Author(s): Ayoub Nainia

Originally published on Towards AI.

Large Language Models can now summarize research papers, analyze data, and even draft academic arguments. Yet behind the flood of progress reports and leaderboard charts, one question remains stubbornly neglected: How do we actually know if these models understand anything?

In my previous article, Production RAG: The Chunking, Retrieval, and Evaluation Strategies That Actually Work, we explored the operational challenges of deploying language models at scale. Today, we want to address an even more fundamental problem: Our evaluation metrics are fundamentally broken.

The uncomfortable truth is that most LLM evaluation relies on metrics designed for entirely different tasks, decades before transformers existed. BLEU scores, originally created for machine translation in 2002, now judge everything from creative writing to code generation. ROUGE metrics, built for document summarization, evaluate conversational AI. It’s like using a thermometer to measure distance, which is technically possible, but profoundly misguided.

This isn’t just an academic debate. The metrics we use to evaluate AI systems directly shape how they’re built, trained, and deployed. When we optimize for the wrong measures, we get systems that excel at gaming tests rather than solving real problems. The consequences ripple through every AI application touching human lives: from medical diagnosis assistants that sound confident but lack understanding, to educational tools that prioritize keyword matching over conceptual clarity.

The field has become trapped in what wecall “metric theater”: the performance of rigorous evaluation without actual rigor. We’ve built an entire infrastructure around measurements that fundamentally misunderstand what makes language models useful, safe, and trustworthy.

1. The Great Evaluation Illusion

Consider this scenario: You ask GPT-4 to explain quantum mechanics to a child, and it responds with a delightful analogy about dancing particles. Meanwhile, another model spills out a textbook definition verbatim.

Which performs better according to BLEU score?

The textbook copy wins every time.

This isn’t a hypothetical problem. Research from Kiela et al. (2021) in “Dynabench: Rethinking Benchmarking in NLP” revealed that models optimized for traditional metrics often fail catastrophically on tasks requiring genuine understanding. They found that adversarial examples designed to fool BLEU and ROUGE scores could achieve near-perfect metric scores while producing completely nonsensical outputs. The issue runs deeper than poor correlation with human judgment. Zhang et al. (2019), in their foundational BERTScore paper, demonstrated that traditional n-gram metrics fail to capture semantic equivalence in over 60% of paraphrase cases. Yet these same metrics continue to drive model development, research funding, and commercial deployment decisions.

But how did we get here?

The story of broken evaluation metrics is also the story of a field that grew too fast for its measurement infrastructure.

2. The Historical Accident: How Translation Metrics Colonized AI

The dominance of BLEU and ROUGE in modern LLM evaluation represents one of the most consequential accidents in AI history. To understand why these metrics persist despite their obvious limitations, we need to trace their origins and see how they escaped their intended domains.

2.1 The Birth of Automatic Evaluation

Before 2002, evaluating machine translation was expensive and subjective. Human evaluators would read translated texts and provide quality ratings, which is a process that was slow, costly, and difficult to standardize across different languages and cultural contexts. Papineni et al. faced a practical problem: IBM’s statistical machine translation systems needed rapid, automatic evaluation to enable iterative improvement.

Their solution was elegantly simple: if a machine translation shared many word sequences (n-grams) with professional human translations, it was probably good. BLEU was born from this pragmatic need, with clear limitations acknowledged by its creators. They explicitly noted that BLEU measured only one aspect of translation quality and should be used alongside human evaluation.

The metric was never intended to be a general measure of language understanding. It was a specific tool for a specific task in a specific research context.

2.2 The Great Metric Migration

What happened next reveals how scientific infrastructure can ossify around convenience rather than validity. As NLP research accelerated in the 2000s and 2010s, researchers needed quick ways to compare systems across different tasks. BLEU offered several attractive properties:

• Deterministic: The same inputs always produced the same scores
• Fast: No need for expensive human evaluation
• Familiar: The community already understood its mathematical properties
• Comparative: Easy to rank systems on leaderboards

Lin (2004) introduced ROUGE for summarization evaluation, following BLEU’s basic philosophy but focusing on recall rather than precision. The pattern was set: take n-gram overlap metrics, adjust them slightly for different tasks, and claim rigorous evaluation.

By 2010, these metrics had spread throughout NLP. Question answering systems were evaluated with BLEU. Dialogue systems used ROUGE. Creative writing tasks employed exact match scoring. The field had developed what Reiter (2018) called “metric fixation”, which is the belief that any quantitative measure was better than qualitative assessment.

2.3 The Transformer Revolution and Metric Lag

The introduction of attention mechanisms and transformer architectures in 2017 created a new problem: models that could generate human-like text in ways that completely broke traditional evaluation assumptions.

GPT-2’s release in 2019 demonstrated the absurdity clearly. The model could write compelling fiction, answer complex questions, and engage in nuanced dialogue, yet scored poorly on many traditional benchmarks because it didn’t match reference texts word-for-word. Conversely, models could achieve high BLEU scores by memorizing training data and regurgitating fragments, despite showing no genuine understanding.

Rogers et al. (2021) documented this disconnect in “Changing the World by Changing the Data”, showing how evaluation metrics failed to capture the qualitative leap in model capabilities. The field was measuring 2023 technology with 2002 metrics and making deployment decisions based on these mismatched assessments.

2.4 Why Surface Metrics Seduce Us ?

BLEU and ROUGE persist not because they work well, but because they feel scientific. They produce clean numbers. They’re deterministic and reproducible. In a field hungry for objective measures of progress, they offer the illusion of mathematical precision.

This seduction operates on multiple levels: psychological, institutional, and economic, creating a self-reinforcing system that’s remarkably resistant to change.

2.5 The Psychology of False Precision

We, human beings, are naturally drawn to quantitative measures, especially in contexts where qualitative assessment feels subjective or unreliable. Kahneman and Tversky’s research on cognitive biases shows that people consistently overweight precise-seeming information, even when it’s less accurate than imprecise but valid data.

BLEU scores feel precise because they’re computed to three decimal places. A score of 0.847 seems more trustworthy than a human evaluator saying “this translation is pretty good”. This precision bias leads researchers and practitioners to treat BLEU scores as more reliable than they actually are.

The problem is compounded by what psychologists call anchoring: once a numerical score is established, it becomes a reference point that’s difficult to abandon. Teams begin optimizing for BLEU improvements of 0.01, treating these changes as meaningful progress even when they’re statistically insignificant or negatively correlated with actual quality.

2.6 Institutional Incentives and Publication Pressure

Academic publishing has created perverse incentives around evaluation metrics. Conference review processes favor papers with clear numerical comparisons over more nuanced qualitative analyses. A paper showing “3.2 BLEU point improvement” is easier to review and more likely to be accepted than one arguing for fundamental changes in evaluation methodology.

Funding agencies compound this problem by requiring quantitative success metrics in grant proposals. Researchers learn to frame their work in terms of metric improvements because that’s what gets funded. Industry labs perpetuate the cycle by publishing leaderboard results that prioritize easily comparable numbers over meaningful assessment.

This creates what Goodhart’s Law predicts:

When a measure becomes a target, it ceases to be a good measure.

The field optimizes for metric improvements rather than actual progress, leading to increasingly sophisticated ways of gaming evaluation systems.

2.7 The Economics of Evaluation

From a business perspective, automatic metrics offer compelling economic advantages:

• Scale: Evaluating millions of model outputs manually would cost prohibitive amounts
• Speed: Rapid iteration requires rapid evaluation
• Consistency: Human evaluators are expensive and show inter-annotator disagreement
• Legibility: Investors and executives understand numbers better than qualitative reports

These economic pressures create what Winner (1980) called “technological momentum”. The tendency for established technological approaches to persist even when better alternatives exist, simply because of the infrastructure built around them.

Evaluation platforms like Papers with Code and Hugging Face’s leaderboards have invested heavily in automatic metric infrastructure. Changing evaluation standards would require rebuilding these systems and recomputing historical results, which happens to be a coordination problem that no single actor wants to solve alone.

3. BLEU: The Accidental Dictator

BLEU (Bilingual Evaluation Understudy) operates on a deceptively simple principle: count matching n-grams between generated and reference texts. The more overlapping word sequences, the higher the score. Originally designed by Papineni et al. (2002) for machine translation, BLEU measures precision, what fraction of the model’s words appear in the reference.

# Simplified BLEU calculation showing the core problem
def simple_bleu(candidate, reference):
 candidate_words = candidate.lower().split()
 reference_words = set(reference.lower().split())
 
 matches = sum(1 for word in candidate_words if word in reference_words)
 return matches / len(candidate_words) if candidate_words else 0

# Examples that expose BLEU's blindness
reference = "The cat sat on the mat"
candidate1 = "The cat sat on the mat" # Perfect score: 1.0
candidate2 = "Cat the mat sat on" # Still high: 0.83
candidate3 = "The feline rested on the rug" # Low score: 0.33

print(f"Candidate 1 (identical): {simple_bleu(candidate1, reference):.2f}")
print(f"Candidate 2 (scrambled): {simple_bleu(candidate2, reference):.2f}") 
print(f"Candidate 3 (paraphrase): {simple_bleu(candidate3, reference):.2f}")

The problem becomes immediately apparent: BLEU treats a grammatical paraphrase as worse than scrambled word salad. This isn’t a bug, it’s the inevitable result of prioritizing surface-level word matching over semantic understanding.

3.1 BLEU’s Systematic Biases

BLEU’s focus on precision creates several systematic biases that become more problematic as language models become more sophisticated:

• Length Bias: BLEU’s brevity penalty was designed to prevent systems from gaming precision by generating very short outputs. However, this penalty often overcompensates, favoring verbose responses that include more potential matches. A concise, elegant answer scores lower than a rambling one that happens to repeat reference words.
• Vocabulary Bias: Models that use the same vocabulary as reference texts score higher, regardless of semantic accuracy. This creates incentives for models to memorize training data rather than develop genuine understanding. Koehn (2004) showed that BLEU scores could be artificially inflated by 15–20 points simply by matching reference vocabulary choices.
• Genre Bias: BLEU performs differently across text types. Reiter (2018) demonstrated that BLEU scores for the same semantic content varied by over 30 points depending on whether the text was formatted as formal prose, dialogue, or bullet points. This makes cross-domain comparisons meaningless.
• Cultural Bias: In multilingual contexts, BLEU systematically favors translations that match Western linguistic patterns. Post (2018) found that BLEU scores for morphologically rich languages (like Finnish or Turkish) showed poor correlation with human judgment compared to English, creating systematic disadvantages for non-English AI systems.

3.2 The Creativity Paradox

Perhaps BLEU’s most damaging bias is against creativity and originality. Models that demonstrate understanding by explaining concepts in novel ways, using metaphors, or adapting to specific audiences are systematically penalized.

Consider these responses to “Explain machine learning to a 5-year-old”:

• Response A (High BLEU): “Machine learning is a method of data analysis that automates analytical model building. It uses algorithms that iteratively learn from data, allowing computers to find hidden insights.”
• Response B (Low BLEU): “Imagine teaching your pet robot to recognize your toys. You show it lots of toy cars and say ‘car,’ then lots of toy dinosaurs and say ‘dinosaur.’ After seeing many examples, the robot learns to tell the difference and can identify new toys you’ve never shown it before!”

Any parent would prefer Response B. It demonstrates genuine understanding by translating complex concepts into age-appropriate language. Yet BLEU systematically ranks Response A higher because it shares more vocabulary with typical reference texts about machine learning.

This creativity paradox becomes more severe as models become more capable. The most impressive model behaviors: genuine understanding, contextual adaptation, and creative problem-solving are exactly what traditional metrics punish most severely.

4. ROUGE: The Recall Trap

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) was designed for automatic summarization evaluation. Unlike BLEU’s precision focus, ROUGE emphasizes recall: how much of the reference content appears in the generated text.

Lin (2004) introduced multiple ROUGE variants:

• ROUGE-1: Unigram (single word) overlap
• ROUGE-2: Bigram (two-word sequence) overlap
• ROUGE-L: Longest Common Subsequence overlap
• ROUGE-S: Skip-bigram overlap (allowing gaps)

The intuition seems reasonable: good summaries should cover the important content from source documents. If a summary includes most of the key words and phrases from the reference, it’s probably capturing the essential information.

4.1 The Reference Dependency Problem

ROUGE’s fundamental flaw lies in its dependence on reference summaries that may not represent the only valid way to summarize a document. Kryscinski et al. (2019) showed that ROUGE scores varied by up to 40 points depending on which human-written reference summary was used for evaluation, even when all references were high-quality.

This variability isn’t just a measurement noise problem. It reflects a deeper issue: summarization is inherently subjective. Different readers prioritize different aspects of a document based on their background, goals, and context. A financial analyst and a policy researcher will write very different summaries of the same economic report, both perfectly valid for their purposes.

ROUGE assumes there’s a “correct” summary and penalizes deviations from it. This creates several problematic dynamics:

• Style Conformity: Models learn to mimic the linguistic patterns of reference summaries rather than developing their own coherent voice. Zhang et al. (2020) found that models optimized for ROUGE scores became increasingly similar in their output style, reducing diversity and creativity.
• Content Bias: ROUGE inherently favors certain types of content based on reference summary choices. If reference summaries emphasize quantitative information, models learn to prioritize numbers over qualitative insights. If references use formal language, conversational summaries are penalized regardless of clarity or appropriateness.
• Length Gaming: ROUGE’s recall focus creates incentives to generate longer summaries that have more opportunities to match reference content. This directly contradicts the goal of summarization, which is to be concise and selective.

4.2 The Extractive Bias

ROUGE systematically favors extractive approaches (selecting sentences from the source) over abstractive approaches (generating new sentences that capture the meaning). This bias has profoundly shaped the development of summarization systems.

See et al. (2017) demonstrated that simple extractive baselines often outperformed sophisticated neural summarization models on ROUGE metrics, despite producing stilted, incoherent outputs. This led to years of research focused on improving extractive approaches rather than developing models that could truly understand and reformulate content.

The extractive bias becomes particularly problematic for cross-domain summarization. Hardy et al. (2019) showed that models trained to optimize ROUGE scores on news articles performed poorly when applied to scientific papers, legal documents, or creative writing, not because they lacked capability, but because they had learned to copy rather than comprehend.

4.3 The Paraphrase Penalty

Like BLEU, ROUGE systematically penalizes paraphrasing and semantic equivalence. Consider these two summaries of a climate change article:

• Summary A: “Global temperatures have increased by 1.1 degrees Celsius since pre-industrial times. Scientists warn that current emission trends will lead to catastrophic warming”
• Summary B: “Earth has warmed by over one degree since the 1800s. Researchers caution that ongoing carbon pollution could trigger devastating climate impacts”

Summary B demonstrates sophisticated understanding by using varied vocabulary while preserving meaning. Yet ROUGE scores it lower because it doesn’t match the exact words and phrases in typical reference summaries.

This paraphrase penalty becomes more severe as models become more sophisticated at natural language generation. The most human-like model behaviors: using varied vocabulary, adapting tone for different audiences, employing metaphors and analogies, are systematically discouraged by ROUGE optimization.

5. Exact Match: The Bluntest Instrument

Perhaps the most primitive metric still in widespread use is Exact Match, a binary score where the generated text either perfectly matches the reference or receives zero points.

def exact_match_score(predictions, references):
 matches = sum(1 for pred, ref in zip(predictions, references) 
 if pred.strip().lower() == ref.strip().lower())
 return matches / len(predictions)

# Examples showing exact match's brittleness
predictions = [
 "Paris",
 "The capital of France is Paris",
 "Paris, France",
 "paris" # Different capitalization
]

references = ["Paris", "Paris", "Paris", "Paris"]

print(f"Exact Match Score: {exact_match_score(predictions, references):.2f}")
# Result: 0.50 (only 2/4 correct despite all being factually accurate)

Exact match fails catastrophically in any domain requiring nuanced understanding. A model that answers “The capital of France is Paris” to the question “What is France’s capital?” receives the same score as one that outputs random gibberish.

5.1 The False Binary

Exact match represents evaluation at its most reductionist: complex language understanding reduced to a single binary decision. This approach ignores several crucial dimensions of response quality:

• Semantic Equivalence: “car” and “automobile” represent the same concept but receive different scores. This problem becomes severe in multilingual contexts where multiple valid transliterations exist.
• Contextual Appropriateness: A formal response and a casual response might both be correct but receive different scores based on arbitrary reference text choices.
• Partial Credit: A response that’s 90% correct receives the same score as complete nonsense. This makes it impossible to distinguish between minor errors and fundamental failures.
• Cultural Variation: Different regions use different terms for the same concepts. A system trained on American English might say “elevator” while a reference uses “lift,” resulting in a false negative.

5.2 The Question Answering Catastrophe

Exact match’s most damaging application has been in question answering evaluation. Rajpurkar et al. (2016) used exact match as a primary metric for the influential SQuAD dataset, creating a generation of QA systems optimized for matching reference text rather than providing helpful answers.

The consequences were predictable and severe:

• Answer Engineering: Models learned to generate responses that matched expected answer formats rather than providing the most helpful information. Jia and Liang (2017) showed that adding distracting sentences to contexts could dramatically reduce exact match scores even when models provided correct information in slightly different formats.
• Brittleness: Systems that scored 90%+ on exact match evaluation often failed completely when deployed in real-world contexts where answer formats varied. Karpukhin et al. (2020) found that many high-scoring QA models were essentially sophisticated pattern matchers with minimal transfer capability.
• User Experience Degradation: Optimizing for exact match led to systems that provided technically correct but unhelpful responses. A model might answer “What year was Obama born?” with “1961” instead of the more helpful “Barack Obama was born in 1961”.

5.3 The Evaluation Gaming Problem

Exact match’s binary nature makes it particularly susceptible to gaming through data engineering rather than model improvement. Kaushik and Lipton (2018) documented several problematic practices:

• Reference Engineering: Adjusting reference answers to match model outputs rather than improving model capabilities. This creates an illusion of progress while degrading evaluation quality.
• Format Standardization: Pre-processing both predictions and references to match arbitrary formatting conventions, eliminating important distinctions in model behavior.
• Cherry-Picking: Selecting evaluation sets where exact match happens to align with meaningful quality differences, then generalizing these results inappropriately.

6. BERT Score: The Semantic Mirage

Recognizing the limitations of surface-level metrics, researchers developed BERTScore by leveraging BERT’s contextual embeddings to measure semantic similarity rather than lexical overlap.

Zhang et al. (2019) introduced BERTScore as a learned metric that could capture semantic equivalence invisible to n-gram approaches. Instead of counting word matches, BERTScore computes contextual embeddings for each token and measures cosine similarity between candidate and reference representations.

# Conceptual illustration of BERTScore approach
def conceptual_bertscore(candidate, reference):
 # Get contextual embeddings for each token
 candidate_embeddings = bert_model.encode(candidate)
 reference_embeddings = bert_model.encode(reference)
 
 # Find best alignment between tokens
 similarities = compute_cosine_similarities(candidate_embeddings, reference_embeddings)
 
 # Compute precision, recall, F1 based on best matches
 precision = mean(max_similarity_per_candidate_token)
 recall = mean(max_similarity_per_reference_token)
 f1 = harmonic_mean(precision, recall)
 
 return f1

BERTScore represents genuine progress. It can recognize that “car” and “automobile” are semantically similar in ways that BLEU cannot. However, it introduces new problems that are more subtle but equally problematic.

6.1 The Model Bias Problem

BERTScore’s most fundamental flaw is its dependence on BERT’s learned representations. These embeddings encode not just semantic relationships but also the biases, limitations, and artifacts of BERT’s training process.

Tenney et al. (2019) showed that BERT embeddings systematically favor certain linguistic patterns over others:

• Frequency Bias: Common words and phrases receive higher similarity scores than rare but accurate alternatives. A model using sophisticated vocabulary is penalized compared to one using simple, frequent terms.
• Training Data Bias: Models that generate text similar to BERT’s training corpus (primarily Wikipedia and BookCorpus) receive artificially inflated scores. This creates circular validation where models are evaluated based on similarity to the same data they were potentially trained on.
• Architectural Bias: Models using transformer architectures similar to BERT show higher BERTScore correlation with human judgment than models using different architectures, regardless of actual quality. Rogers et al. (2020) demonstrated that this bias could account for 10–15% of observed score differences.

6.2 The Hallucination Blindness

BERTScore’s focus on semantic similarity makes it particularly vulnerable to confident hallucinations. Deutsch et al. (2022) found that BERTScore could assign high similarity scores to outputs containing factual errors, as long as they maintained semantic coherence.

Consider these responses to “What is the capital of Australia?”:

• Response A: “The capital of Australia is Sydney, the largest and most famous city”
• Response B: “Australia’s capital is Canberra, located in the Australian Capital Territory”

Response A is factually incorrect but semantically coherent and confident. BERTScore often rates it highly because the embedding space doesn’t distinguish between confident falsehoods and accurate information. Response B, while correct, might score lower if the reference text uses different terminology.

This hallucination blindness becomes particularly dangerous in high-stakes applications where factual accuracy matters more than semantic fluency. Maynez et al. (2020) showed that summarization systems optimized for BERTScore often generated fluent but factually incorrect summaries that were difficult for users to detect.

6.3 The Context Collapse Problem

BERTScore operates at the token level, computing similarity between individual words or subwords. This approach misses crucial document-level patterns that determine overall quality:

• Logical Structure: A response might use semantically similar words but present them in an illogical order. BERTScore captures local similarity while missing global incoherence.
• Discourse Coherence: Scientific or technical writing often depends on precise logical relationships between concepts. BERTScore can miss these relationships while focusing on lexical similarity.
• Pragmatic Appropriateness: The same semantic content might be appropriate in one context and inappropriate in another. BERTScore cannot distinguish between contextually appropriate and inappropriate responses.
• Narrative Flow: Creative writing and storytelling depend on temporal and causal relationships that token-level similarity cannot capture. A story with all the right elements in the wrong order scores similarly to a well-structured narrative.

6.4 The Reference Quality Dependency

Like all reference-based metrics, BERTScore inherits the quality limitations of its reference texts. Bhandari et al. (2020) showed that BERTScore variance could exceed 20 points depending on reference quality, even for the same candidate text.

This dependency creates several problems:

• Reference Bias Amplification: Poor or biased reference texts lead to systematically biased evaluations. Unlike surface metrics where bias is obvious, BERTScore’s semantic approach can hide bias behind apparent sophistication.
• Domain Sensitivity: BERTScore performance varies dramatically across domains based on how well BERT represents domain-specific semantics. Kenton and Toutanova (2019) found that BERTScore correlation with human judgment dropped by 40% or more in specialized domains like legal or medical text.
• Multi-Reference Challenges: When multiple reference texts are available, BERTScore can produce inconsistent results based on which reference is chosen. This inconsistency is often hidden because BERTScore appears more principled than surface metrics.

7. The Real Problem: Optimizing for the Wrong Thing

The fundamental issue isn’t that these metrics are poorly implemented, it’s that they measure the wrong things entirely. Traditional evaluation metrics embody assumptions about language and understanding that predate modern AI capabilities by decades.

Real-world language use is goal-oriented. People communicate to inform, persuade, entertain, educate, or solve problems. The quality of a response depends on how well it achieves these communicative goals, not on how closely it matches a predetermined text. Each response serves different user needs and contexts. Traditional metrics would rank them based on similarity to a reference text rather than appropriateness for the user’s situation.

Traditional metrics conflate two distinct capabilities: understanding (comprehending input and context) and generation (producing appropriate output). A model might understand perfectly but express its understanding in ways that don’t match reference texts.

8. Building Better Evaluation: A Comprehensive Framework

The solution isn’t to abandon quantitative evaluation, but to develop measurement systems that actually assess what we care about: understanding, reasoning, appropriate response generation, and user value creation.

8.1 Principle 1: Multi-Dimensional Assessment

Instead of seeking a single metric to rule them all, we need evaluation frameworks that assess multiple dimensions of quality simultaneously. Real language understanding is inherently multi-faceted, requiring assessment across several independent dimensions:

• Factual Accuracy: Does the response contain verifiable truth? This requires integration with knowledge bases and fact-checking systems, not just pattern matching against reference texts.
• Semantic Coherence: Are ideas logically connected? This involves discourse analysis, causal reasoning assessment, and logical consistency checking.
• Task Completion: Does the response actually accomplish what was requested? This requires understanding user intent and evaluating goal achievement rather than text similarity.
• Contextual Appropriateness: Is the response suitable for the specific context, audience, and purpose? This involves adaptation assessment and situational reasoning evaluation.
• Communicative Effectiveness: Does the response successfully convey information in a way that serves the user’s needs? This requires user experience evaluation and communication success measurement.

8.2 Principle 2: Dynamic Reference Generation

Static reference texts create artificial constraints on acceptable responses. Instead of comparing against fixed targets, we should evaluate whether responses fall within the distribution of reasonable, high-quality outputs.

• Distributional Evaluation: Generate multiple valid responses to the same prompt and evaluate how well the candidate fits within the distribution of quality responses. This approach acknowledges that many different responses can be equally valid.
• Adaptive References: Use large language models to generate contextually appropriate reference responses based on specific user needs, expertise levels, and communication goals. This allows evaluation to adapt to different contexts rather than assuming one-size-fits-all quality standards.
• Consensus-Based Assessment: Use multiple independent evaluators (both human and AI) to establish consensus about response quality. Responses that consistently receive positive evaluation across diverse assessors are more likely to be genuinely high-quality.

8.3 Principle 3: Task-Adaptive Evaluation

Different tasks require fundamentally different evaluation approaches. A one-size-fits-all metric cannot capture the diverse quality dimensions relevant across different applications.

• Question Answering: Focus on accuracy, completeness, and clarity. Evaluate whether the response correctly answers the question with appropriate detail for the user’s expertise level.
• Creative Writing: Assess originality, engagement, stylistic appropriateness, and emotional resonance. Traditional metrics actively work against these qualities.
• Summarization: Evaluate content coverage, conciseness, coherence, and faithfulness to source material. Balance comprehensive coverage with readable presentation.
• Dialogue: Assess contextual appropriateness, conversation flow, empathy, and goal achievement. Consider multi-turn context and relationship building.
• Educational Content: Focus on pedagogical effectiveness, appropriate difficulty level, engaging presentation, and learning objective achievement.

8.4 Principle 4: Human-AI Collaborative Assessment

The future of evaluation lies in intelligent collaboration between human judgment and AI capabilities. Neither humans nor AI systems alone can provide comprehensive evaluation at scale, but their complementary strengths can create robust assessment systems.

• Hierarchical Evaluation: Use AI for initial screening and pattern detection, then employ human evaluators for nuanced judgment and edge cases. This approach leverages AI efficiency with human insight.
• Consensus Building: Combine multiple AI evaluators with diverse architectures and training backgrounds, then validate consensus judgments with human oversight. This reduces individual system bias while maintaining scalability.
• Active Learning for Evaluation: Use human feedback to continuously improve AI evaluation systems. When AI evaluators disagree or express low confidence, flag cases for human review and learn from the results.
• Domain Expertise Integration: Include subject matter experts in evaluation for specialized domains. Medical, legal, and technical content requires domain knowledge that general-purpose evaluators cannot provide.

8.5 Principle 5: Continuous Learning and Adaptation

Evaluation systems should improve over time based on accumulated evidence about what constitutes quality in different contexts. Static evaluation frameworks cannot keep pace with rapidly evolving AI capabilities and changing user needs.

• Feedback Integration: Collect user feedback about AI system performance and use it to improve evaluation metrics. Users are the ultimate judges of system quality.
• Longitudinal Analysis: Track how evaluation metrics correlate with real-world outcomes over time. Adjust metrics based on their predictive validity for actual user satisfaction and task success.
• Cross-Domain Validation: Test evaluation approaches across different domains and applications to ensure generalizability and identify systematic biases.
• Evolutionary Evaluation: Allow evaluation criteria to evolve as AI capabilities advance. What constitutes good performance changes as the possible space of behaviors expands.

Conclusion: Towards Honest Measurement

The current state of LLM evaluation represents more than a technical problem. It’s a crisis of scientific integrity. We’ve built an entire field on the foundation of metrics that fundamentally misunderstand what makes language models useful, safe, and trustworthy.

The path forward requires more than new metrics; it demands a fundamental shift in how we think about measurement in AI. Instead of seeking simple proxies for complex phenomena, we need evaluation systems that embrace the inherent complexity of language understanding and generation.

This transformation won’t be easy. It requires:

• Intellectual Humility: Acknowledging that our current measurement approaches are inadequate and that better evaluation is genuinely difficult.
• Investment in Infrastructure: Building evaluation systems requires significant resources and coordination across the research and industry communities.
• Cultural Change: Shifting from metric optimization to genuine capability development requires changes in how we evaluate research, allocate funding, and make deployment decisions.
• User-Centric Focus: Remembering that the ultimate goal of AI systems is to serve human needs, not to achieve high scores on arbitrary metrics.
• Long-Term Thinking: Evaluation improvement is a long-term investment that may not show immediate benefits but is essential for the field’s future.

The stakes couldn’t be higher. As AI systems become more powerful and more integrated into critical aspects of human life, the quality of our evaluation metrics directly impacts human welfare. Poor evaluation doesn’t just hide problems; it actively creates them by optimizing for the wrong objectives.

The good news is that the technical foundations for better evaluation exist. We have the computational resources, the human expertise, and the methodological knowledge needed to build evaluation systems worthy of modern AI capabilities. What we need now is the collective will to abandon comfortable fictions and embrace the harder work of honest measurement.

The future of AI depends not just on building better models, but on knowing whether they’re actually any good. By fixing how we measure progress, we can guide the field toward building systems that actually serve human needs rather than gaming statistical tests.

The evaluation revolution starts with acknowledging that we can do better. The question isn’t whether we’ll eventually build better evaluation systems, it’s whether we’ll do it before or after the current broken metrics cause more damage to the field and the people it claims to serve.

The next article in this series will explore Constitutional AI and how value-based evaluation frameworks are reshaping our approach to AI alignment and safety. If you’re working on evaluation methodology or have insights about measuring AI system quality, I’d love to hear from you in the comments. The intersection of rigorous evaluation and real-world deployment is where the most important advances in AI will happen.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI