Metrics that matter

Every evaluation metric compresses a complex judgment into a single number. The choice of metric determines what you optimize for and what failures you tolerate. A faithfulness metric ignores fluency. A BLEU score rewards lexical overlap regardless of factual accuracy. Picking the wrong metric is worse than not evaluating — it creates a false sense of quality while the system fails on dimensions you aren't measuring.

Classification metrics for structured outputs

When an LLM produces structured outputs — sentiment labels, intent classification, entity extraction, yes/no decisions — standard classification metrics apply directly.

Precision: TP / (TP + FP). Of everything the model labeled positive, what fraction was actually positive? High precision means few false alarms.
Recall: TP / (TP + FN). Of everything that was actually positive, what fraction did the model find? High recall means few missed cases.
F1: 2 * precision * recall / (precision + recall). Harmonic mean of precision and recall. Penalizes imbalance — a model with 99% precision and 1% recall gets F1 = 0.02.
Macro F1: Unweighted average of per-class F1 scores. Treats rare classes equally with common ones.
Micro F1: Equivalent to accuracy when classes are mutually exclusive.

from sklearn.metrics import classification_report, precision_recall_fscore_support
import json
from openai import OpenAI

client = OpenAI()

test_cases = [
    {"text": "I love this product!", "true_label": "positive"},
    {"text": "Terrible experience, want a refund", "true_label": "negative"},
    {"text": "It arrived on Tuesday", "true_label": "neutral"},
    {"text": "Best purchase I've made all year", "true_label": "positive"},
    {"text": "Completely broken, doesn't work", "true_label": "negative"},
    {"text": "The package was blue", "true_label": "neutral"},
]

predictions = []
for case in test_cases:
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Classify sentiment as positive, negative, or neutral. Reply with only the label."},
            {"role": "user", "content": case["text"]}
        ],
        temperature=0.0
    )
    predictions.append(r.choices[0].message.content.strip().lower())

true_labels = [c["true_label"] for c in test_cases]
print(classification_report(true_labels, predictions, digits=3))

precision, recall, f1, _ = precision_recall_fscore_support(
    true_labels, predictions, average="macro"
)
print(f"Macro P/R/F1: {precision:.3f} / {recall:.3f} / {f1:.3f}")

For extraction tasks (NER, slot filling), compute metrics at the entity level: a predicted entity matches if both the span boundaries and the label are correct (strict match) or if spans overlap (partial match). GPT-4o achieves ~0.89 strict-match F1 on CoNLL-2003 NER zero-shot (Wang et al. 2023).

Generation metrics and their limitations

Traditional generation metrics compare a candidate output against one or more reference strings:

BLEU (Papineni et al. 2002): Geometric mean of n-gram precisions (1-4 grams) with a brevity penalty. Originally designed for machine translation. Range: 0-1 (often reported as 0-100). A BLEU score of 0.30+ is considered good for MT.
ROUGE-L (Lin 2004): Longest common subsequence between candidate and reference, normalized by reference length. Designed for summarization. Captures word order better than BLEU.
BERTScore (Zhang et al. 2020): Computes cosine similarity between contextual embeddings (BERT) of candidate and reference tokens, then takes a greedy alignment. Correlates with human judgment at r=0.73 on WMT17, compared to BLEU's r=0.25.

from evaluate import load
import numpy as np

bleu_metric = load("bleu")
rouge_metric = load("rouge")
bertscore_metric = load("bertscore")

references = ["The cat sat on the mat."]
candidates_good = ["A cat was sitting on the mat."]
candidates_bad = ["The dog ran through the park."]

bleu_good = bleu_metric.compute(predictions=candidates_good, references=[references])
bleu_bad = bleu_metric.compute(predictions=candidates_bad, references=[references])
print(f"BLEU (good): {bleu_good['bleu']:.3f}")  # ~0.35
print(f"BLEU (bad):  {bleu_bad['bleu']:.3f}")   # ~0.05

rouge_good = rouge_metric.compute(predictions=candidates_good, references=references)
rouge_bad = rouge_metric.compute(predictions=candidates_bad, references=references)
print(f"ROUGE-L (good): {rouge_good['rougeL']:.3f}")  # ~0.70
print(f"ROUGE-L (bad):  {rouge_bad['rougeL']:.3f}")   # ~0.20

bert_good = bertscore_metric.compute(
    predictions=candidates_good, references=references, lang="en"
)
bert_bad = bertscore_metric.compute(
    predictions=candidates_bad, references=references, lang="en"
)
print(f"BERTScore F1 (good): {np.mean(bert_good['f1']):.3f}")  # ~0.95
print(f"BERTScore F1 (bad):  {np.mean(bert_bad['f1']):.3f}")   # ~0.85

The critical limitation: these metrics require reference answers. For open-ended generation (creative writing, complex reasoning, multi-step explanations), there is no single correct reference. BLEU and ROUGE correlate poorly with human judgment in these settings — Pearson r of 0.25-0.40 on summarization tasks (Fabbri et al. 2021). BERTScore improves to r=0.50-0.73 depending on the domain but still misses factual errors (a fluent, well-structured hallucination scores high on BERTScore).

Retrieval metrics for RAG

RAG systems have a retrieval stage that selects context chunks before generation. Retrieval quality directly bounds generation quality — if the right information isn't retrieved, the generator cannot produce a correct answer regardless of its capability.

Recall@k: Fraction of relevant documents in the top-k retrieved results. recall@5 = |relevant ∩ retrieved_top5| / |relevant|. The most important retrieval metric for RAG — you need the right context to be in the window.
Precision@k: Fraction of top-k results that are relevant. precision@5 = |relevant ∩ retrieved_top5| / k. Matters when context window is expensive (more irrelevant chunks = more noise for the generator).
MRR (Mean Reciprocal Rank): 1/rank_of_first_relevant_result, averaged across queries. MRR=1.0 means the first result is always relevant.
NDCG@k (Normalized Discounted Cumulative Gain): Accounts for graded relevance (not just binary) and position. DCG@k = sum_{i=1}^{k} (2^{rel_i} - 1) / log2(i+1). Normalized by the ideal ranking's DCG. Range: 0-1.

import numpy as np

def recall_at_k(relevant_ids, retrieved_ids, k):
    retrieved_top_k = set(retrieved_ids[:k])
    relevant_set = set(relevant_ids)
    return len(retrieved_top_k & relevant_set) / len(relevant_set)

def mrr(relevant_ids, retrieved_ids):
    relevant_set = set(relevant_ids)
    for i, doc_id in enumerate(retrieved_ids):
        if doc_id in relevant_set:
            return 1.0 / (i + 1)
    return 0.0

def ndcg_at_k(relevance_scores, k):
    dcg = sum(
        (2**rel - 1) / np.log2(i + 2)
        for i, rel in enumerate(relevance_scores[:k])
    )
    ideal = sorted(relevance_scores, reverse=True)[:k]
    idcg = sum(
        (2**rel - 1) / np.log2(i + 2)
        for i, rel in enumerate(ideal)
    )
    return dcg / idcg if idcg > 0 else 0.0

relevant = ["doc_3", "doc_7", "doc_12"]
retrieved = ["doc_1", "doc_3", "doc_5", "doc_7", "doc_9", "doc_12", "doc_15"]

print(f"Recall@3: {recall_at_k(relevant, retrieved, 3):.3f}")   # 0.333
print(f"Recall@5: {recall_at_k(relevant, retrieved, 5):.3f}")   # 0.667
print(f"Recall@7: {recall_at_k(relevant, retrieved, 7):.3f}")   # 1.000
print(f"MRR: {mrr(relevant, retrieved):.3f}")                     # 0.500

graded_relevance = [0, 2, 0, 2, 0, 1, 0]  # per retrieved doc
print(f"NDCG@5: {ndcg_at_k(graded_relevance, 5):.3f}")          # ~0.72

For production RAG systems, target recall@5 >= 0.85 and MRR >= 0.60. Below these thresholds, the generator is missing critical context on >15% of queries.

Faithfulness, relevance, and coherence

For open-ended generation where reference-based metrics fail, you need reference-free metrics that evaluate intrinsic qualities:

Faithfulness/Groundedness: Is every claim in the output supported by the provided context? Measured by decomposing the output into atomic claims and checking each against the source. A faithfulness score of 0.90 means 10% of claims are hallucinated.
Answer relevance: Does the output actually address the user's question? An output can be faithful (everything it says is true) but irrelevant (it answers a different question).
Coherence: Is the output logically structured and internally consistent? Sentences follow a clear progression, no contradictions, appropriate transitions.
Completeness: Does the output cover all aspects of the question? Relevant but incomplete answers miss key information.

from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    GEval
)
from deepeval.test_case import LLMTestCase

faithfulness = FaithfulnessMetric(threshold=0.8, model="gpt-4o")
relevance = AnswerRelevancyMetric(threshold=0.7, model="gpt-4o")

coherence = GEval(
    name="Coherence",
    criteria="Evaluate whether the output is logically structured with clear transitions between ideas.",
    evaluation_params=["actual_output"],
    model="gpt-4o",
    threshold=0.7
)

test_case = LLMTestCase(
    input="What causes inflation?",
    actual_output="Inflation occurs when the general price level rises. Key causes include demand-pull (excess demand), cost-push (rising production costs), and monetary expansion (increasing money supply faster than GDP growth).",
    retrieval_context=[
        "Inflation is a sustained increase in the general price level. Demand-pull inflation happens when aggregate demand exceeds supply. Cost-push inflation occurs when production costs rise. Monetary inflation results from excess money supply growth."
    ]
)

faithfulness.measure(test_case)
relevance.measure(test_case)
coherence.measure(test_case)

print(f"Faithfulness: {faithfulness.score:.2f}")  # ~0.95
print(f"Relevance: {relevance.score:.2f}")         # ~0.90
print(f"Coherence: {coherence.score:.2f}")         # ~0.85

Choosing metrics by use case

Different application types demand different metric combinations:

Chatbot/QA: Faithfulness (primary), answer relevance, recall@5 for retrieval. Tolerance for hallucination: near zero.
Summarization: ROUGE-L for regression testing, faithfulness for factual accuracy, completeness for coverage. BERTScore as a sanity check.
Code generation: pass@k (functional correctness via test execution), exact match for simple outputs, compilation rate as a floor metric.
Classification/extraction: Precision, recall, F1 per class. Macro F1 as the aggregate.
Creative writing: Coherence, relevance, human preference ranking. No faithfulness requirement.

The cheapest metric that correlates >0.70 with human judgment on your task distribution is the right one for CI. Use more expensive metrics (LLM-as-judge, human review) for periodic audits and calibration.

← Previous