Evaluating RAG

A RAG system fails in two distinct ways: the retriever returns the wrong chunks, or the generator produces a wrong answer from the right chunks. Fixing retrieval failures (better chunking, reranking, hybrid search) is completely different from fixing generation failures (better prompts, stricter grounding instructions, model upgrades). Evaluation must separate these failure modes or you'll apply the wrong fix.

The RAG triad

Es et al. (2023) introduced the RAGAS framework, which decomposes RAG quality into three independent metrics — context relevance, groundedness, and answer relevance. Each measures a different edge of the retrieve-then-generate pipeline:

Context relevance — are the retrieved chunks relevant to the question? Measures retrieval quality. A retriever that returns chunks about "database indexing" when the question asks about "user authentication" fails here regardless of how good the generator is.
Groundedness (faithfulness) — is the generated answer supported by the retrieved context? Measures whether the generator sticks to what it was given. A model that produces correct-sounding text not present in the context is hallucinating.
Answer relevance — does the answer actually address the question? A grounded answer that faithfully summarizes the context but doesn't answer what was asked fails here.

Each metric isolates a specific component. Low context relevance means fix the retriever. Low groundedness means fix the prompt or switch models. Low answer relevance can indicate either — the retriever found tangential context, or the generator went off-topic.

Context precision and recall

Context relevance breaks down further into precision and recall:

Context precision — what fraction of the retrieved chunks are actually relevant? precision = relevant_retrieved / total_retrieved. If you retrieve 10 chunks and only 3 are relevant, precision is 0.3. Low precision wastes context window tokens on irrelevant text and can mislead the generator.
Context recall — what fraction of the necessary information was retrieved? recall = relevant_retrieved / total_relevant. If the corpus contains 5 chunks that answer the question and you retrieved 3 of them, recall is 0.6. Low recall means the answer will be incomplete regardless of generation quality.

These require a ground-truth annotation: for each question, which chunks (or which documents) contain the answer. Building this annotation is the golden test set — discussed below.

Faithfulness scoring

Faithfulness measures whether each claim in the generated answer is supported by the retrieved context. The standard approach:

1. Extract individual claims from the answer

2. For each claim, check whether the context contains supporting evidence

3. faithfulness = supported_claims / total_claims

import anthropic
import json

client = anthropic.Anthropic()

def extract_claims(answer):
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        messages=[{"role": "user", "content": f"""Extract every factual claim from
this answer as a JSON array of strings. Each claim should be a single,
self-contained statement.

Answer: {answer}

Return only the JSON array, no other text."""}]
    )
    return json.loads(response.content[0].text)

def check_claim_support(claim, context):
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=256,
        messages=[{"role": "user", "content": f"""Is the following claim supported
by the provided context? Reply with exactly "supported" or "unsupported".

Claim: {claim}

Context: {context}

Verdict:"""}]
    )
    return response.content[0].text.strip().lower() == "supported"

def score_faithfulness(answer, context):
    claims = extract_claims(answer)
    if not claims:
        return {"score": 1.0, "claims": [], "details": []}

    details = []
    supported = 0
    for claim in claims:
        is_supported = check_claim_support(claim, context)
        details.append({"claim": claim, "supported": is_supported})
        if is_supported:
            supported += 1

    return {
        "score": supported / len(claims),
        "claims": claims,
        "details": details,
    }

This uses Claude as a judge — the same approach RAGAS uses with any LLM. Each claim requires one LLM call, so a 10-claim answer costs 11 calls (1 extraction + 10 verification). At Claude Sonnet pricing, that's roughly $0.02–0.05 per answer evaluated.

Answer relevance scoring

Answer relevance checks whether the response addresses the original question:

def score_answer_relevance(question, answer):
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=256,
        messages=[{"role": "user", "content": f"""Rate how well the answer
addresses the question on a scale of 0 to 1.

- 1.0: The answer directly and completely addresses the question.
- 0.5: The answer partially addresses the question or includes significant
  irrelevant content.
- 0.0: The answer does not address the question at all.

Question: {question}
Answer: {answer}

Return only a number between 0 and 1."""}]
    )
    return float(response.content[0].text.strip())

LLM-as-judge: pitfalls

Using an LLM to evaluate another LLM's output introduces systematic biases:

Position bias — the judge favors the first or last option in a comparison. Zheng et al. (2023) showed that swapping the order of two answers changes the preferred choice 25–30% of the time with GPT-4 as judge. Mitigate by evaluating each answer independently (pointwise) rather than comparing pairs, or by averaging scores across both orderings.
Verbosity bias — longer answers score higher even when the extra content is filler. A 200-word answer that restates the question and adds hedging language ("It's important to note that...") scores higher than a 50-word answer that's correct and complete. Mitigate with explicit rubric instructions: "A concise, correct answer should score higher than a verbose, padded one."
Self-preference — models rate their own outputs higher than outputs from other models. Claude as judge rates Claude-generated answers ~5–10% higher than GPT-generated answers of equivalent quality, and vice versa. Mitigate by using a different model family as judge than the one that generated the answers.
Sycophancy — when the judge sees the context, it's biased toward marking claims as supported. Adding adversarial unsupported claims to the evaluation set measures this: if the judge marks fabricated claims as supported, your faithfulness scores are inflated.

None of these biases are fatal. They mean LLM-as-judge scores are useful for regression testing (did this change make things better or worse?) but unreliable as absolute quality measurements.

Building a golden test set

A golden test set is a curated collection of (question, expected answer, relevant chunks) triples. This is the ground truth that makes evaluation reproducible:

golden_set = [
    {
        "question": "How do I configure HNSW index parameters in pgvector?",
        "expected_answer": "Use CREATE INDEX with hnsw() and specify m and "
                          "ef_construction parameters. Set hnsw.ef_search at "
                          "query time to control recall vs speed.",
        "relevant_doc_ids": ["pgvector-docs", "indexing-guide"],
        "relevant_chunk_indices": [12, 13, 14],
    },
    {
        "question": "What embedding model should I use for code search?",
        "expected_answer": "voyage-code-3 leads code search benchmarks as of "
                          "mid-2026. CodeSage is the main open-source alternative.",
        "relevant_doc_ids": ["embedding-models"],
        "relevant_chunk_indices": [45, 46],
    },
]

Building this set is manual and expensive — someone who understands the corpus must write the questions, identify the correct source chunks, and write reference answers. But it's the only way to measure context recall (you need to know which chunks should be retrieved) and answer correctness (you need to know what the right answer is).

Guidelines for golden set construction:

50–100 questions minimum for reliable aggregate metrics. Fewer than 30 and random variance dominates.
Cover the distribution. Include easy questions (answer is in a single chunk), hard questions (answer requires synthesizing 3+ chunks), adversarial questions (answer is not in the corpus — the system should say "I don't know"), and edge cases (question uses different terminology than the corpus).
Pin the corpus version. The golden set is valid for a specific snapshot of your documents. When the corpus changes significantly, update the relevant chunk indices.
Version the test set. Store it in your repo alongside the code. Changes to the test set are as important as changes to the pipeline — review them with the same rigor.

An eval harness

Combine the metrics into a harness that runs the full pipeline and scores each test case:

import json
from datetime import datetime

def evaluate_rag_pipeline(rag_query_fn, search_fn, golden_set):
    results = []

    for case in golden_set:
        query = case["question"]
        expected_chunks = set(
            (doc_id, idx)
            for doc_id in case["relevant_doc_ids"]
            for idx in case["relevant_chunk_indices"]
        )

        search_results = search_fn(query, top_k=10)
        retrieved_chunks = set(
            (r["doc_id"], r["chunk_index"]) for r in search_results
        )

        relevant_retrieved = retrieved_chunks & expected_chunks
        context_precision = (len(relevant_retrieved) / len(retrieved_chunks)
                            if retrieved_chunks else 0)
        context_recall = (len(relevant_retrieved) / len(expected_chunks)
                         if expected_chunks else 0)

        rag_result = rag_query_fn(query)
        answer = rag_result["answer"]
        context_text = "\n\n".join(r["content"] for r in search_results)

        faithfulness = score_faithfulness(answer, context_text)
        relevance = score_answer_relevance(query, answer)

        results.append({
            "question": query,
            "answer": answer,
            "context_precision": context_precision,
            "context_recall": context_recall,
            "faithfulness": faithfulness["score"],
            "answer_relevance": relevance,
            "faithfulness_details": faithfulness["details"],
        })

    avg = lambda key: sum(r[key] for r in results) / len(results)

    summary = {
        "timestamp": datetime.now().isoformat(),
        "num_cases": len(results),
        "avg_context_precision": avg("context_precision"),
        "avg_context_recall": avg("context_recall"),
        "avg_faithfulness": avg("faithfulness"),
        "avg_answer_relevance": avg("answer_relevance"),
        "results": results,
    }

    return summary

report = evaluate_rag_pipeline(
    rag_query_fn=lambda q: rag_query(conn, q),
    search_fn=lambda q, top_k: search_pgvector(conn, q, top_k),
    golden_set=golden_set,
)

print(f"Context precision: {report['avg_context_precision']:.3f}")
print(f"Context recall:    {report['avg_context_recall']:.3f}")
print(f"Faithfulness:      {report['avg_faithfulness']:.3f}")
print(f"Answer relevance:  {report['avg_answer_relevance']:.3f}")

Save the report as JSON after each run. Over time, you build a history of how pipeline changes affect each metric independently — a chunking change should improve context recall without affecting faithfulness; a prompt change should improve faithfulness without affecting retrieval.

Regression testing a RAG app

Evaluation isn't a one-time activity. Every change to the pipeline — new embedding model, different chunk size, updated prompt, corpus update — can degrade quality in ways that aren't obvious from spot-checking.

def regression_check(current_report, baseline_path, thresholds=None):
    thresholds = thresholds or {
        "avg_context_precision": -0.05,
        "avg_context_recall": -0.05,
        "avg_faithfulness": -0.03,
        "avg_answer_relevance": -0.05,
    }

    with open(baseline_path) as f:
        baseline = json.load(f)

    regressions = []
    for metric, max_drop in thresholds.items():
        current_val = current_report[metric]
        baseline_val = baseline[metric]
        delta = current_val - baseline_val

        if delta < max_drop:
            regressions.append({
                "metric": metric,
                "baseline": baseline_val,
                "current": current_val,
                "delta": delta,
                "threshold": max_drop,
            })

    return {
        "passed": len(regressions) == 0,
        "regressions": regressions,
    }

Wire this into CI: run the eval harness on every PR that touches the RAG pipeline. If any metric drops below the threshold, fail the build. The thresholds are asymmetric — faithfulness has a tighter threshold (-0.03) because hallucination is harder to catch downstream than slightly degraded retrieval.

Tool landscape

Several tools automate parts of this evaluation workflow:

RAGAS (ragas.io) — the reference implementation of the RAG triad metrics. Python library; generates synthetic test questions from your corpus; supports multiple LLM judges. The metrics described in this lesson follow the RAGAS definitions.
Arize Phoenix — open-source observability for LLM applications. Traces each RAG query through retrieval and generation, visualizes embedding clusters, flags retrieval drift. Runs locally or hosted.
LangSmith — LangChain's tracing and evaluation platform. Deep integration with LangChain pipelines; dataset management; annotation queues for human evaluation. Hosted only.
LangFuse — open-source alternative to LangSmith. Tracing, scoring, prompt management. Self-hostable. Works with any framework, not just LangChain.

All four support LLM-as-judge scoring. The differentiator is tracing granularity — Phoenix and LangSmith show you what happened at each step of a query (which chunks were retrieved, what the prompt looked like, what the model returned), which is essential for diagnosing why a specific test case failed.

For teams starting out: run the eval harness from this lesson manually. When you have 100+ test cases and a CI pipeline, adopt RAGAS for standardized metrics and Arize Phoenix or LangFuse for per-query tracing.

Evaluation costs

Running the eval harness costs LLM API calls. For a 100-case golden set with an average of 8 claims per answer:

Claim extraction — 100 calls × ~500 input tokens + ~200 output tokens = ~70K tokens
Claim verification — 800 calls × ~300 input tokens + ~10 output tokens = ~248K tokens
Answer relevance — 100 calls × ~400 input tokens + ~10 output tokens = ~41K tokens
Total — ~360K tokens. At Claude Sonnet pricing ($3/1M input, $15/1M output): roughly $1.10 per full eval run

Running the pipeline itself (embedding the queries + generation) adds another ~$1–2. A full regression run costs under $5 — cheap enough to run on every PR.

Key takeaways

RAG failures split into retrieval failures (wrong chunks) and generation failures (wrong answer from right chunks). Evaluation must measure each independently — context precision/recall for retrieval, faithfulness for generation, answer relevance for end-to-end.
The RAG triad (Es et al., 2023) — context relevance, groundedness, answer relevance — is the standard decomposition. Low scores on each metric point to different fixes.
LLM-as-judge is practical for automated evaluation but introduces position bias, verbosity bias, and self-preference. Use pointwise scoring (not pairwise), enforce concise rubrics, and use a different model family as judge than generator.
A golden test set of 50–100 annotated (question, expected answer, relevant chunks) triples is the minimum for reliable aggregate metrics. Version it alongside your code.
Wire evaluation into CI as a regression gate. Faithfulness regressions need tighter thresholds than retrieval regressions — hallucinations are harder to catch downstream.

← Previous