The Five Questions, and the Metric That Answers Each
A RAG pipeline, at evaluation time, produces exactly four artifacts worth naming. The input is the user's question. The retrieval context is the set of chunks the pipeline pulled and handed to the model. The actual output is what the model said. The expected output is what a competent human says it should have said — the one artifact that does not come from the system, and the one you have to write yourself.
Every RAG metric worth using is a function of some subset of those four. That is not a stylistic observation; it is the whole diagnostic apparatus. Which artifacts a metric reads determines which stage it can blame, and — just as importantly — which stages it is structurally incapable of blaming.
The five metrics
The framework this course uses for worked examples is DeepEval, whose RAG metric set maps onto the stage model cleanly enough to serve as the map. Five metrics, each reading a different subset, each blaming a different stage.
Contextual Recall reads the expected output and the retrieval context. It breaks the ideal answer into statements and asks, of each one, whether it can be attributed to something in the retrieved chunks. The score is the fraction that can. It blames the retriever, specifically for incompleteness: a low score means the information needed to answer was not brought back.
Contextual Precision reads the input, the expected output and the retrieval context, and — uniquely among the five — it reads the order. It marks each retrieved chunk relevant or not, then computes a weighted cumulative precision that divides by rank position, so a relevant chunk at position one contributes more than the same chunk at position five. It blames the reranker. Feed it the same five chunks in a different order and the score moves, which is precisely what makes it the reranker's native metric.
Contextual Relevancy reads the input and the retrieval context. It breaks the retrieved chunks into statements and asks what fraction of them are relevant to the question. It blames the retriever too, but for a different sin: noise. You can score 1.0 on recall and 0.2 on relevancy by retrieving everything, and a generator handed that context has to find the answer inside four-fifths irrelevant text.
Faithfulness reads the actual output and the retrieval context. It extracts the claims the answer makes, extracts the truths the context supports, and scores the fraction of claims not contradicted by those truths. It blames the generator, for hallucination.
Answer Relevancy reads the input and the actual output. It splits the answer into statements and scores the fraction that address the question asked. It blames the generator too, for a different failure: answering a nearby question instead of this one.
Reading a row of five scores
The diagnostic value appears when you look at all five together for one failing example. A few characteristic patterns:
Recall low, everything else high. The answer was never retrieved. The generator did well with what it had — possibly by declining to answer — and no amount of prompt engineering will fix it. Work on chunking, embedding, or k.
Recall high, precision low. The right chunk came back, but buried. Everything the generator needs is present; it is at rank eight behind seven distractors. This is a reranker problem and it is usually the cheapest fix on the list.
Recall and relevancy both high, faithfulness low. The context was good and the model invented anyway. That is a generation problem — prompt, model, or decoding.
Faithfulness high, answer relevancy low. Everything the model said was supported by the context, and none of it answered the question. This is the pattern that produces the most frustrating class of RAG failure, because every sentence is defensible and the reply is useless.
Everything high, answer still wrong. Your expected output is wrong, or your gold labels are. This happens more than anyone admits, and it is why the next two lessons are about building the eval set rather than about metrics.
What reference-based costs, and buys
Two of the five — Contextual Recall and Contextual Precision — require the expected output. Three do not. That split has a practical consequence you will feel immediately.
The three referenceless metrics can run on production traffic, where no one has written an ideal answer and no one ever will. That is what makes online RAG evaluation possible at all. The two reference-based ones can only run on your labelled set, which is why that set is a capital asset rather than a formality.
But referenceless is not free, and the cost is specific: a referenceless metric cannot see what is missing. Contextual Relevancy scores the noise in what you retrieved; it has no opinion whatsoever about the chunk you failed to retrieve, because it never learns that chunk exists. A pipeline that retrieves three tightly relevant chunks and misses the one containing the answer scores beautifully on relevancy and faithfulness, and the answer is wrong. Only Contextual Recall — reading the expected output — catches it.
This is the single most important asymmetry in RAG evaluation, and the reason the honest answer to "can we just run this on production traffic?" is "for three of the five, yes, and those three cannot detect the most common failure."
The rest of this course takes those five apart one at a time, computes each by hand on the running corpus before writing a line of framework code, and ends by putting them back together into the ablation that says where to spend.