Why RAG Failures Are Hard to Attribute
A reader asks your documentation assistant a question with a single correct answer: "What is the default m for the index you recommend above one million vectors?" The system replies, confidently and in fluent prose, "The default m is 16." The documentation says otherwise. Above a million vectors the recommended index is HNSW, and HNSW's default m is 32; the value 16 is the default for the flat-compressed index, which the same page recommends for corpora an order of magnitude smaller. The answer is wrong, it is wrong in a way that will cost the reader an afternoon, and it is wrong in a way no amount of re-reading the reply will explain.
Now find the bug. There are five places it could be, and you cannot tell them apart from the output.
The chunker may have split the recommendation table away from the defaults table, so no single chunk ever contained both halves of the answer. The embedding model may map a bare parameter name like m into a neighbourhood governed by the surrounding prose rather than the symbol, so the query vector lands nowhere near the configuration reference. The retriever may have returned five chunks, none of which was the defaults table. The reranker may have had the right chunk in hand at rank seven and dropped it when the pipeline cut to top-five. Or every stage may have worked perfectly, handed the generator a chunk containing both tables, and the generator read the wrong row.
Here is the property that makes RAG evaluation its own discipline rather than a special case of LLM evaluation: all five failures produce the same artifact. A wrong sentence. There is no stack trace, no exception, no partial output that betrays where the pipeline lost the thread. The system fails silently and completely, and the only evidence it leaves behind is a fluent paragraph that happens not to be true.
Why an end-to-end score tells you almost nothing
The instinct is to score the answer. Mark it wrong, collect a hundred of them, and report 71% accuracy. That number is real and it is worth having, but consider what it can and cannot support. It can tell you the system got worse after a change. It cannot tell you which change. It can tell you that 29 answers out of 100 were wrong. It cannot tell you whether to spend the next sprint on the chunker or the prompt.
An end-to-end score is one bit per example, and the pipeline has five stages. You are trying to solve for five unknowns with one equation, and no quantity of examples fixes that — a thousand wrong answers still give you a thousand copies of the same one bit. This is the arithmetic behind a pattern every team building RAG eventually recognises in itself: months of work, a steadily improving demo, and no ability to say which of the fourteen things they changed was responsible.
Worse, the stages interact in a direction that punishes guessing. Improving the retriever can lower your end-to-end score if the extra chunks it returns are noisy enough to distract the generator. Improving the chunker can lower recall if the smaller chunks now fall below the top-k cut. A team measuring only the final answer will read both of these as regressions and revert an improvement.
The shape of the fix
The fix is to measure each stage against something that stage alone is responsible for. That requires two things this course spends its first part building: ground truth at the level of the stage, not just the answer, and a metric per stage whose score cannot be moved by a different stage's behaviour.
Both are harder than they sound. Ground truth for the generator is an ideal answer, which most teams already write. Ground truth for the retriever is the set of chunks that should have come back — which requires deciding, for every question, which spans of your corpus actually contain the answer, and that decision is genuinely ambiguous often enough to need a policy. Ground truth for the chunker is harder still, and the honest answer is that you evaluate the chunker through its effect on retrieval rather than directly.
The stage-isolated metric is the other half. A metric that scores the generated answer against the retrieved context measures the generator and nothing else — hand it perfect context and a perfect generator scores 1.0, hand it garbage context and a perfect generator still scores 1.0, because it faithfully reported the garbage. That property, which looks like a weakness in isolation, is exactly what makes the metric diagnostic.
What this course does
It builds, in order: an eval set with ground truth at every stage; a metric for each stage that cannot be confounded by its neighbours; the statistical discipline to know when a difference between two runs is real; and finally the ablation procedure that answers the question this lesson opened with, by replacing one stage at a time with an oracle and watching what the score does.
The running example throughout is a corpus you have already met if you took Chunking Strategies for RAG — the two-hundred-page product documentation set, the one whose page 47 discusses twelve topics at once. Here it arrives carrying something that course did not need: a labelled evaluation set of sixty questions, each with a gold answer, the ids of the chunks that contain it, a query type, and a difficulty. Building that set is the subject of the next three lessons, because it is the part everyone skips and the part everything else rests on.
The rest of this course assumes you can already build a RAG pipeline. If retrieval, reranking and generation are not yet familiar as separable stages, Building RAG Systems is the prerequisite; this course is about knowing whether what you built works.