The Corpus, and Five Queries Naive RAG Gets Wrong

Why one corpus, carried the whole way

Techniques taught in isolation compare badly. Every blog post picks the example that flatters its technique, so RAPTOR is always demonstrated on a question that needs a summary and GraphRAG is always demonstrated on a question that needs a traversal, and you learn nothing about when either is wrong.

So this course uses one corpus for all 39 lessons. It is deliberately built so that every technique has something in it to shine on and something to fail on.

Northwind Mutual is a mid-size insurer. Nothing about it is exotic; the point is that it is boring and completely typical, and that its knowledge is scattered across six kinds of artifact that behave very differently under retrieval.

The baseline, and why it is honest

The baseline is naive RAG: split the documents, embed the pieces, retrieve top-k by cosine similarity, stuff them into a prompt.

To make the first failure concrete and checkable, here is a real fragment of a Northwind property policy and a real retrieval run over it. Retrieval below is TF-IDF plus cosine — a real method, exactly reproducible, and one you can run yourself in a second without a GPU. The failure it demonstrates is identical under a dense embedding model, and for exactly the same reason, which is the point worth taking away.

import re
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

# Nine clauses from a Northwind property policy, split on their own numbering.
chunks = chunk_by_clause(POLICY)          # 9 chunks: 4.1-4.4, 6.1, 7.1-7.3

query = "what is the flood sublimit"
vec = TfidfVectorizer(stop_words="english").fit(chunks + [query])
sims = cosine_similarity(vec.transform([query]), vec.transform(chunks))[0]
ranked = sorted(range(len(chunks)), key=lambda i: -sims[i])

The ranking that comes back:

Look at the top hit in full:

4.3 Water Damage and Flood. Loss or damage caused by Flood is covered only
where the Flood Endorsement has been purchased and is shown in the
Declarations. Where covered, the applicable flood sublimit is defined in
Section 7.2 and applies separately to each occurrence.

Scan it for a dollar amount. There isn't one. re.findall(r"\$[\d,]+", top) returns an empty list.

The clause that actually holds the answer — 7.2, which says flood is 25,000 dollars per occurrence and 50,000 in the aggregate — ranks second, at 0.078. That is a seven-fold score gap in favour of the chunk that does not contain the answer.

This is not a bug in TF-IDF and it will not be fixed by a better embedding model. The retriever did its job perfectly. The user asked about the flood sublimit and it returned the passage that is about the flood sublimit. The trouble is that in a well-drafted legal document, the passage about a thing and the passage stating that thing are deliberately different passages, because that is what makes the document maintainable.

Retrieval optimises for aboutness. Answers are not always where the aboutness is.

The five queries

Those are the five questions this course exists to answer. All of them were asked by a Northwind employee in a single week; none is adversarial.

Q1 — "What's the flood sublimit on the Henderson policy?" The baseline answers with a pointer, as shown above. The correct answer sits one clause away, unfetched. Fixed in lesson 15, by a loop that notices the pointer and follows it.

Q2 — "What changed across our policies in 2026?" This has no single chunk to retrieve. The honest answer requires every revised policy at once, which is a query about the corpus rather than a query into it. Fixed in lesson 17, by community summaries.

Q3 — "What's the premium for a 42-year-old non-smoker in Ohio?" The answer is one cell, at the intersection of an age band row and a state column. Flattening that grid into prose at ingest destroys the alignment, and the model then confabulates a plausible number. Fixed in lesson 21, by not flattening it.

Q4 — "Why was claim 88214 denied?" The denial code is easy to retrieve. The reason requires four hops: claim, to policy, to the endorsement that modified it, to the state regulation the endorsement implements. No chunk holds that chain. Fixed in lessons 14 and 18.

Q5 — "How are adjusters trained to photograph hail damage?" It is demonstrated on screen at 14:32 of a training video and never spoken aloud, so it is not in the transcript. The baseline says the material does not specify. Fixed in lesson 24.

What the five have in common

Notice what is not wrong in any of them. The corpus contains the answer every time. The embedding model is fine. The vector index is fine. Nobody needs a bigger model.

Four of the five are failures of structure — the pipeline threw away a relationship at ingest (a cross-reference, a table's geometry, a link between records, a visual channel) and no amount of query-time cleverness can restore it. The fifth is a failure of scope: the question is about the collection, and retrieval only ever returns members of it.

That is the shape of most real RAG failure, and it is why so much of this course is about what you do before a query ever arrives.

The rule for the rest of the course

Every technique from here gets held to these five queries. When a lesson claims a technique helps, it will say which of the five it moves and which it leaves exactly where it was — because a technique that fixes Q2 and does nothing for Q1 is not better or worse, it is a different tool.

In lesson 39 we re-run all five against a system that layers the techniques that earned their place, and show the deltas.

What to measure: write down your own five. Real questions, from real users, where you know the correct answer and can state why the current system gets it wrong. Five is enough to start and far more useful than a benchmark, because you can inspect every failure by hand. This set is the thing you will regret not having built.