Why RAG exists

The model that knows a lot, and nothing in particular

A large language model is a compression of the text it was trained on. During training it reads a vast corpus and adjusts billions of weights until it can predict the next token well. What survives in those weights is a blurry, statistical memory of everything it saw — enough to write fluent prose about photosynthesis, Roman history, or Python decorators.

That memory has two properties that matter for anything you want to ship. It is frozen at the moment training ended, and it is parametric — baked into the weights, with no index, no source list, no way to point at where a fact came from. The model does not store your company handbook or yesterday's incident report. It stores a compressed impression of a general corpus, and when you ask about something outside that corpus, it does what it always does: it predicts a plausible continuation. Sometimes that continuation is true. Often it only sounds true.

Retrieval-Augmented Generation — RAG — is the standard fix. Before the model answers, you fetch relevant text from a source you control and place it in front of the question. The model then answers from that text instead of from memory alone. That single move is the whole idea, and the rest of this course is about doing it well.

Four ways a bare model fails

Every motivation for RAG traces back to one of these.

• Knowledge cutoff. The weights stop learning when training stops. Ask about an event, release, or price from after the cutoff and the model cannot know it — it can only guess in the shape of an answer.
• Private and proprietary data. Your contracts, tickets, wiki, and codebase were never in the training set. To the model, your organization's specifics simply do not exist.
• Hallucination. With no grounding, a model asked a question it cannot answer will still answer. It generates the most statistically likely text, which for a specific factual query is frequently a confident fabrication — right format, wrong content.
• Staleness and the cost of change. Facts move. Policies get rewritten, docs get updated, prices change weekly. Knowledge frozen in weights cannot follow, and the obvious fix — retrain — is expensive and slow.

"Just retrain it" is the wrong tool

A natural instinct is: if the model does not know something, teach it by training on more data. For keeping a system current, this is almost always the wrong lever, for three reasons.

Cost and latency of updates. Continued pretraining or fine-tuning is a batch job measured in hours or days and real money. If your knowledge changes daily, you cannot retrain daily. Retrieval updates the moment you add a document to the index — no training run at all.

It still cannot cite. Even a perfectly fine-tuned model has folded your facts into its weights. Ask "where did that come from?" and there is no answer to give. For anything a user must trust — a legal, medical, financial, or support answer — an unattributable claim is close to worthless.

Knowledge is the wrong thing to fine-tune. Fine-tuning excels at teaching behavior — tone, format, how to follow a schema. It is a poor and unreliable way to inject facts: you risk overfitting, catastrophic forgetting, and a model that has memorized your document without being able to reproduce it faithfully. We will make this trade-off precise in the "RAG vs. fine-tuning" lesson.

The core move: look it up

RAG borrows an idea so obvious we forget it is powerful: you do not have to memorize what you can look up. An open-book exam does not test recall; it tests whether you can find the right passage and reason over it. RAG turns a closed-book model into an open-book one.

Concretely, at query time the system:

• takes the user's question,
• retrieves the most relevant passages from a knowledge base you own,
• assembles those passages together with the question into a single prompt, and
• asks the model to answer using the provided context.

The prompt handed to the model ends up looking like this:

Use ONLY the context below to answer. If the answer isn't in the
context, say you don't know.

Context:
[1] Refunds are issued to the original payment method within 5–7
    business days of approval. (source: policies/refunds.md)
[2] Orders marked "final sale" are not eligible for refund.
    (source: policies/refunds.md)

Question: How long do refunds take, and are sale items refundable?

Nothing about the model changed. We changed what we put in front of it. The knowledge now lives in an index you can update in seconds, and every answer can point back to the passage it came from.

Attribution is a feature, not a footnote

Because the answer is built from retrieved passages, you can show the user which passages. That changes the product, not just the plumbing:

• Trust. A user can verify the claim against its source instead of taking the model's word.
• Auditability. When an answer is wrong, you can see whether retrieval fetched the wrong passage or the model misread a right one — a distinction we will lean on hard when we get to evaluation.
• Governance. You decide what is in the index, so you control what the system is allowed to answer from.

Attribution is one of the main reasons RAG became the default architecture for question-answering over private data, not merely a workaround for the knowledge cutoff.

What RAG does not fix

It is just as important to know the boundaries, so you set the right expectations for the rest of this course.

• It is not a reasoning upgrade. RAG changes what the model knows, not how well it thinks. A hard multi-step inference is still hard.
• It is only as good as its retrieval. Fetch the wrong passages and you have simply given the model confident, irrelevant context — sometimes worse than nothing. Most of the engineering in a real RAG system is in retrieval quality, which is why most of this course is too.
• It does not guarantee faithfulness. A model can ignore the context and answer from memory anyway. Getting it to stay grounded — and detecting when it doesn't — is its own discipline, covered in the Generation and Evaluation sections.

Key takeaways

• A language model's knowledge is frozen at training time and stored in its weights with no sources — so it fails on fresh facts, private data, and specifics, and fills the gap with confident hallucination.
• Retraining is the wrong tool for keeping knowledge current: too slow, too costly, and still unattributable.
• RAG's core move is to retrieve relevant text at query time and let the model answer from it — turning a closed-book model into an open-book one that can cite its sources.
• RAG fixes knowledge, not reasoning, and its quality is bounded by retrieval quality — the theme of everything that follows.