What RAG actually is

Three verbs

The name spells out the architecture: Retrieve, Augment, Generate.

• Retrieve — given a query, find the most relevant pieces of text from a knowledge base.
• Augment — insert those pieces into the prompt alongside the query.
• Generate — have the language model produce an answer conditioned on that augmented prompt.

Every RAG system, from a forty-line script to a platform serving millions of queries, is an elaboration of these three verbs. When a system feels complicated, it is almost always because one of the three has been made more sophisticated — retrieval especially — not because a fourth verb appeared.

The idea traces to Lewis et al. (2020), which coined "retrieval-augmented generation" for combining a retriever with a generator and training them together. Modern practice usually keeps the generator frozen — a general-purpose model like Claude — and invests all the effort in retrieval and prompt construction around it. That shift, from train the whole thing to wrap a frozen model in good retrieval, is what made RAG cheap and ubiquitous.

Two pipelines, not one

The single most useful thing to hold in your head is that a RAG system is really two pipelines that meet at a shared store. Confusing them is the source of a lot of early mistakes.

The index-time pipeline (build once, refresh as data changes)

This runs offline, whenever your documents change — not on the user's request. Its job is to turn a pile of raw documents into a searchable store.

• Load & parse raw sources (PDFs, HTML, tables, database rows) into clean text.
• Chunk that text into passages small enough to retrieve precisely.
• Embed each chunk into a vector that captures its meaning.
• Store the vectors, plus the original text and metadata, in a vector store.

You pay this cost once per document (and again only when a document changes). Get it wrong and no amount of clever query-time work can recover — retrieval can only find what indexing put there. Whole lessons ahead are dedicated to each step, because this pipeline quietly sets the ceiling on the entire system.

The query-time pipeline (runs on every question)

This runs online, once per user query, and must be fast.

• Embed the query into the same vector space as the chunks.
• Search the store for the nearest chunks.
• Assemble the retrieved chunks and the query into a prompt.
• Generate the answer with the model.

The two pipelines share exactly one thing — the vector store — and they must agree on how text is represented. In particular, the same embedding model has to encode both the chunks (at index time) and the query (at query time), or the query's vector lands in a different space than the chunks and search returns noise. This is the most common silent bug in a first RAG build.

The smallest honest RAG

Stripped of every framework, the query-time pipeline is short enough to read in one breath. This is pseudocode, but it is not far from the real thing you will build in the hands-on section.

def answer(query, store, embed, model, k=5):
    # 1. Retrieve: embed the query, find nearest chunks
    q_vec = embed(query)
    chunks = store.search(q_vec, k=k)   # top-k by vector similarity

    # 2. Augment: build a grounded prompt
    context = "\n\n".join(
        f"[{i+1}] {c.text} (source: {c.source})"
        for i, c in enumerate(chunks)
    )
    prompt = (
        "Answer using only the context. Cite sources by number. "
        "If the answer isn't in the context, say you don't know.\n\n"
        f"Context:\n{context}\n\nQuestion: {query}"
    )

    # 3. Generate
    return model.complete(prompt)

Read it against the three verbs: store.search is retrieve, building prompt is augment, model.complete is generate. Everything else in this course makes one of these three lines better — smarter chunks feeding the store, hybrid search and reranking inside search, query rewriting before it, faithfulness checks after complete.

Where the difficulty actually lives

A beginner expects the hard part to be the model. It almost never is — the generator is a solved, purchased component. The difficulty concentrates in retrieval: whether store.search returns the right k chunks.

Consider what has to go right for that one call to succeed. The document had to be parsed without mangling. It had to be chunked so the answer sits inside a single retrievable passage rather than split across two. The chunk and the query had to embed close together despite using different words. The store's approximate search had to actually surface that chunk in the top k. A failure at any step means the model never sees the passage it needed — and a model cannot answer from context it was never given.

That is why the roadmap ahead is weighted the way it is: a little on generation, a lot on getting the right text into that prompt.

Key takeaways

• RAG is three verbs — retrieve, augment, generate — and every system is an elaboration of them.
• It is really two pipelines: an offline index-time pipeline that builds a searchable store, and an online query-time pipeline that reads from it on every question.
• The pipelines meet at the vector store and must share an embedding space — encode chunks and queries with the same model, or search returns noise.
• The generator is the easy part; retrieval quality is where RAG is won or lost, which is where most of this course goes next.