RAG Fundamentals

Why

RAG solves the problem of LLMs relying solely on their frozen parametric memory, which can be outdated, incomplete, or prone to hallucination. By retrieving relevant documents from an external knowledge source at query time and conditioning generation on them, RAG grounds answers in verifiable evidence. This architecture separates knowledge storage from reasoning capability, meaning the knowledge base can be updated independently without retraining the model. The retrieval step acts as a dynamic memory lookup that supplements the model's static training data with current, domain-specific information. Option B is wrong because RAG never modifies model weights; retrieval and generation happen entirely at inference time while the parameters remain frozen throughout. Option C confuses RAG with model compression techniques like quantization or pruning, which target hardware efficiency rather than knowledge access. Option D misunderstands the architecture because RAG still requires a prompt containing both the user's question and the retrieved context, so retrieval augments the prompt rather than replacing it. The underlying principle is the separation of knowledge from computation: the model provides reasoning and language fluency while the retriever provides factual grounding. This matters in practice because enterprise knowledge bases change frequently, and retraining or fine-tuning for every update would be prohibitively expensive and slow. RAG has become the standard approach for building knowledge-intensive applications where accuracy, currency, and source attribution are all required.

Why

The standard RAG pipeline at query time follows three sequential steps: the user's query is embedded into a vector, that vector retrieves the most relevant chunks from the indexed knowledge base, and the retrieved chunks are passed as context to the language model for grounded generation. This embed-retrieve-generate sequence ensures the model sees the most relevant evidence before producing its answer, and each step feeds directly into the next. The embedding step converts the natural language query into the same vector space used during document ingestion, making semantic similarity comparison possible against the pre-computed chunk embeddings. The generation step then assembles the retrieved passages into a prompt that instructs the model to answer based on the provided context. Option A describes a generate-then-verify pattern, which is a different architecture sometimes called self-RAG or corrective RAG and not the standard pipeline. Option C is wrong because fine-tuning is a training-time process that takes hours or days and cannot happen per-query at inference time. Option D inverts the pipeline by suggesting reranking the entire corpus before embedding, which would be computationally infeasible since reranking typically operates only on a small shortlist produced after embedding-based retrieval. Understanding this pipeline order is essential because debugging RAG systems requires knowing which stage failed: if retrieval returns irrelevant chunks, the problem is upstream of generation, and if good chunks produce bad answers, the problem is in prompt construction. In practice, most RAG frameworks implement this exact embed-retrieve-generate sequence as their default retrieval-augmented pipeline.

Why

Chunking gives retrieval the right granularity: passages small enough to be focused, well-embedded semantic units that fit within the model's context budget, but large enough to remain coherent and self-contained. A single embedding must represent the meaning of its chunk, so smaller chunks produce more precise embeddings that closely match a relevant query, whereas a whole-document embedding averages together many topics and dilutes the signal. Chunking also allows the system to select only the most relevant portions of a document rather than spending the entire context window on material that is mostly irrelevant. The chunk size decision involves a tradeoff between precision, which favors smaller pieces, and sufficient context, which favors larger ones. Option B is wrong because chunking is a data preparation step that has no effect on the language model's parameter count, which is fixed at training time. Option C misunderstands embeddings: each chunk gets its own embedding vector rather than being averaged into a single representation of the whole document. Option D gets the goal backwards because chunking deliberately avoids returning whole documents intact, instead surfacing only the most relevant passages. The underlying principle is that retrieval precision depends on the unit of indexing: if you index at the wrong granularity, even a perfect similarity metric cannot surface the right information at the right specificity. In production systems, chunk sizes typically range from 256 to 1024 tokens, tuned based on the nature of the content and the embedding model's effective window.

Why

Overlap between adjacent chunks ensures that a sentence or idea spanning a chunk boundary remains intact in at least one chunk, so retrieval does not miss context that was cut in half by an arbitrary split point. Without overlap, a key fact that straddles two chunks would be incomplete in both, making neither chunk a good match for a query about that fact. A typical overlap of 10 to 20 percent of the chunk size provides enough continuity without excessive duplication. This is especially important for documents with flowing prose where sentence and paragraph boundaries do not align neatly with fixed token counts. Option A is wrong because overlap is not about data redundancy or backup; it is about preserving semantic coherence at boundaries, and each chunk still stores distinct content for most of its length. Option C has the relationship backwards: overlap actually increases the total number of chunks and embeddings that must be stored, since the same text appears in more than one chunk. Option D is incorrect because overlap does not enforce uniform chunk sizes; chunks can still vary in length depending on the splitting strategy used. The underlying principle is that chunking is an artificial partition of continuous text, and any hard boundary risks separating information that belongs together. In practice, tuning the overlap percentage is one of the first levers practitioners adjust when retrieval misses passages that clearly contain the answer.

Why

Dense retrieval works by embedding both the query and every chunk into the same high-dimensional vector space, then ranking chunks by a similarity metric such as cosine similarity or dot product against the query vector. This approach captures semantic meaning beyond exact word matches, so a query about 'car maintenance' can retrieve a chunk discussing 'automobile servicing' even though the words differ. The embedding model learns to place semantically similar texts close together in vector space during its training, which is what makes the similarity score a useful proxy for relevance. At scale, approximate nearest neighbor (ANN) indices like HNSW or IVF make this lookup fast even over millions of chunks. Option A describes lexical retrieval methods like BM25, which count and weight shared keywords but cannot capture paraphrases or synonyms the way dense embeddings do. Option C would require the LLM to process the entire corpus for every query, which is computationally infeasible for any non-trivial knowledge base and defeats the purpose of having a retrieval stage. Option D is wrong because recency has no inherent relationship to relevance; the most recently added chunk is not necessarily the most useful one for a given query. The broader principle is that dense retrieval trades the interpretability of keyword matching for the ability to understand meaning, which is why it dominates modern RAG systems. In practice, the choice of embedding model and similarity metric are among the most important decisions in a RAG pipeline because they directly determine whether the right chunks surface.