Context assembly & grounded generation
Context assembly and grounded generation
Retrieved chunks do not arrive in a format ready for generation. They are unordered fragments — each with its own metadata, possible overlaps with other chunks, and varying relevance to the query. Context assembly is the step between retrieval and generation: selecting, ordering, deduplicating, and formatting chunks into a prompt that maximizes the generator's chance of producing a faithful, cited answer.
Ordering and the lost-in-the-middle effect
Liu et al. 2023 (Lost in the Middle) demonstrated that language models attend unevenly to information based on its position in the context window. In their experiments across 20 models, performance was highest when relevant information appeared at the very beginning or the very end of the input, and degraded significantly when it was placed in the middle — up to 20% accuracy drop on multi-document QA tasks.
The practical implication: place the most relevant chunks first and last in your context window. A common pattern is relevance-descending order (most relevant chunk first), which aligns with the strong primacy effect Liu et al. observed. An alternative is the "sandwich" layout — most relevant chunks at the beginning and end, with lower-relevance supporting context in the middle.
Context window budgeting
Every model has a finite context window. Claude 3.5 Sonnet supports 200K tokens. GPT-4o supports 128K tokens. But filling the full window is rarely optimal — cost scales linearly with input tokens, and Liu et al. showed that more context does not always mean better answers.
A practical budget for a RAG answer:
- System prompt: 200-500 tokens (instructions, persona, grounding rules)
- Retrieved context: 2,000-8,000 tokens (5-15 chunks at 200-500 tokens each)
- Query: 20-100 tokens
- Reserved for generation: 500-2,000 tokens
This totals 3,000-10,000 tokens — a fraction of the available window. The constraint is not capacity but quality: beyond 5-10 highly relevant chunks, additional context tends to introduce noise rather than signal.
Dynamic context sizing. Rather than a fixed chunk count, size the context to the query. A factoid question ("What is the default learning rate?") needs 1-3 chunks. A synthesis question ("Compare the three fine-tuning approaches discussed in chapters 4-6") may need 10-15. Use the reranker scores from L11 as a signal: include chunks above a relevance threshold rather than a fixed top-k.
Deduplication
Overlapping chunks (from sliding-window chunking in L5) can repeat the same passage verbatim. Passing duplicate content to the generator wastes tokens and can cause the model to over-weight the repeated information. Deduplicate by computing pairwise similarity between selected chunks and dropping any chunk whose cosine similarity to an already-included chunk exceeds a threshold (0.90-0.95 works in practice). A simpler alternative: exact substring matching on the overlapping regions.
Prompt construction for grounded generation
The system prompt defines the generator's contract with the retrieved context. Three elements matter: the grounding instruction (answer only from the provided context), the citation format, and the abstention rule.
import anthropic
client = anthropic.Anthropic()
def build_rag_prompt(
query: str,
chunks: list[dict],
max_context_tokens: int = 6000,
) -> list[dict]:
system = """You are a technical assistant. Answer the user's question using ONLY the provided source documents. Follow these rules strictly:
1. Base every claim on the source documents. Do not use prior knowledge.
2. Cite sources inline using [Source N] immediately after the claim they support.
3. If multiple sources support a claim, cite all of them: [Source 1][Source 3].
4. If the provided sources do not contain enough information to answer the question, respond exactly: "I don't have enough information in the provided sources to answer this question."
5. Do not speculate, extrapolate, or fill gaps with information not present in the sources.
6. If sources partially answer the question, answer what you can and state what is missing."""
context_parts = []
token_estimate = 0
for i, chunk in enumerate(chunks):
chunk_text = f"[Source {i+1}] (from: {chunk['source']})\n{chunk['text']}"
chunk_tokens = len(chunk_text.split()) * 1.3
if token_estimate + chunk_tokens > max_context_tokens:
break
context_parts.append(chunk_text)
token_estimate += chunk_tokens
context_block = "\n\n---\n\n".join(context_parts)
messages = [
{
"role": "user",
"content": f"""Source documents:
{context_block}
- - -
Question: {query}""",
}
]
return system, messages
def generate_answer(query: str, chunks: list[dict]) -> str:
system, messages = build_rag_prompt(query, chunks)
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=1024,
system=system,
messages=messages,
)
return response.content[0].text
chunks = [
{
"text": "LoRA (Low-Rank Adaptation) freezes the pretrained model weights and injects trainable rank-decomposition matrices into each layer. For a weight matrix W of shape (d, k), LoRA adds two matrices A (d, r) and B (r, k) where r << min(d, k). The forward pass computes W*x + B*A*x. Typical rank values are 8-64.",
"source": "fine-tuning-guide.pdf, p.12",
},
{
"text": "QLoRA combines 4-bit quantization of the base model with LoRA adapters in full precision. This reduces memory requirements from ~120GB to ~24GB for a 65B parameter model, enabling fine-tuning on a single 24GB GPU. The quantization uses a novel NormalFloat4 data type.",
"source": "fine-tuning-guide.pdf, p.18",
},
{
"text": "Full fine-tuning updates all model parameters and typically requires 2-3x the model size in GPU memory (model weights + gradients + optimizer states). For a 7B parameter model in float16, this means approximately 42GB of GPU memory.",
"source": "training-infrastructure.pdf, p.5",
},
]
answer = generate_answer("How does LoRA reduce memory usage compared to full fine-tuning?", chunks)
print(answer)Abstention: the "I don't know" guard
A RAG system that always produces an answer — even when its retrieved context contains nothing relevant — is a hallucination machine with extra steps. The abstention guard is the system prompt instruction that tells the model to refuse when context is insufficient.
The system prompt above includes the rule: "If the provided sources do not contain enough information to answer the question, respond exactly: 'I don't have enough information in the provided sources to answer this question.'"
Abstention quality depends on two factors:
- Retrieval signal. If the reranker scores all chunks below a confidence threshold (e.g., cross-encoder score < 0.3), you can skip generation entirely and return a canned "no relevant sources found" response. This is faster and more reliable than relying on the model to abstain.
- Model compliance. Claude and GPT-4-class models follow abstention instructions reliably when the system prompt is explicit. Smaller or older models may ignore the instruction and fabricate answers. Test abstention on your specific model with queries that deliberately have no relevant context.
Citation and attribution
Inline citations serve two purposes: they let the user verify claims against the source, and they give the model an anchoring mechanism that reduces hallucination. When the model must attribute each claim to a numbered source, it is structurally harder to introduce unsupported statements.
Citation formats in practice:
- Bracketed references —
[Source 1]— simple, works with any model, easy to parse programmatically for linking back to source documents in the UI - Footnote-style — superscript numbers rendered as links in the frontend
- Inline quotes — the model includes a short verbatim extract from the source alongside the citation, making verification immediate without clicking through
For programmatic extraction, bracketed references are the most reliable. Parse [Source N] from the generated text, map N back to the chunk metadata (URL, page number, document title), and render them as clickable links in the UI.
Hallucination mitigation at the generation step
Beyond the system prompt, three techniques reduce hallucination in the generation step.
Temperature. Set temperature=0 (or near zero) for factual RAG answers. Higher temperatures increase diversity but also increase the probability of the model departing from the provided context. There is no reason to introduce randomness when the answer should be grounded in specific documents.
Max tokens. Cap the response length to prevent the model from rambling past the information in the context. A 500-token answer grounded in 3 chunks is less likely to hallucinate than a 2,000-token answer that exhausts the source material and starts filling.
Structured output. For extractive tasks, ask the model to return JSON with explicit fields for answer, citations, and confidence. This constrains the output format and makes hallucination easier to detect programmatically.
import json
def generate_structured_answer(query: str, chunks: list[dict]) -> dict:
system = """You are a technical assistant. Answer using ONLY the provided sources.
Return a JSON object with this exact structure:
{
"answer": "Your answer text with [Source N] citations inline",
"sources_used": [1, 3],
"confidence": "high" | "medium" | "low",
"abstain": false
}
If the sources do not contain enough information, return:
{"answer": null, "sources_used": [], "confidence": "none", "abstain": true}"""
system_prompt, messages = build_rag_prompt(query, chunks)
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=1024,
temperature=0,
system=system,
messages=messages,
)
return json.loads(response.content[0].text)Streaming responses and time-to-first-token
In a user-facing RAG application, perceived latency matters as much as total latency. The time-to-first-token (TTFT) — how long until the user sees the first word of the response — is the critical UX metric. Production RAG systems target p90 TTFT under 2 seconds.
Server-Sent Events (SSE) stream tokens from the generator to the client as they are produced. The full pipeline latency is: embed query (5ms) + retrieve (10-30ms) + rerank (25-100ms) + context assembly (1-5ms) + TTFT from LLM (200-800ms). The retrieval and reranking steps are not streamable — they must complete before generation begins. But generation itself produces tokens incrementally, and streaming the first token as soon as it is available hides the remaining generation time (1-5 seconds) behind progressive rendering.
import anthropic
client = anthropic.Anthropic()
def stream_rag_answer(query: str, chunks: list[dict]):
system, messages = build_rag_prompt(query, chunks)
with client.messages.stream(
model="claude-sonnet-4-20250514",
max_tokens=1024,
temperature=0,
system=system,
messages=messages,
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
print()For an SSE endpoint serving a web client, the backend streams each token as a data: event. The frontend appends tokens to the rendered response as they arrive. The user sees the answer forming in real time — the same pattern used by ChatGPT, Claude, and every modern chat interface.
TTFT breakdown for a typical RAG query:
- Embedding the query: 5-20ms
- Vector search (HNSW, top-50): 10-30ms
- Reranking (cross-encoder, 50 candidates): 25-100ms
- Context assembly and prompt construction: 1-5ms
- LLM time-to-first-token: 200-800ms (varies by model, load, and prompt length)
- Total TTFT: 250-950ms (well within the 2-second target)
The remaining tokens stream over 1-5 seconds depending on answer length. The user perceives a fast system because the first words appear in under a second.
System prompt patterns for grounding
The system prompt is the primary control surface for grounding behavior. Three patterns appear consistently in production RAG systems.
The strict grounding prompt. The model must answer exclusively from provided context. Used for legal, medical, and compliance applications where unsupported claims are unacceptable. The example above follows this pattern.
The augmented knowledge prompt. The model may use its parametric knowledge to frame and explain, but factual claims must be grounded in the provided sources with citations. Used for educational and technical writing applications where the model's general knowledge adds context without replacing source attribution.
The context-plus-disclaimer prompt. The model answers from context when available and from general knowledge when not, clearly marking which parts come from which. Used in customer support and internal knowledge bases where a partial answer is better than no answer.
Choose the pattern based on the cost of hallucination. For a medical Q&A system, strict grounding with abstention is the only defensible choice. For an internal wiki assistant, the augmented pattern balances helpfulness with accuracy.
Key takeaways
- Place the most relevant chunks first (or first and last) in the context window — models attend unevenly to middle positions (Liu et al. 2023)
- Budget context at 2,000-8,000 tokens (5-15 chunks) rather than filling the full window; more context adds noise beyond the relevant set
- Deduplicate overlapping chunks before assembly to avoid wasting tokens and biasing the model
- Abstention ("I don't know") is a feature, not a failure — implement it at both the retrieval level (score thresholds) and the generation level (system prompt instruction)
- Inline citations (
[Source N]) reduce hallucination structurally by forcing the model to anchor each claim to a specific source - Set
temperature=0for factual RAG answers; there is no benefit to randomness when the answer should be grounded - Stream responses via SSE to achieve p90 TTFT under 2 seconds; the retrieval pipeline completes before generation begins, so streaming hides generation latency
- Choose your grounding pattern (strict, augmented, or context-plus-disclaimer) based on the cost of hallucination in your domain