Advanced indexing & retrieval patterns
Advanced indexing and retrieval patterns
Standard chunking embeds each text fragment in isolation. The chunk boundary strips context — a paragraph about "the transformer architecture" loses the fact that the document is specifically about vision transformers. The embedding captures local semantics but misses document-level positioning. Three families of indexing patterns recover that lost context, each operating at a different point in the pipeline and at different cost.
Parent-document and small-to-big retrieval
Embed small chunks (128–256 tokens) for precise semantic matching but retrieve the parent chunk (1,024–2,048 tokens) for generation. The index stores child embeddings linked to parent document IDs. At query time, retrieve the top-k children by cosine similarity, deduplicate by parent, and pass the full parent text to the LLM.
This exploits a fundamental asymmetry: small chunks produce tighter cosine matches because less irrelevant text dilutes the semantic signal, but the LLM needs surrounding context to generate coherent answers. A 128-token chunk containing a precise definition will match a definitional query better than a 512-token chunk where that definition is buried among adjacent paragraphs. But the LLM generating the answer needs those adjacent paragraphs for coherence.
Recall improves 10–20% on multi-sentence questions compared to fixed-size 512-token chunks in internal benchmarks. The cost is storage: you store both the child embeddings (for search) and the parent text (for generation), roughly 2x the raw text storage.
Sentence-window retrieval
A variant of the parent-document approach: embed individual sentences, but at retrieval time expand to a configurable window of sentences around each hit. Typically +/- 2–3 sentences in each direction.
The advantage over parent-document: finer granularity without the need to pre-define parent boundaries. Particularly effective for QA tasks where the answer spans 2–3 sentences but the surrounding paragraph provides disambiguation.
Implementation requires storing sentence offsets alongside embeddings so the expansion can be computed at retrieval time without re-parsing the document.
Auto-merging retrieval
Auto-merging builds on the parent-child hierarchy: if more than a threshold (typically 50–70%) of a parent node's children appear in the top-k retrieval results, the system merges them and returns the parent instead. This prevents the LLM from seeing fragmented, overlapping snippets of what is effectively one contiguous passage.
The merge threshold is tunable. At 50%, merging is aggressive — useful when questions frequently span multiple adjacent chunks. At 70%, merging is conservative — only triggers when the query clearly targets a single passage. Tune on your eval set by measuring answer completeness.
Contextual retrieval
Anthropic's Contextual Retrieval (2024) takes a fundamentally different approach: instead of recovering context at query time, bake it into the chunk at index time. For each chunk, an LLM call generates a short (50–100 token) description of how the chunk fits within the whole document. That description is prepended to the chunk text before computing the embedding.
A chunk that originally read "The architecture uses 12 attention heads with a dimension of 768" becomes "This chunk describes the GPT-2 model architecture in a paper comparing transformer variants. The architecture uses 12 attention heads with a dimension of 768." The embedding now captures both local content and document-level positioning.
import anthropic
client = anthropic.Anthropic()
CONTEXT_PROMPT = (
"<document>\n{document}\n</document>\n\n"
"<chunk>\n{chunk}\n</chunk>\n\n"
"Give a short succinct context to situate this chunk within "
"the overall document for the purposes of improving search "
"retrieval of the chunk. Answer only with the succinct context "
"and nothing else."
)
def generate_context(document: str, chunk: str) -> str:
"""Generate chunk-specific context using Claude with prompt caching."""
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=150,
messages=[{
"role": "user",
"content": [
{
"type": "text",
"text": f"<document>\n{document}\n</document>",
"cache_control": {"type": "ephemeral"}
},
{
"type": "text",
"text": (
f"<chunk>\n{chunk}\n</chunk>\n\n"
"Give a short succinct context to situate this chunk "
"within the overall document for the purposes of "
"improving search retrieval of the chunk. Answer only "
"with the succinct context and nothing else."
)
}
]
}]
)
return response.content[0].text
def create_contextual_chunks(document: str, chunks: list[str]) -> list[str]:
"""Process all chunks from a document with prompt caching."""
contextual_chunks = []
for chunk in chunks:
context = generate_context(document, chunk)
contextual_chunks.append(f"{context}\n\n{chunk}")
return contextual_chunksThe cache_control parameter on the document block enables Anthropic's prompt caching. When processing multiple chunks from the same document sequentially, the document prefix is cached server-side after the first call, reducing cost by approximately 90% for subsequent chunks in the same document.
Benchmark results from Anthropic's evaluation across diverse knowledge bases (Anthropic, 2024):
- Contextual embeddings alone: 35% reduction in retrieval failure rate (top-20 recall)
- Contextual embeddings + BM25 hybrid: 49% reduction
- Adding a reranker on top of contextual + BM25: 67% reduction
The cost trade-off: one LLM call per chunk at index time. For a 10,000-chunk corpus using Claude Haiku with prompt caching, that is roughly $0.10–0.30 (cached reads at $0.03/M tokens). Without caching, the same job costs $1–3. This is a one-time indexing cost, amortized across all future queries.
RAPTOR hierarchical summaries
RAPTOR (Sarthi et al., 2024) builds a tree of summaries over the corpus. Leaf nodes are the original chunks. The algorithm clusters semantically similar chunks using Gaussian Mixture Models (soft clustering, so a chunk can belong to multiple clusters), then summarizes each cluster with an LLM. Those summaries become the next tree layer. The process repeats recursively until a single root summary exists.
At query time, retrieval searches across all layers simultaneously. A broad question like "what are the main themes of this book" matches high-level summaries near the root. A specific question like "what was the revenue in Q3 2024" matches leaf chunks. The system returns the best-matching nodes regardless of their tree depth.
RAPTOR reported a 20% improvement on QuALITY (long-document QA benchmark) over flat chunking, with the largest gains on questions requiring synthesis across distant passages.
The indexing cost scales as O(n * log(n)) LLM calls for n chunks — each successive layer has fewer nodes, but every node requires a summarization call. For a 1,000-chunk corpus with branching factor 10, that is roughly 1,000 + 100 + 10 + 1 = 1,111 total summarization calls. Worthwhile for stable, infrequently-updated corpora (books, finalized documentation, regulatory filings). Expensive for corpora that change daily, because any leaf update invalidates its entire ancestor chain.
When each pattern pays off
- Parent-document / small-to-big: general-purpose improvement, minimal added cost (just storage), works on any corpus; the default upgrade from naive chunking
- Sentence-window: best for QA tasks where answers span 2–3 sentences and document boundaries are hard to define
- Contextual retrieval: high-value when chunks are ambiguous in isolation (legal clauses, medical records, technical specifications); justify the per-chunk LLM cost against retrieval quality requirements
- RAPTOR: long documents with hierarchical structure, broad synthesis questions, stable corpora that do not change weekly; do not use for rapidly changing content (news, tickets, chat logs)