Chunking strategies

Why chunk at all

An embedding model maps a span of text to a single vector. That vector is the unit of retrieval — you cannot retrieve half a vector. Embed a 50-page document as one vector, and a query either retrieves the whole document or nothing. Embed individual sentences, and each retrieval hit is a sentence with no surrounding context.

Chunking decides the granularity of retrieval. The goal is chunks where:

• Each chunk is self-contained enough to be useful without its neighbors.
• Each chunk is specific enough that a relevant query embeds near it, not near unrelated chunks from the same document.
• Chunks preserve enough context that the model can interpret them correctly.

These goals are in tension. Larger chunks carry more context but dilute specificity. Smaller chunks are more targeted but may lack the context to be interpretable.

Fixed-size chunking

Split text into chunks of a fixed token or character count, with optional overlap.

def fixed_size_chunks(
    text: str,
    chunk_size: int = 512,
    overlap: int = 64,
) -> list[str]:
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap
    return chunks

Token vs. character sizing. Embedding models have maximum input lengths measured in tokens, not characters. text-embedding-3-small accepts 8,191 tokens; all-MiniLM-L6-v2 accepts 256. Character counts are a rough proxy (~4 characters per token for English). For precision, use the model's tokenizer:

import tiktoken

enc = tiktoken.encoding_for_model("text-embedding-3-small")

def token_length(text: str) -> int:
    return len(enc.encode(text))

Overlap mitigates the boundary problem: if a fact spans a chunk boundary, overlap ensures at least one chunk contains it in full. 10–20% overlap is typical. More than that inflates the index without improving recall.

Fixed-size chunking ignores document structure. A chunk might start mid-sentence or split a code block.

Recursive/structure-aware chunking

Recursive chunking splits on progressively finer structural delimiters until each piece fits within the size limit. The hierarchy for prose: \n\n (paragraph) then \n (line) then . (sentence) then (word).

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " ", ""],
    length_function=token_length,
)

chunks = splitter.split_text(document_text)

This preserves paragraph and sentence boundaries wherever possible. A chunk never crosses a \n\n boundary unless a single paragraph exceeds the limit.

For markdown or code, use language-specific separators that split on headings before paragraphs:

from langchain_text_splitters import Language

markdown_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.MARKDOWN,
    chunk_size=1000,
    chunk_overlap=200,
)

Semantic chunking

Semantic chunking uses embedding similarity between consecutive sentences to find natural topic boundaries. Adjacent sentences that are semantically similar stay together; a similarity drop signals a split point.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

semantic_splitter = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=75,
)

chunks = semantic_splitter.split_text(document_text)

The splitter embeds each sentence, computes cosine similarity between consecutive sentences, and places a split where similarity drops below the 25th percentile of all consecutive-pair similarities in that document.

Trade-off: Semantic chunking produces more coherent topic-aligned chunks, but it costs an embedding call per sentence at index time and produces variable-length output that makes retrieval less predictable.

The size-recall-precision tradeoff

Chunk size directly affects retrieval quality in opposing directions:

• Smaller chunks (100–256 tokens): Higher precision — each retrieved chunk is tightly relevant. Lower recall — an answer spanning multiple chunks requires a higher k to capture them all. Short chunks also lose context: "This value should be set to true" is meaningless without knowing what "this value" refers to.
• Larger chunks (512–1024 tokens): Higher recall — more likely that a single chunk contains the complete answer. Lower precision — irrelevant material in the chunk dilutes the embedding, reducing its similarity score against precise queries.

A practical starting range: 256–512 tokens with 10–20% overlap. For code documentation, shorter chunks (128–256 tokens) often work better because individual functions or config parameters are self-contained. Tune based on your retrieval metrics (Lesson 12).

Metadata enrichment

A raw text chunk loses its position in the document hierarchy. Enrich each chunk with metadata at index time:

chunk = {
    "text": "Returns the user's current balance...",
    "source": "api-reference.md",
    "section": "Billing > Account API > get_balance",
    "chunk_index": 3,
    "total_chunks": 47,
}

This enables:

• Filtered retrieval. "search only in the API reference" becomes a metadata filter on source — cheaper and more precise than hoping the embedding separates API docs from tutorials.
• Source citation. Display the section path when citing the passage in the generated answer.
• Context reconstruction. Retrieve neighboring chunks by chunk_index when a single chunk is insufficient.

Choosing a strategy

For most production RAG systems: start with recursive chunking at 400–512 tokens with 10–15% overlap. It handles the common case well, respects document structure, and is cheap. Move to semantic chunking only if your documents have weak structural markers (no headings, no paragraph breaks) and you can absorb the embedding cost at index time.

The right strategy depends on your evaluation metrics more than on first principles. Measure recall@k on a labeled test set (Lesson 12), adjust chunk size, and compare.

Key takeaways

• Chunking determines retrieval granularity — each chunk becomes one vector, retrievable as a unit.
• Fixed-size chunking is simple but structure-blind; recursive chunking respects document boundaries; semantic chunking detects topic shifts but costs more.
• Chunk size trades precision against recall: 256–512 tokens with 10–20% overlap is a practical starting point.
• Enrich chunks with source, section, and positional metadata — it enables filtered search, citation, and context reconstruction.