Chunking strategies

Why chunk at all

An embedding model maps a span of text to a single vector. That vector is the unit of retrieval — you cannot retrieve half a vector. Embed a 50-page document as one vector, and a query either retrieves the whole document or nothing. Embed individual sentences, and each retrieval hit is a sentence with no surrounding context.

Chunking decides the granularity of retrieval. The goal is chunks where:

Each chunk is self-contained enough to be useful without its neighbors.
Each chunk is specific enough that a relevant query embeds near it, not near unrelated chunks from the same document.
Chunks preserve enough context that the model can interpret them correctly.

These goals are in tension. Larger chunks carry more context but dilute specificity. Smaller chunks are more targeted but may lack the context to be interpretable.

The same document chunked three ways: fixed-size, recursive, and semantic

Fixed-size chunking

Split text into chunks of a fixed token or character count, with optional overlap.

python

def fixed_size_chunks(
    text: str,
    chunk_size: int = 512,
    overlap: int = 64,
) -> list[str]:
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap
    return chunks

Token vs. character sizing. Embedding models have maximum input lengths measured in tokens, not characters. text-embedding-3-small accepts 8,191 tokens; all-MiniLM-L6-v2 accepts 256. Character counts are a rough proxy (~4 characters per token for English). For precision, use the model's tokenizer:

python

import tiktoken

enc = tiktoken.encoding_for_model("text-embedding-3-small")

def token_length(text: str) -> int:
    return len(enc.encode(text))

Overlap mitigates the boundary problem: if a fact spans a chunk boundary, overlap ensures at least one chunk contains it in full. 10–20% overlap is typical. More than that inflates the index without improving recall.

Fixed-size chunking ignores document structure. A chunk might start mid-sentence or split a code block.

Recursive/structure-aware chunking

Recursive chunking splits on progressively finer structural delimiters until each piece fits within the size limit. The hierarchy for prose: \n\n (paragraph) then \n (line) then . (sentence) then (word).

python

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " ", ""],
    length_function=token_length,
)

chunks = splitter.split_text(document_text)

This preserves paragraph and sentence boundaries wherever possible. A chunk never crosses a \n\n boundary unless a single paragraph exceeds the limit.

For markdown or code, use language-specific separators that split on headings before paragraphs:

python

from langchain_text_splitters import Language

markdown_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.MARKDOWN,
    chunk_size=1000,
    chunk_overlap=200,
)

Semantic chunking

Semantic chunking uses embedding similarity between consecutive sentences to find natural topic boundaries. Adjacent sentences that are semantically similar stay together; a similarity drop signals a split point.

python

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

semantic_splitter = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=75,
)

chunks = semantic_splitter.split_text(document_text)

The splitter embeds each sentence, computes cosine similarity between consecutive sentences, and places a split where similarity drops below the 25th percentile of all consecutive-pair similarities in that document.

Trade-off: Semantic chunking produces more coherent topic-aligned chunks, but it costs an embedding call per sentence at index time and produces variable-length output that makes retrieval less predictable.

The size-recall-precision tradeoff

Chunk size directly affects retrieval quality in opposing directions:

Smaller chunks (100–256 tokens): Higher precision — each retrieved chunk is tightly relevant. Lower recall — an answer spanning multiple chunks requires a higher k to capture them all. Short chunks also lose context: "This value should be set to true" is meaningless without knowing what "this value" refers to.
Larger chunks (512–1024 tokens): Higher recall — more likely that a single chunk contains the complete answer. Lower precision — irrelevant material in the chunk dilutes the embedding, reducing its similarity score against precise queries.

A practical starting range: 256–512 tokens with 10–20% overlap. For code documentation, shorter chunks (128–256 tokens) often work better because individual functions or config parameters are self-contained. Tune based on your retrieval metrics (Lesson 12).

Metadata enrichment

A raw text chunk loses its position in the document hierarchy. Enrich each chunk with metadata at index time:

python

chunk = {
    "text": "Returns the user's current balance...",
    "source": "api-reference.md",
    "section": "Billing > Account API > get_balance",
    "chunk_index": 3,
    "total_chunks": 47,
}

This enables:

Filtered retrieval. "search only in the API reference" becomes a metadata filter on source — cheaper and more precise than hoping the embedding separates API docs from tutorials.
Source citation. Display the section path when citing the passage in the generated answer.
Context reconstruction. Retrieve neighboring chunks by chunk_index when a single chunk is insufficient.

Choosing a strategy

For most production RAG systems: start with recursive chunking at 400–512 tokens with 10–15% overlap. It handles the common case well, respects document structure, and is cheap. Move to semantic chunking only if your documents have weak structural markers (no headings, no paragraph breaks) and you can absorb the embedding cost at index time.

The right strategy depends on your evaluation metrics more than on first principles. Measure recall@k on a labeled test set (Lesson 12), adjust chunk size, and compare.

Chunking is the single highest-leverage decision in this pipeline, and this lesson is the working subset of it. If it is where your retrieval quality is stuck, our Chunking Strategies for RAG track takes the same ground much further — semantic and late chunking, sentence-window and parent-child retrieval, code and table handling, adaptive sizing, and an evaluation harness that tells you which strategy actually wins on your corpus rather than on a benchmark.

← Previous