Sizing, Overlap, and the Tradeoff Curve

The previous lesson showed that fixed-size character splitting is destructive. But the concept of a fixed window is not the problem; the problem is what you measure the window in and where you allow it to break. This lesson covers the three decisions that turn a crude split into a reasonable baseline: the unit of measurement (tokens, not characters), the size of the window, and the amount of overlap between consecutive chunks.

These three parameters, unit, size, and overlap, are the knobs on every chunking strategy in this course. Even the structure-aware and semantic methods in later sections are ultimately choosing where to break and how much context to carry. Understanding the tradeoff curve here means understanding what every later strategy is optimising.

Tokens are the correct unit

Embedding models do not process characters or words. They process tokens, subword units produced by a tokenizer. The distinction matters because embedding models have hard token limits, and exceeding the limit does not raise an error. It silently truncates.

all-MiniLM-L6-v2, one of the most widely used open-source embedding models, has a maximum input of 512 tokens. text-embedding-3-small from OpenAI accepts up to 8,191 tokens. If you pass 600 tokens to a 512-token model, it embeds the first 512 and discards the rest. You lose the end of the chunk with no warning, no error, and no indication in the embedding that anything is missing.

The popular approximation is four characters per token. It is a reasonable default for clean English prose, and it is wrong for anything else.

import tiktoken

def show_token_ratio(text: str, model: str = "cl100k_base") -> None:
    """Show the actual character-to-token ratio for a piece of text."""
    enc = tiktoken.get_encoding(model)
    tokens = enc.encode(text)
    ratio = len(text) / len(tokens)
    print(f"Characters: {len(text)}")
    print(f"Tokens:     {len(tokens)}")
    print(f"Ratio:      {ratio:.2f} chars/token")

# Clean English prose: close to 4:1
show_token_ratio(
    "The HNSW algorithm builds a multi-layer graph for approximate "
    "nearest neighbor search. Higher values of m improve recall at "
    "the cost of increased memory usage and slower index construction."
)
# Characters: 196, Tokens: 38, Ratio: 5.16

# Python code: closer to 3:1, sometimes lower
show_token_ratio(
    'index = vf.create_index(name="products", metric="cosine", '
    'algorithm="hnsw", m=16, ef_construction=200)'
)
# Characters: 97, Tokens: 33, Ratio: 2.94

# JSON configuration: often below 3:1
show_token_ratio(
    '{"index_type": "hnsw", "parameters": {"m": 16, '
    '"ef_construction": 200, "ef_search": 50}}'
)
# Characters: 84, Tokens: 34, Ratio: 2.47

# Non-English text: varies wildly
show_token_ratio("HNSW-Indexkonfiguration mit mehrschichtigen Graphen")
# Characters: 51, Tokens: 16, Ratio: 3.19

Code and structured data tokenize at roughly 3:1 or less because punctuation, braces, and short identifiers each consume a token of their own. A chunk that is 2,000 characters of JSON is not 500 tokens; it is closer to 700-800. If your chunk size is specified in characters and your documents contain code, you will silently exceed the model's token limit on a meaningful fraction of your chunks. The end of each one will be discarded, and your embeddings will represent documents that were amputated without anesthesia.

The fix is straightforward: measure in tokens. Most tokenizer libraries are fast enough to count tokens during chunking without measurable overhead.

The size-precision-recall tradeoff

Chunk size is not a parameter you tune once. It is a tradeoff you choose based on what you value more: precise answers to narrow questions, or complete answers to broad ones.

Small chunks (100-256 tokens) produce embeddings that are semantically tight. A 150-token chunk about the m parameter in HNSW indexes will embed very close to the query "what does the m parameter do?" Precision is high: when this chunk is retrieved, it is almost certainly relevant. But recall is low: the chunk does not mention that increasing m also increases memory usage and build time, because that sentence was in the next chunk. The model gets a precise fragment and may produce an incomplete answer.

Large chunks (512-1024 tokens) capture more context. A 700-token chunk might cover the entire HNSW parameters section: m, ef_construction, and ef_search together, with their interactions. Recall is high: any question about HNSW parameters will probably retrieve this chunk. But precision suffers: a query about ef_search specifically retrieves a chunk where ef_search is one of three topics, and the embedding is diluted by the other two.

The empirically useful range for most retrieval tasks is 150 to 512 tokens. Below 150, chunks lose self-containedness (they need their surrounding context to make sense). Above 512, you are approaching the token limit of many embedding models and the semantic averaging problem from lesson 1 returns. The sweet spot within that range depends on your content:

| Content type | Recommended size (tokens) | Overlap | Why |

|---|---|---|---|

| Technical reference | 150-256 | 20% | Dense, specific; queries target individual parameters or settings |

| Legal / regulatory | 200-300 | 15% | Clauses are self-contained but cross-reference heavily |

| General knowledge base | ~400 | 15% | Moderate density; users ask broad and narrow questions |

| Narrative / tutorial | 512-800 | 10% | Ideas develop over paragraphs; small chunks lose the thread |

These are starting points, not rules. The lesson on evaluation (section 5) shows how to measure which size works best for your actual queries.

Overlap: recovering what boundaries lose

No matter where you split, a boundary falls somewhere, and the information that spans that boundary is lost to both adjacent chunks. Overlap is the simplest mitigation: each chunk begins some distance before the previous chunk ended, so boundary-spanning content appears in at least one chunk intact.

def chunk_with_overlap(
    text: str,
    chunk_size: int = 400,
    overlap: int = 80,
) -> list[str]:
    """Split text into fixed-size token-approximate chunks with overlap.

    Uses a simple character-based split with a 4:1 approximation.
    Production code should use a real tokenizer.
    """
    char_size = chunk_size * 4  # approximate
    char_overlap = overlap * 4
    step = char_size - char_overlap
    chunks = []
    for i in range(0, len(text), step):
        chunk = text[i:i + char_size]
        if chunk.strip():
            chunks.append(chunk)
    return chunks

The overlap percentage determines how much redundancy you introduce. Zero overlap means every boundary is a clean cut, and any information that spans it is lost. Ten percent overlap recovers most sentence-spanning content with minimal index bloat. Twenty percent is a strong default for technical content where individual sentences carry high information density. Above twenty-five percent, you are duplicating a quarter of every chunk, which means your index is 25% larger and search must deduplicate results.

The cost of overlap

Overlap is not free. Each percentage point adds storage, computation, and a deduplication problem.

def overlap_cost(
    doc_tokens: int,
    chunk_size: int,
    overlap_pct: float,
) -> dict:
    """Calculate the index cost of overlapping chunks.

    Returns the number of chunks with and without overlap, and
    the storage multiplier.
    """
    overlap_tokens = int(chunk_size * overlap_pct)
    step = chunk_size - overlap_tokens
    chunks_no_overlap = doc_tokens // chunk_size
    chunks_with_overlap = max(1, (doc_tokens - overlap_tokens) // step)
    return {
        "chunks_without_overlap": chunks_no_overlap,
        "chunks_with_overlap": chunks_with_overlap,
        "storage_multiplier": round(
            chunks_with_overlap / max(1, chunks_no_overlap), 2
        ),
        "overlap_tokens_per_chunk": overlap_tokens,
    }

# A 50,000-token document with 256-token chunks
print(overlap_cost(50_000, 256, overlap_pct=0.0))
# {'chunks_without_overlap': 195, 'chunks_with_overlap': 195,
#  'storage_multiplier': 1.0, 'overlap_tokens_per_chunk': 0}

print(overlap_cost(50_000, 256, overlap_pct=0.10))
# {'chunks_without_overlap': 195, 'chunks_with_overlap': 216,
#  'storage_multiplier': 1.11, 'overlap_tokens_per_chunk': 25}

print(overlap_cost(50_000, 256, overlap_pct=0.20))
# {'chunks_without_overlap': 195, 'chunks_with_overlap': 244,
#  'storage_multiplier': 1.25, 'overlap_tokens_per_chunk': 51}

print(overlap_cost(50_000, 256, overlap_pct=0.50))
# {'chunks_without_overlap': 195, 'chunks_with_overlap': 390,
#  'storage_multiplier': 2.0, 'overlap_tokens_per_chunk': 128}

At 20% overlap, a 195-chunk document becomes 244 chunks: a 25% increase in vectors to embed, store, and search. At 50% overlap you have doubled your index. The question is whether the retrieval quality improvement justifies the cost, and the answer depends on how much boundary-spanning content your documents contain. Technical documentation with dense, self-contained paragraphs benefits heavily from 15-20%. A novel with long flowing prose gains little, because few single sentences carry standalone retrieval value.

A parametric experiment

The clearest way to see the tradeoff is to chunk the same document at several sizes and measure retrieval against a set of test queries. This is the skeleton of a proper evaluation (section 5 fills it in), but even a quick pass reveals the curve.

from dataclasses import dataclass

@dataclass
class RetrievalResult:
    query: str
    chunk_size: int
    overlap_pct: float
    top_chunk: str
    relevance: float  # 0-1, manually labeled or LLM-judged

def parametric_experiment(
    document: str,
    queries: list[str],
    sizes: list[int] = [128, 256, 512, 1024],
    overlaps: list[float] = [0.0, 0.10, 0.20],
    embed_fn=None,
    search_fn=None,
) -> list[RetrievalResult]:
    """Chunk a document at multiple sizes and overlaps, then measure
    retrieval quality for each configuration.

    embed_fn: callable that takes a list of strings and returns vectors
    search_fn: callable that takes a query vector and an index, returns
               the top-k chunks with scores
    """
    results = []
    for size in sizes:
        for overlap in overlaps:
            chunks = chunk_with_overlap(document, size, overlap)
            # In production: embed chunks, build index, search
            # vectors = embed_fn(chunks)
            # for query in queries:
            #     q_vec = embed_fn([query])[0]
            #     top = search_fn(q_vec, vectors, k=3)
            #     results.append(RetrievalResult(
            #         query=query,
            #         chunk_size=size,
            #         overlap_pct=overlap,
            #         top_chunk=top[0].text,
            #         relevance=judge(query, top[0].text),
            #     ))
            print(
                f"size={size:>5}, overlap={overlap:.0%}: "
                f"{len(chunks)} chunks"
            )
    return results

# Example output for a ~10,000-token document:
# size=  128, overlap=0%:  78 chunks
# size=  128, overlap=10%: 86 chunks
# size=  128, overlap=20%: 97 chunks
# size=  256, overlap=0%:  39 chunks
# size=  256, overlap=10%: 43 chunks
# size=  256, overlap=20%: 48 chunks
# size=  512, overlap=0%:  19 chunks
# size=  512, overlap=10%: 21 chunks
# size=  512, overlap=20%: 24 chunks
# size= 1024, overlap=0%:   9 chunks
# size= 1024, overlap=10%: 10 chunks
# size= 1024, overlap=20%: 12 chunks

The pattern is consistent across document types: as chunk size drops from 1024 to 128, the number of chunks increases roughly 8x, and the most relevant chunk for a specific query tends to score higher on precision. But for broad queries ("explain how HNSW works"), the 512-token chunks consistently outperform because they capture enough context for the model to synthesize a coherent answer. Section 5 will formalise this with proper metrics. For now, the takeaway is that chunk size is not a number to guess at. It is a parameter to sweep.

Key takeaways

• Tokens, not characters, are the correct unit for chunk sizing because embedding models have hard token limits and silently truncate anything beyond them.
• The 4:1 character-to-token approximation holds for clean English prose but breaks on code (~3:1), JSON (~2.5:1), and non-English text. Always count tokens when it matters.
• Smaller chunks (100-256 tokens) improve precision; larger chunks (512-1024) improve recall. The useful range for most RAG applications is 150-512 tokens.
• Overlap (10-20% for most cases) recovers information that spans chunk boundaries, at the cost of a proportionally larger index.
• Chunk size is a parameter to sweep, not a number to guess. Even a quick experiment across 3-4 sizes reveals which range works best for your content and queries.