The Chunking Problem

You are building a support chatbot over your product's 200-page documentation. The simplest approach seems obvious: embed each page as a single vector and store it in your index. A user asks "What does the m parameter do?" The answer is on page 47, buried in a section about HNSW index configuration between a discussion of distance metrics and a graph about recall curves. That page covers twelve topics. Its embedding is a single vector that represents the average of all twelve, and that average is close to none of them. The query vector for "m parameter" is nowhere near it. Retrieval fails, and the model either hallucinates or says it does not know. The documentation is right there, and the system cannot find it.

This is the chunking problem. Not whether to split your documents, but how to split them so that each piece is retrievable on its own terms. This course is about that question, and nothing else.

What a chunk is

A chunk is the atomic unit of retrieval. When a user asks a question and your system searches the vector index, it retrieves chunks, not documents, not paragraphs, not sentences. You cannot retrieve half a vector. Whatever you embedded as one unit comes back as one unit or not at all. That makes the chunk boundary the most consequential decision in a RAG pipeline: it determines what can be found, how precisely it can be found, and how useful it is once it arrives in the model's context window.

The word "chunk" is misleading because it sounds arbitrary, like breaking a chocolate bar into uneven pieces. In practice, a good chunk is a carefully scoped unit of meaning, one that carries enough context to be understood on its own but is specific enough to match a narrow query.

Why granularity matters: three tensions

Every chunking decision is a negotiation between competing pressures. There are three that recur in every strategy this course covers.

Self-containedness vs. specificity. A large chunk (an entire section, a full page) is self-contained: it includes its own context, its own setup, its own definitions. But it is also vague. Its embedding is a blend of many ideas, and it will match many queries weakly rather than one query strongly. A small chunk (a single sentence, a table row) is specific: it embeds one idea tightly. But stripped of its surrounding context, it may be incomprehensible. "Set m to 16 for most workloads" is precise and useless if the reader does not know what m is, what workload means here, or which index type is being configured.

Precision vs. recall. Smaller chunks improve precision: when a chunk matches, it is probably relevant. Larger chunks improve recall: they are more likely to contain the answer even if the match is looser. Neither extreme works. Pure precision retrieves a perfect sentence fragment the model cannot use. Pure recall retrieves a chapter that buries the answer in noise, which brings us to the third tension.

Index cost vs. quality. Every chunk becomes a vector in your index. A 200-page document chunked into sentences might produce 8,000 vectors. Chunked into pages, it produces 200. The sentence index is 40x larger, 40x more expensive to search, and 40x more storage. But it can answer questions the page index cannot. The cost is real, and the quality gap is real, and neither is a rounding error.

The semantic averaging problem

This is the mechanism behind the support-chatbot failure, and it is worth seeing precisely. An embedding model reads a passage of text and produces a single vector in a high-dimensional space. That vector represents the meaning of the passage as a point. Two passages with similar meaning produce vectors that are close together. A query is embedded the same way, and retrieval is a nearest-neighbor search.

When you embed a passage that discusses one topic, the vector lands in a region of the space that corresponds to that topic. When you embed a passage that discusses twelve topics, the vector lands somewhere in between all twelve, in a region that may not correspond closely to any of them. It is a centroid, an average, a point equidistant from twelve ideas and identical to none.

This is not a limitation of the embedding model. It is a mathematical consequence of representing multiple meanings with one point. The model is doing exactly what it should: compressing the passage into a single representation. The problem is that you asked it to compress too much.

The "lost in the middle" problem

You might think: context windows are enormous now. GPT-4o handles 128K tokens, Claude handles 200K. Why not just stuff everything in and let the model sort it out?

Because models do not attend to long contexts uniformly. Research from Stanford and elsewhere has shown that LLMs are measurably less reliable at using information placed in the middle of a long context compared to information at the beginning or end. A 2023 study by Liu et al. ("Lost in the Middle: How Language Models Use Long Contexts") found that performance on a multi-document QA task degraded by up to 20 percentage points when the relevant passage was placed in the middle of the context versus at the top. The problem is not that the model cannot find it. It is that it finds it less reliably, and in production, "less reliably" means "wrong 20% of the time for some users, and you cannot predict which ones."

Chunking is how you ensure the relevant text is not buried. When retrieval selects three focused, relevant chunks and places them at the top of the context, the model attends to them fully. When retrieval fetches one enormous document and pads the context to 50,000 tokens, you are relying on a statistical attention pattern you cannot control.

The running example

Throughout this course, we will chunk the same document with every strategy: a technical API reference for a fictional vector database called VectorForge. The VectorForge documentation contains:

• Narrative prose explaining concepts (what HNSW is, how quantization works)
• Hierarchical headings (H1 product sections, H2 features, H3 parameters)
• Parameter tables with columns for name, type, default, and description
• Code examples in Python showing how to create indexes, insert vectors, and query
• Cross-references between sections ("See the quantization section for memory trade-offs")

This is deliberately messy. Real documentation is never uniform. A chunking strategy that handles only clean paragraphs will shatter tables, split code blocks mid-function, and lose the heading hierarchy. Seeing each strategy applied to the same material makes the trade-offs concrete, not theoretical.

What naive chunking does to this document

Here is the simplest possible chunker: split the text into fixed-size pieces by character count.

def naive_chunk(text: str, chunk_size: int = 500) -> list[str]:
    """Split text into fixed-size character chunks with no awareness of structure."""
    chunks = []
    for i in range(0, len(text), chunk_size):
        chunks.append(text[i:i + chunk_size])
    return chunks

Apply it to a fragment of the VectorForge docs:

sample = """## HNSW Index Configuration

The HNSW (Hierarchical Navigable Small World) algorithm builds a multi-layer
graph for approximate nearest neighbor search.

### Parameters

| Parameter | Type | Default | Description                              |
|-----------|------|---------|------------------------------------------|
| m         | int  | 16      | Max connections per node. Higher values   |
|           |      |         | improve recall but increase memory usage. |
| ef_construction | int | 200 | Size of the dynamic candidate list during |
|                 |     |     | index building. Higher = better quality.  |
| ef_search | int  | 50      | Size of the candidate list during search. |
|           |      |         | Higher = better recall, slower queries.   |

### Example

index = vf.create_index(

name="products",

metric="cosine",

algorithm="hnsw",

m=16,

ef_construction=200

)

chunks = naive_chunk(sample, chunk_size=300)
for i, chunk in enumerate(chunks):
    print(f"--- Chunk {i} ---")
    print(chunk)
    print()

Chunk 0 ends in the middle of "improve recall but increase memory usage." Chunk 1 begins with | | | | improve recall but increase memory usage. and cuts the table mid-row. Chunk 2 starts inside the code block. None of the three chunks is self-contained. None of them embeds well. The heading "HNSW Index Configuration" is in chunk 0 only, so chunks 1 and 2 have lost their section identity entirely.

Measuring the damage

How bad is a set of chunks? We can define a simple quality measure by counting boundary violations: cases where a chunk starts or ends in a way that suggests it was cut mid-thought.

import re

def measure_chunk_quality(chunks: list[str]) -> dict:
    """Count boundary violations across a set of chunks.

    A violation is any signal that a chunk was split mid-structure:
    - starts with a lowercase letter (mid-sentence)
    - starts with a table continuation row (no leading header or separator)
    - ends mid-sentence (no terminal punctuation, code fence, or blank line)
    - contains an unclosed code fence
    """
    violations = {
        "starts_mid_sentence": 0,
        "starts_mid_table": 0,
        "ends_mid_sentence": 0,
        "unclosed_code_fence": 0,
        "total_chunks": len(chunks),
    }

    for chunk in chunks:
        stripped = chunk.strip()
        if not stripped:
            continue

        first_char = stripped[0]
        if first_char.islower():
            violations["starts_mid_sentence"] += 1

        if stripped.startswith("|") and not re.match(
            r"\|[\s\-]+\|", stripped.split("\n")[0]
        ):
            first_line = stripped.split("\n")[0]
            if "---" not in first_line and not any(
                word[0].isupper() for word in first_line.split("|") if word.strip()
            ):
                violations["starts_mid_table"] += 1

        last_char = stripped[-1]
        if last_char not in ".!?:;`\"')]\n" and not stripped.endswith("```"):
            violations["ends_mid_sentence"] += 1

        fence_count = stripped.count("```")
        if fence_count % 2 != 0:
            violations["unclosed_code_fence"] += 1

    violations["total_violations"] = sum(
        v for k, v in violations.items()
        if k not in ("total_chunks",)
    )
    violations["violation_rate"] = (
        violations["total_violations"] / violations["total_chunks"]
        if violations["total_chunks"] > 0
        else 0.0
    )
    return violations

Run this on the naive chunks of the VectorForge sample and the violation rate is above 0.5. Every strategy in this course will be measured against this baseline. A good chunker does not just split text, it splits text at boundaries that preserve meaning.

Key takeaways

• A chunk is the atomic unit of retrieval: you cannot retrieve half a vector, so chunk boundaries determine what can and cannot be found.
• Embedding a multi-topic document into one vector produces a semantic average, a point close to no single topic, which is why page-level embedding fails on specific queries.
• Three tensions govern every chunking decision: self-containedness vs. specificity, precision vs. recall, and index cost vs. quality. No strategy resolves all three; the rest of the course is about choosing where to compromise.
• Even with 128K+ context windows, the "lost in the middle" effect makes retrieval of focused chunks more reliable than stuffing everything into a long prompt.
• Naive fixed-size splitting is the baseline: it is fast, universal, and produces chunks that split sentences, shatter tables, and orphan code blocks. Measuring boundary violations gives a concrete quality score to improve against.

The semantic averaging argument above rests entirely on what an embedding is, and this course takes that as given. If it does not yet feel obvious why one vector can be close to twelve topics and none of them, our Embeddings, End to End track builds that intuition from scratch and then goes considerably further — the full history, the training objectives, evaluation, and the modalities beyond text.