Key takeaways

The through-line

A chunk is the atomic unit of retrieval, and every decision in this course follows from that one fact. You cannot retrieve half a vector. Whatever you embedded as one unit comes back whole or not at all — so the boundary you draw decides what can be found, how precisely, and whether it is still useful once it arrives in the model's context.

The ideas that carry

An embedding of twelve topics is a point close to none of them. Semantic averaging is not a model defect; it is what compressing many meanings into one vector has to do.
Three tensions recur in every strategy — self-containedness against specificity, precision against recall, and index cost against quality. No strategy dissolves them; each one picks a place to stand.
150 to 512 tokens is the useful band for most corpora. Below it chunks stop making sense alone; above it the averaging problem returns.
Overlap is the only defence against a boundary you cannot place perfectly, and 10-20% buys back most boundary-spanning facts for a quarter more index.
Measure in tokens, not characters. Characters are a proxy that drifts with language and code.
Recursive chunking is the right default because it degrades gracefully: it uses structure when structure exists and falls through to finer separators when it does not.
Structure that the author already wrote is free signal. Heading hierarchies, clause numbers and function definitions are boundaries someone else has already justified.
Metadata is what turns a chunk into a citation — and it is the only carrier of a heading once the text no longer contains it.
The unit you embed does not have to be the unit you send to the generator. Sentence-window and parent-child both exist to break that assumption, and it is the single most useful idea in the second half of the course.
Late chunking inverts the order — embed the whole document, then split the token embeddings — so every chunk's vector carries context from outside its own boundaries.
Non-determinism is a chunking bug, not a chunking style. If an LLM picks the boundaries, two rebuilds produce two different indexes and your diffs stop meaning anything. Let the model choose the strategy; keep the splitter deterministic.
Some documents should not be chunked at all. A three-sentence FAQ answer has completeness as its only asset, and splitting it destroys exactly that.
Insert before you delete. Cascading invalidation that removes the old chunks first leaves a window where the correct answer is simply absent from the index.
Content-hash ids make re-indexing idempotent, which is what turns a nightly rebuild from a full re-embed into a diff.
A wide spread in chunk size is not a defect. It is meaning refusing to be uniform, and the evaluation numbers say whether that costs you anything.

What you can now do

Pick a chunking strategy from the shape of your corpus rather than from a blog post, and defend the choice with numbers. Implement fixed-size, sentence, sliding-window, recursive, structure-based and semantic chunking from scratch, and know which failure each one is buying its way out of. Handle the content that breaks naive splitters — code, tables, headings, images. Decouple retrieval units from generation units when precision and context pull in opposite directions. Build an evaluation set of 50-100 labelled pairs, measure keyword coverage, concept integrity, coherence and size consistency, and run a production lifecycle that re-embeds only what actually changed.

If one thing sticks

Start with recursive chunking at 400-512 tokens and 10-15% overlap, then measure. Almost every corpus is served well enough by that, and every escalation past it — semantic, multi-modal, late chunking, routing — costs real money and real complexity. Let the evaluation numbers, not the newest technique, decide when you climb the next step.

← Previous Next →