Key takeaways

The through-line

Tokenization is the one layer of the stack that is entirely a design choice, and every choice in it is a trade you cannot undo later. The vocabulary is frozen when the tokenizer is trained, before the model sees a single gradient — and everything downstream, from context cost to multilingual fairness, inherits it.

The ideas that carry

• Sequence length is bought with vocabulary size. Character-level tokenizers keep vocabularies tiny and make sequences enormous; word-level does the reverse and shatters on the first unseen word. Subword tokenization exists because attention is O(N²) and OOV tokens are information black holes — it is the negotiated settlement between two failure modes, not an optimum.
• BPE, WordPiece and Unigram differ in what they maximise, not in what they produce. BPE merges the most frequent adjacent pair; WordPiece merges the pair that most improves corpus likelihood; Unigram starts from a large vocabulary and prunes downward. Similar-looking vocabularies, different biases — and the bias shows up on rare words and morphology.
• Byte-level BPE is why modern models never emit <UNK>. Operating over the 256 byte values rather than Unicode characters guarantees any input is representable. The cost is that non-Latin scripts fragment into many more tokens.
• That fragmentation is a fairness problem with an invoice attached. The same sentence in Hindi or Thai can cost several times the tokens of its English translation — so speakers of those languages pay more, wait longer, and get less usable context for identical content.
• Special tokens and chat templates are part of the tokenizer, not the prompt. Getting a template wrong doesn't throw an error; it silently produces text the model was never trained to continue, which reads as the model being bad.
• Tokenizers are an attack surface. Adversarial and pathological inputs — repeated Unicode, glitch tokens, unusual whitespace — hit boundary behaviour the model never saw in training.

What you can now do

Count and cost tokens before shipping a feature. Read a tokenizer config and predict how it will behave on your corpus. Train a tokenizer from scratch and justify its vocabulary size. Diagnose the class of bug where the model is fine and the encoding is wrong.

If one thing sticks

The model never sees your text. It sees integers a tokenizer chose for it — and every bad choice in that step is invisible in the loss curve and permanent in the deployed system.