Tokenization — Core

Why

A tokenizer is a deterministic function from text to a list of integers, and each integer is an index into a fixed vocabulary that was built once during tokenizer training. The neural network never sees the original characters; it consumes only these IDs. As its own first internal step, the model maps each ID to a row of its embedding matrix, but that lookup happens inside the model, not in the tokenizer. So the raw-characters option is wrong because the model has no character-level access at all. The embedding-vector option confuses two separate stages: the tokenizer emits IDs, and turning IDs into vectors is the model's job. The bag-of-words option is wrong because token order is preserved exactly, and position matters enormously for attention. Because the vocabulary is frozen after training, the same input always yields the same IDs, which makes tokenization reproducible. This ID sequence is also what context-window limits and per-token billing are measured against, so it is the real currency of an LLM call.

Why

Word-level tokenization fails on out-of-vocabulary words: any word not seen during tokenizer training collapses to a single uninformative UNK token, and new names, misspellings, and morphological variants trigger this constantly. Subword tokenization avoids the problem because rare words split into smaller, frequently-seen pieces, and byte-level variants ultimately bottom out at individual bytes, so nothing is ever truly unknown. That graceful coverage is the decisive reason every major model since GPT-2 uses subword units. The whole-word-entry option is exactly the word-level approach that subword replaced, and it produces huge vocabularies with an OOV cliff. The shorter-than-characters option describes a real property but is not the reason subword won, and subword sequences are actually longer than word-level ones for common text. The hashing option is fiction: subword models still learn an embedding matrix with one row per vocabulary entry. So the core advantage is representing arbitrary input without information-destroying UNK tokens, which keeps the model robust to whatever text a user types.

Why

Transformer self-attention computes a score between every pair of positions, so its cost scales as the square of the sequence length. Going from about 2,700 tokens to about 11,000 tokens for the same article multiplies attention work by roughly sixteen times per layer, which is the dominant penalty of character-level tokenization. The embedding-matrix option is backwards: character vocabularies are tiny, only a few hundred entries, so the embedding matrix actually shrinks rather than grows. The retraining option is wrong because tokenizers of every granularity are trained once and then frozen, and granularity does not change that. The OOV option is also wrong, since character and byte level tokenizers essentially eliminate OOV because any text decomposes into known units. This quadratic scaling is precisely why subword tokenization is preferred: it keeps sequences short enough to control attention cost while still avoiding OOV. It is also why compression ratio, which sets how many tokens a document becomes, is such a central tokenizer design concern.

Why

Compression ratio is text bytes divided by the number of tokens, so a higher ratio means each token carries more bytes and the sequence is shorter for the same text. Larger vocabularies usually compress better because they contain merged entries for more multi-character sequences, letting common substrings become single tokens instead of several. That is why a 200K-entry vocabulary compresses English better than a 50K one. The decode-speed option confuses compression with throughput, which is a separate property driven by the implementation rather than by bytes per token. The special-token-fraction option is unrelated, since special tokens are a tiny handful and do not define compression. The embedding-quality option conflates the tokenizer with the model's learned representations, which are trained jointly but are not what this ratio measures. The tradeoff is that a larger vocabulary grows the embedding matrix linearly, so better compression is paid for in memory. Understanding this ratio directly explains context-window consumption and per-token cost.

Why

Every model family trains and ships its own tokenizer, so the vocabulary, merge rules, and pre-tokenization behavior all differ, and the same string therefore segments into different pieces with different IDs and counts. A model using a roughly 200K vocabulary will generally emit fewer tokens for English than one using a 128K vocabulary. The random-sampling option is wrong for standard inference: production tokenization is deterministic, and although Unigram supports subword-regularization sampling during training, default encoding is fixed. The recount-at-inference option invents a distinction that does not exist, since both families count deterministically the same way every time. The more-parameters option confuses model size with vocabulary and compression, but token count is a property of the tokenizer, not the parameter count. The practical upshot is that you must count tokens with the exact tokenizer of the model you are calling, since those counts drive context limits and billing. This is exactly why token-counting tools are model-specific.