From text to numbers

A tokenizer converts a string of text into a sequence of integers. Each integer is an index into a fixed vocabulary — a lookup table mapping substrings to IDs that was built once during tokenizer training and never modified afterward. The neural network receives these integer sequences as input; it has no access to the original characters.

Three granularity levels

Tokenization operates at one of three levels: character, word, or subword. Each choice fixes the vocabulary size and the sequence length the model must process.

Character-level tokenization

The vocabulary consists of individual characters. For English ASCII, that's roughly 128 entries; extending to full Unicode pushes it to several thousand, but a typical character-level tokenizer constrains itself to ~256 byte values or the few thousand characters observed in training data.

The consequence is sequence length. The word "tokenization" becomes 12 tokens. A 2000-word article becomes roughly 10,000–12,000 tokens. Transformer attention has O(N²) cost in sequence length, so a 12,000-token sequence requires 144 million attention score computations per layer — compared to roughly 4 million for a 2,000-token subword sequence covering the same text.

Character-level models exist (Xue et al. 2022, ByT5), but they pay a steep computational price and have largely been superseded.

Word-level tokenization

Split on whitespace and punctuation. Each unique word form gets its own vocabulary entry. English corpora produce vocabularies in the range of 200,000–500,000 entries after lowercasing, and much more without it.

The fatal problem is out-of-vocabulary (OOV) words. A word-level tokenizer trained on a 2020 corpus has no entry for "ChatGPT" — it maps to an <UNK> token that carries zero information. Misspellings, morphological variants ("tokenizing" vs. "tokenized" vs. "tokenizes"), and code identifiers all produce OOV tokens. In practice, 2–5% of tokens in unseen text hit OOV with a word-level vocabulary, and each one is an information black hole.

Subword tokenization

Subword methods split text into pieces that range from individual characters (or bytes) up to full words. Common words like "the" get a single token. Rare words like "tokenization" might split into "token" + "ization" — two tokens that individually appear frequently in training data.

Vocabulary sizes range from 32,000 (GPT-2, many BERT variants) to 200,000 (GPT-4o's o200k_base). Sequence lengths for typical English text land at roughly 1.3–1.5 tokens per word — short enough for efficient attention, long enough to preserve granular meaning.

Subword tokenization solves OOV completely: any input, no matter how unusual, decomposes into known subword units (and in byte-level schemes, ultimately into individual bytes). This is why every major language model since GPT-2 uses subword tokenization.

The tokenizer is frozen

The vocabulary and merge rules are determined by a separate training process that runs before model pre-training begins. Once the tokenizer is trained, it is frozen: the same vocabulary and rules are used for pre-training, fine-tuning, and inference. Changing the tokenizer would invalidate the learned embedding matrix, which has one row per vocabulary entry.

This means the tokenizer is a fixed function: text → list[int]. It is deterministic — the same input always produces the same output. And it is model-specific — different models use different tokenizers with different vocabularies.

Same text, different tokens

Because each model ships its own tokenizer, the same input string produces different token sequences — different IDs, different token counts, different segmentations.

import tiktoken

enc_gpt4o = tiktoken.encoding_for_model("gpt-4o")
tokens_gpt4o = enc_gpt4o.encode("Tokenization splits text into subword units.")
print(f"GPT-4o (o200k_base): {len(tokens_gpt4o)} tokens")
print(f"  IDs: {tokens_gpt4o}")
print(f"  Pieces: {[enc_gpt4o.decode([t]) for t in tokens_gpt4o]}")

enc_gpt4 = tiktoken.encoding_for_model("gpt-4")
tokens_gpt4 = enc_gpt4.encode("Tokenization splits text into subword units.")
print(f"GPT-4 (cl100k_base): {len(tokens_gpt4)} tokens")
print(f"  IDs: {tokens_gpt4}")
print(f"  Pieces: {[enc_gpt4.decode([t]) for t in tokens_gpt4]}")

Running this produces different token counts and different segmentations despite identical input. GPT-4o uses the o200k_base encoding with a 200,000-token vocabulary; GPT-4 uses cl100k_base with 100,000 tokens. The larger vocabulary tends to merge more aggressively, producing fewer tokens for common English text.

To compare against a SentencePiece-based tokenizer (the family used by Llama 3, Mistral, and Gemma):

from transformers import AutoTokenizer

llama_tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
tokens_llama = llama_tok.encode("Tokenization splits text into subword units.")
print(f"Llama 3: {len(tokens_llama)} tokens")
print(f"  IDs: {tokens_llama}")
print(f"  Pieces: {llama_tok.convert_ids_to_tokens(tokens_llama)}")

Llama 3 uses a byte-level BPE tokenizer with a 128,256-entry vocabulary. On this input it will produce a different token count than either GPT-4 variant. The difference is not a rounding error — it directly affects context window consumption, inference cost (which is priced per token), and the positions the model's attention mechanism operates over.

Vocabulary size and compression ratio

The compression ratio measures how efficiently a tokenizer converts text into tokens: compression_ratio = len(text_bytes) / num_tokens. Higher is better — each token encodes more bytes, so sequences are shorter.

Typical compression ratios on English text:

GPT-2 (r50k_base, 50,257 vocab) — ~3.5 bytes/token
GPT-4 (cl100k_base, 100,256 vocab) — ~4.0 bytes/token
GPT-4o (o200k_base, 200,019 vocab) — ~4.5 bytes/token
Llama 3 (128,256 vocab) — ~4.2 bytes/token

Larger vocabularies generally achieve better compression because they can represent more common multi-character sequences as single tokens. But the embedding matrix grows linearly with vocabulary size (each entry is a d_model-dimensional vector), so there's a memory tradeoff.

import tiktoken

text = "The transformer architecture processes input sequences through layers of self-attention and feed-forward networks."
text_bytes = len(text.encode("utf-8"))

for model_name in ["gpt-4o", "gpt-4", "gpt-3.5-turbo"]:
    enc = tiktoken.encoding_for_model(model_name)
    n_tokens = len(enc.encode(text))
    ratio = text_bytes / n_tokens
    print(f"{model_name}: {n_tokens} tokens, {ratio:.2f} bytes/token")

The two components of a tokenizer

Every tokenizer has exactly two parts:

The algorithm — how text is segmented (BPE, WordPiece, Unigram). This is the procedure that decides where to split.
The trained vocabulary — the specific set of tokens and merge rules learned from a training corpus. This is a data artifact, not code.

The algorithm is shared across many models (GPT-2, GPT-3, GPT-4, and Llama all use BPE). What makes each tokenizer unique is the vocabulary produced by training on different corpora with different hyperparameters. Two BPE tokenizers trained on different data will segment the same input differently.