## The tokenizer pipeline Every tokenizer — whether you're using tiktoken, SentencePiece, or the HuggingFace `tokenizers` library — executes five stages in sequence. The input is a raw string; the output is a list of integer IDs. Each stage transforms the data and passes it to the next. ## Stage 1: Normalization The normalizer applies Unicode and casing transforms to the raw input before any splitting occurs. Common operations: - **Unicode normalization** — NFKC or NFC. NFKC maps compatibility characters to their canonical forms: the ligature "ﬁ" (U+FB01) becomes "fi", the full-width "Ａ" (U+FF21) becomes "A", the superscript "²" (U+00B2) becomes "2". This reduces surface variation that would otherwise produce distinct tokens for semantically identical text. - **Lowercasing** — "The" and "the" become the same token. BERT's uncased models do this; most GPT-family tokenizers do not (case is semantically meaningful in code and proper nouns). - **Accent stripping** — "café" becomes "cafe". Used in some multilingual models to collapse diacritical variants. - **Whitespace normalization** — collapse runs of spaces, strip leading/trailing whitespace. Not every tokenizer applies all of these. GPT-2 and GPT-4 apply no normalization at all — the tokenizer sees exactly the bytes the user typed. BERT applies NFKC + lowercasing + accent stripping. The choice depends on what the model needs to distinguish. ```python from tokenizers import Tokenizer, normalizers normalizer = normalizers.Sequence([ normalizers.NFKC(), normalizers.Lowercase(), normalizers.StripAccents(), ]) raw = "Ｔｈｅ café — naïve ﬁndings²" normalized = normalizer.normalize_str(raw) print(f"Before: {raw}") print(f"After: {normalized}") # After: the cafe — naive findings2 ``` ## Stage 2: Pre-tokenization The pre-tokenizer splits the normalized string into chunks (typically words or word-like units) that the subword algorithm will process independently. This prevents merges from crossing word boundaries — without pre-tokenization, BPE could merge the "e" at the end of "the" with the space before "cat", producing a nonsensical cross-word token. Two major approaches: ### Regex-based splitting (GPT family) GPT-2 through GPT-4o use a regex pattern to split text before BPE runs. The pattern defines what constitutes a "word" for the purpose of bounding merges. GPT-2's pattern (Radford et al. 2019): ```python import regex GPT2_PATTERN = r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""" chunks = regex.findall(GPT2_PATTERN, " The cat's out—42 times!") print(chunks) # [' The', ' cat', "'s", ' out', '—', '42', ' times', '!'] ``` This splits contractions ("cat's" → "cat" + "'s"), separates numbers, and treats punctuation as individual chunks. The leading space is attached to the word that follows it, not the word before — this is a deliberate design choice that lets the tokenizer distinguish "The" at the start of a sentence from " The" mid-sentence. GPT-4o's `o200k_base` uses an expanded regex with additional rules for handling whitespace runs and mixed-script text. ### Whitespace-based splitting (SentencePiece) SentencePiece (Kudo and Richardson 2018) takes a different approach: it treats the input as a raw character stream and uses a special "▁" (U+2581, lower one-eighth block) character to mark word boundaries. Spaces are replaced by "▁" and become explicit characters in the vocabulary. ```python # SentencePiece conceptually transforms: # "The cat sat" → "▁The▁cat▁sat" # Then subword segmentation runs on the result. ``` This means SentencePiece can reconstruct the original text losslessly including whitespace, and it works on languages without spaces (Chinese, Japanese, Thai) without a language-specific word segmenter. ```python from tokenizers import pre_tokenizers # Whitespace-based pre-tokenizer ws_pre = pre_tokenizers.Whitespace() output = ws_pre.pre_tokenize_str("The cat's out—42 times!") print(output) # [('The', (0, 3)), ("cat's", (4, 9)), ('out', (10, 13)), # ('—', (13, 14)), ('42', (14, 16)), ('times', (17, 22)), ('!', (22, 23))] # Byte-level pre-tokenizer (GPT-2 style) bl_pre = pre_tokenizers.ByteLevel(add_prefix_space=False) output = bl_pre.pre_tokenize_str("The cat's out") print(output) # [('The', (0, 3)), ('Ġcat', (3, 7)), ("'s", (7, 9)), ('Ġout', (9, 13))] ``` Notice the "Ġ" character — that's GPT-2's representation of a space byte (0x20). More on this in Lesson 4. ## Stage 3: Subword tokenization The core algorithm runs on each pre-tokenized chunk independently, splitting it into subword units from the trained vocabulary. Three algorithms dominate: - **BPE** (Byte Pair Encoding) — Sennrich et al. 2016. Used by GPT-2, GPT-3, GPT-4, GPT-4o, Llama, Mistral, Claude. Merges are learned greedily by frequency; encoding applies merges in priority order. - **WordPiece** — Schuster and Nakajima 2012. Used by BERT and its derivatives. Similar to BPE but selects merges by maximizing the likelihood of the training corpus under a language model, not raw frequency. - **Unigram** — Kudo 2018. Used by T5, ALBERT, XLNet, Gemma. Starts with a large vocabulary and iteratively removes tokens whose removal least increases the overall corpus loss. Encoding finds the most probable segmentation using the Viterbi algorithm. BPE and WordPiece are constructive (start small, add tokens by merging). Unigram is reductive (start large, remove tokens by pruning). Lessons 3 and 4 cover BPE in detail. ```python from tokenizers import Tokenizer, models # BPE model (empty, untrained — just to show the type) bpe_model = models.BPE() # WordPiece model wp_model = models.WordPiece() # Unigram model uni_model = models.Unigram() ``` ## Stage 4: Post-processing The post-processor adds special tokens that the model architecture expects. These tokens are not part of the input text — they're structural markers inserted by the tokenizer. Common special tokens: - **BOS** (beginning of sequence) — `~~` in Llama, `<|endoftext|>` repurposed in GPT-2 - **EOS** (end of sequence) — `~~` in Llama, `<|endoftext|>` in GPT-2 - **PAD** — fills sequences to uniform length in batches - **SEP** — separates segments in BERT-style models (`[CLS] segment_A [SEP] segment_B [SEP]`) - **System tokens** — GPT-4 uses `<|im_start|>`, `<|im_end|>` to delimit chat messages; Llama 3 uses `<|begin_of_text|>`, `<|start_header_id|>`, etc. ```python from tokenizers import processors # BERT-style post-processing: add [CLS] at start, [SEP] at end bert_post = processors.TemplateProcessing( single="[CLS] $A [SEP]", pair="[CLS] $A [SEP] $B:1 [SEP]:1", special_tokens=[ ("[CLS]", 101), ("[SEP]", 102), ], ) ``` For GPT-family models, post-processing is minimal — typically just an optional BOS token. The chat template (system/user/assistant markers) is applied at a higher level by the serving framework, not by the tokenizer's post-processor. ## Stage 5: Decoding The decoder converts token IDs back to text. This stage must invert any transformations introduced by earlier stages — specifically, it must handle whitespace markers. BPE tokenizers with byte-level pre-tokenization (GPT-2 family) encode spaces as "Ġ" characters. The decoder maps these back to actual spaces: ```python from tokenizers import decoders # ByteLevel decoder: strips the Ġ prefix and restores spaces bl_decoder = decoders.ByteLevel() ``` SentencePiece-based tokenizers use "▁" to mark word boundaries. The decoder replaces leading "▁" characters with spaces: ```python # Metaspace decoder: replaces ▁ with spaces ms_decoder = decoders.Metaspace() ``` WordPiece uses "##" prefixes to mark continuation subwords ("playing" → "play" + "##ing"). The decoder strips these prefixes and concatenates: ```python # WordPiece decoder: strips ## and joins wp_decoder = decoders.WordPiece() ``` ## Assembling the full pipeline The HuggingFace `tokenizers` library exposes all five stages as composable components. Here's a complete pipeline assembled from parts: ```python from tokenizers import Tokenizer, models, normalizers, pre_tokenizers, processors, decoders, trainers # Build a BPE tokenizer from components tokenizer = Tokenizer(models.BPE(unk_token="")) tokenizer.normalizer = normalizers.Sequence([ normalizers.NFKC(), normalizers.StripAccents(), ]) tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False) tokenizer.post_processor = processors.ByteLevel(trim_offsets=False) tokenizer.decoder = decoders.ByteLevel() # Train on sample data trainer = trainers.BpeTrainer( vocab_size=1000, special_tokens=["", "~~", "~~"], min_frequency=2, ) tokenizer.train_from_iterator( ["The cat sat on the mat.", "The dog sat on the log.", "Cats and dogs are animals."], trainer=trainer, ) # Encode → decode round-trip text = "The cat sat" encoding = tokenizer.encode(text) print(f"Tokens: {encoding.tokens}") print(f"IDs: {encoding.ids}") print(f"Decoded: {tokenizer.decode(encoding.ids)}") ``` Each component can be swapped independently. You can use a BPE model with a Metaspace pre-tokenizer, or a Unigram model with a ByteLevel pre-tokenizer — though not all combinations are meaningful. The point is that "tokenizer" is not a monolithic black box; it's a pipeline of five well-defined stages, each with clear inputs and outputs.

Byte Pair Encoding (BPE)

Read this lesson