Nybble™
All courses

Guided Course

understandtheory

Tokenization, Inside and Out

BPE, WordPiece, Unigram, SentencePiece, tiktoken — every algorithm, every tradeoff, every production implication.

0/14
Start course
14 lessons
  1. 1. From text to numbers5 min
  2. 2. The tokenizer pipeline6 min
  3. 3. Byte Pair Encoding (BPE)8 min
  4. 4. Byte-level BPE8 min
  5. 5. WordPiece — likelihood-driven subword merging7 min
  6. 6. Unigram — top-down probabilistic segmentation7 min
  7. 7. SentencePiece and tiktoken — the two dominant tokenizer implementations10 min
  8. 8. Vocabulary design7 min
  9. 9. Special tokens and chat templates6 min
  10. 10. Multilingual tokenization and fairness6 min
  11. 11. Token counting and cost estimation9 min
  12. 12. Tokenizer fragility and adversarial inputs10 min
  13. 13. Training a tokenizer from scratch9 min
  14. 14. The tokenizer-free future13 min

About Nybble™

The AI space moves fast.
Nybble™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack
Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack
Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack