Tokens: how AI reads
Ask a strong AI assistant how many r's are in "strawberry" and there is a decent chance it gets it wrong — a system that can draft contracts, failing a task from a primary-school worksheet. This is not a bug someone forgot to fix. It is a window into something basic: the model never sees letters. Understanding what it sees instead explains this failure, several others, and — as a bonus — why AI pricing is quoted the way it is.
Chunks, not words, not letters
Before any text reaches the model, a pre-processing step chops it into tokens — chunks somewhere between a letter and a word. Common words land as one token each; rarer or longer words get split into pieces; punctuation and even the spaces get bundled in.
Our running sentence — "The parcel doesn't fit in the van because it is too big." — comes out as roughly fourteen tokens: mostly whole words, with "doesn't" split in two ("does" + "n't"). Why chunks? A fixed menu of a hundred-thousand-ish chunks can spell anything — every typo, every new brand name, every Hinglish sentence — while staying small enough to be workable. Frequency earns wholeness: "the" is one token because English uses it constantly; "Nashik" might be two or three. As a rule of thumb, a token is about three-quarters of an English word.
Here is the part that explains the strawberry problem: after this chopping, each token is just an ID number in that menu. "Strawberry" might arrive as two tokens — "straw" and "berry" — and nothing about those two IDs says anything about the letters inside them. Asking the model to count r's is like asking you how many r's are in a word someone spoke aloud in a language you know only by sound. It can often still answer — it has read about spelling — but it is recalling, not looking, because there is nothing to look at. Letter-counting, reversing strings, rhyming precisely, arithmetic on long numbers (chopped into arbitrary digit-chunks): one explanation covers the whole family of clumsy failures.
Why you pay by the token
Everything the model reads and writes is tokens, so tokens are the natural unit of work — and of billing. API prices are quoted per million tokens; your chatbot subscription's usage limits are token budgets wearing a friendlier costume; and the "context window" you'll meet in lesson 8 is a token limit too. Two practical consequences fall out. Output costs more than input (roughly three to five times more, across vendors as of mid-2026), because generating a token takes a full run of the loop from lesson 1 while reading is comparatively cheap — one honest reason assistants aren't penalised for being concise. And non-English text often costs more for the same meaning: token menus are built from mostly-English training data, so a Hindi or Marathi sentence shatters into more, smaller chunks — same sentence, more tokens, more cost. If you work multilingually, you have been paying a quiet tokenization tax without knowing its name.
The takeaway that transfers
Tokens are the first instance of this course's recurring pattern: an everyday AI mystery — "how can something so smart be so bad at that?" — dissolving once you see the machine's actual interface with the world. The model is not a mind reading your words; it is a machine reading chunk-IDs. Keep that firmly in view and the strawberry failure stops being spooky and becomes exactly what you'd expect.
If the chopping algorithms themselves hook you — how the menu gets built, why "doesn't" splits where it does, what this does to code — the Pro shelf's Tokenization, Inside and Out goes all the way down. Entirely optional here.
Try this in your chatbot: ask it to count the r's in "strawberry", then ask it to spell "strawberry" letter by letter first and then count. The second usually works — once the letters exist as separate tokens in the conversation, there is finally something to look at. You've just worked around a real architectural limit, using its own explanation.