Words as numbers

Last lesson left the model holding a row of token IDs — bare catalogue numbers, meaning nothing in themselves. Token #4571 doesn't contain "parcel" any more than a barcode contains a biscuit. So the machine's next move is the one this lesson unpacks: every token ID is swapped for a long list of numbers — called an embedding — and those numbers are where meaning actually lives. This sounds impossible. It is instead the single most beautiful idea in the whole field.

Position is meaning

Forget lists of numbers for a second; think of a map. Two numbers pin any spot on a city map. Three pin a spot in a room. An embedding pins each token to a spot in a space with thousands of directions — unvisualisable, but the principle is just "more map". And the placement follows one rule, learned from reading everything: tokens used in similar surroundings live near each other.

So "van", "truck" and "lorry" sit in a tight cluster — swap one for another in a sentence and the sentence still runs. "Parcel", "package" and "consignment" form their own cluster some distance away; "refund" lives in a different district entirely; "vanilla" is nowhere near "van" despite sharing three letters, because spelling was never the rule — company kept is the rule. The model was never told what a van is. It read enough text to notice that whatever "van" means, sentences treat it the way they treat "truck", and the geometry recorded that.

Directions mean things too

Here is where it gets genuinely uncanny: not just positions but directions in this space carry meaning. The classic party trick from the research that popularised embeddings: take the point for "king", subtract "man", add "woman" — and the nearest point to where you land is "queen". The arrow from "man" to "woman" is a reusable direction that means something like gender, and it works from other starting points too. There are arrows for plural-ness, for past-tense-ness, for country-to-capital. Nobody drew them. They condensed out of prediction practice, the way a river network condenses out of rain — because a model that can reuse "the -er direction" or "the France→Paris direction" predicts better than one memorising each case alone.

This is also your first honest answer to "does it understand anything?" — a question this course will keep sharpening rather than settle in one line. What the model has is relational: it knows how "parcel" behaves relative to "van", "fits", "heavy" — everything's position relative to everything else — without any of it being hooked to an actual parcel in the world. Philosophers argue about whether a rich-enough web of relations just is understanding. You now know precisely what is and isn't there, which puts you ahead of most of the argument.

Where this shows up in your life

Embeddings escaped the inside of chatbots and now power tools you already use. Semantic search — results that match your meaning when the words differ — is embedding distance at work: "parcel misplaced" finds the "lost shipment policy" document because the two land close on the map. "Chat with your PDF" tools find the relevant page the same way (that machinery is lesson 10). Recommendation engines embed products and people alike. Any time software seems to know that two differently-worded things are the same thing, you can now name the mechanism.

And one limit, flagged now because lesson 7 collects on it: closeness on the map is similarity, not truth. "The parcel fits" and "the parcel doesn't fit" embed near each other — same district, opposite claims. A machine whose native geography is plausibility has no neighbourhood called "correct". Hold that thought.

Try this in your chatbot: ask — "Give me five words that behave like 'van' in sentences, then five that contain the letters v-a-n but behave nothing like it. Explain the difference." It will walk you through meaning-space versus spelling-space, from the inside.