How "it" finds its meaning

Time to collect the sentence from lesson 1. "The parcel doesn't fit in the van because it is too big." You read "it" as the parcel, instantly and without noticing a decision. Now swap one word: "The parcel doesn't fit in the van because it is too small." Same pronoun, same position, same grammar — and "it" has silently jumped to the van. Nothing about the word "it" changed; the physics of the situation reached in and reassigned it. To continue such a sentence correctly, a next-word predictor must somehow do what you just did. The mechanism that lets it is called attention, and it is the breakthrough the whole modern AI era is built on.

Every word looks around

The embeddings from lesson 3 gave each token a meaning-point — but a fixed one, the same "it" every time. That cannot be the finished article, because words take their meaning from their neighbours: "bank" by the river is not "bank" with the queue, and our "it" is sometimes a parcel, sometimes a van.

Attention is the fix, and one sentence states it: as the model processes text, every word gets to look at every other word and pull in meaning from the ones that matter to it. Working through "it" in our sentence, the model — figuratively — lets "it" survey the whole line: "parcel"? relevant. "van"? relevant. "fit", "doesn't", "big"? all bear on the question. The word "the"? Barely. Each earns a relevance weight, and "it" absorbs a weighted blend of its neighbours' meanings. In the "too big" sentence, the blend leans strongly toward the parcel's point on the meaning map — big things fail to fit into containers; in the "too small" version the same survey leans toward the van — small containers fail to hold things. After attention, "it" is no longer the bland dictionary "it": it has become contextually parcel-flavoured or van-flavoured, and the next-word machinery downstream inherits that.

Notice what performed the physics: nothing. There is no physics module. The model has simply read millions of sentences about things fitting and not fitting into other things, and the attention weights that predicted those sentences correctly are the ones that survived training. Worldly common sense, distilled into learned habits of where to look.

Many surveys at once, stacked deep

Two upgrades take this from mechanism to machine. First, the model doesn't run one survey — it runs dozens in parallel on the same sentence, each specialised through training: one tracking who-did-what-to-whom, one tracking pronoun references, one watching nearby grammar, one linking distant topics. (The specialisations aren't assigned; they emerge, and researchers discover them afterwards like naturalists finding species.) Second, the whole exercise is stacked in layers — the output of one round of surveys becomes the input to the next, so early layers resolve local grammar, middle layers assemble relationships and references, deep layers hold something like the sentence's overall sense. By the top of the stack, every word's representation has been marinated in everything relevant around it. This — attention plus stacking, the architecture called the transformer — is what the T in "GPT" stands for, and it is the machine you have been talking to all along.

Why this was the breakthrough

Before 2017, language systems read left to right, squeezing everything remembered into a cramped running summary — by word forty, word four was mush. Attention abolished the queue: every word connects directly to every word, whether adjacent or a page apart, and all of it computes in parallel, which is exactly what modern chips are good at. Direct long-range connections plus massive parallelism is why models could suddenly be trained on the whole internet — and that scale-up is the era you're living in. When curiosity bites, this shelf's free Pro courses — Attention Mechanisms and The Transformer Architecture — rebuild today's picture with the actual numbers, and this lesson is honestly most of the intuition they need.

Try this in your chatbot: give it both versions of the parcel sentence and ask "In each sentence, what does 'it' refer to, and how do you know?" Watch it explain fitting and containment — the learned worldly knowledge — and know that you now know the name of the machinery underneath.