Where the knowledge comes from
Ask an assistant about the Treaty of Westphalia, the Mumbai local train map, or why sourdough needs feeding, and it answers like something that has read about all of it. It has. This lesson is about that reading — a process called pretraining — and it settles three everyday mysteries in one pass: why the model knows so much, why its knowledge stops at a certain date, and why it can never tell you where it learned something.
One exam question, trillions of times
Pretraining is mechanically simple to describe. Assemble a colossal pile of text — a filtered sweep of the public web, digitised books, encyclopedias, scientific papers, code, forums; trillions of words. Then run one exercise on repeat: show the model a fragment, hide the next word, and ask it to guess. Wrong guess? Nudge its internal settings a hair toward what would have made the right word likelier. Repeat, billions upon billions of times, for months, on warehouse-scale computing.
That's it. There is no curriculum, no fact-entry step, no team typing knowledge in. Every capability you have watched an assistant display was smuggled in through next-word prediction (lesson 1's point, now at full scale): predicting text about treaties forces absorbing history; predicting recipes forces absorbing cooking; predicting Python forces absorbing programming. The model ends pretraining as a compressed, blurry imprint of its library — billions of numerical settings encoding "how text about the world tends to go".
Frozen at the cutoff
A training run is a batch job, and it ends. Whatever date the library was assembled becomes the model's knowledge cutoff — its built-in picture of the world stops there, frozen, like an encyclopedia printed on that day. The model in your chatbot today was frozen months ago; it has no idea what happened last Tuesday, and — this is the treacherous part — it answers about last Tuesday anyway, plausibly, from patterns of how such things usually go. It will not feel out of date. Nothing about the fluency changes at the cutoff boundary.
Two things follow. Practically: for anything that changes — prices, laws, officeholders, product features — you want the assistant's search mode, where it looks up current pages and works from those (that mechanism is lesson 10's; the habit is preached properly in the everyday-work course). Conceptually: "why doesn't it just learn continuously, like a person?" — because the settings are only safe to change in that careful, controlled batch process; there is no lightweight way to slip one new fact into billions of interlocking numbers without risking damage elsewhere. Updating the model means (some version of) training again.
A blur, not a bookshelf
Now the deepest of the three mysteries: ask the model where it learned a fact and it cannot tell you — not won't, cannot. Pretraining did not file documents; it adjusted settings. Each of those trillions of words nudged billions of dials a hair, and what remains is the accumulated set of nudges — no shelf, no index, no page numbers. The right picture is not a librarian fetching a book; it is the blurred, merged impression of every book at once.
This is why a model asked for its sources so often invents plausible-looking ones — a journal citation with a real journal, a real-sounding author, and no existence. It isn't lying in any meaningful sense; it is doing the only thing it does — producing the text that plausibly follows "the source for this is…" — with no ledger to consult. (When an assistant's search mode cites a live link, that's different and real: the page was actually retrieved. The unreliable citations are the from-memory kind.) The blur also explains why famous facts are solid while obscure ones wobble: a fact repeated across ten thousand documents got ten thousand nudges and sits deep; a fact from one small page got one nudge and may surface half-remembered — fluently, of course. Lesson 7 builds on exactly this.
Try this in your chatbot: first ask "What's your knowledge cutoff?" — it will tell you. Then, with search/browsing off, ask about something announced last week. Watch how the answer feels — hedged, generic, or confidently constructed — and check it against reality. You have just calibrated your own detector for a model running on its frozen library.