The problem embeddings solve

A bicycle shop runs a help desk. Its knowledge base has four articles, and for the sake of seeing every number in this lesson by hand, those four articles are one sentence each.

• D1 — the rear brake squeaks when the rim is wet
• D2 — a squeaking brake usually means the rim needs cleaning
• D3 — the chain squeaks when it is dry and needs oil
• D4 — clean the rim with alcohol to stop brake noise

A customer types: why does my brake squeal in the rain.

You know the answer. It is D1, with D2 close behind. Now watch what a search engine built on words does with it.

Where word matching breaks

Split the question into words and check which of them appear anywhere in the knowledge base. There are eight: why, does, my, brake, squeal, in, the, rain. Exactly two of them occur in any article: brake and the.

Squeal does not appear. The articles say squeaks and squeaking. Rain does not appear either; D1 says wet. The two words that carry the entire question — the symptom and the condition — are invisible to a system that compares strings.

That leaves brake and the to do all the work, and the is useless: it appears in all four articles, so it cannot distinguish between them. Any decent keyword scorer knows this and gives it a weight of exactly zero, for reasons we will derive properly in Lesson 6. So the entire query collapses to the single word brake — which appears in D1, D2 and D4 alike.

Score those four articles with classical TF-IDF and this is what comes back:

D1  0.0320   matched: brake, the
D2  0.0320   matched: brake, the
D4  0.0320   matched: brake, the
D3  0.0000   matched: the

Three-way tie. The system cannot tell that D1 is the answer, because from where it is standing D1, D2 and D4 are indistinguishable. It is not ranking badly; it has no information to rank with.

This is the lexical gap: the distance between the words a person uses and the words a document happens to contain. It is not an edge case. It is the normal condition of search. People ask about squealing when the manual says squeaking, about rain when the manual says wet, about my laptop won't turn on when the article is titled Diagnosing power delivery faults. Every synonym, every paraphrase, every difference in register opens the gap a little wider.

You can paper over some of it. Stemming collapses squeaks and squeaking to squeak, which helps here and does nothing for squeal. A synonym dictionary can be told that rain implies wet, and then must also be told about damp, drizzle, puddle, humid, and every other word anyone might use, in every language you serve, forever. These are patches on a representation that was never able to carry meaning in the first place.

The idea

Suppose that instead of storing each article as a bag of words, you stored it as a point in space — a list of a few hundred numbers, chosen so that articles about similar things land near each other.

D1 and D2 are both about a squeaking brake and a wet or dirty rim, so they land close together. D3 is about a dry chain, a different failure with a different fix, so it lands somewhere else. Now the customer's question gets turned into a point in the same space, by the same procedure. Squeal is a noise complaint about a brake in bad conditions, and so is D1, so the question lands near D1 — not because they share a word, but because they mean nearly the same thing.

Search stops being string comparison and becomes geometry: turn the query into a point, then return the points nearest to it.

That list of numbers is an embedding. The model that produces it is an embedding model. Everything else in this course is detail on top of those two sentences: how the numbers get chosen, why the choosing has been redesigned five or six times since 2013, and how you decide which of today's models to actually use.

What an embedding is, precisely

An embedding is a fixed-length list of real numbers that represents a piece of data, positioned so that geometric closeness corresponds to semantic similarity.

Three parts of that sentence are load-bearing.

Fixed-length. D1 is nine words, D4 is ten, a customer's question might be forty. All three become vectors of exactly the same length — 384, or 768, or 1536, depending on the model. This is what makes them comparable at all, and it is also the first thing you give up: a fixed-size container cannot hold arbitrarily much, which is why long documents get split before they are embedded (our Chunking Strategies for RAG track is entirely about that decision).

Real numbers. Not counts, not flags. Continuous values, usually somewhere between about −1 and 1, that can express degrees of things. This is why an embedding can place squeal near squeak without either word having been listed as the other's synonym.

Geometric closeness corresponds to semantic similarity. This is the promise, and it is a trained property, not a mathematical guarantee. An embedding model is good precisely to the extent that this correspondence holds on your data. Lesson 4 makes that idea sharp enough to measure, and Lesson 22 turns it into numbers you can put in a dashboard.

Two words worth separating now

People say "embedding" for two related but different objects, and conflating them causes real confusion later.

A token embedding is a row in a lookup table inside a language model. Token 4,921 has a vector; that vector is fetched and fed into the first layer. It is an input to a computation. Our The Transformer Architecture track covers that table in detail.

A text embedding is the output of a whole model that has read an entire sentence or paragraph and compressed it into one vector, meant to be stored, indexed and compared. That is what a search system stores.

They share a name, a shape and an ancestry, and they do different jobs. Lesson 11 walks the path from one to the other, and Lesson 12 explains why the model that produces a good text embedding is usually not the model that writes your chat replies.

What this course covers

Five sections, each answering a question you can state in one line.

What an embedding is (Lessons 1–4) — the idea, the geometry, your first real vectors, and what separates a good embedding space from a bad one.

Where they came from (Lessons 5–10) — one-hot, bag of words, TF-IDF, word2vec, GloVe, fastText, ELMo, BERT and Sentence-BERT. Each with its formula, an example small enough to check by hand, its honest advantages, and the specific failure that forced the next technique into existence. This is the section most treatments skip, and it is the one that makes the rest predictable rather than magical.

Modern models and LLMs (Lessons 11–16) — where embeddings live inside a transformer, how an embedding model differs from a chat model, how today's embedders are trained, how to fine-tune one on your own data end to end, and working code.

Beyond text (Lessons 17–19) — images, audio, video, graphs, users, items and code.

Choosing, measuring and shipping (Lessons 20–23) — how to pick a model, how to read MTEB without being misled by it, the retrieval metrics that decide whether you were right, and the production practices that keep a vector index honest over time.

By the end you should be able to look at the bike shop's help desk, pick a model with reasons, measure whether it works, and know what to do when it stops working.