Key takeaways

The through-line

Every technique in this course, from one-hot in the 1970s to a decoder-based embedder in 2026, answers the same question: what should be near what? One-hot answered "nothing is near anything". TF-IDF weighted that answer by rarity. word2vec derived it from company, BERT made it depend on context, and Sentence-BERT finally trained the space directly for it.

The ideas that carry

• An embedding is a coordinate system for meaning — fixed-length, continuous, and positioned so that geometric closeness is semantic similarity. That last property is trained, not guaranteed.
• Nobody chooses the axes. You define what closeness should mean and let optimisation find coordinates that satisfy it, which is why no single dimension means anything on its own.
• Alignment and uniformity pull against each other, and every modern objective sits at a deliberate point between them. A model that maps every input to one point has perfect alignment and is useless.
• Anisotropy is the failure you will actually meet. Raw language-model vectors crowd into a narrow cone, so every cosine comes out near 0.9 and no threshold means anything.
• TF-IDF gives "the" a weight of exactly zero — not heuristically, but because log(N/N) is zero. The mathematics deletes stopwords for you.
• Negative sampling replaced a 50,000-way softmax with six binary decisions, and that question — is this pair real, or a distractor? — is still the objective every embedding model is trained on.
• *GloVe's insight is that the ratio of co-occurrence probabilities carries meaning* where the probabilities themselves mostly carry frequency.
• Static embeddings die on polysemy. One vector for bank averages the river and the money, and no amount of data fixes a representation that cannot express ambiguity.
• Raw BERT was worse than averaged GloVe at sentence similarity — 54.81 against 58.02, and the [CLS] vector worst of all at 29.19. Optimising for one thing and evaluating on another is the whole lesson.
• The bi-encoder is why vector databases exist. A cross-encoder is more accurate per pair and produces nothing you can store; 65 hours becomes 5 seconds.
• Batch size is negative count, which is why embedding models train with batches that look absurd.
• Query and document are different roles. Skipping the prefixes raises no error and quietly costs several points.
• Matryoshka makes dimension a deployment choice rather than a training one, and int8 quantization is a 4x saving for well under a point.
• The coupling constraint is absolute. Two models produce two unrelated coordinate systems; comparing them is not less accurate, it is meaningless — and it fails silently.
• Fine-tuning is a data problem, not a training problem. The run is thirty lines; the pairs are the work — and they are usually already in your click logs.
• Fine-tune only when retrieval is the measured bottleneck. The same procedure that nearly doubles recall on a corpus the base model was failing at buys almost nothing downstream on one where retrieval already worked.
• Skip the top ten results when mining hard negatives. They are the likeliest unlabelled positives, and training on them teaches the model that a right answer is wrong — silently, without moving the loss.
• A fine-tuned model at 64 dimensions can beat its own base model at 768. Fine-tuning does not only raise quality; paired with Matryoshka it moves the whole cost curve.
• Fifty labelled queries from your own logs beat any leaderboard, because a leaderboard measured someone else's corpus.

What you can now do

Explain what an embedding is and why it works, from one-hot through TF-IDF, word2vec, GloVe, fastText, ELMo, BERT and Sentence-BERT to today's decoder-based embedders — with each formula and the specific failure that forced the next technique. Fine-tune an embedding model on your own corpus end to end: harvest or generate the pairs, mine hard negatives without poisoning them, train with the right loss and batch sampler, and prove the result against the base model. Read a real embedding with sentence-transformers, and build search, clustering, deduplication, classification and anomaly detection on one array of vectors. Choose a model by elimination rather than by leaderboard position, read MTEB without being misled by it, and measure the choice with recall@k, MRR and nDCG. Recognise embeddings outside text — images, audio, video, graphs, users, items and categorical columns — and ship the whole thing with model identity stored beside every vector.

If one thing sticks

Start with a small, well-established model. Write down fifty real queries and the documents that should answer them. Measure, then read the failures one by one — the aggregate tells you that something is wrong and only the individual cases tell you what. The models will keep changing; those fifty queries are what tell you whether the change helped.