Embeddings — Core

Dense vectors, word2vec/GloVe/fastText, static vs contextual, sentence embeddings and SBERT, bi- vs cross-encoders, similarity metrics, contrastive learning, and Matryoshka.

Medium50 questions

Sample questions

What is the core idea of representing a word or sentence as an embedding?

A lookup id that indexes into a fixed dictionary of definitions
A compressed image of the token rendered as pixels
A one-hot vector with a single 1 marking the token's position
A dense vector whose geometry reflects meaning; similar items sit close

Why

An embedding maps an item to a dense, continuous vector so that semantic similarity becomes geometric closeness (small distance / high cosine). It is not a dictionary id, an image, or a sparse one-hot vector (embeddings are the dense alternative to one-hot).

Efficient Estimation of Word Representations Sentence-Transformers docs

Which principle underlies learned word embeddings like word2vec?

Words are embedded by hashing their characters into buckets
Words in similar contexts should have similar vectors
Every word must be orthogonal to every other word
Word meaning is fixed by a hand-built ontology

Why

This is the distributional hypothesis: 'a word is known by the company it keeps', so word2vec learns similar vectors for words sharing contexts. It is not character hashing, forced orthogonality (that would erase similarity), or a manual ontology.

Efficient Estimation of Word Representations Distributed Representations of Words

In word2vec, how do the skip-gram and CBOW objectives differ?

Skip-gram uses characters; CBOW uses whole words as tokens
Skip-gram: word predicts context; CBOW: context predicts word
Skip-gram trains on labels; CBOW is fully unsupervised
Skip-gram outputs sentences; CBOW outputs single vectors

Why

Skip-gram predicts surrounding context words given the center word, while CBOW does the reverse — predict the center word from its averaged context. Both are self-supervised over word tokens; neither is character-based or supervised with labels.

Efficient Estimation of Word Representations Distributed Representations of Words

Why is negative sampling used when training word2vec?

It labels each word with its part of speech during training
It removes rare words from the vocabulary before training
It avoids a full softmax by scoring a few sampled negatives
It replaces the vectors with their negated values each step

Why

A softmax over the whole vocabulary is expensive; negative sampling turns it into cheap binary classification against a handful of sampled 'negative' words each step. It is not POS labeling, vocabulary pruning, or vector negation.

Distributed Representations of Words Efficient Estimation of Word Representations

The famous 'king - man + woman ≈ queen' result illustrates what property of word embeddings?

Embeddings memorize a fixed table of analogy answers
Words are stored as exact integer coordinates
Vector arithmetic can capture some linear semantic relationships
The model was explicitly trained on analogy questions

Why

Learned embeddings encode some relationships as roughly linear directions (e.g. a 'gender' offset), so analogies fall out of vector arithmetic — an emergent property, not memorization, integer coordinates, or supervised analogy training.

Efficient Estimation of Word Representations GloVe: Global Vectors

Free account

Take the full module

These are the first few of 50 questions. A free account opens the rest as a scored drill.

Every question in this module
Instant feedback and supporting reading
Your score and progress, saved

Free · your email is used for progress only.

About Nybble^™

The AI space moves fast.
Nybble^™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack

Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack

Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack