Nybble™
← Back to modules

Embeddings — Advanced

Contrastive-loss internals, anisotropy and whitening, hard-negative mining (ANCE), DPR/ColBERT late interaction, Matryoshka, decoder embeddings (LLM2Vec), and evaluation trade-offs.

Hard50 questions

Sample questions

1

The InfoNCE contrastive loss is essentially which kind of objective over an anchor, its positive, and the negatives?

  • A softmax cross-entropy selecting the positive from the candidates
  • A regression of the anchor onto the positive's coordinates
  • A margin-only hinge with no probabilistic interpretation
  • A reconstruction loss that rebuilds the anchor from noise

Why

InfoNCE treats the positive as the correct class among {positive, negatives} and applies a temperature-scaled softmax cross-entropy, maximizing the positive's similarity relative to the negatives. It is not coordinate regression, a pure hinge, or a denoising reconstruction.

Representation Learning with Contrastive Predictive CodingSimCSE
2

Wang & Isola decompose contrastive learning into two properties. What are alignment and uniformity?

  • Alignment sorts vectors by norm; uniformity equalizes their magnitudes
  • Alignment merges classes; uniformity collapses everything to a point
  • Alignment: positives close; uniformity: features spread on the sphere
  • Alignment increases dimensionality; uniformity reduces it

Why

Alignment measures how close positive pairs map, and uniformity measures how evenly normalized features spread on the unit hypersphere (preserving information); good contrastive embeddings optimize both. The other options misstate these into norm-sorting, collapse, or dimensionality changes.

Alignment and Uniformity on the HypersphereSimCSE
3

What is the 'anisotropy' problem in raw BERT sentence embeddings?

  • The embeddings are too high-dimensional to index efficiently
  • The embeddings change every time the model is reloaded
  • The vectors are sparse, so most dimensions are exactly zero
  • The vectors occupy a narrow cone, inflating cosine similarities

Why

Anisotropy means the embeddings cluster in a narrow cone of the space, so even unrelated sentences have high cosine similarity and the metric loses discriminative power. It is not about dimensionality, reload instability, or sparsity.

Whitening Sentence RepresentationsSimCSE
4

How does a whitening transformation improve anisotropic sentence embeddings?

  • It quantizes the vectors to int8 for faster search
  • It appends extra dimensions to increase capacity
  • It drops the smallest dimensions to compress the vectors
  • It linearly transforms them toward an isotropic (spherical) distribution

Why

Whitening applies a linear map that removes correlations and equalizes variance, making the distribution more isotropic so cosine comparisons are better behaved. It is not quantization, dimensionality growth, or naive truncation.

Whitening Sentence RepresentationsAlignment and Uniformity on the Hypersphere
5

In supervised SimCSE, natural-language-inference data provides positives and hard negatives. How are they assigned?

  • Neutral pairs are positives; entailment pairs are negatives
  • Entailment pairs are positives; contradictions are hard negatives
  • All three relations are treated as equally positive
  • Contradictions are positives; entailments are negatives

Why

Supervised SimCSE uses NLI 'entailment' pairs as positives and 'contradiction' pairs as informative hard negatives, sharpening the space. The other assignments invert or flatten these relationships.

SimCSESentence-BERT
Free account

Take the full module

These are the first few of 50 questions. A free account opens the rest as a scored drill.

  • Every question in this module
  • Instant feedback and supporting reading
  • Your score and progress, saved

Free · your email is used for progress only.

About Nybble™

The AI space moves fast.
Nybble™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack
Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack
Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack