Distance metrics and vector spaces

Distance metrics and vector spaces

Every vector search operation reduces to: compute a numeric score between the query vector and each candidate, then rank by that score. The choice of scoring function — the distance metric — determines what "similar" means geometrically. Using the wrong metric for your embedding model silently degrades retrieval quality with no error message.

Cosine similarity

Cosine similarity measures the angle between two vectors, ignoring their magnitudes:

cos(a, b) = (a · b) / (||a|| × ||b||)

• Range: -1 (antiparallel) to +1 (identical direction). 0 means orthogonal.
• What it captures: Directional alignment. Two vectors pointing in the same direction score 1.0 regardless of whether one is 10x longer than the other.
• Computational cost: One dot product plus two norms. For d-dimensional vectors: 3d multiplications, 3d-2 additions, 2 square roots, 1 division.

When to use cosine

Most text embedding models (OpenAI text-embedding-3-small, Cohere embed-v3, sentence-transformers models) are trained with cosine similarity as the loss objective. Their vectors are either pre-normalized to unit length or designed so that angular separation encodes semantic distance.

For pre-normalized vectors (unit L2 norm), cosine similarity simplifies to the dot product — the normalization denominators become 1. This is why many vector databases internally normalize vectors and switch to dot product computation.

Dot product (inner product)

a · b = Σ(a_i × b_i) for i = 1 to d

• Range: Unbounded. Can be any real number.
• What it captures: A combination of directional alignment and magnitude. If both vectors are long and point in the same direction, the dot product is large.
• Computational cost: d multiplications, d-1 additions. Cheapest of the three metrics.

When dot product diverges from cosine

For normalized vectors, dot product and cosine produce identical rankings. They diverge when magnitudes vary:

import numpy as np

a = np.array([1.0, 0.0, 0.0])
b = np.array([0.9, 0.1, 0.0])  # close in direction to a
c = np.array([5.0, 0.5, 0.0])  # same direction as b, but longer

cos_ab = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))  # 0.994
cos_ac = np.dot(a, c) / (np.linalg.norm(a) * np.linalg.norm(c))  # 0.995

dot_ab = np.dot(a, b)  # 0.9
dot_ac = np.dot(a, c)  # 5.0

Cosine ranks b and c nearly identically (both point roughly the same direction as a). Dot product strongly prefers c because it is longer.

When magnitude is information

Some retrieval models intentionally encode importance or confidence in vector magnitude. ColBERT-style late-interaction models and certain maximum inner product search (MIPS) formulations rely on this: a document embedding is longer when the model is more confident the document is relevant. For these models, dot product is the correct metric — using cosine discards the magnitude signal the model learned to emit.

Euclidean distance (L2)

||a - b|| = sqrt(Σ(a_i - b_i)²) for i = 1 to d

• Range: 0 (identical) to infinity. Lower = more similar.
• What it captures: Straight-line geometric distance in the embedding space.
• Computational cost: d subtractions, d multiplications, d-1 additions, 1 square root. In practice, indexes skip the square root and compare squared distances (preserves ranking).

The relationship to cosine for normalized vectors

When both vectors have unit L2 norm:

||a - b||² = ||a||² + ||b||² - 2(a · b) = 2 - 2·cos(a, b) = 2(1 - cos(a, b))

This means L2 distance and cosine similarity produce inverse but monotonically related rankings for normalized vectors. Minimizing L2² is equivalent to maximizing cosine. FAISS exploits this: its IndexFlatIP (inner product) on normalized vectors gives the same results as IndexFlatL2.

When L2 is the wrong choice

L2 distance is sensitive to magnitude. If your embedding model produces vectors of varying length (common for unnormalized models), L2 will penalize magnitude differences that may be semantically irrelevant. A vector [0.1, 0.1] is far from [1.0, 1.0] in L2 despite pointing in the exact same direction. If direction is what matters, normalize first or use cosine.

Metric agreement and disagreement

import numpy as np
from numpy.linalg import norm

np.random.seed(42)
d = 768

query = np.random.randn(d)
query_norm = query / norm(query)

# Generate candidates: some normalized, some not
candidates_raw = np.random.randn(100, d)
candidates_norm = candidates_raw / norm(candidates_raw, axis=1, keepdims=True)

# Rankings on normalized vectors — all three agree
cos_scores = candidates_norm @ query_norm
dot_scores = candidates_norm @ query_norm  # same as cos for unit vectors
l2_dists = norm(candidates_norm - query_norm, axis=1)

cos_rank = np.argsort(-cos_scores)[:10]
dot_rank = np.argsort(-dot_scores)[:10]
l2_rank = np.argsort(l2_dists)[:10]

assert np.array_equal(cos_rank, dot_rank)  # True
assert np.array_equal(cos_rank, l2_rank)   # True

# Rankings on unnormalized vectors — they diverge
cos_scores_raw = (candidates_raw @ query) / (norm(candidates_raw, axis=1) * norm(query))
dot_scores_raw = candidates_raw @ query
l2_dists_raw = norm(candidates_raw - query, axis=1)

cos_rank_raw = np.argsort(-cos_scores_raw)[:10]
dot_rank_raw = np.argsort(-dot_scores_raw)[:10]
l2_rank_raw = np.argsort(l2_dists_raw)[:10]

overlap_cos_dot = len(set(cos_rank_raw) & set(dot_rank_raw))  # typically 3-6 out of 10
overlap_cos_l2 = len(set(cos_rank_raw) & set(l2_rank_raw))    # typically 4-7 out of 10

On normalized vectors, all three metrics produce identical top-10 rankings. On unnormalized vectors, overlap drops to 30-70% — a silent retrieval quality degradation.

The normalization trap

The most common vector search misconfiguration: using L2 or dot product with a model that assumes cosine similarity, without normalizing vectors at ingestion time.

How it manifests

• You use text-embedding-ada-002 (trained for cosine) with a FAISS IndexFlatL2.
• Vectors from that model have norms ranging from ~0.9 to ~1.1 (near-unit but not exact).
• For most queries, L2 and cosine agree because norms are close to 1.
• Occasionally, a document with an unusually high or low norm gets misranked — promoted or suppressed based on magnitude rather than semantic relevance.
• Retrieval quality drops by 2-5% recall@10 compared to proper cosine search.
• No error is raised. The system appears to work. Only careful evaluation reveals the regression.

The fix

Normalize all vectors to unit length before insertion:

vector = vector / np.linalg.norm(vector)

Then use inner product (dot product) as your metric. This gives you cosine similarity semantics at dot product computational cost — the best of both worlds.

Binary and set-based metrics

For specialized use cases, two additional metrics appear:

Hamming distance

hamming(a, b) = number of positions where a_i ≠ b_i

Used with binary vectors (each dimension is 0 or 1). Relevant when vectors are binary hash codes from locality-sensitive hashing or binary quantization. Extremely fast — computable with XOR and popcount CPU instructions. A 256-bit Hamming comparison takes ~1 nanosecond.

Jaccard similarity

jaccard(A, B) = |A ∩ B| / |A ∪ B|

Used when vectors represent sets (e.g., shingle sets for document deduplication). Range 0 to 1. Not applicable to dense floating-point embeddings — mentioned here because some vector databases support it for hybrid search scenarios involving sparse set representations.

Choosing the right metric

• You use OpenAI, Cohere, or sentence-transformers embeddings: Normalize and use dot product (equivalent to cosine). These models are trained for cosine similarity.
• You use a MIPS model (ColBERT, some recommendation models): Use dot product without normalization. Magnitude is signal.
• You use a model with L2-trained loss (some custom metric learning models): Use L2 distance.
• You are unsure: Normalize and use dot product. This is correct for the vast majority of embedding models and never catastrophically wrong.

Key takeaways

• Cosine similarity measures angular alignment and ignores magnitude; it is the training objective for most text embedding models (OpenAI, Cohere, sentence-transformers).
• For unit-normalized vectors, cosine, dot product, and L2 distance produce identical rankings — normalize at ingestion and use dot product for the cheapest computation.
• Dot product without normalization incorporates magnitude, which is correct only for models that intentionally encode importance in vector length (MIPS setting).
• The normalization trap — using the wrong metric without normalizing — silently degrades recall@10 by 2-5% with no error raised.
• When in doubt, normalize vectors and use inner product: it is correct for the majority of embeddings and avoids the normalization trap entirely.