Dense retrieval & semantic search

A bi-encoder embeds the query and each document chunk independently, then scores candidates by vector similarity. The query encoder and document encoder share the same architecture — often the same weights — producing vectors in a shared space where cosine similarity approximates semantic relevance.

This is the core retrieval mechanism in most RAG systems. Karpukhin et al. 2020 (DPR) showed that a fine-tuned BERT bi-encoder outperformed BM25 on open-domain QA by 9–19 points on top-20 retrieval accuracy, establishing dense retrieval as the default starting point.

Why bi-encoders, not cross-encoders

A cross-encoder takes the query and a document as a single concatenated input and outputs a relevance score. It sees both texts simultaneously, so it captures fine-grained token interactions — negation, coreference, conditional relationships. Cross-encoders produce more accurate relevance scores than bi-encoders.

The tradeoff is computational. A cross-encoder must run a full forward pass for every (query, document) pair. With 1 million chunks, that is 1 million forward passes per query — minutes of GPU time. A bi-encoder embeds the query once (~5ms), then computes cosine similarity against pre-computed chunk vectors using a dot product — microseconds per comparison. This makes the bi-encoder the only viable architecture for first-stage retrieval at scale. Cross-encoders appear later in the pipeline as rerankers (L11), scoring only the top 20–100 candidates the bi-encoder surfaces.

The retrieval path

The full dense retrieval path over a pgvector index looks like this:

from openai import OpenAI
import psycopg2

client = OpenAI()

def embed_query(text: str, model: str = "text-embedding-3-small") -> list[float]:
    response = client.embeddings.create(input=[text], model=model)
    return response.data[0].embedding

def dense_retrieve(
    query: str,
    conn,
    k: int = 5,
    source_filter: str | None = None,
) -> list[dict]:
    query_vec = embed_query(query)

    sql = """
        SELECT id, content, source, page,
               1 - (embedding <=> %s::vector) AS score
        FROM chunks
    """
    params = [query_vec]

    if source_filter:
        sql += " WHERE source = %s"
        params.append(source_filter)

    sql += " ORDER BY embedding <=> %s::vector LIMIT %s"
    params.extend([query_vec, k])

    with conn.cursor() as cur:
        cur.execute(sql, params)
        rows = cur.fetchall()

    return [
        {"id": r[0], "content": r[1], "source": r[2], "page": r[3], "score": r[4]}
        for r in rows
    ]

The <=> operator computes cosine distance. pgvector's HNSW index (built in L7) accelerates this from a full scan to an approximate graph traversal. The 1 - distance conversion gives cosine similarity on a 0-to-1 scale.

Interpreting similarity scores

The raw cosine similarity between a query and a chunk is a number between -1 and 1 (in practice, 0 to 1 for normalized embeddings from models like text-embedding-3-small). Higher means more similar. But what constitutes a "good" score depends entirely on the model, the corpus, and the query distribution.

Typical score ranges for text-embedding-3-small on English technical documentation:

0.85+ — near-exact match or paraphrase of the query
0.70–0.85 — semantically relevant, likely useful context
0.55–0.70 — tangentially related, may or may not help
below 0.55 — probably noise

These ranges are not universal. A model trained on biomedical text produces different score distributions than one trained on web text. A corpus of short FAQ answers produces higher top scores than a corpus of long regulatory filings. The only reliable way to set a threshold is to sample query-result pairs from your own system and label them.

Score thresholds vs. fixed top-k

Two strategies for deciding which chunks to pass to the generator:

Fixed top-k returns exactly k chunks regardless of their scores. Simple, predictable token budget, but includes irrelevant chunks when the query has fewer than k good matches — and those irrelevant chunks can mislead the generator (Liu et al. 2023, "Lost in the Middle").

Score threshold returns only chunks above a minimum similarity. Adapts to query difficulty — easy queries with many matches return more context, hard queries with no good match return nothing. The risk: a threshold tuned on one query distribution may be too aggressive or too lenient on another.

In practice, combine both — retrieve top-k, then filter by a minimum score:

def retrieve_with_threshold(
    query: str,
    conn,
    k: int = 10,
    min_score: float = 0.60,
) -> list[dict]:
    results = dense_retrieve(query, conn, k=k)
    return [r for r in results if r["score"] >= min_score]

Start with k=10 and min_score=0.60, then adjust based on labeled retrieval examples. Track the percentage of queries that return zero results after filtering — if it exceeds 10–15%, the threshold is too high or the corpus has coverage gaps.

Score calibration

Cosine similarity is not a probability. A score of 0.75 does not mean "75% relevant." Two practical consequences:

Scores are not comparable across models. A 0.82 from text-embedding-3-small and a 0.82 from all-MiniLM-L6-v2 indicate different levels of semantic match. If you switch embedding models, all thresholds need recalibration.

Score distributions shift with corpus size. Adding more documents to the index increases the chance that some chunk gets a moderately high score for any query, even if it is not relevant. A threshold that works at 10K chunks may admit noise at 1M chunks.

To calibrate: embed 50–100 representative queries, retrieve top-20, and have a human label each result as relevant or not. Plot precision at each score cutoff. The threshold is the point where precision drops below your tolerance — typically 80–90% for RAG applications.

Where dense retrieval fails

Dense retrieval matches meaning, not tokens. This is its strength and its blind spot. Three failure modes appear consistently:

Exact identifiers. A query like "error code E-4012" or "part number AX-7890-B" requires exact lexical matching. Dense models encode these as generic "error code" or "part number" semantics, losing the specific identifier. The embedding for "E-4012" is nearly identical to the embedding for "E-5033" — both are error codes.

Rare proper nouns. Names, acronyms, or domain-specific terms that appear rarely in the embedding model's training data get poor vector representations. A query for "Kagra interferometer" may retrieve chunks about LIGO or gravitational waves in general — semantically adjacent but not what was asked.

Negation and contrast. "Models that do NOT use attention" and "models that use attention" produce nearly identical embeddings. Dense models encode topic, not logical structure. A query seeking the absence of a concept retrieves chunks about the presence of that concept.

from openai import OpenAI
import numpy as np

client = OpenAI()

def cos_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

texts = [
    "error code E-4012: connection timeout",
    "error code E-5033: authentication failure",
    "models that use attention mechanisms",
    "models that do not use attention mechanisms",
]
response = client.embeddings.create(
    input=texts, model="text-embedding-3-small"
)
vecs = [e.embedding for e in response.data]

print(f"E-4012 vs E-5033: {cos_sim(vecs[0], vecs[1]):.3f}")
print(f"use attention vs NOT use attention: {cos_sim(vecs[2], vecs[3]):.3f}")

Typical output: the two error codes score ~0.92 similarity. The attention/no-attention pair scores ~0.96. Dense retrieval cannot distinguish them.

These failure modes are not fixable by using a better dense model — they are structural limitations of the bi-encoder architecture, which compresses the full meaning of a text into a single vector before comparison. The next lesson covers how sparse retrieval and hybrid fusion address these gaps.

Key takeaways

Dense retrieval embeds query and documents independently with a bi-encoder, then ranks by cosine similarity. It handles paraphrase, synonymy, and fuzzy matching well — but that is all it sees.
Similarity scores are not probabilities. Thresholds are dataset-dependent: calibrate on labeled query-result pairs from your own system, not on general ranges.
Combine top-k with a minimum score threshold. Track zero-result rates to detect when the threshold is too aggressive or the corpus has coverage gaps.
Dense retrieval fails on exact identifiers (error codes, part numbers), rare proper nouns, and negation-sensitive queries. These are structural limitations of the bi-encoder, not bugs in any particular model.

← Previous