Reranking

Reranking

A bi-encoder embeds the query and each document independently, then compares them by cosine similarity. This independence is what makes bi-encoders fast — you precompute document embeddings once and compare against a single query vector at search time. But it also means the model never sees the query and document together. It scores them in the same space without reading them as a pair.

A cross-encoder takes a different approach. It concatenates the query and document into a single input — [CLS] query [SEP] document [SEP] — and passes the full sequence through a transformer. The model attends across both texts simultaneously, producing a single relevance score. This joint attention is why cross-encoders consistently outscore bi-encoders on precision: the model can detect fine-grained relationships between the query and specific passages within the document.

The precision gain is measurable. On the MS MARCO passage ranking benchmark, bi-encoders achieve MRR@10 of ~0.33-0.38. Cross-encoders on the same dataset reach MRR@10 of ~0.39-0.43 — a 5-10 point improvement that translates directly to better context windows in RAG.

The tradeoff is computational. A bi-encoder computes one forward pass per query (the document embeddings already exist). A cross-encoder computes one forward pass per query-document pair. Scoring 1 million documents with a cross-encoder means 1 million forward passes — at ~5ms each on a GPU, that is 83 minutes per query. Cross-encoders cannot serve as first-stage retrievers at any reasonable scale.

The retrieve-then-rerank pattern

The standard solution is a two-stage pipeline: retrieve a wide candidate set with a fast bi-encoder (or hybrid search), then rescore the top candidates with a cross-encoder.

• Stage 1 (retrieval): bi-encoder or hybrid search returns top-50 to top-100 candidates in 10-50ms
• Stage 2 (reranking): cross-encoder rescores those 50-100 candidates and returns the top-5 or top-10 in 50-200ms

The combined latency is 60-250ms — well within interactive budgets. The precision gain is substantial: production systems consistently report 10-30% precision improvement at the final top-k after adding a reranker, making reranking the single highest-ROI quality improvement after basic retrieval is working.

Cross-encoder models

Three families dominate production reranking.

ms-marco-MiniLM and ms-marco cross-encoders. Trained on the MS MARCO passage ranking dataset (~530K query-passage pairs with relevance labels). cross-encoder/ms-marco-MiniLM-L-6-v2 is the workhorse: 22M parameters, 6 layers, ~5ms per pair on GPU. Its larger sibling ms-marco-MiniLM-L-12-v2 (33M parameters) gains ~1-2 points on nDCG@10 at roughly double the latency. Both are open-weight and run locally.

BGE-Reranker (BAAI). BAAI/bge-reranker-v2-m3 supports multilingual reranking across 100+ languages. bge-reranker-v2-gemma (based on Gemma-2B) is a larger model that trades latency for higher accuracy — 15-20ms per pair on GPU, but competitive with API-based rerankers on BEIR benchmarks.

Cohere Rerank. An API-based reranker: send a query and a list of documents, receive relevance scores. rerank-v3.5 handles up to 100 documents per call in English and multilingual. Latency is ~100-200ms for 50 documents (includes network round-trip). Pricing is per-search (not per-document), which makes it cost-effective for moderate-volume applications. The API trade-off: no model to host, but you send your documents to a third party.

Code: retrieve top-50, rerank to top-5

from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np

bi_encoder = SentenceTransformer("all-MiniLM-L6-v2")
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

chunks = [
    "RAG retrieves relevant documents and feeds them to an LLM for generation.",
    "The transformer architecture uses self-attention to process sequences.",
    "Cross-encoders score query-document pairs jointly through full attention.",
    "Vector databases store embeddings and support approximate nearest neighbor search.",
    "Reranking improves precision by rescoring retrieval candidates with a cross-encoder.",
    "BM25 uses term frequency and inverse document frequency for lexical matching.",
    "Fine-tuning updates model weights on domain-specific training data.",
    "Cosine similarity measures the angle between two vectors in embedding space.",
    "Hybrid search combines dense and sparse retrieval with reciprocal rank fusion.",
    "Prompt engineering structures inputs to guide LLM behavior without weight updates.",
]

query = "How does reranking improve retrieval quality?"

query_embedding = bi_encoder.encode(query)
chunk_embeddings = bi_encoder.encode(chunks)
cosine_scores = np.dot(chunk_embeddings, query_embedding) / (
    np.linalg.norm(chunk_embeddings, axis=1) * np.linalg.norm(query_embedding)
)

top_50_indices = np.argsort(cosine_scores)[::-1][:50]

pairs = [(query, chunks[i]) for i in top_50_indices]
cross_encoder_scores = cross_encoder.predict(pairs)

reranked_order = np.argsort(cross_encoder_scores)[::-1]
top_5_indices = [top_50_indices[i] for i in reranked_order[:5]]

print("Bi-encoder ranking (top 5):")
for rank, idx in enumerate(top_50_indices[:5]):
    print(f"  {rank+1}. [{cosine_scores[idx]:.3f}] {chunks[idx][:80]}")

print("\nCross-encoder reranking (top 5):")
for rank, idx in enumerate(top_5_indices):
    ce_rank = list(top_50_indices).index(idx)
    print(f"  {rank+1}. [was #{ce_rank+1}] {chunks[idx][:80]}")

The reordering is the point. A chunk that the bi-encoder ranked 8th or 15th may land at position 1 after the cross-encoder reads it alongside the query. The cross-encoder sees token-level interactions the bi-encoder missed — a paraphrase, a negation, a conditional statement that changes relevance.

Using the Cohere Rerank API

For applications where hosting a cross-encoder is impractical, the Cohere API handles reranking as a service.

import cohere

co = cohere.ClientV2()

results = co.rerank(
    model="rerank-v3.5",
    query="How does reranking improve retrieval quality?",
    documents=[chunk for chunk in chunks],
    top_n=5,
)

for result in results.results:
    print(f"  index={result.index} score={result.relevance_score:.4f} "
          f"{chunks[result.index][:60]}")

The API returns relevance scores between 0 and 1 (not cosine similarity — these are calibrated relevance probabilities). A score above 0.5 generally indicates strong relevance; below 0.1 indicates the document is likely irrelevant. These calibrated scores are more interpretable than raw cross-encoder logits and can serve double duty as confidence signals for the abstention guard in the generation step.

Latency and cost budgets

Reranking adds a fixed cost per query, independent of corpus size (since you only rescore the top-N candidates, not the full index).

• 50 candidates, ms-marco-MiniLM-L-6: ~25ms on an A10G GPU, ~250ms on CPU
• 50 candidates, bge-reranker-v2-m3: ~75ms on GPU
• 50 candidates, Cohere rerank-v3.5: ~100-200ms (API, includes network)
• 100 candidates, ms-marco-MiniLM-L-6: ~50ms on GPU

For the majority of RAG applications — internal knowledge bases, customer support, document search — the total query budget is 500ms-2s. Reranking at 50-100ms fits comfortably. The calculus changes at very high QPS (>1,000 queries/second), where the GPU cost of running a cross-encoder on every query may justify a dedicated serving instance.

When reranking is not worth it

Reranking adds complexity and latency. Skip it when:

• Retrieval precision is already high. If your bi-encoder consistently puts the relevant chunk in the top-3, a reranker has little to improve.
• The corpus is small. With <1,000 chunks, dense retrieval is often sufficient — the search space is small enough that the right answer surfaces without reranking.
• Latency budget is extremely tight. Sub-50ms total query time leaves no room for a second scoring pass.
• You have not measured retrieval quality yet. Add metrics first (next lesson), then decide if reranking is needed based on where precision drops.

The retrieve-rerank-generate pipeline

In a full RAG system, the pipeline is: embed query (5ms) -> retrieve top-50 (10-30ms) -> rerank to top-5 (25-100ms) -> assemble context and generate (500ms-2s). Reranking occupies a small fraction of total latency but produces the tightest, most relevant context window for the generator — and context quality is the single largest lever on answer quality.

Key takeaways

• Cross-encoders process query-document pairs jointly through full transformer attention, achieving higher precision than bi-encoders at the cost of per-pair computation
• The retrieve-then-rerank pattern combines bi-encoder speed (full corpus) with cross-encoder precision (top candidates) — typically retrieve top-50, rerank to top-5
• Reranking delivers 10-30% precision improvement for 25-200ms of added latency, making it the highest-ROI retrieval quality improvement after basic search
• Open-weight models (ms-marco-MiniLM, BGE-Reranker) run locally; API rerankers (Cohere) trade hosting for a network dependency
• Measure retrieval quality before adding a reranker — if precision at top-k is already high, the added complexity may not pay off