Query transformations
Query transformations
The user's query and the documents in the corpus are written for different purposes. The user asks a question; the documents state facts. A query like "why does my RAG pipeline return irrelevant results?" is semantically distant from the chunk that actually answers it: "Low retrieval precision is commonly caused by embedding model mismatch, insufficient chunking granularity, or missing metadata filters." The query uses question syntax and colloquial phrasing; the answer uses declarative, technical prose. This is the query-document gap.
Query transformations rewrite the user's query before it hits the retriever, producing one or more reformulated queries that are closer to the language of the target documents. The retriever itself is unchanged — you are improving the input, not the algorithm.
Multi-query expansion
Generate multiple reformulations of the original query, retrieve for each, and union the results. This widens the net: different phrasings activate different regions of the embedding space and match different lexical patterns.
from anthropic import Anthropic
anthropic = Anthropic()
def generate_multi_queries(original_query: str, n: int = 3) -> list[str]:
response = anthropic.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=300,
messages=[{
"role": "user",
"content": (
f"Generate {n} alternative search queries for retrieving "
f"documents relevant to this question. Return only the queries, "
f"one per line, no numbering.\n\n"
f"Original query: {original_query}"
),
}],
)
queries = response.content[0].text.strip().split("\n")
return [q.strip() for q in queries if q.strip()]
def multi_query_retrieve(
original_query: str,
conn,
k: int = 5,
) -> list[dict]:
queries = [original_query] + generate_multi_queries(original_query, n=3)
seen_ids = set()
all_results = []
for query in queries:
results = dense_retrieve(query, conn, k=k)
for r in results:
if r["id"] not in seen_ids:
seen_ids.add(r["id"])
all_results.append(r)
all_results.sort(key=lambda x: x["score"], reverse=True)
return all_results[:k]For the query "why does my RAG pipeline return irrelevant results?", the LLM might generate:
- "causes of low retrieval precision in RAG systems"
- "how to debug poor retrieval quality in vector search"
- "embedding model mismatch chunking strategy retrieval failure"
The first two are semantic reformulations that may surface different chunks from the embedding space. The third is keyword-dense, which helps if you're running hybrid search — BM25 matches on "embedding model mismatch" directly.
Multi-query adds one LLM call of latency (~200–500ms with a fast model). The retrieval calls can run in parallel, so the total retrieval latency is roughly max(retrieval_time_per_query) plus the generation time, not the sum.
HyDE: Hypothetical Document Embeddings
HyDE, introduced by Gao et al. 2022, takes a different approach. Instead of rewriting the query, it generates a hypothetical answer — a passage that would appear in a document that answers the query — and embeds that passage instead of the query.
The intuition: a hypothetical answer is written in the same register as the actual documents (declarative, factual prose), so its embedding lands closer to the relevant chunks in vector space than the original question does.
from anthropic import Anthropic
from openai import OpenAI
anthropic = Anthropic()
openai_client = OpenAI()
def hyde_retrieve(
query: str,
conn,
k: int = 5,
) -> list[dict]:
response = anthropic.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=300,
messages=[{
"role": "user",
"content": (
"Write a short technical passage (3-4 sentences) that would "
"appear in a document answering this question. Write it as "
"factual prose, not as a response to a question.\n\n"
f"Question: {query}"
),
}],
)
hypothetical_doc = response.content[0].text.strip()
hyde_vec = openai_client.embeddings.create(
input=[hypothetical_doc], model="text-embedding-3-small"
).data[0].embedding
with conn.cursor() as cur:
cur.execute("""
SELECT id, content, 1 - (embedding <=> %s::vector) AS score
FROM chunks
ORDER BY embedding <=> %s::vector
LIMIT %s
""", [hyde_vec, hyde_vec, k])
return [
{"id": r[0], "content": r[1], "score": r[2]}
for r in cur.fetchall()
]For the query "why does my RAG pipeline return irrelevant results?", HyDE might generate:
> "Low retrieval precision in RAG systems is commonly caused by three factors: embedding model mismatch between the query and document encoders, chunking granularity that is too coarse or too fine for the target queries, and missing metadata filters that allow semantically similar but contextually irrelevant documents to surface."
This passage is far closer in embedding space to the actual documentation chunks than the original question was. Gao et al. 2022 showed HyDE improving recall@20 by 4–12 points on BEIR benchmarks compared to direct query embedding, with the largest gains on datasets where the query-document register gap is widest.
When HyDE hurts
HyDE has a failure mode: the hypothetical answer can be wrong. If the LLM hallucinates a plausible but incorrect passage, the embedding points to the wrong neighborhood in vector space and retrieval gets worse, not better.
This is most likely when:
- The query is about a fact the LLM doesn't know (obscure internal documentation, recent events)
- The domain is highly specialized (medical, legal) and the LLM generates generically correct but domain-irrelevant prose
- The query contains specific identifiers that the hypothetical answer paraphrases away
HyDE is most effective for conceptual questions where the LLM can generate a reasonable approximation of the answer's language, even if the details are wrong. For identifier-heavy or factual queries, multi-query expansion or direct retrieval with hybrid search is safer.
Step-back prompting
Zhou et al. introduced step-back prompting as a way to handle overly specific queries that miss relevant context. Instead of searching for the specific question, generate a broader "step-back" question and retrieve for both.
def stepback_retrieve(
query: str,
conn,
k: int = 5,
) -> list[dict]:
response = anthropic.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=150,
messages=[{
"role": "user",
"content": (
"Given this specific question, generate a broader 'step-back' "
"question that captures the underlying concept. Return only the "
"step-back question.\n\n"
f"Specific question: {query}"
),
}],
)
stepback_query = response.content[0].text.strip()
original_results = dense_retrieve(query, conn, k=k)
stepback_results = dense_retrieve(stepback_query, conn, k=k)
seen_ids = set()
merged = []
for r in original_results + stepback_results:
if r["id"] not in seen_ids:
seen_ids.add(r["id"])
merged.append(r)
merged.sort(key=lambda x: x["score"], reverse=True)
return merged[:k]Example: "What is the maximum ef_search value for HNSW indexes in pgvector 0.7?" steps back to "How does HNSW index tuning work in pgvector?" The specific query might miss if no chunk mentions the exact version and parameter limit. The step-back query retrieves the HNSW tuning documentation, which likely contains the answer.
Query decomposition for multi-hop questions
Some queries require information from multiple chunks that do not appear in the same retrieval window. "How does the chunking strategy affect reranking latency?" spans two topics: chunking (which determines chunk count and size) and reranking (which processes each retrieved chunk through a cross-encoder). A single retrieval pass may return chunks about one topic or the other, but rarely both.
Decomposition splits the query into sub-questions, retrieves for each, and provides all retrieved chunks to the generator:
def decompose_and_retrieve(
query: str,
conn,
k_per_subquery: int = 3,
) -> list[dict]:
response = anthropic.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=300,
messages=[{
"role": "user",
"content": (
"Break this question into 2-3 independent sub-questions that "
"can each be answered by searching a knowledge base. Return "
"only the sub-questions, one per line.\n\n"
f"Question: {query}"
),
}],
)
sub_queries = [
q.strip() for q in response.content[0].text.strip().split("\n")
if q.strip()
]
seen_ids = set()
all_results = []
for sq in sub_queries:
results = dense_retrieve(sq, conn, k=k_per_subquery)
for r in results:
if r["id"] not in seen_ids:
seen_ids.add(r["id"])
all_results.append(r)
return all_resultsFor "How does the chunking strategy affect reranking latency?", the decomposition might produce:
- "What chunking strategies are used in RAG systems and how do they affect chunk count?"
- "How does reranking latency scale with the number of retrieved chunks?"
Each sub-query retrieves focused, relevant chunks. The generator receives context covering both topics and can synthesize the cross-cutting answer.
Choosing a transformation strategy
Each transformation adds an LLM call (~200–500ms) before retrieval. Use the simplest one that addresses your query distribution:
- No transformation — queries are well-formed and match document language. Most keyword-style queries.
- Multi-query — queries use varied vocabulary; you want broader recall without changing the retrieval architecture. Low risk, moderate gain.
- HyDE — conceptual questions with a large query-document register gap. Higher gain on these queries, but risks hallucination on factual/identifier queries.
- Step-back — overly specific queries that miss broader context. Useful when users ask about narrow details and need surrounding concepts.
- Decomposition — multi-hop questions that span topics. Necessary when single-pass retrieval cannot surface all required context.
These are not mutually exclusive. A production system can route queries to different strategies based on query classification — short keyword queries go straight to hybrid search, conceptual questions go through HyDE, and complex multi-part questions get decomposed. L19 (Agentic RAG) covers this routing pattern in detail.
Measuring the improvement
Query transformations increase recall at the cost of latency and LLM calls. Measure both:
import time
def compare_retrieval(query, conn, relevant_ids, k=10):
start = time.time()
direct = dense_retrieve(query, conn, k=k)
direct_time = time.time() - start
direct_hits = len(set(r["id"] for r in direct) & set(relevant_ids))
start = time.time()
hyde = hyde_retrieve(query, conn, k=k)
hyde_time = time.time() - start
hyde_hits = len(set(r["id"] for r in hyde) & set(relevant_ids))
print(f"Direct: recall={direct_hits}/{len(relevant_ids)}, "
f"latency={direct_time*1000:.0f}ms")
print(f"HyDE: recall={hyde_hits}/{len(relevant_ids)}, "
f"latency={hyde_time*1000:.0f}ms")On a typical setup with text-embedding-3-small and pgvector, direct retrieval completes in 10–30ms. HyDE adds 300–600ms for the LLM generation step. If HyDE improves recall@10 from 0.6 to 0.8, the latency cost is usually worth it — the generator produces a better answer with better context. If recall barely moves, skip the transformation and save the latency.
Key takeaways
- The query-document gap — questions vs. declarative prose — is a consistent source of retrieval failure. Query transformations rewrite the input to close this gap before it reaches the retriever.
- Multi-query expansion generates variant phrasings and unions the results. Low risk, broadens recall, and parallelizes well across retrieval calls.
- HyDE generates a hypothetical answer and embeds that instead of the query. Most effective for conceptual questions; risky when the LLM hallucinates facts the corpus contradicts.
- Step-back prompting broadens overly specific queries; decomposition splits multi-hop queries into retrievable sub-questions. Both are routing problems — choose the transformation that matches the query type.
- Every transformation adds an LLM call (~200–500ms). Measure recall improvement against the latency cost on your own query distribution before committing to a strategy.