Agentic RAG & query routing

Agentic RAG and query routing

A single retrieve-then-generate pass assumes every query can be answered by the same index using the same strategy. In practice, some queries need multiple sources, some retrieved documents are irrelevant, and some generated answers hallucinate despite good retrieval. Agentic RAG introduces decision points — nodes that evaluate intermediate results and conditionally route execution through different paths.

Query routing

The simplest agentic pattern: classify the incoming query and route it to the appropriate retrieval backend. A query about API syntax goes to the code documentation index. A query about company policy goes to the HR knowledge base. A query about current events bypasses the corpus entirely and goes to web search.

Implementation is a classifier (LLM-based or embedding-based) that maps query intent to one or more retrieval sources. An LLM-based router uses a few-shot prompt that outputs a structured label. An embedding-based router computes similarity between the query and reference embeddings for each source (one reference embedding per topic cluster). The embedding approach adds ~50ms; the LLM approach adds ~300ms but handles ambiguous or multi-topic queries better.

For multi-source queries, the router can trigger parallel retrieval from multiple backends, then merge results before generation. This is strictly superior to picking one source for queries that span domains.

Corrective RAG (CRAG)

CRAG (Yan et al., 2024) inserts a document-relevance grading step between retrieval and generation. After retrieving candidate documents, a lightweight evaluator scores each document's relevance to the query on a three-point scale: Correct, Ambiguous, or Incorrect.

The grading signal determines the next action:

• If at least one document is Correct: proceed to generation using only the correct documents
• If all documents are Ambiguous: trigger a query rewrite and re-retrieve from the same index
• If all documents are Incorrect: fall back to web search (the corpus does not contain the answer)

This prevents a specific and common failure mode: the retriever returns documents that are plausible (high cosine similarity) but off-topic, and the LLM confidently generates a wrong answer from them. Without grading, the user sees a fluent, well-cited, incorrect response — the worst kind of failure because it erodes trust invisibly.

The grading step can be implemented as a lightweight LLM call (Claude Haiku, ~100ms) or a fine-tuned classifier. The grader does not need to be perfect — even 70% accuracy on the Correct/Incorrect distinction catches the worst failures.

Self-RAG

Self-RAG (Asai et al., 2023) takes a more radical approach: the generation model itself evaluates its own output inline. The model is trained to emit special reflection tokens that signal:

• Whether retrieval is needed for the current segment
• Whether a retrieved document is relevant (ISREL token)
• Whether the generated text is supported by the retrieved documents (ISSUP token)
• Whether the response overall is useful (ISUSE token)

At inference time, these reflection signals enable tree-based decoding: the model generates multiple candidate continuations at each decision point, scores each path on groundedness and relevance, and selects the highest-scoring path. This is not a post-hoc check — the evaluation is woven into the generation process itself.

Self-RAG reported +10–15% improvement on factuality benchmarks (ASQA, FactScore) over standard RAG with comparable 7B/13B model sizes. The cost: approximately 2–3x more tokens generated per query due to the reflection overhead and candidate branching.

Adaptive RAG

Adaptive RAG combines query routing with complexity estimation to select the appropriate pipeline depth. A query complexity classifier (small fine-tuned model or LLM judge) assigns each query to a tier:

• Simple (single-hop, single-source): single retrieval pass, direct generation
• Moderate (multi-fact, single-source): retrieval + reranking + generation
• Complex (multi-hop, multi-source): full agentic pipeline with CRAG grading and potential rewriting

This prevents over-processing simple queries (wasting latency and cost on unnecessary grading/rewriting steps) while ensuring complex queries get the pipeline depth they need.

Implementation with LangGraph

LangGraph provides stateful, cyclic graph execution — the control-flow primitive that agentic RAG requires. Unlike a linear chain, a LangGraph workflow can loop (rewrite → re-retrieve → re-grade), branch (route based on grading outcome), and accumulate state across steps.

from langgraph.graph import StateGraph, END
from typing import TypedDict, List
from langchain_anthropic import ChatAnthropic
from langchain_core.documents import Document

class CRAGState(TypedDict):
    query: str
    original_query: str
    documents: List[Document]
    generation: str
    relevance_scores: List[str]
    retry_count: int

llm = ChatAnthropic(model="claude-sonnet-4-20250514")
grader_llm = ChatAnthropic(model="claude-haiku-4-20250414")

def retrieve(state: CRAGState) -> CRAGState:
    """Retrieve documents from vector store."""
    docs = vector_store.similarity_search(state["query"], k=5)
    return {"documents": docs}

def grade_documents(state: CRAGState) -> CRAGState:
    """Grade each retrieved document for relevance."""
    scores = []
    for doc in state["documents"]:
        response = grader_llm.invoke(
            f"Query: {state['query']}\n\n"
            f"Document: {doc.page_content[:1000]}\n\n"
            "Is this document relevant to the query? "
            "Answer exactly 'relevant' or 'irrelevant'."
        )
        score = "relevant" if "relevant" in response.content.lower() else "irrelevant"
        scores.append(score)
    return {"relevance_scores": scores}

def filter_relevant(state: CRAGState) -> CRAGState:
    """Keep only documents graded as relevant."""
    filtered = [
        doc for doc, score in zip(state["documents"], state["relevance_scores"])
        if score == "relevant"
    ]
    return {"documents": filtered if filtered else state["documents"]}

def rewrite_query(state: CRAGState) -> CRAGState:
    """Rewrite the query for better retrieval."""
    response = llm.invoke(
        f"The following query did not retrieve relevant documents from our "
        f"knowledge base. Rewrite it to be more specific and likely to match "
        f"relevant content.\n\n"
        f"Original query: {state['original_query']}\n"
        f"Previous attempt: {state['query']}\n\n"
        "Return only the rewritten query, nothing else."
    )
    return {"query": response.content, "retry_count": state["retry_count"] + 1}

def generate(state: CRAGState) -> CRAGState:
    """Generate answer from relevant documents."""
    context = "\n\n".join(doc.page_content for doc in state["documents"])
    response = llm.invoke(
        f"Answer the question based on the following context. "
        f"If the context is insufficient, say so.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {state['original_query']}"
    )
    return {"generation": response.content}

def route_after_grading(state: CRAGState) -> str:
    """Route based on grading results."""
    relevant_count = state["relevance_scores"].count("relevant")
    total = len(state["relevance_scores"])

    if relevant_count > 0:
        return "filter_and_generate"
    if state["retry_count"] >= 2:
        return "filter_and_generate"
    return "rewrite"

# Build the CRAG graph
workflow = StateGraph(CRAGState)
workflow.add_node("retrieve", retrieve)
workflow.add_node("grade", grade_documents)
workflow.add_node("filter", filter_relevant)
workflow.add_node("rewrite", rewrite_query)
workflow.add_node("generate", generate)

workflow.set_entry_point("retrieve")
workflow.add_edge("retrieve", "grade")
workflow.add_conditional_edges("grade", route_after_grading, {
    "filter_and_generate": "filter",
    "rewrite": "rewrite"
})
workflow.add_edge("filter", "generate")
workflow.add_edge("rewrite", "retrieve")
workflow.add_edge("generate", END)

app = workflow.compile()

result = app.invoke({
    "query": "How does RLHF work in language models?",
    "original_query": "How does RLHF work in language models?",
    "documents": [],
    "generation": "",
    "relevance_scores": [],
    "retry_count": 0
})

Iteration caps and cost implications

Every agentic pattern multiplies inference calls. A single CRAG loop with one rewrite costs approximately 8 LLM calls: 5 grading calls (one per document) + 1 rewrite + 5 more grading calls + 1 generate = 12 calls in the worst case. Compare to basic RAG: 1 generate call.

Enforce hard limits to prevent runaway execution:

• Maximum retry/rewrite count: 2–3 iterations
• Total step budget per query: cap at 5–8 LLM calls total
• Wall-clock timeout: 10–15 seconds for the full pipeline
• Cost circuit breaker: if accumulated cost exceeds $0.05, terminate and return best-effort result

Without these caps, a query with no good answer in the corpus can spiral: grade (all irrelevant) → rewrite → retrieve → grade (still irrelevant) → rewrite → infinitely. The retry_count in the implementation above enforces a hard stop at 2 rewrites — after that, generate with whatever documents are available and let the LLM acknowledge insufficient evidence.

Key takeaways

• Agentic RAG adds decision nodes between retrieval and generation — grading, routing, rewriting, and reflection
• CRAG (Yan et al., 2024) grades retrieved documents and falls back to query rewriting or web search when relevance is low
• Self-RAG (Asai et al., 2023) embeds reflection tokens into generation itself, enabling tree-based selection of grounded outputs (+10–15% factuality)
• LangGraph provides the stateful cyclic execution model required for conditional loops and branching
• Adaptive RAG routes by query complexity — simple queries skip the agentic overhead entirely
• Every agentic loop multiplies cost and latency — enforce step budgets, iteration caps, and wall-clock timeouts to prevent runaway execution