RAG Evaluation & Advanced RAG — Core

Faithfulness, answer/context relevance, retrieval metrics, RAGAS, LLM-as-judge, HyDE, multi-query, Self-RAG, CRAG, RAPTOR, contextual retrieval.

Medium50 questions

Sample questions

Your RAG assistant returns a wrong answer even though the correct passage is clearly present in the retrieved context. Which component should you investigate first?

The retriever, which failed to fetch the relevant passage
The embedding model, which encodes the query wrongly
The chunker, which split the passage too small
The generator, which isn't grounding on the retrieved passage

Why

If the gold passage is in the context but the answer is wrong, retrieval already succeeded — the fault is generation ignoring or misusing the evidence (a faithfulness failure). Splitting retrieval and generation metrics localizes exactly this; embedding or chunking bugs would instead show up as the passage being absent.

Ragas metrics documentation TruLens RAG Triad

In RAG evaluation, what does the metric 'faithfulness' (also called groundedness) measure?

Whether the answer's claims are supported by the retrieved context
Whether the answer is directly relevant to the user's actual question
Whether the retrieved chunks are relevant to the query
Whether the answer matches a human-written reference

Why

Faithfulness checks that every claim in the generated answer can be traced back to the retrieved context — the core hallucination guard. Answer relevance targets the question, context relevance targets the chunks, and matching a reference is answer correctness; all are separate axes.

Ragas metrics documentation TruLens RAG Triad

What is the main advantage of RAGAS-style reference-free metrics over reference-based ones like exact match?

They require a large labeled test set to compute
They only work for extractive question answering
They score faithfulness and relevance without gold answers
They measure retrieval latency instead of answer quality

Why

RAGAS uses an LLM to judge faithfulness, answer relevance and context relevance from just (question, context, answer), so it needs no human-written gold answers and enables fast iteration. Exact match and F1 require references, and RAGAS measures quality, not latency.

Ragas: Automated Evaluation of RAG Ragas metrics documentation

You measure Recall@5 as high, yet answers keep missing a fact that sits in the 5th (last) retrieved chunk. What is the most likely explanation?

The embedding model has too few dimensions
Recall@5 is the wrong metric here; you should use Precision@5
The model under-attends to late context ('lost in the middle')
The retriever keeps returning duplicate chunks

Why

High recall means the fact was retrieved, but LLMs attend most to the start and end of the context and can neglect the middle — and a marginal last position is easy to skip. Reordering the strongest chunk toward an edge helps. It isn't a metric choice, a dimensionality issue, or duplicates hiding a present fact.

Lost in the Middle Ragas metrics documentation

In retrieval evaluation, what does Recall@k measure?

The fraction of the top-k retrieved results that are relevant
The rank position of the first relevant document
The fraction of relevant documents present in the top-k results
The graded relevance of results, discounted by rank

Why

Recall@k is (relevant items in top-k) / (all relevant items) — how completely the evidence made it into the window. Option 2 defines Precision@k, option 3 is the reciprocal-rank idea behind MRR, and option 4 describes NDCG.

Ragas metrics documentation Discounted Cumulative Gain (NDCG)

Free account

Take the full module

These are the first few of 50 questions. A free account opens the rest as a scored drill.

Every question in this module
Instant feedback and supporting reading
Your score and progress, saved

Free · your email is used for progress only.

About Nybble^™

The AI space moves fast.
Nybble^™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack

Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack

Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack