RAG Evaluation & Advanced RAG — Advanced

Faithfulness mechanics, RAGAS/ARES internals, judge bias, RGB, Self-RAG reflection tokens, CRAG, RAPTOR trees, FLARE, contextual retrieval, FiD, long-context trade-offs.

Hard50 questions

Sample questions

RAGAS estimates faithfulness by first decomposing the answer into individual claims. Why is that decomposition step important?

It lets each atomic claim be verified against the context separately
It shortens the answer to fit within the judge's limited context window
It removes stop words so scoring is cheaper
It translates claims into a canonical language

Why

Breaking the answer into atomic claims lets the judge check support for each one, so a single unsupported sentence lowers the score even if the rest is grounded. It is not about truncation, stop-word removal, or translation — those wouldn't localize which claim is unsupported.

Ragas: Automated Evaluation of RAG Ragas metrics documentation

Your faithfulness score is high but users still get wrong answers, and you confirm the retrieved context itself contains errors. Which axis is failing?

Answer relevance is too low
Context precision is too low
Retrieval latency is far too high under peak load
Answer correctness — faithful to a wrong context

Why

Faithfulness only checks agreement with the retrieved context; if that context is wrong, the answer is faithful yet incorrect, so answer correctness (vs ground truth) is the failing axis and the corpus needs fixing. It isn't relevance, precision ordering, or latency.

Benchmarking LLMs in RAG (RGB)Ragas metrics documentation

Self-RAG's 'ISSUP' (fully/partially/no support) reflection token is used to judge what?

Whether retrieval should be triggered for the segment
Whether the retrieved passage is relevant to the query
Whether the overall answer is useful to the user
Whether the generated statement is supported by the passage

Why

ISSUP grades how well a generated segment is entailed by its cited passage — Self-RAG's built-in faithfulness check. Deciding to retrieve is the Retrieve token, passage relevance is ISREL, and overall usefulness is ISUSE.

Self-RAG (Asai et al., 2023)TruLens RAG Triad

Compared with standard RAG that always retrieves a fixed number of passages, Self-RAG's on-demand retrieval mainly helps by doing what?

Guaranteeing lower latency on every single query
Skipping retrieval when parametric knowledge suffices
Eliminating the need for any generator model
Removing the need to evaluate faithfulness

Why

Self-RAG learns to retrieve only when needed, avoiding irrelevant passages for questions the model already knows and cutting noise. It doesn't guarantee lower latency on all queries, remove the generator, or make faithfulness evaluation unnecessary.

Self-RAG (Asai et al., 2023)FLARE: Active RAG

CRAG's 'decompose-then-recompose' step on retrieved documents is designed to do what?

Split the query into unrelated sub-queries
Extract key strips and filter distracting text within docs
Compress the model weights for faster serving
Re-embed the entire corpus using a much smaller model

Why

CRAG breaks retrieved documents into fine-grained strips, keeps the relevant ones, and recomposes them, denoising the context before generation. It isn't query splitting, weight compression, or re-embedding the corpus.

Corrective RAG (CRAG)Self-RAG (Asai et al., 2023)

Free account

Take the full module

These are the first few of 50 questions. A free account opens the rest as a scored drill.

Every question in this module
Instant feedback and supporting reading
Your score and progress, saved

Free · your email is used for progress only.

About Nybble^™

The AI space moves fast.
Nybble^™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack

Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack

Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack