Nybble™
← Back to modules

RAG Fundamentals — Advanced

Chunking and embedding trade-offs, hybrid fusion (RRF), query transforms (HyDE, multi-query), retrieval metrics (MRR, nDCG, context precision/recall), faithfulness vs relevancy, reranking cost, and failure diagnosis.

Hard50 questions

Sample questions

1

Recall@10 on your eval set is 0.92, but end-to-end answer accuracy is only 0.55. Where is the bottleneck most likely, and why?

  • Retrieval, because high recall means the pipeline is already optimal
  • The embedding model, because recall cannot be trusted
  • The vector index, because it must be silently dropping results
  • Generation or context assembly, since retrieved chunks aren't used well

Why

High recall@10 means the relevant chunk is usually retrieved, so the loss happens downstream — grounding, context ordering (lost-in-the-middle), reranking, or prompt design. Retrieval is clearly working, so the embedder and index aren't the first suspects.

Lost in the MiddleRAGAS: Evaluation of RAG
2

How does Reciprocal Rank Fusion (RRF) combine a BM25 ranking and a dense ranking without tuning score scales?

  • It sums 1/(k+rank) across the lists, using ranks only
  • It normalizes both score distributions to [0,1] and averages them
  • It keeps whichever list has the higher top-1 score
  • It multiplies the two raw similarity scores together

Why

RRF scores each document by summing 1/(k + rank_i) over the lists it appears in, relying only on rank positions, so it sidesteps the incomparable score scales of BM25 vs cosine. It doesn't normalize/average raw scores, pick one list, or multiply scores.

Reciprocal Rank Fusion (Cormack et al.)RAG for LLMs: A Survey
3

HyDE (Hypothetical Document Embeddings) improves retrieval for vague queries by doing what?

  • Embedding the query several times and averaging the vectors
  • Fine-tuning the embedder on the query at runtime
  • Draft a hypothetical answer, embed it, retrieve with it
  • Expanding the query with synonyms from a thesaurus

Why

HyDE asks an LLM to draft a plausible answer, then embeds that answer — which often sits closer to real documents in vector space than a short query — and retrieves with it. It isn't query averaging, runtime fine-tuning, or thesaurus expansion.

HyDE: Hypothetical Document EmbeddingsRAG for LLMs: A Survey
4

What does nDCG reward that plain recall@k does not?

  • Returning results faster within the top k
  • Ranking better results higher, with position-discounted gains
  • Retrieving a larger number of total documents
  • Reducing the embedding dimension for efficiency

Why

nDCG uses graded relevance and discounts gains by rank position, so ranking the best documents first scores higher — unlike recall@k, which only checks presence in the top-k set. It's unrelated to latency, corpus size, or embedding dimension.

RAG for LLMs: A SurveyRAGAS: Evaluation of RAG
5

MRR (Mean Reciprocal Rank) is most informative when you care about what?

  • Rank of the first relevant result, averaged
  • The total number of relevant documents retrieved
  • The fluency of the generated answer
  • The diversity of the retrieved set

Why

MRR averages 1/rank of the first relevant hit, so it emphasizes surfacing at least one good result early — useful when the generator mostly needs the top passage. It ignores total relevant count, answer fluency, and diversity.

Dense Passage Retrieval (DPR)RAG for LLMs: A Survey
Free account

Take the full module

These are the first few of 50 questions. A free account opens the rest as a scored drill.

  • Every question in this module
  • Instant feedback and supporting reading
  • Your score and progress, saved

Free · your email is used for progress only.

About Nybble™

The AI space moves fast.
Nybble™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack
Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack
Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack