Nybble™
SnackStackHack
LLM Evaluation and Testing

Lesson 19 of 27

Lessons

  1. 1. Why evaluation is different for AI
  2. 2. Metrics that matter
  3. 3. Choosing metrics for what you actually built
  4. 4. Single-turn scoring: the unit of evaluation
  5. 5. Consistency: the score that moves when nothing changed
  6. 6. Golden datasets
  7. 7. Building a golden set that survives contact
  8. 8. Human annotation and agreement
  9. 9. LLM-as-a-judge
  10. 10. Judge designs: pointwise, pairwise, reference-guided
  11. 11. Judge bias, measured and mitigated
  12. 12. Calibrating and meta-evaluating a judge
  13. 13. Multi-turn evaluation
  14. 14. User simulators
  15. 15. The RAG evaluation triad
  16. 16. Component-level evaluation
  17. 17. Agent evaluation
  18. 18. Regression testing for AI
  19. 19. Slices, peeking, and the multiple-comparisons trap
  20. 20. Building an eval harness
  21. 21. Reproducible runs: the eval manifest
  22. 22. CI/CD gates for AI quality
  23. 23. When offline and online disagree
  24. 24. A/B testing in production
  25. 25. Online monitoring and drift detection
  26. 26. Bias, safety, and adversarial evaluation
  27. 27. Capstone — an evaluation pipeline
  28. Key takeaways
  29. How to get certified
  30. Your certificate

Slices, peeking, and the multiple-comparisons trap

Free account

Read this lesson

The opening lessons of every course are free — this one needs an account. Sign in and the full course opens.

  • Every lesson, start to finish
  • Progress saved across devices
  • Bits per lesson, plus a bonus for finishing

Free · your email is used for progress only.

Nybble™ — built for the people building AI.

AboutTermsPrivacyContact
SnackStackHack
Message Nybble