Nybble™
SnackStackHack
LLM Evaluation and Testing

Course wrap-up

Lessons

  1. 1. Why evaluation is different for AI
  2. 2. Metrics that matter
  3. 3. Choosing metrics for what you actually built
  4. 4. Single-turn scoring: the unit of evaluation
  5. 5. Consistency: the score that moves when nothing changed
  6. 6. Golden datasets
  7. 7. Building a golden set that survives contact
  8. 8. Human annotation and agreement
  9. 9. LLM-as-a-judge
  10. 10. Judge designs: pointwise, pairwise, reference-guided
  11. 11. Judge bias, measured and mitigated
  12. 12. Calibrating and meta-evaluating a judge
  13. 13. Multi-turn evaluation
  14. 14. User simulators
  15. 15. The RAG evaluation triad
  16. 16. Component-level evaluation
  17. 17. Agent evaluation
  18. 18. Regression testing for AI
  19. 19. Slices, peeking, and the multiple-comparisons trap
  20. 20. Building an eval harness
  21. 21. Reproducible runs: the eval manifest
  22. 22. CI/CD gates for AI quality
  23. 23. When offline and online disagree
  24. 24. A/B testing in production
  25. 25. Online monitoring and drift detection
  26. 26. Bias, safety, and adversarial evaluation
  27. 27. Capstone — an evaluation pipeline
  28. Key takeaways
  29. How to get certified
  30. Your certificate

Your certificate

Certificates are issued to an account — it's what ties the credential to a name a reader can check. Sign in to see where you stand on LLM Evaluation and Testing.

← PreviousBack to course

Nybble™ — built for the people building AI.

AboutTermsPrivacyContact
SnackStackHack
Message Nybble