Key takeaways
The through-line
AI evaluation is different because the system is non-deterministic and the ground truth is often a judgement call. You cannot assert equality. What you can do is measure distributions, and build a harness that tells you whether today's change made things worse.
The ideas that carry
- A golden dataset is the asset; everything else is tooling. It should be small, real, adversarial where it counts, and versioned. Most teams over-invest in frameworks and under-invest in the hundred examples that would have caught the regression.
- Pick metrics that move when quality moves. BLEU/ROUGE measure surface overlap and are nearly useless for open generation. Task-specific and rubric-based metrics correlate with what you actually care about.
- LLM-as-a-judge works, with discipline. It needs a rubric, position-bias control, and calibration against human labels — otherwise you have automated your assumptions.
- The RAG triad localises failure: context relevance (retrieval), groundedness (did it stick to context), answer relevance (did it answer). One number can't tell you which broke.
- Component-level evaluation is what makes a pipeline debuggable. End-to-end scores tell you something is wrong; component scores tell you where.
- Regression testing must tolerate non-determinism — thresholds and distributions, not exact match, or the suite gets muted within a fortnight.
- CI gates are where evaluation becomes real. An eval nobody blocks a deploy on is a dashboard.
- Offline evaluation ends at the deploy boundary. A/B tests, online monitoring and drift detection cover the part where reality disagrees with your dataset.
What you can now do
Build a golden dataset that earns its keep. Stand up a harness that runs in CI and blocks a bad merge. Calibrate an LLM judge against humans. Detect drift before users report it.
If one thing sticks
If you can't say what would have to be true for you to roll back, you don't have an evaluation — you have a report.