Key takeaways

The through-line

AI evaluation is different because the system is non-deterministic and the ground truth is often a judgement call. You cannot assert equality. What you can do is turn a pile of single scores into a claim you can defend in a deploy review — which is what the twenty-seven lessons above are for.

The ideas that carry

• Start from what you built, not from the metric catalogue. Does it retrieve? Does it converse? Each yes adds a family of failures the others cannot see, and each no lets you delete a whole section of the list. A summarizer needs two metrics; a RAG chatbot needs eleven.
• A golden dataset is the asset; everything else is tooling. Stratify it rather than mirroring traffic, size it from the smallest difference you need to detect, and hold a slice back. A 200-example suite resolves about seven points — if someone asks you to prove one, the honest answer is that the suite cannot.
• Reference-free metrics measure internal consistency, never correctness. A system that retrieves the wrong document and faithfully summarises it scores 1.0 on faithfulness and is wrong.
• Measure your noise floor before you compare anything. Run the suite twice unchanged; the gap is the smallest difference you are entitled to call real. Most reverted "improvements" were inside it.
• Reliability is not capability. pass@k asks whether any attempt worked; pass^k asks whether all of them did. Users live in the second one, because they do not get to silently retry.
• Two runs can both be right and still disagree. Correct, grounded, and each citing a different subset of the same supporting facts — invisible to correctness and to faithfulness. Score which facts each run used: the overlap between runs, and the stable core every run includes. A stable core of zero means there is no such thing as the answer in your system, only a distribution of them.
• Comparison beats valuation. "Is this good?" has no stable answer and the scale wanders; "do these agree?" is nearly objective. Where you have labels, a reference-guided judge is the design that agrees with humans most.
• A judge is a classifier, so test it like one. Build a small labelled meta-set, report kappa rather than raw agreement, compare against the majority-class baseline and the human ceiling, and pin its version — an unpinned judge silently splits your trend line in two.
• Bias is measured, not assumed. Swap every pairwise comparison and count only the responses that win both orders. That is the one mitigation that addresses position bias at the root.
• The RAG triad localises failure: context relevance (retrieval), groundedness (did it stick to context), answer relevance (did it answer). Faithfulness high and recall low means the retriever; the reverse means the generator.
• Per-turn quality is necessary and not sufficient. Constraint decay, self-contradiction and dropped intentions are relations between turns, so every turn can pass while the conversation fails. Knowledge retention and constraint decay are different failures and both need measuring.
• A simulated user that never fails is measuring its own politeness. Point the suite at a deliberately broken assistant; if completion does not collapse, every number it has produced is worthless.
• Every extra slice is another chance to be wrong. Twelve slices at a 5% threshold raise a false alarm about half the time on an unchanged system. Name the primary metric before the run.
• A score belongs to a configuration. Change the system and you have an experiment; change the dataset, judge or rubric and the delta is meaningless. Emit a manifest so that distinction is a check rather than an argument.
• Offline is a screen; online is the decision. Compute the correlation between the two across shipped changes — until you have, your gate's authority is a claim rather than a fact.

What you can now do

Choose the right metrics for the system you actually built. Build a golden dataset that earns its keep, and know what it can and cannot resolve. Calibrate a judge against humans and catch it drifting. Score a conversation, not just its turns. Stand up a harness that runs in CI, emits a manifest, and blocks a bad merge. Tell a real regression from a slice that was always going to alarm.

If one thing sticks

If you can't say what would have to be true for you to roll back, you don't have an evaluation — you have a report.