Why evaluation is different for AI

A traditional software function sort(list) returns the same output every time for the same input. You write an assertion, it passes or fails, and that binary signal is sufficient for a test suite. An LLM call openai.chat.completions.create(model="gpt-4o", messages=[...], temperature=0.7) returns a different string on every invocation — even with identical inputs. The sampling procedure selects tokens probabilistically from the model's output distribution, and temperature controls the entropy of that distribution. At temperature=0.0, the model takes the argmax at each step (greedy decoding), which is nearly deterministic but still not guaranteed across API versions or hardware. At temperature=0.7, the top-p nucleus (Holtzman et al. 2020) samples from a truncated distribution, producing genuinely different outputs each run.

from openai import OpenAI

client = OpenAI()

responses = []
for _ in range(5):
    r = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Explain gradient descent in one sentence."}],
        temperature=0.7,
        max_tokens=100
    )
    responses.append(r.choices[0].message.content)

for i, resp in enumerate(responses):
    print(f"Run {i}: {resp[:80]}...")

Five runs produce five different explanations — all correct, none identical. The character-level edit distance between any two responses averages 40-60% of the string length. Traditional assertEqual is useless here. This is the foundational problem: evaluation for generative AI must tolerate semantic equivalence across surface-level variation.

The evaluation spectrum

Evaluation methods form a spectrum from cheapest/most brittle to most expensive/most flexible:

Exact match: output == expected. Works only for classification, extraction, or constrained generation (JSON keys, enum values). Cost: zero. Failure mode: rejects valid paraphrases.
Pattern/regex match: re.search(pattern, output). Captures structural requirements (contains a date, starts with a number). Cost: zero. Failure mode: over-constrains format.
Token overlap (BLEU, ROUGE): Measures n-gram overlap between output and reference. Cost: negligible. Failure mode: low correlation with human judgment for open-ended generation (r=0.25-0.40 on summarization, Liu et al. 2023).
Embedding similarity: cosine(embed(output), embed(reference)). Captures semantic meaning. Cost: one embedding call per comparison (~$0.0001). Failure mode: high similarity doesn't guarantee factual correctness.
LLM-as-judge: A strong model (GPT-4o, Claude Sonnet 4) scores the output against criteria. Cost: $0.005-0.02 per judgment. Failure mode: systematic biases (position, verbosity, self-preference).
Human review: Domain expert evaluates output. Cost: $0.50-5.00 per example. Failure mode: inter-annotator disagreement, doesn't scale.

The art of evaluation engineering is choosing the cheapest method that correlates sufficiently with human judgment for your specific task. A customer service bot that must extract order IDs can use regex. A creative writing assistant requires LLM-as-judge or human review.

Offline vs online evaluation

Offline evaluation runs before deployment against a fixed dataset. You maintain a golden dataset — curated input-output pairs representing your production workload — and run your pipeline against it on every code change. This catches regressions: a prompt edit that improves summarization quality but breaks extraction accuracy.

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

metric = AnswerRelevancyMetric(threshold=0.7, model="gpt-4o")

test_cases = [
    LLMTestCase(
        input="What is the capital of France?",
        actual_output="Paris is the capital of France.",
    ),
    LLMTestCase(
        input="Explain photosynthesis briefly.",
        actual_output="Plants convert CO2 and water into glucose using sunlight.",
    ),
]

results = evaluate(test_cases, [metric])
print(f"Pass rate: {sum(r.success for r in results)}/{len(results)}")

Online evaluation runs in production on live traffic. You sample a percentage of requests (typically 1-10%) and score them asynchronously. This catches distribution shift — when real user queries diverge from your golden dataset — and gradual model degradation (provider model updates, context window changes, upstream data drift).

The two are complementary. Offline evaluation is your CI gate: it blocks bad deployments. Online evaluation is your monitoring system: it detects problems that your test set didn't anticipate.

The cost of not evaluating

Silent quality degradation is the default failure mode for LLM applications. Unlike a crash or a 500 error, a subtly worse response generates no alert. Consider a RAG system for internal documentation:

Week 1: 92% of answers are factually grounded in the retrieved context.
Week 4: A model provider update changes how the model handles long contexts. Grounding drops to 78%.
Week 8: Users stop trusting the system and revert to manual search. Usage drops 40%.

No metric moved. No alert fired. The product silently degraded because nobody measured faithfulness in production.

The OpenAI API has had at least 3 documented capability regressions in GPT-4 between March 2023 and March 2024 (Chen et al. 2023, "How Is ChatGPT's Behavior Changing over Time?"). Code generation pass@1 dropped from 67.0% to 33.0% between March and June 2023 on the same benchmark. Without continuous evaluation, you discover these regressions from user complaints weeks after they begin.

Non-determinism in practice

Even at temperature=0, outputs can vary due to:

Floating-point non-determinism: GPU parallel reductions accumulate floating-point errors in different orders depending on scheduling. This can flip the argmax at low-confidence positions.
API versioning: Model providers update weights behind stable model names. gpt-4o on January 15 is a different checkpoint than gpt-4o on March 15.
Batching effects: Some providers batch requests, and the batch composition can affect outputs through attention over padding tokens.

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
from openai import OpenAI

client = OpenAI()

def get_embedding(text):
    r = client.embeddings.create(model="text-embedding-3-small", input=text)
    return np.array(r.data[0].embedding)

prompt = "List three benefits of unit testing."
outputs = []
for _ in range(10):
    r = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
        seed=42
    )
    outputs.append(r.choices[0].message.content)

unique_outputs = set(outputs)
print(f"Unique outputs at temp=0 with seed: {len(unique_outputs)}/10")

embeddings = np.array([get_embedding(o) for o in outputs])
sims = cosine_similarity(embeddings)
print(f"Min pairwise similarity: {sims[np.triu_indices_from(sims, k=1)].min():.4f}")
print(f"Mean pairwise similarity: {sims[np.triu_indices_from(sims, k=1)].mean():.4f}")

With seed=42 and temperature=0, you typically get 1-2 unique outputs across 10 runs (the system_fingerprint in the response header changes when the backend rotates). The cosine similarity between variants is usually >0.97 — semantically identical but not string-identical. This is why evaluation must operate at the semantic level, not the string level.

Evaluation as infrastructure

Evaluation for LLM systems is not a one-time check. It is infrastructure that must be maintained alongside the application:

Version your metrics alongside your prompts. A prompt change that optimizes for conciseness will tank a verbosity-rewarding metric.
Track metric distributions, not just means. A mean faithfulness score of 0.85 hides a bimodal distribution where 15% of responses are completely ungrounded.
Set alert thresholds on rolling windows. A 5% drop in answer relevance over a 24-hour window should page someone.
Budget for evaluation costs. Running GPT-4o-as-judge on 10% of traffic at 1000 req/day costs ~$1.50/day. Running on 100% costs ~$15/day. These are infrastructure costs, not optional extras.

import statistics

scores = [0.92, 0.88, 0.95, 0.31, 0.89, 0.93, 0.87, 0.12, 0.91, 0.90]

mean_score = statistics.mean(scores)
median_score = statistics.median(scores)
p10_score = sorted(scores)[1]  # 10th percentile approximation

print(f"Mean: {mean_score:.2f}")    # 0.77 — looks acceptable
print(f"Median: {median_score:.2f}") # 0.89 — most responses are fine
print(f"P10: {p10_score:.2f}")       # 0.12 — but the tail is catastrophic

The mean hides the failure cases. Production evaluation must track percentiles — specifically the bottom 10th percentile — because that is where user trust erodes. A system that is excellent 85% of the time and catastrophically wrong 15% of the time is worse than one that is mediocre 100% of the time, because users cannot predict when to trust it.