Evals: scoring model output
You can't improve what you don't measure, and "it seemed fine when I tried it" is not measurement. An eval (evaluation) is the unit test of AI engineering: a fixed set of test cases run through your model, each output scored against an expected answer, summarised as a pass-rate.
The pieces:
- Cases — a list of
{input, expected}pairs. Your held-out truth. - Metric — a function that scores one prediction against its expected answer. The simplest is exact match (equal after normalising case and whitespace); richer ones check for a substring, a number within tolerance, or use another model as a judge.
- Harness — runs every case through the model, applies the metric, and records
{input, expected, predicted, passed}for each. - Pass-rate — the fraction that passed. A single number you can track over time.
- Regression detection — the failing cases. A regression is something that used to work and now doesn't; surfacing those rows is how you catch a change that quietly broke things *before* it ships.
The goal isn't a perfect score on day one — it's a number you can watch. If the pass-rate drops after a prompt tweak or a model swap, your eval just earned its keep.
Syntax
def exact_match(predicted, expected):
return predicted.strip().lower() == expected.strip().lower()
results = run_eval(cases, model_fn, exact_match) # list of {input,expected,predicted,passed}
rate = pass_rate(results) # passed / total (0.0 if empty)
regressions = [r for r in results if not r["passed"]]Worked examples
A metric, then the eval harness
def exact_match(predicted, expected):
return predicted.strip().lower() == expected.strip().lower()
def run_eval(cases, model_fn, metric):
results = []
for case in cases:
prediction = model_fn(case["input"])
results.append({
"input": case["input"],
"expected": case["expected"],
"predicted": prediction,
"passed": metric(prediction, case["expected"]),
})
return results
print(exact_match(" Paris ", "paris")) # True (case + whitespace normalised)
print(exact_match("Paris", "Madrid")) # FalsePass-rate and regression detection
def pass_rate(results):
if not results:
return 0.0
return sum(1 for r in results if r["passed"]) / len(results)
def find_regressions(results):
return [r for r in results if not r["passed"]]
# A stub model where the Spain answer is deliberately wrong:
def model_stub(q):
return {"capital of france": "Paris", "capital of japan": "Tokyo",
"capital of italy": "Rome", "capital of spain": "Barcelona"}.get(q.lower(), "unknown")
cases = [
{"input": "capital of france", "expected": "Paris"},
{"input": "capital of japan", "expected": "Tokyo"},
{"input": "capital of italy", "expected": "Rome"},
{"input": "capital of spain", "expected": "Madrid"}, # model says Barcelona
]
results = run_eval(cases, model_stub, exact_match)
print(pass_rate(results)) # 0.75 (3 of 4)
print([r["input"] for r in find_regressions(results)]) # ['capital of spain']Empty cases must not crash (guard the divide)
# pass_rate([]) returns 0.0 thanks to the `if not results` guard —
# without it, dividing by len([]) == 0 raises ZeroDivisionError.
print(pass_rate([])) # 0.0Common mistakes & gotchas
Exact match is brutal — pick the metric to fit the task
exact_match rejects "Paris." against "Paris" over a full stop. For free-form answers that's too strict; a substring check, a number-with-tolerance, or an LLM-as-judge metric fits better. Choosing the wrong metric makes a good model look broken (or a broken one look fine). The metric is a design decision, not a default.
Divide-by-zero on an empty case list
passed / len(results) raises ZeroDivisionError when there are no cases. Always guard with if not results: return 0.0. An empty eval should report 0.0 (or be flagged), never crash the harness that's supposed to be checking everything else.
An eval is only as honest as its cases
If your test cases are too easy, too few, or accidentally leaked into the prompt, a high pass-rate means nothing. Evals catch regressions only on what they actually cover — broaden the cases as you find new failure modes, and treat the pass-rate as a floor of confidence, not a guarantee.
Why it matters
Evals are what turn AI work from vibes into engineering. They let you swap a model, tweak a prompt, or change retrieval and *know* whether it got better or worse — and they catch the regression that would otherwise reach a user. Every serious AI team runs evals on every change; it's the difference between shipping with confidence and shipping with crossed fingers.