Your AI works in the demo because the demo is one input you already know the answer to. An eval is what tells you it still works across the thousand inputs you have not tried yet — and whether the change you shipped this morning quietly broke a hundred of them. This guide is the working discipline: what evals measure, the four types a production system needs, and a harness blueprint you can build this week, wire into CI, and use to block a bad release before it reaches a user.
What an eval is — and why one good demo is not one
An eval is a repeatable, scored test over a fixed set of inputs. You define the inputs, define what an acceptable output looks like, run the system, and get a number: pass rate, mean score, count of regressions. Same inputs next week, same scoring, comparable number. That comparability is the whole point.
A demo has none of it. A demo is one input, chosen because it works, scored by a human nodding. It cannot tell you the pass rate on the inputs you did not pick, cannot be re-run to detect a regression, and cannot gate a deploy. “It looked great when I tried it” does not survive a prompt change, a model version bump, a new retrieval index, or a temperature tweak — each silently shifts behavior across your input distribution, and only a scored, re-runnable eval catches the shift.
The mental model: treat the LLM feature as a system under test whose output is probabilistic. You are not asserting one exact string the way a unit test does — you measure a distribution, what fraction of inputs land inside the band you require, and watch that fraction move when something changes.
The four eval types a production system needs
Different failures need different evals. A production AI feature needs all four; many teams build only the first and get surprised by the other three.
- Correctness — does the output do the task? For RAG: is the answer faithful to the retrieved context. For extraction: are the fields right. Scored against a golden set of known-good outputs. This is the eval everyone builds.
- Safety — does the system refuse and resist what it must? Prompt injection (OWASP LLM01), sensitive information disclosure (OWASP LLM02), jailbreaks, and tool misuse in agents (OWASP Agentic Top 10). Scored against an adversarial set written to break the system, mapped to the OWASP LLM Top 10 and MITRE ATLAS so coverage stays honest. Deep-dive: /guides/prompt-injection-testing.
- Regression — did a change break something that used to work? A diff, not an absolute score: run the fixed suite before and after the change, flag every case that went from pass to fail. One prompt edit routinely regresses five behaviors; without this eval you ship the trade blind.
- Drift — is production behavior leaving the baseline you tested? Inputs, the provider’s weights, and the retrieved corpus all change under you. Drift evals run on a schedule against live-shaped traffic. Covered below.
How to build the harness
An eval harness is not a framework purchase. It is five components you own, wired so one command produces a scored, comparable result.
1. Golden sets
A golden set is a curated list of inputs paired with the output you require — or with a rubric that scores the output when there is no single right answer. Rules that keep it honest:
- Source from reality. Draw inputs from real production traffic, not cases you invented to pass. The hard, ambiguous, and adversarial inputs are the ones that matter; a set of easy questions proves nothing.
- Version it in git. The set is code — every input, expected output, and rubric reviewed and diffed like any other change, so you can trace why a score moved.
- Keep a frozen holdout. Reserve a slice you never look at while tuning prompts, or you overfit your prompt to your own test set. The holdout is your out-of-sample read.
- Size for signal. A few dozen well-labeled cases per behavior beats thousands of noisy ones. Grow it every time production surprises you — each incident becomes a permanent case.
2. Judges — and judge calibration
Some outputs score themselves: exact match, JSON-schema validity, a regex, a retrieval-precision number. Use those first — free, fast, deterministic. When the criterion is qualitative (“is this answer faithful to the context”, “is the tone appropriate”), you reach for an LLM-as-judge: a model scoring output against a rubric.
An uncalibrated judge is a liability. It disagrees with your reviewers, drifts with the provider’s model updates, and will happily pass output a human would reject. Calibrate before you trust it:
- Hand-label a slice of outputs with human reviewers — the ground truth.
- Run the judge on the same slice and measure agreement. Low agreement means the rubric is ambiguous or the task is not judge-able — fix the rubric or fall back to human review.
- Re-calibrate whenever you change the judge model or its prompt. A judge is a model under test too, and gets its own regression eval.
Reserve the judge for criteria it scores consistently; keep a human in the loop for the rest. A judge that has never been checked against a human is not evidence — it is a second opinion of unknown quality dressed up as a number.
3. Replay tests
Capture real production request/response traces — inputs, retrieved context, tool calls, final output — and replay them through the current build. Replay reproduces a specific past failure on demand and proves a fix holds without waiting for the input to recur. It is also the bridge to regression: today’s incident, captured as a trace, becomes tomorrow’s permanent regression case. Reproducible evidence — the actual request trace, not a screenshot of a vibe — is what makes a fix verifiable.
4. Gates in CI
The harness runs on every pull request that touches a prompt, a model version, a retrieval config, or agent tooling. It runs the fixed suite, scores it, diffs against the baseline, and returns pass or fail. A red eval blocks the merge the same way a red unit test does. Without a gate, the harness is a dashboard nobody reads; with one, it is a control.
The blueprint
A minimal harness has five parts, each catching a distinct class of failure. Build all five or leave a blind spot.
| Component | What it is | What it catches |
|---|---|---|
| Golden set (+ frozen holdout) | Versioned inputs with known-good outputs or rubrics, plus an untouched out-of-sample slice | Correctness failures; overfitting to your own test set |
| Deterministic scorers | Exact match, JSON-schema validation, regex, retrieval precision/recall | Format breaks, malformed tool calls, missing fields |
| Calibrated judge | LLM-as-judge scored against a human-labeled slice, agreement measured | Qualitative failures (faithfulness, tone) a rule cannot express |
| Adversarial set | Attack inputs mapped to OWASP LLM/Agentic Top 10 and MITRE ATLAS | Prompt injection, data leakage, jailbreaks, unsafe tool use |
| Replay + regression diff | Captured production traces re-run, diffed against the prior run | Silent regressions; reproduces a past failure on demand |
Wire the five behind one command, run it in CI, and store every run so scores stay comparable over time. That store is what turns “it felt better” into “correctness held at baseline, safety pass rate up four cases, zero regressions.”
Monitoring drift in production
Passing every gate at release does not keep you passing. Three things move under you: the input mix (users ask new things), the model (providers update weights and deprecate versions), and the retrieved corpus (your own documents change). Any one drifts live behavior away from the baseline you signed off on.
Run drift evals on a schedule against live-shaped traffic and compare to the release baseline. Track a few stable signals — correctness pass rate on a sampled slice, safety pass rate on the adversarial set, output-format validity, refusal rate, and the judge’s agreement with periodic human spot-checks. Alert on a sustained move, not a single noisy run. When a drift eval crosses your threshold, that is the signal to re-assess and re-baseline before a user finds the gap. This is continuous assurance, not a one-time audit — the discipline behind our BUILD & RUN work: /services/ai-saas-build-run.
Release gates: when a FAIL blocks the ship
A gate is only real if it can stop a release. Decide the thresholds before you are under pressure to ship, and write them down. A workable default:
- Safety regressions block, always. A new prompt injection or data-disclosure failure that was passing is a hard stop — no override without a named owner accepting the risk in writing. This is where the OWASP LLM Top 10 mapping earns its keep: the gate names which control regressed.
- Correctness has a floor and a no-regression rule. Pass rate must stay at or above baseline on the frozen holdout, and no prior-passing case may flip to fail without review. A change that improves the average while silently breaking a known case does not ship on the average alone.
- Format and schema failures block hard. If downstream code parses the output, a malformed-output regression is a production incident waiting to happen — and deterministic scorers make this gate cheap.
- Drift alerts trigger re-assessment, not an auto-block. The gate is the release pipeline; the alert is the monitor.
The honest part: an eval proves what you tested, on the inputs you chose, at the moment you ran it. It does not certify that your AI is “safe” — no eval can, and we will never tell you otherwise. What it gives you is a defensible, reproducible answer to “what did you test, and what happened” — the same answer an enterprise security reviewer will ask you for. That gap between “tested” and “safe” is exactly why the assessment is the door: /services/ai-product-readiness-assessment.
We run this on our own product
Phixe runs this discipline on faben, our own AI product — golden sets, a calibrated judge, replay tests, and gates that block our own ship when a FAIL earns it. We do not sell a practice we do not run ourselves. See how it shows up in our work: /work.
If your AI has to pass an enterprise security review, or you want to know where it breaks before a customer does, an assessment is the fastest way to a real answer — reproducible evidence, mapped to the frameworks the reviewer already trusts: /services/ai-product-readiness-assessment. To go deeper, read /guides/llm-red-teaming and /guides/ai-production-readiness-checklist, and see how we run the engagement at /methodology.