AI evals: what tells you a system still works

Evaluation usually enters AI projects late.

The workflow is designed, built, and shipped, and only once it is running does someone wire up evals as part of monitoring. At that stage evaluation becomes a quality check on a finished system.

In practice, though, the important moment comes earlier.

Not at the start of the project, but at the first point where an atomic workflow produces a real operational result. That may be a classified support ticket, an extracted contract clause, a translated product description, or a generated compliance summary. The workflow is no longer theoretical at that stage. It has begun interacting with operational reality.

This article argues that this is exactly the moment to stop and measure.

The value of doing so is not only that it creates a benchmark for later comparison. Measuring forces a kind of attention that building does not. While developing a workflow, teams focus mostly on making it produce output at all. Once that output is measured deliberately against defined parameters, other things begin to surface: inconsistencies, hidden assumptions, edge cases, operational risks, and eventually questions of responsibility.

Teams often intend to discuss responsible AI. But teams can only discuss what they can see. Evaluation is part of what makes the unseen visible early enough to act on it. Most of these rarely show up during enthusiastic prototyping. Measurement is often the first thing that makes them concrete enough to design around.

What to measureWhat you are actually measuring

At the first real result, record how the workflow performs against parameters chosen in advance. Not “it looks right.” Actual measurements, written down.

The parameters depend on the workflow: latency, consistency, escalation frequency, extraction accuracy, retrieval quality, confidence behaviour, reviewer correction rates, or task completion stability. But one category matters more than most: how the system behaves when it is wrong.

Every AI system is sometimes wrong, yet prototyping tends to skip past that reality. Teams judge workflows by their successful outputs and leave the failures for later.

Measurement does not let you do that.

A demo says the workflow is possible. A benchmark says it extracts the correct clause ninety percent of the time, silently misses renewal conditions in seven percent of cases, escalates appropriately in four, and confidently invents unsupported values in two. Once those numbers exist, someone has to ask about the remaining behaviour.

That question is the point.

It surfaces hidden assumptions, operational risks, and responsibility gaps while the system is still small enough to change cheaply. The earlier those behaviours become visible, the easier they are to design around instead of explain away after launch.

This is also why a single accuracy score is rarely enough. Extraction quality is not one number. Pulling party names correctly and missing a liability cap are not the same failure, and an aggregate score hides the difference. The Stanford HELM work (Liang et al., 2022)⁵ made this argument at the level of language models generally: a model has to be evaluated across several properties at once, including robustness and calibration, because a single headline metric conceals as much as it reveals. The same holds for a workflow. What you measure has to be broken out by what the output is actually for.

This is also where an important distinction appears: consistency is not correctness. A legal extraction workflow can return structurally valid outputs, repeatedly, while consistently misclassifying uncommon contractual language. The behaviour looks stable in operation, but the interpretation underneath it is wrong. A benchmark built only on repeatability will not tell the two apart.

Designing benchmarksHow to design a benchmark that does not lie to you

The easiest benchmark to build is one that confirms what the team already believes.

The workflow works on the clean examples. The extraction looks correct. The demo succeeds. Accuracy appears high. Very quickly, the benchmark stops measuring operational reliability and starts measuring how well the system handles the exact conditions it was designed around.

That is not evaluation. It is self confirmation.

A useful benchmark does something harder. It forces the workflow into the conditions where its assumptions begin to break.

Take a document intelligence workflow extracting key terms from legal contracts. If every benchmark document is cleanly scanned, consistently structured, and written in predictable language, the workflow is only being tested against the easiest version of reality.

Production rarely looks like that for long. Contracts arrive with missing sections, inconsistent formatting, mixed jurisdictions, poor OCR quality, contradictory clauses, unusual legal phrasing, and vendor specific templates that were never part of the original dataset. The difficult cases are not edge conditions in practice. Over time, they become normal operational conditions.

This is where many benchmarks quietly become misleading. High benchmark scores often reflect familiarity more than robustness. The system learns the structure of the benchmark instead of learning the task itself, and benchmark performance diverges from production performance because the workflow has been optimised for the measurement environment rather than the operational one.

This has a name. Goodhart’s law, first set out by the economist Charles Goodhart in 1975,¹ holds that when a measure becomes a target, it stops being a good measure. A benchmark is a measure. The moment a workflow is tuned to maximise its benchmark score, the score begins to describe the workflow’s fit to the benchmark rather than its fit to the task. Improvement against the number and improvement in reality quietly stop being the same thing.

Production conditions shift over time as well, so a benchmark fixed to a single snapshot of reality ages out of usefulness. A benchmark that stays honest has variation deliberately built in. Not enough to make the task impossible. Enough to expose where the workflow stops generalising reliably.

That changes the purpose of evaluation entirely. The benchmark is no longer a scoreboard proving the system works. It becomes an instrument for locating the edges of operational reliability before production discovers them first.

Offline vs productionTwo kinds of evaluation

One distinction matters here because it changes what can actually be measured.

Offline evaluation checks correctness against known answers. A workflow runs against a prepared set of inputs where the expected outputs already exist. This is the stage where you can directly measure whether the system is right. For a legal extraction workflow, this means running contracts through the system where reviewers have already validated the correct clause boundaries, obligations, dates, and classifications.

Production evaluation works differently because production has no answer key.

Live contracts are new. Nobody has pre-computed the correct extraction output, and often nobody can do so fast enough to matter operationally. Production evaluation therefore cannot continuously ask: “is this exact output correct?” Instead, it asks whether the workflow is still behaving like the system that was benchmarked earlier.

That changes what gets measured. Offline evaluation measures correctness directly. Production evaluation measures behavioural properties: consistency, escalation rates, reviewer correction frequency, formatting stability, confidence alignment, retrieval quality, and drift over time.

This is why the benchmark captured at the first real result matters beyond testing. Offline, it proves the workflow is operationally sound under known conditions. In production, it becomes the behavioural reference the live system is continuously compared against.

Two kinds of evaluation — offline vs production

Silent failuresThe failure modes measurement makes visible

Some failures announce themselves. The workflow crashes. An error is thrown. A dashboard turns red. These are easy failures because the system signals them clearly.

The failures that matter here are the quiet ones. The output still looks correct: fluent, structured, confident. Nothing in the response itself signals that anything is wrong.

Silent failure is the broad case. A workflow extracts the wrong clause while preserving the correct structure, format, and tone. Hallucination is a more specific version of this, where the system invents a detail, citation, obligation, or value and presents it with full confidence.

Drift behaves differently again. The workflow performs well initially, then slowly degrades as production inputs stop resembling the conditions it was originally benchmarked against. This is not a vague idea. It is a studied phenomenon: the machine learning literature calls it dataset shift, the condition where the data a system meets in production diverges from the data it was trained and evaluated on (Quiñonero-Candela et al., 2009).² There is no obvious break point. The system simply becomes less reliable as the shift widens.

What makes these failures hard to catch is that none is detectable from a single output viewed alone. They show only as a gap from a standard — a known answer offline, a benchmarked behaviour pattern in production. Without that reference, a quiet failure is found by a user, late.

Operational driftWhat teams usually get wrong operationally

Most operational failures in AI systems do not come from models suddenly becoming unintelligent. They emerge from small assumptions accumulating quietly across the workflow.

A retrieval pipeline changes its chunking behaviour, an OCR layer degrades document quality, a prompt update improves one extraction category while weakening another, an escalation threshold is relaxed to reduce queue volume, and operators gradually stop checking fields that usually look correct. None of these changes individually appears catastrophic. Together, they slowly alter the behaviour of the system.

This is a well-documented property of production machine learning. Sculley et al., in their 2015 paper on hidden technical debt in machine learning systems,³ described how ML systems accumulate operational complexity not in the model but in everything around it — pipelines, dependencies, feedback loops, and the coupling between components. Sambasivan et al., in their 2021 study of data cascades,⁴ showed the related pattern from the data side: small, under-attended problems in the data pipeline compound into significant downstream failures in deployed, high-stakes AI systems. Both describe the same shape of failure the article is pointing at — not a dramatic break, but quiet accumulation.

This is one reason production reliability often fails in the seams between components rather than inside the model itself. In practice, the benchmark rarely measures the model alone. It measures the entire workflow around it: OCR quality, retrieval quality, schema constraints, escalation logic, review behaviour, post-processing rules, and how operators interact with the outputs.

Production systems also do not stay purely technical for long. People adapt around them continuously. Reviewers learn which outputs are usually safe, operators invent informal escalation patterns, teams restructure documents to improve extraction quality, and certain failure categories quietly become accepted operational debt. After enough time, the human behaviour becomes part of the workflow itself.

Offline benchmark datasets rarely capture these adaptations, yet they strongly influence real-world reliability. The workflow changes even when nobody believes the system itself has changed.

Benchmarks as governanceBenchmarks become operational contracts

The benchmark becomes an operational contract: a definition of what the organisation believes acceptable system behaviour looks like under real conditions.

As workflows mature, benchmarks begin influencing operational decisions directly.

Thresholds define when outputs are auto-approved, when escalation is mandatory, which documents require review, what confidence level is acceptable, and what failure rate the organisation will tolerate. At that point, evaluation stops functioning as a purely technical measurement layer. The benchmark becomes part of workflow governance.

In document intelligence systems, benchmark definitions eventually shape auditability requirements, review staffing, compliance procedures, risk tolerance, escalation design, and operational accountability itself. The benchmark becomes an operational contract: a definition of what the organisation believes acceptable system behaviour looks like under real conditions. It is rarely written in a single place, but every escalation threshold, approval rule, reviewer override, and confidence cutoff is a clause of it.

That contract is not signed once. As models change, document distributions drift, and operators adapt around the system, the contract describes conditions that have started to move. A benchmark nobody revisits slowly governs a system that no longer exists. The contract remains necessary. It cannot remain static.

This is also why benchmark design cannot be separated from operational responsibility. A poorly designed benchmark does not merely produce misleading scores. It gradually shapes misleading operational behaviour around the system itself.

Attribution and ownershipResponsibility emerges through measurement

Before workflows are measured systematically, failures remain difficult to classify consistently. After evaluation exists, behaviour becomes attributable.

Teams can now ask why a threshold was set where it was, why recall was prioritised over precision, why reviewer correction rates rose after a workflow update, why escalation was disabled for one document category, why extraction quality declined after the retrieval pipeline changed.

Each of those questions has an owner and an answer only because the measurement exists to make it askable. That is what turns a benchmark into more than a score. It is the first point where an AI workflow becomes operationally understandable — a reference stable enough for a team to reason about behaviour before the workflow becomes too large, too distributed, and too embedded to understand clearly.

The closeClosing perspective

Return to the question at the heart of this article. How do you know your AI still works?

Not by looking at its output. The output looks right even when it is wrong — that is the defining property of the failures that matter. You know the system still works only by measuring it against a reference: a benchmark established at the first real result, and kept current as the system and its conditions change.

That is the shift this article argues for. Evaluation is not a late-stage check on a finished system. It is the instrument that makes an AI workflow observable at all — the point where behaviour becomes measurable, failure becomes classifiable, and the assumptions buried in the system become explicit enough to take responsibility for.

The benchmark does not monitor the workflow from outside. Over time, it becomes part of how the workflow is defined, and part of how the team knows it is still the workflow they built.