Almost every AI feature that stalls between prototype and production stalls for the same reason: nobody agreed what good enough meant. The demo worked, a few people tried it, the output looked plausible, and the project moved forward on that basis. Then someone asked how often it is wrong, and there was no answer: not a bad answer, no answer at all. Evaluation turns that question into a number you can act on, and it is cheapest to set up before you build.
Define acceptance before you build
The first artifact is not a prompt or a model choice. It is a sentence in business language that says what the feature has to do and how often it has to do it. For example: this classifier must route at least 95 percent of inbound requests to the correct queue, and the ones it routes wrongly must be recoverable within one business day. That sentence is testable. It also forces the conversation about consequences, which is the part teams skip.
Setting the threshold requires knowing the current baseline, and that number is usually uncomfortable to collect. People assume manual work is close to perfect. It rarely is. Measure the existing process on the same cases you will use to measure the model. If a person misroutes 8 percent of tickets today, a model at 6 percent is an improvement even though 94 percent sounds unimpressive in isolation. Without a baseline you are comparing against an imagined standard nobody currently meets.
- What decision or output does the feature produce, stated in one sentence
- How often it must be correct, as a number, against a defined case set
- What a wrong answer costs, and who absorbs that cost
- How a wrong answer is detected and corrected, and how fast
- What the current manual process scores on the same cases
Build an evaluation set from real cases
An evaluation set is a fixed collection of real inputs with the answers you would accept, held constant so that every change to the prompt, the model, the retrieval step, or the surrounding code can be scored the same way. Fifty to two hundred well-chosen cases is usually enough to make decisions. Thousands of cases that all look alike are worth less than eighty that cover the range.
The cases people naturally collect are the easy ones, because those are the ones sitting at the top of the queue. Deliberately go and find the hard ones. Pull the records that took a person the longest to handle, the ones that were escalated, the ones two experienced staff disagreed about, and the ones that were handled incorrectly and later corrected. Add adversarial inputs: empty fields, truncated documents, the wrong document type entirely, contradictory information in the same record, and text that instructs the system to ignore its instructions. If your evaluation set contains no case that the feature fails, it is not measuring anything.
Labelling those cases is the expensive part, and it belongs to the people who do the work rather than to the engineers. Where two reviewers disagree on the correct answer, that disagreement is information: the task is ambiguous, and an ambiguous task needs a human in the loop or a tighter definition before it needs a model.
Why reviewing output by feel does not survive a change
Informal review works right up until something moves. Then a prompt is edited to fix one complaint, a model version is retired by the provider, a retrieval index is rebuilt, or a new document type enters the pipeline. Nobody can say whether overall quality went up or down, because there was never a score to compare against. Teams in this position tend to freeze: they stop improving the feature because they cannot prove a change is safe.
The second problem is that these systems are not deterministic. The same input can produce different output on two runs, so a single failed case proves nothing and a single successful case proves less. Regression testing has to be statistical. Run the evaluation set several times, record the score and its spread, and treat a change as an improvement only when it moves the score by more than the run-to-run variation you already observe. Set the pass condition as a threshold on the aggregate, plus a small set of cases that must never fail: the ones with a legal, financial, or safety consequence.
Choose the metric that matches the task shape
There is no general measure of AI quality. There is only the measure that matches what the feature is asked to do, and picking the wrong one produces a number that improves while the feature gets worse. Extraction and classification have a right answer, so score them against it. Triage decides what a person looks at next, so missing a real case is far more expensive than sending a false alarm. Optimize recall and accept the review cost. Drafting has no single right answer, so the only meaningful comparison is whether a qualified reviewer prefers the output to what the current process produces, and how much editing it takes to make it usable.
| Task shape | Primary metric | Threshold to set before building | Common mistake |
|---|---|---|---|
| Structured extraction from documents | Exact match per field, scored field by field | Per-field accuracy on the fields that matter, plus a hard rule that a low-confidence field is left blank rather than guessed | Reporting one document-level accuracy number that hides the two fields everyone actually depends on |
| Classification and routing | Accuracy per class, with a confusion matrix | Overall accuracy above the measured manual baseline, and no single class below a floor you can live with | Averaging across classes so a rare, high-consequence category disappears into the mean |
| Triage and flagging for review | Recall on true positives, with false-positive rate as a cost | Recall high enough that missed cases are tolerable, then tune precision to fit reviewer capacity | Optimizing precision because it makes the demo look clean, while real cases are silently dropped |
| Summarization of source material | Faithfulness to the source, checked against cited passages | Zero tolerance for claims not present in the source; fluency is not scored | Judging summaries on how well they read rather than whether they are supported |
| Drafting and first-pass content | Human preference versus the current process, plus edit distance to final | Preferred by reviewers more often than not, and faster end to end than writing from scratch | Measuring output volume instead of the time it takes to get to something sendable |
| Retrieval-backed question answering | Answer correctness plus whether the right source was retrieved | Both scored separately, so a wrong answer from correct sources is distinguishable from a retrieval failure | Scoring only the final answer, then tuning the model when the retrieval step is the actual problem |
Cost and latency are part of acceptance
Quality is one of three acceptance dimensions and it is the only one most teams measure. A configuration that scores two points higher but takes eleven seconds inside a workflow where the user expects an instant response has not won. Neither has one that clears the quality bar at a per-item cost that exceeds the labor it replaces. Record cost per item and response time on the same evaluation run that produces the quality score, so every comparison is made across all three at once.
Doing this early also stops a late-stage surprise. Techniques that raise quality (larger context, multiple passes, self-checking, retrieval over more material) raise cost and latency too. Measured only after the quality target is met, a feature can pass evaluation and still be unshippable at volume.
Set the failure behaviour deliberately
Every AI feature will be wrong sometimes. The design question is what it does when it is. There are three honest options: refuse and say nothing, escalate to a person with the context they need, or degrade to whatever the process was before. Any of the three can be right. What is never right is producing a confident answer with no signal that it might be wrong, because the user has no way to know which outputs deserve scrutiny.
Whichever behaviour you choose, it needs the same evaluation discipline as the success path. Confirm that low-confidence cases actually route to the escalation queue, that someone owns that queue, and that the fallback still works after months of nobody using it. A fallback that has silently been broken since launch is the same as having none.
How to run the evaluation, and keep running it
- 1
Write the acceptance sentence: what the feature outputs, how often it must be right, and what a wrong answer costs. Get the operations owner to agree to it in writing.
- 2
Measure the current manual process on a sample of real cases so you have a baseline to compare against.
- 3
Assemble 50 to 200 labelled cases, weighted toward the hard, ambiguous, and adversarial ones, and freeze the set.
- 4
Choose the metric that fits the task shape, and instrument cost per item and response time on the same run.
- 5
Establish run-to-run variation by scoring the same configuration several times before you start comparing configurations.
- 6
Make one change at a time, rescore, and keep only changes that move the aggregate by more than that variation.
- 7
Define and test the failure behaviour, including who receives escalations and how quickly they act.
- 8
Ship behind a review step or a limited user group, log every input and output, and compare production scores to evaluation scores weekly.
- 9
Add every production failure to the evaluation set so the same mistake cannot return unnoticed.
Evaluation does not stop at launch
An evaluation set is a snapshot of the inputs you had when you built it. Inputs drift. A vendor changes a form layout, a new region sends documents in a different format, or a policy change alters what the correct answer is. Quality degrades quietly in all of these cases because nothing errors. The output is still fluent and still arrives on time.
The practical monitoring set is small: score a sample of production cases on a schedule, track how often humans override or correct the output, watch cost and latency for drift, and rerun the full evaluation set whenever the model, prompt, or retrieval layer changes. Name an owner for the number. A quality metric with no owner stops being looked at within a quarter.
Common questions
- How many test cases do we need for a useful evaluation set?
- Fifty to two hundred labelled cases is usually enough to make shipping decisions, provided they span the real range of inputs rather than the easy ones. Coverage matters more than volume: a set of eighty cases that includes the ambiguous, rare, and adversarial examples will catch more problems than a thousand routine ones. Grow the set over time by adding every production failure to it.
- How do you regression test something that is not deterministic?
- Score in aggregate rather than case by case. Run the evaluation set several times against an unchanged configuration first to learn how much the score naturally varies, then treat a change as real only when it moves the score by more than that. Keep a small set of high-consequence cases that must pass every run, and treat any failure there as a blocker regardless of the aggregate.
- What accuracy is good enough to ship?
- It depends entirely on what a wrong answer costs and how quickly it is caught. A drafting aid that a person reviews every time can ship well below the bar required for anything that moves money or makes a commitment to a customer. Set the number against the measured performance of the current process, not against an abstract standard.
- Can we use a model to grade the model's output?
- Yes, for tasks where human grading does not scale, but only after you have checked the grader against human labels on a sample and know how often the two agree. An automated grader with unmeasured agreement gives you a number that moves without telling you whether quality moved. Keep humans grading a sample permanently so grader drift is visible.
- Who should own evaluation, engineering or the business?
- The acceptance threshold and the labelled answers belong to the people who do the work, because they are the only ones who can say what a correct output is. Engineering owns the harness that runs the set and reports the score. When engineering owns both, the metric tends to drift toward what is easy to measure rather than what matters.
- How often should we re-run evaluation after launch?
- Run the full set on any change to the model, prompt, or retrieval layer, and on a fixed schedule regardless. Monthly is a reasonable default for most operational workflows. Between full runs, monitor the human override rate on production output, which is the earliest signal that inputs have drifted away from what you tested.