Evaluation harness: proving an LLM feature still works after each prompt change

Someone changes one sentence in a prompt to fix a complaint, and three weeks later finance notices that supplier names have been wrong since the Tuesday after. An evaluation harness turns that discovery into a failed test before the merge, and it is the minimum equipment for any team running an LLM feature that other people depend on.

What is an evaluation harness?

Four parts: a reference dataset of real inputs with known correct outputs, a runner that pushes each input through the actual feature (same prompt, model, retrieval and tools), a scorer that compares the output with the expected answer, and a report that shows the score against the last run. It is the unit test suite of an AI feature, with one difference: individual cases are scored, not passed or failed, because the model is probabilistic. Pass and fail apply to the aggregate — "urgent-ticket recall stayed above 95%" — not to a single ticket.

It is not a benchmark: a benchmark says which model is generally better at something; a harness says whether your feature, on your data, still does the job after the change you are about to ship.

Building the reference set from real cases

The cases come from production or the archive: last quarter's invoices, a random sample of tickets, the documents support remembers because they went wrong. Take them as they are — scanned, forwarded three times, with a signature block in the middle. Synthetic cases are easy to generate and easy to pass; they hide the failures that matter.

Each case gets a label: the correct extracted fields, the correct class, or a reference answer graded by a rubric. Two people label independently for at least part of the set, and the disagreement rate is recorded. If two humans agree on urgency only 90% of the time, a model at 92% is not underperforming; the label is the limit. Start with 50–100 cases, grow to 300–500 as failures are found, and stratify: the common shapes, the rare ones, and the hostile ones from red-teaming. Anonymize before storing. Version the set in the repository or a Postgres table, and hold back a portion that is never used while tuning, so its score stays honest.

Metrics per task type

One number for the whole feature is close to useless. "94% accurate" on invoice extraction can mean amounts at 99% and bank account numbers at 86%, which is the one field where 86% is unacceptable. Score per field, per class, per rubric item.

LLM-as-judge is the practical way to grade free text at scale, with non-optional caveats. Calibrate the judge first: have humans grade 50 outputs, run the judge on the same 50, and use it only once the two agree at a rate you can live with. Use a different model from the one being judged. Expect bias toward longer, more confident answers. And never let the judge decide a safety case — whether a payment proposal was correct is a deterministic comparison against the source of truth, not an opinion.

Running it on every change

The harness runs whenever the prompt, the model version, the retrieval configuration, a tool schema or the post-processing code changes. In practice it is a CI job that takes a few minutes and costs a few euros per run for a few hundred cases. The output is a score per metric, the difference from the baseline run, and the list of cases that flipped either way.

The regression gate is the part that makes it matter. A merge is blocked if any gated metric falls below its threshold or drops more than an agreed margin from the baseline, even if the overall number went up. Flipped cases are looked at by a person before the gate is overridden; the override is a decision with a name on it, not a button.

Closing the loop from production

The set grows from two sources. Red-team findings become permanent cases the moment they are found. Production failures — the sampled human review in monitoring, the outputs users rejected or heavily edited in the interface — are reviewed weekly, labeled and added. A failure that is not in the set will happen again on the next model upgrade; a failure that is in the set cannot ship silently.

A worked example: ticket triage for an online retailer

An online retailer handles about 1,800 support tickets a week. A classifier assigns one of seven intents and one of three urgency levels. In Hilluter's build the harness was set up before the prompt was finished: 400 tickets from the previous quarter, labeled by two support agents, 6% disagreement resolved by a team lead. Gates: urgent recall at or above 95%, no intent class below 85% precision, macro-F1 not to drop more than two points from baseline.

Two months in, a newer model version became available. The run showed overall accuracy up 1.5 points and urgent recall down from 96% to 91%. The flipped cases were all the same shape: the new model read "whenever you get a chance, but my event is tomorrow" as calm. The gate blocked the upgrade. Two examples were added to the prompt, one re-run put urgent recall at 97%, and the upgrade merged the same afternoon. Without the harness it would have looked like an improvement, and about 90 urgent tickets a week would have waited in the normal queue.

Steps to build your first harness

1. Pick the task and the fields that carry consequences. Not everything needs a gate; the bank account number does.
2. Collect 50–100 real cases. From production or the archive, anonymized, ugly ones included.
3. Label them with two people. Record the disagreement; it sets the ceiling.
4. Write the runner. Input in, the real feature in the middle, structured output out. No shortcuts around the actual code path.
5. Score per field or class and set thresholds. Thresholds come from business consequence, not from what the model currently scores.
6. Wire it into CI with a gate. Block on threshold and on drop from baseline.
7. Add every finding. Red-team results and production failures go in the week they are found.

This is the second step in every AI integration project we run, before the prompt is tuned, because tuning without a score is guessing with extra steps. If you have an LLM feature in production without one, describe the task and we will suggest the first 50 cases and the metrics to score them with.