The Evaluation Set Is the Most Valuable Asset You Are Not Building

There is a strange asymmetry in how teams invest in AI features. Weeks go into prompt iteration, days into picking a provider, and roughly one afternoon into deciding whether any of it is working. Then a model version changes, quality shifts in some direction nobody can characterise, and the team argues from anecdotes for a fortnight.

Why anecdotes lose

Human memory of model output is unreliable and heavily biased toward the last bad case. Without a fixed set of examples and expected outcomes, every change becomes a matter of opinion, and the loudest opinion wins. Worse, you cannot answer the only question that matters during an upgrade: did this get better or just different?

What a useful set looks like

It does not need to be large. Between fifty and two hundred cases covers most product surfaces if they are chosen well. What matters is composition:

• Golden path cases that must never regress, drawn from your most common real requests.
• Known failures, every one of them added the day someone reported it. This is how a bug report becomes a permanent guard rail.
• Adversarial cases: prompt injection attempts, contradictory instructions, requests for things the system must refuse.
• Abstention cases where the correct answer is that the system does not know. If you never test this, your product will never learn to say it.

Grade what you can grade mechanically

Where the task has a checkable output, assert on it: JSON that must validate, a field that must match, a citation that must resolve to a real document. Mechanical checks are fast, free, and never drift. Reserve model-graded evaluation for genuinely subjective qualities like tone, and when you use it, pin the grader version, because a moving grader makes your history meaningless.

Run it where it will actually be run

An evaluation suite that takes forty minutes and lives in someone's notebook will be run twice. The set belongs in CI, triggered on any change to a prompt, a tool definition, a retrieval parameter, or a model identifier, with results posted where the team already looks. Slow suites get sampled per commit and run fully nightly.

Every incident should end with a new row in the evaluation set. That is the difference between a team that learns and a team that keeps rediscovering the same failure.

The compounding effect

Six months in, this set is worth more than any individual prompt you have written. It lets you evaluate a new model in an afternoon instead of a sprint, lets a new engineer change the system without fear, and turns provider competition into something you can exploit rather than something that threatens you. Prompts are disposable. Cases are not.