You are shipping an LLM feature and there is no evaluation set. How do you build one?

Sit down to write cases and you will write the ones you already handle

The default approach is to open a spreadsheet and think of test inputs. It produces a set with a characteristic shape: well-formed questions, in the author's own language, on the main path, phrased the way somebody who understands the system phrases things. Every one of them passes. The set then reports 97 per cent and provides no information, because it is a sample from the author's mental model rather than from the world, and the failures live outside that model — the two-word query, the pasted email thread with a signature block, the question in Hindi, the request that is out of scope, the ambiguous pronoun, the document that is a scanned photograph.

So the discipline is to harvest rather than invent. Almost every feature has a source of real inputs before it has any of its own.

If the feature automates work people do today, that work has a log: the support queue, the tickets, the search box, the email inbox, the spreadsheet an analyst maintains. Sample from it, stratified rather than uniformly, so that rare-but-important categories are present in numbers you can actually measure. If the feature is genuinely new, shadow it: run it over live inputs without showing the output to anyone, and label what comes back. A week of shadow traffic yields a better set than a month of composition, and it is the single most valuable thing to do before launch.

Then mine what has already gone wrong. Every prototype failure, every embarrassing demo, every complaint from the pilot users becomes a permanent case with the ticket number attached. This is the mechanism that makes the set improve rather than merely grow, and it is the same reflex as writing a failing test for a bug before fixing it.

Expert-authored cases still have a place, but as targeted coverage rather than as the bulk: the regulated scenario that must be refused, the safety case, the adversarial injection attempt, the input class you know exists and have no traffic for yet. Write those deliberately and label them as such.

Every case records why it exists

A case is not an input. It is an input, an expected behaviour, and the reason it is in the set. The third field is the one people skip and the one that keeps the set alive.

Six months on, someone will find a case that fails and have to decide whether the expectation is still right. Without provenance the choice is unresolvable, and the path of least resistance is to relax the expectation until it passes — which is how a suite quietly stops guarding anything. With added after incident 2419, model cited the superseded refund policy written next to it, the answer is obvious and the case survives.

Record the category too, so you can report by segment rather than in aggregate. Aggregate scores hide exactly the shifts you need to see; a suite that reports 91 per cent overall while the non-English segment has dropped from 88 to 61 has told you nothing useful.

Two kinds of check, kept apart

Some expectations are decidable by code. Does the output parse against the schema, is every required field present, does every factual sentence carry a citation, is the citation to a document that was actually retrieved, is there no email address in a field that must not contain one, is the reply under the length cap. These are assertions. They are cheap, deterministic, run on every change, and belong in continuous integration alongside your unit tests.

Other expectations require judgement: is this answer correct, is it complete, is the tone right, did it decline appropriately. These need a written rubric with concrete criteria and worked examples of a pass and a fail, and a grader — human, or a judge model you have first validated against human labels on a sample. Keep them in a separate suite because they cost money and time, and because a mixed suite gets run at the frequency of its slowest half.

Do not let the graded half swallow cases that could have been assertions. Every check you can make deterministic is one you can run a hundred times a day for nothing.

A frozen slice and a working slice

Split the set from the beginning. The working slice is what you iterate against while tuning a prompt; you will look at its failures in detail and change things until they pass, and that is fine. The frozen regression slice is what you run before shipping, and you do not look at its individual failures while designing — you look at the number.

The reason is the same as for a test set in any other machine-learning work. A set you tune against stops measuring the feature and starts measuring your fit to that set, and because prompt iteration is fast and unstructured, the overfitting happens within a single afternoon. Keeping a slice you never optimise against is what preserves your ability to say the last change was an improvement.

Start smaller than feels rigorous. Thirty to fifty well-chosen, correctly labelled cases with real provenance beat five hundred generated ones, because the labelling effort is the binding constraint and a wrongly labelled case is worse than a missing one — it trains the team to distrust the suite. Grow it from incidents, keep the stratification honest, and re-review the labels when the product's expected behaviour changes rather than when the suite goes red.

On synthetic cases

Generating inputs with a model is legitimate for one purpose: producing volume and variety in the input distribution, particularly paraphrases and edge shapes a human would not think to type. It is not legitimate for producing the labels, because then your evaluation measures agreement between two models rather than correctness, and the blind spots are correlated by construction. Have a person select, correct and label anything a model produced, and mark those cases as synthetic so their contribution to any score is visible rather than assumed.