You have fine-tuned a model for a task. How do you prove it helped?

Loss went down, which tells you almost nothing

The first thing to refuse is the training curve as evidence. Loss measures how well the model predicts the next token in your training distribution. It falls because that is what the optimiser is for, and it can fall while the outputs get worse for users — by memorising the idiosyncrasies of whoever wrote your labels, by collapsing onto a single safe phrasing that scores well on likelihood and reads as evasive, or by learning artefacts of your formatting rather than the task. Validation loss on a held-out split of the same export is better and still not the thing you care about, because it measures agreement with your labels and not correctness in the product.

What counts as proof is a comparison, on a set the model has never seen, using a metric that corresponds to what the feature is for. Everything else in this answer is about making that comparison honest.

The baseline has to be a real attempt

The most common dishonesty in fine-tuning results is not fabricated numbers, it is a weak baseline. A prompt written in ten minutes, no few-shot examples, no schema, on a smaller model than the fine-tuned one, judged on a metric chosen after seeing the results — against that, almost any fine-tune wins, and the result means nothing because the alternative you rejected was never on the table.

A defensible baseline is the best prompted configuration you can build with comparable effort to the fine-tune, on a model you would actually be willing to deploy, with the same retrieval context and the same output schema. If your fine-tune took two weeks, the baseline deserves more than an afternoon. Two additional comparisons are worth including when they apply: the base model of your fine-tune with the same prompt, which isolates what the training contributed from what the model already did; and a small prompted model, which is often the real competitor on cost.

Fix the metric and the test set before you run anything. Choosing either afterwards converts the exercise into a search for a framing in which your work looks good, and everyone can tell.

Where the test set comes from decides what the number means

Draw the evaluation set from production traffic, not from the same curation pass that produced the training data. Those two sets share the author's blind spots, so an evaluation built from them measures performance on the cases you already thought of. Real inputs bring the malformed, the ambiguous, the multilingual and the out-of-scope, and those are where a fine-tune most often disappoints.

Then check for leakage properly, because a random split will not save you. Exact duplicates are easy; near-duplicates are the problem — the same ticket resubmitted, boilerplate paraphrased, one document appearing in twelve chunk variants. Deduplicate by similarity across the split boundary and drop matches from the test side. If the underlying data is grouped, split by group: by customer, by document, by session. Splitting individual examples from one customer's corpus across train and test lets the model score well on a document it has effectively already read. And if the feature will run on tomorrow's inputs, prefer a time-based split with the test set drawn from later than the training window, because that is the only split shape that reflects deployment.

Measure the target task and the collateral damage

Two suites, run together, every time.

The task suite measures the thing you trained for, with a metric appropriate to it: exact match or field-level accuracy for extraction, a confusion matrix rather than bare accuracy for classification, schema-validity and required-field rates for structured output. For open-ended generation where no reference answer exists, use blind pairwise comparison — human or a judge you have validated against human agreement — with the two candidates presented in randomised order, since judges favour whichever came first.

The regression suite measures everything you did not train for and still depend on. Fine-tuning on a narrow distribution reliably erodes behaviour outside it, and the erosion is invisible if you only look at the target metric. Include general instruction following, refusals and safety behaviour, inputs in the other languages you serve, adjacent tasks the same endpoint handles, tone, and adherence to output format on inputs unlike the training examples. A fine-tune that gains six points on extraction and quietly stops declining out-of-scope requests is a net loss, and the ticket for it will arrive as a customer complaint rather than as a metric.

Report cost and latency in the same table. Half the reason to fine-tune a smaller model is that it is cheaper and faster with a shorter prompt, and that gain is part of "helped" — as is the reverse, if your fine-tune needs a larger base than the prompted baseline did.

Say how sure you are

Two accuracy figures with no uncertainty attached are the other place these reports mislead. On 200 examples, a two-point difference is well inside the noise, and a reviewer who knows that will discount the whole document. Give the interval, and prefer a paired analysis: run both systems on the identical examples and count where they disagree, which removes the variance from item difficulty and needs far fewer cases to detect a real difference. Also report the disagreements qualitatively — the cases the fine-tune fixed and the cases it broke — because a wash on the aggregate can hide a trade you would not accept, such as fixing sixty easy cases and breaking twenty important ones.

Then close the loop online. Offline evaluation establishes that shipping is defensible; it does not establish improvement in the product. Roll out behind a flag against the metric the feature exists for — resolution rate, edit distance between the model's draft and what the user sent, escalation rate, abandonment — and keep the prompted baseline reachable by configuration so that reverting is a switch rather than a project. A fine-tune that cannot be turned off has not been proven, it has been committed to.