A more complex model beats your simple one by a small margin offline. How do you decide whether it is worth shipping?

A metric delta is not a benefit until it changes a decision

The first move is to refuse the comparison as posed. Two numbers on a validation set are not a business case, because a model's output only matters where it feeds an action, and most actions are taken through a threshold. If credit applications above a score are declined, a model that reorders applicants far from the threshold has improved a ranking metric while changing nothing at all, because every case that was declined is still declined. So the question to answer is not how much the metric moved but how many decisions flipped, in which direction, and what each flip is worth.

That reframing is usually decisive on its own. It turns an argument about model families into a countable comparison: the number of cases that change hands, the value of a correct decision and the cost of each kind of error. Where the two models are compared at their operating points rather than by an aggregate score, a large-looking gain often shrinks to a handful of borderline cases, and occasionally the reverse happens and a modest average gain turns out to sit precisely where the money is.

Is the gain even real?

Before pricing anything, establish that the difference exceeds the variance of your own evaluation. A single split gives you one sample of the metric, and the spread across folds, across random seeds and across time periods is frequently wider than the margin being celebrated. Repeat the comparison with different splits and different seeds, report the spread rather than the point, and check whether the ordering is stable when the evaluation window moves. If the complex model wins on average but loses on two of five folds, you have not measured an improvement, you have measured a coin.

The evaluation also has to be the harder one rather than the convenient one. Compare on the most recent period held out entirely, since a model that wins on a random split and loses on the last month has learned something about the past that is no longer true, and that is the pattern that will greet you in production.

What the complex model commits you to

The cost side is systematically underestimated because most of it is not compute. Assume the model ships and count what the team owns from then on.

Serving cost and latency come first, and latency is often a hard constraint rather than a price. Then retraining: a model with more moving parts has more that can drift, more hyperparameters to re-select, and a longer training run to schedule and pay for. Then dependencies, since extra features mean extra upstream pipelines that can be late, wrong or quietly renamed, and each one is a new way for the model to fail with no error. Then explanation, because when a customer or a regulator asks why a decision was made, the answer has to come out of whatever you deployed. And finally people: an ensemble of several models with a blending layer is understood by the person who built it, and the on-call engineer at three in the morning is somebody else.

Every one of those is a recurring cost paid for as long as the model lives, set against a one-off improvement measured on a fixed dataset. That asymmetry is the reason the honest answer to a small gain is so often no.

The ensemble that won the Netflix Prize and was never deployed

In 2009 the ensemble that won the Netflix Prize was never put into production, because the engineering and maintenance cost of running it outweighed the accuracy it bought over the simpler models Netflix already had. It is the cleanest illustration available of the whole argument: the winning solution was genuinely more accurate on the stated metric, the metric was the one the competition was scored on, and the decision not to ship it was still correct.

Volunteering that distinction unprompted is what an interviewer is listening for. An offline metric improvement is a claim about a leaderboard. The number that decides whether something ships is the gain net of everything the team is committing to maintain indefinitely, and those two quantities can point in opposite directions without either of them being wrong.

Where complexity earns its keep anyway

The symmetric failure is refusing complexity as a matter of taste, so be specific about when the answer flips. A small aggregate gain is worth paying for when it is concentrated where the value is, such as a fraud model that barely moves overall precision but catches a category of high-value loss the simple model misses entirely. It is worth paying for when the simple model's errors are structured rather than random, because a systematic failure on one segment is a fairness and reputation problem that averages conceal. And it is worth paying for when the complex model unlocks a capability rather than a percentage, such as calibrated probabilities that let a downstream system reason about expected value where the incumbent only produced a ranking.

Framing the incumbent as the thing that must be displaced, rather than as a baseline to beat on a table, is what makes this a judgement rather than a preference. The simple model is already deployed, already understood and already trusted, and those are real assets that do not appear in either model's score.

Price the gain in flipped decisions rather than in metric points, count the cost over the model's whole life rather than at training time, and be willing to recommend keeping the simpler model even when the complex one genuinely wins the evaluation.