A model has been degrading for months. How do you decide it should be retired rather than retrained again?

Retraining is a bet that the premise still holds

Every retrain rests on an assumption that is rarely stated: that the relationship the model learned is still the right shape, and only its parameters have gone stale. When that is true, fresh data restores performance and the loop works. The reason a model can be retrained repeatedly and keep degrading is that the assumption has quietly stopped being true, and no amount of newer data fixes an invalidated premise. Distinguishing those two situations is the whole question, and it is answered by looking at the target rather than at the metrics.

So the first thing to establish is not how far performance has fallen but whether the last few retrains bought anything. A model whose metric recovers after each retrain and decays again over a predictable interval has a healthy relationship and a retraining cadence that is too slow. A model whose metric no longer recovers to its previous level after a retrain is telling you that the thing it is fitting has changed, and that is a retirement conversation rather than a scheduling one.

Three signals that the premise has gone, not the parameters

The clearest is that the target has been redefined underneath you. The label the model was trained on is produced by some process — a manual review queue, a chargeback window, a support ticket taxonomy — and when that process changes, historical labels and current labels stop meaning the same thing. Training on a mixture of the two produces a model that fits neither, and the fix is a new problem definition, not a new run.

The second is that a feature the model leaned on has stopped carrying information. Not drifted, stopped: the upstream product removed the interaction that generated it, or the field is now populated by a default, or the behaviour it measured was an artefact of a user interface that has since been redesigned. If the model's importance was concentrated in that feature, what remains is a different and much weaker model, and it is worth asking whether the weaker version clears the bar the original cleared.

The third and hardest is that the proxy has stopped standing for the outcome. Most production models predict something convenient rather than something you care about, because the thing you care about is unobservable or arrives too late. Clicks stand for interest, review flags stand for fraud, search volume stands for demand. Those substitutions are all reasonable at the moment they are made and all of them can decay independently of the model, because the world that made the proxy a good stand-in is not obliged to stay still.

Google Flu Trends and the proxy that stopped being one

Google Flu Trends, launched in 2008, estimated influenza activity from the volume of related search queries. It tracked reported case data well at first and degraded over the following years as search behaviour, and the search product itself, changed around the model; Google eventually stopped publishing the estimates.

What makes it the right example here is that the failure was not in the modelling and would not have been fixed by retraining on newer queries. The premise was that search volume for a set of terms was a stable proxy for people falling ill. Media coverage that drives searches by well people, and product changes such as query suggestions that alter which terms people type, both break that proxy without changing anything about the illness. Once the input is partly a function of the platform's own behaviour and of news cycles, the mapping from queries to cases is no longer a fixed relationship to be estimated, and a model of it has no premise left to stand on.

The generalisable point an interviewer wants is that a deployed model's accuracy is a property of the world it runs in rather than of the artefact you shipped. That cuts both ways: it is why monitoring is mandatory, and it is why some degradations are terminal.

You cannot switch it off until you know what happens instead

Retirement is a design task, not a deletion. Something consumes the model's output, and that consumer has an implicit contract that must still be satisfied when the predictions stop.

Decide what the fallback is and make it explicit: a rules engine, a population-level default, the previous behaviour of the product, or a path that routes the case to a human queue. Then check that the fallback can carry the volume, because a manual review process sized for the cases the model was unsure about will not absorb every case. Then measure the difference the removal makes to the business metric rather than to the model metric, because the honest comparison at retirement is model versus fallback, not model versus its own past.

Ramp it down rather than cutting it, for the same reasons you would ramp a launch: send a fraction of traffic to the fallback, watch the guardrails, and increase it. And keep the artefact, the training data reference and the prediction logs for as long as the decisions it made can be questioned, because retiring a model does not retire your obligation to explain what it did while it was live.

Why marginal models are never retired, and what to do about it

The reason this question is asked in senior interviews is that the organisational default is to keep everything. Nobody is promoted for switching a model off, the cost of keeping it alive is spread across a serving bill, a nightly pipeline and a rota nobody itemises, and the person who understood it left two years ago. So models accumulate, and the accumulation is paid for in on-call load and in the attention of a team that has fewer people than models.

The counter is to make the cost visible and the decision scheduled. Give every production model a named owner and a periodic review with a stated bar: what it is worth against its fallback, what it costs to run and to maintain, and whether anyone would build it today. A model that cannot clear that bar should be retired even if it is not obviously broken, and being willing to say so, in an interview and in the review, is the signal that separates an engineer who operates models from one who only ships them.

Retraining fixes stale parameters. It cannot fix a target that has been redefined, a feature that has stopped existing, or a proxy that no longer stands for the outcome — and in those cases the correct engineering answer is a designed retirement with an explicit fallback.