How do you roll back a model in production?

The instinctive answer and what it misses

Asked this question, most candidates describe re-pointing the serving alias at the previous registry version, perhaps with a gradual traffic shift, and stop. That is the mechanism, and the mechanism is the easy part. The question is whether the previous model, once it is receiving live traffic again, is in the same situation it was in when it worked.

Frequently it is not, because a model release is rarely just a model. The release that introduced version 18 probably also changed the code that computes its features, added a column to the request schema, or altered a transformation's default. Reverting the artefact without reverting those leaves version 17 being fed inputs it has never seen — which is training-serving skew, deliberately introduced, at the worst possible moment. The rollback then produces a model that is not obviously broken, is quietly wrong, and is now the thing you are relying on to end the incident.

Three ways the ground moves under a rollback

Feature transformation drift. Version 18 shipped with a change to how a categorical is encoded, or a window narrowed from thirty days to twenty-eight, or a new imputation default for a column that had started arriving null. Version 17 was trained against the old behaviour. Point traffic back at it and every prediction is computed from subtly different numbers. Nothing errors: the encoding produces a valid integer, the window produces a valid average. The model's scores are simply no longer the scores it was validated on.

Schema movement. Suppose version 18 consumed a feature that did not exist when 17 was trained, and the feature pipeline was cleaned up afterwards to remove one that 17 needed and 18 did not. Version 17 now cannot be served at all, or is served with a default in place of a real input, which is worse because it works. The same class of problem appears at the artefact level: 17's preprocessing may depend on a library version that was upgraded in the meantime, and a serialised model that no longer deserialises is a rollback plan that fails at the moment you invoke it.

Downstream calibration. This is the one that most often surprises people, because the coupling is outside your service. Something consumes your score and has been tuned to its distribution. A collections team works a queue of the top two per cent by risk. A pricing rule applies a surcharge above 0.7. An alerting threshold was chosen to yield a manageable number of reviews per day. If version 18 shifted the score distribution — and a retrained model almost always shifts it somewhat, even at identical ranking quality — those consumers were re-tuned, formally or informally, to 18. Roll back to 17 and every one of them is now mis-tuned in the opposite direction. Case volumes double or collapse, and the incident you were resolving is replaced by a different one in another team.

release 18 shipped:   model 18  +  features v9  +  schema v3
                                                     |
downstream re-tuned:  queue cut-off 0.62 ---> 0.71  (informally, by an analyst)

naive rollback:       model 17  +  features v9  +  schema v3
                      ^ trained on features v7          ^ cut-off still 0.71

The point of that sketch is that only one of the four things changed, and the other three are now inconsistent with it.

Make the release unit bigger than the model

The design conclusion is that the deployable unit is a bundle: the model artefact, the exact version of the feature transformation code, the input schema it expects, and the contract it publishes about its output. Roll all of them together or none of them. Concretely, the registry entry for a model version should pin the feature-code version it was trained against and the serving image digest, and the serving path should refuse to load a model whose pinned feature version does not match the one it is running. That refusal is a good failure: an alert at deploy time rather than silent skew at request time.

For the downstream coupling, the fix is contractual rather than technical. Publish the score distribution as part of the release — its quantiles on the evaluation set — so consumers can see when it moves. Better still, where the consumer's real need is a rank or a rate rather than an absolute number, give them a percentile or a decile rather than a raw probability, so a distribution shift does not change what the top two per cent means. Teams that do this find that a whole category of cross-team incident disappears, because the interface stopped exposing a number that was never stable.

Keep the old version genuinely runnable

A rollback plan that has never been executed is a hypothesis. Two things make it real. First, retention: the previous version's artefact, its feature code, its container image and its schema must all still exist, and a retention policy that prunes images or artefacts on age will eventually delete your rollback target. Second, exercise it. Roll back deliberately in a lower environment on a schedule, or better, occasionally roll production back and forward during quiet hours. What you are testing is not the traffic switch, which is trivially reliable, but whether the old bundle still assembles.

Keeping a shadow or standby deployment of the previous version receiving mirrored traffic is the strongest form of this. It costs compute, and in return the rollback target is a running service producing scores you can compare against the live one, rather than an artefact you hope will load.

Sometimes rollback is not the right move

Worth saying unprompted, because it shows you are reasoning about the incident rather than reciting a procedure. If the problem is bad input data rather than a bad model, rolling back the model changes nothing and costs you time. If the problem is that the new model is unsafe in a way the old one also was, rollback restores a different bug. If the previous version cannot be safely fed by the current feature pipeline, the faster and more honest mitigation may be to fall back to a deterministic rules path or a conservative default decision, accept the loss of model quality for an hour, and fix forward. And if the release included a schema migration downstream systems have already adopted, rolling the model back without rolling that back may be impossible in the time available.

The judgement is about which state you can reach quickly and describe confidently, not about which state is nominally the previous one. A rollback that leaves the system in a configuration nobody has ever tested is not a return to safety, it is a new deployment made under time pressure with no evaluation.

The unit of rollback is the bundle: model, feature code, schema and published output contract. Reverting the artefact alone puts an old model behind new inputs and a stale threshold, and produces a failure quieter than the one you were fixing.