hardScenarioDesignCase StudyMidSeniorStaffLead
How do you roll back a model in production?
6 min readUpdated 2026-08-15
Answer
Short answer
Re-pointing traffic at the previous version is only safe if nothing else moved with it.The feature transformations, the input schema and any consumer calibrated to the current score distribution may have changed, so the unit of rollback is the model plus its features and its contract.
Mention model cicd where it changes the risk, the owner, or the next check. A useful model cicd point should make the answer more testable, not merely longer. Tie model cicd back to the scenario so the interviewer can see why it matters here.
Was this answer helpful?
Likely follow-ups
- The previous model needs a feature that was deleted in the meantime. What are your options?
- How would you make rollback boring enough to do at 03:00 without a design discussion?
- A downstream team hard-coded a threshold against the current model's scores. How does that change your plan?
- When is shifting traffic to a rules-based fallback better than rolling back to the previous model?
Related questions
- What has to be in place before you would let a model serve live traffic?hardAlso on training-serving-skew and model-registry6 min
- Design the gate that decides whether a newly trained model gets promoted.hardAlso on model-cicd6 min
- Is a feature store worth its operational cost, or can you argue against one?hardAlso on training-serving-skew5 min
- An upstream team renames a field in the events you consume and nobody tells you. What should your pipeline have done?hardAlso on schema-evolution5 min