hardDesignScenarioMidSeniorStaffLead
What has to be in place before you would let a model serve live traffic?
6 min readUpdated 2026-08-15
Answer
Short answer
Features must be computable at decision time, training and serving must share one transformation implementation, the model and dataset version must be pinned together, drift and realised performance monitored separately because labels arrive late, and rollback must move model and features as one unit.
Was this answer helpful?
Likely follow-ups
- Your labels arrive 60 days after the prediction. What do you monitor in the meantime, and what can you conclude from it?
- A feature is drifting but offline metrics are unchanged. Do you retrain, and what would change your mind?
- How would you detect that a model has been receiving a default value in place of a real feature for a week?
- Where does a champion-challenger setup sit relative to a canary deployment, and can you run both?
Related questions
- A model scored well in validation and performs badly in production, with no errors anywhere. What do you check first?hardAlso on mlops and training-serving-skew5 min
- How do you roll back a model in production?hardAlso on training-serving-skew and model-registry6 min
- Where does train-serve skew come from, and how do you stop it?hardAlso on feature-store and mlops6 min
- Is a feature store worth its operational cost, or can you argue against one?hardAlso on training-serving-skew and mlops5 min