hardScenarioDesignMidSeniorStaff
A model scored well in validation and performs badly in production, with no errors anywhere. What do you check first?
5 min readUpdated 2026-08-15
Answer
Short answer
Check whether training and serving compute the same features from the same definitions, because the usual cause is two implementations that disagree slightly. Nothing raises an error when they diverge - the model receives plausible numbers that mean something different from the ones it learned on.
Was this answer helpful?
Likely follow-ups
- Training averages a customer's last 30 days. What does serving have to be careful about?
- How would you detect this without waiting for the business metric to fall?
- What is the difference between this and drift, and does the fix differ?
- Where would a feature store help, and where is it overkill?
Related questions
- What has to be in place before you would let a model serve live traffic?hardAlso on mlops and training-serving-skew6 min
- Your labels arrive months after the prediction, or never. What do you monitor?hardAlso on monitoring and drift5 min
- How do you decide when to retrain a model?mediumAlso on drift and monitoring5 min
- Is a feature store worth its operational cost, or can you argue against one?hardAlso on training-serving-skew and mlops5 min