A model scored well in validation and performs badly in production, with no errors anywhere. What do you check first?

Suspect the plumbing, not the model

The shape of the report is the diagnosis. Validation good, production bad, no errors, no exceptions, nothing in the logs. Models do not silently get worse; inputs do.

The overwhelmingly likely cause is that the features the model receives in production are not the features it was trained on. Not missing, not malformed — computed slightly differently, by different code, so every value is plausible and some are wrong. Nothing validates it because nothing can: a number is a number.

This is training-serving skew, and the reason it dominates is structural. Training reads history in bulk, usually in SQL or a dataframe over a warehouse. Serving computes one row at a time, in application code, from live state. Two implementations, two languages, two authors, one definition — and the definition lives in neither of them.

How the two paths diverge

The specific divergences recur, which makes them worth knowing by name.

Time windows. Training computes "average order value over the last 30 days" by grouping historical rows, and quietly includes the current day because the WHERE clause used <=. Serving excludes today, because today is not finished. The feature is systematically different, and in training it partly encodes the outcome.

Aggregation state. Training sees a customer's complete history. Serving sees whatever the cache had at request time, which for a new customer is nothing and for a busy one may be a few seconds stale.

Missing-value handling. Training fills nulls with the column mean, computed over the training set. Serving fills with zero, because that is what the code that a different team wrote does. Every absent value now sits at a different point in the distribution.

Categorical encoding. A category unseen during training maps to a new index at serving time, or to a different index because the encoder was refitted rather than loaded.

Units and types. Amounts in pence in one path and pounds in the other is the crude version. Subtler is a timestamp that is UTC in the warehouse and local in the service, which shifts every time-derived feature by hours.

-- Training. `<=` includes today's partial orders, which the model then
-- learns from and which serving cannot reproduce.
select avg(amount) from orders
where customer_id = :id and created_at <= :as_of;

-- What serving can actually compute: strictly before the scoring moment.
select avg(amount) from orders
where customer_id = :id and created_at < :as_of;

The fix is one definition, not two careful ones

The instinct is to review both implementations until they agree. That works until someone changes one of them, which is a matter of months.

The durable fix is that the feature is defined once and both paths call it. What that looks like depends on scale.

At the small end, a shared library: one function per feature, imported by the training job and by the service, with tests. This is unglamorous and it removes most of the problem for most teams.

At the larger end, a feature store, whose actual value is precisely this — a single definition, materialised to a batch store for training and an online store for serving, with point-in-time correctness handled for you so that a training row can only see values that existed at that row's timestamp. That last property is the hard part to build yourself and the main reason to buy rather than assemble.

The overkill test is honest: a feature store is infrastructure with an operating cost, and a team with one model and eight features does not need one. A shared module and a test that asserts both paths produce the same value for a fixed input covers the same failure at a fraction of the cost.

Detect it before the business metric moves

Waiting for revenue to fall is a slow detector. Three cheaper ones, in order of how quickly they pay for themselves.

Log the features at serving time, not just the prediction. Then compare their distributions against the training set — mean, standard deviation, null rate, cardinality per feature. A feature whose null rate is 0.1% in training and 40% in production is the whole answer, found in an afternoon.

Score the same rows both ways. Take a sample of production requests, replay them through the training pipeline, and diff the feature vectors. Any non-zero difference is a bug. This is the most direct test and the one teams skip.

Monitor the prediction distribution. Cheaper than monitoring outcomes, which may take weeks to arrive, and it moves when something upstream changes even before you know what.

Skew is not drift

Worth separating, because they present identically and the responses differ.

Skew is a discrepancy between two pipelines. It is a bug, it was there on day one, and retraining does not fix it — you retrain on the training pipeline and serve through the broken one, so the new model is wrong in the same way.

Drift is the world changing: the input distribution moves, or the relationship between inputs and outcome does. Nothing is broken, the model is simply describing a situation that no longer holds, and retraining is exactly the right response.

The distinguishing question is whether performance was ever good in production. Bad from launch is skew. Good and decaying is drift. Getting this the wrong way round leads teams to retrain repeatedly against a pipeline bug, which produces a series of models that each disappoint in the same way.

Validation good, production bad, nothing in the logs. That combination is almost always two pieces of code computing the same feature slightly differently, and no amount of retraining will fix it.