Your model scored 0.91 AUC in cross-validation and is barely beating the old rules engine in production. How do you work out why?

First establish that the two numbers are comparable

Before diagnosing a gap, confirm there is one. Offline AUC was computed on a curated sample with complete labels; the production figure is often computed on a filtered population, over a shorter window, with labels that are incomplete because they have not matured yet. Each of those differences can manufacture a gap on its own.

Three questions settle it. Which rows are in the production denominator, and does that population match the offline sample, or has an upstream rule already removed the easy negatives? How are production labels obtained, and what is the delay, because a chargeback or a churn event observed at 20 days is a censored version of the 90-day label you trained on. And is the production metric the same metric, since a threshold-dependent number like precision cannot be compared to a threshold-free number like AUC at all. Roughly a third of these investigations end here, with an apparent regression that was a measurement artefact.

One experiment that splits the space

Assuming the gap is real, run the cheapest discriminating test available. Take the rows production actually scored, replay them through your offline pipeline using the same model artefact, and compare the two scores row by row.

# The logged feature vector the server used, versus the vector the
# training pipeline computes for that same entity and timestamp.
replay = offline_pipeline.transform(entities, as_of=served_at)
served = logged_feature_vectors            # captured at request time

drift = (replay - served).abs().mean()      # per-feature disagreement
score_gap = (model.predict_proba(replay)[:, 1] - logged_scores).abs()

If the offline-computed scores match the logged scores closely but the metric is still poor, the serving path is faithful and the problem lies in the data or the labels: leakage, split structure or shift. If they differ materially, you have found train-serve skew and the per-feature disagreement tells you which column. Either branch eliminates half the hypotheses for the cost of one batch job.

flowchart TD
    A[Offline 0.91<br/>Production poor] --> B{Same population<br/>and matured labels}
    B -- no --> C[Measurement artefact<br/>fix the comparison]
    B -- yes --> D{Replay scores match<br/>logged scores}
    D -- no --> E[Train-serve skew<br/>diff the feature vectors]
    D -- yes --> F{Score holds on a<br/>later untouched period}
    F -- no --> G[Leakage or dishonest split]
    F -- yes --> H[Distribution shift]

Note the asymmetry in that tree: the skew branch identifies a specific column and is therefore quick to close, while the right-hand branch only narrows the cause and still needs the ordered investigation below.

Leakage, because it is the most common and the most flattering

A 0.91 on a problem the business has never solved well is itself evidence. Leakage means a feature carries information that would not have existed at prediction time, and it produces exactly this signature: superb offline, worthless live.

Screen for it mechanically. Score every feature on its own and look for any single column that reaches near the full model's performance. Inspect permutation importance and be suspicious when one feature holds most of the mass. Then take the top handful and answer one question for each: at what moment is this column written, and by which process. A field populated by the case-management system that reviews the very event you are predicting is a label in disguise. So is an aggregate whose window includes the event, such as an account's average balance computed over all rows rather than as of the decision time.

The split that was never honest

The second cause is a validation scheme that broke the independence it assumed, and it deserves to be ranked this high because it is silent. Nothing looks wrong, the gap between training and validation folds is small, and the number is still fiction.

The arithmetic is stark for grouped data. Suppose 500 customers contributing roughly 40 rows each, 20,000 rows in total, split at random 80/20. The chance that a given customer's rows land entirely in training is 0.8 to the power of 40, which is about 0.00013, and entirely in validation is negligible. So essentially every customer appears on both sides, and the model can score well by recognising customers rather than learning behaviour. Production shows it new customers and the performance evaporates. The fix is a group-aware split on the customer key, and for anything time-ordered a forward-chaining split with a gap wide enough to cover the label window.

Skew and shift, and why they are last

Train-serve skew is a defect rather than a phenomenon: two implementations of one transformation, differing default handling of nulls, an aggregation window measured in calendar days offline and rolling hours online, or a feature that is available in the warehouse and simply not available in the request. It is ranked below leakage only because the replay experiment finds it immediately, not because it is rare.

Genuine distribution shift is real and is where most candidates start, which is why starting there is a weak signal. It also usually degrades gradually, so a model that was never good is more likely to have been broken at build time than shifted at run time. Distinguish the two kinds when you do get here. Covariate shift means the inputs moved while the relationship held, and often just needs retraining on recent data. Concept shift means the relationship itself changed, so retraining helps only until it changes again, and the response is architectural rather than a refit.

Ordering the hypotheses is the skill being examined

The weak answer lists every possible cause. It is complete, it is correct, and it demonstrates nothing, because an interviewer wants to know what you would do on Monday morning. The strong answer commits to an order, justifies the order by base rates and by cost of testing, and names one experiment that eliminates several branches at once. Rank by likelihood multiplied by how cheap the test is, say so out loud, and then be willing to revise the order when the first result comes back.

A model that was excellent offline and useless live was usually broken before it shipped. Investigate build-time causes first, and earn the right to blame the world only after replaying production rows through your own pipeline.