Why the offline number was never real
Training data is usually assembled by joining tables on a customer key. That join has no concept of time unless you give it one. A cancellation_reason_code, a final_invoice_flag, a retention_call_outcome — each is written by a process that only runs once the outcome is known, and each lands in the same row as the label with nothing to mark it as arriving later.
A random train/test split cannot catch this. Both halves contain rows built the same way, so the leaked feature is equally available in both, and the held-out score is just as inflated as the training score. The validation you trusted was structurally incapable of detecting the problem, which is why the failure only appeared in production. That is the sentence worth saying in an interview: the split did not fail to catch it by bad luck, it was the wrong instrument.
Finding it quickly
Two checks locate almost every instance of this.
Rank features by importance and read the top of the list. Leakage is rarely subtle. A feature carrying implausible predictive power — one that alone gets you most of the way to the reported AUC — is a leak until proven otherwise. If a single feature explains churn better than everything else combined, the honest first hypothesis is that it is churn.
Ask, for each top feature, when its value is physically written. Not when it is queryable in the warehouse, but when the upstream process populates it relative to the event being predicted. This question is uncomfortable because it usually has to be asked of another team, and it is the one that settles the matter.
A quick confirmation: score the model using a snapshot of the feature store as it stood at prediction time rather than as it stands now. If performance collapses, you have reproduced production offline and you are done diagnosing.
The fix: define features by availability, not by existence
Rebuild the training set so that every row contains only what was knowable at its prediction timestamp. This is a point-in-time or as-of join: for each labelled example, take each feature's value as of the cutoff, not its current value.
SELECT s.customer_id,
s.score_date,
s.churned_within_30d AS label,
f.value AS support_tickets_90d
FROM scoring_events s
LEFT JOIN LATERAL (
SELECT value
FROM customer_features f
WHERE f.customer_id = s.customer_id
AND f.valid_from <= s.score_date
ORDER BY f.valid_from DESC
LIMIT 1
) f ON true;
The requirement this imposes on your data platform is that features carry a valid_from — the time the value became known — separate from any business timestamp. Feature stores exist largely to provide this guarantee; without it, point-in-time correctness has to be hand-rolled per pipeline and quietly breaks.
Then validate on a forward temporal split: train on everything up to a date, test on what came after. This mirrors how the model is actually used and will surface any remaining leak as a gap between random-split and time-split scores.
Features that arrive late but not too late
The follow-up worth anticipating is the feature that is legitimate but delayed. If support_tickets_last_7d is genuinely available six hours after the fact, it is not leakage, but it is only usable if your scoring job also runs at least six hours behind — otherwise training sees it and serving does not. The rule is to align the training cutoff with the worst-case serving freshness, not the average, and to treat a feature that is sometimes late as sometimes missing, training the model with that missingness present so it learns to cope.
Leakage that has nothing to do with timestamps
Three others come up often enough to name, because an interviewer will usually ask for one:
Fitting a transform on the full dataset. Computing a scaler's mean and variance, or a target encoding, before splitting means test-set information has flowed into the training features. Every transform must be fitted on the training fold alone and applied to the others.
Duplicate or near-duplicate rows spanning the split. The same customer appearing in both train and test, or multiple rows per entity, lets the model memorise rather than generalise. Grouped splitting by entity id is the fix.
A proxy for the label. No timestamp violation, just a feature that is downstream of the outcome by construction — account_status = 'closed' predicting churn, or a discount code only issued to customers already flagged for retention.
Setting expectations after the fix
Be direct that the honest model will score far worse, because the person who saw 0.94 will remember it. The framing that works is that 0.94 was never a measurement of anything — it described a model with access to the answer, and it predicted nothing about production, which is exactly what production then demonstrated. A 0.71 that holds up on a forward time split is a real number you can make decisions with, and it is the first one you have had.
The durable prevention is a training-serving skew check: for a sample of live predictions, compare the feature vector the model actually received against the one the training pipeline would have produced for the same entity at the same moment. Systematic differences — nulls in production that were populated in training — fail the check. That comparison is what catches the next instance before it reaches a stakeholder.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).