What is target leakage, and how would you catch it before the model ships?

Defined by timing, not by correlation

Target leakage is the presence of information in the training features that would not be available at the moment the prediction has to be made. The definition is about availability at a point in time, and it matters that you phrase it that way, because the more common phrasing, "a feature too closely related to the target", would also condemn every genuinely predictive feature you have.

The consequence is that leakage cannot be detected by looking at correlation alone. It is detected by asking, for each column, when the value was written and by which process. A column that is populated by the same workflow that determines the outcome is a label wearing a different name, however innocent its title.

Form one, the column written after the event

The first family is a field that only acquires its value once the outcome has happened. A case_closed_reason on a fraud investigation, a discharge_code on a hospital admission, a cancellation_channel on a subscription, a collections_stage on a loan. Every one of these is present in the warehouse row and absent at decision time, because at decision time the event has not occurred.

These are easy to find once you look and almost impossible to find if you do not, because the column names sound like features and the pipeline that assembled the table has flattened away the fact that the fields were written weeks apart.

Form two, the aggregate whose window swallows the label

The second family is subtler and survives review far more often, because the feature is legitimate in principle and only the computation is wrong. You want "average transaction amount over the trailing 30 days", which is a perfectly reasonable feature. Someone computes it as a group-by over the whole table rather than as of each row's own timestamp.

Consider a customer with 40 ordinary transactions averaging £35, and then one fraudulent transaction of £4,000 which is the row you are trying to classify. The point-in-time correct feature for that row is the mean of the 40 prior transactions, £35. The naive group-by over all 41 rows gives:

(40 x 35 + 4000) / 41  =  (1400 + 4000) / 41  =  5400 / 41  =  £131.71

So for every fraudulent row the feature reads roughly £132 and for ordinary rows it reads close to £35. The model has been handed a fraud flag scaled by amount. Offline it will look superb. Live, the aggregate is computed only over genuinely prior transactions, the feature reads £35 for everything, and the model has nothing.

The same defect appears in any target-encoded categorical fitted on the full dataset, in any "number of claims per policy" computed across the whole history, and in any standardiser fitted before the split, which leaks the validation set's mean and variance rather than the label but is the same class of mistake.

A screen you can run every time

Make the check mechanical rather than a resolution to be careful.

Start with suspicion of the score itself. If the model reaches a level nobody in the business has ever reached on this problem, treat that as a defect report. This inverts the natural reaction and is most of the discipline.

Then fit each candidate feature alone against the label and rank by single-feature performance. Any column that on its own approaches the full model is a suspect. Follow it with permutation importance on the fitted model and look for mass concentrated in one or two columns rather than spread across many, which is the usual leakage signature.

For each suspect, answer the timing question in writing: which system writes this column, at what point in the entity's lifecycle, and is that point before or after the moment of prediction. This is a conversation with whoever owns the source table and it cannot be shortcut by reading the data dictionary, which describes the field and not the write time.

Finally, validate on a period the pipeline has never touched. Train on everything up to a cut-off and score the following weeks with the model receiving only what a live caller could have supplied. A leaked feature that survives the earlier screens usually fails here, because the leakage is a property of how the historical table was assembled and the forward period was assembled the same way only if you have made the mistake consistently.

Why the validation gap will not save you

The standard defence against overfitting is the gap between training and validation scores, and it is exactly the wrong instrument here. A leaked feature generalises perfectly across folds, because the leak is present in every row of the dataset, both sides of every split. Training 0.94, validation 0.93, a small honest-looking gap, and the whole thing worthless. That is what makes leakage the most expensive defect in applied machine learning: it defeats the one check most teams rely on, and it is usually discovered by a stakeholder rather than by an engineer.

Cross-validation answers the question "does this pattern hold on rows I did not fit". Leakage detection answers a different question, "did this pattern exist yet", and no split can answer it because the answer is not in the data.

Building the answer into the pipeline

Prevention beats screening. Attach an event timestamp to every feature and enforce that every join is as-of that timestamp rather than a plain key join, which is the discipline a feature store's point-in-time correctness gives you. Require a declared availability lag on each feature, and refuse to include any feature whose lag exceeds the decision horizon, which turns leakage from a judgement call into a schema constraint a test can enforce. And keep an allow-list of source tables rather than selecting everything the warehouse offers, since most leakage arrives as a column nobody consciously chose.

Leakage is a question about clocks, not about correlations. For every feature, know which system writes it and when, and treat a metric that is too good as a bug report rather than a result.