Top 11 Data leakage Interview Questions
Browse 11 real-world technical and behavioral interview questions about Data leakage. Review scenarios, edge cases, and architectural best practices.
A churn model scores 0.94 AUC offline and barely beats the baseline in production. One of its strongest features is populated by a process that runs after the customer has already churned. Explain what happened and how you would fix it.
The feature encodes the outcome, so the model reads the answer rather than predicting it. Offline the label and the feature come from the same snapshot, which is why validation looks excellent; in production the feature is empty at scoring time and the signal vanishes. Fix by defining features by when they were available and validating on a forward time split.
How would you design a validation strategy that respects the structure of the data you have?
Build the split to imitate the gap between training and deployment. Group all of an entity's rows onto one side when rows repeat per entity, split forward in time for anything temporal with an embargo covering the label window, and treat a plain random shuffle as the default that silently inflates almost every score.
How do you tell that a model is overfitting, and what do you do about it?
Detect overfitting by comparing training and validation performance on an honest split. Fix it with more or better data, simpler models, regularisation, early stopping and a final test set that is read only once. It also connects model validation to the point an interviewer is testing.
One of your features is a postcode with about 40,000 distinct values. How do you encode it?
One-hot is unusable at that width, so the options are frequency encoding, smoothed target encoding, hashing, or a library's native categorical handling. Target encoding is the strongest and the most dangerous, because it must be fitted inside each fold or it leaks the label. Use this categorical encoding answer to show the decision, trade-off, and evidence rather than a memorised definition.
You have fine-tuned a model for a task. How do you prove it helped?
Compare it against a genuinely strong prompted baseline on a held-out set drawn from real traffic, using the task metric your product cares about, and run a broad regression set alongside to catch capability lost elsewhere. Falling training loss is evidence that training worked, not that the product improved.
Your model scored 0.91 AUC in cross-validation and is barely beating the old rules engine in production. How do you work out why?
Rank the causes by how often they are to blame and test them in that order: target leakage, a validation split that ignored group or time structure, train-serve skew, then genuine distribution shift. One cheap experiment separates most of them, which is rescoring logged production rows offline with the same artefact.
What does point-in-time correctness mean, and how does a naive join break it?
A training row must contain only feature values that were knowable at its own event timestamp. A naive equality join on entity id attaches whatever the feature table holds now, including values derived from after the event, which leaks the future and produces offline metrics that collapse in production.
Where does train-serve skew come from, and how do you stop it?
Train-serve skew is a mismatch between features used in training and features produced at serving time. Prevent it with shared transformations, point-in-time joins, feature logging, schema checks and monitoring of served feature distributions. It also connects feature store to the point an interviewer is testing.
What is target leakage, and how would you catch it before the model ships?
Target leakage happens when a feature contains information that would not be available at prediction time. Catch it by auditing feature creation times, window boundaries, post-label columns and suspiciously strong validation performance. Use this data leakage answer to show the decision, trade-off, and evidence rather than a memorised definition.
Why does a random train-test split ruin a forecasting model?
A random train-test split ruins a forecasting model by leaking future observations into training. Use time-based holdouts or rolling-origin backtesting, and build every feature only from data available before the forecast timestamp. Use this time series answer to show the decision, trade-off, and evidence rather than a memorised definition.
Most predictive maintenance projects never reach production. Why, and what would you do differently?
They fail on labels rather than models: failures are recorded as repair dates in free-text work orders, the positive class is nearly empty, feature windows leak the outcome, and no baseline was established, so an encouraging first result cannot be trusted or beaten.