The gap between training and validation
The primary signal is the difference between the score on data the model fitted and the score on data it has never seen. A model at 0.98 on training and 0.71 on validation has memorised something specific to the training rows that does not generalise. The absolute numbers matter far less than the divergence, which is why quoting one accuracy figure tells an interviewer nothing about whether the model works.
The most informative version of this is a learning curve: plot both scores as training progresses, or as you feed in more data. Overfitting has a characteristic shape, where training loss keeps falling while validation loss flattens and then turns upward. That turning point is diagnostic in a way a single pair of numbers is not, because it distinguishes a model that has finished learning from one that has started memorising.
Be careful about the opposite claim. Some gap is normal, because the model has genuinely seen the training rows and the validation set is a finite sample with its own noise. A two-point gap on a well-sized validation set is usually not worth acting on, and chasing a zero gap tends to produce an underfitted model instead - the more expensive mistake, because it discards signal you paid to collect.
When the split itself is the leak
A single random split silently assumes your rows are independent and interchangeable. When they are not, the split leaks and the gap you rely on disappears - the model looks fine on validation and then fails in production, which is the worst failure mode because nothing warned you.
Two cases account for most of it. With time-ordered data, a random split puts future rows into training and past rows into validation, so the model gets to see the outcome period it is meant to predict. You must split forward in time, training on everything before a cut-off and validating after it, and repeat that across several cut-offs rather than trusting one. With grouped data - many rows per patient, per customer, per device, per document - a random split scatters one group's rows across both sides, and the model can score well by recognising the group rather than learning the phenomenon. The fix is to split on the group key so that every row belonging to a group lands wholly on one side.
from sklearn.model_selection import TimeSeriesSplit, GroupKFold, cross_val_score
scores = cross_val_score(model, X, y, cv=TimeSeriesSplit(n_splits=5))
scores = cross_val_score(model, X, y, groups=customer_id, cv=GroupKFold(n_splits=5))
The same reasoning extends to preprocessing. Fitting a scaler, an imputer or a target encoder on the full dataset before splitting leaks statistics from validation into training, so those steps belong inside a pipeline that is refitted per fold.
What to try, in order
Start with data, because it has the largest effect and the fewest side effects. More labelled examples is the direct remedy for a model that has enough capacity to memorise a small set. Better data often beats more of it: fixing mislabelled rows, removing near-duplicates that inflate your validation score, and dropping features that are proxies for the target.
If more data is not available, reduce the model's capacity to memorise - fewer trees or shallower ones, fewer parameters, fewer engineered features. Then apply regularisation, which penalises complexity rather than forbidding it: L2 or L1 penalties on linear models, weight decay and dropout in networks, max_depth and min_samples_leaf on trees. Early stopping is the cheapest intervention of all when the learning curve turns upward, because it simply keeps the weights from the epoch where validation loss was lowest. Augmentation belongs last in this ordering, not because it is weak - it is very effective for images and audio - but because it requires transformations that genuinely preserve the label, and inventing those for tabular data is harder than it looks.
Tuning quietly overfits the validation set
Every decision you make by looking at the validation score spends a little of that set's independence. Try forty hyperparameter configurations and pick the best, and part of that winning score is the configuration having got lucky on this particular sample. The reported number is then optimistic by an amount you cannot measure from inside the process, and the effect compounds when the same set also guided feature selection, threshold choice and architecture.
This is the failure that separates a strong answer from an adequate one, because it is invisible: nothing looks wrong, the gap is small, and the model still underperforms once deployed. The defence is structural. Hold out a test set at the very beginning, do all searching against validation folds, and read the test set once when you believe you are finished. If the test result disappoints you and you go back to tune further, that set is now part of your training loop and you need a fresh one. On small datasets, nested cross-validation gives the same guarantee without permanently sacrificing rows, at the cost of an outer loop of refits.
The train/validation gap only means something if the split is honest, and the validation score only means something if you have not read it a hundred times. Overfitting to your own evaluation procedure is more common in practice than overfitting to the data.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).