How would you design a validation strategy that respects the structure of the data you have?

The split is a model of the deployment gap

Validation exists to estimate how the model will do on the rows it will meet in production. Every splitting scheme is therefore a hypothesis about how those rows differ from the ones you fitted. A plain random shuffle asserts that production rows are exchangeable with training rows, drawn from the same pool, independent of one another and of time. That assertion is almost never true of business data, and when it is false the estimate is not merely noisy, it is biased upwards.

So the design question is not "which splitter" but "in what respect will production data be new". If it will be new customers, the split must be by customer. If it will be next month, the split must be by time. If it will be a new region or a new device model, the split must hold one out. Answer that first and the splitter follows mechanically.

Group splits, and how large the leak is

Take rows that repeat per entity: 500 patients contributing roughly 40 visits each, 20,000 rows, split at random 80/20. The probability that all 40 of a given patient's rows land in the training side is 0.8 to the power 40:

0.8^40  =  e^(40 x ln 0.8)  =  e^(-8.926)  =  0.00013

and the probability they all land in validation is 0.2^40, which is vanishingly small. So roughly 99.99% of patients have rows on both sides of the split. The model does not have to learn the phenomenon; it can learn to recognise the patient and recall their outcome. On any patient-level feature that is stable across visits, the validation set is effectively a memory test.

The fix is a group-aware split on the patient key, so an entity is wholly in training or wholly in validation. The number that matters afterwards is how the score changes. A drop from 0.93 to 0.74 when you switch from random to grouped folds is the honest measurement, and reporting both is a stronger answer than reporting only the corrected one, because the size of the gap tells you how much of the model was entity recognition.

Groups are also often nested. Visits sit inside patients, patients sit inside hospitals, and if the deployment target is new hospitals then the grouping key is the hospital, not the patient. Choose the key from the deployment question, not from the most granular identifier available.

Time splits, and the window the labels occupy

For anything temporal, train on a prefix and validate on the following period, repeated over several cut-offs so you see how the estimate behaves across regimes rather than trusting one arbitrary boundary. Forward chaining is the standard construction, where each successive fold extends the training prefix and validates on the next block.

The part that gets missed is that the label has a duration. If you are predicting 30-day churn from daily rows, then a row dated 5 June carries an outcome determined by events up to 5 July. Cut training at 30 June and validate from 1 July, and the last thirty days of training rows have labels that were resolved inside the validation window. The information flows the wrong way across a boundary that looked clean. The remedy is an embargo: drop training rows whose label period extends past the cut-off, which for a 30-day label means discarding the final 30 days of training data. It costs data and it is the difference between a forward split and an honest one.

When both constraints apply

Real problems usually impose two. A subscription churn model has customers who recur and a strong time dimension. The order to reason in is to satisfy the constraint that the deployment gap makes primary and then satisfy the other one as far as the data allows.

If you will score existing customers next month, time is primary: split forward, and accept that a customer appears in both the training prefix and the validation window, because that is precisely what production looks like. If you will score customers you have never seen, the group constraint is primary and the split must be grouped and forward at once, holding out both later periods and unseen customer identifiers. That combination shrinks the usable data considerably, which is a real cost and worth stating rather than hiding.

What the folds tell you beyond their mean

A five-fold mean of 0.81 with folds at 0.80, 0.81, 0.81, 0.82, 0.81 and a five-fold mean of 0.81 with folds at 0.68, 0.75, 0.84, 0.88, 0.90 are the same headline and completely different situations. The second is either a fold containing a genuinely different regime, which under a time split usually means the relationship is drifting, or a sample too small for the estimate to be stable. Report the spread alongside the mean, and on a temporal split look at the trend across folds specifically, because monotonically improving or degrading fold scores are a statement about the world rather than about the model.

Fold count is a variance and cost trade-off rather than a matter of taste. More folds means each model trains on more data, so the estimate is less pessimistic, and it costs proportionally more compute while the folds become more correlated. On small data, repeated stratified splits reduce the variance of the estimate at the price of more refits, and nested cross-validation is what keeps hyperparameter selection from contaminating the estimate when you cannot afford a permanent holdout.

The shuffle that nobody questions

The default is the failure. train_test_split with a fixed random seed is the first line of most notebooks, it runs without complaint on grouped and temporal data alike, and it produces a number that is wrong in the flattering direction. Nothing in the output signals the problem: the training and validation scores are close, the learning curve looks healthy, and the model is being graded on rows it has effectively already seen.

That is why the design belongs at the start of a project rather than at the end. Decide what production novelty looks like, encode it in the splitter, and write it down as the reason. A candidate who describes the split before describing the model is demonstrating the habit the question is testing.

Ask what a row represents and how production rows will differ from it, then let that answer choose the splitter. A random shuffle is a claim about exchangeability that most business data does not satisfy, and it fails without any warning.