Why does a random train-test split ruin a forecasting model?

The split answers the wrong question

Shuffling rows and holding out twenty per cent asks the model to fill in gaps. Every held-out day sits between two training days, so the model can reach the right answer by noticing that yesterday and tomorrow were both around 400 units. That is interpolation, and it is easy. Forecasting is extrapolation: at run time there is nothing after the point you are predicting, because it has not happened.

The gap between the two is large on precisely the series that matter. Anything with strong autocorrelation — daily demand, traffic, load — is nearly predictable from its immediate neighbours, so a shuffled split can produce an error figure a factor better than what the model will achieve in production, and the failure is silent because every diagnostic you ran looked healthy.

It also breaks the second thing a forecast has to handle, which is that the world changes. A random split spreads every regime across both halves, so a model trained on it has already seen the post-price-rise period, the pandemic dip and the new product launch. The whole reason to hold out the most recent period is to find out whether the model copes with a period it was not fitted to, and shuffling removes exactly that test.

Rolling-origin backtesting

The correct evaluation mirrors how the model will be used. Pick a cut-off, train on everything up to it, forecast the next h periods, and score against what actually happened. Then move the cut-off forward and repeat. Each cut-off is an origin, and averaging the errors across many origins gives you an estimate with some spread to it rather than one number that depends on which fortnight you happened to hold out.

Three parameters define the protocol and each is a decision to state.

The horizon h must match the decision the forecast supports. If procurement orders six weeks ahead, evaluate at six weeks, and report error by lead time rather than as a single average — accuracy at one week ahead tells you almost nothing about accuracy at six, and quoting the blended figure hides the degradation that matters.

The window is either expanding, where each origin trains on all history, or sliding, where it trains on a fixed recent span. Expanding uses more data and is usually right when the process is stable. Sliding adapts faster after a structural break and is right when old history is actively misleading. Testing both is cheap and the comparison is informative in itself.

The gap is the awkward one. If features arrive with a lag in production — yesterday's sales are not in the warehouse until the following morning — the backtest must leave the same lag between the end of training and the start of the forecast. Otherwise you have built a model that depends on data it will not have.

from sklearn.model_selection import TimeSeriesSplit

# Fixed 30-period test window per fold, each fold training only on earlier data.
# gap leaves the production data-arrival lag between train and test.
cv = TimeSeriesSplit(n_splits=6, test_size=30, gap=1)

Features have to respect the clock too

Fixing the split and leaving the features alone is the more common version of this mistake, and it is harder to see because the split now looks correct.

The rule is that every value used to predict time t must have been knowable strictly before t, with production availability lags included. Under that rule a rolling seven-day mean must exclude the current day, so it is the mean of t-7 to t-1 and never t-6 to t. In pandas terms a rolling window includes the current row by default, so it needs a shift(1) after it.

Aggregates computed over the whole dataset are the same error at a different scale. A store-level average sales figure, a category median or a target encoding of a promotion flag, computed once across all rows and joined back, carries information from the test period into the training features. These have to be computed as-of the prediction time, expanding forward, which is more work and is the difference between a model that works and one that only appeared to.

Preprocessing counts as a feature. Fit a scaler on the full series and its mean and variance embed the level of the test period. Impute a gap with the series median and the same thing happens. Both belong inside a pipeline refitted at each origin.

The last one to check is the target. If the label is "did this customer churn in the next 30 days", then the row for a customer at time t requires 30 days of future data to label at all, so rows within 30 days of your training cut-off cannot be labelled and must be dropped rather than labelled optimistically. And any feature derived from the outcome window — the number of support tickets during the churn window, say — is the answer in disguise.

Sanity checks that catch it

Two habits catch most instances before review does. First, compare against a naive baseline under the same backtest: last observed value, or the value from the same weekday last week for a seasonal series. A sophisticated model that cannot beat last week's value on a rolling-origin backtest is either not working or not needed, and models that beat it by an implausible margin usually have a leak.

Second, be suspicious of improvement itself. If a new feature drops error by a third, find out what it is made of before celebrating. On forecasting problems a large unexplained gain is more often a leak than a discovery, and the cost of checking is an hour against a quarter of trusting a number that was never real.