The split answers the wrong question
Shuffling rows and holding out twenty per cent asks the model to fill in gaps.
Every held-out day sits between two training days, so the model can reach the
right answer by noticing that yesterday and tomorrow were both around 400 units.
That is interpolation, and it is easy. Forecasting is extrapolation: at run time
there is nothing after the point you are predicting, because it has not happened.
The gap between the two is large on precisely the series that matter. Anything
with strong autocorrelation — daily demand, traffic, load — is nearly predictable
from its immediate neighbours, so a shuffled split can produce an error figure a
factor better than what the model will achieve in production, and the failure is
silent because every diagnostic you ran looked healthy.
It also breaks the second thing a forecast has to handle, which is that the world
changes. A random split spreads every regime across both halves, so a model
trained on it has already seen the post-price-rise period, the pandemic dip and
the new product launch. The whole reason to hold out the most recent period is to
find out whether the model copes with a period it was not fitted to, and shuffling
removes exactly that test.
Rolling-origin backtesting
The correct evaluation mirrors how the model will be used. Pick a cut-off, train
on everything up to it, forecast the next h periods, and score against what
actually happened. Then move the cut-off forward and repeat. Each cut-off is an
origin, and averaging the errors across many origins gives you an estimate with
some spread to it rather than one number that depends on which fortnight you
happened to hold out.
Three parameters define the protocol and each is a decision to state.
The horizon h must match the decision the forecast supports. If procurement
orders six weeks ahead, evaluate at six weeks, and report error by lead time
rather than as a single average — accuracy at one week ahead tells you almost
nothing about accuracy at six, and quoting the blended figure hides the
degradation that matters.
The window is either expanding, where each origin trains on all history, or
sliding, where it trains on a fixed recent span. Expanding uses more data and is
usually right when the process is stable. Sliding adapts faster after a structural
break and is right when old history is actively misleading. Testing both is cheap
and the comparison is informative in itself.
The gap is the awkward one. If features arrive with a lag in production —
yesterday's sales are not in the warehouse until the following morning — the
backtest must leave the same lag between the end of training and the start of the
forecast. Otherwise you have built a model that depends on data it will not have.
from sklearn.model_selection import TimeSeriesSplit
cv = TimeSeriesSplit(n_splits=6, test_size=30, gap=1)
Features have to respect the clock too
Fixing the split and leaving the features alone is the more common version of
this mistake, and it is harder to see because the split now looks correct.
The rule is that every value used to predict time t must have been knowable
strictly before t, with production availability lags included. Under that rule a
rolling seven-day mean must exclude the current day, so it is the mean of t-7 to
t-1 and never t-6 to t. In pandas terms a rolling window includes the current
row by default, so it needs a shift(1) after it.
Aggregates computed over the whole dataset are the same error at a different
scale. A store-level average sales figure, a category median or a target encoding
of a promotion flag, computed once across all rows and joined back, carries
information from the test period into the training features. These have to be
computed as-of the prediction time, expanding forward, which is more work and is
the difference between a model that works and one that only appeared to.
Preprocessing counts as a feature. Fit a scaler on the full series and its mean
and variance embed the level of the test period. Impute a gap with the series
median and the same thing happens. Both belong inside a pipeline refitted at each
origin.
The last one to check is the target. If the label is "did this customer churn in
the next 30 days", then the row for a customer at time t requires 30 days of
future data to label at all, so rows within 30 days of your training cut-off
cannot be labelled and must be dropped rather than labelled optimistically. And
any feature derived from the outcome window — the number of support tickets during
the churn window, say — is the answer in disguise.
Sanity checks that catch it
Two habits catch most instances before review does. First, compare against a
naive baseline under the same backtest: last observed value, or the value from the
same weekday last week for a seasonal series. A sophisticated model that cannot
beat last week's value on a rolling-origin backtest is either not working or not
needed, and models that beat it by an implausible margin usually have a leak.
Second, be suspicious of improvement itself. If a new feature drops error by a
third, find out what it is made of before celebrating. On forecasting problems a
large unexplained gain is more often a leak than a discovery, and the cost of
checking is an hour against a quarter of trusting a number that was never real.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).