How do you decide when to retrain a model?

The condition, stated properly

A retrain is worth doing when two things hold at once: there is evidence the relationship the model encodes no longer matches the world, and you have data that contains the new relationship. Both halves matter, and the second is the one that gets skipped.

The first half means an outcome signal, not an input signal. Realised performance falling on labelled data, a proxy outcome deteriorating, a decision rate that has moved in a way the business can feel. A shift in an input distribution is not evidence that the mapping is wrong — it is evidence the population changed, which may leave a correctly-learned mapping entirely intact.

The second half means enough recent, correct, labelled examples of the new regime to actually learn from. If the world changed three weeks ago and your labels take sixty days, then three weeks after the change you have no labelled data from the new regime, and retraining will produce a model fitted to the old one with slightly different noise. Recognising that retraining is not yet possible is a legitimate and frequently correct answer, and the interim response is a different lever: adjust the decision threshold, add a rule for the cases you know are being handled badly, or route them to a human.

What a schedule is for

Monthly retraining is not a decision, it is a default that people adopt because deciding is harder. That said, it is a defensible default in exactly one circumstance: when your outcome signal is too delayed or too noisy to trigger on, a regular cadence bounds how stale the model can get. Say so in those terms and the answer reads as a judgement rather than a habit.

The failure modes of an unexamined cadence run in both directions. Too frequent, and you are shipping models nobody has scrutinised, paying for training you did not need, and adding variance: two models trained on overlapping data with different random seeds will disagree on borderline cases, so your decisions become less stable without becoming better. Too infrequent, and a step change in the world sits unaddressed until the calendar comes round.

A cadence also lets a slow problem hide. If a feature has been degrading for six weeks and the monthly retrain absorbs the degraded data as if it were normal, each new model is fitted to slightly worse inputs, the offline metrics look reasonable because the evaluation set has the same defect, and nothing ever surfaces. Scheduled retraining without data validation is how a pipeline bug becomes permanent.

The alert that means your pipeline broke

Now the part that separates the answers. A drift alert has at least two explanations and the less interesting one is far more common: the world changed, or your data about the world changed.

An upstream job that started failing silently, a schema change that renamed a column so the join now yields nulls, a third-party enrichment API returning defaults after a contract change, a unit conversion, a backfill that rewrote history, a timezone shift after a deployment. Every one of these registers as drift on any distributional test, because the distribution of the data genuinely moved. None of them is a reason to retrain. All of them are reasons to fix something.

Retraining on that data is worse than doing nothing, and it is worth being concrete about why. Suppose an enrichment service began returning a default value for a feature the model relies on. Before the retrain, you have a good model receiving one broken feature — bad, and recoverable the moment the feature is fixed. After an automatic retrain on the broken data, you have a model that has learned the constant, redistributed its weight onto other features to compensate, and been validated against a held-out set drawn from the same broken window, so it passes every gate. When the upstream service is repaired, the feature starts arriving correctly again and the new model is now wrong in a fresh way that no alert predicted. You have converted a transient input fault into a baked-in model fault and destroyed your last known-good baseline in the same step.

So the order of operations is not negotiable: validate the data, then decide about the model.

The validation that has to run first

Before a retrain is allowed to proceed, the training data has to pass checks against the previous training set rather than merely being present. Row counts and their expected range, so a partial extract is caught. Null rates per column against the historical rate. Numeric ranges and cardinality per categorical. Distribution comparison against the previous window, with a threshold that halts the pipeline rather than logging a warning. Label base rate, which is frequently the earliest signal that something in the outcome-recording path changed. And freshness — the maximum timestamp present, since a pipeline reading a stale partition is a common and undetectable-by-eye failure.

flowchart TD
  A[Performance drop<br/>or drift alert] --> B[Validate current data]
  B -->|checks fail| C[Fix upstream<br/>no retrain]
  B -->|checks pass| D{Outcome signal moved}
  D -->|no| E[Investigate segments<br/>then stop]
  D -->|yes| F[Enough labelled data<br/>from new regime]
  F -->|no| G[Adjust threshold<br/>or add rules]
  F -->|yes| H[Retrain and gate<br/>against champion]

Two of those branches end without a retrain, which is the shape a healthy process has.

The gate at the end matters as much as the trigger at the start. A retrained model is a candidate, not a replacement. It has to beat the incumbent on a held-out set that includes recent data, hold up per segment rather than only in aggregate, and — because offline agreement is weak evidence — earn a shadow or canary period before it takes traffic. A candidate that wins overall while losing on a segment that matters is a decision for a person, not a threshold. And the champion stays deployed and servable throughout, because the fastest fix for a bad retrain is routing traffic back, not another retrain under pressure.

The window nobody specifies

One more choice usually left implicit: how much history to train on. A long window averages over regime changes and produces a model that is mediocre in every regime, including the current one. A short window tracks the present and is noisier, and it may not contain enough of a rare class to learn it — a fraud model trained on six weeks of a rare label may see very few positives. Recency weighting is the compromise, and whichever you pick, the window length is a parameter with a reason behind it, not a constant somebody typed once.

Two questions in order, every time: has my data changed, or has the world changed? Only the second is a reason to retrain, and confusing them is how a broken upstream job gets permanently compiled into a model.