The delay sets the clock for everything downstream
If the ground truth for a prediction is only known sixty days later, then the newest data you can train on is sixty days old, the newest performance figure you have describes a model as it behaved sixty days ago, and any change you make today cannot be assessed on real outcomes for another sixty days. That single number propagates into the design of the retraining schedule, the evaluation harness, the monitoring stack and the release process, and a candidate who does not trace it through all four has not understood the constraint.
Credit default is the canonical case, with delays measured in months. Churn is similar: a customer has not churned, they have merely not been seen recently, and the label is a definition applied after a window elapses. Fraud chargebacks arrive on the card network's timetable. Medical outcomes and long-cycle sales are worse. Even in short-cycle domains the delay is rarely zero — an advertising conversion may take days, and the tail is long.
Retraining faster than labels arrive is theatre
Retraining weekly when labels take sixty days does not make the model fresher. Each weekly run sees a training set that has advanced by one week at the far end, meaning roughly a week of genuinely new labelled data appended to a window that is already two months behind. The model that emerges is a marginal variation on last week's, retrained mostly on the same rows, and the churn it creates — a new artefact, a new promotion decision, a new deployment — is cost without corresponding benefit.
Worse, it distorts evaluation. If you retrain weekly and evaluate each candidate on the most recent labelled window, that window barely moves between runs, so you are effectively re-testing on the same data repeatedly and selecting the model that best fits it. Over enough cycles that is overfitting to the holdout by another name.
The defensible cadence is derived, not chosen. Set it from the label delay and from how fast the input distribution genuinely moves, taking the slower of the two. A useful sanity check is whether a retrain has enough new labelled rows to plausibly change the model: if a cycle adds two per cent of your training set, it will not.
There is a legitimate exception. Retraining frequently on unchanged label semantics but a shifting feature distribution — where the point is to re-fit to a new input regime rather than to learn from new outcomes — can be worth it. But say that is what you are doing, because it is a different justification.
Every recent evaluation window is censored
This is the subtlety most answers miss. When you evaluate on the last thirty days, you are not evaluating on a small sample of the last thirty days. You are evaluating on the non-random subset whose labels have already arrived, and the arrival process is usually correlated with the outcome.
Consider fraud: confirmed fraudulent transactions are often reported quickly by the cardholder, while legitimate transactions are only known to be legitimate by the passage of time without dispute. Look at the last two weeks and the positive rate appears wildly elevated, because positives resolve faster than negatives. A model evaluated on that window looks miscalibrated when it is not, and a model trained on that window learns an inflated base rate.
Predictions made in week 0
|
|-- labels resolved by week 1 : fast resolvers <- biased subset
|-- labels resolved by week 4 : most cases
|-- labels resolved by week 12 : slow tail <- often the hard cases
v
Evaluating at week 1 measures the fast resolvers only.
The disciplines that fix this are unglamorous. Evaluate on cohorts defined by prediction date and only when that cohort has matured past the delay, so every comparison is like-for-like. Track label completeness per cohort as a first-class metric, and refuse to report a headline number for a cohort below a threshold. Where you must report early, report the maturation curve alongside it so the number carries its own caveat. Survival-analysis framing helps if the delay distribution is a genuine part of the problem rather than a nuisance.
Proxy labels, and being honest about the bias
The pressure to move faster is real, so proxies are legitimate — provided you say what they cost. A proxy is a signal that arrives sooner and correlates with the outcome: a first missed payment standing in for default, a thirty-day inactivity flag standing in for churn, a click standing in for a purchase, an internal analyst review standing in for a confirmed chargeback.
The risk is not that a proxy is noisy. Noise you can reason about. The risk is that it is systematically different from the target, so optimising it moves you somewhere you did not intend. Missed payments over-represent people with cash-flow timing problems who never default; a model tuned on them learns to detect irregular payers rather than bad credit. Clicks over-represent whatever is most attention-grabbing; a model tuned on them learns to bait. And the bias is directional, so it does not average out with more data — more data makes you more confident in the wrong objective.
Two practices make proxies safe enough to use. First, keep collecting the true label and periodically measure the proxy against it on matured cohorts, tracking the agreement over time rather than validating it once. Agreement degrading is a signal in its own right, and it often degrades precisely because your intervention changed behaviour. Second, keep the true label as the promotion criterion even when the proxy drives training: candidate models may be selected quickly on the proxy, but the decision to keep a model in production is settled on matured outcomes. That split — fast iteration on the proxy, slow adjudication on the truth — is the shape most successful teams converge on.
One trap deserves naming on its own. If the model's prediction triggers an intervention, the intervention corrupts the label. Decline the applications the model thinks will default and you never learn whether they would have; the population you observe outcomes for is the population the model approved. Your labels are now conditioned on the previous model's decisions, and each retrain narrows the observed world further. Mitigations exist — hold out a small randomised slice that bypasses the model, or log the score at the decision boundary and reason about the discarded region — but they cost money and have to be argued for before the feedback loop tightens, not after.
Derive the retraining cadence from the label delay, and treat any recent evaluation window as a biased sample rather than a small one. The subset whose labels have arrived is almost never a random subset.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).