Order the signals by when they arrive
The constraint is temporal, so design around it explicitly: for each thing you could measure, ask how long after the prediction it becomes available. That ordering is the monitoring architecture.
Operational and data-quality signals arrive in the same request. Input distributions arrive within a window as short as you are willing to make it. The prediction distribution arrives just as fast. Proxy outcomes arrive in hours or days. The real label arrives when the business says it does — a chargeback in weeks, a credit default in months, a churn event only once someone stops paying, and for some decisions never at all, because you declined the applicant and will never learn what they would have done.
Being clear that the last layer is genuinely absent for now is worth more in an interview than listing five drift tests. Everything before it is a leading indicator, and a leading indicator can tell you something changed. It cannot tell you the model got worse.
Data quality first, because it fails more often than statistics do
The most common cause of a model quietly serving nonsense is not a subtle distributional shift, it is a broken upstream job. A column arrives null, a join starts missing, an enumeration gains a value the encoder has never seen, a unit changes from pence to pounds, a timestamp arrives in the wrong zone. Every one of these produces plausible numbers and no errors.
So monitor per feature, at serving time, on the vector the model actually received: null rate, the share of values falling outside the range seen in training, cardinality for categoricals, the share of categoricals mapping to the unknown bucket, and the rate at which any default or imputed value was substituted. That last one catches the failure that is otherwise invisible: a feature service timing out, the code substituting a default, and the model receiving a constant for a week while every distributional test passes because a constant has a perfectly stable distribution.
These are threshold checks against known-good bounds, not hypothesis tests, and they should page someone. A feature whose null rate went from nought point one per cent to forty per cent is not a statistical question.
Against a fixed reference sample from the training period, compare each feature's current distribution: a divergence or population-stability measure on binned continuous features, frequency comparison on categoricals. Weight your attention by how much the model relies on the feature — a shift in the top three features by importance matters, a shift in the fortieth does not, and treating all features as equally alarming is how a drift dashboard becomes wallpaper.
The prediction distribution is the cheapest high-value signal in the whole set, and the most under-used. Watch the mean predicted score, its spread, and the share of predictions crossing each decision threshold. If the score distribution moves and the input distributions did not, something in the serving path changed rather than the world: a model version, a preprocessing step, a feature definition. That inference — inputs stable, outputs moved, therefore the change is ours — is a genuinely useful diagnostic and it needs no labels at all.
Watch the decision rate too, not just the score. A small shift in a score distribution around a threshold produces a large shift in the proportion of customers declined, and the business feels the decision rate long before anyone notices the scores.
Proxy outcomes, chosen honestly
A proxy is an observable event that correlates with the eventual label and arrives sooner. For a credit model whose real label is default at twelve months, early delinquency at thirty days is a proxy. For churn, a decline in engagement is a proxy. For a fraud model, a manual review team's decision arrives in hours where a chargeback takes weeks. For a recommendation model, a click arrives in seconds where the purchase or the return arrives in days.
The discipline is to state what the proxy tracks and where it diverges, because every proxy has a regime in which it moves the wrong way. Clicks are the clearest example: a model that learns to surface provocative items raises clicks and lowers satisfaction, so the proxy improves while the outcome worsens. The mitigation is to monitor the proxy alongside at least one slower measure that captures the thing the proxy is standing in for, and to treat a divergence between them as an alert in its own right rather than as noise.
There is one more source of ground truth worth paying for: withhold a small random slice of traffic from the model's control. If the model's decision determines whether you ever observe the outcome — you decline an application, so you never learn whether it would have defaulted — then your labelled data is selected by the model itself, and it becomes progressively less able to tell you about the region the model has stopped approving. A small randomised holdout, or a policy of approving a few marginal cases you would otherwise refuse, is the only clean way to get an unbiased signal there. It costs real money and it needs a decision from someone who owns the risk, which is exactly why it belongs in this answer rather than being assumed.
What an aggregate hides
Do all of the above per segment, and defend it with arithmetic rather than as good practice.
Take a model with an aggregate accuracy-style metric of 0.85. A segment that is three per cent of traffic collapses from 0.80 to 0.40. The aggregate moves by roughly three per cent of that forty-point fall — about 0.012, from 0.85 to 0.838. That is comfortably inside the week-to-week variation of the aggregate, so nothing alerts, and meanwhile forty per cent of a real group of customers is being handled badly. If that segment is a geography, a new product line, or a protected group, the fact that the aggregate looked fine is not a defence.
Choosing segments is the hard part, since monitoring every combination of every dimension both explodes the alert count and guarantees spurious findings. Pick a small fixed set for which you can articulate why it matters: the dimensions along which the business is managed, the ones a regulator would ask about, new cohorts that were thin in training, and the top few values of the features the model leans on hardest. Fix that list, monitor it consistently, and set a minimum segment size below which you report a wide interval rather than a point estimate — a segment with two hundred rows and ten positives cannot support a stable error-rate estimate, and pretending otherwise generates alerts that are pure sampling noise.
Before the labels arrive you can prove that something changed and often prove that it was your own pipeline. You cannot prove the model still works, and the strongest version of this answer says so rather than dressing leading indicators up as performance.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).