What is the difference between data drift and concept drift, and does it change what you do?

Which distribution moved

A supervised model learns a mapping from inputs to an outcome. Written as distributions, the data it was trained on has an input distribution and a conditional distribution of the outcome given the inputs. Drift is one of those two moving, and which one it is determines almost everything about your response.

Data drift — also called covariate shift — is the input distribution changing while the relationship between inputs and outcome holds. The same customer profile still implies the same probability of default; it is simply that you now see more of one profile and less of another. Your traffic mix changed, a new channel started sending you a different population, a product launched in a new country.

Concept drift is the conditional distribution changing: the same inputs now imply a different outcome. What a given set of features meant last quarter is not what it means now. Fraud tactics adapting is the archetype, because the adversary's whole purpose is to make the old mapping wrong. A pricing change that alters how customers respond to the same offer does it too, as does a policy change upstream — if the credit team tightens who gets approved, the meaning of "approved and then defaulted" has moved even though the applicants have not.

A third case is worth naming because it is common and it is neither: label drift, where the base rate of the outcome shifts. That may be a symptom of either of the other two, or of a change in how the label is recorded, which is a data-quality problem wearing drift's clothes.

The detection asymmetry is the practical difference

Data drift is detectable from inputs alone, which means immediately and cheaply. You hold a reference sample from the training period and compare the current window against it, feature by feature and ideally jointly: a two-sample test for continuous features, a divergence measure or population-stability index for binned ones, a comparison of category frequencies and cardinality for discrete ones. None of that requires knowing what happened.

Concept drift is not detectable from inputs alone, and this is the part that separates a confident answer from a recited one. The inputs can be perfectly stationary while the mapping rots underneath, and no test on the input distribution will ever see it. To observe concept drift you need something that reflects the outcome: realised labels compared against predictions, or a proxy that correlates with the outcome and arrives sooner, or a deliberately held-out slice of traffic where you can measure what happens when the model is not in control.

Prediction-distribution monitoring sits usefully between the two: available immediately, like input monitoring, and it moves when either the inputs or the serving path change. What it cannot do is distinguish a model that is wrong from a population that genuinely became riskier, so it triggers an investigation rather than settling one.

Only one of them makes the model wrong

Here is the conclusion the question is fishing for, and getting it the wrong way round is the usual failure.

Under pure data drift, the mapping the model learned is still correct. If more of your traffic now looks like a segment the model already handles well, nothing about the model is invalid — its aggregate metrics may even change, because the mix of easy and hard cases changed, without any loss of skill on any individual case. Retraining is not indicated by the drift itself. What data drift does do is move traffic towards regions of the input space where the training data was thin, and there the model's behaviour was never really established. So the honest position is that data drift is a prompt to check whether the new population is inside the support of your training data, and to check per-segment performance for the segment that grew. If the growing segment is well represented in training, you have a reporting problem, not a model problem.

Concept drift is different in kind. The relationship the model encodes is no longer the relationship in the world, so the model is wrong in a way no amount of infrastructure fixes. The only remedy is to learn the new relationship, which means retraining on data that contains it — and that in turn means your training window has to be short enough, or weighted recently enough, to reflect the change rather than average it away with two years of history.

Where the vocabulary misleads

The word "drift" implies slow, and it makes people build slow detectors. Concept drift is frequently not gradual: a regulatory change, a competitor's price move, a change to your own upstream policy all shift the mapping abruptly on a known date. Sudden change is far easier to diagnose, because there is a date to correlate against a deployment log — but only if your monitoring window is short enough to show a step rather than smear it.

The second misleading effect is statistical. Run a two-sample test per feature on a large window and it will reject, because with enough samples a test detects differences too small to matter. Three hundred features tested hourly at a five per cent significance level yields around fifteen alerts an hour from chance alone, which trains everyone to ignore the dashboard. Either monitor a few aggregated quantities, or report an effect size against a threshold you can defend, or correct for multiple comparisons — and in every case weight attention towards the handful of features the model leans on.

Data drift changes who you are seeing; concept drift changes what they mean. The first is detectable without labels and does not by itself invalidate the model; the second needs an outcome signal and cannot be fixed by anything except learning the new relationship.