When does ROC-AUC mislead you, and when does a precision-recall curve mislead you instead?

What each curve is a function of

A ROC curve plots the true-positive rate against the false-positive rate as the threshold sweeps. Both axes are conditioned on the true class: TPR is TP/(TP+FN), computed entirely within the positives, and FPR is FP/(FP+TN), computed entirely within the negatives. Because each axis lives inside one class, neither depends on the ratio between the classes. The area under the curve has a clean interpretation: it is the probability that a randomly chosen positive receives a higher score than a randomly chosen negative. It is a pure statement about ranking.

A precision-recall curve plots precision, TP/(TP+FP), against recall, which is the same quantity as TPR. Precision mixes the classes in its denominator, so it depends directly on how many negatives there are. That difference in denominators is the whole answer, and the arithmetic makes it concrete.

The arithmetic of why ROC flatters an imbalanced problem

Take 100,000 transactions with 200 fraudulent, a prevalence of 0.2%, leaving 99,800 negatives. Suppose at a chosen threshold the model raises 2,000 alerts, of which 150 are genuine fraud.

Quantity	Calculation	Value
True positives	given	150
False positives	2,000 alerts − 150	1,850
Recall / TPR	150 / 200	0.750
False-positive rate	1,850 / 99,800	0.0185
Precision	150 / 2,000	0.075

The ROC point is (0.019, 0.750), which sits close to the top-left corner and looks like a strong classifier. The precision at the same operating point is 7.5%, meaning 92.5% of everything you send to the review team is wrong.

Push it further to see how insensitive the FPR is. To move the FPR from 0.0185 to 0.037, still a respectable-looking figure, you would need 3,700 false positives. The negative class is so large that the denominator absorbs almost any volume of mistakes. Precision, whose denominator is the alert count itself, halves from 7.5% to about 3.9% over that same range. The same model, the same predictions, and one metric registers the damage while the other does not.

Note also that the areas start from different floors. A random model has ROC-AUC 0.5, so 0.5 is the anchor most people carry in their heads. A random model's precision-recall curve is a horizontal line at the prevalence, 0.002 here. A PR-AUC of 0.15 on this problem is therefore roughly seventy-five times the random baseline and a genuinely good result, while looking terrible to anyone reading it against the 0.5 anchor.

The direction nobody mentions, where precision-recall misleads

Because precision depends on prevalence, PR numbers are not portable. Two consequences follow and both bite in production.

You cannot compare PR-AUC across datasets. A model scoring 0.40 on a 5% prevalence sample and a model scoring 0.15 on a 0.2% sample may be identical in ranking quality; the difference is arithmetic, not skill. Any benchmark table that compares average precision across problems with different base rates is comparing the problems, not the models.

Worse, you cannot compare it across time on the same problem. If fraud prevalence falls by half because an upstream rule started blocking a category of attack, every precision figure falls with it and your monitoring dashboard reports a model degradation that did not happen. ROC-AUC is invariant to that shift, which is exactly why it is the better instrument for answering "has the model's ranking got worse". This is the reason to track both: ROC-AUC tells you whether the model changed, precision at your operating point tells you whether the system is still worth running.

There are also cases where the negatives genuinely matter and ROC is the honest choice. When you will re-threshold for a deployment population whose prevalence differs from your sample, when you are comparing candidate models purely on ranking before any threshold exists, and when the downstream consumer is another scoring stage rather than a human queue, the prevalence-independence is a feature rather than a blind spot.

Neither number is the thing you deploy

Both are threshold-free summaries, obtained by integrating over operating points you will never use. The model runs at one threshold, chosen from capacity or from a cost ratio, so the number that predicts the experience of the people receiving the alerts is precision at that point together with the recall it buys. Report that pair, and quote the areas only as model-comparison instruments. In the worked example, "75% recall at 2,000 alerts per 100,000 transactions, 7.5% precision" is a sentence a fraud manager can act on. "0.94 ROC-AUC" ends the conversation without informing it.

Where the answer usually goes wrong

The common failure is not choosing wrongly, it is treating the choice as a rule to be memorised. A candidate who says "use PR for imbalanced data" has the right heuristic and cannot defend it, and will therefore also apply it when tracking drift, where it produces a false alarm every time the base rate moves. The strong answer is mechanical: name the denominators, say which contains the large count, derive the insensitivity from that, then observe that the same dependence which makes precision informative about alert quality makes it unstable as prevalence moves.

The false-positive rate divides by every negative in the dataset, so on a rare-event problem it cannot register the volume of wrong alerts. Precision can, because its denominator is the alert count, and that is also why precision moves when prevalence moves and the model has not changed at all.