What does it mean for a model to be calibrated, and why is an uncalibrated probability dangerous?

Calibration is a claim about groups, not about cases

A model is calibrated if, for every score value it emits, the observed frequency of positives among cases receiving that score equals the score. Gather the 1,000 cases the model scored near 0.30; if about 300 of them turned out positive, the model is calibrated in that region. No single prediction can be checked, because a single case is either positive or not. Calibration is only ever measured over a bucket.

That framing matters because it makes calibration testable with nothing more than a held-out set and a group-by. Bin the scores, compute the mean predicted probability and the observed positive rate in each bin, and plot one against the other. A calibrated model traces the diagonal. A model whose curve sits below the diagonal is overconfident, promising more risk than materialises; above it, underconfident. Expected calibration error summarises the gap as a weighted mean absolute deviation across bins, and the Brier score, the mean squared error of the probability against the 0/1 outcome, gives a single number that penalises both miscalibration and lack of discrimination together.

Ranking and probability are independent properties

Take any calibrated model and pass its outputs through a monotone squashing function, so 0.30 becomes 0.03 and 0.90 becomes 0.09. Every pairwise ordering is preserved, so ROC-AUC is unchanged and identical to before. Every probability is now wrong by a factor of ten. AUC is a function of the ranking alone and is structurally incapable of detecting this, which is why a model report consisting of AUC says nothing about whether the numbers mean anything.

This is the fact that reframes the question. Calibration is not a refinement of accuracy, it is an orthogonal axis, and a model can be excellent on one and useless on the other.

What a boosted tree's score is not

A gradient-boosted classifier trained on log loss emits a number between 0 and 1 obtained by passing the ensemble's additive score through a logistic function. It is not a posterior probability, and treating it as one is the commonest version of this mistake because the number looks exactly like one and the API method is called predict_proba.

Several mechanisms push it away from calibration. Boosting to many rounds drives training log loss down by pushing scores towards the extremes, so a well-fitted ensemble is typically overconfident at both ends. A random forest averages votes and tends the other way, pulled towards the middle because a bagged average rarely reaches 0 or 1. A support vector machine emits a margin with no probabilistic interpretation at all. And any model trained on resampled or class-weighted data has been fitted to a prior that does not match the world, so its scores are shifted by a known and correctable amount.

Only a well-specified logistic regression on the correct prior is calibrated more or less for free, because fitting it is fitting the probability directly.

The arithmetic of what miscalibration costs

Calibration only matters where a number rather than an ordering drives the decision, and threshold selection from costs is the standard case. Suppose a missed failure costs £200 and an unnecessary inspection costs £5. Expected cost of inspecting is £5; expected cost of not inspecting is p × 200. Inspect when:

p x 200 > 5      ->      p > 5 / 200  =  0.025

So the cost-optimal threshold is 2.5%. Now suppose the model is overconfident and reports 0.040 for a set of cases whose true positive rate is 0.015. Those cases clear the 0.025 threshold, so you inspect them. The real arithmetic on that group is:

expected loss avoided  =  0.015 x 200  =  £3.00
cost incurred          =              =  £5.00
net                    =  3.00 - 5.00 =  -£2.00 per case

Every one of those inspections destroys £2 of value, and nothing in your monitoring says so. The recall looks fine, the AUC looks fine, and the threshold was derived correctly from costs that were correct. The error entered through the probability. Run 40,000 such inspections a year and the miscalibration has cost £80,000 while every dashboard stayed green.

Where a real probability is required

Expected-value decisions of the kind above are the first case. Any threshold derived from a cost ratio is only valid if the score is a probability, otherwise you have derived a correct number and applied it to the wrong scale.

The second is combination. Multiplying a model's probability by an expected loss amount, or by another model's probability, or feeding it into a portfolio-level aggregate such as expected credit loss, propagates the error multiplicatively. Aggregating uncalibrated scores across a book gives a total with no meaning.

The third is communication. When a clinician, an underwriter or a credit officer is told "38% chance", they will act on that as a frequency, and they are entitled to. Quoting a rank score as a percentage to a human decision-maker is the most consequential form of this error because the human cannot audit it.

By contrast, if the only use is to sort a queue and hand the top N to a review team, the ranking is all you need and calibration is optional. Say so, because knowing when it does not matter is part of knowing what it is.

Repairing it, and the set you fit the repair on

Both standard remedies learn a monotone map from raw scores to calibrated probabilities, which leaves the ranking and therefore the AUC untouched.

Platt scaling fits a one-dimensional logistic regression on the scores, so it has two parameters and can only apply a sigmoid-shaped correction. It is the right choice with a few hundred calibration points because it cannot overfit much. Isotonic regression fits an arbitrary non-decreasing step function, so it can correct any monotone distortion, and it needs materially more data because with few points it will overfit the calibration set and produce a staircase with wide flat regions.

The critical constraint is the data you fit on. Calibrating on the training scores learns the map from a distribution the model has already memorised, which is optimistic in exactly the region you are trying to correct. Use a dedicated held-out calibration set, or cross-validated calibration where the map is fitted out of fold, and then verify on a third set you did not touch. And recalibrate whenever the base rate moves, because a shift in prevalence invalidates the map even when the ranking is untouched.

Calibration and discrimination are independent, and AUC measures only the second. If any decision multiplies your score by a cost, the score has to be a frequency, and you have to have checked that it is one.