How do you choose the decision threshold for a classifier, and what makes it move?

Ask what happens to a positive before assigning any number

The threshold cannot be chosen from the data, because it encodes a preference between two kinds of mistake, and that preference lives in the business. The question that unlocks it is what the system does with a positive prediction. A flag that blocks a payment, a flag that opens a case for a human, and a flag that adds a verification step have false-positive costs differing by two orders of magnitude, and the same model therefore has three different correct thresholds.

So the first move is to name the action, then to price the two errors in the same units.

The derivation

Let p be the model's probability that the case is positive, C_FP the cost of acting on a case that turns out negative, and C_FN the cost of not acting on one that turns out positive. Assume for clarity that a correct decision of either kind costs nothing, which is a simplification you can drop by using cost differences instead.

expected cost of acting        =  (1 - p) x C_FP
expected cost of not acting    =       p  x C_FN

Act when the first is smaller:

(1 - p) x C_FP  <  p x C_FN
C_FP - p x C_FP <  p x C_FN
C_FP            <  p x (C_FN + C_FP)
p               >  C_FP / (C_FP + C_FN)

The threshold is the false-positive cost as a share of the total cost of the two errors. It contains no reference to prevalence, and that absence is the most informative feature of the formula.

The same model, two thresholds, one order of magnitude apart

Take a fraud model and price the errors two ways.

Deployment	C_FP	C_FN	t = C_FP / (C_FP + C_FN)
Flag opens an analyst case, four minutes at £60/hour	£4	£180	4 / 184 = 0.022
Flag blocks the payment, cost of a wrongly declined customer	£60	£180	60 / 240 = 0.250

The threshold moves from 2.2% to 25%, a factor of eleven, with no change to the model and no change to the data. That is the number to carry out of this question, because it shows that a default of 0.5 is not a neutral choice but an assertion that the two errors cost the same amount, which they almost never do. At 0.5 the implied claim here is C_FP = C_FN, so the payment-blocking deployment is behaving as though a declined customer costs £180.

The cost figures themselves come from the business and should be sourced explicitly. The analyst cost is a loaded hourly rate divided by cases per hour, both of which finance and operations already know. The fraud loss is the average net loss per confirmed fraud, which the fraud team reports. The cost of a wrongly declined customer is the one nobody has, and the honest handling is to solve for it: state that at a threshold of 0.25 you are implicitly valuing a false decline at £60, and ask whether that is too high or too low. People who cannot produce a number can almost always react to one.

Calibration is the precondition, not a refinement

The derivation compares p × C_FN against a monetary amount, so p must be a probability rather than a rank score. Feed an uncalibrated score into a cost-derived threshold and you have combined a correct formula with a wrong scale.

Suppose the model reports 0.04 for a set of cases whose true positive rate is 0.015, and the threshold is 0.022. Those cases clear the threshold, so you act. The real arithmetic on the group, using the analyst-review costs, is 0.015 × 180 = £2.70 of loss avoided against £4 of review cost, a net loss of £1.30 per case. Nothing in the monitoring reports it, because the threshold was derived correctly and the recall looks fine. Calibrate first, then derive the threshold, and re-derive it after any recalibration.

Two regimes, and why they respond to prevalence differently

This is where the answer separates. If the constraint is genuinely the cost ratio and the probabilities are calibrated, the threshold does not move when prevalence changes. The formula has no prevalence term. What moves is the volume of cases clearing it, because the score distribution has shifted. A doubling of prevalence from 0.05% to 0.1% roughly doubles the alert count from 9,000 to about 18,000 a month, and the correct response is to keep the threshold and get more capacity, because each of those additional alerts still has positive expected value by construction.

If the constraint is a fixed review capacity, the threshold must move. Capacity is 9,000 a month, prevalence doubles, and holding the threshold at 0.022 produces a backlog that ages until cases become worthless. So you raise the threshold until the volume fits, precision rises, and recall relative to the new fraud population falls. That is a real loss and it should be reported as one: the cases you are now declining to review had positive expected value, and the gap between the cost-optimal volume and the affordable volume is the business case for hiring.

The two regimes therefore need different monitoring. Under cost optimisation, watch the alert volume and the calibration, and alarm when volume drifts. Under a budget, watch the threshold itself, because it is now a moving quantity whose drift is the signal, and watch the expected value of the marginal case you are dropping.

Fitting the threshold without overfitting it

The threshold is a parameter and it is selected by looking at data, so it is subject to the same discipline as any other. Choose it on validation folds, not on the test set, and prefer a plateau to a peak: if the cost surface is flat between 0.02 and 0.03, take a value in the middle rather than the exact argmin, which is fitted to noise in the tail of the score distribution. On a rare-event problem the number of positives above the threshold is small, so the estimate is noisy by construction, and a confidence interval on the resulting precision is more honest than a point estimate.

Leaving the default in place is a decision

The failure mode is not choosing badly, it is not choosing. A model shipped at predict rather than predict_proba has a threshold of 0.5 that nobody discussed, nobody owns and nobody wrote down, and on a rare-event problem it will flag almost nothing at all because very few cases ever reach 0.5. The team then concludes the model does not work and tunes the model, which is the wrong artefact. Make the threshold a configured, versioned, reviewable value with the cost assumptions recorded next to it, so that when the costs change somebody can change the number without retraining anything.

The threshold is the false-positive cost divided by the sum of both error costs, and 0.5 is the special case where the two errors cost the same. Whether prevalence should move it depends entirely on whether your real constraint is the cost ratio or the size of the review team.