hardConceptScenarioMidSeniorStaff
How do you choose the decision threshold for a classifier, and what makes it move?
Choose a classifier decision threshold from calibrated probability, false-positive cost, false-negative cost and operational capacity. The threshold changes when costs, prevalence, calibration or alert budget changes. It also connects cost sensitive learning to the point an interviewer is testing.
6 min readUpdated 2026-08-15
Answer
Short answer
A classifier threshold is a business decision, not always 0.5. If probabilities are calibrated, act when expected benefit exceeds expected cost; with limited review capacity, choose the threshold that fills the alert budget. Revisit it when prevalence, costs, calibration or capacity changes.
Was this answer helpful?
Likely follow-ups
- Prevalence doubles overnight and your budget has not. What do you change and what do you tell the business?
- How do you set a threshold when the cost of a false positive is reputational and nobody will quantify it?
- Would you ever run two thresholds on one model, and what would that give you?
- How do you choose the threshold on the validation set without overfitting it?
Related questions
- What does it mean for a model to be calibrated, and why is an uncalibrated probability dangerous?mediumAlso on calibration and evaluation5 min
- Your fraud model is 99.4% accurate. Is that good?mediumAlso on class-imbalance and evaluation4 min
- Class weights, oversampling, undersampling, SMOTE. What does each one cost you?hardAlso on class-imbalance and calibration5 min
- When does ROC-AUC mislead you, and when does a precision-recall curve mislead you instead?mediumAlso on evaluation and class-imbalance4 min