One transaction in two thousand is fraudulent and you have been asked to build the detector. How do you approach it?
Work backwards from the review capacity. Fix the alert budget the team can clear, evaluate precision and recall at exactly that budget on data at the real prevalence, define the label and its arrival delay before modelling, and never report accuracy, which a do-nothing model scores 99.95% on.
5 min readUpdated 2026-08-15
0
Answer
Short answer
Work backwards from the review capacity.95% on.
Was this answer helpful?
Likely follow-ups
→Chargebacks take months to confirm. What do you use as a label in the meantime?
→Blocked transactions never generate an outcome. How does that corrupt next quarter's training set?
→The review team doubles in size. What changes, and what does not?
→How would you decide between one model and a fast filter feeding a slower model?
A rare-event detector is a capacity problem wearing a modelling problem's clothes, so start from the constraint that will not move. Suppose four million transactions a month at a prevalence of one in two thousand, which is 2,000 fraudulent transactions to find. The review team is five analysts, each clearing a case in about four minutes, working roughly six productive hours a day over twenty working days.
cases per analyst perday=6 x 60/4=90
team capacity perday=5 x 90=450
team capacity permonth=450 x 20=9,000
So the system may raise 9,000 alerts a month against 2,000 real frauds, and every design decision follows from that ratio. Even a perfect model would leave the team with 7,000 spare slots; a realistic one will fill them with false positives. If the model catches 900 of the 2,000 at that budget, recall is 45% and precision is 900/9,000 = 10%, meaning nine of every ten cases an analyst opens is innocent. That is the sentence to put in front of the fraud manager, because it is the one they can act on.
The number also tells you when to stop tuning. Moving PR-AUC by 0.01 is worth nothing if the alert budget is unchanged and the marginal alerts were already below the value of an analyst's four minutes.
Accuracy is arithmetically disqualified
A model that predicts "not fraud" for every transaction is correct on 3,998,000 of 4,000,000 rows, which is 99.95%. It catches nothing, costs nothing to build, and beats most first attempts on that metric. Quote that figure once, early, and the metric conversation is over.
The same reasoning disqualifies any single threshold-free summary as the headline. Report precision and recall at the 9,000-alert budget, report the recall you would gain from a 12,000 budget so the capacity conversation is informed, and use average precision or PR-AUC only to compare candidate models against one another.
The label is the hard part and it arrives late
Before any modelling, settle what a positive is. Confirmed chargeback, analyst-confirmed fraud, customer-reported fraud and rule-blocked transaction are four different labels with different populations and different delays, and a model trained on one will be evaluated against another unless somebody writes it down.
Delay is the operational constraint. Confirmed fraud outcomes can take weeks or months to resolve, so the most recent data has an artificially low positive rate simply because its labels have not matured. Training on a window that includes immature labels teaches the model that recent transactions are safe. Two defences: exclude any period whose labels are not yet mature from both training and evaluation, and track the label maturation curve so you know how much of a recent period's positives are still outstanding.
Where a fast proxy exists, use it as a leading indicator for monitoring rather than as a training label, and be explicit that it is a proxy.
The pipeline is not one model
At four million transactions a month with a latency budget, a two-stage shape is usually right, and the stages are graded differently.
Loading diagram…
The interesting edge is the arrow from the analyst outcome back into training: only queued cases ever get an outcome, so the labelled data drifts towards whatever the current model already flags.
The first stage exists to be cheap and to preserve recall, so it is tuned to discard the obviously clean majority with a very high recall threshold and is evaluated on how much of the population it removes at a fixed recall loss. The second stage ranks the survivors and is evaluated on precision at the budget. Deterministic rules sit in front of both because some patterns are policy rather than prediction, and a rule that blocks a known compromised card should not be at the mercy of a score.
The feedback loop that eats your labels
Once the system is live, the only cases with outcomes are the ones it selected. Blocked transactions never complete, so they have no genuine label at all, and un-alerted transactions are recorded as clean when many of them were undetected fraud. Train the next model on that data and it learns the current model's decision boundary rather than fraud.
Three mitigations are worth naming. Hold out a small randomised sample that bypasses the model entirely and is reviewed regardless, which is expensive and is the only source of unbiased prevalence and unbiased negatives. Record the score and the decision for every transaction so the selection is at least observable, allowing you to reweight. And treat blocked cases as censored rather than as positive or negative, because assuming a block was correct is assuming the conclusion.
Evaluate on the world, not on the training distribution
Resampling and class weighting are legitimate training devices and neither changes what you must evaluate on. If you undersample negatives to a 1:1 ratio for training, the model's scores now reflect a 50% prior and its precision measured on that sample is a number from a world where half of all transactions are fraud. Recalibrate the scores back to the true prior, then measure precision and recall on a held-out sample at the real 1-in-2,000 prevalence, split forward in time.
This is the point where most otherwise-good answers fail, and it fails quietly: the reported precision of 0.72 is arithmetically correct on the resampled evaluation set and bears no relationship to what the analysts will see. The rule is simple enough to state as a constraint. Training distribution is yours to choose; evaluation distribution is not.
Monitoring after launch
Watch the alert volume and the positive rate as first-class signals alongside precision, because precision falls when prevalence falls even though the model is unchanged, and distinguishing those two situations decides whether you retrain or re-threshold. Watch the queue depth, since a model that becomes more sensitive silently converts into a worse model when the team cannot clear the backlog and cases age out. And watch the score distribution rather than only the metrics, since it moves before the labels arrive and is often the earliest warning available on a problem where labels are months behind.
Fix the alert budget from the team's real capacity, then report precision and recall at exactly that budget on data at the true prevalence. Every other number on a rare-event problem is either flattering or irrelevant.