Compute the baseline first
Before discussing the model at all, work out what doing nothing scores. If 0.6%
of transactions are fraudulent, a model that predicts "not fraud" unconditionally
is right 99.4% of the time. It has learned nothing, it catches no fraud, and it
beats or matches the number you were just given.
That single calculation is most of the answer, and volunteering it unprompted is
what the question is testing. Accuracy is the proportion of predictions that are
correct, so on a skewed problem it is dominated by the majority class and is
almost insensitive to performance on the class you care about.
The habit generalises beyond fraud: quote any metric alongside what the trivial
baseline scores on it, because a number without a floor under it means nothing.
The confusion matrix is the vocabulary
Every useful metric here is a ratio of four counts, and being fluent in them is
what makes the conversation precise.
| Predicted fraud | Predicted clean |
|---|
| Actually fraud | True positive | False negative — fraud you missed |
| Actually clean | False positive — a customer you blocked | True negative |
Precision is TP / (TP + FP): of the cases you flagged, how many were real.
It answers "how much of my team's time is wasted" and "how many honest customers
did I inconvenience".
Recall is TP / (TP + FN): of the real cases, how many you caught. It answers
"how much fraud got through".
They trade against each other, and the trade is not a property of the model —
it is a property of where you put the threshold.
Ask what happens to a flagged case
Which metric to optimise cannot be answered from the data. It follows from the
cost of each error, and that means asking what the system does with a positive.
If a flag blocks a transaction, a false positive is a customer whose
legitimate card was declined at a checkout. That is expensive in a way that does
not appear in any dataset: they call support, and some of them stop using you.
Precision matters, and the threshold should be high.
If a flag opens a case for a human reviewer, a false positive costs a few
minutes of analyst time. If a missed fraud costs the average chargeback, and
review costs a few pounds, the arithmetic favours recall heavily — until the
queue exceeds what the team can clear, at which point the binding constraint is
review capacity rather than the metric.
If a flag adds a verification step, the cost is friction rather than refusal,
and you can afford to be more aggressive still.
Same model, three different correct thresholds. A candidate who asks what
happens downstream before choosing is demonstrating the thing being examined.
The threshold is a separate decision
Most classifiers produce a score, and the label comes from comparing it to a
threshold. That threshold is not part of training and should not be left at its
default.
from sklearn.metrics import precision_recall_curve
precision, recall, thresholds = precision_recall_curve(y_true, y_scores)
viable = [(p, r, t) for p, r, t in zip(precision, recall, thresholds) if p >= 0.30]
best = max(viable, key=lambda row: row[1])
Reporting a single F1 score hides this. F1 is the harmonic mean of precision and
recall, which weights them equally — a choice, and usually the wrong one, since
the whole point is that the two errors cost different amounts. If you want one
number, F-beta lets you say how much more you care about recall than precision,
and stating the beta forces the conversation the single number was hiding.
Why ROC AUC misleads here
ROC AUC is the default in a great many notebooks and it is the wrong default
under heavy imbalance.
It plots the true positive rate against the false positive rate, and the false
positive rate has the number of true negatives in its denominator. With 99.4% of
cases negative, that denominator is enormous, so thousands of false positives
barely move it. A model can produce an impressive-looking ROC curve while the
cases it flags are overwhelmingly wrong.
The precision-recall curve has no true negatives in either axis, so it
degrades honestly as false positives accumulate. Under imbalance, quote PR AUC
and say why, and note that its baseline is the positive class rate rather than
0.5 — a PR AUC of 0.15 on a 0.6% problem is a real result, not a bad one.
Two things that follow
Resampling changes the numbers you report. Over-sampling the minority class
or under-sampling the majority is a legitimate training technique, and it alters
the base rate. Evaluate on data with the real distribution, or your precision is
a number from a world that does not exist.
The base rate moves. Fraud rates change with campaigns, seasons and
attackers. Precision depends directly on the base rate, so a model whose
precision falls may be unchanged while the world moved — which is why monitoring
the positive rate alongside the metric is what tells you whether to retrain or
to re-threshold.
Compute what predicting the majority class would score before you say anything
about the model. Then choose the metric from what a flag costs, because
nothing in the data can tell you that.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).