Your fraud model is 99.4% accurate. Is that good?

Compute the baseline first

Before discussing the model at all, work out what doing nothing scores. If 0.6% of transactions are fraudulent, a model that predicts "not fraud" unconditionally is right 99.4% of the time. It has learned nothing, it catches no fraud, and it beats or matches the number you were just given.

That single calculation is most of the answer, and volunteering it unprompted is what the question is testing. Accuracy is the proportion of predictions that are correct, so on a skewed problem it is dominated by the majority class and is almost insensitive to performance on the class you care about.

The habit generalises beyond fraud: quote any metric alongside what the trivial baseline scores on it, because a number without a floor under it means nothing.

The confusion matrix is the vocabulary

Every useful metric here is a ratio of four counts, and being fluent in them is what makes the conversation precise.

	Predicted fraud	Predicted clean
Actually fraud	True positive	False negative — fraud you missed
Actually clean	False positive — a customer you blocked	True negative

Precision is TP / (TP + FP): of the cases you flagged, how many were real. It answers "how much of my team's time is wasted" and "how many honest customers did I inconvenience".

Recall is TP / (TP + FN): of the real cases, how many you caught. It answers "how much fraud got through".

They trade against each other, and the trade is not a property of the model — it is a property of where you put the threshold.

Ask what happens to a flagged case

Which metric to optimise cannot be answered from the data. It follows from the cost of each error, and that means asking what the system does with a positive.

If a flag blocks a transaction, a false positive is a customer whose legitimate card was declined at a checkout. That is expensive in a way that does not appear in any dataset: they call support, and some of them stop using you. Precision matters, and the threshold should be high.

If a flag opens a case for a human reviewer, a false positive costs a few minutes of analyst time. If a missed fraud costs the average chargeback, and review costs a few pounds, the arithmetic favours recall heavily — until the queue exceeds what the team can clear, at which point the binding constraint is review capacity rather than the metric.

If a flag adds a verification step, the cost is friction rather than refusal, and you can afford to be more aggressive still.

Same model, three different correct thresholds. A candidate who asks what happens downstream before choosing is demonstrating the thing being examined.

The threshold is a separate decision

Most classifiers produce a score, and the label comes from comparing it to a threshold. That threshold is not part of training and should not be left at its default.

from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_true, y_scores)

# Pick the operating point from the constraint, not from a formula.
# Here: the highest recall we can reach while keeping precision at 0.30,
# because the review team can absorb roughly three false positives per catch.
viable = [(p, r, t) for p, r, t in zip(precision, recall, thresholds) if p >= 0.30]
best = max(viable, key=lambda row: row[1])

Reporting a single F1 score hides this. F1 is the harmonic mean of precision and recall, which weights them equally — a choice, and usually the wrong one, since the whole point is that the two errors cost different amounts. If you want one number, F-beta lets you say how much more you care about recall than precision, and stating the beta forces the conversation the single number was hiding.

Why ROC AUC misleads here

ROC AUC is the default in a great many notebooks and it is the wrong default under heavy imbalance.

It plots the true positive rate against the false positive rate, and the false positive rate has the number of true negatives in its denominator. With 99.4% of cases negative, that denominator is enormous, so thousands of false positives barely move it. A model can produce an impressive-looking ROC curve while the cases it flags are overwhelmingly wrong.

The precision-recall curve has no true negatives in either axis, so it degrades honestly as false positives accumulate. Under imbalance, quote PR AUC and say why, and note that its baseline is the positive class rate rather than 0.5 — a PR AUC of 0.15 on a 0.6% problem is a real result, not a bad one.

Two things that follow

Resampling changes the numbers you report. Over-sampling the minority class or under-sampling the majority is a legitimate training technique, and it alters the base rate. Evaluate on data with the real distribution, or your precision is a number from a world that does not exist.

The base rate moves. Fraud rates change with campaigns, seasons and attackers. Precision depends directly on the base rate, so a model whose precision falls may be unchanged while the world moved — which is why monitoring the positive rate alongside the metric is what tells you whether to retrain or to re-threshold.

Compute what predicting the majority class would score before you say anything about the model. Then choose the metric from what a flag costs, because nothing in the data can tell you that.