Your fraud score holds up a claim from a customer of eleven years, the investigation finds nothing, and settlement is six weeks late. What did that cost you, and what should the design have done differently?
6 min readUpdated 2026-08-15Target archetype: Enterprise Captive, Product Startup
0
Answer
Short answer
It cost a customer, a complaint, and a training example you have mislabelled as clean. The fix is to keep the score ranking an investigation queue rather than gating a payment, to time-box every referral with an interim payment path, and to record why a referral was cleared instead of merely that it was.
Was this answer helpful?
Likely follow-ups
→How would you choose the threshold if somebody handed you the cost of a missed fraud and the cost of a delayed honest claim?
→What would you insist is recorded when a referral is cleared, so the model can learn something from it?
→One postcode band turns out to carry much of the model's signal. What do you do about it?
→How do you stop the score becoming the reason a handler declines, when it was only ever meant to order a queue?
The investigator hours are the smallest item on the bill, and a candidate who stops there has answered a resourcing question rather than a design one.
The customer has been treated as a suspect for six weeks over a claim that was honest, at the point in the relationship where they were finally using the product they had paid for over eleven years. Some proportion of customers in that position leave, and the ones who do are disproportionately the profitable long-tenure renewals your book depends on. Some proportion complain, and a complaint about delay on a claim that was ultimately paid in full is a complaint you will struggle to defend, because the substance of it is true.
Insurers are generally under an obligation to handle claims promptly and to give reasons for a decision, and a six-week hold produced by a model score is difficult to justify as prompt when the eventual answer was to pay.
There is also a real cost in the claim itself. Delay makes claims more expensive: the hire car runs longer, the building dries out more slowly, the injury goes untreated, the legal costs accrue. And there is a cost you cannot see, which is the next honest customer whose claim the same threshold will hold up.
A cleared referral is not a clean label
The more insidious cost is to the model. An investigation that finds nothing has not established that the claim was honest; it has established that nobody found evidence in the time available. If that outcome is written back as a negative label, you are teaching the model that the pattern it correctly identified as unusual is normal, and its future performance on exactly that pattern degrades. Do it consistently and the model gradually learns the shape of your own investigative capability rather than the shape of fraud.
So the outcome recorded has to be richer than a binary. Distinguish confirmed fraud from a withdrawn or abandoned claim, from a claim settled below the amount presented, from a referral closed for lack of evidence, and from a referral closed because nobody had time. Each of those means something different, and only the first is a positive label. The rest are censored observations, and treating them honestly is the difference between a model that is evaluated and one that is merely scored.
The score orders a queue, it does not gate a payment
This is the architectural point the scenario is really about. A score is a ranking device: it tells you which of today's claims deserve a human's attention first, given that you can only look at a fixed number of them. It is not a finding, and it must not sit anywhere in the system where a payment cannot proceed without it clearing.
Wire it that way and two things follow. First, the score's job becomes capacity allocation, so the operating question is how many referrals the investigation team can genuinely work in a day, and the threshold is set to fill that capacity rather than chosen for a statistic. Second, the decision to delay a payment becomes a separate, human, recorded act with its own justification, which is what makes it defensible to a regulator, an ombudsman or a court. A design where the model's output is the reason for the delay has put an unexplainable artefact in the causal chain of a decision you are obliged to explain.
Loading diagram…
The branch that matters is the one from the investigation back to an interim payment, because a design without it has only one way to express suspicion, and that way is to make the customer wait.
Time-box the referral and keep paying where you can
Two mechanisms remove most of the harm in this scenario without weakening detection. The first is a hard limit on how long a referral can sit before it either produces a substantiated reason to continue or returns to the normal path, with the clock visible to somebody senior rather than buried in a queue. Referrals do not usually fail by being wrong; they fail by being forgotten, and the six weeks in the question is far more likely to be three days of work and five weeks of waiting.
The second is separating the parts of the claim that are genuinely in question from the parts that are not. Where cover plainly applies and only the quantum of one head of loss is doubted, the rest can be paid. Where liability is not in dispute at all, an interim payment can go out while the investigation continues. This is ordinary claims practice, and its absence is usually a systems limitation rather than a policy one, because a payment engine that can only settle a claim in full leaves the handler with no option between paying everything and paying nothing.
What the customer is told
You cannot tell a customer their claim was flagged by a model, both because it is not a reason and because saying so invites the next question you cannot answer. What you can tell them is what you require, by when, and why it is relevant: a document, an engineer's inspection, a repairer's report. That converts an unexplainable delay into a specific request with an end, which is a very different customer experience and a very different complaint file.
That requirement, incidentally, disciplines the model design. If a referral cannot be turned into a concrete, articulable request for information, it should probably not have been raised, because nothing about it can be progressed or explained. The strongest referral queues therefore surface the evidence alongside the score, and a score with no accompanying feature the investigator can act on is a defect in the pipeline rather than a hard case.
Where the threshold comes from
Nobody can set a threshold from a precision and recall curve alone, because the two errors are not commensurable. The way to make the argument is to put both costs on the same footing and reason about the ratio openly: the expected saving from catching a fraudulent claim of a given size against the expected cost of delaying an honest one, including retention, complaint handling, and the extra claim cost that delay itself creates. You will not get precise figures, and that is not the point. The point is that a threshold argued from that comparison can be defended and revisited, while a threshold chosen because it looked reasonable on a validation set will drift until somebody complains loudly enough to move it.
The cost of a false positive in fraud is paid by a customer who did nothing wrong and recorded as evidence that your model was too cautious, and both of those consequences land somewhere no one is measuring unless you have deliberately built the measurement.