How do you measure hallucination in a way you would put in front of a customer?

Two failures wearing one word

"Hallucination" is used for two things that need different measurements, different owners and different fixes, and conflating them is how a single reported percentage becomes indefensible under questioning.

The first is ungroundedness. The system was given sources and produced a claim those sources do not support. The judgement is closed-book: everything needed to decide it is in the provided passages, so it can be evaluated consistently, cheaply and at scale, and it is entirely within your control because you chose what to retrieve and how to instruct the model.

The second is factual error. The claim is wrong about the world. Deciding this requires knowledge outside the request, often a specialist, and sometimes there is no settled answer at all.

These come apart in both directions, which is why one number cannot cover them. An answer can be perfectly grounded and factually wrong, because your document is out of date or was wrong when written — the model behaved exactly as designed, and the defect is in the corpus. And an answer can be factually correct yet ungrounded, because the model supplied something true from its parameters that no retrieved passage supported. That second case is the dangerous one to score as a pass: the behaviour that produced a correct unsupported claim is the same behaviour that produces an incorrect one, and you have rewarded it.

So report two rates, with two names, and say which one you are talking about every time. Groundedness is the number you can move this quarter. Factual accuracy is the number the customer cares about, and it is bounded above by the quality of your corpus.

Measure claims, not answers

An answer of six sentences is not one unit of truth. Marking a whole response as hallucinated because one clause was unsupported destroys most of the signal and makes the metric hypersensitive to answer length; marking it as clean because the substance was right hides the clause that will cause the incident.

Decompose each answer into atomic claims — single verifiable assertions — and adjudicate each against the sources. The rate is then unsupported claims over total claims, which is stable against length and tells you severity rather than mere incidence. Be honest that decomposition is itself a lossy, judgement-laden step: a decomposer will split some claims too finely and merge others, and it tends to drop hedges and attributions, so "the report suggests revenue may have fallen" can be flattened into a bare assertion that then scores as unsupported. Fix the decomposition procedure, review a sample of it by hand, and keep it constant between measurements, because a change to the decomposer moves every number you have.

Two derived metrics are worth reporting next to it. Citation precision asks whether each cited passage genuinely supports the claim attached to it, which catches the very common failure of a plausible citation to a passage about something adjacent. Citation recall asks whether every claim that needed support carried a citation at all.

Silence is not accuracy

A system that answers "I cannot find this in the available documents" every time has a zero ungroundedness rate and no value whatsoever. Any hallucination figure published without a coverage figure beside it is therefore uninterpretable, and it is the easiest way to appear to have improved: tighten the refusal threshold, watch the rate fall, ship the announcement.

Report the triple for a fixed set of queries: the proportion answered and grounded, the proportion answered with at least one unsupported claim, and the proportion abstained. Then split the abstentions by whether the answer was in fact available in the corpus, because an abstention on retrievable material is a failure of a different kind — an unhelpfulness that users experience as the product not working. The trade-off between the two is a product decision that belongs to whoever owns the feature, and it can only be made if both sides are visible.

Attribute the failure to the right stage

When an answer is unsupported, the next question is whether the passages contained the answer at all. If retrieval returned nothing relevant and the model filled the gap, there are two defects: a retrieval miss, and a generation that should have abstained. If retrieval returned the answer and the model contradicted it, the generation stage owns it alone.

Recording that attribution changes what you do next, and it also stops the metric from being blamed on the wrong team. A pipeline where most unsupported claims trace to retrieval misses will not be improved by any amount of prompt work on the generator, and a pipeline where the evidence was present and ignored will not be improved by a better reranker.

How to grade it at scale without fooling yourself

Human annotation on a stratified sample is the anchor. Write a rubric that defines supported, contradicted, and not-addressed as distinct verdicts, have two annotators overlap on a portion of it, and report their agreement, because a metric produced by graders who agree two-thirds of the time has an error bar wider than most of the changes you will make.

Automated grading then scales it, by entailment checking or a judge model, but the setup detail decides whether it measures anything. The grader must see only the claim and the retrieved passages when judging groundedness. Give it the question and let it reason freely and it will use its own world knowledge, at which point it is silently scoring factuality instead — the exact conflation you built two metrics to avoid. Validate the automated grader against the human labels and publish that agreement with the results.

Finally, state the denominator. A hallucination rate is a property of a system and a query distribution jointly, and it moves by a large factor between a set of well-covered factual lookups and a set of multi-hop comparative questions about thinly documented edge cases. Keep a fixed, versioned query set for trending over time so the series means something, break the results down by query category, and refuse to quote a single headline figure without saying which set produced it.

The one number nobody can argue with does not exist. Two rates, a coverage figure and a named query set is the honest form, and it is also the form that tells you what to fix.