Which metrics tell you whether retrieval is working?

Recall at k is the load-bearing number

The retriever's job is to put the answer-bearing passage inside the set of chunks the prompt will carry. If it does not, no prompt or model setting recovers it, which makes recall at k — the fraction of gold questions whose answer-bearing passage appears in the top k — the one metric causally linked to whether the product can work.

Two properties make it the place to start. It needs no model to grade: with a gold set labelled by a quoted answer string a hit is a substring check, so the number is deterministic and reproducible in a test suite. And k is not arbitrary; it is however many passages the prompt really carries after deduplication and truncation. Reporting recall at 20 when the prompt receives five measures a system you did not build, and it is a common way for a dashboard to look healthy while users are failed.

The rest of the retrieval metrics, and what each is for

Metric	The question it answers	Where it misleads
Recall at k	Did the passage reach the prompt at all	Says nothing about how much noise came with it
Precision at k	How much of the window was wasted	A low value is fine if the window is large
MRR	How high the first relevant passage ranked	Ignores everything after the first hit
NDCG at k	Is the ordering good, with graded labels	Needs graded judgements to mean anything

Precision at k matters more than people expect, and not because the model cannot ignore a bad passage. Every irrelevant chunk occupies window space, costs tokens on every request, and adds a plausible-looking source the model may cite, so a retriever reaching high recall by returning fifty near-duplicates has bought it with cost and confusion.

Mean reciprocal rank suits the case where there is essentially one correct passage per question and its position is what you care about, such as a "find me the clause" tool; it rewards getting one thing to the top and is blind to what followed. NDCG suits the case where several passages contribute in different degrees, since it discounts by rank and consumes graded relevance. Reporting NDCG over binary labels is a common error: it collapses to something less interpretable than recall while sounding more rigorous.

Whichever you choose, report it per query class rather than only in aggregate: an average over a mixed gold set hides the case where identifier queries collapsed while paraphrase queries improved, which is the commonest shape of a change that looks positive and feels negative. Two operational numbers belong beside these — the refusal rate on the deliberately unanswerable portion of the gold set, because a system that answers everything is worse than one that declines, and latency at the tail, since quality bought with a p99 nobody will wait for has not been bought.

Groundedness is a different measurement

None of the above says whether the final answer is right, and the answer is the product. So retrieval metrics are read next to two answer-level ones. Groundedness asks whether every claim is supported by the passages supplied — not whether it is true in general, which is a separate and harder question. Citation accuracy asks whether the cited passage contains the claim attached to it, which is the check users perform themselves and the one that destroys trust fastest.

Both are judgements rather than substring checks, so they are graded by humans on a sample or by a model prompted to verify each claim against the supplied context. A model grader must be validated before it is trusted: run it against a few dozen human-labelled examples and measure agreement first, because a grader nobody has calibrated produces a number that moves for reasons unrelated to the system.

When recall is high and the answers are still wrong

This combination is the diagnostic case, and reading it correctly is the thing being scored. If the answer-bearing passage was in the top k and the answer is wrong, retrieval has done its job and the fault is downstream. There are three places to look, in order.

The passage was retrieved but did not reach the model. Deduplication, a token budget, truncation or a post-retrieval filter can drop it between the retriever's output and the rendered prompt, and that code is rarely instrumented. Log the passage identifiers present in the prompt, not the ones the retriever returned; the two lists disagreeing is a bug nothing else reveals.

The passage reached the model and its position worked against it. Models are sensitive to where in a long context the relevant material sits, so a supporting passage buried among many others carries less weight than the same passage near the beginning or the end. Ordering what you pass is therefore a real intervention, and one your retrieval metrics are blind to because the set was identical.

The passage reached the model and the answer contradicts it. That is generation: prompt instructions, a model preferring its parametric knowledge to the supplied context, or passages that disagree with each other. Conflicting sources deserve separate treatment, because the right behaviour is to surface the conflict rather than pick a side, and no retrieval metric will tell you it happened.

Recall at k with k set to what the prompt really carries is the metric that decides whether the system can work. Every other number tells you how well it is working, or which stage to blame when it is not.