A user reports a wrong answer. How do you work out whether retrieval or generation failed?

Find the passage that should have been used

Before blaming any stage, establish what the right answer was and where it lives. Ask whoever reported the failure, or search the corpus by hand, until you have a specific document and a specific sentence that answers the question. That single step resolves a surprising share of reports, because the answer is frequently not in the corpus at all — the document was never ingested, it failed to parse, it was filtered out by a permission the user does not hold, or the content genuinely does not exist and someone assumed it did. None of those is a retrieval bug and all of them get reported as one.

If the sentence exists and is indexed, you now have a probe: a string that must appear in the chunks retrieved for this query. Everything after this is checking where it stopped.

Three gates, in order

Gate	What you check	Conclusion if it fails
1	Does the answer string appear in the retriever's top k for this query	Retrieval: chunking, index, query, or filters
2	Is that chunk present in the prompt as rendered and sent	The plumbing between the two stages
3	Does the answer contradict or ignore the passage that was present	Generation: prompt, model, or conflicting sources

The order matters because each gate presupposes the previous one. Debugging a prompt when gate one failed is the single most common way weeks disappear on a RAG project, and gate one is a substring check that takes a minute.

Gate one. Run the exact query as the user's request ran it — same filters, same user identity, same k — and look for the string. If it is absent, the fault is upstream of the model and the sub-diagnosis follows the shape of the miss. Search the index for the chunk directly: if no chunk contains the string, the document was never chunked in a way that preserved it, or was never indexed. If a chunk contains the string but does not rank, ask where it does rank — at 40 rather than 5 means a ranking problem a reranker can fix, whereas nowhere in the first few hundred means the query and the chunk simply do not match, which is a chunking, embedding or hybrid-retrieval problem. And check the filters: a scope, date or permission predicate that excludes the document produces a retrieval miss that looks like a relevance failure and is neither.

Gate two. This is the gate people do not know exists. Between the retriever's result list and the text sent to the model there is code: deduplication, a token budget, truncation of long passages, a minimum-score cut-off, reordering, formatting. Any of it can drop the chunk that gate one proved was retrieved. So the check is not "what did the retriever return" but "what was in the string we sent", and answering that requires having logged the rendered prompt or at least the passage identifiers it contained. When the two lists disagree, the bug is in that glue code, and it is usually a budget that silently discards the tail of the list — which means the passage ranked fifth of five is the first casualty and the failure is intermittent by construction.

Gate three. The passage was there and the answer is still wrong. Now read the answer against the passage and characterise the disagreement, because different disagreements have different causes. An answer that states something the passage denies suggests the model is drawing on parametric knowledge rather than the context, which is a prompt-instruction and model-choice problem. An answer that hedges or declines when the passage is explicit suggests the passage was buried among many others, since position within a long context affects how strongly material is used. An answer that is half right often means the passage was truncated at a boundary and the model faithfully used the half it received, which sends you back to chunking with a much better hypothesis than you started with. And where two retrieved passages disagree — an old policy and its replacement, both indexed — the model picking one is not a bug in the model; the bug is that both were passed with nothing to distinguish them, and the correct behaviour is either to filter the superseded one or to surface the conflict.

The diagnosis depends on what you logged

All of the above is cheap if the request left a trace and expensive if it did not, because reproducing a request from a screenshot means guessing the filters, the user's permissions, the index version and the model version, any of which changes the outcome.

{
  "request_id": "r_44f1",
  "query": "how long to return a priority item",
  "rewritten_query": "priority delivery returns window days",
  "filters": {"region": "emea", "effective_on": "2026-07-28"},
  "index_version": "v2",
  "retrieved": [{"chunk_id": "doc_8821:v3:p14:c2", "score": 0.71, "rank": 1}],
  "prompt_chunk_ids": ["doc_8821:v3:p14:c2"],
  "model": "generator-v4",
  "answer_hash": "a91f..."
}

The two fields that make the difference are retrieved and prompt_chunk_ids recorded separately, because gate two is precisely the comparison between them and no other artefact reveals it. The rewritten query matters for the same reason: if a rewriting step mangled the query, gate one fails for a reason that is invisible when you only logged what the user typed. Index and model versions matter because "it worked last week" is answerable only if you know what changed.

Closing the loop

A diagnosis that ends in a fix and nothing else will be repeated. Once you know the gate, add the case to the gold set as a labelled question with its answer quote, in the class it belongs to, so the specific failure becomes a permanent check. Over time the gold set accumulates precisely the questions your system is bad at, which is the most valuable fixture you can own and one you cannot buy.

It is also worth reporting the gate rather than the fix, because the distribution of gates across reports tells you where to invest. Many gate-one failures on literal queries argues for hybrid retrieval. Many gate-two failures argues that the plumbing needs tests rather than that the retriever needs work. Many gate-three failures with good passages argues for prompt and context-ordering work, and for a groundedness check that would have caught the answers before a user did.

Never debug the prompt before checking whether the passage was retrieved, and never trust the retriever's output as evidence of what the model received. The gap between those two lists is a real failure mode and the only thing that reveals it is logging both.