Your RAG system returns confident but wrong answers. How do you debug it?

Bisect before you fix

A confidently wrong answer has exactly two possible origins, and they demand opposite remedies. Either the passage containing the correct fact never reached the model, in which case the model was doing its best with what it had, or the passage did reach the model and the model ignored, misread, or embellished it. Every hour spent tuning prompts when the real problem is that the relevant chunk was never in the top-k is wasted, and the reverse is equally true. So the first action is not a change at all: it is an inspection.

Take twenty or thirty of the observed bad answers and, for each one, log the full retrieved context alongside the answer. Then ask one question per case: was the correct passage present in that context? Answering this requires you to know what the correct passage is, which is why you need a small set of queries with a known gold source document. The split you get from this exercise is the whole diagnosis. If the gold passage is absent in most cases you have a retrieval problem. If it is present and the answer is still wrong you have a generation problem. A useful confirmation is the oracle test: paste the gold passage into the context by hand and re-run. If the answer becomes correct, retrieval is the bottleneck and nothing else needs attention yet.

When retrieval is the problem

Chunking is the most common culprit and the least glamorous. Fixed-size splits cut definitions away from their subject, orphan tables from their headers, and strand a pronoun in one chunk while its referent sits in the previous one. A chunk that is not self-contained cannot be retrieved by a query about the thing it describes. The fix is structure-aware splitting on headings or clause boundaries, plus small overlaps, and often prepending the document title and section path to each chunk so it carries its own context.

Next is the mismatch between how questions are phrased and how documents are written. A short interrogative query and a long declarative paragraph occupy different regions of embedding space, and a general-purpose embedding model has never seen your domain's acronyms, part numbers, or internal product names. This is where a purely dense retriever fails hardest: it will happily return topically similar text while missing the one chunk that contains the exact error code the user typed. Adding lexical retrieval, typically BM25, and fusing the two result lists is the highest-value single change for that class of failure, because exact-token matching is precisely what dense vectors are bad at.

Then reranking. Bi-encoder similarity is an approximation chosen for speed; it scores the query and the document independently and never lets them interact. A cross-encoder reranker reads the pair together and reorders a larger candidate set, so you retrieve broadly at k of fifty or a hundred and hand the model only the top few after reranking. Without it, top-k tuning is a trap in both directions: too small and the gold passage falls just outside the cut, too large and you flood the context with plausible near-misses that actively mislead the generator.

Finally, look at the corpus. Stale documents, three versions of the same runbook, and a deprecated policy that still ranks well will produce a confidently wrong answer from a retrieval pipeline that is working perfectly. Deduplication, versioning, and metadata filters on recency and tenancy belong in the debugging checklist, not below it.

When generation is the problem

If the gold passage was in context and the answer is still wrong, work through a shorter list. The prompt probably never told the model it is allowed to refuse. A model given a question and some documents will answer; abstention has to be granted explicitly, with an instruction to say it cannot determine the answer when the context does not support one, and ideally with few-shot examples of doing so. Second, position matters: attention to material in the middle of a long context is measurably weaker than to material at the start or end, so a gold passage buried at rank eight of twelve may as well not be there. Fewer, better-ordered chunks beat more of them. Third, require inline citations to chunk identifiers and validate them programmatically. This does not just help the user trust the answer, it converts silent fabrication into a detectable event, because a claim with no citation or a citation to a chunk that does not contain it is a defect you can count.

Measuring it: two scoreboards, never one

You cannot manage this with a single quality score, because a blended number cannot tell you which half to fix. Build a labelled set of a few hundred queries, each with the gold source chunks and a reference answer, and deliberately include unanswerable questions where the corpus genuinely lacks the fact. Then score two things separately.

from statistics import mean

# Retrieval: did the gold chunk make the cut? Independent of the LLM entirely.
recall_at_k = mean(
    any(g in retrieved[q][:k] for g in gold_ids[q]) for q in queries
)

# Generation: is every claim in the answer entailed by the context that was
# actually supplied? Note it is scored against retrieved context, NOT the truth.
faithfulness = mean(
    supported_claims[a] / total_claims[a] for a in answers
)

# Correctness is a third, separate number: answer vs reference answer.
# Faithfulness high + correctness low == retrieval fed it the wrong document.

Recall@k, and nDCG or MRR if rank position matters, tell you whether the retriever is doing its job. Faithfulness or groundedness tells you whether the generator stayed inside its evidence. Answer correctness against the reference tells you whether the pair got the user what they needed. Add abstention accuracy on the unanswerable slice, since a system that never says "I don't know" is the direct cause of confident wrongness. With these in place, every subsequent change becomes an experiment with a measurable outcome instead of an opinion.

The trap

The trap is treating "confident but wrong" as a synonym for hallucination and going straight to the prompt. Most of these failures are retrieval failures, and a meaningful share are corpus failures where the model was perfectly faithful to a document that was outdated or contradicted elsewhere. That is why faithfulness and correctness must be separate metrics: a system can score close to perfect on faithfulness and still be wrong on every answer, and if you only track one of them you will conclude the generator is fine and keep tuning the wrong layer. Candidates who name that gap are demonstrating they have actually operated one of these systems.

Never change a RAG pipeline before you know whether the right passage was in the context, and never trust a single quality score to tell you.