hardScenarioDesignMidSeniorStaff
Your RAG system returns confident but wrong answers. How do you debug it?
Debug a RAG system with confident wrong answers by separating retrieval failure from generation failure. Check whether the gold passage was retrieved, then measure recall@k, reranking quality, faithfulness and abstention on a labelled eval set. It also connects retrieval evaluation to the point an interviewer is testing.
5 min readUpdated 2026-08-15
Answer
Short answer
To debug confident wrong RAG answers, first inspect whether the correct source passage was retrieved. If it was absent, fix chunking, hybrid retrieval, metadata filters or reranking. If it was present, fix generation with better context ordering, abstention, citations and faithfulness checks. Score retrieval and generation separately.
Was this answer helpful?
Likely follow-ups
- How would you build the labelled eval set when you have no annotated queries at all?
- Where exactly would you put a reranker, and what latency and cost does it add?
- How do you handle two retrieved passages that contradict each other?
- What would make you conclude that RAG is the wrong architecture and you need fine-tuning or an agentic tool call instead?
Related questions
- Your RAG assistant cites the wrong document, and sounds completely sure of itself. How do you find which stage failed?hardAlso on rag and retrieval-evaluation5 min
- You are building a support assistant over a knowledge base that changes every week. RAG or fine-tuning, and how do you decide?mediumAlso on rag5 min
- Your RAG system cannot answer a question whose answer is definitely in the documents. Where is the fault?hardAlso on rag4 min
- A better embedding model has come out. What does moving to it cost you?hardAlso on rag5 min