hardScenarioDesignMidSeniorStaff
Your RAG system cannot answer a question whose answer is definitely in the documents. Where is the fault?
4 min readUpdated 2026-08-15
Answer
Short answer
Almost always in retrieval rather than generation, and most often in chunking: if the answer was split across a boundary, or sits in a chunk whose surrounding context was stripped, no prompt can recover it.
Was this answer helpful?
Likely follow-ups
- How would you measure whether retrieval is the problem, in an afternoon?
- Your chunks are 500 tokens with no overlap. What does that lose?
- Nearest neighbour returns the right topic and the wrong passage. What now?
- When is RAG the wrong tool for the question being asked?
Related questions
- How do you build a gold set for a retrieval system?mediumAlso on evaluation and retrieval5 min
- How do you choose a chunking strategy for a document corpus?mediumAlso on chunking and retrieval4 min
- What does a text embedding encode, and what does it fail to encode?mediumAlso on embeddings and retrieval4 min
- You are building a support assistant over a knowledge base that changes every week. RAG or fine-tuning, and how do you decide?mediumAlso on rag and evaluation5 min