Your RAG system cannot answer a question whose answer is definitely in the documents. Where is the fault?

Split the system before debugging it

A RAG system is two systems: one finds passages, one writes an answer from them. They fail differently and only one of them is usually at fault.

So the first move is not to look at the prompt. It is to ask a question with a yes-or-no answer: was the passage containing the answer among the chunks retrieved?

# The cheapest useful evaluation in this whole area.
hits = 0
for question, answer_substring in gold:
    chunks = retrieve(question, k=5)
    if any(answer_substring in c.text for c in chunks):
        hits += 1
print(f"retrieval recall@5: {hits}/{len(gold)}")

No model grades this and no judgement is involved. If recall is low, the generator was never given the material and every hour spent on the prompt is wasted. If recall is high and answers are still wrong, the problem genuinely is generation, and now you know.

Teams skip this and tune prompts against a retrieval failure, which is the single most common way months disappear in this area.

Chunking decides what can ever be found

Assuming retrieval is the problem, chunking is the usual cause, because the chunk is the unit that gets embedded and returned. A boundary in the wrong place destroys information before any search happens.

The answer straddles a boundary. A fixed 500-token split lands mid-procedure, so half the steps are in one chunk and half in the next. Either chunk retrieved alone is an incomplete answer, and the model confidently produces the half it was given.

The chunk lost its context. A table row reading Standard | 14 days | £4.99 is meaningless without the heading three hundred tokens above it that said what the table is about. Embedded alone, it matches almost nothing and explains nothing when retrieved.

The chunk is too large. A 2,000-token chunk covering four topics has an embedding that is the average of four things, which is close to none of them. Similarity search rewards focus, so oversized chunks are quietly unfindable.

The mitigations are unglamorous and effective. Overlap — a couple of hundred tokens shared between neighbours — means a straddling answer appears whole in at least one chunk. Structure-aware splitting on headings, list boundaries and paragraph breaks rather than a fixed token count keeps semantic units intact. Contextual prefixes, where each chunk carries its document title and section heading in the embedded text, fix the orphaned-table-row case directly.

Similarity is not relevance

The second retrieval failure is subtler: the right topic comes back and the right passage does not.

Nearest-neighbour search returns what is closest in embedding space. That correlates with relevance and is not the same thing. Embeddings capture topical similarity well and struggle with the things that distinguish two passages about the same topic — negation, a specific figure, a product name, a date.

Two standard responses:

Hybrid search. Combine dense vector search with a keyword method such as BM25. Keyword search is excellent at exactly what embeddings are weak at: product codes, error identifiers, proper nouns, anything where the literal token matters. Merging the two candidate lists recovers a class of query that pure vector search misses entirely.

Reranking. Retrieve a wider candidate set — twenty or fifty rather than five — then score each candidate against the query with a cross-encoder that reads the pair together rather than comparing two independently computed vectors. It is slower per candidate, which is why it runs over a shortlist, and it is usually the cheapest available quality improvement.

The questions RAG cannot answer

Worth raising unprompted, because it is the limit of the architecture rather than a bug in an implementation.

Retrieval finds passages similar to the question. That works for "what is the returns window for standard delivery". It does not work for aggregation — "how many of our policies mention arbitration" — because no single passage contains the answer and the model cannot count what it was not shown. It does not work for whole-document reasoning, such as summarising a contract, where the answer depends on everything rather than on the top five chunks. And it handles negation and absence badly: "which products do not support this" retrieves passages about products that do.

Recognising that a question is outside what retrieval can serve — and routing it to a query against structured data instead — is a better answer than improving the retriever.

Two operational things that are not the model

Permissions. An index built over a company's documents inherits none of the access control those documents had. Filtering must happen at retrieval time against the requesting user's entitlements, and it must happen before the chunks reach the prompt. A great many internal projects stall here.

Freshness. A document that changed yesterday is answered from whatever was indexed, and a document that was deleted is still answerable until it is removed from the index. Deletion in particular is easy to omit, because nothing breaks — the system simply keeps quoting a document that no longer exists.

Before touching the prompt, check whether the answer was in the retrieved chunks at all. If it was not, the fault is upstream of anything a prompt can reach, and it is usually where you drew the boundary.