Split the system before debugging it
A RAG system is two systems: one finds passages, one writes an answer from them.
They fail differently and only one of them is usually at fault.
So the first move is not to look at the prompt. It is to ask a question with a
yes-or-no answer: was the passage containing the answer among the chunks
retrieved?
hits = 0
for question, answer_substring in gold:
chunks = retrieve(question, k=5)
if any(answer_substring in c.text for c in chunks):
hits += 1
print(f"retrieval recall@5: {hits}/{len(gold)}")
No model grades this and no judgement is involved. If recall is low, the
generator was never given the material and every hour spent on the prompt is
wasted. If recall is high and answers are still wrong, the problem genuinely is
generation, and now you know.
Teams skip this and tune prompts against a retrieval failure, which is the
single most common way months disappear in this area.
Chunking decides what can ever be found
Assuming retrieval is the problem, chunking is the usual cause, because the
chunk is the unit that gets embedded and returned. A boundary in the wrong place
destroys information before any search happens.
The answer straddles a boundary. A fixed 500-token split lands mid-procedure,
so half the steps are in one chunk and half in the next. Either chunk retrieved
alone is an incomplete answer, and the model confidently produces the half it
was given.
The chunk lost its context. A table row reading Standard | 14 days | £4.99 is meaningless without the heading three hundred tokens above it that
said what the table is about. Embedded alone, it matches almost nothing and
explains nothing when retrieved.
The chunk is too large. A 2,000-token chunk covering four topics has an
embedding that is the average of four things, which is close to none of them.
Similarity search rewards focus, so oversized chunks are quietly unfindable.
The mitigations are unglamorous and effective. Overlap — a couple of hundred
tokens shared between neighbours — means a straddling answer appears whole in at
least one chunk. Structure-aware splitting on headings, list boundaries and
paragraph breaks rather than a fixed token count keeps semantic units intact.
Contextual prefixes, where each chunk carries its document title and section
heading in the embedded text, fix the orphaned-table-row case directly.
Similarity is not relevance
The second retrieval failure is subtler: the right topic comes back and the
right passage does not.
Nearest-neighbour search returns what is closest in embedding space. That
correlates with relevance and is not the same thing. Embeddings capture topical
similarity well and struggle with the things that distinguish two passages about
the same topic — negation, a specific figure, a product name, a date.
Two standard responses:
Hybrid search. Combine dense vector search with a keyword method such as
BM25. Keyword search is excellent at exactly what embeddings are weak at:
product codes, error identifiers, proper nouns, anything where the literal token
matters. Merging the two candidate lists recovers a class of query that pure
vector search misses entirely.
Reranking. Retrieve a wider candidate set — twenty or fifty rather than five
— then score each candidate against the query with a cross-encoder that reads
the pair together rather than comparing two independently computed vectors. It
is slower per candidate, which is why it runs over a shortlist, and it is
usually the cheapest available quality improvement.
The questions RAG cannot answer
Worth raising unprompted, because it is the limit of the architecture rather
than a bug in an implementation.
Retrieval finds passages similar to the question. That works for "what is the
returns window for standard delivery". It does not work for aggregation —
"how many of our policies mention arbitration" — because no single passage
contains the answer and the model cannot count what it was not shown. It does
not work for whole-document reasoning, such as summarising a contract, where
the answer depends on everything rather than on the top five chunks. And it
handles negation and absence badly: "which products do not support this"
retrieves passages about products that do.
Recognising that a question is outside what retrieval can serve — and routing it
to a query against structured data instead — is a better answer than improving
the retriever.
Two operational things that are not the model
Permissions. An index built over a company's documents inherits none of the
access control those documents had. Filtering must happen at retrieval time
against the requesting user's entitlements, and it must happen before the chunks
reach the prompt. A great many internal projects stall here.
Freshness. A document that changed yesterday is answered from whatever was
indexed, and a document that was deleted is still answerable until it is removed
from the index. Deletion in particular is easy to omit, because nothing breaks —
the system simply keeps quoting a document that no longer exists.
Before touching the prompt, check whether the answer was in the retrieved
chunks at all. If it was not, the fault is upstream of anything a prompt can
reach, and it is usually where you drew the boundary.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).