RAG, retrieval and agents fundamentals

A model knows only what its weights absorbed during training, and it has no way to tell you which of those things it absorbed reliably. Retrieval puts the specific text an answer depends on into the prompt at request time, so the answer is conditioned on documents you control, dated when you indexed them, and attributable to a source the user can open. Those are three separate wins — recency, private data and citation — and together they are why retrieval is almost always cheaper than any form of training for the same effect. The cost is that you now operate a search system in front of the model, with its own latency, its own failure modes and its own evaluation problem, none of which the model's own metrics will tell you about.

Because retrieval changes what is available, not what the model is obliged to use. The generation step still writes fluent text, and when the retrieved passages do not contain the answer it will produce something plausible from its weights in exactly the same voice as a grounded sentence. Nothing in the architecture forces a claim to trace back to a passage. You reduce fabrication by instructing the model to answer only from the supplied context and to say when it cannot, by requiring a citation per claim, and by checking afterwards that each cited span actually supports what was written. Only the last of those detects the failure rather than discouraging it, which is why it belongs in the pipeline and not in the prompt.

Each stage exists because the one before it cannot do that job, and each one can fail independently.

The rewrite exists because a user's query is often not a good search query — it carries pronouns from the previous turn, abbreviations the corpus spells out, and no keywords at all when it is a follow-up. Retrieval is hybrid because dense and lexical search fail on disjoint sets of queries, which the fusion item below works through. Reranking exists because retrieval must optimise for recall over a large candidate list, and recall in an unhelpful order is wasted on a generator that attends unevenly across a long prompt.

Assembly is the stage people leave out of the diagram and then debug for a week. It decides how many passages fit, in what order, whether neighbouring chunks are merged, whether near-duplicates are dropped, and what happens when the budget is exceeded. A passage that was retrieved and reranked correctly and then truncated mid-sentence is indistinguishable, from the answer, from one that was never found.

Verification closes the loop by checking the generated claims against the passages that were supplied, which is the only stage that can catch a confident answer built on nothing. Treat it as optional for latency reasons if you must, but log enough to run it offline, because otherwise you have no evidence about groundedness at all.

The consequence of the shape is that every quality question splits into "which stage" before it splits into "what fix". A system with one number for answer quality and no per-stage measurement cannot be improved except by guessing.

Training compresses a corpus into weights, so the facts end up entangled, unattributable and frozen at the training cut-off. Retrieval keeps the corpus as text, so a fact can be added, corrected, deleted or permission-checked in seconds. The practical distinction is who owns the update path. Anything that changes weekly, differs per customer, or must be shown to the user as a source belongs in retrieval. Anything that is general capability — language, reasoning, following a format — lives in the weights and cannot be retrieved into a model that lacks it. A candidate proposing fine-tuning to teach the model a set of facts has usually confused the two, and will discover that the model has learned the style of the documents rather than their content.

When the question is not answerable from a passage. Aggregations over structured data — how many, what is the total, which is largest — need a query against a database, and putting rows into a prompt as text means arithmetic performed by a language model over whichever sample it happened to retrieve. Retrieval is also wrong when the whole corpus fits comfortably in the context window, because you have added a search system whose only possible contribution is to lose recall you already had. And it is wrong when the task needs the entire document rather than the relevant part of it, such as summarising a contract or checking a policy for internal contradictions, where chunking destroys exactly the completeness the task requires.

Because they change what the user has to take on trust. Without a citation the only way to check an answer is to already know it, which makes the feature useful precisely where it is not needed. A link into the source passage turns an unverifiable assertion into a claim with evidence attached, and that is what makes the system usable in a domain where being wrong is expensive. The engineering consequence is that citation is a retrieval requirement rather than a formatting one: stable document identifiers and offsets must be carried from ingestion onwards, because you cannot reconstruct where a chunk came from after the fact. Retrofitting citations to a pipeline that discarded provenance means reingesting the corpus.

Mostly into passages, and that pressure shapes every other decision. A fixed budget divided by chunk size gives the number of independent sources you can pass, so larger chunks mean fewer of them and smaller chunks mean more but choppier ones. Instructions, conversation history and the space reserved for the output all draw on the same total, and history grows turn by turn while the passage allowance does not, so a long conversation quietly starves retrieval. When something must be dropped the choice is fewer sources or truncated ones, and truncating is the worse option because it tends to remove the exact sentence a citation points at. Deciding that policy explicitly, and logging when it fires, is worth more than any prompt wording.