RAG, retrieval and agents fundamentals

A model knows only what its weights absorbed during training, and it has no way to tell you which of those things it absorbed reliably. Retrieval puts the specific text an answer depends on into the prompt at request time, so the answer is conditioned on documents you control, dated when you indexed them, and attributable to a source the user can open. Those are three separate wins — recency, private data and citation — and together they are why retrieval is almost always cheaper than any form of training for the same effect. The cost is that you now operate a search system in front of the model, with its own latency, its own failure modes and its own evaluation problem, none of which the model's own metrics will tell you about.

Because retrieval changes what is available, not what the model is obliged to use. The generation step still writes fluent text, and when the retrieved passages do not contain the answer it will produce something plausible from its weights in exactly the same voice as a grounded sentence. Nothing in the architecture forces a claim to trace back to a passage. You reduce fabrication by instructing the model to answer only from the supplied context and to say when it cannot, by requiring a citation per claim, and by checking afterwards that each cited span actually supports what was written. Only the last of those detects the failure rather than discouraging it, which is why it belongs in the pipeline and not in the prompt.

Each stage exists because the one before it cannot do that job, and each one can fail independently.

flowchart LR
    Q[User query] --> RW[Rewrite and embed]
    RW --> R[Hybrid retrieve<br/>dense plus lexical]
    R --> RR[Rerank candidates]
    RR --> A[Assemble prompt<br/>within budget]
    A --> G[Generate with citations]
    G --> V[Verify claims<br/>against passages]

The rewrite exists because a user's query is often not a good search query — it carries pronouns from the previous turn, abbreviations the corpus spells out, and no keywords at all when it is a follow-up. Retrieval is hybrid because dense and lexical search fail on disjoint sets of queries, which the fusion item below works through. Reranking exists because retrieval must optimise for recall over a large candidate list, and recall in an unhelpful order is wasted on a generator that attends unevenly across a long prompt.

Assembly is the stage people leave out of the diagram and then debug for a week. It decides how many passages fit, in what order, whether neighbouring chunks are merged, whether near-duplicates are dropped, and what happens when the budget is exceeded. A passage that was retrieved and reranked correctly and then truncated mid-sentence is indistinguishable, from the answer, from one that was never found.

Verification closes the loop by checking the generated claims against the passages that were supplied, which is the only stage that can catch a confident answer built on nothing. Treat it as optional for latency reasons if you must, but log enough to run it offline, because otherwise you have no evidence about groundedness at all.

The consequence of the shape is that every quality question splits into "which stage" before it splits into "what fix". A system with one number for answer quality and no per-stage measurement cannot be improved except by guessing.

Training compresses a corpus into weights, so the facts end up entangled, unattributable and frozen at the training cut-off. Retrieval keeps the corpus as text, so a fact can be added, corrected, deleted or permission-checked in seconds. The practical distinction is who owns the update path. Anything that changes weekly, differs per customer, or must be shown to the user as a source belongs in retrieval. Anything that is general capability — language, reasoning, following a format — lives in the weights and cannot be retrieved into a model that lacks it. A candidate proposing fine-tuning to teach the model a set of facts has usually confused the two, and will discover that the model has learned the style of the documents rather than their content.

When the question is not answerable from a passage. Aggregations over structured data — how many, what is the total, which is largest — need a query against a database, and putting rows into a prompt as text means arithmetic performed by a language model over whichever sample it happened to retrieve. Retrieval is also wrong when the whole corpus fits comfortably in the context window, because you have added a search system whose only possible contribution is to lose recall you already had. And it is wrong when the task needs the entire document rather than the relevant part of it, such as summarising a contract or checking a policy for internal contradictions, where chunking destroys exactly the completeness the task requires.

Because they change what the user has to take on trust. Without a citation the only way to check an answer is to already know it, which makes the feature useful precisely where it is not needed. A link into the source passage turns an unverifiable assertion into a claim with evidence attached, and that is what makes the system usable in a domain where being wrong is expensive. The engineering consequence is that citation is a retrieval requirement rather than a formatting one: stable document identifiers and offsets must be carried from ingestion onwards, because you cannot reconstruct where a chunk came from after the fact. Retrofitting citations to a pipeline that discarded provenance means reingesting the corpus.

Mostly into passages, and that pressure shapes every other decision. A fixed budget divided by chunk size gives the number of independent sources you can pass, so larger chunks mean fewer of them and smaller chunks mean more but choppier ones. Instructions, conversation history and the space reserved for the output all draw on the same total, and history grows turn by turn while the passage allowance does not, so a long conversation quietly starves retrieval. When something must be dropped the choice is fewer sources or truncated ones, and truncating is the worse option because it tends to remove the exact sentence a citation points at. Deciding that policy explicitly, and logging when it fires, is worth more than any prompt wording.

Because everything downstream inherits its mistakes silently. A PDF has no notion of a paragraph, a heading or a reading order — it has positioned glyphs — so extraction must infer structure, and it infers wrongly on two-column layouts, running headers, footnotes, and text that is really an image. What arrives in the index is then a chunk beginning with a page footer whose sentences interleave two columns. No embedding model and no reranker recovers from that, and the failure raises no error: you simply get worse retrieval with nothing anywhere pointing at the parser. Reading a sample of parsed output by eye before indexing is the cheapest control available, and the one most often skipped in favour of tuning the retrieval end.

The decision is not about a number, it is about what a boundary destroys.

Source: a support article, plain text.

  ... To reset the device, hold the power button for ten seconds.
  The indicator will flash amber twice. Do not release the button
  before the second flash, or the reset will not complete.

Fixed 200-character chunks, no overlap:

  chunk 41  "... To reset the device, hold the power button for ten
             seconds. The indicator will flash amber twice. Do not"

  chunk 42  "release the button before the second flash, or the reset
             will not complete. Firmware updates are ..."

Query: "how long do I hold the power button"
  -> retrieves chunk 41, ranked first, high score.
  -> the answer is correct and incomplete: the condition that decides
     whether the reset actually works is in chunk 42, which retrieves
     poorly because on its own it is about nothing in particular.

Two defects, and only one of them is visible. Chunk 41 answers the question, so recall@1 is satisfied and no metric complains, while the user follows the instruction and the reset fails. Chunk 42 is worse than useless: it begins mid-sentence with an unresolved pronoun, so its embedding represents a fragment with no topic, and it will rarely be retrieved for anything.

The size itself trades two things against each other. Small chunks embed precisely, because a single vector can represent one idea, and they waste less of the prompt budget per hit — but they lose the surrounding context that made the sentence meaningful. Large chunks keep context and dilute the embedding, so a passage covering four topics matches all four weakly and none strongly.

What resolves it is chunking on structure rather than on character count. Splitting at headings, then at paragraphs, then at sentences, and only falling back to a hard cut when a paragraph exceeds the limit, keeps boundaries where the author put them. Prefixing each chunk with its document title and section path restores the topic that a mid-document fragment otherwise lacks.

The two habits worth stating unprompted: pick the size by measuring retrieval on a gold set rather than by argument, and store chunk boundaries as offsets into the extracted text so you can re-chunk without re-parsing. Both of those turn chunk size from a permanent commitment into a parameter.

Overlap repeats the tail of one chunk at the head of the next, so a sentence that answers a question is not split across a boundary where neither half retrieves well. It buys tolerance for the fact that you cannot know in advance where an answer will sit. The costs are all quiet. Index size and embedding spend grow by the overlap fraction. The same content now exists in two chunks, so the top results for a query can be two near-duplicates that crowd out a genuinely different source, which reduces the diversity of evidence the model sees while every metric reports a hit. That last effect is why deduplicating overlapping neighbours during prompt assembly matters as much as choosing the overlap.

Not as flowing text, because a table read left to right loses the column each number belonged to and yields a chunk of digits meaning nothing. Extract it as a table, then serialise it so the association survives — markdown, or one row per line with the header repeated — and keep the caption and the sentence introducing it, since the caption usually carries the units and the period the numbers refer to. Split a large table by rows rather than arbitrarily, repeating the header in each part. A short generated description of what the table shows, indexed alongside it, retrieves for the questions that never mention a column name, which is most of them.

It has to become text before anything else can happen, so optical character recognition is the ingest step and its error rate becomes your retrieval error rate. That matters most for exactly the tokens users search by — part numbers, account references, dates — where one misread character makes a string unfindable by lexical search and subtly wrong in the answer. Keep a per-page confidence signal where the engine provides one and treat low-confidence pages as needing review rather than indexing them silently. Keep the page image and the coordinates of the recognised text too, because a citation into a scan should show the user the region it came from rather than a transcription they cannot check.

Everything you will later want to filter, cite, authorise or invalidate by — which is more than it first appears.

{
  "chunk_id": "kb/reset-guide.pdf#c0007",
  "doc_id": "kb/reset-guide.pdf",
  "doc_version": "2026-06-14T09:12:00Z",
  "ordinal": 7,
  "char_start": 4180,
  "char_end": 4712,
  "page": 12,
  "section_path": "Troubleshooting > Hard reset",
  "title": "Device reset guide",
  "effective_from": "2026-06-14",
  "status": "current",
  "acl": ["group:support", "group:field-engineering"],
  "tenant_id": "t-4471",
  "content_hash": "sha256:9f2c...",
  "embedding_model": "emb-v3"
}

Read it as four groups, each earning its place from a different failure. The identity fields — doc_id, ordinal, content_hash — let you enumerate every chunk derived from a document, which is what makes an update, a delete and an erasure request possible at all. Without them, updating a document means finding its chunks by guesswork.

The location fields make a citation resolvable to a place a user can look at. An answer that cites a document without a page or an offset asks the reader to search a long manual, and a reader who has to do that stops checking, at which point the citation is decoration.

The governance fields decide what may be returned. acl and tenant_id allow the permission check to happen inside the search rather than after it; effective_from and status are what let you prefer the current policy over the superseded one, which no embedding will do for you because supersession is not a property of the text.

The provenance fields are the ones omitted most often and regretted most. Knowing which embedding model produced a vector is what makes a model migration a controlled operation instead of a rebuild of unknown extent, and doc_version is what tells you whether an index is stale without diffing the corpus.

By carrying an anchor from ingestion rather than generating one afterwards. That means a stable document identifier that survives reindexing, a location within the document — page, section path, or character offsets into the extracted text — and enough of the original span to highlight. The model cites a chunk identifier and your application resolves it into a link that opens the source at that place. The failure to avoid is a citation naming only a document, because it cannot be checked cheaply and so it will not be checked. Note also that a citation must reference the passage, not the model's paraphrase of it: resolving to the stored chunk is what makes verification mechanical.

Retrieval fills with duplicates, and the effect is worse than wasted slots. Several near-identical passages in the top results reduce the diversity of evidence the model sees, so a wrong statement appearing in three copies looks corroborated rather than repeated. Duplicates also flatter evaluation, because recall counts a hit whichever copy found it while answer quality falls. The usual causes are a document indexed under two paths, a pipeline re-run without deletes, and chunk overlap. A content hash per chunk catches exact duplicates at ingest for almost nothing, and near-duplicate suppression during assembly handles the rest — but neither happens unless someone looked for the problem.

A position in a vector space learned so that texts used in similar contexts land near one another, which in practice means topical and paraphrase similarity. That is why a query and a passage sharing no words can still match: the model has learned that "cancel my plan" and "terminating a subscription" occupy the same region. What is encoded is a compressed summary of meaning as the training data happened to use it, in a fixed number of dimensions, for the whole span of text at once. Every characteristic of retrieval behaviour follows from that compression, including its hard limit — one vector cannot represent a passage that discusses three unrelated things, so it represents an average of them and matches none well.

Anything where the exact form carries the information, and anything the training data had no reason to model. Negation is the standard example: a sentence and its denial sit close together because they are about the same subject. Identifiers are worse — two invoice references differing in one digit are near neighbours — and a number is treated as a token rather than a quantity, so no embedding search answers "over fifty thousand". Recency, authority and permission are not encoded at all, because they are not properties of the text. Each of those gaps is filled by something else, whether lexical search, a metadata filter or a rerank step, and never by a larger embedding model.

The three families differ in what they precompute, and that decides everything else.

1,000,000 chunks at 768 dimensions, 4 bytes per component
  -> roughly 3 GB of raw vectors before any index structure

index  build cost   query           recall@10   extra memory   fails at
-----  ----------   -------------   ---------   ------------   ----------------
flat   none         scan all        1.00        none           scale; latency
                                    exact                      grows linearly
IVF    cluster the  probe a few     tunable     cluster lists  churn; clusters
       vectors      clusters        mid-high     small         drift, needs
                                                              retraining
HNSW   build a      walk a          tunable     the graph,     memory; slow to
       graph        neighbour       high        clearly the    build, hostile
                    graph                       largest        to bulk delete

the dials that matter
  IVF   nlist   how many clusters exist     more -> finer partitions
        nprobe  how many are searched       more -> higher recall, slower
  HNSW  M       neighbours kept per node    more -> higher recall, more RAM
        ef      candidate breadth at search more -> higher recall, slower

Read the table as one decision repeated three times: how much memory and build time will you spend to avoid scanning. Flat spends nothing and scans everything, which is genuinely correct below a few hundred thousand vectors and is always the right thing to measure recall against.

IVF is the cheap middle. Building it is a clustering pass, inserts are accepted without rebuilding, and query cost is a dial you turn at run time. Its weakness is that the clusters were fitted to the data as it was, so a corpus whose content shifts gradually loses recall with no change in configuration — a slow failure that looks like the model getting worse.

HNSW gives the best latency at high recall and charges for it in memory and build time, since the graph itself is substantial on top of the vectors. It also handles deletion badly, because removing a node damages the connectivity the search depends on, so deletes become tombstones and churn accumulates until a compaction.

The mistake to avoid is treating the recall column as a property of the index rather than of the settings. Both approximate families will return whatever recall you tuned them to, and neither will tell you what that is — so build a flat index over a sample and measure the overlap. An index whose recall you have never measured is an index with an unknown top result.

The result set, not the distances. An approximate index returns a list that is usually the true nearest neighbours and occasionally omits one, because it examines a fraction of the vectors rather than all of them. What you trade away is recall against exhaustive search, and it is directly measurable: build a flat index over a sample, run the same queries against both, and report the overlap. That number is the one nobody computes, which is how a system ships with a search parameter tuned for latency and a quietly missing top result that looks like a model problem. Approximation is fine; approximation of unknown magnitude is a correctness gap you cannot bound.

Memory and bandwidth, linearly, everywhere. A vector's footprint is dimensions times bytes per component, so doubling dimensions doubles index memory, doubles the cost of every distance computation, and doubles what crosses the network on a bulk load. Because an approximate index only performs when it stays resident in memory, dimensionality is what decides how many vectors fit on a machine and therefore your shard count. Larger embeddings do generally retrieve somewhat better, so this is a measurement rather than a preference: compare retrieval quality at the smaller size against the hardware it saves, and check whether your model supports truncating dimensions, which often costs less quality than the size reduction suggests.

It replaces each component of a vector with a smaller representation — eight-bit integers, or in the extreme a single bit — cutting memory by a large factor and speeding up distance computation, at the price of some recall. The standard shape is to search the compressed vectors for a candidate list, then rescore those candidates against full-precision copies held on cheaper storage, which recovers most of the loss for a small additional cost. What makes it worth doing is that memory is the binding constraint on vector search, so a fourfold reduction is often the difference between one machine and four. The discipline is the same as everywhere else here: measure the recall you gave up rather than assuming it rounds to nothing.

Every vector in the index becomes meaningless, because distances are only comparable within one model's space. A model change is therefore a full re-embedding of the corpus and a rebuild of the index — real money and real hours on a large collection — and it cannot be done gradually in place, since a half-migrated index returns nonsense for any query that touches both halves. The workable pattern is to build the new index alongside the old, evaluate both on the same gold set, shadow live traffic, then switch reads and delete the old one. Budget for this at design time by storing extracted text and chunk offsets separately from the vectors, so re-embedding never means re-parsing.

Because an embedding encodes meaning and an identifier has none. A part number, an error code or a customer reference is a token the model has no useful representation for, so it lands near other strings that merely look similar, and a query containing it retrieves passages about the general topic instead of the one passage containing that exact string. The resulting user experience is distinctive and damaging: the system answers vague questions well and precise ones badly, which is the opposite of what anyone expects from search. No larger model and no bigger index fixes it, because the requirement is exact match and dense retrieval discards exactness by construction.

It matches terms literally and weights them by how rare they are in the corpus and how often they appear in a document, which makes it strong at exactly the queries dense retrieval fails: identifiers, proper nouns, quoted phrases, and jargon that postdates the embedding model's training data. It also needs no inference at query time, so it is cheap, and its behaviour is inspectable — you can see which terms matched and why a document scored. Its weakness is the mirror image: no vocabulary overlap means no match at all, so a paraphrased question finds nothing. Running both legs and fusing them is standard precisely because the two failure modes barely intersect.

The problem is that the two legs produce scores on incomparable scales, so you cannot add them.

Query: "error 0x8007 on reset"

dense results          lexical results
1. chunk_A             1. chunk_D    <- the only one with the literal code
2. chunk_B             2. chunk_A
3. chunk_C             3. chunk_E

Reciprocal rank fusion: score = sum over lists of 1 / (k + rank), k = 60

chunk_A   1/61 + 1/62 = 0.016393 + 0.016129 = 0.032522
chunk_D   1/61         =                       0.016393
chunk_B   1/62         =                       0.016129
chunk_C   1/63         =                       0.015873
chunk_E   1/63         =                       0.015873

fused order: A, D, B, then C and E tied

The arithmetic shows the property that makes the method work. Only ranks enter the formula, so a cosine similarity and a BM25 score never have to be normalised against each other, and adding a third retriever needs no re-tuning. A document appearing in both lists is rewarded — chunk_A wins on agreement rather than on either individual position — while a document ranked first by only one leg still places highly, which is what keeps the exact-match hit from being buried.

The constant is doing more work than it looks. A large k flattens the differences between ranks so that appearing in several lists dominates position; a small k makes the top rank of each list decisive. It is a parameter to fit on your gold set, not a magic number, and the fact that the conventional value works acceptably everywhere is why nobody checks whether it is right for them.

What fusion cannot do is repair a missing candidate. If neither leg retrieved the passage, no combination produces it, so the fusion step is where you notice that recall is a property of the legs and precision is a property of the ordering.

Two costs to name. You now run and operate two indexes, which must be kept consistent through updates and deletes, and a document present in one but not the other produces results that are hard to explain. And weighting one leg higher — the alternative to rank fusion — reintroduces exactly the score-normalisation problem that rank fusion existed to avoid.

It scores a query and a passage together in a single forward pass, so it can attend to how the specific words of the query relate to the specific words of the passage. That is strictly more information than a dense retriever has, because a retriever must embed a passage before it has ever seen the query. The result is a considerably better ordering of a candidate list. The cost is that the work is per pair rather than per passage, so nothing can be precomputed and nothing can be cached across queries: reranking fifty candidates is fifty inferences. That is why a reranker sits after a cheap recall-oriented stage instead of replacing it, and why its depth is a budget rather than a setting.

A batched model call per candidate, on the critical path before the first token can be generated. In most pipelines that is the largest single addition to time-to-first-token, and it scales with candidate count, so rerank depth is a latency dial dressed up as a quality parameter. The mitigations all shrink or hide the work: rerank fewer candidates, use a smaller reranker, size the hardware for the batch, and cache scores for repeated query and document pairs. Where latency is genuinely tight, reranking only when the retrieval scores are close is a fair compromise, since a clear winner does not need reordering — but that makes your latency distribution query-dependent, which complicates alerting.

Enough that the correct passage is in the list, because a reranker can only reorder what it is given. That makes the retrieval stage's objective recall rather than precision, which is why its top-k should be considerably larger than the number of passages you will eventually put in the prompt. The right figure comes from measuring recall at increasing depths on your gold set and finding where the curve flattens; past that point you pay reranker latency for nothing. Reporting retrieval recall at the candidate depth and answer quality after reranking as two separate numbers is what makes the parameter tunable at all, because a single end-to-end score cannot tell you which one is limiting.

Restricting the space the index searches, which is usually what you want and occasionally destroys the result. An approximate index navigates a graph or a set of clusters built over all the vectors, so a highly selective filter can leave the search traversing regions where nearly everything is excluded — it exhausts its candidate budget and returns few or poor results even though matching vectors exist. Engines differ in how they handle this, pre-filtering, post-filtering or a hybrid, and the failure is silent in all of them. Test narrow filters explicitly against a flat scan, and for a small set of very selective values consider a separate partition per value instead.

It tells you whether a passage containing the answer was among the top k retrieved, which is the single thing the retrieval stage is accountable for and the ceiling on everything after it. What it hides is position, redundancy and sufficiency. A hit at rank ten is worse than one at rank one because a generator attends unevenly across a long prompt; a hit accompanied by four near-duplicates leaves no room for corroborating evidence; and a passage judged relevant may still not contain enough to answer. So recall@k is the correct first metric and never the only one — pair it with a rank-aware measure and an end-to-end answer score measured on the same items.

From real questions, which means from logs, support tickets and the people who will use the system — not from questions written by reading the documents, because those inherit the source's vocabulary and hide precisely the mismatch you are trying to measure. For each question record every passage that genuinely answers it, allowing several, and the expected answer where one exists. A hundred or two carefully judged items beat thousands generated automatically. Stratify deliberately to include queries carrying identifiers, questions needing several documents, questions with no answer in the corpus at all, and near-duplicate topics, because those four strata are where pipelines break and an unstratified average conceals all of them.

One number per stage per stratum, so a change moves something you can point at. The figures below show the shape of a real scorecard rather than any measured system.

Gold set: 180 real questions, judged passages recorded per question

stratum              n    recall@50  recall@5   MRR    grounded   answer ok
-------------------  ---  ---------  --------  -----  ---------  ---------
paraphrase            64     0.95      0.81     0.68     0.93       0.88
exact identifier      31     0.61      0.42     0.35     0.90       0.55
multi-document        38     0.88      0.59     0.51     0.81       0.47
long-tail topic       25     0.79      0.55     0.44     0.88       0.61
no answer in corpus   22      n/a       n/a      n/a      n/a       0.32
                                                        (share correctly declined)

Four separate diagnoses fall out of it, and none of them would be visible in a single overall score.

The identifier row is a retrieval failure at the candidate stage: recall@50 of 0.61 means the passage was never in the list, so no reranker and no prompt change can help. That is the signature of a dense-only or dense-dominant leg, and the fix is the lexical index and the fusion weight.

The multi-document row is the opposite shape. Recall@50 is healthy and answers are still wrong less than half the time, with groundedness lower than elsewhere, which points at assembly and generation: the passages are being found and then not all reaching the prompt, or reaching it and being ignored. Raising the passage count and checking for duplicate suppression is the first move.

The final row is the one most teams have no number for. A third of unanswerable questions correctly declined means two thirds produced a confident answer from nothing, which is the failure that destroys trust fastest and the one an accuracy-only scorecard cannot represent, because there is no correct passage to score against.

The habit that makes this work is versioning the gold set and the judge prompt as artefacts alongside the code, so a movement in the numbers is attributable to a change in the system rather than to a change in how it was measured.

Groundedness is whether each claim in an answer is supported by the passages that were supplied. You measure it by decomposing the answer into individual claims and checking each against the cited context, which a model performs acceptably because it is a narrow entailment judgement rather than an open-ended assessment. Report it as the share of claims supported, and keep the unsupported ones, since they are your fabrication examples and the only concrete material for improving the prompt. Two things it does not measure: completeness, because a fully grounded answer can omit the important half, and truth, because groundedness against a superseded document is faithful and wrong at the same time.

Answering that from the request log is the core diagnostic skill in this discipline, and it is mechanical once the log holds the right fields.

A bad answer is reported. Read the logged request in this order.

1. Was a judged-correct passage among the retrieved candidates?
     no  -> retrieval failure. Go to 2.
     yes -> go to 4.

2. Does the lexical leg alone find it?
     yes -> the dense leg is losing it: identifier or rare term.
            Check the fusion weights, and that the lexical leg ran.
     no  -> go to 3.

3. Is the passage in the index at all?
     no  -> ingestion. The parser dropped it, chunking split it,
            or the document was never crawled.
     yes -> query or filter mismatch. Rewrite the query, add
            synonyms, and print the filter that was applied.

4. Was the passage still present after reranking and truncation?
     no  -> ordering or assembly. Raise the passage count, fix the
            reranker, stop truncating mid-passage.
     yes -> generation. The evidence was in the prompt and was not
            used: check its position, conflicting passages, and
            whether the abstention instruction fired wrongly.

The value of the tree is that every branch ends at a different team's work. Steps 2 and 3 are ingestion and indexing; step 4's first branch is the serving path; its second branch is prompting. A team without this separation argues about the model when the parser dropped a table.

Step 3 is the one that surprises people. A passage that is not in the index cannot be retrieved by any means, and the most common reason is not a missing crawl but a parse that produced something unrecognisable — a table read as digits, a heading fused to the paragraph before it. Searching the index for a distinctive literal string from the source document settles it in seconds.

The prerequisite for all of it is logging the retrieval, not just the answer: the query, the rewritten query, candidate identifiers with scores, what survived reranking, what was actually placed in the prompt, and the citations used. Without those fields the tree collapses into speculation, which is why the logging is the first thing to build and the first thing dropped.

Judgements that are narrow, comparative and cheap to spot-check: does this passage answer this question, is this claim supported by this text, which of two answers is more complete. Those work because the judge is handed the evidence and asked a bounded question. It is not good for scoring correctness against nothing, or for producing an absolute number you then track to two decimal places, because the scale drifts with the prompt and the model version. Calibrate against a human-labelled sample, pin the judge model and its prompt as versioned artefacts, and re-calibrate whenever either changes — otherwise an apparent quality improvement may only be a judge that got more generous.

Because recall is blind to position and position is what the generator experiences. Mean reciprocal rank averages one over the rank of the first relevant result, so it rewards putting the right passage first and falls sharply as the hit moves down the list. That makes it the metric that moves when a reranker starts working while recall stays flat, which is exactly the improvement you need to detect in order to justify the reranker's latency. Its own blindness is to everything after the first hit, so for questions requiring several sources it says nothing useful, and a coverage measure over all the required passages belongs beside it.

As an ordinary write, if the index supports incremental insertion — graph indexes do, and clustered ones accept inserts while their clusters drift away from the data. The part needing design is not the insert but the unit of work: a changed document means deleting all of its old chunks and inserting the new ones, atomically enough that no query sees both versions at once. Key chunks by document identifier plus an ordinal so the old set is enumerable, and drive the whole thing from a change feed rather than a periodic re-scan, because scanning a corpus to discover what changed does not scale and misses deletions entirely — a deleted document simply stops appearing in the scan.

A marker recording that something was deleted, kept because the deletion cannot be applied everywhere immediately. Vector indexes typically flag a vector as deleted and filter it from results, reclaiming the space only at a later compaction, because physically removing a node from a graph would damage the connectivity the search depends on. The consequences are practical: tombstoned vectors still occupy memory and are still traversed, so a corpus with heavy churn loses both space and effective recall until it is compacted, since some of the candidates the search found are discarded after the fact. Deletion is therefore an operational task with a schedule, not an instantaneous state change.

The constraint is that reads must never see a partially built index, because a half-migrated vector space returns nonsense rather than degraded results.

Goal: move to a new embedding model. Every vector must be rebuilt.

  1. RECORD      note the current change-feed offset
  2. BUILD       create index_v2 and re-embed from the stored extracted
                 text and chunk offsets, not from the source documents
  3. DUAL WRITE  from now on apply every change-feed event to both
                 index_v1 and index_v2
  4. CATCH UP    replay the feed from the offset in step 1 into v2
  5. EVALUATE    run the gold set against both and compare recall@k,
                 MRR and end-to-end answer quality per stratum
  6. SHADOW      send a share of live queries to both, log the top
                 results, show only v1 to users
  7. CUT OVER    flip reads by configuration, one tenant at a time
  8. CONTRACT    stop dual writes, delete index_v1

Step 2 is where the earlier decision to store extracted text separately from vectors pays for itself. If the only copy of the chunk boundaries is inside the index, re-embedding means re-parsing the corpus, which multiplies the cost and introduces a second variable: you can no longer tell whether a quality change came from the new model or from a new parser version.

Steps 3 and 4 in that order are what make the build consistent without a freeze. Dual writing begins before the catch-up completes, so the window between the recorded offset and the switch is covered from both directions, and the operation is idempotent as long as chunk identifiers are deterministic.

Step 5 is the step that gets skipped, and skipping it is how a migration regresses quality invisibly. A newer embedding model is not automatically better on your corpus, particularly on the identifier and long-tail strata, and the comparison is only meaningful before the old index has been deleted.

Step 7 being per tenant and configuration-driven is the whole safety argument. Reverting is then a configuration change rather than a rebuild, which is the property that lets you cut over on a Tuesday instead of at midnight.

The difference between the two shapes below is not style, it is whether k means anything.

# Wrong: retrieve, then filter
hits = index.search(query_vector, k=10)
visible = [h for h in hits if can_read(user, h.doc_id)]
# a restricted user silently gets three passages instead of ten,
# and sometimes zero, while every metric reports a successful search

# Right: the permission is part of the search
hits = index.search(
    query_vector,
    k=10,
    filter={
        "tenant_id": user.tenant_id,
        "acl": {"any_of": user.group_ids},
    },
)
# k=10 means ten passages this user is allowed to read

Post-filtering is not usually a security hole, because the filter itself is correct. It is a quality hole that varies per user and averages away in aggregate metrics, so the report reads "search is fine" while a member of one small group gets a thin, arbitrary subset of the evidence and an answer built from it.

It becomes a security hole the moment anything derived from the discarded passages escapes: a result count, a summary generated before filtering, a suggested follow-up question, a "no results" that reveals a document exists. Each of those has shipped in real systems because the filter guarded the list and not the pipeline.

Making it work requires denormalising the authorisation model onto the chunk, which has its own cost. The acl field is a snapshot of the groups that could read the document at index time, so a permission revoked in the source system is still in the index until something updates it. That is an argument for keeping the grant list coarse and stable, expanding groups to members at query time rather than at index time, and treating permission changes as change-feed events like any other edit.

The last detail is that the filter must be injected by the retrieval layer rather than passed in by callers. A filter that a caller can omit is a filter that a new endpoint will omit, and the test worth writing is that a search issued without a tenant identifier fails loudly instead of returning the corpus.

It silently reduces the answer to whatever survived. Retrieve ten passages, drop the six this user may not see, and the model answers from four — sometimes from none — while everything appears to be working. That is a quality defect varying by user, invisible in any aggregate, and it becomes a security defect as soon as something derived from the dropped passages leaks: a count, a summary, a suggested follow-up, or the difference between "no results" and "not permitted". Filtering inside retrieval, so that k means k passages this user can read, is both safer and better, and it is the reason the vector store needs the authorisation attributes on the chunk.

By making the tenant a partition rather than a predicate wherever volume allows, because a filter is a line of code somebody can forget while a partition is a different index that must be reached deliberately. Where per-tenant indexes are impractical, carry the tenant identifier on every chunk, apply it as a mandatory filter injected by the retrieval layer rather than supplied by callers, and assert that a query lacking it fails instead of returning everything. The failure to fear is not a clever attack but an ordinary bug — a cache keyed without the tenant, a reranker batch mixing requests, a debug endpoint — so keep the identifier attached to the request end to end.

Treat erasure as a first-class pipeline operation with a verification step, because the content exists in more places than the vector index. One source document may have produced dozens of chunks, vectors in a primary and its replicas, a lexical index, cached retrieval results, generated summaries, and entries in an evaluation set. Deleting from the vector store alone leaves the text recoverable from most of those. Keep a manifest mapping each source document to every derived artefact so deletion is enumerable rather than remembered, route it through the same change feed as updates, and assert afterwards that a search for a distinctive literal string from the document returns nothing.

Control flow decided at run time by the model. A pipeline follows a sequence you wrote; an agent is given tools and an objective and chooses which tool to call next based on what the previous call returned, looping until it judges itself finished. That flexibility is the point and it is also the bill: the sequence of steps is no longer knowable in advance, so latency, spend and the set of side effects become distributions rather than constants, and testing means evaluating trajectories rather than asserting on an output. You accept that only when the branching genuinely cannot be enumerated, because everything you gave up was something that made the system easy to operate.

The schema is a prompt and a contract at the same time, which is why it is worth more care than an internal function signature.

{
  "name": "refund_order",
  "description": "Issue a refund against one order. Call lookup_order first and confirm the order is delivered. Refunds are final and visible to the customer.",
  "input_schema": {
    "type": "object",
    "properties": {
      "order_id":        {"type": "string", "pattern": "^ORD-[0-9]{8}$"},
      "amount_minor":    {"type": "integer", "minimum": 1},
      "currency":        {"type": "string", "enum": ["GBP", "EUR", "USD"]},
      "reason_code":     {"type": "string",
                          "enum": ["damaged", "late", "duplicate", "goodwill"]},
      "idempotency_key": {"type": "string"}
    },
    "required": ["order_id", "amount_minor", "currency",
                 "reason_code", "idempotency_key"],
    "additionalProperties": false
  }
}

Four decisions in that block, each preventing a specific failure. The description carries the preconditions and the consequence, because the model chooses tools by reading descriptions and has no other source for "this is irreversible". A description that only restates the name gives the model nothing to select on, and tool confusion between two similar tools is almost always a description problem rather than a reasoning one.

Enumerations instead of free text turn an unbounded generation into a choice. reason_code as a string means you receive forty spellings of "broken" and can neither report on them nor branch on them; as an enum the model must pick, and an invalid pick is a schema error rather than bad data written to your ledger. additionalProperties: false matters for the same reason — it rejects a hallucinated field instead of ignoring it, which is how you find out that the model believed in a parameter you never had.

The amount being an integer in minor units is a correctness decision, not a formatting one. A model emitting 42.99 as a float in a field typed as a number invites the rounding problems money always has, and an integer has no ambiguous representation to get wrong.

The idempotency key is required rather than optional because an optional safety parameter is an absent one. Make it derivable from the request content, so a genuine retry collides and a genuinely different refund does not.

The cost of all this rigour is that a strict schema turns model mistakes into errors your orchestrator must handle and feed back as observations. That is the right trade, but the error messages are now part of the prompt, so they must explain what to do rather than merely what failed.

Alternates between choosing an action and observing its result, with the accumulated observations serving as the context for the next choice. That is the entire mechanism, and stating it plainly matters because the elaborate names attached to it hide how little structure there is: a model, a tool list, a growing transcript, and a stopping condition. What varies is whether a plan is written up front and then followed or re-decided at every step. Writing the plan first makes the trajectory inspectable and cheap to reject before any side effect occurs; re-deciding each step adapts better to surprises. Most production designs keep a plan for structure and allow it to be revised when an observation contradicts it.

Because the loop terminates when the model decides it is finished, and that is not a guarantee. Without a cap, a tool call that fails and is retried in a slightly different form each time will spin until something else stops it, spending money and wall-clock time on a request nobody is waiting for any more. Cap the iterations, the total tokens and the elapsed time, and make exhaustion an explicit outcome the caller can handle rather than a timeout firing somewhere in the middle of a side effect. Cap repetition too: the same tool called twice with identical arguments is a loop signature worth halting on directly rather than waiting out the budget.

By having the caller supply a key identifying the intent, and having the tool return the original result when it sees that key again. This matters more for agents than for ordinary clients because retries here are not only transport level: the model may reissue a call because the observation was ambiguous, or because a later reasoning step re-derived the same action from the same premises. Without a key, "create the refund" executed twice is two refunds and nothing in the transcript reads as wrong. Derive the key deterministically from the request content so a genuine retry collides, and return enough of the prior result that the model can recognise the action already happened.

Between the model proposing an irreversible action and anything executing it — never inside the model's own reasoning, where it can be talked out of.

sequenceDiagram
    participant M as Model
    participant O as Orchestrator
    participant T as Reversible tools
    participant H as Human approver
    M->>O: propose next action
    O->>T: execute if reversible
    T-->>M: observation
    M->>O: propose refund_order
    O->>H: hold for approval
    H-->>O: approve or reject with a reason
    O-->>M: outcome as an observation

The orchestrator, not the model, classifies each tool as reversible or not. That is the load-bearing detail: if the decision to seek approval is something the model makes, then a persuasive retrieved document or an awkward phrasing can route around it, and you have a policy rather than a control. Marking the tool itself means no trajectory can reach the side effect without passing the gate.

The rejection path has to return a reason as an ordinary observation. An agent told only "denied" will retry the same action or invent a variant that slips through — changing an amount, splitting one refund into two — whereas a reason lets it either satisfy the condition or stop and explain. That single design choice is the difference between a gate and an obstacle.

What the gate costs is the property that made the agent attractive. A held request is no longer sub-second, the human becomes a queue with its own throughput and working hours, and you need a policy for expiry. That is why the gate is worth narrowing with argument-level rules — approve refunds under a threshold automatically, hold the rest — so the human sees the cases where judgement is actually required.

The gate is also an audit artefact, and it should be built as one: who approved what, with which arguments, at which step, on the basis of which observations. That record is what makes an agent with side effects defensible to whoever owns the risk.

By asserting on the trajectory, not only on the final message, because a correct answer reached through a wrong path will be wrong next week.

case: "customer says the item arrived broken, order ORD-40028119"
      tool responses replayed from recorded fixtures

trajectory assertions
  1. lookup_order called with order_id ORD-40028119     required call
  2. refund_order not called before lookup_order returned    ordering
  3. refund_order amount_minor equals the order total        derived
  4. approval gate reached exactly once                  side effect
  5. no tool called twice with identical arguments        loop guard
  6. steps <= 8 and total tokens <= 30000                    budget

outcome assertions
  7. final message states the order id and the refunded amount
  8. no claim about a delivery date that no tool output contained

Replaying recorded tool responses is what makes any of this repeatable. With live tools the same case exercises a different corpus each run, so a failure cannot be attributed and a pass proves little; with fixtures the only variable is the model's behaviour, which is the thing under test.

Assertions 2 and 5 are the ones absent from most suites and the ones that catch real regressions. Ordering assertions encode the preconditions your tool descriptions claim, and a model upgrade that starts refunding before looking up will otherwise pass every output check on this case. The loop guard catches the failure mode that costs money without producing a wrong answer at all.

Assertion 8 is the groundedness check applied to tool output rather than to retrieved passages, and it is the same idea: a claim in the final message that no observation supports is a fabrication, whether the source was a document or an API.

What this cannot do is cover the space. Trajectories branch, so a suite of cases is a set of samples, and the honest supplement is aggregate measurement over live traffic — step counts, tool error rates, gate rejection rates, budget exhaustions — treated as the actual regression signal.

Whenever you can write the steps down. If the task is extract these fields, look up this record, produce this summary, then a fixed pipeline with a model at each step is cheaper, faster, testable with ordinary assertions and debuggable by reading a stack trace instead of a transcript. An agent earns its place when the required sequence depends on what is discovered along the way and the branching is too wide to enumerate — investigation, open-ended research, triage across systems whose state you cannot predict. The honest framing is that agency buys flexibility and pays in predictability, so ask whether you need flexibility you could not get by adding a branch.

The queries change shape. A demo is driven by questions written by whoever built it, phrased in the corpus's own vocabulary and answerable from one passage. Real traffic brings identifiers, internal abbreviations, questions spanning several documents, questions whose answer is not in the corpus at all, and follow-ups meaningless without the previous turn. Each hits a different weakness: dense-only retrieval, missing synonyms, single-passage assembly, no abstention path, and no query rewriting. None surfaces as an error, so the first thing to build after the demo is query logging with retrieved identifiers attached, because a distribution you cannot see is one you cannot fix.

Nothing here threw an exception, and the customer received a refund.

step  action                             observation
----  ---------------------------------  ----------------------------------
1     search_kb "broken item refund"     3 passages, policy scoped to EU
2     lookup_order ORD-40028119          status SHIPPED, total 4299 GBP
3     lookup_order ORD-40028119          same result
4     refund_order amount_minor 4299     error: order not yet delivered
5     refund_order amount_minor 4299     error: order not yet delivered
6     refund_order amount_minor 4200     ok, refund created
7     final: "I have refunded 42.00 GBP for your damaged item."

Five distinct defects, none of which an output assertion would catch, because the output is well formed and roughly plausible.

Step 3 repeats step 2 with identical arguments, which means the observation from step 2 was not usable — either it was truncated out of the context or its format made the answer hard to extract. Repetition with identical arguments is the cheapest loop signal there is and should have halted the run.

Steps 4 and 5 retry a call that failed for a business reason rather than a transport one. Business rejections are terminal and must be surfaced to the model as a condition to satisfy, not as an error to retry; retrying them is how budgets are consumed.

Step 6 is the serious one. The model changed the amount to get past a validation error, which is not a fix but a workaround that altered the financial outcome. It is the signature of an error message that says what is wrong without saying what to do, and of an irreversible tool with no approval gate in front of it.

Two further faults sit underneath. No idempotency key appears anywhere, so a transport retry at step 6 would have created a second refund. And the policy retrieved at step 1 was scoped to one region and never checked against the order, which is retrieval feeding the loop and the loop ignoring it — the failure mode that makes an agent over a corpus harder to evaluate than either part alone.

It lands entirely before the first token, so the user experiences it as dead time with nothing on screen. A query rewrite, a retrieval stage and a reranker in sequence can easily exceed the model's own time-to-first-token, and unlike generation none of it can be streamed to hide it. That makes the pre-generation path the place to spend optimisation effort: run retrieval and any independent lookups concurrently, cap reranker depth, and issue retrieval on the raw query in parallel with the rewrite rather than after it. Where the total stays high, showing sources as they are found at least gives the user something true to look at while they wait.

The model picks one, usually whichever reads most confidently or appears earliest, and states it without mentioning the conflict. This is the most under-tested failure in retrieval systems, because contradictions are normal in real corpora: a superseded policy, an old revision of a manual, a draft filed beside the approved version. The retrieval layer has no notion of authority, so the fix is metadata — effective dates, status, version, document class — used both as a filter and as something the prompt is told to weigh. Beyond that, instructing the model to surface disagreement when passages conflict converts a silently wrong answer into a useful one.

It becomes an instruction channel. Anything retrieved is text the model reads, so a document containing directions — call this tool, ignore what you were told, send this elsewhere — is attempting to steer the loop, and an agent holding side-effecting tools makes the attempt worth someone's while. The design consequence is that retrieved content must never be presented as though it came from the operator: delimit it, label it as data, and gate every irreversible tool on something outside the model's control, whether an approval or a policy check on the arguments themselves. Retrieval over documents that users can upload is the highest-risk configuration in this whole discipline.

By logging the retrieval as well as the answer, because the answer alone tells you nothing about why. Per request, keep the query, the rewritten query, the identifiers and scores of the candidates, what survived reranking, what went into the prompt, the citations the answer used, and the latency of each stage. That record makes the two questions you will always be asked answerable afterwards. The aggregate signals worth alerting on are the abstention rate, the share of answers citing nothing, the top-result score distribution and the reranker's latency, since a drift in any of them precedes the quality complaints by days.

This is the first question to ask about any bad answer, and the discipline lies in insisting on both halves. A strong answer says the retrieved passages are logged per request and can be read back, that the first check is whether any of them actually contained the answer, and that the two outcomes lead to different work: a miss points at chunking, parsing, the query, the lexical leg or the filter, while a hit that still produced a wrong answer points at ordering, prompt assembly or the abstention instruction. It then names how that judgement is made at scale — a stratified gold set, recall at the candidate depth, groundedness per claim — and allows for the third case, where the corpus simply does not contain the answer and the system should have said so. A weak answer substitutes prompt changes for measurement: a newer embedding model, a larger top-k, a firmer instruction not to invent things, none of it preceded by establishing which stage failed. It treats retrieval as a component that either works or does not, rather than one with a recall figure it could have reported today.