A document was updated an hour ago and the assistant is still quoting the old version. Walk me through the diagnosis.
6 min readUpdated 2026-08-15
0
Answer
Short answer
Ask the index directly whether the old text is still there, because that single check splits the problem into change detection, queue or write failure, a retire step that left duplicate chunks, or a downstream cache.
Was this answer helpful?
Likely follow-ups
→The manifest says the new version was indexed but retrieval returns the old text. Where do you look next?
→How would you distinguish a genuinely dropped change event from a change event you never had a right to expect?
→What does a freshness alert look like that does not fire constantly during a bulk migration?
→The user's conversation started before the edit. What is the correct behaviour for turn six?
Two questions before diagnosis. Which document, by source identifier, and what exactly did the assistant say — the quoted passage and the citation it attached. And when did the conversation start, because a session that began before the edit may be repeating a passage it retrieved forty minutes ago, in which case nothing in the index is wrong at all.
Then the check that does the most work for the least effort: query the vector store directly for the chunks belonging to that document identifier and read them, along with their indexed_at and their stored source version. You are not looking for relevance here, you are taking an inventory. The result puts you in exactly one of four places, and each one has a different owner.
Loading diagram…
The branch to dwell on is the middle one. Both versions present is the most dangerous state and the least likely to be noticed, because retrieval still returns something plausible and the pipeline reports success.
Both versions present means the retire step failed
This happens when chunk identity is positional and the edit moved boundaries, so new chunks were written under new identifiers while the old ones were never deleted. It also happens when the upsert and the retire are separate operations and only the first succeeded, and when a re-parse derived a different document identifier than the original ingest did — different casing, a trailing slash, a versioned URL — so the reconciliation matched nothing and deleted nothing.
The immediate fix is to retire that document's orphans. The real fix is the reconciliation: any chunk in the index whose identifier is not in the manifest for its document is by definition garbage. Sweep the whole corpus for it once you have found one instance, because whatever caused it was not document-specific, and count what the sweep finds. That number is the size of the problem you thought you did not have.
Only the old version, and the manifest agrees with it
If the manifest still records the previous version or content hash, the pipeline never learned about the edit, and the interesting question is why. Change events get dropped: a webhook delivery failed and was not retried, a consumer acknowledged a message and then crashed before committing the write, a crawl cursor advanced past a page whose listing was paginated inconsistently.
Change events also get never sent, which is a different failure with a different fix — some sources do not emit an event when a field is edited through an API rather than a file write, some do not bump a modification timestamp on metadata-only changes, and some emit an event whose payload is the document identifier only, so a pipeline that trusts the payload's own content field sees nothing changed.
Check the source's own view: does it show a new version, and would the credentials your connector uses see it? A revision visible to the author but not yet published, or a change in a folder your service principal quietly lost access to, presents exactly as a dropped event and is not one — and the fix is a permission, not a retry.
The manifest has the new version and retrieval does not
Now it is a write or a lag problem, and the order of checks is mechanical. Look at consumer lag on the ingestion topic or queue, and at the oldest unprocessed offset rather than the average latency, because a backlog with a fast tail looks healthy on a mean. Look at the dead-letter queue, where one document that fails to embed on a content-type edge case sits quietly forever.
Check whether embedding succeeded but the index write was rejected — a metadata field of unexpected type, an oversized payload, a vector of the wrong dimensionality after a model change. Check whether you are reading a replica or a secondary region that has not caught up, and whether the alias your reader resolves is pointing at the index generation your writer is filling. A rebuild that cut over reads but not writes, or the reverse, produces precisely this symptom and survives every stage-level health check.
Below the index, the caches nobody drew on the diagram
If the index is clean and current, everything remaining is a cache, and there are more of them than the architecture diagram admits. A retrieval cache keyed on the query string will happily return the chunk identifiers and text it captured an hour ago. An answer cache keyed on question plus tenant will return the whole response. A reranker or embedding cache keyed on text is safe for text that has not changed, and stale for text that has. An edge or CDN layer in front of a response endpoint counts. So does the browser.
The context-assembly layer is the subtle one and now the common one. When you optimise for prompt caching by keeping the start of the prompt byte-identical across requests, what you are stabilising is usually a block of retrieved or pinned documents. A token-prefix cache cannot itself serve different content — it either matches the prefix you sent or it does not — but the code written to preserve the hit reuses a previously assembled context block rather than reassembling it from fresh retrieval, and that freezes the documents for the life of the block. Whatever refresh interval that reuse has is your real freshness bound, and it is usually undocumented, chosen for cost, and longer than an hour.
Conversation history is the last one and it is not a bug you can flush. Once the old passage is in the transcript it keeps influencing answers until it leaves the window, and the model has no way to know it was superseded. If freshness matters more than continuity, re-retrieve each turn and drop stale passages from the transcript rather than trusting the model to prefer the newer of two contradictory quotes.
The case where the index is right and the world is wrong
Before closing the ticket as a pipeline defect, confirm that the passage came from the document the user means. Corpora are full of near-duplicates: an exported PDF of a page that is still being edited, an attachment emailed in March, a copy in a second team's space, a translated version updated on its own schedule. If the citation points at one of those, retrieval is working perfectly and you have a corpus-hygiene problem, whose fixes are different in kind: deduplicate, mark canonical sources and prefer them in ranking, or stop ingesting the derived copy.
Make the next one an alert
This was reported by a user, and that is the actual defect. Three instruments earn their keep. Emit a freshness histogram of indexed_at minus source_updated_at per source, so lag is a distribution you watch rather than a number you assume. Alert on the age of the oldest unprocessed change and the depth of the dead-letter queue, not on throughput, because throughput stays healthy while one poisoned document rots.
And run a canary: a scheduled job edits a document in each source system with a fresh token, then queries for that token through the ordinary retrieval path — the same client, the same caches, the same alias — and reports the round-trip time. Anything less than end-to-end will show you green stages and a stale answer, which is exactly the situation you just spent an afternoon in.
Then make the state visible in the product. Carrying the indexed date on a citation and rendering it costs almost nothing and turns "the assistant is wrong" into "this says it is from Tuesday", which is a report you can act on in minutes.