Capacity is not the constraint that retrieval was solving
The argument for retrieval is usually presented as fitting: documents are big, windows are small, so select. If that were the whole argument, a large enough window would end it. But three other constraints are untouched by window size, and they are the ones that bite in production.
The first is that you pay for what you send, in money and in time. Every token in the prompt is billed and every token in the prompt is processed before the first output token appears, so a request carrying 400,000 tokens of context costs a multiple of one carrying 6,000 and makes the user wait for the difference. Prefill is parallel and fast per token, but it is not free per token, and time-to-first-token is the latency users perceive. Sending a whole handbook to answer a question about clause 4 is the same design mistake as SELECT * from a table you then filter in application code: it works, and it works worse every quarter.
The second is that a large window is not uniformly usable. Retrieval accuracy inside the context depends on where the relevant span sits, and the well-documented pattern is that material at the beginning and the end of a long input is used more reliably than material in the middle. A model that scores well on synthetic single-fact retrieval can still degrade when the task requires combining several facts scattered through hundreds of thousands of tokens, because the difficulty is not locating one string, it is attending to six things at once. Padding the context with plausible but irrelevant neighbours also makes the task harder, not neutral, since the model now has to discriminate rather than simply read.
The third is arithmetic that no window survives. A corpus is unbounded and a window is a constant. A company wiki, a ticket history, a decade of contracts, a code repository — these grow, and any policy of the form "send everything" has a date on which it stops working, usually shortly after the feature becomes popular enough to have more data.
Retrieval does jobs that have nothing to do with size
Even where everything would fit, the retrieval layer is where several other requirements are met, and removing it means reimplementing them somewhere less convenient.
Permissions are the clearest case. In any multi-tenant or enterprise setting the set of documents this user may see is a query-time property, and the filter belongs in retrieval where it can be applied against an index. Sending the whole corpus and instructing the model to disregard documents the user is not entitled to is not access control; it is asking a probabilistic system to keep a secret it has been handed.
Freshness is similar. Retrieval is the point at which you decide that the superseded version of a policy is excluded and the current one is not. Hand the model both and it will sometimes cite the wrong one, and you will have no mechanism to prevent it other than prompt wording. Citations, too, are easier when you know what you sent: a system that provides eight identified passages can attribute a claim to one of them and verify the attribution, while a system that provides a million tokens of undifferentiated text gets citations that are themselves generated rather than grounded.
What long context genuinely changed
Answering this question by defending retrieval on every axis is the weaker response, because a lot did change and an interviewer knows it.
Chunking a single medium-sized document is often no longer necessary or wise. Where a contract, a specification or a source file fits comfortably, sending it whole preserves the structure and cross-references that chunking destroys, and it removes the chunk-size tuning that used to dominate these projects. The retrieval unit can now be a document rather than a fragment, which is a better unit.
Recall pressure eases considerably. When the window was tight, precision was everything, because a wrong passage displaced the right one and the answer was lost. With room to spare you can retrieve generously — a wider candidate set, several rankings unioned — and let the model do the final filtering, which it is good at. That converts a hard ranking problem into an easier one, and it is a real simplification.
Prompt caching changes the economics in specific shapes. A stable corpus small enough to fit, queried repeatedly, can be sent once as a cached prefix and reused across requests at a fraction of the input cost and prefill time. For a product-manual assistant over a fixed handbook that can genuinely beat maintaining an index, and dismissing it out of hand is as wrong as claiming it generalises. It does not generalise, because the moment the corpus is per-user, changes often, or exceeds the window, the cached-prefix trick evaporates and you are back to selection.
So the accurate position is that long context absorbed chunking and relaxed ranking, while selection, filtering and attribution remain. The design question stopped being "how do I fit this" and became "what is the smallest set of material that fully supports the answer", which was always the better question.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).