The window is a budget with one growing line item
Everything in a request competes for the same allowance: the system prompt, the tool schemas, the few-shot examples, the retrieved passages, the whole conversation so far, the current user message, and the answer the model has yet to write. The reason this needs designing rather than monitoring is that all of those are roughly constant per turn except one. The system prompt does not grow. The retrieved set does not grow, because you choose how many passages to include. Conversation history grows on every single turn, twice — the user's message and the model's reply — and it grows fastest in exactly the sessions that matter most, the long troubleshooting ones where the user is engaged.
That asymmetry is the whole design. A feature that works beautifully in testing, where nobody has a conversation longer than six turns, degrades at turn forty when history has crowded out the retrieved evidence, and the symptom is that the assistant starts answering from memory instead of from documents. Nothing errored. The retrieval step ran, found the right passages, and they were the thing your truncation logic dropped.
Reserve the answer before you allocate anything
The output shares the window, so the space for it has to be subtracted at the top, not discovered at the bottom. Decide the maximum reply length the feature needs, in tokens, and treat that as spent before you compose the prompt. If what remains cannot hold the fixed prompt plus a usable amount of evidence, the feature needs a smaller reply, fewer passages, or a bigger model — but you find that out while designing rather than when a user asks a question that requires a long answer and receives one cut off mid-sentence.
Choose a working budget below the model's advertised limit as well. Filling a large window is billable and slow: you pay for every token you send, prefill time rises with input length, and time-to-first-token is the latency your user experiences. A budget set by what the answer actually needs is usually far under the limit, and that is a feature.
A worked allocation
Take a model with a 128,000-token window and a deliberate working budget of 24,000 tokens, chosen because that is where cost and time-to-first-token stay acceptable for an interactive feature. Assume a system prompt, tool schemas and examples measuring 1,500 tokens, a maximum reply of 1,200 tokens, and retrieved passages averaging 500 tokens each with 200 tokens of citation metadata across the set.
| Line item | Tokens | Behaviour over a session |
|---|
| Working budget | 24,000 | chosen, not the window limit |
| Fixed prompt | 1,500 | constant |
| Answer reservation | 1,200 | constant |
| Retrieved passages, 8 at ~500 | 4,000 | constant by policy |
| Citation metadata | 200 | constant |
| Remaining for history | 17,100 | shrinks every turn |
If an average turn pair costs 250 tokens, history exhausts its 17,100 at around turn sixty-eight, and that number is now something you designed rather than something you will learn from an incident. It also tells you what to negotiate: dropping to five passages buys another twelve turns, and halving the reply reservation buys a similar amount at the cost of truncating long answers. Those are product decisions, and having the arithmetic in front of you is what makes them decidable.
Count tokens, not messages or passages
Budget in tokens measured with the tokeniser the target model actually uses, and measure rather than estimate. The characters-per-token rules of thumb are calibrated on English prose and break in the two places chat features live: JSON and code tokenise far more densely because punctuation and identifier fragments cost tokens individually, and non-Latin scripts can consume several tokens per character. A budget expressed as "the last twenty messages" or "the top eight chunks" has no idea how big those things are, so the same policy that fits comfortably for one user overflows for another whose documents are in Japanese.
The pre-flight check matters as much as the estimate. Compose the prompt, count it, and if it exceeds the budget, evict until it does not — then send. A system that trusts its own estimates will eventually send an over-length request, and the provider's error is a hard failure at the worst moment rather than a degraded answer.
Eviction order is where the judgement is
When something has to go, the order should be deliberate and should follow how much information each item carries per token. Verbatim tool outputs and long pasted blobs go first, replaced by a short note recording that they existed and what they were, because a 4,000-token API response is usually three useful facts. Old turn pairs go next, oldest first, and this is where a rolling summary earns its place: fold the evicted turns into a compact running summary of decisions, entities and constraints, so the thread retains its state without retaining its transcript. Retrieved evidence for the current question should be nearly last, because it is the reason the feature answers correctly at all. The system prompt and the most recent turns are not evictable.
Two details keep this from misbehaving. Always keep the current user message and the immediately preceding exchange intact, or the model loses the thread of a follow-up like "and the second one?" — the most common form of this bug. And treat the summary as lossy and monotonic: each re-summarisation compounds the loss of the last, so pin the facts you cannot afford to lose — identifiers, the user's stated goal, agreed constraints — into a structured slot that is carried forward verbatim rather than re-paraphrased. Anything that survives only inside prose the model rewrites every twenty turns will eventually not survive.
What to instrument
Log the composed size of each component per request, not just the total, because the total tells you nothing about who is squeezing whom. A dashboard of fixed-prompt, history, retrieval and reply tokens over session length shows the crossover point where retrieval starts being evicted, and that curve is what you tune against. Log eviction events with their category as well: a sudden rise in evicted retrieval is a quality regression that will otherwise reach you as vague complaints that the assistant has become less accurate.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).