The bill is a sum of lines, and token accounting finds the one that moved
At this volume a doubled bill has many possible causes, and they need different fixes. Write the LLM cost of one request as a sum, per model:
cost per request =
uncached input tokens x input rate
+ cached input tokens x cached rate (plus any cache-write premium)
+ output tokens x output rate
then multiply by (1 + retry rate) and add calls you did not plan for
Token accounting means recording each of those quantities on every request, tagged with feature, prompt version and model. Most providers report cached and uncached input tokens separately in the usage data they return, so the data usually exists. What is missing is the habit of storing it per feature. Without that split, the only number you have is the invoice, and you cannot tell growth from regression.
A worked example in relative units
Prices vary by provider and change often, so model LLM cost with placeholder rates you replace from your own price card. Assume uncached input costs 1 unit per 1,000 tokens, cached input 0.1 units, and output 4 units. The output multiple reflects that output is priced at several times input. The cached discount varies by provider, so check yours.
Assume 2 million requests an hour. Each has a 4,000-token fixed prefix of system prompt, tool schemas and examples, then 2,000 tokens of retrieved passages and user turn, and a 300-token reply. Last month the prefix hit the cache 90 per cent of the time.
cached prefix 3,600 tokens x 0.1 / 1,000 = 0.36
uncached prefix 400 tokens x 1.0 / 1,000 = 0.40
variable input 2,000 tokens x 1.0 / 1,000 = 2.00
output 300 tokens x 4.0 / 1,000 = 1.20
total = 3.96 units per request
x 2,000,000 per hour = 7.92 million units per hour
Now apply plausible changes one at a time and compare each with 3.96:
| Change | New cost per request | Ratio |
|---|
| Traffic up 20 per cent | 3.96, but 20 per cent more requests | 1.20 |
| Replies grow from 300 to 600 tokens | 3.96 + 1.20 = 5.16 | 1.30 |
| Retrieval sends 4,000 tokens, not 2,000 | 3.96 + 2.00 = 5.96 | 1.51 |
| Cache hit rate falls to zero | 4.00 + 2.00 + 1.20 = 7.20 | 1.82 |
Combine the last row with a validation regression that pushes the retry rate to 10 per cent: 7.20 × 1.10 = 7.92 units, exactly double. A prompt edit that put today's date and the user's name on line one would do it. Every request now differs in its first few tokens, so no prefix matches, with no change in traffic at all.
Where else LLM cost hides
Retries multiply the whole request, not just the part that failed. A repair loop that resends the prompt after a schema failure is a surcharge on every token above.
Hidden callers come next. An LLM judge added to grade a sample of outputs, an eval suite now running on every commit, a batch job someone pointed at the interactive endpoint, or an agent whose average step count crept up: none of these show up in the main feature's request count. For agents, each step resends the growing transcript. More steps raise input cost faster than linearly, because each step resends everything before it.
Abandoned streams are the quiet one. A user who navigates away halfway through a reply still pays for whatever was generated before the stream was cancelled, or for all of it if cancellation never reaches the model call.
What to cut, in order of risk
Start with the LLM cost fixes that cannot change answer quality. Restore the prompt layout so stable material comes first and per-request material comes last, then alert on cache hit rate per prompt version. Fix whatever is driving retries. Make sure stopped and abandoned streams cancel the upstream call. In the example, the cache and retry fixes alone take 7.92 back to 3.96.
Next come cuts that need an eval run but are usually safe. Set an explicit output cap and ask for the shape you need, such as three named fields rather than an explanation containing them. Trim retrieval to the number of passages your recall measurements justify; in the example, halving the variable input saves 1.00 unit, a quarter of the bill. Move work nobody waits for, like nightly summaries and backfills, to a provider's asynchronous batch tier if one exists, since these are usually discounted in exchange for a completion window of hours. Cache whole responses only for exact repeats. A semantic cache that returns the answer to a similar question can return a wrong one.
Model routing comes last, because it changes quality and adds a second system to evaluate.
The cascade arithmetic most answers skip
A cascade sends every request to a cheap model first and escalates the ones it cannot handle. Assume the cheap model costs one fifth of the large one per request, and a fraction e escalates. Every request pays for the cheap call, and escalated ones pay for both:
relative cost = 0.2 + e x 1.0
e = 0.3 -> 0.5 half the original bill
e = 0.6 -> 0.8
e = 0.8 -> 1.0 break-even: all the complexity, none of the saving
So the claim "route to a smaller model" is only an argument once you know the escalation rate on real traffic. A cheap model that is confidently wrong also never escalates, which saves money by losing quality. Choosing between a large model and a small one with retrieval covers how to evaluate that second system. If you host the model yourself, the equivalent lever is utilisation, which batching controls.
Cutting before attributing
The weak answer jumps to the most visible change: switch to a cheaper model, or shorten the system prompt. In the example, shortening a cached 4,000-token prefix by a quarter saves 0.19 units per request at a 90 per cent hit rate: 0.09 on the 900 cached tokens and 0.10 on the 100 that miss. That is under 5 per cent of the bill, for a change that risks behaviour. The strong answer asks for the per-line breakdown first and finds that the cache stopped working. It then fixes the cause and adds the alert that would have caught it on day one. The forecast side of the same arithmetic is in estimating an LLM feature's cost.
A doubled LLM bill can be a regression rather than growth, and only a per-line breakdown tells you which one you have.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).