Before we build this, how do you estimate what the LLM feature will cost to run?

Build the estimate from assumptions you can be wrong about in public

A cost estimate for an LLM feature is a short arithmetic exercise with a small number of inputs, and the value of it is not the total. It is that each assumption becomes a named, checkable quantity, so when the bill arrives at three times the forecast you can say which assumption was wrong instead of concluding that LLMs are expensive.

Five quantities are enough to start: requests per day, input tokens per request, output tokens per request, the retry and overhead multiplier, and the two unit prices. Everything else is refinement.

Take a worked example, with the unit prices as placeholders you replace from the provider's current price card — never quote rates from memory, they change. Assume a support-assistant feature at 20,000 requests per day. The composed prompt is 1,500 tokens of fixed system material, 4,000 tokens of retrieved passages and 2,500 tokens of conversation history, so 8,000 input tokens. Replies average 400 tokens. Assume the card says $3 per million input tokens and $15 per million output tokens.

Input: 20,000 × 8,000 = 160 million tokens per day, so 4.8 billion over a 30-day month. At $3 per million that is 4,800 × $3 = $14,400.

Output: 20,000 × 400 = 8 million tokens per day, 240 million per month. At $15 per million that is 240 × $15 = $3,600.

Total around $18,000 per month, or $0.03 per request. Now notice the shape of it: output is 5 per cent of the tokens and 20 per cent of the cost. That ratio is the thing to internalise, because it inverts the intuition that a long prompt is the expensive part.

Why output tokens cost several times input

The price gap is not a commercial preference, it reflects how inference runs. Serving a request has two phases with different characteristics.

Prefill processes the entire prompt in one pass. Because all the input positions are known up front, the work is a batch of large matrix multiplications over the whole sequence, which saturates the accelerator's arithmetic units and gets excellent utilisation. Thousands of input tokens can be absorbed in the time it takes to do a handful of big multiplies.

Decode produces one token per forward pass, and cannot do otherwise, because the next token depends on the one just sampled. Each of those passes has to stream the model's weights and the growing key-value cache out of memory to compute a single position, so the work per token is dominated by memory bandwidth rather than arithmetic and the hardware sits well below its compute ceiling. Generating a thousand tokens means a thousand of those passes. Batching across concurrent requests is what makes it economic at all, and it is why output remains several times dearer per token however the batching is tuned.

The design consequences follow immediately. Verbose output is the most expensive habit you can build in: asking the model to restate the question, echo the input, or explain itself at length is billed at the output rate. Extractive tasks that return identifiers or short structured fields are cheap; tasks that regenerate a document you already have are not, and rewriting only the changed portion is a real optimisation rather than a micro-one.

The multipliers people leave out

The naive estimate is almost always low, and predictably so.

Retries and repair loops multiply the whole request, not the delta. A validation failure that triggers one repair attempt resends the entire prompt, so a 10 per cent repair rate is a 10 per cent surcharge on input, and an unbounded loop is unbounded cost. Streams that the user abandons are billed for the tokens already generated, so a feature where people read the first two lines and navigate away pays for the other forty.

Agent loops are the big one, because the transcript is resent at every step. A tool-using agent that takes eight steps does not send 8,000 tokens once; it sends a prompt that grows with each observation, so the total input is closer to the sum of a growing series than to eight times the first prompt. If each step adds 800 tokens of tool output to an 8,000-token base, the eight requests total roughly 8 × 8,000 plus the accumulating observations, and the growth term is what surprises people. Modelling an agent feature with a single-request estimate understates it badly.

Then there is everything around the request: embedding the corpus at ingestion and re-embedding it when you change models, evaluation runs over your test set on every prompt change, and LLM-as-judge calls, which are themselves LLM calls at the same rates. A guarded feature that judges every output has doubled its request count by definition.

Prompt caching changes the arithmetic, with conditions

In the example above, 1,500 tokens of fixed prompt sit at the front of every request, and if the retrieved set is stable within a session more may be cacheable too. Providers bill cached prefix tokens at a materially reduced input rate, so if 5,500 of the 8,000 input tokens are a byte-identical prefix and you achieve an 80 per cent hit rate, the majority of the input line moves onto the cheaper rate and the input estimate falls substantially — which, since input was 80 per cent of the bill, moves the total more than any prompt-shortening exercise would.

Model it as a hit rate rather than a certainty, and check three things against the provider's documentation rather than assuming: whether cache writes carry a premium over normal input, how long an entry survives without use, and what granularity the match requires. All three decide whether the saving is real. Low-traffic tenants can lose money on caching if entries expire between requests and every write is billed at a premium, and a prefix that varies — a timestamp, a user's name interpolated at the top — has a hit rate of zero no matter how much of the prompt below it is identical.

Finish with a sensitivity check

Vary each assumption and see what moves. In this example, doubling the volume doubles everything, halving the retrieved passages removes about a quarter of the input line, and halving the reply length saves $1,800. The retrieved-passage count is the largest single lever and the cheapest to change, which tells you where to spend the first day of tuning. If instead the numbers had come out output-dominated — a summarisation or drafting feature, where replies run to thousands of tokens — the priorities would reverse and reply length would be the only thing worth optimising.

Say plainly which number you are least sure of, and how you would find out cheaply. Usually it is tokens per request, because nobody knows what the retrieved set really measures until it exists; a half-day spike that composes real prompts over real documents and counts them with the provider's tokeniser replaces the shakiest assumption in the model with a measurement.

The estimate is not the deliverable. The named assumptions are, because they are what make the eventual invoice explainable.