Build the number from an identity, not a price list
Cost per prediction is a ratio: what you pay for the serving capacity over some interval, divided by the useful predictions that capacity delivered in the same interval. Everything difficult about the calculation sits in the denominator, because capacity is billed for whether you use it or not.
So work with your own bill rather than a published rate, and state each assumption as an assumption. Suppose your finance data says one inference instance costs C per hour all-in, including the instance itself, the storage attached to it, and its share of the cluster overhead. Call the sustained throughput of that instance under production conditions T predictions per second — measured with real feature vectors at your latency target, not from a synthetic benchmark. And call your average utilisation U, the fraction of that throughput you are actually using across the day.
The compute term is then C divided by 3600 × T × U. Put numbers in it to see what dominates. Take C as £2 per hour, T as 200 predictions per second, and U as 25 per cent, which is what a service provisioned for a four-times daily peak looks like. Delivered predictions per hour are 3600 × 200 × 0.25 = 180,000. The compute cost per prediction is £2 / 180,000 ≈ £0.0000111, or about £11 per million predictions.
Then notice the sensitivity. Hold everything else and raise utilisation to 50 per cent, and the figure halves to about £5.60 per million. Nothing about the model changed. That is the finding worth carrying into the conversation: for most inference services the unit cost is a statement about utilisation, not about the model.
The terms people leave out
Compute is rarely the whole marginal cost, and the omissions are systematic.
Feature retrieval is the big one. If each prediction performs three reads against an online store and that store bills per read or per request unit, the arithmetic is straightforward and it scales exactly with traffic, which makes it a genuinely marginal cost in a way the instance is not. At high volume this term routinely exceeds the compute term, and it is the one that appears on a different line of the bill from the model, which is why it goes unattributed. Add cache reads, which are cheaper per operation but not free, and any cross-zone data transfer if your feature store is not co-located with your serving pods.
Logging is next. If you log the served feature vector for every request — and you should, because it is how skew and drift are detected — you are paying for ingestion, storage and retention of one row per prediction. At a million predictions a day with a two-kilobyte vector that is two gigabytes a day of write volume, and retention for a year is a real storage line whose size follows directly from those two stated numbers.
Then the ancillaries: the load balancer, the model registry, the artefact storage, the monitoring pipeline that computes drift statistics over the logged vectors, and the training runs that produced the model. Those belong in the total cost of the model, and they behave very differently from the ones above.
Fixed and marginal, and why the distinction earns its keep
Split every line into two buckets by asking one question: if one more prediction were served this second, would this cost change?
Marginal costs answer yes — the accelerator time or CPU time the request consumes, the feature-store reads, the log row, the egress. Fixed costs answer no — the training runs, the experiment tracking, the registry, the on-call rota, the engineers, the baseline monitoring, and crucially the warm capacity you provisioned for peak and are not using at this moment.
That last item is why the distinction matters rather than being an accounting nicety. On a service running at 25 per cent utilisation, the marginal cost of one more prediction is close to zero, because the capacity is already paid for and idle. So the £11 per million derived above is a planning figure — the right number for forecasting what ten times the traffic will cost, or for comparing two architectures — and the wrong number for deciding whether to serve one more request. Confusing the two produces bad decisions in both directions: throttling a feature that is effectively free at current load, or assuming a tenfold traffic increase is free because the marginal request was.
It also tells you what a cost reduction has to attack. If eighty per cent of your monthly model spend is fixed, then making the model twice as fast reduces the bill by at most a tenth. Raising utilisation, consolidating models onto shared capacity, or shortening log retention are the levers with leverage, and you can only see that once the split exists.
Attribution across shared capacity
Once several models share an accelerator or a cluster, cost per prediction stops being derivable from the bill and needs instrumentation. Emit, per request, the model identity and version, the compute time consumed, the number of feature reads and the bytes logged. Aggregating that by model gives you a defensible allocation of shared capacity in proportion to consumed resource, which is both the input to the unit-cost figure and the thing that makes a cost regression diagnosable later.
Without it you have a total and a guess. With it you can answer the question that actually gets asked in a review, which is not what a prediction costs but which model got more expensive and when.
Connect it to the value of the decision
A unit cost is only interesting next to the value of what it decides. A prediction costing a hundredth of a penny is irrelevant if it routes a five-pound transaction and decisive if it ranks a free content tile served fifty times per page view. So the useful output of this exercise is a comparison: unit cost against expected value per prediction, per surface.
Where that comparison comes out badly, the fix is usually not a cheaper model. It is serving fewer predictions — cache the score for entities that have not changed, precompute for the population that will be scored anyway, skip the model where a rule settles the case, and stop scoring the ninety per cent of the catalogue that never gets seen. Reducing the denominator is the lever people reach for last and it is frequently the largest one available.
The number is dominated by utilisation, and most of what it contains is fixed. Derive it from your own throughput and utilisation, split it into fixed and marginal, and then use the marginal half for operational decisions and the whole for planning ones.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).