Your inference bill tripled last month and traffic was flat. How do you find out why?

Restate the problem as an arithmetic identity

The bill is requests multiplied by cost per request. Traffic is flat, so requests did not change, so cost per request rose by a factor of three. That reframing is worth stating out loud because it eliminates half the hypotheses immediately and directs the whole investigation at one question: what does each request cause to happen now that it did not cause before?

Before hypothesising, get the shape of the curve. A step on a single day points at a deployment, a configuration change or a flag rollout, and you can find it by intersecting the cost curve with your change log. A gradual ramp over the month points at something growing with data rather than with traffic — a cache hit rate decaying as key cardinality grows, a retention window filling, a candidate set expanding as the catalogue does. A curve that steps up and stays flat at the new level, with no corresponding step in any traffic metric, is the classic signature of capacity that was added and never removed.

Also confirm that traffic really is flat at the level that matters. Requests to your service can be flat while predictions are not: a batch endpoint whose average batch size grew, or a page that started calling the ranker four times instead of once, both hold request count constant while tripling the work. Check predictions served, not just HTTP requests.

A larger model behind an unchanged endpoint

The first place to look, because it is both the most common cause and the easiest to miss from the outside. The endpoint is unchanged, the contract is unchanged, the client code is unchanged — and the model behind it is now three times the size, or runs at a longer sequence length, or lost its quantised variant when someone rebuilt the artefact from the training checkpoint.

The evidence is a rise in compute time per request with no change in request count, so look at accelerator or CPU milliseconds per prediction over the month, broken down by model version, and correlate the step against the registry's promotion history. If per-request compute tripled on the day version 12 was promoted, you have your answer in one query.

Adjacent causes in the same family: batching disabled or its window shortened, so throughput per device falls; a precision change from int8 back to fp16 or fp32; a runtime upgrade that lost a fused-kernel optimisation; and a change from a single model to an ensemble of three, which is a threefold cost increase that reads as an accuracy improvement in the pull request.

Feature-fetch amplification

The second family is work the model causes rather than work the model does. Count the downstream calls per prediction and compare against the previous month, because this number grows quietly.

A new feature was added and it needs a lookup in a different entity group, so reads per prediction went from three to five. A multi-get was refactored into a loop, so one call became forty. A cache time-to-live was shortened for a correctness fix, so the hit rate fell from ninety per cent to fifty and the reads reaching the backing store went from one in ten to one in two — a fivefold increase in store cost from a change whose description said nothing about cost. A feature store was moved to a different availability zone, so every read now crosses a zone boundary and attracts transfer charges that did not exist before.

The arithmetic here is unforgiving because it multiplies. Reads per prediction rising from three to five is 1.67 times. A cache hit-rate fall taking backing-store reads up fivefold on top of that is 8.3 times on that line of the bill. Two innocuous-looking changes can produce a cost outcome neither author would have predicted, which is exactly why reads per prediction deserves to be a monitored metric and not a thing you count during an investigation.

Retry storms and self-inflicted load

Retries turn one client request into several downstream requests, and they are invisible at the edge because the edge sees one request.

The mechanism to check: a dependency became slower, timeouts started firing, clients retried, and each retry re-executed the whole chain including the feature fetches and the model call. If two per cent of requests fail and each is retried up to three times, calls rise only modestly. If a dependency's error rate reached thirty per cent and retries are unbudgeted, and the retrying client is itself behind another retrying client, the multiplication compounds per layer — two layers of three attempts is up to nine executions of the innermost call for one user action.

Hedged requests do the same thing by design: sending a second attempt after a delay to improve tail latency costs extra load in proportion to how often the hedge fires, and a hedge threshold set below your p50 fires on most requests. Look for the ratio of downstream calls to edge requests and for the share of predictions that were later discarded because a duplicate answer arrived first. Discarded work is pure cost and it appears nowhere in a success-rate dashboard.

flowchart TD
  A[Bill tripled<br/>traffic flat] --> B[Cost per request rose]
  B --> C[Compute per prediction<br/>by model version]
  B --> D[Downstream calls<br/>per prediction]
  B --> E[Provisioned but idle<br/>capacity]
  C --> F[Correlate against<br/>registry and deploy log]
  D --> F
  E --> F

The three middle branches are the only three places the extra money can be, which is what makes this tractable rather than open-ended.

Capacity that is running and serving nothing

The last family produces no signal anywhere in the application, because nothing is happening. A shadow deployment set up to evaluate a candidate against live traffic and left running afterwards doubles inference compute while every latency and error metric stays perfect. A load test's capacity was never scaled back. An autoscaler's minimum replica count was raised during an incident and never lowered. A GPU node pool was created for a migration and the old one was not deleted.

Find these by comparing provisioned capacity against delivered predictions over the month rather than by reading application metrics. If instance-hours rose while predictions per instance-hour fell proportionally, you are paying for idle, and the cause is configuration rather than code. This family is also the most cheaply prevented: an expiry on shadow deployments, a budget alert per environment, and periodic reconciliation of provisioned capacity against a declared expectation.

The instrumentation that makes this a query rather than an investigation

Everything above is reconstruction after the fact, and the reason it takes a week is that cost data and request data live in different systems with no shared key. The fix is per-request cost attribution, and it is a modest amount of work.

Emit, on every prediction, the model name and version, the compute time consumed on the device that served it, the number of downstream feature reads, whether each was a cache hit, the retry or attempt number, whether the result was discarded, and the calling surface. Then combine that with the hourly cost of the capacity each model ran on to produce cost per prediction by version, by surface and by tenant. With that in place, "which model version got more expensive on which day" is a single grouped query, and a threefold rise is caught by an alert on cost per prediction the day it happens rather than by a finance review a month later.

The rule that generalises: alert on cost per unit of work, not on the total. A total rising with traffic is a business succeeding. A total rising with flat traffic is a defect, and only the ratio distinguishes them.

Flat traffic and a tripled bill is not a capacity question, it is a per-request question, and it has exactly three answers: each request buys more compute, causes more downstream calls, or you are paying for capacity that no request is using.