Decompose the budget before optimising anything
Fifty milliseconds is not a model requirement, it is a sum. Write the sum out before touching anything, because the term people optimise is usually not the term that dominates.
A realistic decomposition for a single online prediction has five parts: network and serialisation to reach the model service, feature retrieval, any transformation applied to those features, the forward pass, and the response hop back. Measure each. The common finding is that the forward pass for a gradient-boosted tree ensemble or a small neural network is a small fraction of the total, and feature retrieval is most of it. A model that runs in two milliseconds behind twenty-eight milliseconds of feature fetching does not have a model latency problem.
Once you have the split, the budget can be allocated deliberately: give the fetch path a hard ceiling, give the model whatever remains after reserving the network hops, and keep a few milliseconds of slack so that the total does not breach on a bad garbage-collection pause.
Fan-out is what makes the tail bad
Sequential lookups add. Five feature lookups at twelve milliseconds each, executed one after another, is sixty milliseconds and the budget is gone before the model runs. Issuing them concurrently reduces the mean to roughly the slowest single call, and that is the first fix — but it changes the tail in a way worth being explicit about.
When you fan out to five independent lookups and wait for all of them, your latency is the maximum of five samples, not one. Suppose each lookup independently has a one per cent chance of exceeding twenty milliseconds. The probability that at least one of the five does is one minus 0.99 to the fifth power, about 4.9 per cent. Your p99 per-call dependency has become roughly a p95 problem at the request level. Fan-out converts a rare slow call into a common slow request, and that is why request p99 so often looks nothing like the p99 of any component.
The mitigations are ordinary once the mechanism is clear. Reduce the number of round trips by batching keys into one multi-get where the store supports it. Make optional features genuinely optional, so a slow lookup can be abandoned and the model receives its declared missing-value representation rather than the request waiting. Hedge only where the backend is idempotent and you can afford the extra load, because hedging trades tail latency for throughput.
Caching, and what is safe to cache
Two things are cacheable and they behave differently. Features that change slowly — a customer's tenure band, a merchant's category, an item embedding — can be cached aggressively with a time-to-live measured in minutes or hours, and the cache hit rate does most of the work. Whole predictions can also be cached, but only where the input space is small and repetitive enough for a hit rate to materialise, which usually means the entity does not change within the window.
The decision that needs stating out loud is how stale a cached feature may be, because a cache time-to-live is a freshness decision dressed as an infrastructure setting. A five-minute cache on a fraud velocity counter defeats the counter. A five-minute cache on account age is free. Set the time-to-live per feature, from the same reasoning that decided the serving mode, and record it next to the feature definition so it is reviewable.
Making the model itself cheaper
If the forward pass really is the dominant term, there are three levers and they trade differently.
Quantisation reduces the numeric precision of weights and activations. The arithmetic is simple: an eight-bit weight occupies a quarter of the space of a thirty-two-bit one, so a model holding four gigabytes of fp32 weights holds about one gigabyte in int8. That shrinks memory traffic, which is what most inference is bound by, and it usually costs a small amount of accuracy that you must measure on your own evaluation set rather than assume. Post-training quantisation is the cheap version; quantisation-aware training recovers more of the accuracy at the cost of a training run.
Distillation trains a smaller model to reproduce the larger model's outputs, which can be a much bigger win than quantisation because it changes the architecture rather than the representation, and a much bigger project.
Batching raises throughput per accelerator but adds latency, because a request waits for its batch to fill. Dynamic batching with a maximum wait of a few milliseconds is often worth it; a batching window comparable to your whole budget is not. This is the one lever that can make your latency worse while making your dashboard look better, because throughput improves.
A timeout without a fallback is just a different failure
Set the client timeout below the budget, not at it — if the budget is fifty milliseconds and you time out at fifty, you have guaranteed that a timeout also breaches the budget. Time out at, say, thirty-five, leaving room for whatever you do next.
What you do next is the part candidates leave out, and it is where the strong answer lives. A timeout has to resolve to a decision, because the calling service still owes someone a response. So the design needs a named degraded path: a cached previous score for this entity, a population-level default, or a simple rule set that stands in for the model. Each of those has a different business consequence, and choosing between them is not an engineering decision alone — someone who owns the outcome has to agree that approving a transaction on a rule when the model is unavailable is preferable to declining it, or the reverse.
Two things then need instrumenting. The fallback rate must be a first-class metric with an alert, because the failure mode of a good fallback is that it works quietly for a week while the model serves nothing. And the fallback path needs to appear in your evaluation, because a path that handles two per cent of traffic and has never been measured is an unmeasured two per cent of your decisions.
The budget is a sum, and the term that dominates it is usually feature retrieval rather than the model. Optimise the sum you measured, and make the timeout resolve to a decision somebody has agreed to.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).