How would you autoscale a GPU inference service?

The cold start is a sum you can measure

Autoscaling a stateless web service works because a new replica is useful within a second or two. A GPU inference replica is not, and the whole design follows from how long it takes to become useful. Break that time into its parts, because each has a different fix.

Scheduling comes first: if no GPU node is free, the pod waits for a node to be provisioned, which is cluster-level capacity rather than pod-level scaling. Then the container image is pulled, and inference images carrying CUDA libraries and a framework are large. Then the model weights are fetched from object storage. Then the weights are loaded into GPU memory and any compilation or kernel autotuning happens, and only then does the replica pass a readiness probe.

The weight transfer is the term worth doing arithmetic on, because it scales with model size and network throughput and nothing else. Take a twenty-gigabyte checkpoint. At an effective one gigabit per second, that is 160 gigabits over 1 Gb/s, roughly 160 seconds. At an effective one gigabyte per second it is about twenty seconds. Two orders of magnitude of design consequence sit in that one assumption, which is why measuring your actual object-store throughput from a fresh node is the first thing to do rather than the last.

The fixes follow the decomposition. Bake weights into the image, or into a pre-warmed node-local volume, so the pull and the fetch collapse into one. Keep a pool of pre-provisioned GPU nodes so scheduling is not on the critical path. Load lazily only the parts of the model you need. None of this makes the replica instant; it makes the number small enough to plan around.

Why request rate is the wrong signal

Request rate is a lagging indicator of saturation. It tells you how much work arrived, not whether you can absorb it, and it crosses a threshold at the same moment your existing replicas begin to fall behind. If a replica needs three minutes to become ready, a scaling decision taken when the queue starts growing arrives three minutes after users started waiting — and by then the queue has grown by three minutes of surplus arrivals, which the new replica must now drain on top of current traffic.

Queueing makes this worse than linear. For a single server with random arrivals, mean waiting time in queue is the service time multiplied by utilisation over one minus utilisation. At fifty per cent utilisation that factor is one — an average request waits about as long as it takes to serve. At eighty per cent it is four. At ninety-five per cent it is nineteen. The interval between "comfortable" and "queue exploding" is a small change in arrival rate, and a lagging signal cannot cover it.

So scale on something that leads. In-flight concurrency per replica, or queue depth, or time-in-queue, all move before latency does and all express the thing you actually care about: how much unabsorbed work exists right now. If your serving stack exposes a batch queue, its depth is the best single signal available, because it is the literal backlog. Whatever the signal, use a metric normalised per replica so the target does not have to be recomputed every time the fleet resizes.

flowchart TD
  A[Arrivals rise] --> B[Queue depth grows]
  B --> C{Depth above target}
  C -->|yes| D[Request new replica]
  D --> E[Node schedule<br/>image pull<br/>weight load]
  E --> F[Replica ready]
  B --> G[Latency rises<br/>and requests time out]
  G -.->|users already affected| F

The dotted edge is the point: the path through latency completes before the path through readiness, so anything that waits for latency to move has already lost.

Warm capacity is a purchase, not an oversight

Because the replica cannot arrive in time, you keep spare capacity warm. The honest framing is that you are buying insurance and you should know the premium.

Suppose steady-state demand needs eight replicas and you hold two idle so a burst is absorbed while scaling catches up. You are running ten to do the work of eight, so the serving bill is twenty-five per cent above its floor. Whether that is right depends on the cost of the alternative: if a burst without headroom means a minute of timeouts on a checkout path, twenty-five per cent is cheap; if it means a background scoring job finishes late, it is waste. Put the number in front of whoever owns the outcome rather than deciding it silently.

Two refinements make the premium smaller. Scheduled scaling handles demand you can predict — a diurnal cycle, a marketing send, a market open — and pre-warming ahead of a known peak costs nothing at the trough. And where several small models must be served, sharing one GPU across them, or keeping several models resident in one replica's memory, raises utilisation without raising the number of accelerators.

Scaling to zero deserves a direct answer rather than avoidance. It is right when the first request after idle can tolerate the full cold start you measured — internal tools, batch-ish endpoints, evaluation services. It is wrong for anything user-facing, unless you can serve the first requests from a small always-on CPU fallback while the GPU replica loads.

When scaling cannot arrive in time

There is a class of burst no autoscaler can serve, and the answer is not a faster autoscaler. If demand triples in ten seconds and readiness takes three minutes, some requests will not be served well, and the only decision left is which ones.

That makes admission control part of the scaling design rather than a separate concern. Bound the queue: an unbounded queue does not prevent failure, it converts a fast rejection into a slow timeout after the client has already given up, while the GPU spends its time on work nobody is waiting for any more. Reject or shed beyond a depth derived from your latency budget — if the budget is two seconds and a request takes fifty milliseconds on the accelerator, a queue deeper than about forty per replica is holding work that will breach the budget by the time it runs. Shed by priority where the traffic has a priority, so paid traffic survives and speculative prefetching does not.

The related trap is self-inflicted load. A client that retries on timeout multiplies arrivals exactly when the service is saturated, and a queue-depth autoscaler will faithfully scale up in response to a storm it is also feeding. Retry budgets and circuit breakers on the caller are therefore part of the serving design, and it is worth checking whether a scale-up event was caused by real demand or by your own retries before concluding you need more GPUs.

The whole design is set by one measured number: how long a replica takes to become ready. That number decides your scaling signal, how much warm capacity you buy, and where you have to shed instead.