Design an autoscaling scheme for a self-hosted CI runner fleet. What are you actually trading off, and how do you decide the numbers?

Short answer

Runner autoscaling trades idle compute cost against queue time, and the two levers that decide the trade are a warm pool sized to your usual concurrency and a scale-down delay long enough to absorb the next job without a fresh cold start.

The real constraint is cold-start time, not capacity

Autoscaling a CI runner fleet looks, at first glance, like any other autoscaling problem: watch a queue, add capacity when it grows, remove it when it doesn't. What makes CI runners different is that a runner is rarely useful the instant it exists.

Provisioning a fresh VM or container, pulling a base image, installing whatever the job's environment needs and registering with the CI control plane can take anywhere from tens of seconds to several minutes depending on the image and the platform, and that delay lands directly on a developer waiting for feedback on a pull request. A queue-based autoscaler that reacts only after jobs pile up guarantees that the first jobs in every burst pay the full cold-start cost, because there was nothing warm to hand them.

This reframes the actual design question. It isn't "how do I add capacity when load rises" — it's "how much capacity do I keep warm, doing nothing, so the common case never pays for a cold start, and how do I handle the uncommon case when warm capacity runs out."

Sizing the warm pool against typical concurrency, not peak

A warm pool is a fixed number of runners kept alive and idle, ready to pick up a job the instant it's queued. Sizing it too small means most bursts still hit a cold start; sizing it too large means paying for compute that sits idle most of the day, which is the whole cost story stakeholders will ask about first.

The workable heuristic is to size the warm pool to typical concurrent demand — the number of jobs usually running at once during normal working hours — and let a burst above that trigger on-demand scaling that accepts the cold-start cost as the price of handling the tail. This deliberately accepts slower feedback during a genuine spike, such as a mass rebase after a long-lived branch merges, in exchange for not paying idle cost for capacity that would only be used during rare spikes.

Typical concurrent jobs, working hours     : ~40
Warm pool size                             : 40, always on, near-zero cold start
Burst above 40 (e.g. mass rebase, 300 PRs) : scale on demand, accept cold start
                                              for jobs beyond the warm pool

The scale-down delay is the second lever, and it's counterintuitive

Once a burst subsides, the instinct is to tear down the extra capacity immediately to stop paying for it. That instinct produces a specific bad pattern: a runner finishes a job, gets torn down within seconds because the queue is momentarily empty, and thirty seconds later a new job arrives and pays a full cold start for capacity that had just been destroyed.

The fix is a scale-down delay — keeping a runner alive and idle for some minutes after its last job before terminating it — sized against how bursty your actual traffic is, not against a desire to minimise idle spend. A team merging in bursts throughout the day, where a five-minute gap between pushes is normal, needs a delay measured in minutes; a team with genuinely sparse, isolated jobs can tear down faster without much cost. Getting this number right is mostly a measurement exercise: look at the actual gap distribution between consecutive job arrivals and set the delay to cover the bulk of that distribution, accepting that the tail will still pay for a cold start.

flowchart TD
    A[Job queued] --> B{Idle runner<br/>in warm pool}
    B -- Yes --> C[Assigned immediately]
    B -- No --> D[Provision new runner<br/>cold start cost paid]
    D --> C
    C --> E[Job completes]
    E --> F[Runner idle for<br/>scale-down delay]
    F --> G{New job arrives<br/>within delay}
    G -- Yes --> C
    G -- No --> H[Runner terminated]

Separate pools by resource profile

A single undifferentiated pool works only when every job has roughly the same resource needs. In practice, most fleets have a handful of profiles: lightweight lint and unit-test jobs that finish in under a minute, integration-test jobs needing a database sidecar, and occasionally a resource-heavy job needing significantly more memory or a specialised accelerator. Provisioning every job against the largest profile wastes capacity on the common case; provisioning against the smallest profile means the heavy jobs queue behind lightweight ones or fail outright.

Splitting the fleet into pools keyed by job type, each with its own warm-pool size and scale-down delay tuned to that type's actual demand pattern, avoids both problems at the cost of some operational complexity: you now have several pools to size and monitor instead of one.

The measurement that tells you whether any of this is working is job wait time broken out by whether the job hit a warm runner or triggered a cold start, tracked as its own metric rather than inferred from cost dashboards. A scheme that looks cheap in the cost report but shows a rising share of cold starts is quietly making every developer's feedback loop worse while looking like a success everywhere else.

Autoscaling a CI fleet is really deciding how much idle cost you're willing to pay to avoid cold starts, and the honest way to set that number is against your measured burst pattern, not against a target utilisation percentage.

© 2026 Preptima. Originally published at preptima.com.