Loading...
Loading...
Browse 3 real-world technical and behavioral interview questions about Autoscaling. Review scenarios, edge cases, and architectural best practices.
Runner autoscaling trades idle compute cost against queue time, and the two levers that decide the trade are a warm pool sized to your usual concurrency and a scale-down delay long enough to absorb the next job without a fresh cold start.
Scale on queue depth or in-flight concurrency rather than request rate, because a GPU replica that takes minutes to load weights cannot be added after the queue has already formed. Then decide deliberately how much warm capacity to keep, since that is what you are paying to avoid cold starts.
Reactive scaling has a latency of its own — metric window, evaluation, provisioning, warm-up, health checks — and a surge that arrives faster than that latency is served entirely by the capacity you already had. The fix is a scheduled floor plus a saturation-based signal, not a more aggressive policy.