Throttling algorithms, circuit breakers, retries with jitter, bulkheads, and graceful degradation.
7 questions in System Design.
Rate limiting an LLM gateway by request count fails because cost is tokens, and output tokens are unknown at admission. Budget tokens per tenant, reserve input plus expected output, settle against reported usage when the stream ends, cap concurrent streams, and cancel upstream on disconnect.
Redis gives twenty gateways one counter mutated atomically through INCR or a Lua script, so rate limiting is fleet-wide. What breaks first is usually latency, not throughput: a slow Redis stalls every request for its timeout. Hot keys, caller clocks inside scripts and counts lost on async failover follow.
A global limit protects the service from overload but says nothing about how capacity is divided, so one tenant can consume most of it while staying under the ceiling. Fairness needs per-tenant accounting: a token bucket for the steady-state contract, a concurrency limit so expensive requests cannot hog workers, and round-robin queueing so a burst queues behind itself.
No retry policy or circuit breaker helps here, because the change was valid and the defect was already deployed. What bounds the damage is treating configuration as a rollout with stages and bake time, a kill switch whose own dependencies cannot fail with it, and a measured rollback time.
Analyzing architectural resilience patterns, including circuit breakers, bulkheads, adaptive load shedding, and the mitigation of retry storms. Use this distributed systems answer to show the decision, trade-off, and evidence rather than a memorised definition. It also connects resilience to the point an interviewer is testing.
Global rate limiting fails when each region owns a local counter, because five regions each allowing 100 requests creates a 500-request ceiling. Use one atomic shared counter, route each user to an owning region, or lease regional budget deliberately. It also connects distributed counters to the point an interviewer is testing.
Fixed windows permit twice the limit across a boundary; the two sliding windows fix that at a memory or an accuracy cost; token bucket permits a bounded burst and leaky bucket smooths output instead. Across a fleet the counter must sit in one store mutated atomically, or each node silently enforces its own limit.