Why the global limit did not fire
Suppose the service handles 10,000 requests per second and the global limit is set there. Normally forty tenants send 100 rps each. One bursts to 4,000 rps and total load reaches 13,900 — above capacity, so latency climbs for everyone, but the limiter only rejects the excess above 10,000 and does so indiscriminately. Every tenant's requests are equally likely to be shed, and the bursting tenant, sending forty times more, wins forty times more of the surviving capacity.
The limiter did exactly what it was built to do. It was simply never asked the fairness question.
Rate is not the only axis
The instinct is to add a per-tenant request rate, and that is necessary but insufficient. Consider two tenants each sending 100 rps: one requests a cached lookup taking 2ms, the other a report that takes 4 seconds and holds a worker thread throughout. Identical rates, wildly different consumption — the second tenant occupies 400 concurrent workers while the first occupies less than one.
So the design needs both:
A token bucket per tenant for the sustained contract. Tokens refill at the tenant's contracted rate; bucket depth defines how large a burst is tolerated. Depth is the parameter people leave at a default and should not — a depth equal to one second of refill forbids any burst and breaks legitimate batch clients; a depth of sixty seconds lets a tenant sit idle for a minute and then dump a minute's traffic instantly, which is precisely the incident in the question.
A concurrency limit per tenant for the resources rate cannot see. A simple semaphore of N in-flight requests per tenant caps how much of the worker pool any one tenant can hold, regardless of how cheap or expensive their requests turn out to be. This is the control that actually protects tail latency, and it is the one most often missing.
Weighting requests by expected cost — charging a report ten tokens and a lookup one — is the refinement that unifies the two, and it is worth mentioning as the more precise version once the basic shape is in place.
The part usually missed: the queue
Admission control decides who gets rejected. It does not decide who waits behind whom, and that is what the other tenants actually experienced.
With a single FIFO queue, 4,000 queued requests from the bursting tenant sit in front of the next request from every other tenant. Those tenants are inside their limits, are not being rejected, and are still waiting behind someone else's backlog. Their latency doubled without a single 429 being issued.
The fix is a queue per tenant with round-robin — or weighted fair — dequeuing:
┌── tenant A ──┐ ┌─ 1 request ─┐
├── tenant B ──┤ →│ each round │→ workers
├── tenant C ──┤ └─────────────┘
└── tenant D ──┘ (D's 4,000-deep backlog drains one per round,
so it queues behind itself, not in front of A, B, C)
Each tenant's backlog delays only that tenant. This is the same idea as fair queueing in network schedulers, and naming that lineage is a good signal. Per-tenant queues should also be bounded and shed load when full, so a burst produces fast rejections rather than an unbounded memory-resident backlog.
Distributed counters and the honesty about exactness
Thirty pods each enforcing "100 rps per tenant" locally permits 3,000 rps if traffic is balanced — and a lower effective limit if it is not, since a tenant whose requests land unevenly gets throttled on the busy pod while quota sits unused elsewhere. Both errors are real and neither is acceptable to explain to a customer.
The options are worth stating with their costs. A central store — Redis with an atomic script implementing the bucket — is exact and adds a network hop plus a hard dependency on the critical path. Local buckets with periodic reconciliation, where each pod is given a share and pods exchange usage every few hundred milliseconds, is approximate, resilient, and the common production choice. Consistent routing of a tenant to a subset of pods makes local counting correct but concentrates that tenant's load and complicates rebalancing.
The reasonable default is approximate local enforcement with a global backstop, because being 10% wrong on a rate limit is nearly always cheaper than adding a synchronous dependency to every request.
Do not waste idle capacity
Strict per-tenant caps are simple and wasteful: if thirty-nine tenants are quiet, the fortieth is still throttled to its share of a mostly idle service. A work-conserving design lets a tenant borrow unused capacity while reclaiming it quickly when others return — soft limits that only bind under contention, or a two-tier scheme where a tenant's guaranteed share is reserved and everything above it is best-effort and shed first.
That distinction is what to communicate to customers too: the guaranteed rate is a floor they can build against, and the burst above it is available but not promised.
What the client sees
The limiter is part of the API contract, so it needs to be legible. Reject with 429 Too Many Requests and a Retry-After header, and publish the budget on every response — remaining tokens, limit, and reset time — so a well-behaved client can pace itself instead of discovering the limit by hitting it. Distinguish the tenant's own limit from a global overload, since the correct client behaviour differs: back off and slow down in the first case, retry with jitter in the second.
Rolling this out to existing customers is a migration, not a switch. Log what each tenant would have been limited to for a couple of weeks, find the ones who would break, raise their limits or talk to them, and only then enforce. Turning on fairness without that step converts an incident affecting one tenant into an incident affecting the customers who were quietly over the line all along.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).