Where the hundred times comes from
Take two requests through the same gateway. Request A is a classification prompt: 200 input tokens and 50 output tokens, 250 in total. Request B is a document summary: 20,000 input tokens and 5,000 output, 25,000 in total. Both count as one request. One costs a hundred times the other.
Price weighting widens the spread further. Assume output tokens are priced at four times input tokens, which is the shape of many published price lists, though the ratio varies by provider and model. Request C, a long-form draft with 500 input and 4,000 output tokens, is 4,500 raw tokens but 500 + 16,000 = 16,500 input-equivalent tokens. A budget in raw tokens undercharges output-heavy traffic by more than three times. Where models with different prices share a gateway, budget in a weighted unit or in currency, not in raw tokens.
The cost is unknown when you have to decide
This is the part that separates rate limiting answers. Input tokens can be counted before admission, with the model's tokenizer where it is published or the provider's token-counting endpoint where it is not. Output tokens cannot be known until generation stops. The request's max_tokens parameter gives an upper bound, nothing more.
Charging nothing up front lets a tenant start fifty large generations at once and overspend before any of them finishes. Charging the full upper bound is safe and wasteful. Suppose a tenant has 100,000 tokens a minute, sends 2,000 input tokens per request, and sets max_tokens to 8,000 out of habit while typically using 300. Reserving the bound charges 10,000 per request, so the tenant is admitted 10 requests a minute. Its real usage of about 2,300 each would allow roughly 43. You have throttled a well-behaved tenant to a quarter of what it paid for.
The middle path is estimate, then settle. Reserve counted input plus an expected output, such as the route's or tenant's recent p95, capped at max_tokens. When the response ends, read the reported usage and adjust the balance by the difference. An overrun can leave the balance negative. That debt is bounded, because no request can exceed its input plus max_tokens, and it blocks the tenant's next admission until refill pays it back.
class TokenBudget {
constructor(tokensPerMinute) {
this.capacity = tokensPerMinute;
this.ratePerMs = tokensPerMinute / 60_000;
this.balance = tokensPerMinute;
this.updated = performance.now();
}
refill() {
const now = performance.now();
this.balance = Math.min(this.capacity, this.balance + (now - this.updated) * this.ratePerMs);
this.updated = now;
}
reserve(estimate) {
this.refill();
if (this.balance < estimate) return false;
this.balance -= estimate;
return true;
}
settle() {
.();
. += estimate - actual;
}
}
budget = ();
estimate = + ;
(budget.(estimate)) {
actual = + ;
budget.(estimate, actual);
}
.(.(budget.));
This runs as written in Node and shows one process. Across a gateway fleet the same reserve and settle steps move into the shared store, and the counter problems described in which rate-limiting algorithm, and where the counter lives apply unchanged.
Concurrency is a second limit
Token-based rate limiting controls spend per minute. It does not control how many generations a tenant holds open. Assume a model streams 50 output tokens a second. A 5,000-token response then occupies a slot for 100 seconds. A tenant can stay inside its minute budget while holding dozens of slots on a self-hosted model, starving everyone else. Put a per-tenant cap on in-flight requests alongside the token budget, as in tenant burst fairness in a shared API.
Streaming cancellation
When a user closes the tab, the gateway sees the downstream connection drop. If it does not cancel the upstream call, the model keeps generating tokens nobody will read, and they are still spent. Propagate the disconnect: abort the upstream request. On a self-hosted model that frees the slot; on a hosted API it is the only signal you can send that the output is no longer wanted.
Then settle honestly. Count the output tokens streamed before the stop and charge those. Whether a hosted provider bills for tokens generated after you disconnect is a question for that provider's terms, so check it before assuming the cancellation saved money.
The provider has limits too
A hosted provider limits your account, typically by requests and tokens per minute. Those limits are a quota shared by every tenant behind your gateway, so rate limiting here is allocation. The gateway's job is to divide that quota, which means the sum of tenant budgets should not promise more than the upstream allows, or should do so knowingly and shed lowest priority work first.
When the upstream returns a 429, do not pass it straight through. Hold the request briefly with backoff, route it to another deployment or region if you have one, and tell the tenant only when the wait would exceed their timeout. Return your own 429 with a Retry-After for tenant budget exhaustion, and keep it distinguishable from an upstream capacity problem.
An LLM request's price is decided by the model while it runs, so admission can only make a bet. Reserve a realistic estimate, settle against reported usage, and let bounded debt absorb the error rather than charging every caller for the worst case.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).