The trigger for the outage is gone and you have restarted the service, and it collapses again within a minute of taking traffic. Why will it not recover on its own?

Removing the cause is not the same as removing the effect

The thing to name first is that the system is in a state which sustains itself. The original trigger — a slow dependency, a bad deploy, a traffic spike — pushed the service into a regime where its own behaviour generates enough extra load to keep it there, so restoring the trigger's absence restores nothing. The load that is killing you now is load the outage created, and it will keep being created until you break the loop deliberately.

Three effects do almost all of this work, and a good answer separates them because each has a different remedy.

The retries that accumulated while you were down

While the service was failing, callers did not stop. Each of them retried, and the retries queued in client-side buffers, in message brokers, in mobile apps that will try again on the next screen, and in upstream services that treat your failure as transient. When you come back you are not offered the steady-state arrival rate. You are offered the steady-state rate plus the accumulated demand of the outage window, arriving in the first seconds, synchronised by the fact that everyone learned you were healthy at the same moment.

Three minutes down at 200 requests per second is on the order of tens of thousands of pending attempts. Nothing in your capacity plan covers that, and the first thing it does is fill your queues. This is why jitter on client retries matters more during recovery than during the failure, and why the honest server-side answer is that you cannot rely on clients behaving well and must be prepared to refuse the excess yourself.

The cache that is now empty

The restart threw away everything warm. Suppose your cache normally serves 95 percent of reads, so a workload of 200 reads per second reaches the database as roughly 10. With an empty cache, the same 200 reads per second all reach the database, a twentyfold increase against a database that was provisioned for the served fraction. It cannot keep up, so latency rises, so requests time out before they can populate the cache, so the next request for the same key is also a miss. The cache never fills, because filling it requires successful requests and there are none.

That loop is the clearest example of the general shape: an effect that would be self-correcting under light load is self-reinforcing under heavy load. Duplicate concurrent misses for the same key make it worse, which is why coalescing identical in-flight fetches into one is a recovery mechanism and not merely an efficiency. The other lever is to warm deliberately before accepting user traffic, from a replay of recent keys or a background loader, and to accept that this makes your restart slower on purpose.

flowchart TD
  A[Restart with empty cache] --> B[Every read reaches the database]
  B --> C[Database latency rises]
  C --> D[Requests time out before caching a result]
  D --> E[Clients retry]
  E --> B
  C --> F[Threads held waiting]
  F --> D

The loop closes without the original trigger appearing anywhere in it, which is the whole reason the service does not heal by itself.

Throughput without goodput

The measurement that reframes this scenario is the difference between work completed and work completed in time to be useful. If callers give up after two seconds and your responses are arriving after five, then your service is fully utilised and delivering nothing. Every unit of capacity is spent on a result that will be discarded by a client that has already retried, and that retry is more load.

Once you state it that way, the remedy follows and it is counter-intuitive enough that interviewers watch for it: you must refuse work. Serving fewer requests well is strictly better than serving all of them too late, because the ones you refuse stop consuming capacity and the ones you accept produce responses somebody reads.

Making that real needs three things. A bound on the queue, so overload becomes a rejection at admission rather than an unbounded latency; a rejection path cheap enough that shedding is not itself expensive, which means deciding before you authenticate against a remote service or open a database connection; and a deadline attached to each request, checked before each stage of work, so anything that cannot finish in time is dropped rather than half-completed. Propagating that deadline downstream is what stops your dependencies from spending their capacity on your abandoned requests too.

There is a queue-discipline subtlety worth volunteering. Under sustained overload, first-in-first-out serves the oldest request first, and the oldest request is the one most likely to have been abandoned already, so a FIFO queue in overload maximises wasted work. Serving the newest arrival instead means the requests you do serve are the ones whose callers are still waiting. It feels unfair, and it converts a total brownout into a partial outage, which is the trade you want when you cannot serve everyone.

What to shed, not just how much

Shedding is only defensible if the choice of victim is deliberate. Rank by what the request is for, not by which connection happened to arrive first: health checks and control-plane calls must always be admitted, a checkout must outrank a recommendation carousel, a synchronous user-facing read must outrank a batch job that can run at 3am, and a retry should be admitted after a first attempt because a retry is by definition load that has already been paid for once. Carrying a criticality label on the request makes that decision cheap; deriving it from the URL at admission time is the pragmatic version.

The related lever is concurrency, not rate. Limits expressed as a maximum number of in-flight requests adapt to changing service times in a way that a fixed requests-per-second cap does not, which matters exactly here, because during recovery your service times are unusually long and a rate limit tuned for healthy latency will admit far more concurrent work than the machine can hold.

Feature flags belong in the same conversation. A design in which you can switch off personalisation, recommendations and non-essential enrichment without a deploy gives you the ability to reduce your own load by a large factor in seconds, and that is often the fastest way out of the loop.

Your own health checks may be holding you down

Two probe mistakes turn a recovery into a cycle. If readiness fails under load, the orchestrator removes the instance from the load balancer, its share of traffic is redistributed to the remaining instances, they now fail readiness too, and the fleet removes itself one pod at a time while each pod is individually recovering. If liveness checks the same thing, the instance is killed and restarted, which discards whatever warmth it had accumulated and returns it to the cold-cache state you were trying to escape.

The rules that follow are that liveness should test only whether the process is fundamentally stuck, readiness should reflect whether this instance can accept more work rather than whether the whole system is healthy, and neither should call a downstream dependency, because a dependency's failure then removes every instance you own simultaneously.

Coming back on purpose

Given all of the above, recovery is a controlled procedure rather than a restart. Bring the service up without traffic, warm what needs warming, then admit load in stages with a concurrency limit you raise as latency stays inside the deadline, watching goodput rather than CPU. Where a broker backlog exists, drain it with capacity separate from the online path, and make an explicit decision about whether the oldest items are still worth processing at all. Where clients are retrying hard, a rejection carrying an explicit wait hint does more for you than silently dropping the connection, because it is the only way to influence when they come back.

The thing to say last is the design conclusion: a service that cannot shed load has no recovery mode, only a hope that demand falls below capacity on its own. Everything above is available in a library or a proxy configuration, and the reason it is so often absent is that it is invisible until the day it is the only thing that would have worked.