Skip to content
Preptima
hardScenarioDesignSeniorStaffLead

The trigger for the outage is gone and you have restarted the service, and it collapses again within a minute of taking traffic. Why will it not recover on its own?

7 min readUpdated 2026-08-15Target archetype: Big Tech, Enterprise Captive, Product Startup

Answer

Short answer

The system is being held down by effects it now generates itself: accumulated client retries, an empty cache multiplying database load, and a queue of requests whose callers have already given up.

This is load shedding and admission control during recovery. Good load shedding protects the service from work it cannot finish, while admission control decides which requests are allowed in before queues explode. Weak load shedding starts only after latency is already catastrophic. Strong admission control lets the system warm caches, rebuild pools, and serve the highest-value traffic instead of accepting everything and failing everything.

Load shedding should begin before queues and thread pools are already saturated. Load shedding is a recovery control, not only a last-ditch defence.

Was this answer helpful?

Likely follow-ups

  • What do you shed first, and how does a request carry enough information for that decision to be made cheaply?
  • Your queue is full of work that is minutes old. Do you serve the oldest request or the newest, and why?
  • How would you ramp traffic back up, given that the load balancer will send you everything the moment you report healthy?
  • What would you build now so that the next recovery does not need a human deciding when to let traffic back in?

Related questions

Masterclass guide

Further reading