The trigger for the outage is gone and you have restarted the service, and it collapses again within a minute of taking traffic. Why will it not recover on its own?
Answer
Short answer
The system is being held down by effects it now generates itself: accumulated client retries, an empty cache multiplying database load, and a queue of requests whose callers have already given up.
This is load shedding and admission control during recovery. Good load shedding protects the service from work it cannot finish, while admission control decides which requests are allowed in before queues explode. Weak load shedding starts only after latency is already catastrophic. Strong admission control lets the system warm caches, rebuild pools, and serve the highest-value traffic instead of accepting everything and failing everything.
Load shedding should begin before queues and thread pools are already saturated. Load shedding is a recovery control, not only a last-ditch defence.
Was this answer helpful?
Likely follow-ups
- What do you shed first, and how does a request carry enough information for that decision to be made cheaply?
- Your queue is full of work that is minutes old. Do you serve the oldest request or the newest, and why?
- How would you ramp traffic back up, given that the load balancer will send you everything the moment you report healthy?
- What would you build now so that the next recovery does not need a human deciding when to let traffic back in?
Related questions
- Traffic is a hundred times normal and some of it is real customers. What do you drop first?hardAlso on load-shedding and admission-control5 min
- You need to enforce 100 requests per minute per API key across a fleet of forty servers. Which limiting algorithm, and where does the counter live?hardAlso on load-shedding and admission-control8 min
- Your rate limiter runs in five regions with a cap of 100 requests a minute per user. One user gets 500 through. Where did the counting go wrong?hardAlso on load-shedding5 min
- Why is one flip-flop not enough to bring a signal into another clock domain?hardAlso on metastability7 min