Timeouts, retries, idempotency keys, dead-letter queues, and designing for partial failure.
3 questions in Backend Engineering.
A timeout shorter than the caller's patience is the only defence that works without cooperation. Retries need idempotency, jitter and a budget or they amplify the outage; a breaker must trip on latency as well as errors; and a bulkhead is what stops one slow dependency consuming every thread.
The system is being held down by effects it now generates itself: accumulated client retries, an empty cache multiplying database load, and a queue of requests whose callers have already given up. Recovery needs shed load and a controlled ramp, not more capacity.
Take an idempotency key from the client and store it in a uniquely-constrained row written in the same transaction as the effect, so the dedupe record and the work commit together. Replay the stored response on a repeat, refuse a retry arriving mid-flight, and expire keys past the retry horizon.