A dependency that normally answers in 80ms starts taking eight seconds. What in your service reacts, and in what order?

The failure mode is resource exhaustion, not the slow call

Eight seconds instead of eighty milliseconds is a hundredfold increase in how long each in-flight request holds a thread, a connection and a slot in whatever pool you use. Arrival rate has not changed. Little's law does the rest: concurrency equals arrival rate times service time, so a service comfortably handling 200 requests per second at 80ms was holding about 16 concurrent requests, and at 8 seconds it needs 1,600. It does not have 1,600 of anything. The pool fills, new requests queue for a thread, and requests that never touch the slow dependency at all begin timing out.

That is why the answer starts with the timeout rather than with the breaker. Everything else in this list mitigates; the timeout is the only mechanism that bounds the resource an unhealthy dependency can consume from you, and it works without the dependency's cooperation.

Pick it from the distribution, not from intuition. If p99 is 150ms in normal operation, a timeout somewhere around two to three times that is defensible: high enough that a normal slow request survives, low enough that a hung one is cut. A ten-second timeout on an 80ms call is the same as having no timeout, and the default in most HTTP clients is worse than ten seconds — it is none at all, which means a request can be held until the operating system gives up on the socket. Name that default explicitly, because "we relied on the client library's default" is the actual root cause of a large share of real incidents.

The refinement worth volunteering is a deadline rather than a per-hop timeout. Propagate the remaining budget with the request — gRPC does this natively — so a downstream call that cannot possibly finish before the caller gives up is never started. Per-hop timeouts that sum to more than the caller's patience mean you spend effort on work whose result nobody will read.

Retries help sometimes and make this worse

Retrying a call that failed because the network dropped a packet is free recovery. Retrying a call that failed because the dependency is saturated adds load to a saturated system, and if every caller does it the dependency cannot recover even after the original trigger is gone. That is a retry storm, and its signature is inbound request volume at the dependency exceeding what callers believe they are sending.

Three rules keep retries from becoming the outage. First, retry only what is safe to retry: an idempotent operation, or a non-idempotent one carrying an idempotency key. A retried POST /payments without a key is a second payment. Second, back off exponentially with jitter, because synchronised retries from a hundred instances arrive as a spike no matter how long you waited.

// Full jitter. The cap matters as much as the base: without it, attempt 10
// waits for minutes. The random draw is what desynchronises callers that
// all failed at the same instant.
long window = Math.min(CAP_MS, BASE_MS * (1L << attempt));
long delay  = ThreadLocalRandom.current().nextLong(window + 1);

Third, bound the total. Two or three attempts, and a budget rather than a per-call count: allow retries only while they are under some small fraction of overall request volume, which is what gRPC's retry throttling implements with a token bucket. The reason a budget beats a count is nesting. Three layers each retrying three times is up to 27 calls for one user request, and no layer can see the multiplication because each of them is behaving reasonably in isolation. Retry at one layer — ideally the one closest to the failure that still knows whether the operation is safe — and let the others fail fast.

The breaker, and the state everyone forgets to justify

A circuit breaker converts a slow failure into a fast one. While it is open, calls are rejected locally in microseconds, which releases the threads and lets the rest of your service work; it also removes load from the dependency so it has a chance to recover.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: error or slow-call rate over threshold
    Open --> HalfOpen: cool-down elapsed
    HalfOpen --> Closed: probe calls succeed
    HalfOpen --> Open: any probe fails

The transition to argue about is half-open back to open on a single failing probe: the cool-down restarts from scratch rather than the breaker lingering half-open and leaking traffic into something still broken. That asymmetry — many failures to open, few successes to close, one failure to reopen — is the whole design, and a candidate who explains why it is deliberately asymmetric is describing something they have tuned rather than configured.

For this scenario the essential detail is that the breaker must count slowness as a failure. A dependency answering in eight seconds is returning 200s, so an error-rate-only breaker never trips and you get no protection at all. Resilience4j exposes this as a slow-call-rate threshold alongside the failure-rate threshold; whatever the library, check that the knob exists. Sizing matters too: a minimum number of calls before the rate is evaluated, otherwise two failures at low traffic open the circuit, and a window measured in requests or in time depending on whether your traffic is steady.

Breakers are also the wrong tool for a per-tenant or per-key problem. A single hot partition failing will not move a fleet-wide error rate enough to trip anything, and if it does trip you have taken out every tenant to protect one.

Isolation is what limits the blast radius

Timeouts bound how long one call holds a resource; a bulkhead bounds how many of your resources any one dependency can hold at once. Give each downstream its own connection pool and its own concurrency limit, so recommendations going bad cannot consume the threads that checkout needs. Without it, "one non-critical dependency degraded" and "the whole service is down" are the same event, which is the thing this pattern exists to prevent.

Then decide what a rejected call returns. A fallback is right when a stale or empty answer is genuinely usable — cached recommendations, a default shipping estimate, a feature quietly absent from the page. It is wrong when it fabricates a fact: returning "no outstanding balance" because the balance service is down is worse than an error, because the caller cannot distinguish it from truth. Say which of your dependencies are on the critical path and therefore have no fallback, because a design where everything degrades gracefully usually means something is lying.

Finally, protect yourself from the inside as well. Bound your own inbound queues and shed load rather than accepting work you cannot complete within your callers' deadlines — an unbounded queue converts an overload into a latency spiral where every response arrives after the client gave up, so the work is done and thrown away. And drop the retry-after-recovery trap: a service that comes back and is immediately hit by every client's accumulated retries has not recovered, so the cool-down and the jitter are as much about the recovery as about the outage.