A gRPC call three services deep hangs, and so does everything waiting on it. What was missing?
A gRPC deadline. gRPC sets none by default, so a client waits as long as the server takes, and every caller up the chain holds a thread or stream with it. Set a deadline at the edge, propagate the remaining budget on every hop, stop work on cancellation, and retry only inside what is left.
5 min readUpdated 2026-10-09Target archetype: Big Tech, Product Startup
0
Answer
Short answer
A gRPC deadline was missing. gRPC does not set one by default, so a client with no deadline waits for as long as the server takes, which can be forever. When service C blocks, B waits on C, A waits on B, and each holds a thread, a stream and memory while it does. The fix is a deadline set at the edge, with deadline propagation passing a shrinking budget through every hop. Handlers must stop work when the call is cancelled, and retries must fit inside the time remaining.
Was this answer helpful?
Likely follow-ups
→The edge has a two-second budget and the deepest service needs a database call. How much of the budget should it be allowed to spend?
→A service in the middle publishes to a queue and a worker calls the next gRPC service. What happens to the deadline at that boundary?
→How would you find, across forty services, the call sites that make outgoing gRPC calls without any deadline?
→Keepalive pings are already configured. Why did they not detect this hang?
Suppose C is waiting on a row lock that a crashed transaction will hold until someone kills it. C's handler is not failing. It is simply not returning. B called C with no gRPC deadline, so B's request is waiting too, and so is A's call to B. Nothing in the chain ever produces an error, which is why the dashboards show low error rates while latency climbs without limit.
The damage spreads sideways. In a thread-per-request server, each stuck request pins a worker. New requests that need C pile up behind the first, and soon requests that never touch C cannot get a thread either. The same arithmetic as any slow dependency applies, covered in timeouts, retries and breakers. The gRPC-specific point is that the protocol has the machinery to stop this, and it does nothing unless you use it.
Keepalive does not help here. Keepalive pings detect a dead connection, and this connection is healthy. The transport answers pings while the handler sits on a lock, and only a deadline bounds how long a live but stuck call may run.
A deadline is a budget that travels
A gRPC deadline is a point in time after which the client no longer wants the answer. On the wire, gRPC over HTTP/2 sends it in the grpc-timeout header as a relative duration, so hops do not need synchronised clocks. Each hop receives the time remaining, not the original figure.
Deadline propagation means a server that calls onward passes its own remaining budget into the outgoing call. That way C learns that A's user gave up, instead of only learning what B chose on its own.
Loading diagram…
Look at the budget shrinking on the way down. When the edge's two seconds run out, every hop expires at about the same moment, and each one releases its thread.
How deadline propagation happens depends on the language. In Go, a handler receives a context.Context carrying the caller's deadline. Passing that same context to the outgoing call carries the deadline and cancellation along. Deriving a tighter timeout from it keeps whichever deadline is earlier, so a hop can shorten its budget but cannot extend it.
func(s *quoteServer) GetQuote(ctx context.Context, req *pb.QuoteRequest) (*pb.Quote, error) {
// ctx carries the caller's remaining deadline; passing it on propagates it.
price, err := s.pricing.GetPrice(ctx, &pb.PriceRequest{Sku: req.Sku})
// The bug that hangs the chain: a fresh context has no deadline at all.// price, err := s.pricing.GetPrice(context.Background(), &pb.PriceRequest{Sku: req.Sku})if err != nil {
returnnil, err
}
return &pb.Quote{Price: price.Amount}, nil
}
At the edge in Go, the deadline comes from context.WithTimeout. In Java, a client sets one with withDeadlineAfter on the stub. Inside a server, grpc-java keeps the incoming deadline in io.grpc.Context, and calls made within that context inherit it.
Where deadline propagation quietly breaks
Setting a gRPC deadline at the edge is the part everyone does. A strong answer finds the hop where it was lost, because the chain is only as bounded as its weakest link.
The common breaks are ordinary code. In Go, someone calls context.Background() because the function signature did not take a context. In Java, the context is bound to the current thread, so work handed to another executor loses the deadline unless the task is wrapped, for example with Context.current().wrap(...). Any boundary that is not gRPC also drops it: a REST call in the middle, or a message queue between two services. On those hops, you must carry the remaining budget yourself or set a local timeout.
A server-side backstop helps with code you do not control. An interceptor can apply a default maximum deadline to any incoming call that arrives without one. Then a forgetful client gets an error after a bounded time instead of an indefinite wait.
Cancellation is the other half
When a gRPC deadline expires, the client gets DEADLINE_EXCEEDED and gRPC cancels the call, so the server sees it as cancelled. The library does not stop your handler, though. A handler that ignores cancellation keeps querying the database and calling onward for a caller that has already left.
Handlers should check for cancellation before expensive steps. In Go that means watching ctx.Done(), and in Java checking Context.current().isCancelled(). Passing the context into database drivers that accept one lets the query itself be cancelled. Without this, deadlines protect the caller's threads while the servers keep doing work nobody will read.
Retries inside the remaining budget
Retrying a call that timed out is where good intentions make things worse. A retry loop that gives each attempt a fresh two-second deadline turns a two-second budget into six seconds. Upstream callers have already given up, so all of that work is wasted. gRPC's built-in retry policy counts every attempt against the single call deadline. Hand-written loops need the same rule: retry only while time remains, and never past it.
DEADLINE_EXCEEDED also does not mean the operation failed. The gRPC status code documentation notes that, for an operation that changes state, it may arrive even though the server completed the work. A payment call that timed out may have charged the card. Retry it safely only with an idempotency key, or check the outcome before trying again.
Every gRPC call needs a deadline, and every server must pass on the remaining budget rather than start a new one. Then a stuck call fails in seconds, at every hop at once, instead of holding the whole chain open.