A gRPC call three services deep hangs, and so does everything waiting on it. What was missing?
A gRPC deadline. gRPC sets none by default, so a client waits as long as the server takes, and every caller up the chain holds a thread or stream with it. Set a deadline at the edge, propagate the remaining budget on every hop, stop work on cancellation, and retry only inside what is left.