hardScenarioDesignMidSeniorStaff
You move an internal service to gRPC behind the same load balancer your REST services use, and one pod ends up serving most of the traffic. What is happening, and what do you change?
6 min readUpdated 2026-08-15Target archetype: Big Tech, Enterprise Captive, Product Startup
Answer
Short answer
gRPC multiplexes many calls over one long-lived HTTP/2 connection, so a balancer that distributes connections pins each client to one backend for its lifetime. Balance per request with an HTTP/2-aware proxy or client-side subchannels, and bound connection age so scaling out actually moves traffic.
The hidden transport detail is HTTP2. gRPC multiplexes many calls over one HTTP2 connection, so a load balancer that balances only at connection creation may send all traffic to one pod. Good HTTP2-aware routing either balances streams correctly, limits connection lifetime, or makes clients open enough connections. Weak HTTP2 answers scale pods without changing why the load balancer keeps choosing the same one.
Was this answer helpful?
Likely follow-ups
- Your RPCs vary from two milliseconds to two seconds. Does round robin still distribute load evenly?
- A rolling deploy replaces every server pod. What does a client with open subchannels experience, call by call?
- How do you keep a connection alive through a proxy that closes idle sockets, and how do you detect one that has died silently?
- Where would you enforce a per-caller rate limit, given that every call arrives on an existing connection?
Related questions
- The same domain has to be exposed to a mobile app, a partner integration and internal service-to-service traffic. Where does GraphQL fit, where does gRPC, and where does neither?hardAlso on grpc and streaming7 min
- Rate-limit an LLM gateway where one request can cost a hundred times another.hardAlso on streaming5 min
- Your autoscaling is configured exactly as designed and the service still browns out every Monday at nine. Where is the time going?hardAlso on load-balancing6 min
- How do you achieve true exactly-once semantics in Flink across source, state, and sink without cratering throughput?hardAlso on streaming2 min