You move an internal service to gRPC behind the same load balancer your REST services use, and one pod ends up serving most of the traffic. What is happening, and what do you change?

Connections are what your balancer distributes, and gRPC only opens one

The REST services worked because each HTTP/1.1 request could be sent on its own connection, or on a short-lived keep-alive connection, so distributing connections was very nearly the same thing as distributing requests. gRPC runs on HTTP/2, which multiplexes many concurrent streams over a single TCP connection and then keeps that connection open indefinitely. A transport-layer balancer picks a backend once, at connection establishment, and every call the client makes for the rest of the process's life travels down that one pinned path.

Put numbers on it. Six client instances, four server pods, one connection per client chosen at random. The expected count is one and a half connections per pod, but the variance is what you feel: it is entirely ordinary for one pod to hold three of the six connections and for another to hold none. Then weight it by real traffic, because clients are not identical, and the pod holding the connection from your busiest caller serves a share of requests that has nothing to do with its share of connections.

flowchart LR
  C1[Client A busy] --> P1[Pod 1]
  C2[Client B] --> P1
  C3[Client C] --> P2[Pod 2]
  P3[Pod 3 idle]
  P4[Pod 4 new after scale out]

The two pods on the right are the point: nothing is wrong with them, no health check is failing, and they will stay idle until some client is forced to resolve and dial again.

Why autoscaling makes this worse rather than better

The imbalance is self-reinforcing under load. Pod 1 is hot, so average latency rises, so an autoscaler adds capacity. The new pod receives no connections, because every client already has one and has no reason to open another. Aggregate CPU across the deployment falls, which looks like the scale-out worked, while the pod that was overloaded is exactly as overloaded as before. Meanwhile the metric that would have shown you this — requests per second per pod, viewed as a distribution rather than a total — is usually the one nobody has dashboarded, because for the REST services it was never interesting.

The same mechanism explains the other complaint that arrives with it: after a deploy, load is briefly even and then drifts back into a lopsided shape over hours as connections are re-established at different moments and never move again.

The two places balancing can live

The first option is to balance at layer seven. A proxy that speaks HTTP/2 terminates the client's connection, reads individual streams, and forwards each call to a backend of its choosing, so the unit of distribution becomes the RPC. This is what an ingress or mesh proxy configured for gRPC does, and it is usually the least invasive change because it needs nothing from client code. You pay for it: an extra network hop on every internal call, a component that must parse HTTP/2 frames at your full internal request rate, and a proxy whose own connection pool to the backends now needs the attention you were hoping to avoid.

The second option is to balance in the client. gRPC's name resolution can return every server address rather than one, and the client keeps a subchannel to each, choosing per call. In Kubernetes that means resolving a headless service so the answer is the set of pod addresses instead of a single virtual address, or running a lookaside control plane such as xDS that pushes endpoint membership to clients. The behaviour you get without choosing is the single-connection behaviour: gRPC ships a policy that connects to one reachable address and sends everything there, and a round-robin policy that maintains a subchannel per address and rotates between them. The distinction matters because the naive setup is not misconfigured, it is simply unconfigured.

Client-side balancing removes the hop and gives the client the freshest possible view of latency, and it costs you a full mesh of connections. Forty client instances and twenty pods is eight hundred connections, each with its own HTTP/2 state and flow-control windows on both ends, and every client must now learn about membership changes promptly or it will keep dialling pods that have gone. It also puts balancing policy in every language runtime you deploy, which is a real tax if that number is above two.

Bounding connection lifetime is the fix nobody remembers

Whichever you choose, add a maximum connection age on the server. gRPC servers can be told to retire a connection after some interval, which sends an HTTP/2 GOAWAY so the client finishes its in-flight streams, re-resolves, and dials again. That single setting converts a static, indefinitely pinned assignment into a slowly rotating one, which is what makes a scale-out event eventually reach the new pod even with a connection-level balancer in front.

It is also the mechanism that makes graceful shutdown coherent. GOAWAY tells the client to stop opening new streams on this connection while allowing existing ones to complete, so a pod being drained during a deploy stops attracting work without failing calls that are already running. Without it, the choices are to kill in-flight RPCs or to wait for connections that will never close on their own.

Pair it with keepalive pings. A connection that sits idle through a NAT device or a proxy with its own idle timeout can be discarded silently, and the client will not discover this until it tries to send and waits. Periodic pings both hold the path open and turn a dead connection into a prompt error.

Streams break the request-level model again

Per-request balancing works because requests are short. A server-streaming or bidirectional call is a single long-lived stream, so once it is placed it stays where it was placed, and a client holding a subscription for an hour is pinned for an hour no matter how sophisticated your proxy is. Streaming reintroduces exactly the problem you just solved, at a coarser grain.

The practical answers are to bound stream lifetime deliberately and have clients reconnect with jitter, so placement is periodically reconsidered, and to be sceptical of streams whose real purpose is fan-out to many consumers, which a broker handles better than a pinned connection ever will. Say this before the interviewer asks, because it is the part of gRPC operations that surprises teams after the migration is already done.

Finally, choose the policy with knowledge of your call cost distribution. Round robin assumes calls are roughly equal, and when a service mixes two-millisecond lookups with two-second reports, an even count of requests is not an even distribution of work. A least-outstanding-request policy tracks the thing you actually care about, and the reason to name it here is that it is the only one of these choices you can make without changing your topology.