Every rolling update drops a small number of requests. Where do they go?
4 min readUpdated 2026-08-15
0
Answer
Short answer
Removing a pod from its EndpointSlice and sending it SIGTERM happen concurrently, and the data-plane update takes time to propagate, so requests keep arriving at a pod that has already begun shutting down.
Was this answer helpful?
Likely follow-ups
→Your image is distroless and has no sleep binary. How do you implement the preStop delay?
→terminationGracePeriodSeconds is 30 and your drain takes 45. What actually happens at second 30?
→Does a PodDisruptionBudget protect you during a Deployment rollout? Explain either way.
→How would you prove the fix worked rather than assert it, given the losses were only a handful per rollout?
When the Deployment controller scales down an old ReplicaSet, the API server marks the pod with a deletion timestamp and moves it out of the Ready state. Two independent chains of work then start, and the fact that they are independent is the whole answer.
On one side the kubelet notices the pod is terminating, runs the preStop hook if there is one, and then sends SIGTERM to the container's PID 1. On the other side the EndpointSlice controller notices the pod is no longer ready and removes it from the slice, after which every consumer of that slice has to be told: kube-proxy on every node rewriting iptables or nftables rules, an ingress controller updating its own upstream list, a service mesh sidecar reloading its cluster config.
Nothing sequences those two chains. The kubelet does not wait for the data plane to converge, and the data plane does not know the container is already shutting down.
Loading diagram…
The interesting part is the ordering of the last three messages: the request is routed correctly according to rules that are correct but stale, and the update that would have prevented it arrives afterwards. On a large cluster that window is commonly hundreds of milliseconds to a few seconds, and it scales with node count.
The delay is for the data plane, not for you
This is why the standard fix is a preStop hook that does nothing but wait. The hook runs beforeSIGTERM, so the container stays fully healthy and serving throughout it, which is exactly what you want while the endpoint removal propagates. It is not a pause before shutdown work; it is a period of continued normal service that overlaps the propagation window.
lifecycle:preStop:sleep:seconds:10# native action, added in 1.29 and on by default from 1.30terminationGracePeriodSeconds:45
Before that native action existed the idiom was exec: ["sleep", "10"], which requires a sleep binary in the image and therefore fails silently useful-looking on a distroless or scratch base. If you are on an older cluster and a minimal image, the workable options are handling SIGTERM in the application with your own delay before you stop accepting connections, or shipping a static sleep in the final image stage.
The grace period is a budget both phases spend
terminationGracePeriodSeconds, 30 by default, starts counting when deletion begins, and the preStop hook spends from it. A 10-second hook and a 30-second grace period leaves your process 20 seconds between SIGTERM and the unconditional SIGKILL. If the drain needs longer than that, requests are cut off mid-flight and you have reintroduced the problem you were fixing, one layer down. So size it as hook plus longest plausible in-flight request plus margin, and raise the grace period rather than shortening the hook.
What the application must do on SIGTERM is specific and often only half implemented: stop accepting new connections, let in-flight requests finish, close idle keep-alive connections so clients do not reuse them, flush anything buffered, then exit zero. A server that exits immediately on SIGTERM is the second most common cause of exactly this symptom, and it is worth checking before blaming the data plane. It is also worth checking that SIGTERM reaches your process at all — a shell-form entrypoint can leave a shell as PID 1 that never forwards it, in which case nothing graceful happens and the pod is killed at the grace period every time.
Connections that will not drain themselves
The model above assumes short HTTP requests. It does not hold for anything long-lived. A WebSocket or a gRPC stream stays open past the grace period unless the server actively closes it, so those need an application-level shutdown that tells peers to reconnect — for gRPC, sending GOAWAY so clients re-resolve rather than waiting for the socket to break. Keep-alive HTTP connections are a milder version of the same problem: a client pool holding an idle connection to a pod that is going away will use it for the next request, and only a Connection: close on the way out prevents that.
The other end of the rollout
Requests lost at start-up look identical from a dashboard. A pod is added to the EndpointSlice as soon as it is ready, so if the readiness probe passes before the process can genuinely serve — a warm cache still filling, a connection pool still opening, a JIT still cold — traffic arrives into a pod that is technically up and effectively not. A readiness probe that checks a real dependency path, plus minReadySeconds so the rollout does not race ahead of its own new pods, closes that side. The rollout defaults of 25% maxUnavailable and 25% maxSurge also mean a quarter of capacity can be gone at once, which turns a small per-pod loss into a visible one.
Two clarifications worth volunteering, because both are commonly stated wrongly. A PodDisruptionBudget constrains voluntary disruptions that go through the eviction API, such as a node drain; a Deployment rolling update deletes pods directly and is governed by maxUnavailable, not by the PDB. And an ingress controller that programmes pod IPs itself has its own convergence delay independent of kube-proxy, so tuning kube-proxy alone can leave the window almost unchanged.