Your database fails over and is accepting connections again within thirty seconds, but your service returns errors for ten minutes. Where is that time going?
6 min readUpdated 2026-08-15Target archetype: Big Tech, Enterprise Captive, Product Startup
0
Answer
Short answer
The database recovered; your clients did not. Sockets to the old primary hang rather than fail, a cached DNS answer keeps pointing at it, the promoted node refuses writes while it finishes, and then the whole fleet reconnects at once.
A failed recovery often sits in the connection pool. Good connection pool design notices dead connections, drains them, and recreates capacity after failover instead of waiting for every request to discover the problem. Weak connection pool behaviour keeps handing out sockets that were valid before the database moved. Strong connection pool recovery treats failover as a normal event the service must survive, not a reason to restart the fleet.
The connection pool should recover as deliberately as the database does.
Was this answer helpful?
Likely follow-ups
→Which errors from your driver do you retry, and which must never be retried after a failover?
→How would you rehearse this in a way that would have caught the stale address before a customer did?
→Your liveness probe fails during the failover and the orchestrator restarts every instance. Is that helping?
→You read from replicas as well. What happens to traffic that was reading from the node that has just been promoted?
Two recovery times, and only one of them is on the database dashboard
The database team's number is honest. The node was promoted, it accepted a connection, and their measurement stopped there. Your number is the one users experienced, and it covers everything between the old primary going away and your service completing a write again. The gap between the two lives almost entirely inside your own process: a pool still holding sockets to a host that no longer answers, a resolver answer cached before the change, retry logic that classified a transient error as permanent, and the stampede your fleet creates when it all reconnects in the same instant.
So answer by attributing the ten minutes to those stages rather than by listing fixes. Every fix follows once you know which stage owns the time, and at least two of the plausible fixes make the others worse.
The pool is holding sockets to a machine that is gone
Take a service running twenty pooled connections per instance across forty instances, so eight hundred established sockets to the primary. A failover does not close them politely. When a node is demoted or its network path is withdrawn, packets are dropped rather than refused, so an in-flight query receives no error at all. It waits. A socket read with no timeout configured waits until the kernel exhausts its retransmission budget, which is considerably longer than the failover took, and a blocked write behaves the same way. This is the most common reason a thirty-second failover becomes a ten-minute outage: nothing in your process has been told that anything happened.
Hence the first thing to configure is a bounded wait on the socket itself, distinct from any statement timeout, so that a connection to a vanished host becomes an error on a schedule you chose. The pool needs the same treatment. A pool that validates a connection only at creation will keep handing out dead sockets for as long as they remain in it, so validation belongs on borrow, with a maximum connection lifetime and a test on idle connections as well. When one connection fails validation, evict its siblings to the same host too, because otherwise the next request simply finds another corpse.
There is a second-order effect worth volunteering. While requests are blocked on dead sockets, each one holds a worker thread. The arrival rate has not changed, so the thread pool fills, and requests that never touch the database at all start failing. That is the real argument for bounded waits: they keep the failure inside the database call path instead of letting it consume the service.
Loading diagram…
The loop between the resolution check and the reconnect is where a fleet spends most of a long outage, and no amount of retry tuning escapes it, because every attempt is correctly reaching a host that is genuinely no longer there.
The cached name
Managed databases usually fail over by moving a name to the new node, which only helps you if your client resolves that name again. Two layers can prevent it. The record carries a time to live, and a resolver or sidecar that honoured a generous one will keep returning the old address.
Above that, the runtime may cache independently of the record: on the JVM, address lookups are cached according to the networkaddress.cache.ttl security property, so a service left on a long-lived value keeps dialling the dead host for as long as that value says, whatever the record now contains. A candidate who has been through this failure names the runtime cache without being prompted, because it is the layer that is invisible from the database side and from the network side alike.
The remedy is to know both numbers and to keep the client-side one small enough that it is not the dominant term. Better still, prefer a driver that is handed several candidate hosts and works out which one is writable, which is what PostgreSQL's target_session_attrs does: give the connection string every node and let the client establish the primary rather than trusting a name to be current. Where the driver cannot do that, put a proxy in front of the database so exactly one component has to learn the new topology, instead of every instance of every service learning it independently.
The promoted node is not yet the node you had
Two things are true of a new primary in its first minutes. It may still be completing promotion, during which reads succeed and writes are refused, so your service returns a confusing mixture rather than a clean outage. And its caches are cold, so a workload that was being served from memory now reads from disk, and the same traffic arrives at a temporarily slower machine.
The first is an error-classification problem, and it is where most services lose time they did not need to lose. A driver error meaning the node will not accept writes is transient and worth retrying after a short backoff. A constraint violation is permanent and must never be retried. A timeout on a non-idempotent write is neither, because the work may have committed before the connection died, so the safe response is to check the outcome rather than repeat the call.
Services that funnel every database exception into one handler either retry the unsafe cases or abandon the safe ones, and this scenario exposes whichever mistake you made. Transactions in flight at the moment of failover are rolled back, so anything mid-commit must be either genuinely repeatable or reported as failed.
The second is a capacity problem that interacts badly with your recovery. Eight hundred connections re-establishing at once means eight hundred TCP handshakes, TLS negotiations and authentication round trips landing on a cold node in the same second, immediately followed by the backlog of user requests that queued while everything was hanging. If every instance retries on the same schedule, they arrive as synchronised waves. Jitter the reconnection, cap how many new connections an instance will open per second, and shed inbound work rather than accepting a queue you cannot drain within your callers' patience. Without that, the node is knocked over each time it returns, and the graph shows a recovery that keeps relapsing for reasons that have nothing to do with the original fault.
What settles the attribution
The measurement that decides this is not the error rate, which looks the same for all four stages. It is the time spent acquiring a connection, recorded as a distribution rather than a mean, alongside the count of established connections as seen from the database, and a breakdown of driver errors by class rather than by count. Those three together tell you whether you were waiting on a dead socket, dialling a stale address, being refused as read-only, or queueing behind your own reconnection storm.
In practice it is rarely one stage. A minute of hanging sockets, several minutes of a cached address, a burst of write rejections that the application reported as hard failures, and a final stretch of self-inflicted load. The reason to insist on the breakdown before proposing anything is that a shorter socket timeout fixes the first and worsens the last.