Top 4 Failover Interview Questions

Browse 4 real-world technical and behavioral interview questions about Failover. Review scenarios, edge cases, and architectural best practices.

A network blip lasting under a minute leaves two database primaries having each accepted writes, and you spend the next day degraded. Why does a short fault produce a long recovery, and what would you have changed?

Recovery time is set by how much state diverged, not by how long the fault lasted, so seconds of dual writes can take a day to reconcile. The prevention is a single writer enforced by quorum and fencing; the mitigation is designing the write path so repair is mechanical rather than forensic.

hardscenario

You want to prove the system survives losing a database, in production, on a Tuesday afternoon. How do you run that without being fired?

Turn it into an experiment: a steady state you can measure in under a minute, a blast radius you chose, an abort that is one command with a named person who can call it, and written approval from whoever owns the revenue. Tuesday afternoon is the right time because everyone who understands the system is awake, and the announcement goes to the channel your own alerts land in.

hardscenario

Your database fails over and is accepting connections again within thirty seconds, but your service returns errors for ten minutes. Where is that time going?

The database recovered; your clients did not. Sockets to the old primary hang rather than fail, a cached DNS answer keeps pointing at it, the promoted node refuses writes while it finishes, and then the whole fleet reconnects at once. Attribute the ten minutes to those stages before tuning anything.

hardscenario

You are about to change the address a hostname points to. What does the TTL actually promise you, and why do clients keep reaching the old address anyway?

A TTL is an upper bound on how long a resolver may cache a record, not a promise about when clients will notice a change. Lowering it only takes effect after one old TTL has elapsed, negative answers are cached under a different rule, and pooled connections keep using an address they resolved hours ago.

mediumconcept