hardScenarioCase StudySeniorStaffLead
A network blip lasting under a minute leaves two database primaries having each accepted writes, and you spend the next day degraded. Why does a short fault produce a long recovery, and what would you have changed?
6 min readUpdated 2026-08-15Target archetype: Big Tech, Enterprise Captive, Product Startup
Answer
Short answer
Recovery time is set by how much state diverged, not by how long the fault lasted, so seconds of dual writes can take a day to reconcile. The prevention is a single writer enforced by quorum and fencing; the mitigation is designing the write path so repair is mechanical rather than forensic.
Was this answer helpful?
Likely follow-ups
- Your failover controller spans both data centres so it can see both sides. Why is that both necessary and dangerous?
- What does semi-synchronous replication buy you here, and what does it cost on a cross-region link?
- You must choose between several hours read-only and minutes of possible data loss. How do you decide, and who owns that decision?
- How would you design the write path so that reconciling two divergent copies is mechanical rather than forensic?
Related questions
- One managed service in your primary region is having a bad day and your product is down with it. What can you do during the outage, and what should you have designed before it?hardAlso on graceful-degradation6 min
- You want to prove the system survives losing a database, in production, on a Tuesday afternoon. How do you run that without being fired?hardAlso on failover7 min
- Your database fails over and is accepting connections again within thirty seconds, but your service returns errors for ten minutes. Where is that time going?hardAlso on failover6 min
- Checkout calls pricing, tax, fraud, inventory and a session store. Which of those are you willing to carry on without, and what does the shopper see?hardAlso on graceful-degradation6 min