hardScenarioDesignSeniorStaff
When a transatlantic cable is severed and your multi-region database inevitably enters split-brain, how do you resolve the conflicting writes without data loss?
3 min readUpdated 2026-08-15
Answer
Short answer
An uncompromising examination of split-brain resolution, quorum consensus, fencing tokens, and why timestamp-based last-write-wins is fundamentally flawed.
Use distributed systems as the thread through the resolving split brain explanation: name the signal, then say what changes when it moves.
Was this answer helpful?
Likely follow-ups
- What happens if the witness node itself becomes unreachable from both regions during the partition?
- How does your design change if the two regions are running different, slightly incompatible schema versions when the partition heals?
- An engineer pushes a fencing-token implementation that reuses tokens after a service restart instead of persisting the counter. How do you detect and recover from the resulting split-brain risk?
Related questions
- How do you isolate a degraded dependency and halt a cascading failure before thread pool exhaustion takes down the entire microservice ecosystem?hardAlso on distributed-systems and resilience2 min
- How do you prevent a malformed Kafka payload from poison-pilling a critical partition without violating strict financial audit requirements?hardAlso on distributed-systems and resilience2 min
- One tenant bursts to ten times their normal traffic and every other customer's latency doubles. Your global rate limit was never hit. How would you design for fairness instead?hardAlso on resilience5 min
- A dependency that normally answers in 80ms starts taking eight seconds. What in your service reacts, and in what order?hardAlso on resilience7 min