A network blip lasting under a minute leaves two database primaries having each accepted writes, and you spend the next day degraded. Why does a short fault produce a long recovery, and what would you have changed?

The ratio is the whole lesson

In October 2018 a 43-second loss of connectivity between two GitHub data centres left MySQL accepting writes on both sides of the partition, and reconciling those writes safely took roughly 24 hours of degraded service. Three orders of magnitude separate the trigger from the recovery, and no amount of faster detection or faster alerting would have closed that gap, because none of the time was spent noticing.

That ratio is what makes this a reliability question rather than a networking one. Mean time to recover is usually taught as detect, diagnose, mitigate, and each of those is measured in minutes. But when a fault produces divergent state, recovery time becomes a function of how much state diverged and how hard it is to tell the two copies apart. It is bounded by the size of the repair, and the repair grows with write throughput multiplied by the duration of the divergence. Seconds of dual writes on a busy system is a lot of rows.

So the first thing to say is that you cannot fix this class of incident by improving your response. You fix it by preventing divergence, and where you cannot prevent it, by making the repair cheap before you need it.

Why a partition produces two writers at all

The mechanism is worth being precise about, because the vague version - "split brain" - hides the design decision. A replicated database has one node accepting writes. When something decides that node is unreachable, it promotes another. The decision is the problem: unreachable from where, and decided by whom.

If the promoting component sits on one side of a partition, it sees the primary as dead and promotes locally, while the original primary is alive, reachable by its own clients, and still accepting writes. If the promoting component spans both sides so that it can compare, then it is itself partitioned and its members must agree, which is why consensus turns up here. And if promotion is automatic and fast, it will act during exactly the brief blips where doing nothing would have been correct.

flowchart TD
    A[Link between sites drops] --> B{Who decides the primary is dead}
    B -- local view only --> C[Promote locally while old primary still writes]
    C --> D[Two writers, state diverges]
    B -- quorum across sites --> E{Minority side}
    E -- yes --> F[Old primary demoted or fenced]
    E -- no --> G[Single writer preserved]

The branch that matters is the fencing step rather than the promotion step: promotion without demotion is what creates two writers, and most systems make promotion easy and demotion optional.

Prevent it: one writer, enforced rather than assumed

Three properties together give you a single writer, and an answer that names only the first is incomplete.

Quorum means the decision to promote requires a majority of voting members, so a minority partition cannot elect anything. That in turn means an odd number of members and, for a two-site deployment, a third location holding a witness or arbiter, because two sites cannot form a majority when they cannot see each other. Anyone proposing cross-site failover with exactly two sites should be asked where the tie is broken.

Fencing means the old primary is actively prevented from accepting writes, not merely asked to stop. The old primary is by definition unable to hear you, so the mechanism has to work from the outside or from within itself: revoking its access to shared storage, withdrawing its route or its proxy backend entry, or having it demote itself when it observes that it has lost quorum. Self-demotion on quorum loss is the cheapest and most reliable of these, and it is the one most often left out because it means the database stops serving during a partition, which feels like making availability worse.

Synchronous or semi-synchronous commit means a write is not acknowledged until a second node has it, which bounds how much can be lost on promotion. It is the property that converts "we do not know what diverged" into "we know the new primary has everything acknowledged". Its cost is honest and unavoidable: every commit now waits for at least one network round trip to the other node, so across regions the write latency floor becomes the round-trip time, and the availability of writes becomes dependent on a second node being reachable. Within a metropolitan area that is often an easy trade. Across continents it usually is not, which is a real argument for keeping the synchronous replica close and accepting asynchrony for the distant one.

Accept that the remaining choice is unpleasant

Once divergence has happened, there are only two families of outcome and both cost something. You can discard one side's writes and be consistent quickly, which means telling some users their work is gone and possibly reversing payments, shipments or messages that were already acknowledged. Or you can reconcile, keeping everything and merging by hand or by script, which preserves the data and extends the outage for as long as the merge takes.

Which is right depends entirely on what the data is, and that judgement belongs to the business rather than to the engineer on call. What belongs to you is having asked in advance, per dataset, and having written the answer down. A team that has decided beforehand that acknowledged payments are never discarded and that draft state may be, spends the incident executing rather than deliberating.

The middle path that is usually underrated is serving read-only. Rejecting writes with a clear signal while reads continue keeps most of a product usable, keeps the divergence from growing, and buys time for the repair. It requires that the application has a real read-only mode, that clients handle write rejection without corrupting local state, and that someone can turn it on quickly. All three are things you build in advance or not at all.

Design the write path so repair is mechanical

The reason reconciliation takes a day is usually that the data cannot tell you where it came from. Rows have no provenance, updates overwrote their predecessors, and reconstructing intent means correlating application logs against table contents by hand. Several cheap design choices remove most of that work.

Give every externally originated write an idempotency key, so the same logical operation arriving from both sides is identifiable as one operation rather than two. Record an append-only log of accepted operations alongside the mutable state, so the history exists independently of the rows it produced. Stamp writes with the identity of the node that accepted them and a monotonic sequence, so a merge can be filtered by origin. And prefer additive event-style writes over destructive updates for the domains where you can, since divergent appends merge and divergent overwrites conflict.

None of this is free, and it is not right everywhere. But the argument for it is precisely the ratio at the top: the incident you are insuring against is rare and its cost is dominated by repair, so spending a little on making repair mechanical has a better return than spending a lot on making the fault less likely.

Rehearse the repair, not just the trigger

Most failover testing verifies that promotion works. That tests the easy half. The half that hurts is the one nobody exercises: partition the sides so both take writes, then practise detecting divergence, deciding, and merging. Do it in a copy of production with production-shaped data, and time it. The number that comes out is your real recovery objective for this failure mode, and if it is wildly worse than what the business believes, that discrepancy is the most valuable thing you will produce all quarter.

A fault's duration tells you almost nothing about how long recovery will take; the volume of divergent state does. Prevent two writers with quorum plus actual fencing, bound possible loss with a synchronous replica where the latency allows, decide in advance which datasets may be discarded, and rehearse the merge rather than the promotion.