One node stops answering. How does the rest of the cluster decide whether it has crashed or is merely slow, and what does that decision cost you?
5 min readUpdated 2026-08-15Target archetype: Big Tech, Enterprise Captive, Product Startup
0
Answer
Short answer
You cannot distinguish a crashed process from a slow one, so every failure detector is a timeout and every timeout is a guess. Choose which error you can afford: declaring a live node dead risks two writers, waiting risks a stall, and only fencing makes the first survivable.
Failure detection is a trade-off, not a fact. Good failure detection chooses timeout values, leases, heartbeats, and fencing based on the cost of false suspicion versus slow recovery. Weak leases reasoning assumes a process is dead because it is quiet. Strong failure detection treats leases as a way to bound authority, not as proof that the old owner has vanished.
Failure detection should be tuned to the damage caused by a wrong suspicion.
Was this answer helpful?
Likely follow-ups
→The node that everyone declared dead wakes up and resumes writing. What stops it?
→Would you let each node form its own opinion about who is alive, or route that decision through consensus, and what does each cost?
→How does your answer change if the node is reachable from half the cluster and not the other half?
→Where does a lease's clock live, given that the holder and the grantor measure time separately?
The evidence is identical, so the question has no correct answer
A node that has crashed sends nothing. A node whose process is paused for garbage collection sends nothing. A node whose packets are being dropped by a switch sends nothing, and so does one that has been descheduled on an oversubscribed host, or one that is alive and healthy behind a link that is delivering your heartbeats a second late. From the outside these are the same observation: no message has arrived. In an asynchronous system there is no bound on message delay, so no amount of waiting turns absence of evidence into evidence of death.
This is why the honest answer starts by refusing the premise. The cluster does not decide whether the node has crashed. It decides whether to act as though it has, on the basis of a timeout, and that timeout is a policy choice about which mistake you would rather make. Candidates who reach immediately for a better health check, a second probe from a different network path, or a shorter heartbeat interval are optimising the guess. Those are worthwhile, and none of them removes the guess.
The two errors are not symmetric
Declaring a live node dead is a false positive. The consequences are concrete: if the node was the leader, you now have two processes that each believe they hold the write role. The suspected node did not agree to be replaced, is not aware of the election, and will happily continue accepting writes and acknowledging them to clients. That is split brain, and it produces divergence that no later reconciliation can fully undo, because both branches contain acknowledged work.
Failing to declare a dead node dead is a false negative, and it costs availability. Writes queue behind a leader that will never respond, or a shard sits without a replica until somebody notices. It is unpleasant and it is visible, but nothing is lost that was not going to be lost anyway.
Tightening the timeout trades the second error for the first. A detection window shorter than your worst realistic pause makes every long pause look like a death, and long pauses are common: a stop-the-world collection, a saturated disk, a host under memory pressure, a noisy neighbour. Loosening it does the opposite. So the tuning question is not "what is the right timeout" but "what is the longest pause a healthy node in this deployment can suffer", and the answer to that comes from measurement rather than from a default. Detectors that report a suspicion level rather than a boolean, as accrual-style detectors do, help because they let the reaction be proportionate, but they shift the boundary rather than removing it.
Make the false positive harmless instead of rare
Since you cannot drive the false positive rate to zero, the design that survives is one where a wrongly-evicted node cannot cause damage when it returns. That means the resource being protected must reject the stale writer itself, rather than the cluster trying to ensure only one writer exists.
The mechanism is a monotonically increasing number issued with the role: a term, an epoch, a fencing token. Every write carries it, and the storage layer refuses any write bearing a number lower than the highest it has already seen. Now the paused leader can wake up believing anything it likes; its writes are rejected at the door, because a newer term has been observed.
Loading diagram…
What to look for is that the rejection happens at the storage layer and not at the coordinator. If the old leader can reach the data without passing the check, the token bought you nothing, and that is the usual gap between a design that names fencing and one that implements it.
Leases follow the same logic with time attached, and inherit a subtlety worth stating: the holder and the grantor measure the lease with separate clocks, so a holder that checks its own remaining validity and then pauses may act after expiry believing it is still inside the window. A lease is therefore a liveness optimisation, and the fencing check is what makes it safe.
Who is allowed to hold the opinion
The last decision is where the judgement lives. If every node independently decides who is alive, you get disagreement, and disagreement about membership is itself a source of split brain: two subsets each with a majority of their own view. Routing the decision through a consensus group means there is one authoritative sequence of membership changes, which is precisely what a term number in Raft encodes, and it is why an election in such a protocol both removes the old leader's authority and stamps every subsequent write with proof of the change.
The unavoidable trade remains. A majority can only act while it can talk to itself, so under a partition the minority side must stop serving writes even though its nodes are perfectly healthy. That is the price of never having two writers, and stating it plainly is the difference between a candidate who has read about consensus and one who has operated it.