One node stops answering. How does the rest of the cluster decide whether it has crashed or is merely slow, and what does that decision cost you?

The evidence is identical, so the question has no correct answer

A node that has crashed sends nothing. A node whose process is paused for garbage collection sends nothing. A node whose packets are being dropped by a switch sends nothing, and so does one that has been descheduled on an oversubscribed host, or one that is alive and healthy behind a link that is delivering your heartbeats a second late. From the outside these are the same observation: no message has arrived. In an asynchronous system there is no bound on message delay, so no amount of waiting turns absence of evidence into evidence of death.

This is why the honest answer starts by refusing the premise. The cluster does not decide whether the node has crashed. It decides whether to act as though it has, on the basis of a timeout, and that timeout is a policy choice about which mistake you would rather make. Candidates who reach immediately for a better health check, a second probe from a different network path, or a shorter heartbeat interval are optimising the guess. Those are worthwhile, and none of them removes the guess.

The two errors are not symmetric

Declaring a live node dead is a false positive. The consequences are concrete: if the node was the leader, you now have two processes that each believe they hold the write role. The suspected node did not agree to be replaced, is not aware of the election, and will happily continue accepting writes and acknowledging them to clients. That is split brain, and it produces divergence that no later reconciliation can fully undo, because both branches contain acknowledged work.

Failing to declare a dead node dead is a false negative, and it costs availability. Writes queue behind a leader that will never respond, or a shard sits without a replica until somebody notices. It is unpleasant and it is visible, but nothing is lost that was not going to be lost anyway.

Tightening the timeout trades the second error for the first. A detection window shorter than your worst realistic pause makes every long pause look like a death, and long pauses are common: a stop-the-world collection, a saturated disk, a host under memory pressure, a noisy neighbour. Loosening it does the opposite. So the tuning question is not "what is the right timeout" but "what is the longest pause a healthy node in this deployment can suffer", and the answer to that comes from measurement rather than from a default. Detectors that report a suspicion level rather than a boolean, as accrual-style detectors do, help because they let the reaction be proportionate, but they shift the boundary rather than removing it.

Make the false positive harmless instead of rare

Since you cannot drive the false positive rate to zero, the design that survives is one where a wrongly-evicted node cannot cause damage when it returns. That means the resource being protected must reject the stale writer itself, rather than the cluster trying to ensure only one writer exists.

The mechanism is a monotonically increasing number issued with the role: a term, an epoch, a fencing token. Every write carries it, and the storage layer refuses any write bearing a number lower than the highest it has already seen. Now the paused leader can wake up believing anything it likes; its writes are rejected at the door, because a newer term has been observed.

sequenceDiagram
    participant L1 as Leader term 7
    participant M as Coordinator
    participant L2 as Leader term 8
    participant S as Storage
    L1->>M: heartbeat then long pause
    M->>L2: grant leadership term 8
    L2->>S: write with term 8
    L1->>S: write with term 7 after waking
    S-->>L1: rejected, stale term

What to look for is that the rejection happens at the storage layer and not at the coordinator. If the old leader can reach the data without passing the check, the token bought you nothing, and that is the usual gap between a design that names fencing and one that implements it.

Leases follow the same logic with time attached, and inherit a subtlety worth stating: the holder and the grantor measure the lease with separate clocks, so a holder that checks its own remaining validity and then pauses may act after expiry believing it is still inside the window. A lease is therefore a liveness optimisation, and the fencing check is what makes it safe.

Who is allowed to hold the opinion

The last decision is where the judgement lives. If every node independently decides who is alive, you get disagreement, and disagreement about membership is itself a source of split brain: two subsets each with a majority of their own view. Routing the decision through a consensus group means there is one authoritative sequence of membership changes, which is precisely what a term number in Raft encodes, and it is why an election in such a protocol both removes the old leader's authority and stamps every subsequent write with proof of the change.

The unavoidable trade remains. A majority can only act while it can talk to itself, so under a partition the minority side must stop serving writes even though its nodes are perfectly healthy. That is the price of never having two writers, and stating it plainly is the difference between a candidate who has read about consensus and one who has operated it.