Two servers each write a record stamped with its own clock. Why can you not use those timestamps to decide which write happened first, and what do you use instead?

What a wall clock is actually reporting

A machine's wall clock is an estimate of civil time maintained by a local oscillator that drifts, corrected periodically by a synchronisation daemon. Three properties of that arrangement make it unusable as an ordering primitive.

The error is unbounded from the application's point of view. A well-synchronised fleet may agree within a few milliseconds; a machine whose network path to its time source is congested, or whose virtual clock was suspended while the hypervisor migrated it, can be off by seconds or minutes, and nothing in the API you call reports how wrong the value is. You cannot write a correct comparison against an unknown error term.

It is not monotonic. Correction can step the clock, including backwards, so two reads in sequence within one process can return a decreasing pair of values. Any code that subtracts one wall-clock reading from another to measure a duration can therefore compute a negative elapsed time, and a timeout implemented that way can wait effectively forever. The discipline is simple and worth stating because it is a real bug you will meet: use the monotonic clock for durations and the wall clock only for values a human will read or a system will report. In Java that is System.nanoTime() for elapsed time and System.currentTimeMillis() for a timestamp, and mixing them up is one of the most common latent defects in retry code.

And a timestamp comparison implies nothing about causality even when both clocks are accurate. If A wrote at 12:00:00.100 and B wrote at 12:00:00.090, ten milliseconds apart on two machines, the ordering you have inferred is a property of two oscillators, not of your system. If A's write was a response to reading B's, then B genuinely preceded A and your timestamps say the opposite.

The consequence appears in real datastores. A store that resolves concurrent writes by keeping the higher timestamp — last-write-wins over wall clock, as Cassandra does by default — will silently discard a later write whose originating node's clock ran slightly behind. There is no error and no conflict reported; the update simply is not there, which is the worst available failure mode because it is invisible until someone notices missing data weeks later.

Ordering by causality instead

Lamport's insight is that you do not need to know when things happened to order them, only what could have influenced what. Define happened-before as: events in the same process are ordered by their local sequence, and a send always precedes its receive. Anything not related by a chain of those two rules is concurrent, and asking which came first is not a meaningful question.

A Lamport clock implements this with one integer per process. Increment it on every local event; attach it to every outgoing message; on receipt set the local counter to the maximum of its current value and the received one, then increment. The resulting numbers have the property that if A happened-before B then C(A) < C(B).

The half that gets missed, and the thing an interviewer is listening for, is that the implication does not run the other way. C(A) < C(B) does not mean A happened-before B; they may be entirely unrelated. So a Lamport clock is enough to produce a consistent total order — break ties on node id and every node agrees on the same sequence — but it cannot tell you whether two events conflict, which is exactly what you need in order to detect a concurrent update.

Vector clocks recover that. Each node keeps a counter per node, increments its own on a local event, and merges by taking the element-wise maximum on receipt. Comparing two vectors now has four outcomes rather than two: one dominates, the other dominates, they are equal, or neither dominates and the events are genuinely concurrent. That fourth case is the useful one, because it lets a store say "these two writes conflict" and either keep both as siblings for the application to reconcile or apply a deterministic merge, instead of silently dropping one. The cost is that the vector grows with the number of writers and travels with every message, which is why systems that use them restrict who counts as a writer — replica nodes rather than clients — and prune stale entries.

Node A: [A:2, B:1]      Node B: [A:1, B:3]
        neither dominates -> concurrent, so this is a real conflict

Node A: [A:2, B:1]      Node B: [A:3, B:1]
        B dominates      -> B's write descends from A's, safe to overwrite

The legitimate use of physical time

It would be dishonest to say physical time is never usable for ordering, because one production system does exactly that, and describing how it manages it is the strongest possible demonstration that you understand the constraint.

Google's Spanner uses TrueTime, which does not return a timestamp but an interval guaranteed to contain the true time, made narrow by GPS receivers and atomic clocks in every datacentre. Because the uncertainty is bounded and known, a transaction can pick a commit timestamp and then simply wait until that timestamp is definitely in the past everywhere before making its writes visible. The ordering guarantee is purchased with that deliberate wait plus specialised hardware, and the shape of the solution tells you what the problem was: ordinary systems cannot order by physical time not because clocks are inaccurate but because their inaccuracy is unquantified.

The pragmatic middle ground is a hybrid logical clock, which pairs a physical component with a logical counter, advances the physical part when the local clock moves forward and falls back to incrementing the counter otherwise. The timestamps stay close to wall-clock time, so they remain meaningful to humans and to time-range queries, while never going backwards and while still respecting causality through the message-passing merge.

Quorums give overlap, not order

For a replicated register across N replicas, requiring W acknowledgements to write and R responses to read means that when R + W > N the two sets must share at least one replica, so a read is guaranteed to touch a node that saw the latest completed write. Be precise about what that buys you: the guarantee is overlap, which means the freshest value is present among the responses. Selecting it still requires versioning, which brings you back to the clocks above, and making the result durable for subsequent readers requires the reader to write back what it found.

Two common qualifications are worth volunteering. A sloppy quorum, where a write that cannot reach its intended replicas is accepted by whichever nodes are available and handed off later, keeps you writeable during a partition and voids the overlap arithmetic entirely — you have chosen availability, and the read guarantee is gone while the handoff is outstanding. And quorum reads and writes alone do not give you linearizability; a read concurrent with a write can legitimately return either value, and consecutive reads can go backwards unless the protocol repairs as it reads.

A lease does not stop the old leader writing

Leader election looks like the easy part and contains the sharpest trap in the topic. Electing a leader safely requires a majority, because two disjoint majorities cannot exist and that is what prevents two nodes both believing they lead. Systems normally attach a lease so a leader that loses contact eventually stops acting, and this is where the reasoning usually stops.

It should not. Consider a leader that acquires a thirty-second lease, then enters a stop-the-world garbage collection pause — or is descheduled, or has its VM frozen for a live migration — for forty seconds. The rest of the cluster times it out and elects a new leader, correctly, by majority. The old process then resumes. From inside, no time has passed and it is still the leader, so it proceeds to write to the shared storage with complete confidence. You now have two writers and a corrupted resource, and no amount of care in the election protocol prevented it, because the fault was in the assumption that a process can detect its own suspension.

The fix has to live at the resource. Every election increments a monotonically rising number — a term in Raft, an epoch elsewhere — and the leader presents it with every write. The storage layer records the highest number it has seen and rejects any write bearing a lower one. That is a fencing token, and it converts an unbounded pause from a correctness bug into a rejected request. It also means the resource being protected has to participate: a lock service that hands out a lease and a resource that does not check the token has given you mutual exclusion under the assumption that nothing ever pauses, which is precisely the assumption distributed systems exist to abandon.

Never compare timestamps from two machines to decide order, and never trust a leader's own belief that it still leads; the substitutes are causality tracked in the data with logical clocks and a monotonically increasing token that the resource itself enforces.