Your service writes a record, reads it straight back, and sometimes gets the old value. What is happening and what do you change?

The write was acknowledged, not replicated

Nothing is broken here, and saying so is the first mark. In a store that replicates a record to several nodes, the acknowledgement you received tells you how many replicas had the new version at that instant, and that number is a parameter you chose. Acknowledge on one replica and the write is durable on one replica; the other copies receive it a moment later. If the subsequent read is routed to a node that has not received it yet, returning the previous version is the system behaving as configured.

sequenceDiagram
  participant App
  participant W as Write coordinator
  participant R1 as Replica 1
  participant R2 as Replica 2
  App->>W: write v2 acknowledged by one replica
  W->>R1: store v2
  W-->>App: ack
  App->>R2: read routed elsewhere
  R2-->>App: v1
  Note over R1,R2: v2 still in flight to R2

What to notice there is that the gap is not an error path. Every arrow succeeded, and the stale answer is produced by a healthy system, which is why this class of bug survives testing on a single-node development database and appears the week you scale out.

Locate the staleness before you fix it

There are three quite different places the old value can come from, and the fixes do not transfer between them.

The first is a lagging replica, as above. The second is a secondary index or materialised view, which in most distributed stores is maintained asynchronously from the base record even when the base record itself is read strongly. DynamoDB is explicit about this: a global secondary index supports eventually consistent reads only, so a strongly consistent read of the item does not help a query that goes through the index. Cassandra's materialised views and search indexes carry the same asymmetry. The third is a cache — your own, a shared one, or an HTTP layer you do not control — where the value is not stale by microseconds but by the whole time-to-live.

Establishing which one you have is a matter of reading the same key by primary key and by index, from a pinned node and from any node, and seeing which combination returns the old value. That takes minutes and prevents you from raising a consistency level that was never the problem.

Buying read-your-writes on the replica path

For the replica case the underlying arithmetic is worth being able to derive rather than quote. With a record replicated to N nodes, a write acknowledged by W of them and a read that consults R of them, the read is guaranteed to see the write when R + W > N, because the two sets cannot then be disjoint and at least one node in the read set holds the new version. With five replicas, writing to three and reading from three overlaps in at least one node; writing to one and reading from one does not.

Cassandra exposes exactly this as a per-statement choice, which is the useful property. You can write and read at QUORUM for the operation that needs the guarantee and leave everything else at ONE. Where the cluster spans regions, LOCAL_QUORUM gives the same overlap within one datacentre without paying a cross-region round trip on every operation — and it is worth stating clearly that this buys you nothing at all for a client whose read is routed to a different region.

Stores with a single writer per shard offer a cheaper version of the same thing. A MongoDB replica set has one primary, so directing this particular read to the primary rather than a secondary removes the staleness by removing the second copy from the path; the general mechanism, when reads must be spread, is a causally consistent session, where the driver carries a token from the write and the server waits until the node it reads from has caught up to it. DynamoDB's per-request strongly consistent read is the same idea expressed as a flag.

What the guarantee costs

Every one of these fixes is a purchase, and being able to price it is what separates a considered answer.

A quorum read costs latency, because you now wait on the slowest of several nodes rather than the fastest of one, and tail latency is where that lands. It costs throughput, since each logical read consumes several nodes' worth of work. And it costs availability in the precise sense that matters during an incident: if enough replicas of that partition are unreachable, a quorum read cannot be satisfied and the correct behaviour is to fail rather than to answer from an under-replicated set. A design that reads at quorum everywhere has decided, usually without noticing, that it prefers errors to stale data on every code path — including the ones where stale data was perfectly acceptable.

That is why the answer is per-operation rather than global. The read after a user's own write needs the guarantee, because a form that appears not to have saved generates a support ticket and a duplicate submission. The list of everybody else's records a second later does not.

The two cheap fixes people forget

Before reaching for consistency levels, consider not doing the read at all. If the write path already knows the new state, return it from the write, and have the client render what it just submitted rather than re-fetching. That eliminates the window entirely, costs nothing, and is available on every store.

The second is routing. If the same client's requests are directed to the same replica or the same region for the duration of a session, that client observes its own writes and also stops seeing time go backwards between two successive reads. Session affinity is a blunter instrument than a causal token and it does not survive a node failure, but it turns a class of user-visible weirdness into a rare one.

Keep this problem distinct from its neighbour. Read-your-writes is about visibility of your own completed write. A lost update — two clients read the same version, both modify it, and the second silently overwrites the first — is not fixed by any read consistency level, because both reads were correct. That one needs a conditional write: a compare-and-set on a version attribute, expressed as a conditional expression in DynamoDB, or a lightweight transaction in Cassandra with the consensus cost that implies. Confusing the two is the most common wrong turn on this question.

An acknowledged write is a promise about the replicas that acknowledged it and nothing more. Decide which individual operations need to see their own writes, buy the guarantee only there, and know what that read does when a replica set is degraded.