A user saves their profile, the page reloads, and the old name is back. What is happening and how do you fix it?
The write committed on the primary and the reload was served by a replica that had not applied it yet. Fixing it means giving that session a read-your-writes guarantee, either by routing its reads to the primary for a bounded window or by carrying the commit position and refusing to read a replica behind it.
What the interviewer is scoring
- Does the candidate diagnose this as replica lag rather than as a caching or browser problem
- Whether read-your-writes is named as a session guarantee, distinct from making the whole system strongly consistent
- That routing reads to the primary is priced as load moved onto the primary, not as a free correctness fix
- Whether the candidate knows a replica's applied position must be compared against the write's, per replica
- Does the candidate raise the second user-visible anomaly, a read that goes backwards in time
Answer
Diagnose it as lag, not as a cache
Asynchronous replication acknowledges a write once the primary has it durably, then streams it to replicas. So there is a window, usually milliseconds and occasionally minutes, during which the primary and a replica disagree, and a read routed to a replica inside that window legitimately returns the old value. Nothing is broken and no data is lost — the read is correct with respect to a state that was true a moment ago.
The reason this surfaces as a bug report rather than as a metric is that the user performed the write. Staleness is invisible when you are reading somebody else's data and glaring when you are reading your own, because you know what the value should be. That asymmetry is the whole insight: the guarantee you need is not global recency, it is a per-session promise that a client never sees a state older than one it has already caused.
That guarantee has a name — read-your-writes, one of the session guarantees, alongside monotonic reads, monotonic writes and writes-follow-reads. Naming it matters because it tells the interviewer you are proposing something far cheaper than linearisability. You are not making every read see the latest write from every client; you are making one client's reads consistent with its own history, and that can be done without a single extra round trip between replicas.
sequenceDiagram
participant U as User session
participant P as Primary
participant RA as Replica A
participant RB as Replica B
U->>P: write name, commit at position 900
P->>RA: stream, applied 900
P->>RB: stream, still applied 850
U->>RB: read profile
RB-->>U: stale name from 850
U->>RA: read again carrying token 900
RA-->>U: name as writtenThe thing to look at is the difference between the two stream arrows: the replicas are at different positions at the same instant, so "read from a replica" is not one behaviour, and any fix that does not consult a specific replica's applied position is guessing.
The three fixes, in ascending order of cost and correctness
Route the session's reads to the primary for a window. After a write, mark the session — a flag with a short expiry in a shared store, or a signed cookie — and send its reads to the primary until the flag lapses. Simple, correct, and it works today. The cost is load: if 5 percent of requests are writes and each pins the session to the primary for 10 seconds, you have moved a share of read traffic onto the one node you were offloading, and that share grows with how write-heavy the workload is. Choose the window from your observed replication lag, and understand that you have coupled correctness to a timeout — if lag exceeds the window you are back to the original bug, silently.
Narrowing the pinning helps a lot. It is rarely the whole session that needs the primary, just reads of the objects the session wrote, so pin per entity rather than per user where you can distinguish them.
Sticky routing to one replica. Send a session's reads consistently to the same replica. This is cheap and buys you monotonic reads, which is a genuine improvement, but it does not buy read-your-writes: the replica you are stuck to may be the lagging one. It also fails when that replica is removed from rotation, at which point the session moves to a replica that may be behind the one it was reading, and time appears to run backwards. Useful as a supporting measure, wrong as the primary answer.
Carry the commit position. The correct mechanism is to make the client's causal history explicit. The write returns the log position at which it committed; the client carries that token; a read presents it, and the read is only served by a replica whose applied position is at or beyond it, waiting briefly or falling back to the primary otherwise. This is exact rather than probabilistic, it costs nothing when replicas are caught up, and it degrades into the first fix precisely when lag is the problem.
Some systems expose this directly. MongoDB's causally consistent sessions carry a cluster time, and a read can specify afterClusterTime so the server waits until it has applied at least that point. In PostgreSQL the pieces are there but the wiring is yours: pg_current_wal_lsn() on the primary gives you the write position, pg_last_wal_replay_lsn() on a standby gives you its applied position, and pg_wal_lsn_diff() compares them, so an application-level router can track each replica's position and route accordingly. What you must not do is compare wall-clock timestamps between machines and call it a position; clock skew makes that a source of subtle, unreproducible staleness rather than a fix for it.
The second anomaly, which is the one that gets missed
Read-your-writes only covers state the session itself created. There is a distinct failure with no write in it at all: two consecutive reads land on two replicas at different positions, so the second read returns an older state than the first. A count decrements, an item reappears in a list after being seen gone, a page reloads to a previous version. That is a violation of monotonic reads, and it is arguably more confusing to users because there is no action of theirs to explain it.
The fixes overlap but are not the same. Sticky routing solves monotonic reads and not read-your-writes. Position tokens solve both, if the client advances its token to the highest position it has observed, not merely the highest it has written. That distinction — tracking reads as well as writes into the causal token — is the difference between a design that handles the reported bug and one that handles the class.
The related third case is ordering across objects. A comment visible while the post it replies to is not means a causal dependency crossed a replication boundary in the wrong order, which is what writes-follow-reads addresses: a write made after reading some state must not become visible before that state does. This is the guarantee that matters when the two objects live in different partitions or different services, and it is the reason "eventual consistency is fine, users won't notice" is a claim that needs testing rather than asserting.
What to say about scope and about measurement
Two limits are worth volunteering. A session guarantee is scoped to the session, so a token in a cookie does nothing when the user writes on their phone and reads on their laptop; users perceive consistency per person, not per session, and closing that gap means holding the position server-side against the account, which starts to cost what you were avoiding. And these guarantees say nothing about two different users, so if user A writes and user B must see it — a chat message, an approval, a transferred balance — you need a stronger model and should say so rather than stretching a session guarantee to cover it.
On measurement: lag must be monitored per replica and as a distribution, not as a fleet average, because the average hides the one standby that is minutes behind and serving a share of your reads. Watch it in both units. Byte lag tells you how much data is outstanding, which detects a network or disk bottleneck; time lag tells you how stale a read can be, which is the number your window is set from. An idle primary makes byte lag zero while a stalled replay leaves time lag large, and an alert on only one of them misses that. Then set the alert threshold below the pinning window you chose, so the first symptom is a page rather than a support ticket.
Ask which guarantee the user's complaint actually violates before choosing a mechanism. "Save then reload shows the old value" is read-your-writes; "refresh shows an older value than the previous refresh" is monotonic reads; and the sticky-routing fix that solves the second does nothing for the first.
Likely follow-ups
- Two consecutive reads hit two replicas at different positions. What does the user see, and what is that guarantee called?
- The write and the read happen on two different devices. Which of your fixes still works?
- How would you monitor lag so that this is caught before a user reports it?
- A comment is visible but the post it replies to is not. Which guarantee did you violate?
Related questions
- The monthly bill run dies two thirds of the way through and the cycle closes tomorrow. What do you do?hardSame kind of round: scenario5 min
- A corporate action is confirmed with an effective date in the past, after you have already struck positions and sent statements. How does your system cope?hardSame kind of round: design5 min
- A trade was booked with the wrong quantity and you find out after it has been confirmed, reported and sent for clearing. What has to happen?hardSame kind of round: scenario6 min
- A family plan shares 100 GB across five SIMs with each SIM capped at 30 GB. How does charging enforce both limits at once?hardSame kind of round: design6 min
- A fill arrives for an order your system has already marked cancelled. What do you do?hardSame kind of round: scenario5 min
- The MES is unreachable for four hours and the line keeps producing. What happens to the production record, and how do you reconcile afterwards?hardSame kind of round: scenario5 min
- The operations team says there is no rule for this, they just use judgement. How do you model that?hardSame kind of round: design6 min
- A subscriber's service is working on the network but no subscription exists in billing. How did that happen, and what do you do about it?hardSame kind of round: scenario5 min