How do you safely rebuild a corrupted read model in an event-sourced architecture without incurring significant downtime?
Rebuild a corrupted CQRS read model by replaying the event store into a fresh projection, applying upcasters for old event versions, validating parity, and cutting over with a blue-green deployment.
What the interviewer is scoring
- Does the candidate understand the separation of concerns inherent in CQRS and event sourcing?
- Whether they can articulate a strategy for replaying the event store to generate a new projection.
- That they explain how to handle event schema versioning, such as using upcasters.
- Whether the candidate knows how to transition traffic from the old read model to the new one using a blue-green deployment approach.
- Does the candidate consider optimisations like snapshotting to accelerate the replay process?
Answer
Short answer
Rebuild a corrupted read model by replaying the immutable event log into a fresh projection, using upcasters for older event versions and a blue-green cutover to keep reads available. Validate the new projection against expected state, catch it up to the live stream, then switch traffic only after parity is proven.
This is data engineering inside an event sourcing system. Good data engineering asks whether the source events are still trustworthy, whether the projection is deterministic, and how to rebuild without serving mixed old and new state. Event sourcing helps only if the event log can replay the truth. Weak data engineering work patches the read model directly and destroys the audit trail.
Why patching the read model in place is the wrong move
The amateur approach is to attempt an in-place mutation or a series of complex data migrations directly on the live read model database. They try to write scripts that patch the bad data, hoping to reverse-engineer the corruption without dropping the tables.
This fundamentally violates the principles of event sourcing. In a CQRS system, the read model is entirely ephemeral and derivative. Attempting to repair a derivative state manually is a path to irrecoverable data inconsistency and inevitable API downtime.
Rebuilding reality in isolation
The only correct action is a full replay of the event store utilizing a blue-green deployment strategy. A new, isolated projection service (the "green" environment) is spun up, pointed at a completely fresh database collection, and set to consume the event stream from the beginning. The corrupted "blue" environment continues to serve live reads, satisfying availability requirements even if the data is temporarily flawed.
However, historical event replay immediately unearths the reality of schema evolution. Over years of development, event payloads mutate. A modern projection cannot process a legacy event schema natively. This necessitates upcasters—middleware interceptors that catch older event versions and sequentially transform them through intermediate states until they match the current schema definition. This keeps legacy structural debt entirely isolated from the modern projection logic.
Rebuilding from snapshots
Replaying hundreds of millions of events from the system's genesis is rarely fast enough for incident response. Relying entirely on a linear replay guarantees unacceptable delays.
Optimized systems rely on a robust snapshotting mechanism. By periodically persisting the state of the aggregate into a separate snapshot store, the green projection can load the latest uncorrupted snapshot and process only the delta of events that occurred subsequently. Once the green projection catches up to the tip of the stream and achieves parity with live traffic, strict mathematical validation ensures integrity. Only then is the API gateway routed to the green cluster, and the corrupted blue read model is destroyed.
flowchart TD
A["Immutable Event Store"] --> B["Blue Projection (Corrupted)"]
B --> C["Blue Read Model (Live API)"]
A --> D["Event Upcaster"]
D --> E["Green Projection (Rebuilding)"]
E --> F["Green Read Model (Standby)"]
G["Snapshot Store"] -.-> EThe resilience of event sourcing lies in its capacity for time travel; by treating state as a derivative of history, you can always rewrite the present to correct the mistakes of the past.
© 2026 Preptima. Originally published at preptima.com.
Likely follow-ups
- How would you validate that the green projection is equivalent to what the blue projection should have produced, before cutting traffic over?
- What changes in your rebuild strategy if the corruption originated in the event store itself rather than in the projection logic?
- How do you handle upcasters that need to combine or split events when an aggregate's event schema is restructured, not just renamed?
Related questions
- How would you design a data lakehouse to handle petabytes of data with frequent GDPR right-to-be-forgotten requests and rapid schema evolution without sacrificing query latency?hardAlso on data-engineering3 min
- How do you implement dynamic PII masking across a highly decentralised data mesh without destroying the analytical utility of the data for downstream machine learning workloads?hardAlso on data-engineering2 min
- How do you execute the Inverse Conway Maneuver on a 200-person engineering org that accidentally built a distributed monolith mirroring their dysfunctional silos?hardAlso on architecture3 min
- How do you implement local-first collaborative editing without the unbounded memory growth inherent to CRDTs crashing the client?hardAlso on architecture2 min