How do you safely rebuild a corrupted read model in an event-sourced architecture without incurring significant downtime?
Rebuild a corrupted CQRS read model by replaying the event store into a fresh projection, applying upcasters for old event versions, validating parity, and cutting over with a blue-green deployment.
Answer
Short answer
Rebuild a corrupted read model by replaying the immutable event log into a fresh projection, using upcasters for older event versions and a blue-green cutover to keep reads available. Validate the new projection against expected state, catch it up to the live stream, then switch traffic only after parity is proven.
This is data engineering inside an event sourcing system. Good data engineering asks whether the source events are still trustworthy, whether the projection is deterministic, and how to rebuild without serving mixed old and new state. Event sourcing helps only if the event log can replay the truth. Weak data engineering work patches the read model directly and destroys the audit trail.
Was this answer helpful?
Likely follow-ups
- How would you validate that the green projection is equivalent to what the blue projection should have produced, before cutting traffic over?
- What changes in your rebuild strategy if the corruption originated in the event store itself rather than in the projection logic?
- How do you handle upcasters that need to combine or split events when an aggregate's event schema is restructured, not just renamed?
Related questions
- How would you design a data lakehouse to handle petabytes of data with frequent GDPR right-to-be-forgotten requests and rapid schema evolution without sacrificing query latency?hardAlso on data-engineering3 min
- How do you implement dynamic PII masking across a highly decentralised data mesh without destroying the analytical utility of the data for downstream machine learning workloads?hardAlso on data-engineering2 min
- How do you execute the Inverse Conway Maneuver on a 200-person engineering org that accidentally built a distributed monolith mirroring their dysfunctional silos?hardAlso on architecture3 min
- How do you implement local-first collaborative editing without the unbounded memory growth inherent to CRDTs crashing the client?hardAlso on architecture2 min