hardScenarioDesignConceptMidSeniorStaff
A transform has been writing wrong revenue figures for three days and six downstream tables have consumed it. How do you backfill the corrected data without double-counting anything?
Make the write idempotent before you make it correct: partition-level replace rather than append, so rerunning a day produces the same result whether it runs once or five times. Then work out how far the contamination spread, because a downstream table that accumulated rather than recomputed will not fix itself when the source is repaired.
4 min readUpdated 2026-08-16Target archetype: Big Tech, Enterprise Captive, Fintech
Answer
Short answer
Double-counting is not caused by rerunning the job — it is caused by the job appending. Fix the write semantics first so that processing a day produces the same end state no matter how many times it runs, then rerun the affected partitions. After that, the harder half: work out which downstream tables merely read the bad data, which will self-correct, and which accumulated it, which will not.
Was this answer helpful?
Likely follow-ups
- Your target table is append-only and you cannot change that. What are your options?
- A downstream table keeps a running total rather than recomputing. What do you do about it?
- How do you decide the boundaries of the backfill window?
- Finance already filed a report using the wrong figures. Does that change your sequencing?
- What would you add so the next bad transform is caught before three days pass?
Related questions
- Events for a daily revenue table arrive up to three days late. How do you design the pipeline and the backfill?hardAlso on backfill and data-pipelines4 min
- A DAG that finished in an hour now takes nine and blocks everything behind it. How do you find out where the time went?hardAlso on data-pipelines6 min
- A small percentage of your transaction reports are rejected every day and the team resubmits them the next morning. Is that acceptable?hardAlso on reconciliation6 min
- An upstream team renames a field in the events you consume and nobody tells you. What should your pipeline have done?hardAlso on data-pipelines5 min