hardDesignScenarioMidSeniorStaff
How would you design a data pipeline you can safely re-run?
Design a safely rerunnable data pipeline by making each output partition a deterministic result of its input partition, overwriting instead of appending, and writing into idempotent sinks that tolerate retries and late events.
6 min readUpdated 2026-08-15
Answer
Short answer
A safely rerunnable data pipeline is idempotent: rerunning the same partition should replace the same output, not append duplicates. Use deterministic partition keys, overwrite or merge by natural key, track watermarks for late events, and make sinks tolerate at-least-once delivery.
Was this answer helpful?
Likely follow-ups
- A late event arrives for a partition you already closed and reported on downstream. What do you do?
- How do you make the backfill of 400 partitions safe to run concurrently with the nightly schedule?
- Where would you put the deduplication key, and what retention does that state need?
- What changes if the sink is a REST API or a payment system rather than a table?
Related questions
- An event for 23:58 yesterday arrives at 00:20 today, after you have already published yesterday's totals. What should your pipeline do?hardAlso on event-time and watermarks4 min
- Events for a daily revenue table arrive up to three days late. How do you design the pipeline and the backfill?hardAlso on backfill and event-time4 min
- You must ingest all of English Wikipedia every day, and the source does not want to be hammered. What runs, how often, and what do you do about the run that fails at hour six?hardAlso on idempotency6 min
- A transform has been writing wrong revenue figures for three days and six downstream tables have consumed it. How do you backfill the corrected data without double-counting anything?hardAlso on backfill4 min