How would you design a data pipeline you can safely re-run?

Make the unit of work a partition, not a batch of rows

Take a concrete target: a fact_orders table partitioned by event_date, loaded daily from an orders topic landed in object storage. The design decision that makes everything else tractable is that a run is identified by the partition it produces. Run 2026-07-14 and the contract is that when the job finishes, the partition event_date=2026-07-14 contains exactly the rows that belong to that date, regardless of what it contained before and regardless of how many times you have run it.

That contract is only enforceable if the write replaces the partition. An append-only load has no notion of "what this run wrote", so a second run adds a second copy of everything and you end up maintaining a deduplication step downstream to repair damage the write path caused. Overwriting makes a re-run a no-op in the outcome even though it is real work in the compute.

-- Idempotent: the partition is the unit replaced, so a second run
-- produces the same table state as the first.
INSERT OVERWRITE TABLE fact_orders PARTITION (event_date = '2026-07-14')
SELECT order_id, customer_id, amount_minor, currency
FROM staging_orders
WHERE event_date = '2026-07-14';

In Spark this needs spark.sql.sources.partitionOverwriteMode=dynamic if you want a multi-partition statement to replace only the partitions the query actually produced rather than truncating the whole table, which is the default static behaviour and a classic way to delete four years of history during a two-day rerun. On Delta Lake the equivalent is a write with replaceWhere scoped to the same predicate as the query; on Iceberg, the snapshot-based commit gives you the same replace semantics plus an atomic swap, so readers never observe the partition mid-rewrite. Whichever engine, the property you are buying is the same: the commit is atomic at the partition level, and the partition key is derived from the data, not from when the job ran.

Why a backfill is the honest test

Nobody discovers a pipeline is not re-runnable during normal operation, because normal operation runs each date once. You discover it the day a bug is found in the revenue calculation and someone asks for the last fourteen months to be recomputed. A backfill of 400 partitions exercises every assumption the daily happy path lets you hide.

It surfaces hidden dependence on the clock: any current_date(), now(), or "yesterday" computed inside the job rather than passed in as a parameter makes the 400 runs all produce today's answer. It surfaces order dependence, because a backfill will often run partitions in parallel or in reverse. It surfaces reads of mutable upstream state, such as joining to a dim_customer table that holds only current values, which means a 2025 partition gets recomputed with 2026 attributes and the historical numbers silently change. And it surfaces accumulator logic, where a job reads its own previous output to add to a running total, so a single re-run double-counts.

The practical rule is to treat every job as a function of (partition_key, immutable inputs). If you can express the run that way, the backfill is a loop and the daily schedule is a special case of it. If you cannot, the backfill is a bespoke script written under incident pressure, and that script is where the real data corruption comes from.

Event time, processing time, and the lookback window

Partitioning by event_date and not by ingestion date is what makes the output correct, and it is also what creates the problem. An order placed at 23:58 on the 14th may reach your storage at 00:07 on the 15th; a mobile client that was offline may deliver its event three days late; a retried producer may deliver events out of order within a batch. If you partition by processing time, none of this matters and every report is wrong in a way that is hard to see. If you partition by event time, the reports are right but a partition is never definitively finished.

The bounded answer is a lookback: each nightly run reprocesses not one partition but the last N days by event time, using the same overwrite-by-partition write. Choose N from the observed lateness distribution rather than intuition, and be explicit that anything later than N days is dropped or routed to a side table for manual repair. Because the write is idempotent, reprocessing three unchanged partitions costs compute and nothing else, which is precisely why the idempotency work pays for itself here.

In streaming the same idea is formalised as a watermark. withWatermark("event_time", "3 hours") in Structured Streaming, or an equivalent watermark strategy in Flink, tells the engine to assume no event will arrive more than three hours behind the maximum event time seen so far, so windows older than that can be emitted and their state discarded. The watermark is not a correctness mechanism, it is a declared trade: you buy the ability to close a window and free state, and you pay for it with the events that exceed the bound. Naming the bound and saying what happens to events beyond it is the part that distinguishes a considered design from a copied configuration.

Schema evolution that does not break the rerun

Because re-runs rewrite old partitions with current code, schema change interacts with idempotency in a way that catches people out. Additive, nullable columns are safe: an old partition rewritten by new code gains the column as null, and readers that do not know about it are unaffected. Renames, type narrowing, and reordering are not safe, and a rename in particular is a drop plus an add as far as the storage layer is concerned, which means rewriting a 2025 partition can silently null out a column that had values. Format matters here too: columnar formats resolve by field name when the table format tracks column IDs, as Iceberg does, but a raw Parquet directory read positionally will happily shift every value one column to the left.

The stance that holds up is a contract on the boundary rather than a hope about the producers: register the input schema, treat unknown fields as tolerable and missing required fields as a failure that quarantines the batch, and version the transformation so you can state which code produced which partition. If a new column must be populated for history, that is a deliberate backfill with a documented default, not a side effect of someone rerunning last March.

At-least-once with an idempotent sink beats a claimed exactly-once

The distinction worth being precise about is that end-to-end exactly-once delivery is not achievable over an unreliable network, and every system that advertises it is describing something narrower. Kafka's transactional guarantee, enabled in Kafka Streams via processing.guarantee=exactly_once_v2, is genuinely atomic for the read-process-write cycle where the destination is also Kafka, because the offset commit and the output records land in one transaction. Flink and Spark achieve effectively-once by pairing a checkpoint or offset log with a sink that either supports a two-phase commit or overwrites deterministically. In every case the guarantee holds inside a scope, and the scope ends at the first sink that cannot participate in the commit.

So describe at-least-once delivery into a sink that makes duplicates harmless: an overwrite of a partition, an upsert keyed on a business identifier or a deterministic hash, or a request carrying an idempotency key the receiver deduplicates on. That survives the failure modes exactly-once marketing does not cover, including a task retried after it had already written output but before it acknowledged, and an operator rerunning a job because the first attempt's logs were ambiguous. You cannot prevent that duplicate. You can make it not matter.

The candidates who do well here stop treating re-runnability as an operational nicety and describe it as a property of the write path. Once the sink makes duplicate work harmless, backfills, late data, and retries all collapse into the same solved problem.