hardDesignScenarioMidSeniorStaff
Events for a daily revenue table arrive up to three days late. How do you design the pipeline and the backfill?
Land raw events unchanged by arrival date, then rebuild each revenue day from that raw layer by event date. Each run overwrites only the days its new arrivals touched, a day is provisional until the three-day window closes, and the backfill is the same job run over a date range, never a separate script.
4 min readUpdated 2026-10-10
Answer
Short answer
Keep two layers. Raw events are appended unchanged, partitioned by the date they arrived, with both timestamps. The revenue table is partitioned by event date and rebuilt from the raw layer. Each run finds which event dates the new arrivals touched, within the three-day bound, and overwrites only those days. Late-arriving data therefore causes a clean recompute, not a patch. The backfill is the same job called with a list of dates.
Was this answer helpful?
Likely follow-ups
- An event arrives eleven days late for a day finance has already closed. Walk me through what happens to it.
- How do you stop the nightly run and a backfill from overwriting the same revenue day at once?
- Refunds arrive as separate events, sometimes weeks after the sale. Which day do they belong to?
- How would you notice that the three-day bound had quietly become five?
Related questions
- A transform has been writing wrong revenue figures for three days and six downstream tables have consumed it. How do you backfill the corrected data without double-counting anything?hardAlso on backfill and data-pipelines4 min
- How would you design a data pipeline you can safely re-run?hardAlso on backfill and event-time6 min
- An event for 23:58 yesterday arrives at 00:20 today, after you have already published yesterday's totals. What should your pipeline do?hardAlso on data-pipelines and event-time4 min
- A DAG that finished in an hour now takes nine and blocks everything behind it. How do you find out where the time went?hardAlso on data-pipelines6 min