Two clocks, and the pipeline must know which it uses
Every event has at least two timestamps. Event time is when the thing
happened. Ingestion time is when your system received it. They disagree
whenever a device was offline, a queue backed up, a batch was retried, or a
partner sent a file late.
A pipeline that groups by ingestion time produces figures that are always
internally consistent and frequently wrong: yesterday's total contains events
from the day before, and the number changes depending on how busy the queue was.
A pipeline that groups by event time produces figures that are correct and
occasionally incomplete, because more of yesterday can still arrive.
Only the second is defensible, and it is the one that creates this question.
Choosing it means accepting that a window is not finished when the clock passes
midnight — it is finished when you decide it is.
Three responses, and the choice is not technical
Wait longer. Hold the window open for an allowance — an hour, six hours —
before publishing. Simple, and it moves the problem rather than solving it:
anything later than the allowance is still late, and every consumer now waits.
The allowance should come from the observed lateness distribution, not from a
round number, and it is worth measuring: the ninety-ninth percentile of
ingestion_time - event_time over the last month tells you what you are
actually buying.
Accept and restate. Publish on schedule, and when late data arrives,
recompute the affected window and republish. The figure changes after the fact,
which is fine for an internal dashboard and unacceptable for something already
sent to a regulator. This is the honest option for most analytical workloads and
it has a hard prerequisite, below.
Drop, and count what you dropped. Refuse anything behind the boundary. Fast
and stable, and only defensible if the dropped volume is reported somewhere a
human sees, because a pipeline silently discarding 3% of events looks identical
to one discarding none.
The decision belongs to whoever consumes the number. The question to ask is
whether a published figure is allowed to change — and the answer differs between
a marketing dashboard, a finance close and a regulatory return, in the same
company, from the same pipeline.
Watermarks make the boundary explicit
A watermark is the pipeline's assertion that it does not expect event-time data
older than a given point. It is a heuristic, not a guarantee, and stating it
that way is the point: it converts an implicit assumption into a value you can
configure, log and alert on.
(events
.withWatermark("event_time", "6 hours")
.groupBy(window("event_time", "1 day"), "product_id")
.agg(sum("amount").alias("total")))
Two things follow. The engine can release state for windows behind the
watermark, which is what stops a streaming job's memory growing forever — the
watermark is as much a resource decision as a correctness one. And records
arriving behind it are dropped by default, which is the third option above
chosen implicitly, so if you did not mean to choose it you must route them
somewhere.
The habit worth having is to send late records to a side output rather than
letting them vanish. A table of what arrived too late, with how late it was, is
both the alert and the input to revising the allowance.
Restatement needs idempotent writes
"Recompute the window" is easy to say and depends entirely on the write being
safe to repeat. A pipeline that appends cannot restate: rerunning it adds a
second set of rows and doubles the total.
The two workable shapes:
Overwrite the partition. Recomputing a day means deleting and rewriting that
day's partition atomically. Simple, and it requires the partition key to be
event time rather than ingestion date — which is the same decision as before,
appearing again in the physical layout.
Upsert on a natural key. Each output row has a deterministic key, and the
write is a merge rather than an insert. Slower and more flexible, and it copes
with corrections that change a row rather than adding one.
Either way the property to state plainly is that the job produces the same
result whether it runs once or five times. That is the difference between a
pipeline you can restate and one where every correction is a manual repair.
merge into daily_totals t
using recomputed r
on t.day = r.day and t.product_id = r.product_id
when matched then update total r.total, restated_at
matched (, product_id, total) (r.day, r.product_id, r.total);
What to say when asked
The strong answer names the two clocks first, states that the response is a
business decision about whether a published figure may change, gives the three
options with what each costs, and then commits to one for the case in front of
you with a reason. The weak answer picks a watermark duration immediately.
And the detail that shows experience: whatever you choose, record on each output
row when it was computed and whether it has been restated. A consumer who can
see that yesterday's total was revised at 06:00 will trust the pipeline; one who
notices a number changed and cannot find out why will stop trusting all of them.
Arrival order is not chronology. Once you group by event time you have to say
out loud when a window closes, what happens to anything behind it, and whether
a published number is allowed to change.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).