Your services talk through events and one consumer has been down for an hour. What has it missed, and how does it catch up?

Short answer

If the events are in a retained log, a consumer down for an hour has an offset lag, not automatically lost messages. Check whether log retention and committed-offset retention still cover the outage, restart from the last committed offset, and calculate catch-up time from backlog divided by surplus throughput. Handlers must be idempotent because replay can redeliver, and time logic must use event time rather than wall-clock processing time.

Nothing was missed, and that is the point of a log

The word "missed" is the first thing to push back on. If the events flow through a retained log rather than a fire-and-forget bus, the broker kept every one of them. The consumer stopped reading. Its last committed offset is still recorded, the log has grown past it, and the hour of events sits on disk waiting.

So the answer to "what has it missed" is a subtraction. Log end offset minus committed offset, per partition. That is the backlog, it is a number you can query, and it is the only honest description of the gap. On restart the consumer resumes from the committed offset and works forwards.

This is why the log-versus-queue distinction matters more than any diagram. A broker that deletes on acknowledgement has to hold undelivered work in memory or in a per-consumer structure, and a consumer that is down cannot be reasoned about the same way. A log decouples reading from delivery: retention is what protects you, not the broker's memory of who has seen what.

Which leads to the question that decides whether this is a routine restart or an incident. Is the retention window longer than the outage?

Two clocks race: your outage and your retention

If retention is measured in days and the outage was an hour, the committed offset is still inside the log and recovery is a restart. If retention was shorter than the outage, the oldest events the consumer needed are gone, and the committed offset now points at a position that no longer exists.

What happens then is configuration, not luck. In Kafka the auto.offset.reset setting decides where a consumer goes when it has no valid offset: earliest reprocesses everything the log still holds, latest jumps to the head and silently skips the gap, and none raises an error. The default in many client setups is latest, which means a retention overrun does not look like a failure. The consumer starts cleanly, lag reads zero, and an hour of business events was quietly dropped. That is the worst of the three outcomes and it is the one that produces no alert.

There is a second retention to know about, because it catches people out. Committed offsets for a group that has gone inactive are themselves subject to expiry, on a setting separate from log retention. A group that is down long enough can lose its offsets while the data is still there, which puts it back in the same reset-policy decision with a full log in front of it.

Name both windows in the answer. An outage of an hour against six-hour log retention is comfortable; the same outage against a group-offset expiry you never configured is not.

How long the catch-up takes, from stated numbers

Do the arithmetic out loud, because it turns a vague "it catches up" into a plan. Suppose the topic takes 2,000 events a second and the consumer's steady-state capacity is also close to 2,000 a second, which is how most consumers are sized. An hour down is 2,000 × 3,600, so 7.2 million events behind.

If the consumer can only manage the arrival rate, it never catches up. Lag stays at 7.2 million forever, because every event it processes is replaced by a new one. Catch-up requires surplus. At 6,000 a second it drains 4,000 a second net, so 7,200,000 / 4,000 is 1,800 seconds: half an hour of degraded but shrinking lag.

Where that surplus comes from is the design question. Consumer parallelism is capped by the partition count, so adding instances beyond the number of partitions does nothing at all. Within that cap you can add instances, increase in-flight batching, or temporarily raise the fetch size. If the partition count is the binding constraint you cannot fix it during the incident, which is the argument for over-partitioning at design time.

The other lever is what you skip. Some backlogs contain events that are worthless by the time they are read: a cache-invalidation event for a key that has been invalidated forty times since, or a "driver moved" event from an hour ago. Compaction, or a handler that discards anything older than a threshold when a newer version of the same key exists, turns a 7.2-million-event drain into something much smaller.

Replay is redelivery, so the handler decides whether it is safe

At-least-once delivery means catching up can reprocess events the consumer had already handled before it died, because the commit and the work are not one atomic act. Every handler on the catch-up path has to be safe to run twice.

Two shapes of handler behave very differently here. A handler that upserts state keyed by the event's own identifier is naturally replay-safe: writing the same fact twice leaves the same fact. A handler with an external side effect is not. Replaying an hour of "order shipped" events into a notification service sends every one of those emails a second time, and the customers who receive them are the visible cost of your recovery.

So separate them before you drain. Handlers that mutate your own state can run freely. Handlers that emit to the outside world need either a deduplication record keyed on the event id, or a suppression rule based on event age, or a deliberate decision to skip that segment of the backlog and accept the missing notifications. A strong candidate volunteers this without being asked: "Before I drain seven million events, I want to know which of my handlers sends something a human sees."

The bug that only appears during a catch-up

Here is the failure that a design review misses and a backlog run finds. Any handler that reads the wall clock computes the wrong answer during catch-up, because the events are an hour old and the clock is not.

A window keyed on processing time puts an hour of events into one bucket. A rule that says "if the event is less than five minutes old, alert" fires on nothing. A rate limiter counting per current minute sees a minute containing 7.2 million events and rejects most of them. A TTL computed as now-plus-an-hour gives a stale event a fresh lifetime. None of this shows up in steady state, because in steady state event time and processing time are within milliseconds of each other, and the two only diverge when something has gone wrong.

The discipline is to take time from the event, not from the machine. Every event carries the timestamp of the thing that happened, windows and expiries are computed from it, and the consumer tracks a watermark so it knows how far behind it is rather than assuming it is current. That also gives you the honest signal to expose: not just lag in messages, but lag in seconds of event time, which is the number a business owner understands.

The last piece is ordering. Partition assignment guarantees ordering within a partition, so a backlog replays in the original order per key, and that is usually enough. It is not enough if two related entities live in different partitions, and the fix is to choose a partition key that keeps things that must be ordered together in the same partition rather than to reorder afterwards.

The gap is an offset range, the drain time is backlog divided by surplus throughput, and the two things that turn a clean catch-up into an incident are handlers that resend to humans and logic that trusts the wall clock.

---

© 2026 Preptima. Originally published at preptima.com.