Your services talk through events and one consumer has been down for an hour. What has it missed, and how does it catch up?
A Kafka-style consumer down for an hour has missed an offset range, not necessarily lost data. If retention still covers the gap, it resumes from the committed offset; catch-up needs surplus throughput, idempotent handlers, event-time logic and a plan for retention expiry. Use this event driven answer to show the decision, trade-off, and evidence rather than a memorised definition.
Answer
Short answer
If the events are in a retained log, a consumer down for an hour has an offset lag, not automatically lost messages. Check whether log retention and committed-offset retention still cover the outage, restart from the last committed offset, and calculate catch-up time from backlog divided by surplus throughput. Handlers must be idempotent because replay can redeliver, and time logic must use event time rather than wall-clock processing time.
Was this answer helpful?
Likely follow-ups
- The consumer committed offsets before processing rather than after. What is now permanently lost, and how would you find out how much?
- Retention was six hours and the outage lasted eight. Walk through the recovery and what you tell the business.
- How would you replay one day of history into a new consumer without disturbing the existing group?
- The backlog includes two updates to the same entity, and the older one is processed after the newer. What in your handler prevents the stale write?
Related questions
- You need to add one field to an event and remove another, and five teams consume it. How does that roll out?hardAlso on event-driven and kafka6 min
- You must ingest all of English Wikipedia every day, and the source does not want to be hammered. What runs, how often, and what do you do about the run that fails at hour six?hardAlso on idempotency6 min
- An Android app queues actions while offline. After the process is killed and restarted, some actions sync twice and the user sees duplicate orders. How do you design the fix?hardAlso on idempotency4 min
- The monthly bill run dies two thirds of the way through and the cycle closes tomorrow. What do you do?hardAlso on idempotency5 min