Your broker stops accepting writes for twenty minutes. What does each producer do, and what should you have decided before it happened?

The broker is a shared failure domain, so this question is about your service

The instinct is to answer with how you would make the broker more reliable. That is the wrong half of the problem, and interviewers are asking the other half deliberately: the broker will be unavailable sometimes, and what they want to know is whether the twenty minutes shows up as delayed events or as a customer-facing outage.

The public case worth citing is the AWS Kinesis event in November 2020, where the front-end fleet in us-east-1 hit an operating system thread limit as it scaled out, and the failure propagated into other AWS services that consumed it. The lesson is not about thread limits. It is that a streaming backbone becomes a shared failure domain for everything downstream of it, so the questions that matter are what a producer does when the stream stops accepting writes and whether its own retry behaviour makes the recovery slower than the original fault.

Three answers, and you need a different one per event class

A producer facing a broker that will not accept writes has exactly three options, and every real system uses all three for different data.

Block the calling thread until the write is accepted. This is what most client libraries do by default when their internal buffer is full, and it is the option that causes the outage. Kafka's producer buffers records in buffer.memory and, once that is exhausted, send waits up to max.block.ms before throwing. If your HTTP handler publishes an event before responding, that wait is now inside your request latency: the handler holds its thread, the thread pool fills, health checks time out, the load balancer takes instances out of service, and a broker problem has become a web-tier problem. This is worth saying explicitly, because the failure arrives as your service being down and the cause is a library default nobody chose.

Buffer somewhere with capacity, and accept that the capacity is finite. In-memory buffering is bounded by heap and lost entirely if the process restarts, which during an incident it very likely will. Spilling to local disk survives a restart and gives you far more room, at the cost of an ordering and duplication problem when several instances flush independently afterwards. Either way the important sentence is what happens when the buffer is full, because "we buffer" without that sentence is just blocking with extra steps.

Drop, deliberately, and count what you dropped. For high-volume telemetry this is the correct answer and needs no apology: a metrics sample or a click event lost during a twenty-minute outage costs you a gap in a graph. Dropping silently is the defect; dropping with a counter you can alert on and a note in the dashboard is a design.

The point of the classification is that these are the same three options with wildly different correct answers depending on the payload. Analytics events drop. Audit records must not. An order confirmation must not be lost and must also not block checkout, which means neither dropping nor blocking is available and the answer has to be structural.

For events that cannot be lost, the broker cannot be on the write path

If losing an event is unacceptable, publishing to the broker inside the request is unsound regardless of how you configure the client, because there is no configuration under which an unavailable broker both accepts your write and lets your request succeed.

The structural answer is to write the event into the same transaction as the state change it describes, into a table in your own database, and have a separate process read that table and publish. The outbox commits atomically with the order row, so an event can never exist without its state change or the reverse, and the publisher's failure to reach the broker delays delivery without failing the customer's checkout. What you have done is move the durability requirement from the broker to the database you were already committing to, which is a dependency you cannot drop anyway.

Two details are worth having ready. The publisher must tolerate republishing after a crash, so consumers must be idempotent — the outbox gives you at-least-once, not exactly-once. And the outbox table is now a queue in your primary database, so it needs an index that makes the unpublished rows cheap to find and a retention policy, or it becomes the largest table you own.

Retries are how a twenty-minute outage becomes an hour

The part that separates a senior answer is recognising that the fleet's own behaviour extends the outage. When the broker starts refusing writes, every producer retries. If retries are immediate, or exponential without jitter, the fleet synchronises into waves that arrive together. The broker, already unhealthy and now attempting to recover, receives more load than it did in steady state precisely while it has less capacity, and cannot get far enough ahead to stabilise. The system has reached a state where the load induced by the failure is sufficient to sustain the failure, and removing the original trigger no longer helps.

Three controls prevent it. Exponential backoff with full jitter, so retry attempts spread rather than align. A cap on total in-flight retries per producer, so a stuck client cannot amplify itself without limit. And a circuit breaker that stops attempting after a threshold of failures and probes occasionally, which converts thousands of doomed requests per second into a handful of health checks.

The same shape appears on the recovery. Twenty minutes of buffered events across a large fleet flush the instant the broker returns, arriving as a spike far above normal throughput, and the broker fails again. Rate-limit the drain — flush at a bounded multiple of your steady-state rate rather than as fast as the buffer allows — and stagger the resumption across instances.

What the consumers see afterwards is the second incident

Writes resuming is not recovery. The backlog now has to be consumed, and consumers that were sized for steady-state throughput cannot catch up on twenty minutes of arrears while also handling live traffic, so lag keeps growing after the underlying fault is fixed. Say what the recovery plan is, because it is a real decision with three defensible answers.

Adding consumer instances helps only up to the partition count, since a partition is consumed by one member of a group at a time; if you have eight partitions and eight consumers, more instances do nothing, which is an argument for over-partitioning at design time when the work is parallelisable. For data where only the latest value matters — a cache refresh, a search index update — skipping to the end of the log and abandoning the backlog is correct and fast, and the ability to say that some streams are safely skippable is a good signal. For everything else, plan for the catch-up window and monitor lag in time rather than in message count, because "eleven minutes behind" is actionable and "four hundred thousand messages behind" is not.

Every producer in your fleet already has an answer to a broker outage, and if you did not choose it, it is blocking on a full buffer — so classify your events into block, buffer and drop deliberately, put the ones you cannot lose behind an outbox in your own transaction, and jitter both the retries and the drain so the fleet does not extend the outage on the broker's behalf.