What is the difference between a queue and a log, and what does a broker mean when it claims exactly-once?

Distributing work versus keeping a record

A classic queue exists to hand out work. A producer enqueues, one of several competing consumers takes the message, does something, acknowledges it, and the broker deletes it. Parallelism is elastic because any consumer can take any message, so you scale by adding consumers and the broker balances for you. The cost is that the message is gone: there is no second reader, no replay, and nothing to re-derive state from if the consumer had a bug. Redelivery exists only as a failure mechanism, driven by an acknowledgement deadline, with a dead-letter destination for messages that keep failing.

A log inverts that. Records are appended to a partitioned, retained sequence and are not deleted on consumption; each consumer group tracks its own offset. That single change is what buys the properties people actually want from a log. Multiple unrelated consumers read the same stream without coordinating, so adding a new downstream system is a deployment rather than a producer change. Retention is a policy independent of who has read what, so you can reset a consumer to an earlier offset and reprocess after fixing a bug, which is the only practical way to repair a corrupted derived store. And because a partition is an ordered sequence, you get ordering guarantees a work queue cannot offer.

What a log gives up is per-message dispatch flexibility. Consumer parallelism within a group is bounded by the partition count, one partition to one consumer at a time, so you cannot scale consumers past the number of partitions and you cannot let a slow message be picked up by a different consumer. A single stuck record blocks its partition's progress in a way a queue's per-message acknowledgement would not.

The design question, then, is not which product is better. It is whether you need work distribution with independent per-message lifecycles, or a retained ordered record that several consumers replay. Task fan-out — send the email, resize the image, charge the card — is queue-shaped. Event streams that feed a search index, an analytics store and a notification service from the same source are log-shaped. Plenty of systems need both, and saying so is more credible than picking a side.

The guarantees, stated where they actually live

The guarantee is decided by where the acknowledgement sits relative to the work, not by a broker setting.

Acknowledge before processing and you have at-most-once: crash after the ack and the message is gone, unprocessed, silently. This is defensible only when the data is loss-tolerant, such as high-volume telemetry where a dropped sample changes nothing.

Acknowledge after processing and you have at-least-once: crash after processing but before the ack, and the message is redelivered and processed twice. Nearly every real system runs here, because the broker cannot know whether your side effects completed and must therefore assume they did not.

There is no third option in the general case. Delivering a message and recording that it was delivered are two separate actions in two separate failure domains, and no protocol makes them one atomic action across that boundary. The sender can never distinguish a lost message from a lost acknowledgement, so it must choose which error to prefer.

Ordering belongs to the partition

A log orders records within a partition and makes no promise across partitions, so "the topic is ordered" is not a thing you can rely on. In practice you get the ordering you need by choosing a partition key — customer id, account id, aggregate id — so all records for one entity land in one partition and are therefore totally ordered relative to each other. That is almost always sufficient, because the invariants that care about ordering are usually per-entity.

Several things quietly break it. Changing the partition count remaps keys, so records for one key can exist in two partitions and their relative order across the change is undefined. Producer retries can reorder records if multiple requests are in flight, which is why brokers offering idempotent production tag records with a producer identity and sequence number and reject out-of-order writes. Consumer-side concurrency defeats it too: hand records from one partition to a thread pool and you have re-introduced arbitrary interleaving inside your own process. And routing a failed record to a dead-letter queue while continuing past it is an explicit decision to give up ordering for that key, which is often right but must be said out loud rather than discovered later.

Global total ordering means one partition, which means one consumer and no horizontal scaling. If someone asks for it, the useful reply is to ask which invariant needs it, because the answer is nearly always satisfiable per key.

Unpacking the exactly-once claim

When a broker advertises exactly-once, the correct response is to ask what the guarantee spans, because the honest version is narrow and genuinely useful, and the marketing version is neither.

What is real is atomicity within one system. A broker can offer a transaction that atomically commits the records a consumer produced and the consumer's new read offsets, as one unit. If the process dies mid-transaction the transaction aborts, the offsets are not advanced, and the partial output is never visible to readers configured to see only committed data. That composes into a genuine end-to-end guarantee for a read-process-write topology where input, output and offsets all live in the same transactional system. It is the foundation of exactly-once stream processing and it works.

What is not real is exactly-once when the effect leaves that system. If your consumer charges a card, sends an email, or writes to an unrelated database, the broker has no way to make its offset commit atomic with that external effect. Between "call the payment API" and "commit the offset" there is a window in which the process can die, and on restart the consumer cannot tell whether the charge went through. Duplicate delivery is not eliminated; it has been pushed to the boundary where it is your problem.

Some queues offer a weaker but practical version: a deduplication window in which a resent message with the same deduplication identifier is discarded. That is real and it helps, but it is bounded — a retry arriving after the window is a new message — so it is a mitigation, not a guarantee.

Therefore: build the consumer to survive redelivery

Since at-least-once is what you actually have, the engineering work is making a second delivery harmless, and this is the part interviewers want to hear. Give every message a stable business identifier, generated by the producer, not the broker, and not a timestamp. Make the effect idempotent against that identifier: a conditional insert on a unique constraint, an upsert keyed on it, or a processed-messages table written in the same local transaction as the effect so committing the work and recording that it was done are one atomic act. When the effect is a call to someone else's API, pass the identifier as an idempotency key and let them deduplicate; when they will not, keep your own record of the outcome before you call and reconcile afterwards.

The same reasoning applies on the producing side, which is where the transactional outbox earns its keep. Writing a row to your database and publishing an event about it are two systems again, so doing them separately means either publishing an event for a transaction that rolled back or committing a change nobody hears about. Writing the event into an outbox table inside the same local transaction makes the two atomic, and a relay then publishes from the outbox with at-least-once semantics that your idempotent consumers already tolerate.

Finally, keep lag as the primary health signal. Both shapes fail in the same direction under sustained overload: the backlog absorbs the mismatch and converts a throughput problem into a latency problem that no error rate will show you. A pull-based consumer applies backpressure naturally by simply fetching more slowly; a push-based broker needs an explicit prefetch or credit limit, or a single slow consumer will be handed more work than it can hold.