A job queue is backed up four hours and half the jobs are now pointless. What do you drain, and what do you drop?

Short answer

For a four-hour job queue backlog, first calculate drain time from backlog size and surplus throughput, then classify jobs by value. Expired jobs should be shed with an audit record, superseded jobs should collapse to the newest desired state, and required jobs such as invoices or webhooks must drain fully. You can only make that decision safely if each job carries a deadline, type and collapse key from enqueue time.

Work out how long draining takes, because it decides everything else

Before touching anything, put arithmetic on the board. Say jobs arrive at 1,000 a second and the fleet processes 1,200. Four hours of backlog at that arrival rate is about 14.4 million jobs, and you are clearing them with 200 jobs a second of spare capacity. That is twenty hours to catch up, assuming arrivals do not rise, which they will.

Now the same sum answers the "just scale the workers" reflex. Clearing 14.4 million jobs in thirty minutes needs 8,000 jobs a second of surplus, which is roughly eight times the current fleet. Sometimes that is available and it is the right answer. Often it is not, and knowing which within a minute is the difference between a plan and a hope.

The number also tells you whether dropping is even necessary. A backlog that clears in twenty minutes needs patience. One that clears in twenty hours needs a decision, because the four-hour-old notifications at the head of the queue will be a day old by the time anybody sees them.

Check the bottleneck before adding consumers

More workers only help if the workers are the constraint. Look at consumer CPU first. If they are busy, the fleet is the limit and scaling out works. If they are idle and the backlog is still growing, they are blocked on something downstream, and every worker you add takes another connection from a database that is already the problem.

This is the sequence where scaling out makes the incident worse. A saturated dependency responds more slowly under more concurrency, so throughput falls as you add consumers, and the backlog grows faster after your intervention than before it. Rate-limit towards the dependency instead and accept the slower drain, because there is no configuration in which a queue drains faster than its slowest downstream call allows.

Also rule out a poison message. A job that fails, is redelivered, fails again and holds a partition or a visibility window can produce a growing backlog with healthy-looking consumers and no obvious cause. If the failure rate is high and the backlog is growing, the retry loop may be the entire workload.

Three categories, and only one of them is drained

Sort the backlog by whether the work still has value.

• Expired. The job had a moment and the moment has gone. A push saying a driver is arriving, a two-minute one-time code, a "someone is typing" indicator. Delivering these late is not neutral, it is wrong, and every one you deliver costs a user's trust and possibly a support ticket.
• Superseded. A later message about the same subject makes the earlier one irrelevant. Six updates to one search-index document, four recalculations of one basket total, a sequence of presence changes ending in offline. You need the last one. The other five are pure cost.
• Still required. Receipts, invoices, ledger postings, webhooks a customer's system is waiting on, anything with an audit obligation. Late is bad. Never is a breach, so these drain in full however long that takes.

A kitchen with two hundred unserved tickets works the same way. Some tables have left, several tickets are the same table changing its mind, and a few are large parties still waiting patiently. The analogy fails at the moment of judgement: a waiter can walk out and look at the room, and your consumer cannot. It only knows what the message says.

Which is why the message has to say

Here is the part that decides whether the previous section is a plan or a wish. A job that carries no deadline and no collapse key cannot be classified as expired or superseded, because the information required to make that call was never written down. At 3am, facing fourteen million opaque payloads, your only options are drain everything or drop blindly.

So both fields go on at enqueue time.

{
  "type": "push.driver_arriving",
  "ride_id": "r-8842",
  "not_valid_after": "2026-08-12T14:36:00Z",
  "collapse_key": "push.driver_arriving:r-8842"
}

The not_valid_after field lets a consumer discard the job in microseconds with no downstream call, which also means an expired backlog drains at enormous speed rather than at your dependency's speed. The collapse_key lets you keep only the newest job per key, so six index updates for one document become one. Both are cheap to add and impossible to retrofit into messages already in the queue, which is the argument for adding them before you need them.

Two cautions worth voicing. Collapsing changes ordering semantics, so it is only safe where the payload is a full desired state rather than a delta; collapsing two increments loses one of them. And expiry has to be evaluated against the deadline in the message, not against a duration since enqueue that a redelivery would reset.

Dropping is a recorded action, not a deletion

Purging a queue is the fastest way to end an incident and the fastest way to lose an argument the following week. Somebody will ask what was in it. "We do not know" is not an answer that survives contact with a customer.

Shed to somewhere durable instead. Move the jobs you are abandoning into a separate topic, table or bucket, tagged with the reason and the time. That gives you three things you would otherwise lack: a count to report, the ability to replay a subset once you understand it, and evidence for the question about a specific customer's job. It costs one write per shed job, which is far cheaper than processing it.

Also separate dropping from deferring. Some work is not pointless, merely badly timed, and moving it to a low-priority queue that drains overnight preserves it without holding up the work that matters now. Deciding case by case, and saying which is which out loud, is the whole skill: "Expired pushes I shed with a record. Index updates I collapse. Invoices I move to a dedicated queue and drain in full, and I will tell finance they are four hours late."

The reason one backlog delayed everything

Finally, name the structural cause, because the interviewer is waiting for it. If notifications, index updates and invoices share a queue, then a slow dependency behind any one of them blocks all three. That is head-of-line blocking, and it is the reason a four-hour backlog is a company-wide event rather than a feature being slow.

Split by class of work, not by convenience. Different queues get different consumer pools, different concurrency limits and different expiry policies, and a failure in one becomes a failure in one. Do the same for tenants if a single customer can generate enough work to fill a queue, since otherwise their bad afternoon is everybody's.

A backlog is not a volume problem, it is a triage problem, and triage needs data. The deadline and the collapse key you add today are the only reason a future incident has any option other than drain everything or purge everything.

---

© 2026 Preptima. Originally published at preptima.com.