Consumer lag on your orders topic climbs every morning and never clears before the next peak. Work through it.

Lag is an integral, so only two things can cause it

Consumer lag is the difference between the log end offset and the committed offset, which means it is the accumulated gap between how fast messages arrive and how fast you process them. It rises whenever arrival rate exceeds service rate and falls whenever the reverse holds. There is no third mechanism. Every hypothesis you form has to land on one side or the other, and saying that out loud stops the diagnosis wandering.

"Climbs every morning and never clears" says something more specific: the daily peak exceeds capacity, and the trough is not deep enough or long enough to drain what accumulated. So you are either short of throughput at peak, or you have lost throughput recently and the trough that used to be sufficient no longer is. Look at when the pattern started and whether it correlates with a deploy, a traffic increase, or a change in the shape of the messages rather than their count.

Is it slow, or is it stopped?

These look identical on a lag graph and need entirely different fixes, so separate them before anything else. A slow consumer is committing offsets steadily, just not fast enough. A stalled one is not committing at all, or is committing and then re-consuming the same offsets.

The tell is the offset commit rate and the records-consumed rate together. If the consumer is processing thousands of records per second and lag still climbs, it is a throughput shortfall. If records are being consumed but the committed offset barely advances, or advances and jumps backwards, you have a rebalance loop, and that is worth checking early because it is self-sustaining.

flowchart TD
    A[Batch takes longer than max.poll.interval.ms] --> B[Broker evicts the consumer]
    B --> C[Group rebalances]
    C --> D[Partitions reassigned, uncommitted work redone]
    D --> E[Effective throughput drops further]
    E --> A

What makes this loop vicious is the last arrow rather than the first: every pass through it destroys work already done, so the condition that triggered it gets worse each time round and the group can spend most of its life rebalancing rather than consuming. The signature in the logs is repeated group-join messages and offsets that go backwards, and the fix is either processing fewer records per poll or raising max.poll.interval.ms to exceed the worst-case batch time — not adding instances, which makes rebalances more frequent.

Per-partition lag, not group lag

The aggregate number hides the most common real cause. Kafka guarantees ordering within a partition and distributes by key hash, so an unbalanced key space concentrates traffic. One tenant generating a third of your events puts a third of your volume on one partition, and one consumer instance owns that partition, so the group's total capacity is irrelevant to it. Twenty-three partitions at zero lag and one at four million is a completely different problem from twenty-four partitions at 170,000 each, and only the second one is a capacity problem.

Check the assignment too. With more consumers than partitions the extras are idle, holding no assignment at all, which is why "we scaled to forty pods and nothing improved" is such a common report. The partition count is a hard ceiling on consumer parallelism, and raising it does not retroactively redistribute the messages already sitting in the old partitions — those still have to be drained by their current owners.

Where the time actually goes

Once you know it is a throughput shortfall, the work is per-message cost, and the cost is almost never in Kafka. Profile or trace one message end to end. The usual finding is a per-message synchronous database round trip, or several, so the consumer spends its life waiting on I/O rather than on CPU. The fix is batching: accumulate the records from one poll, do one bulk insert or one multi-row upsert, commit once. An order-of-magnitude improvement from batching is routine, and it costs nothing in ordering because the batch comes from one partition anyway.

The second usual finding is a downstream call — an HTTP request per message to a service that answers in 40ms, which caps you at 25 messages per second per thread no matter what you do to the consumer. Either batch that call as well or increase in-flight concurrency deliberately, and be explicit about what concurrency does to ordering: processing several messages from one partition in parallel abandons per-key ordering unless you shard the in-process work by key. If ordering per key is a real requirement, say so and shard; if it is not, say that too, because a lot of ordering requirements are inherited rather than needed.

The third is a poison message. If one record fails and the consumer retries it forever, the partition stops advancing and lag on that partition grows linearly while every other partition looks healthy. That is head-of-line blocking, and the answer is a bounded retry count followed by routing the record to a dead-letter topic with enough context to reprocess it. The part that is usually missed is the alarm: a dead-letter topic nobody watches is a data-loss mechanism with extra steps.

Fixing the backlog is not the same job as fixing the rate

Two distinct decisions, and conflating them is what leaves teams in this state for weeks. Restoring the steady-state rate is the engineering fix above. Draining four hours of accumulated backlog while still serving live traffic needs more capacity than steady state does, temporarily, and the levers are different: raise the partition count for future traffic, spin up a separate group to reprocess the backlog into a staging area, or accept that live processing takes priority and the backlog drains over the following night.

Be careful about what you break in a hurry. Increasing partitions changes the key-to-partition mapping, so messages for a key can land in a different partition from the ones still queued for it, and ordering across the transition is not preserved. That is often acceptable and it is never acceptable silently.

Two errors that show up in almost every answer

The first is reaching for consumer count. It is the intuitive lever, it is the one exposed in every deployment tool, and beyond the partition count it does exactly nothing while making rebalances more frequent and each one more expensive. Establish the assignment picture first.

The second is treating lag as a latency metric. Lag in messages tells you how much work is outstanding, not how stale the data is, and those diverge exactly when it matters: 100,000 messages is a few seconds of delay at peak and half an hour in the overnight trough. If the business cares about staleness, measure the age of the record you just processed — the difference between now and its producer timestamp — and alert on that. Lag in messages is the diagnostic; time lag is the thing anyone outside the team actually cares about.