CPU saturates on every node minutes after a release that contained forty commits. How do you find the cause?

Mitigate before you diagnose

The first decision is not technical. A saturated fleet is an outage, and diagnosis on a burning system is slow, stressful and prone to the wrong conclusion. If the release is revertible, revert it, and confirm CPU returns to its previous level — that both restores service and establishes causation, which is worth a great deal on its own. If the change sits behind a flag, turn the flag off. If neither is available, buy headroom by adding capacity or shedding load, accepting that this treats the symptom.

Say this out loud in an interview, because a candidate who begins profiling a production incident without stabilising it first has signalled something about how they behave under pressure. Then be explicit that the revert is not the end: an unexplained regression that was rolled back will be reintroduced by the same commit next week.

Preserve evidence on the way out. Capture a profile, a thread dump or the equivalent for your runtime, and a snapshot of the relevant metrics before the rollback removes the conditions. Rolling back without capturing anything means the investigation restarts from a clean environment where the problem no longer exists.

Rule out the simplest explanation first

Before assuming the code changed for the worse, check whether the work changed. CPU is a product of the amount of work per unit time, so it rises either because there is more work or because each unit costs more.

Request rate is the first thing to look at, along with the mix. A deploy that coincides with a marketing send, a retry storm from a client whose timeouts the release shortened, or a cache that was flushed on restart and is now missing on every request, all produce saturation without a single inefficient line of code. The cold-cache case is particularly worth naming, because it looks exactly like a code regression, resolves itself if you wait, and is caused by the deploy rather than by the diff.

If the rate and mix are unchanged and CPU per request has risen, the cost of a request went up, and now the code is genuinely implicated. Establishing that split takes a minute and prevents an hour of reading the wrong forty commits.

Profile the process, do not read the diff

With forty commits, reading the diff produces a list of plausible suspects with no way to rank them, and the temptation is to pick the change that looks most complicated. That is guessing dressed as analysis. A CPU profile of the saturated process answers the question directly: it tells you which functions are consuming the cycles, in what proportion, and along which call paths.

Take an on-CPU profile from an affected node while it is saturated, aggregate it as a flame graph, and read the widest frames. This is the whole technique. The width of a frame is the share of samples in which it was on the stack, so a single dominant frame gives you the answer immediately, and the call path above it tells you which request type reaches it. From there, comparing against a profile from a node that has not yet been upgraded, if you still have one, isolates the difference with no interpretation needed.

The July 2019 Cloudflare outage is the clean illustration of why this order matters. A newly deployed WAF rule contained a regular expression that backtracked catastrophically, CPU saturated across their global edge, and traffic was affected until the rule set was disabled. The cause was a single expression on a hot path, in a deployment that had many components, and it was located because the profile pointed at it rather than because anybody guessed which rule was expensive. Their own account of it is worth reading, since it also shows mitigation and diagnosis being handled as two separate activities.

flowchart TD
  A[CPU saturated after release] --> B{Request rate or mix changed}
  B -- yes --> C[Load or retry problem not code]
  B -- no --> D[Cost per request rose]
  D --> E[Profile a saturated node]
  E --> F{One dominant frame}
  F -- yes --> G[Map frame to a commit]
  F -- no --> H[Look at GC lock contention or a cold cache]

The branch that costs teams the most time is the middle one, because skipping it means an hour spent profiling a system whose code is fine.

When the profile is flat

A profile with no dominant frame is informative rather than a dead end. It usually means the cost is spread across something that touches everything: serialisation on a payload that grew, logging turned up to a more verbose level, a validation or encryption step newly applied to every request, or a client library that now retries internally so each logical request does two or three times the work.

It can also mean the CPU is not being burnt in your application code at all. Garbage collection shows as time in the collector or, in some tooling, as time attributed nowhere useful, and the giveaway is allocation rate rather than any single hot method. Spinning on a contended lock burns CPU while achieving nothing, and the tell is that throughput has not risen with the CPU. Time in the kernel points at system calls, which points at network or file activity per request. Each of these has a different fix, and the profile plus one supporting metric is enough to tell them apart.

Naming the commit, and proving the fix

Once the profile names a function, the commit follows from git log on that file, and the argument is over. Where the profile names a function that several commits touched, or the hot path is in a dependency whose version moved, a bisect over the release in a load-test environment is the fallback, and it is a fallback because it costs many builds and a reproducible load profile rather than ten minutes with a profiler.

Then prove the fix rather than declaring it. Re-run the same load against the patched build and compare CPU per request to the pre-release baseline, not to the broken build, because beating the broken build proves almost nothing. Keep the profile from the incident attached to the postmortem, since it is the only durable evidence of what the code was doing.

The habit underneath all of this is measuring before hypothesising. It feels slower for the first ten minutes and is far faster over the hour, and it is the difference between finding a regression and finding a change you already suspected.