An unchanged p50 with a tenfold p99 is a strong piece of evidence, and the first thing to do is say what it excludes. A uniform slowdown is out: a new synchronous call, a serialisation change, a slower framework version, or a smaller instance size would shift the whole distribution and drag the median with it. Nothing that happens on every request is responsible.
That leaves two shapes, and separating them is the entire diagnosis. Either the tail is a distinct population of requests that were always going to be slower and have now become much slower, or every request is drawing from the same distribution and a small fraction of them are being caught by a stall that has nothing to do with what they are asking for. The first is a bimodal population: large tenants, a particular endpoint, cache misses, requests with big payloads. The second is a queueing or contention effect: a garbage-collection pause, a saturated connection pool, a lock, a thread pool with a backlog. Both produce the same picture on a p50/p99 graph, which is why the graph alone cannot tell you which one you have.
Partition the histogram before proposing a cause
The first move is not a hypothesis, it is a slice. If the metrics are exported as histograms rather than pre-computed quantiles, you can recompute the tail along any dimension you already label, and each slice answers a question.
Splitting by instance tells you whether one pod is dragging the fleet. If a single instance out of twenty is slow, the tail is close to 5% of traffic, which is enough to move p99 substantially while leaving the median untouched, and you are now debugging one machine rather than a code change. Splitting by endpoint tells you whether the regression is confined to one handler, which points at the diff. Splitting by tenant or account tells you whether it tracks data volume, which is the signature of a query plan that changed or an index that is no longer being used for the accounts big enough to care.
# Fleet-wide p99: the number in the alert, and the least informative view.
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# The same tail, per instance. One slow pod is invisible above and
# obvious here. Note the aggregation happens on the raw buckets -
# summing counters and then estimating is correct; estimating per
# instance and then averaging the results is not a percentile at all.
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, instance))
Matching each mechanism to the evidence that confirms it
Once a slice narrows things down, each candidate cause has a measurement that settles it, and you should name the measurement rather than the cause.
Garbage collection is worth checking early because a deploy is a plausible trigger: a change that raises allocation rate increases pause frequency, and more frequent pauses mean a larger share of requests land inside one. The confirming evidence is GC logs correlated in time with the slow requests, not a hunch about heap size. If pause timestamps line up with the tail and the pauses are the right order of magnitude, you have it; if the pauses are two milliseconds and the tail is two seconds, GC is exonerated and you stop talking about it.
Connection-pool saturation shows up as time spent waiting to acquire, before any query runs. Traces make this visible as a gap between the start of the database span's parent and the query itself, and most pool implementations expose an acquisition-wait timer directly. The important subtlety is that pool exhaustion is usually a symptom: connections are held longer because queries got slower, or because a code path now holds one across a remote call. Raising the pool size moves the queue from your application into the database.
A missing or newly unused index is confirmed by looking at the actual plan for the slow tenant's parameters, not the fast one's. Plans that switch on row-count estimates are exactly the mechanism that produces a clean bimodal split, because the same statement is fast for a tenant with hundreds of rows and terrible for one with millions.
A cold cache and JIT warm-up both fit "after a deploy" and share a distinguishing signature: they decay. If the tail improves steadily over minutes after each instance starts serving, and does so again on the next restart, you are looking at warm-up rather than a defect in the change. A retry storm has the opposite signature — it is self-sustaining, and it shows as inbound request count at the dependency exceeding what callers report sending.
Lock contention is the one that hides from request-scoped metrics. Thread dumps taken a few seconds apart while the tail is elevated will show threads parked on the same monitor, and a wall-clock profiler attributes the time properly where a CPU profiler shows an idle box.
Confirming rather than guessing
The discipline being scored is whether you commit to a falsifiable statement before touching anything. "If this is pool saturation, the slow traces spend most of their time before the first query, and the wait timer is elevated on the affected instances only" is a claim that can be wrong. "It's probably GC, let's bump the heap" is not, and if the latency happens to improve you have learned nothing and may have masked the real cause.
Where the evidence for a mechanism is genuinely absent, the honest next step is to say what instrumentation you would add — exemplars linking the slow histogram buckets to specific trace IDs are the cheapest way to turn an aggregate into an example you can read. And the deploy itself is a hypothesis, not a given: a rollback that fixes it confirms the change was involved, while a rollback that does not tells you something coincided with the deploy window, such as a batch job, a traffic-mix shift, or a dependency of your own being deployed.
Where this diagnosis usually goes wrong
Almost everyone names a cause in the first sentence. The tell of a weaker answer is not picking the wrong mechanism, it is treating the fleet-wide p99 as a single number that describes one thing, and then reasoning about that number instead of about the requests underneath it. Two specific errors follow from it. The first is arithmetic: percentiles do not average, so a dashboard that computes a p99 per instance or per minute and then means them together reports a value that no request experienced, and can hide the single bad instance you are looking for. Merge the underlying histograms, then estimate once.
The second is scope. A tail measured over requests is not a tail measured over users. If a page issues thirty backend calls, a 1% per-call failure to meet latency touches roughly a quarter of page loads, so "only 1% of requests" is not the reassurance it sounds like — and it also explains why an unchanged p50 does not mean users are unaffected. The candidates who do well hold both facts: the median tells you the common path is intact, and it tells you nothing whatsoever about how many people noticed.
Diagnose the shape before the cause: a flat median means you are looking for either a subset of requests or a queue, and every hypothesis you name should come with the measurement that would prove it wrong.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).