A load test shows p99 latency climbing sharply past 500 concurrent users. How do you work out why?

Read the result before you touch a profiler

The instinct this question is designed to catch is jumping straight to a tool. Before you open anything, the result you already have answers most of the question, provided you plot two curves against concurrency rather than one.

Plot throughput and latency together. If throughput rose with concurrency up to some point and then went flat while latency kept climbing, you have found a saturation knee: the system reached its maximum rate of work somewhere below 500 users, and every user you added after that is standing in a queue. Latency past the knee is mostly waiting time, and it will grow roughly in proportion to the number of users you keep adding, because each of them is behind the same fixed-rate server. If instead throughput is still rising in line with concurrency and latency has merely got worse, nothing is saturated yet and you are looking at something else: a resource getting slower under load rather than running out, such as a query whose plan degrades as a cache stops fitting, or lock contention that grows with the number of concurrent writers.

If throughput actually fell after the knee, that is a third and more serious signal. Falling throughput under rising load means the system is spending capacity on overhead rather than work: thread thrashing, garbage-collection death spiral, retry storms amplifying the original load, or timeouts causing clients to abandon and resubmit.

Saturation is not the same observation as slowness

This is the distinction the interview is really probing. Utilisation is how busy a resource is. Saturation is how much work is queued waiting for it. They come apart in both directions, and confusing them is what produces wrong conclusions.

A resource at 100 percent utilisation with no queue is fine — it is fully used, which is what you paid for. A resource at 40 percent utilisation with a growing queue is your bottleneck, and this is the case candidates miss. A connection pool of 50 with 400 threads waiting for one is completely saturated while the database it fronts sits nearly idle, because the pool is the constraint and the database never sees the load. The same shape appears with a fixed worker thread pool, a semaphore-limited downstream client, or a single-threaded consumer on a partition.

Queueing theory gives the intuition for why the curve turns up so violently. For a simple single-server queue, response time is service time divided by one minus utilisation. At 50 percent utilisation you wait about as long as you are served; at 90 percent you wait nine times as long; at 95 percent, nineteen. Response time does not degrade linearly as you approach the limit, it goes vertical, which is exactly why "it was fine at 400 users" tells you very little about 500.

Walk the resources, do not guess them

Take every resource in the request path and ask three questions of each: how utilised is it, is work queued for it, and is it returning errors. Do that in order along the path rather than starting where you suspect.

Start with the load generator, because a saturated generator is common and produces exactly this graph. If the generator's own CPU is pinned, or it is running out of ephemeral ports or file descriptors, or its measurement thread is being descheduled, the latency it reports includes its own queueing and the system under test may be perfectly healthy. The cheap check is to run the same load from two generators and see whether the knee moves.

Then, at the application tier: CPU, and specifically whether run-queue length exceeds core count; heap behaviour and garbage-collection pause time and frequency, which show up as a latency profile with a clean bimodal shape rather than a smooth curve; thread state, where a thread dump taken three times ten seconds apart tells you immediately whether threads are runnable, blocked on a monitor, or parked waiting on a pool.

Then the connection pool between app and database, which deserves naming separately because it is the single most common answer to this exact question. Look at pool wait time, not pool size. If the mean time to acquire a connection is climbing, the bottleneck is behind the pool or the pool is too small, and those need different fixes.

Then the database: active sessions versus configured maximum, lock waits, the top queries by total time rather than by mean time, buffer cache hit rate, and disk latency. And finally the network and any downstream service, where the useful metric is the difference between the time the app measures for a call and the time the callee measures for the same call.

Confirm by intervention, not by correlation

Correlated graphs are how wrong root causes get published. At the knee, everything looks bad simultaneously, because saturation propagates: a slow database makes app threads pile up, which makes CPU look busy, which makes the generator's numbers worse. The resource that is causing it is the one whose queue started growing first, so line the metrics up on a single time axis and look at ordering rather than at magnitude.

Then prove it by changing one thing. Raise the pool from 50 to 100 and re-run: if the knee moves to a higher concurrency, the pool was the constraint. If the knee stays at exactly the same place and the database now shows lock waits, you have simply pushed the queue one layer down, which is progress but not a fix. A performance investigation is finished when you can state the limiting resource, the number it is limited at, and what the next limit will be after you raise it.

concurrency:  100   200   300   400   500   600
throughput:   480   940  1380  1520  1530  1490   <- flat from 400: this is the knee
p99 (ms):      95   110   180   410  1250  3100   <- rising because of queueing, not work
pool wait:      1     2     9    140   890  2600   <- started growing first: the constraint

The failure that makes this worse instead of better

The specific mistake to avoid is prescribing capacity before identifying the constrained resource. "p99 is bad past 500 users, so we add app servers" is the answer that gets the incident repeated, because if the constraint is a shared database or a fixed pool of licences or a rate-limited downstream, adding instances in front of it increases the number of clients contending for the same fixed thing. Latency gets worse, not better, and now it does so at lower concurrency per node so the graphs become harder to read.

The related version is treating the client-side p99 as though it were a statement about the server. It is a statement about the whole path, generator included, and until you have a server-side latency histogram to compare it with, you do not know how much of that 99th percentile is time your code spent and how much is time your load tool spent waiting to send.

A latency curve on its own is a symptom; the throughput curve next to it tells you whether you are looking at a queue or at work getting slower, and those two findings lead to completely different fixes.