Performance Diagnosis and Optimisation Interviews: A Masterclass
How senior interviewers grade the question every backend loop eventually asks: why is this slow, and how would you find out. Percentiles, queueing, profiling, the slow-query path, tail latency in a fan-out, metastable collapse, autoscaling that browns out, and how to report a finding a business can decide on.
Masterclass·104 min read
What it is
A performance interview is a reasoning interview wearing a systems costume. The interviewer says some version of "it got slow, what do you do", and everything they learn about you in the next forty minutes comes from the order in which you do things rather than from the list of things you know. Two candidates can name the same eight causes. One of them names them as a checklist and starts guessing; the other establishes what the symptom already rules out, picks the measurement that separates the surviving hypotheses, and only then says a word about fixes. The second candidate gets the offer, and frequently the first one had deeper knowledge of the individual mechanisms.
That is the whole subject. Performance work is not a body of tricks about indexes and caches. It is the discipline of finding out where time is spent, in a system that will not tell you unless you ask correctly, and then proving that a change moved the thing you claimed it would move. The tricks are real and you need them, but they are the second half. Candidates fail this round overwhelmingly on the first half, and the failure is almost never "did not know about connection pooling". It is "started optimising before establishing what was slow".
The subject has a stable shape that is worth holding in your head as a shape rather than as a list, because under interview pressure a list collapses and a shape does not. A symptom arrives, usually from someone who is not an engineer and whose description is not reliable. You establish what is actually true and reproduce it, or find a way to observe it in production if it will not reproduce. You localise it to a layer, which mostly means ruling layers out. Inside that layer you find where time is going, which means separating time spent working from time spent waiting. You form a hypothesis specific enough to be wrong, and you test it. You change one thing. You verify against the original measurement, in the same units, on the same population. And when the change did not help, which is common and is not a failure, you go back and re-measure rather than stacking a second change on top of the first.
Loading diagram…
The edge worth looking at is the loop back from verification to measurement, not the straight run down the middle. Everyone can describe the straight run; the candidates who are trusted with a real production problem are the ones who say out loud what they do when the fix did not work, because that is the branch the job spends most of its time in.
Measurement before change, and what that phrase actually commits you to
"Measure before you change" is repeated so often that it has stopped carrying information. It is worth unpacking into the three commitments it makes, because interviewers probe each of them separately.
The first commitment is that you have a number before you touch anything, and that the number is of the thing the complainant cares about. If the report is "the dashboard is slow", the baseline is time to a usable dashboard for the users complaining, not average server-side handler duration across all endpoints. Those can move in opposite directions. A candidate who baselines the wrong quantity will later demonstrate an improvement nobody experiences, which is a specific and common professional embarrassment.
The second commitment is that the measurement is repeatable. A single observation of a slow request is an anecdote. You need to be able to take the measurement again, later, after the change, under comparable conditions, and know that the difference between the two numbers is the change rather than the hour of day. This is why load-test-based performance work has such a strong pull despite its many problems: it buys repeatability. It is also why production measurement has to be paired with a stated population and window, because "the p99 last Tuesday" and "the p99 this morning" may be measuring two different traffic mixes.
The third commitment, and the one candidates skip, is that you can attribute the change. If you deploy an index, a cache and a pool-size bump together and latency halves, you have learned that one of three things worked and you now carry two changes of unknown value for the life of the system. One of them may be actively harmful in a way that will surface in six months under a different load shape. Changing one thing at a time is slower and it is the only way the second change is ever grounded in evidence rather than in the momentum of the first.
The vocabulary that keeps a diagnosis honest
A handful of terms do most of the work in this domain, and using them precisely is itself a graded signal because imprecision here is how people talk themselves into the wrong fix.
Service time is how long an operation takes when it is being worked on, with no contention. Wait time, or queueing delay, is how long it sat before anyone touched it. Response time, which is what your user experiences and what your latency metric usually records, is the sum. These are three different quantities and they respond to completely different interventions: service time falls when you make the work cheaper, and wait time falls when you reduce arrival rate, add servers, or shorten the queue. Optimising code to reduce service time when the problem is a queue is the single most common wasted week in this field.
Utilisation is the fraction of time a resource is busy. Saturation is the extent to which work is queued for it. The two are related but a resource can be at high utilisation with no queue and be perfectly healthy, and a resource can be at moderate utilisation with a growing queue because the work arrives in bursts. Reporting utilisation and calling it saturation is how a team concludes that a CPU at seventy per cent is fine while requests are visibly waiting to be scheduled.
Throughput is completed work per unit time. Goodput is completed work that somebody still wanted by the time it completed. A service under overload can have healthy-looking throughput and near-zero goodput, because every response arrives after the caller has timed out and retried. That distinction is the hinge of the metastable-failure discussion later, and volunteering it early marks you as someone who has been in one.
Concurrency is how many things are in the system at once. It is not a rate, and confusing it with a rate is a specific error with a specific cost, because limits expressed in requests per second do not adapt when service times change and limits expressed in in-flight requests do.
Performance questions have become a standard part of senior backend loops for reasons that are worth understanding, because they tell you what the interviewer is trying to find out and therefore what to show them.
The first reason is that the skill does not correlate with the skills the rest of the loop measures. Someone can be excellent at designing a system, fluent in the language, sound on data modelling, and still be unable to work out why the thing they built is slow, because diagnosis is a separate discipline with its own habits. Every team that has run an incident knows the difference between the engineer who narrows and the engineer who guesses, and hiring managers have learned to test for it directly rather than hoping it comes bundled.
The second is that the cost of getting it wrong is asymmetric and lands on the business. A guessed fix that appears to work is worse than no fix, because it closes the investigation. The team believes the problem is solved, the real cause continues to accumulate, and the next occurrence arrives with a changed signature and a stale explanation attached to it. Interviewers who have lived through that specifically hunt for the habit of confirming rather than assuming.
The third is that modern systems hide the answer better than they used to. When an application was a process on a machine talking to a database on another machine, the set of places time could go was small enough to enumerate by hand. Now a single user action can traverse a CDN, an edge function, an ingress, a service mesh sidecar, three services, two caches, a message broker and a managed database, each with its own pool, its own retry policy and its own idea of a timeout, running on shared hardware with noisy neighbours and CPU quotas that throttle in ways the process cannot see. The number of places time can hide has grown faster than most engineers' mental models, and the ones whose models kept up are worth identifying.
The fourth is that performance is where engineering meets money in a way that is unusually legible. Latency affects conversion, capacity affects cloud spend, and both are numbers a finance function understands. An engineer who can say "this change removes a third of the database load, which lets us defer the instance-class upgrade" is having a different conversation from one who says "the query is now faster". Senior loops test for the first conversation, and the reporting section at the end of this page exists because candidates who are technically strong routinely lose the offer on it.
The fifth is that this is where discipline is visible under pressure. Almost every other interview round lets you think in silence. A performance scenario is deliberately open, deliberately underspecified, and deliberately delivered as a vague complaint, because that reproduces the conditions of the job. What the interviewer sees is how you behave when you do not know the answer, which is the thing they most want to know and the hardest thing to fake.
Percentiles, and why the average is worse than useless
The average response time is not merely a weak summary. It is actively misleading, and the reasons are worth being able to state cleanly because "averages hide the tail" on its own is a slogan that interviewers hear from everybody.
Latency distributions are not symmetric. They have a floor, because no request can take less than the work it requires, and they have no ceiling, because a request can wait arbitrarily long. That produces a long right tail as a matter of structure rather than as an anomaly. The mean of such a distribution sits above the typical experience and below the bad experience, describing neither. It is pulled around by a small number of extreme values, so it moves when the tail moves and moves when the mode moves, and you cannot tell from the number which happened.
The second problem is that averages compose wrongly. If a page issues several backend calls and each call has a small chance of being slow, the chance that the page contains at least one slow call is much higher than the per-call chance, and it grows with fan-out. Averaging per-call latency tells you nothing about that. The arithmetic is worth being able to do out loud as an illustration: if each of thirty calls independently has a one-in-a-hundred chance of exceeding some threshold, the probability that none of them does is roughly ninety-nine hundredths raised to the thirtieth power, which is a little under three quarters. So a "one per cent" per-call tail touches something like a quarter of page loads. State it as illustrative arithmetic from assumed independence, not as a measurement, and note that real calls are not independent because they share the same slow instance.
The third problem is that percentiles do not average either, which is a distinct and less well-known error. You cannot compute a p99 per instance and take the mean, and you cannot compute a p99 per minute and take the mean over an hour. Neither result is a percentile of anything. If your metrics are exported as histograms you can merge the underlying buckets and estimate once, which is correct; if your metrics pipeline pre-computes quantiles per instance, the fleet-wide number on your dashboard is arithmetic nonsense and, worse, it specifically hides the single bad instance you are usually looking for. This is exactly the failure explored in the p99 spike with a flat p50, and volunteering it unprompted is a reliable senior signal.
Statistic
What it tells you
Who cares about it
Median
Whether the common path is intact
Engineers checking that a change did not shift the bulk of traffic
p90 to p95
The experience of the unlucky but not rare request
Product, because this is what a regular user hits occasionally
p99
Where queueing, GC and contention show up first
On-call, because it moves before the median does
p999 and max
Individual pathologies rather than a population
Whoever owns the specific pathology; rarely worth an SLO
Distribution shape
Whether you have one population or two
Anyone diagnosing, because bimodality changes the whole approach
The row that earns you credit is the last one. Percentiles are summary statistics and a summary can conceal the single most useful fact about a latency distribution, which is whether it has one hump or two. A bimodal distribution means there are two populations of request being served by one endpoint: cache hits and misses, small tenants and large ones, the fast path and the fallback. When the distribution is bimodal, the p99 is mostly telling you what fraction of traffic is in the slow population, and moving it means either shrinking that population or fixing the slow path, which are different projects. A heatmap or a histogram shows this in a second and a percentile line graph never will. Asking to see the distribution rather than the percentile is a small move that separates people.
One more distinction that matters in an interview: percentile of what population, over what window. A p99 over requests is not a p99 over users, because heavy users issue more requests and are therefore over-represented. A p99 over five minutes is not comparable to a p99 over a day. And a percentile computed only over successful responses excludes the requests that timed out, which are the slowest ones there are, so an endpoint can improve its latency percentile by getting worse. If you say nothing else about measurement hygiene, say that one, because it is the mechanism by which a dashboard lies while every individual number on it is correct.
Where the time actually goes
Once you have established that something is slow and how slow, the next question is not "what is the cause" but "in what proportion is this time spent working versus waiting". Everything downstream depends on that split, and there is a reliable way to reason about it that does not require inventing figures.
Queueing is why the graph bends
The single most important intuition in performance work is that response time does not rise linearly with load. It stays roughly flat, then bends, then goes vertical. The bend is queueing, and the reason it is a bend rather than a slope is that as utilisation approaches one, waiting time grows without bound: a resource that is busy almost all the time makes arriving work wait for almost all the work in front of it, and the amount in front of it keeps growing.
The practical consequences of that shape are what an interviewer wants to hear, and there are three.
The first is that a small increase in load near the bend produces a large increase in latency, which is why systems appear to fall over rather than degrade. Nothing changed qualitatively; you moved along a curve that was always there. It also means that a system running comfortably at some utilisation has far less headroom than the number suggests, and that headroom shrinks non-linearly.
The second is that variability moves the bend leftwards. If arrivals are bursty rather than smooth, or if service times vary widely, queues form at utilisations well below saturation, because a burst creates a queue that has not drained before the next burst arrives. This is why real systems queue at utilisations that a naive calculation says are fine, and why reducing the variance of service time is often a bigger win than reducing its mean. Making the slow requests less slow helps everyone waiting behind them; making the fast requests faster helps almost nobody.
The third is that adding capacity fixes wait time and does nothing for service time. If a request takes a long time because the work is expensive, doubling the instance count does not make any individual request faster; it merely lets you do more of them at once. Candidates who reach for horizontal scaling as a latency fix are revealing that they have not separated the two, and interviewers ask "would adding another replica help" precisely to find out.
Little's Law as a reasoning tool, not a formula to recite
The relationship that ties concurrency, throughput and latency together is that the average number of items in a system equals the average arrival rate multiplied by the average time each item spends in the system. It holds for any stable system regardless of the distribution of arrivals or service times, which is what makes it so useful: you can apply it to a thread pool, a connection pool, a queue, a whole service, or a single database.
Use it in an interview as a consistency check rather than as a calculation. The three quantities are linked, so if you know two you know the third, and if someone tells you all three and they do not agree then one of them is being measured wrong. That is frequently the most valuable thing it does. A team reports a throughput, a latency and a concurrency limit that are mutually inconsistent, and the resolution is usually that the latency is measured only over successful fast responses, or that the concurrency limit is not the binding one because a smaller pool sits behind it.
Use it also to reason about limits without needing numbers you do not have. If a downstream call takes some amount of time and your pool holds some number of connections, then the maximum rate at which you can complete those calls is the pool size divided by the call duration, and no amount of upstream capacity changes that. When the downstream slows, that ceiling drops proportionally, which means the pool that was ample at the normal service time becomes the bottleneck at the degraded one, without anybody changing a configuration. That single derivation explains most connection-pool exhaustion incidents, and it is far more convincing than saying "the pool was too small".
Use it, finally, to explain why a queue is not a solution to insufficient capacity. Adding a buffer in front of an under-provisioned consumer does not increase the rate at which work is completed; it increases the time each item spends in the system, which by the same relationship is exactly what latency is. A queue converts rejection into delay. That is sometimes what you want, if the arrivals are bursty and the buffer drains between bursts, and it is never what you want if the arrival rate exceeds the service rate on average, because then the queue grows until something breaks. Consumer lag that never catches up is that situation stated as a scenario.
USE and RED, and why you need both
Two framings are widely used to decide what to look at, and they answer different questions. Knowing both and knowing when each applies is worth more than fluency in either.
The USE method is resource-oriented: for every resource, look at utilisation, saturation and errors. Resources here means the physical and logical things that can be exhausted, and the value of the method is that it is a checklist over a complete set rather than an intuition about where to look. It is the right framing when you have a machine or a component that you suspect and need to find which resource is limiting.
The RED method is request-oriented: for every service, look at request rate, error rate and duration. It is the right framing when you have a user-visible symptom and need to work out which service owns it, because it is expressed in the units the symptom arrived in.
USE
RED
Unit of analysis
A resource such as CPU, disk, a pool, a lock
A service or an endpoint
The three signals
Utilisation, saturation, errors
Rate, errors, duration
Answers
Which resource is the constraint
Which service is slow, and how slow
Best used
Once you have localised to a component
At the start, to localise
Blind to
Work that is waiting on something not modelled as a resource
Which resource inside the service is responsible
The practical method is to start with RED to find the service, then USE inside it to find the resource, and the reason to say that out loud is that it demonstrates you understand them as complements rather than as competing brands. The other thing worth naming is what both miss: a request can be slow because of a resource nobody thought to model, most often a lock, a rate limiter, a leader election, or a shared external quota. Neither framework enumerates those for you. This is where distributed tracing earns its place, because a trace does not care whether you modelled something as a resource; it shows the gap in the timeline regardless.
A first-move table
The most useful thing you can carry into a diagnostic interview is not a list of causes but a mapping from symptom to first measurement, because the interviewer's first question is always effectively "where do you look first".
Symptom
Layer that usually owns it
First measurement to take
Everything is uniformly slower
Shared resource, or a change on every path
Compare the whole distribution before and after; check for a deploy or config change
Median flat, tail much worse
Queueing, contention, or a second population
Split the histogram by instance, endpoint and tenant
Slow only for some customers
Data volume, plan choice, or a per-tenant limit
Latency by tenant, then the plan for a big tenant's parameters
Slow only at a time of day
Batch job, cache expiry, or a scheduled peak
Overlay latency with concurrency and with the job schedule
Fast in isolation, slow under load
Queueing at some resource
Concurrency and wait-time metrics, not per-request timing
Slow after a restart, recovers
Cold cache, JIT warm-up, connection establishment
Latency as a function of time since instance start
Client sees slow, server reports fast
Network, TLS, proxy, serialisation, or client rendering
Timing from the client, and at each hop between
CPU low, latency high
Waiting, not working
Off-CPU or wall-clock profile, plus pool wait timers
Trend the latency against a size metric such as table rows or heap after GC
The row worth expanding is the client-versus-server disagreement, because it catches out a lot of otherwise strong candidates. If the server records a short handler duration and the user experiences seconds, the time is somewhere your server-side metric does not span: connection setup and TLS negotiation, DNS resolution, a proxy or mesh sidecar, time spent in the accept queue before your handler was entered, response body transfer over a slow link, or entirely client-side work after the bytes arrived. Server-side handler duration typically starts when the framework begins processing a request that has already been read, which means it structurally cannot see the queue in front of it. Saying that sentence is worth a great deal, because it demonstrates you know what your instrument excludes.
Profiling, and what each technique cannot see
Once you know a process is where the time goes, you profile it. The mistake that dominates this area is treating "profiling" as one activity when it is at least four, each of which is blind to what the others see.
A CPU profiler samples what code is executing on-CPU. It answers "what is this process burning cycles on" and it is the right tool when the box is busy and the work is computational. It is close to useless when the process is waiting, because a thread blocked on a socket or a lock is not on-CPU and simply does not appear. This is the single most common profiling error: the service is slow, the CPU profile is examined, nothing stands out, and the engineer concludes there is no hotspot when in fact there is no CPU work because everything is waiting.
A wall-clock or off-CPU profile samples where threads are spending elapsed time including time blocked. It answers "what are we waiting for", and it is the correct first choice for a service that is slow while the machine is idle. The trade is that it is noisier and includes threads that are legitimately parked, so it needs more interpretation.
An allocation profile samples where objects are being created rather than where CPU is spent. This matters because allocation is how you cause garbage collection, and GC cost frequently shows up in a CPU profile as time inside the collector with no indication of which code created the garbage. Fixing GC pressure means finding the allocation site, and only an allocation profile points at it.
A lock or contention profile records where threads block acquiring monitors and locks. Contention is close to invisible in the other views: the CPU profile shows an idle machine, the wall-clock profile shows threads parked without necessarily saying on what, and the request-scoped metrics show latency with no attribution. Taking two or three thread dumps a few seconds apart while the problem is happening is the crude version and is often enough, because a genuine contention problem shows the same monitor at the top of many stacks.
Technique
What it sees
What it is blind to
CPU sampling profiler
Hot code paths burning cycles
All blocking, all waiting, anything off-CPU
Wall-clock or off-CPU profile
Elapsed time including blocked time
Attribution when many threads park legitimately
Allocation profiler
Which code creates garbage
The cost of collecting it, and pause timing
Lock and contention profile
Which monitor threads pile up on
Contention inside the kernel or a native library
System-wide profiler with kernel symbols
Syscalls, page faults, scheduler waits, kernel time
Application-level semantics, which request is which
Distributed trace
Time attributed across services and spans
Anything inside a span with no child spans
Query-level statistics in the database
Which statement shape consumes total time
Time spent in the application between statements
The row that most often changes an investigation is the last one on tracing. A trace is only as informative as its instrumentation, and the classic disappointing trace is one where the whole request is a single span of several seconds with nothing inside it. That is not a trace telling you there is no detail; it is a trace telling you nobody instrumented the detail. The follow-up an interviewer likes is "what would you add", and the good answer is spans around the boundaries that can wait: pool acquisition, remote calls, serialisation of large payloads, and any lock or semaphore you hold across IO.
Two practical notes worth having. Sampling profilers in managed runtimes have historically suffered from safepoint bias, where samples can only be taken at points the runtime is willing to pause and those points are not uniformly distributed through the code, so the profile is skewed towards methods that happen to contain them. Profilers that sample using operating system signals and walk the stack without waiting for a safepoint avoid this, which is why they are preferred for serious work. And profiling in production is normal and expected at senior level; a candidate who says they would reproduce the load in staging and profile there has to explain how the staging data volume, the traffic mix and the neighbour noise are going to match, and usually cannot.
# A wall-clock profile, not a CPU profile: the difference is the whole point when# the box is idle and requests are still slow. 'wall' includes blocked threads.
./profiler.sh -e wall -d 60 -f /tmp/wall.html <pid>
# The same process, allocation instead: this is what points at the code# creating the garbage, which a CPU profile of the collector never will.
./profiler.sh -e alloc -d 60 -f /tmp/alloc.html <pid>
The general habit to display is stating, before you run anything, what you expect the profile to show if your hypothesis is right. A profile examined without a prior expectation always contains something that looks suspicious, and the human tendency is to optimise it whether or not it was on the critical path.
The slow-query path, from a saturated database to a plan
Databases deserve their own treatment because they are the most common home of the missing time and because the diagnostic path there is unusually well-defined. The path runs from the whole instance down to a single node in a plan, and the discipline is not to skip levels.
Loading diagram…
The step people skip is the second one, and skipping it is why so much query tuning achieves nothing.
Establish what sessions are waiting on before you look at any SQL
A database that is "slow" is a database whose sessions are waiting, and the first question is what for. The categories are broadly the same across engines even though the names differ: waiting for CPU, waiting for a read from storage, waiting for a write to be durable, waiting on a lock held by another session, waiting on an internal latch or buffer, or waiting on the client to send the next thing. Each of those points at a completely different remedy, and the SQL text is only relevant for some of them.
If sessions are waiting on locks, no amount of index tuning helps, because the statements are not slow; they are blocked. That is a concurrency and transaction-scope problem, and the fix lives in how long transactions are held open and in what order rows are touched, as in deadlocks under load. If sessions are waiting on storage reads, the working set no longer fits in memory and you are looking at a caching, memory-sizing or data-volume problem before you are looking at a query problem. If sessions are waiting on commit durability, the bottleneck is write path and the fix may be batching or grouping rather than anything about reads. Only if sessions are burning CPU inside query execution is the plan the primary suspect.
Saying this ordering out loud is a strong signal because it is the difference between someone who has operated a database and someone who has tuned queries on a laptop.
Rank by total time, not by worst case
Once you are looking at statements, rank them by the total time each statement shape consumed over a window, which is the number of executions multiplied by the mean duration, rather than by the slowest single execution. The cheap statement executed constantly usually outranks the report that takes ten seconds once an hour, and the cheap statement is also frequently the one causing the saturation that makes everything else slow. This is the argument developed in deciding which queries are worth tuning, and it is worth pairing with the observation that a statement is only worth tuning if the resource it consumes is the one that is scarce. Halving the CPU cost of a query on a system bottlenecked by storage reads changes nothing.
Two measurement subtleties are worth volunteering. Statement statistics normalise parameters, so a shape that is fast for most parameter values and terrible for a few is reported as an average that describes neither, and you need per-execution logging or a sampling mechanism to find the bad values. And the statistics are cumulative since some reset point, so comparing two windows requires taking a snapshot at each end and subtracting rather than reading the totals.
Read the plan against real parameters
Now, and only now, look at the plan. The single most useful thing in an execution plan is not the operator names but the comparison between the rows the planner estimated and the rows the operation actually produced, which is why an executed plan is worth far more than an estimated one. A large gap means the planner made its choice on a false premise, and every decision above that node in the tree inherits the error. A nested loop chosen because the inner side was estimated at a handful of rows is catastrophic when the inner side returns a great many, and the operator is not the problem; the estimate is.
-- Executed plan, abbreviated. The gap on the inner side is the finding.
Nested Loop (actual rows=482913 loops=1)
-> Seq Scan on orders o (actual rows=91 loops=1)
Filter: (status = 'PENDING')
-> Index Scan using idx_items_order on order_items i
-- planner expected a handful per order, got thousands, and the
-- loop count multiplies it: this is where the time went
(actual rows=5307 loops=91)
The specific reading habits that separate a strong answer are that a node's reported per-loop cost has to be multiplied by its loop count to get its real contribution, that times shown are typically cumulative for the subtree so the interesting quantity is a node's cost minus its children's, and that the plan tells you what the database did rather than why it chose to, so a suspicious choice sends you to statistics, to the predicate's sargability, or to parameter sensitivity. How to read a PostgreSQL execution plan works through the mechanics properly.
Estimate errors have a small number of usual causes and naming them is worth credit: stale statistics after a bulk load, correlated predicates that the planner assumes are independent so it multiplies selectivities that should not be multiplied, a function or expression wrapped around a column so no statistics apply, a data-type mismatch forcing a cast that prevents index use, or a parameter distribution where one plan is right for common values and another for rare ones. The last is why the same statement can be fast from a console and slow from an application, which is the whole subject of the same query fast in psql and slow from the application, and the mechanism is that the application sends bind parameters so the planner may be choosing a plan without the literal values.
The related trap is the index that exists and is not used, which has its own set of causes ranging from the predicate not being usable against the index, to the planner correctly deciding a scan is cheaper for the fraction of rows involved, to a leading-column mismatch on a composite index. Why is my index not being used and column order in a composite index cover the ground. And the reflex to add an index needs the counterweight that indexes are not free: they are maintained on every write, which is the argument in fourteen indexes and slower writes. An interviewer who hears "I would add an index" and no mention of write cost has learned something.
There is one further database-side cause that is invisible from the query text entirely, which is bloat and version accumulation under multi-version concurrency control. A table whose dead row versions are not being reclaimed grows, so scans read more pages for the same logical rows and the working set stops fitting in memory, and the immediate cause is often a long-running transaction holding back cleanup. What a row version costs under MVCC and the oldest transaction id that keeps climbing are the two halves of that story, and mentioning it in a query-tuning discussion is a strong signal because it explains a class of gradual slowdown where every individual query looks reasonable.
Application-level pathologies that a query-level view will never show
A database can be entirely healthy, every statement fast, and the application still slow, because the problem is not any one operation but the number of them.
N+1, and the reason it survives review
The N+1 pattern is that code fetches a collection and then, for each member, issues another query or call to fetch something related. Every individual statement is fast, the database reports no slow queries, the code reads naturally, and the endpoint takes seconds. It is the archetypal case of a problem that is invisible at the level everyone is monitoring.
The mechanism to name is that the cost is dominated by round trips rather than by work. Each call pays network latency, protocol overhead, and in many stacks a pool acquisition and a transaction check, and those fixed costs are paid once per member of a collection whose size is determined by data rather than by code. That is why the endpoint is fast in development, where the fixture has three rows and the database is on the same machine, and pathological in production, where a customer has thousands and the database is a network hop away. The illustrative arithmetic is trivial and worth doing aloud: some small per-call overhead multiplied by a collection size that grows with customer success.
The fixes are enumerable. Fetch the related data in one statement with a join or a batched predicate. Fetch it in a second statement keyed by the collected identifiers, which is two round trips regardless of collection size and is often preferable to a join because it avoids row multiplication. Use whatever batching facility the data access layer offers, and know that most object-relational mappers default to lazy loading precisely because it is convenient, which is why the pattern is so common in code written with one. Or restructure so the collection is bounded, which is where pagination belongs; unbounded collection endpoints are a latency problem before they are anything else, as paginating a collection endpoint at scale sets out.
The detection answer matters as much as the fix. You find N+1 by counting queries per request rather than by timing them, and a request-scoped counter that logs or alerts when the count exceeds a threshold catches the whole class permanently. Interviewers like this because it generalises: the instrument that catches a class is worth more than the fix for an instance.
The same shape between services
The distributed version is the chatty interaction, and it is worse for two reasons. The per-call cost is higher, because a service call carries serialisation, connection handling, authentication, and possibly a mesh hop on each side. And the failure semantics compound, because each additional call is another opportunity for a timeout or a retry.
The remedies mirror the database case but with an extra one: change the interface. If a caller consistently needs three fields from a related entity for every member of a list, an endpoint that returns them is a legitimate design response rather than a leaky abstraction, and refusing it on purity grounds while the page takes seconds is the wrong trade. Where the composition genuinely spans owners and includes a filter or a sort, the composition may not be possible at all at request time, which is the argument in one screen, three services and a 200ms budget: either one service drives and the rest hydrate its page of identifiers, or you build a read model and pay for staleness deliberately.
Connection pool exhaustion and thread starvation
These are two names for the same shape at different layers, and they are worth treating together because the reasoning transfers.
A pool of limited size sits in front of a resource. Work arrives, takes a slot, holds it while it does something, and returns it. By the relationship discussed earlier, the maximum completion rate is the pool size divided by the hold time. When hold time rises, the ceiling falls, and if arrivals continue at the previous rate the excess queues at the pool. Requests that would previously have gone straight through now spend most of their duration waiting to acquire, and the latency added is not the downstream's slowdown; it is the queueing that the slowdown produced. The two are different magnitudes and it is common for the queueing to dominate.
The diagnostic signature is specific and worth memorising: time spent before the first downstream operation begins. A trace shows a gap between the parent span starting and the child span starting. Most pool implementations expose an acquisition-wait timer directly, and it is one of the highest-value metrics a backend service can emit, because it converts an invisible queue into a number.
The instinct is to raise the pool size, and the reason that is usually wrong is worth stating precisely: the pool is a rate limiter on your access to a shared resource, and removing it does not add capacity to the resource. Raising the size moves the queue from your application, where it is visible and bounded and cheap, into the database, where it is less visible, competes with every other client, and consumes memory and scheduling on the most expensive machine you own. Sometimes raising it is correct, specifically when the pool was sized for a hold time that no longer applies and the resource has genuine headroom. The evidence for that is resource-side, not application-side.
Thread starvation is the same story with threads as the pool. A fixed-size thread pool handling requests, each of which blocks on IO, has an in-flight limit equal to the pool size, and when downstream latency rises the pool fills and the accept queue grows. The particularly nasty version is when the same pool serves both the incoming requests and the callbacks or continuations that complete them, so a full pool cannot process the completions that would free it: a deadlock by exhaustion rather than by lock ordering. The structural fix is a bulkhead, meaning separate pools per dependency so one slow downstream cannot consume the capacity needed by every other path, which sits alongside timeouts and breakers in what reacts when a dependency slows down.
Two related traps are worth naming because interviewers use them. Holding a database connection across a remote call is the archetypal way to convert someone else's latency into your own pool exhaustion, and it is easy to do accidentally when a transaction wraps a service call. And a pool sized per instance interacts with replica count: the database sees the pool size multiplied by the number of instances, so an autoscaler that adds replicas under load also multiplies the connection demand on the database at exactly the moment the database is struggling.
Garbage collection pauses that are longer than the collection
Managed runtimes stop application threads to do some of their collection work, and those pauses appear in latency as requests that were doing nothing wrong. The important nuance, and the one that gets candidates credit, is that a pause is not the same as the collection work inside it.
The pause has three parts in the general case: getting all the application threads to a point where they can be stopped safely, doing the work, and resuming. The first part is not proportional to heap size or garbage volume. It is bounded by the slowest thread to reach a stopping point, and a thread executing a long loop the runtime cannot interrupt, or a thread that has been descheduled by the operating system, or a thread blocked in a system call, can extend that phase far beyond the collection itself. A collector reporting a small collection time while the application observes a much longer stall is the signature, and it points at the time-to-safepoint rather than at the collector. Chasing that with heap tuning and collector flags achieves nothing.
The second nuance is that pauses are not the only way collection shows up. Concurrent collectors do most of their work alongside the application, which trades pause time for throughput: the collector threads consume CPU that the application wanted, so under CPU pressure a concurrent collector makes everything slightly slower rather than some things much slower. If the collector cannot keep up with the allocation rate it falls back to a stop-the-world collection, which is why a system can look fine until allocation crosses a threshold and then look very bad.
The third is the interaction with containers, which is where a lot of real incidents live. A runtime that sizes its heap from the machine's memory rather than the container's limit will happily grow past what the container is allowed, and the process is killed rather than collected. A container with a CPU quota can be throttled mid-collection, stretching a pause by the length of the throttling period. And a memory limit that the runtime respects still leaves non-heap usage outside it, so the process can be killed with plenty of heap headroom. What happens when a container hits its CPU and memory limits is worth having read before any cloud-native performance interview, because the throttling mechanism produces latency that looks exactly like contention and is not.
The diagnosis, in order: are the pauses real, from the collector's own logs with timestamps; do they correlate in time with the slow requests, rather than merely coexisting; is the observed stall longer than the reported collection, which redirects you to safepoints and to CPU starvation; and if the collection work itself is genuinely large, is that a heap-sizing problem or an allocation-rate problem, which is what an allocation profile answers. Reaching for a bigger heap first is the classic reflex and it frequently makes pauses longer.
Caching that moves the problem instead of solving it
Caching is the most reached-for optimisation and the one with the most ways to be wrong, and interviewers use it as a maturity probe because the naive answer is so available.
The honest framing is that a cache does not make anything faster. It avoids work by serving a previous result, which means it converts a correctness problem into a latency improvement and the exchange rate is staleness. Every question about caching is really a question about how stale is acceptable and who decides.
The specific ways a cache moves rather than solves the problem are worth being able to enumerate, because that enumeration is the graded content.
It can hide a pathology rather than remove it. A cache in front of a query that is slow because it lacks an index means the query still runs on every miss, and misses happen at exactly the worst time: during a deploy, after a restart, when a new customer's data has never been requested. The system's behaviour under stress is now governed by the uncached path, which nobody is measuring because the cached path looks fine.
It can concentrate load. A restart or a mass eviction sends the full read rate at a dependency provisioned for the miss rate. If a cache normally absorbs the great majority of reads, the uncached load is larger by a factor of one over the miss rate, which for a high hit ratio is a large multiplier. That is the mechanism behind the cold-cache collapse in it collapses a minute after every restart, and it is why cache warming and request coalescing are recovery mechanisms rather than efficiencies.
It can synchronise. Entries created at the same moment with the same time-to-live expire at the same moment, so a periodic thundering herd appears at the expiry interval. Jittering the expiry is a one-line fix that almost nobody applies until it has happened.
It can create a new consistency problem in a place with no owner. A write path that updates the database and then deletes the cache entry has a window in which a concurrent read repopulates the entry with the old value, and the entry then persists until it expires. The result is a stale read that is not transient, which is a much worse failure than a slow read and one whose reports arrive weeks later as "the data is sometimes wrong". You write then read back the old value covers the general shape of this class.
It can add a network hop to the fast path. A remote cache is faster than the database and slower than memory, and if the cached computation is cheap, the cache is a pessimisation. Measuring the cost of the lookup against the cost of the work is a step almost nobody takes.
And it can move the bottleneck to itself. A cache serving the majority of a system's reads is now the most heavily loaded component in it, with its own capacity, its own hot keys, its own single-threaded operations that block everything, and its own failure mode where losing it takes down everything at once because nothing is provisioned for the uncached load.
The senior position to hold is that caching is legitimate, frequently correct, and should be reached for after you know where the time goes rather than instead of finding out, and that every cache introduced needs a stated staleness tolerance, a stated behaviour when it is unavailable, and a stated plan for the cold start.
Tail latency, fan-out and head-of-line blocking
One slow dependency dominates a fan-out
If a request requires responses from several backends before it can complete, its latency is the maximum of theirs, not the average. That single sentence reorganises how you think about a service-oriented architecture, because the maximum of several draws from a distribution sits far out in that distribution's tail.
The reasoning to display, with illustrative arithmetic rather than invented measurements: if a call independently has some small probability of exceeding a threshold, the probability that none of a set of calls exceeds it is that complement raised to the power of the fan-out. Fan-out therefore converts a rare per-call event into a common per-request event, and it does so faster than intuition suggests. The consequence is that the per-call percentile you can tolerate gets tighter as fan-out grows, and a team that adds one more backend call to a page without touching any latency budget has quietly moved the page's tail.
The remedies are worth knowing as a set because interviewers ask for more than one. Reduce the fan-out, which is a design change and usually the largest win. Make the calls concurrent rather than sequential, which changes the sum to a maximum and is often the cheapest large improvement available. Set a deadline for the whole request and propagate it, so a slow backend is cut off rather than allowed to define the page's latency. Return partial results where the product allows it, degrading a section of the page instead of the page. Hedge, meaning issue a second request to a different replica after a short delay and take whichever answers first, which trades a small amount of extra load for a large reduction in tail and only works when the operation is safe to duplicate. And reduce the variance of the backends themselves, because tail latency in a fan-out is a property of variance rather than of the mean.
The hedging answer needs its caveats stated or it reads as naive. Hedging on a non-idempotent operation duplicates side effects, which is why the discussion connects to making a POST endpoint idempotent. Hedging under overload adds load to a system that is already the bottleneck and can be the thing that tips it over, so it needs a budget expressed as a fraction of traffic. And hedging at a small delay percentile is the version that works; hedging immediately is just duplicating every request.
Head-of-line blocking, at four layers
Head-of-line blocking is the situation where one item at the front of a queue prevents everything behind it from being served, even when those items could have been served independently. It appears at enough layers that recognising the shape is more valuable than knowing any one instance.
At the transport layer, an ordered stream delivers bytes in order, so a lost segment stalls delivery of everything after it until it is retransmitted, regardless of whether the later bytes belong to a different logical request. This is the specific limitation that multiplexing several requests over one ordered connection does not remove, because the multiplexing is above the ordering.
At the connection layer, a client that pipelines requests over a connection that processes them in order will have a slow request delay the responses to fast ones behind it.
At the application layer, a work queue serving items in arrival order lets one expensive item block many cheap ones. This is why mixing workloads of very different service times in one queue is a design error, and why separating them into different queues or pools, with different concurrency, is such a reliable improvement. The general statement is that queue discipline is a design decision and defaulting to first-in-first-out is a decision even when nobody made it.
At the partitioned-log layer, an ordered partition consumed in order has exactly this property: one poison or slow message blocks its partition, and because ordering is the point of the partition you cannot simply skip. The mitigations are a dead-letter path with a bound on attempts, and concurrency within a partition where the ordering requirement is per-key rather than per-partition.
There is also a load-balancing variant worth naming, because it is a recurring production surprise: a long-lived multiplexed connection is balanced once, at connection time, so a connection-level load balancer distributes connections evenly while distributing requests very unevenly. The symptom is one instance hot and the rest idle with a load balancer that reports a perfect spread, which is the situation in gRPC behind a load balancer hitting one pod.
Retries, amplification, and the metastable failure
The most interesting failure mode in this entire subject is the one where a system, having been pushed into a bad state by some trigger, stays in that state after the trigger is gone, because its own behaviour now generates enough load to sustain it. The classic presentation is a service that is restarted, looks healthy for a few seconds, and collapses again.
Loading diagram…
The step to point at is the note rather than any single message: the loop closes without the original trigger appearing in it, which is why the service does not heal when the trigger is removed.
Three amplifiers do almost all the work and it is worth separating them because they have different remedies. Accumulated retries mean the offered load at recovery is the steady-state rate plus everything that queued during the outage, arriving at once because every client learned you were healthy at the same moment. A cold cache multiplies the load reaching the datastore by roughly the inverse of the miss rate, and the loop is self-sealing because filling the cache requires successful requests. And work completed after the caller gave up is capacity spent on nothing, so throughput can look healthy while goodput is zero.
The remedy is counter-intuitive enough that interviewers specifically listen for it: you have to refuse work. Serving fewer requests well is strictly better than serving all of them too late, because refused requests stop consuming capacity and accepted ones produce responses somebody reads. Making that real requires a bounded queue so overload becomes rejection rather than unbounded latency, a rejection path cheap enough that shedding is not itself expensive, a deadline on each request checked before each stage, and a ranking so that what you shed is chosen rather than arbitrary. The full treatment is in it collapses a minute after every restart.
Retry policy deserves its own scrutiny because it is where well-intentioned resilience becomes the outage. A retry is a decision to multiply load at the exact moment load is the problem. The safeguards are exponential backoff so the multiplication decays, jitter so retries do not synchronise, a cap on attempts, and a retry budget expressed as a fraction of total traffic so the system cannot spend more than a small share of its capacity on retries however many individual clients decide to try again. The one that is most often missing is the budget, and it is the only one that bounds the aggregate rather than the individual.
The other amplifier is retries at multiple layers. If a client retries three times, and the gateway it calls retries three times, and the service behind that retries three times, a single user action can become a large number of downstream calls, and each layer believes it is being modestly resilient. The rule to state is that retries belong at one layer, chosen deliberately, usually the one closest to the failure that has enough context to know the operation is safe to repeat.
A related recovery pathology is that health checks can hold a system down. If readiness fails under load, the orchestrator removes the instance, its traffic redistributes to the survivors, they fail readiness too, and the fleet removes itself one instance at a time while each is individually recovering. If liveness checks the same condition, instances are killed and restarted, discarding whatever warmth they had. The rules that follow are that liveness should test only whether the process is fundamentally stuck, readiness should reflect whether this instance can take more work rather than whether the system is healthy, and neither should call a downstream dependency, because then one dependency's failure removes every instance simultaneously. A parallel version of this shows up in deploys, where connection draining and probe timing interact badly enough to drop requests on every rollout, as in a rolling update that drops requests every time.
Capacity, autoscaling and load testing
Autoscaling that is configured correctly and still browns out
Autoscaling is presented as the answer to capacity and is a partial answer with several structural limitations, and the ability to name them is a reliable senior signal because the naive position is so common.
The first is that scaling has a latency of its own, and it is the sum of several delays: the metric's collection and aggregation window, the controller's evaluation interval and any stabilisation period, the time to provision an instance, and the time for that instance to become useful, which includes runtime start-up, connection establishment, cache warming and any JIT warm-up. That total is frequently longer than the traffic event that triggered it. A scale-out that completes after the spike has passed has cost money and helped nobody, and the scale-in that follows can arrive just in time for the next spike.
The second is that the scaling signal is usually a lagging or indirect one. CPU utilisation is the default and it is a poor proxy for a service that is slow because it is waiting, since a service blocked on a downstream has low CPU precisely while it is failing. Scaling on a saturation signal such as queue depth, in-flight concurrency or pool-acquisition wait tracks the actual constraint far better, and saying so is worth credit.
The third, and the one that produces the brownout, is that the thing you are scaling is frequently not the constraint. Adding stateless replicas in front of a saturated database adds connections and load to the saturated component. The autoscaler behaves exactly as configured, capacity goes up, and latency gets worse, because the bottleneck moved down a layer and scaling the layer above it makes the problem larger. This is worth stating as a rule: scaling helps only where the bottleneck is the thing being scaled, and every autoscaling policy should come with an explicit statement of what it assumes the constraint is.
The fourth is oscillation. A policy that scales out on latency will, once the new capacity reduces latency, scale in, which raises latency again. Without a stabilisation window and asymmetric thresholds, the system hunts, and each cycle pays the cold-start cost. Scaling in should be slower and more conservative than scaling out, because the cost of being briefly over-provisioned is money and the cost of being briefly under-provisioned is an outage.
The fifth is the ceiling nobody configured. Quotas, subnet address space, licence counts, database connection limits and downstream partner rate limits all impose a maximum that is invisible until the autoscaler reaches it, at which point the failure is sudden. A capacity plan that does not enumerate the limits between current load and the maximum the policy will attempt is incomplete.
Load testing that finds the breaking point rather than confirming a target
There are two kinds of load test and conflating them is the commonest waste of effort in performance engineering.
A validation test asks whether the system meets a stated target at a stated load. It has a pass and a fail, it is appropriate as a gate, and it teaches you almost nothing when it passes. A characterisation test asks where the system breaks and how, by ramping load until the response-time curve bends and continuing past it. It has no pass or fail; its output is a description. The second is the one that informs design and capacity, and it is the one candidates rarely propose.
Test shape
The question it answers
What it will not tell you
Validation against a target
Does it meet the number today
Where the margin is, or what fails first
Ramp to breaking point
Which resource saturates, and at what load
Behaviour under a realistic mixed workload
Soak at steady load
Leaks, unbounded growth, slow degradation
Anything about peak behaviour
Spike
Behaviour when load arrives faster than scaling
Sustained-load bottlenecks
Recovery after overload
Whether it comes back on its own
Whether it degrades gracefully on the way in
The measurement mistakes in load testing are severe enough that they deserve naming, because a test with these defects reports numbers that are worse than none.
Coordinated omission is the important one. If a load generator sends the next request only after the previous one returns, then when the system slows the generator slows with it, so the offered load falls exactly when you needed it constant, and every request that a real user would have sent during the stall is never sent and therefore never recorded as slow. The reported latency distribution is dramatically better than reality, and the error is largest precisely in the tail you were measuring. The fix is to generate at a fixed rate independent of responses and to record latency against intended send time, and most serious tools have an option for this that is not the default.
The second is measuring at the wrong point. A load generator inside the same network, bypassing the proxy, the TLS termination and the mesh, measures a system nobody uses.
The third is unrealistic data and cache behaviour. A test that requests the same handful of entities repeatedly measures a cache, and a test with a small dataset measures a query plan that will not be chosen in production. Both produce optimistic numbers that fail in the direction of confidence.
The fourth is a single-endpoint test on a system that serves a mix. Contention only appears when the workloads that contend are both present, and the interesting failures are usually interactions.
The fifth is testing only the entry point of a system whose dependencies are stubbed, which measures your service against a downstream that never slows down. The most valuable load tests deliberately inject downstream latency, because that is the condition under which pools exhaust and threads starve.
The habit that turns a load test into evidence is to state a prediction first. "At this concurrency I expect the database connection pool to saturate, and I expect to see it as acquisition wait rather than as query time" is a claim the test can refute. Running a ramp with no prediction produces a chart and a shrug.
Capacity as a statement about headroom
The output of capacity work is not a maximum throughput number. It is a statement of how much headroom exists before the response-time curve bends, expressed against the growth you expect and the failure you must survive. A system sized to its current peak has no headroom for a failed availability zone, a retry storm or a marketing event, and a system sized to its theoretical maximum throughput is sized to a point at which its latency is already unacceptable.
The related discipline is that capacity is per-resource rather than global. There is a first constraint, and behind it a second, and knowing the order matters because relieving the first exposes the second and the second may be much harder. A team that removes a database bottleneck and immediately hits a connection-count limit, and then a licence limit, has not planned; it has discovered. Naming the next two constraints after the current one is a strong architectural signal.
Finally, capacity and reliability targets belong together. An error budget makes the trade explicit: it tells you how much unreliability you may spend, which converts arguments about whether to add capacity into arithmetic about whether the budget can afford not to. Setting an SLO and using an error budget is the mechanics, and a one-minute blip and a day degraded is the useful complication that a short total outage and a long partial degradation can consume the same budget while feeling completely different to users and to the business.
What interviewers ask
Performance rounds come in a small number of shapes, and each is graded on different observable behaviour. Recognising the shape tells you which behaviour to display, and candidates who bring scenario behaviour to a concept question, or the reverse, are marked down for reasons they never learn.
Archetype
Typical opening
The signal being graded
The open scenario
It is slow, what do you do
Whether you narrow before you guess, and whether each hypothesis has a test
The constrained scenario
Median flat, tail tenfold
Whether you reason from what the evidence excludes
The instrument question
How do you know where the time goes
Whether you know what your tools cannot see
The mechanism question
Why does a full cache make a restart dangerous
Whether you can derive the multiplier rather than assert it
The tempting fix
Would you add a cache here
Whether you ask what the staleness tolerance is before agreeing
The measurement trap
Your load test showed it was fine
Whether you interrogate the test rather than the system
The escalation
Explain this to the VP who wants a date
Whether the finding survives translation into a decision
The prevention question
How does this take two minutes next time
Whether you propose instrumentation rather than heroics
The open scenario
Someone says a system is slow and gives you nothing else. This is the flagship question of the round and almost everything is decided in your first ninety seconds.
What is graded is whether you ask before you answer. Not endless clarification, which reads as stalling, but the three or four questions that genuinely change the approach: what specifically is slow and for whom, when did it start and was it sudden or gradual, is it all traffic or a subset, and what changed. A candidate who asks "sudden or gradual" has already halved the hypothesis space, because sudden points at a change and gradual points at growth.
The second signal is whether you distinguish the report from the symptom. "The app is slow" from a stakeholder might mean one page, one customer, one time of day, or the login flow specifically. Establishing the observable before diagnosing it is not pedantry; it is the difference between investigating the right system and the wrong one.
The third is whether you narrow by exclusion. Strong candidates spend their early moves ruling layers out, and they say what each observation excludes. Weak candidates spend their early moves listing everything that could cause slowness, which sounds knowledgeable and makes no progress.
The fourth is whether you commit to falsifiable statements. Every hypothesis should arrive with the measurement that would kill it. "If it is pool saturation, the slow traces spend their time before the first query and the wait timer is elevated on the affected instances only" can be wrong. "It is probably GC, let us raise the heap" cannot, and if latency happens to improve you have learned nothing.
The constrained scenario
You are given a specific piece of evidence and asked to reason from it. The evidence is chosen so that it excludes a lot, and the whole point is whether you notice.
The most common is the flat median with a much worse tail, which rules out any uniform slowdown, because anything happening on every request would drag the median. What remains is either a distinct population of slower requests or a queueing effect that catches a fraction of otherwise ordinary requests, and separating those two is the diagnosis. Other variants: throughput flat while latency rises, which is the signature of a saturated resource rather than of expensive work; latency rising with no change in traffic, which points at data growth or a plan change; errors and latency rising together, which usually means timeouts rather than two problems.
What is graded is whether the first sentence out of your mouth is about what the evidence excludes rather than about a cause. Candidates who lead with a cause are, at best, right by luck, and interviewers know the difference.
The instrument question
Asked how you would find out where the time goes, and the signal is whether you know the limits of your tools. Naming a profiler is table stakes. Saying that a CPU profiler shows nothing when the service is blocked, that server-side handler duration cannot see the accept queue in front of it, that a trace with one span and no children is an instrumentation gap rather than an absence of detail, and that percentiles cannot be averaged across instances is the answer that separates.
The follow-up is usually "and what if you do not have that". A strong candidate has a degraded path: repeated thread dumps as a poor man's contention profiler, logging query counts per request to catch N+1, timing added around the suspect boundary and deployed, or a comparison against a known-good instance. The candidate who says they would need better observability first and stops there has answered the question in a way that would not have resolved the incident.
The tempting fix
You are offered an obvious optimisation and watched to see whether you take it. Would you add a cache. Would you add an index. Would you raise the pool size. Should we scale out.
Each of these is sometimes right. What is graded is whether you ask the question that determines it. For a cache, what staleness is acceptable and what happens when it is cold or unavailable. For an index, what the write cost is and whether the query is actually the constraint. For a pool, whether the downstream has headroom, because otherwise you are moving the queue. For scaling out, whether the thing you are scaling is the bottleneck.
An interviewer offering a tempting fix is running a small integrity test, and the failure mode is agreeing quickly to demonstrate decisiveness.
The measurement trap
A version of the scenario where the misleading thing is the measurement rather than the system. The load test passed and production is slow. The dashboard shows a healthy p99 and customers are complaining. CPU is low so the machine is fine. Latency improved after the change.
What is graded is whether you interrogate the instrument. The load test may have coordinated omission, or an unrealistic dataset, or stubbed dependencies that never slow down. The dashboard may be averaging percentiles, or measuring only successful responses, or measuring requests rather than users. Low CPU is consistent with a service that is entirely blocked. And latency improving after a change is not evidence the change caused it unless you controlled for everything else that moves.
The escalation
You have found something and must now tell someone who will make a decision about money or a date. This is graded harder for senior and staff roles than most candidates expect, and it is where technically excellent people lose offers.
The signals are whether you lead with the impact rather than the mechanism, whether you attach a number the business recognises, whether you state your confidence honestly and say what would change it, and whether you present options with costs rather than a single recommendation that leaves no decision to make. A finding delivered as "the N+1 in the order service issues one query per line item" has handed a manager a puzzle. The same finding delivered as "checkout is slow for large baskets, it affects the customers who spend most, the fix is a day and here is how we would verify it" has handed them a decision.
The prevention question
Frequently the last question, and it separates people who resolve incidents from people who reduce them. Asked how you would make this diagnosis fast next time, the good answers are specific instruments rather than aspirations: histograms rather than pre-computed quantiles so the tail can be sliced after the fact, exemplars linking slow buckets to trace identifiers, pool acquisition-wait timers, query counts per request, a saturation signal for every pool and queue, and GC logs with timestamps that can be overlaid on latency. The weak answer is "better monitoring" and the very weak answer is a runbook, because a runbook helps the person who already knows which runbook to open.
How the same question is graded at different levels
At mid level, a correct mechanism is the bar. Knowing what a connection pool does, why an index helps, what a GC pause is. At senior level the mechanisms are assumed and grading moves to method and to trade-offs: you are expected to narrow systematically, to say what each measurement excludes, and to volunteer the cost of your own proposal. At staff level it moves again, to whether you can bound the problem across a system and an organisation: which team owns the constraint, whether the right answer is an architectural change rather than a fix, what you would stop doing to fund it, and whether the problem is worth solving at all. A candidate who brings deeper mechanism to a staff loop is commonly downlevelled with feedback that reads as "strong engineer, did not demonstrate scope".
Tells that you have not done this in production
Interviewers keep an informal list. Naming a cause before asking a question. Proposing a fix and a measurement in the same breath, with the fix first. Treating a fleet-wide percentile as a description of one thing. Reaching for a cache without asking about staleness. Assuming a CPU profile is informative on an idle box. Believing a load test that stubbed the dependencies. Suggesting horizontal scaling for a service-time problem. Never mentioning what happens when the change does not work. Talking about optimisation with no reference to what the user experiences. None is disqualifying alone, and all are noted.
Questions
These are phrased the way interviewers phrase them. The answers are the substance of a strong response, not a script.
Method and measurement
A stakeholder says the application is slow. Where do you start?
By converting a complaint into an observable. Ask what specifically is slow, for whom, since when, and whether it was sudden or gradual, because those four answers eliminate most of the hypothesis space before you touch anything. Sudden points at a change: a deploy, a configuration edit, a dependency's own release, a traffic shift, a data migration. Gradual points at growth: a table that outgrew memory, a collection that outgrew its pagination, a leak, an index whose selectivity decayed. All-traffic points at a shared resource; a subset points at a data-dependent path or a single bad instance.
Then establish the baseline in the units the complainant cares about, and state the population and window. From there the sequence is localise, attribute, hypothesise, test, change one thing, verify against the same measurement. Say the last part explicitly, because verifying against the original baseline rather than against a fresh feeling of improvement is the step that most often gets skipped and is the one that makes the work defensible.
Finish by saying what you would do if the first change did not help, which is to go back to measurement rather than to add a second change. Interviewers are listening for that loop.
Why is the average response time a bad metric?
Because the distribution it summarises is skewed with a hard floor and no ceiling, so the mean sits between the typical experience and the bad one and describes neither. It is pulled by extreme values, so it moves when the tail moves and when the bulk moves, and the number does not tell you which.
Then add the two things that are less commonly said. Averages compose wrongly across a fan-out, because a request needing several calls experiences the maximum rather than the mean, so a per-call average is silent about the page. And percentiles do not average either: computing a p99 per instance or per minute and taking the mean of those produces a number that is not a percentile of anything and specifically conceals the single bad instance. If your metrics are histograms you can merge buckets and estimate once; if they are pre-aggregated quantiles, you cannot recover the truth.
The strongest version of this answer ends by saying what you would look at instead, which is the distribution itself, because a percentile is a summary and the most useful fact about a latency distribution is often whether it is bimodal, which no percentile will tell you.
What does the p99 tell you that the p50 does not, and what does neither tell you?
The median tells you the common path is intact and nothing else. The p99 is where queueing, garbage collection, lock contention and retries show up first, because those affect a fraction of requests rather than all of them, so it is the early-warning statistic.
Neither tells you how many people noticed. A tail over requests is not a tail over users, because heavy users issue more requests and are over-represented, and because a session containing many requests is far more likely to contain a slow one than a single request is. Neither tells you whether the slow requests are one population or two. And neither includes requests that never completed, so an endpoint whose slowest requests start timing out shows an improving latency percentile while getting worse. Saying that last one is worth a lot, because it is the mechanism by which a dashboard can be entirely accurate and entirely misleading.
How do you tell the difference between a system that is doing slow work and a system that is waiting?
By looking at utilisation and saturation of the resources rather than at request timing, and by choosing a profiler that can see off-CPU time. A system doing slow work has a resource pinned and a CPU profile with a hotspot in it. A system that is waiting has idle resources, a CPU profile with nothing in it, and time visible in a wall-clock profile or in a trace as a gap between spans.
The reason the distinction is worth this much attention is that the two respond to opposite interventions. Waiting is reduced by removing the contention, shortening the queue, adding servers at the constrained resource, or reducing the arrival rate. Slow work is reduced by making the work cheaper, and adding capacity does nothing for it. Most wasted optimisation effort is code being made faster while the time was being spent in a queue.
The practical first measurements are the saturation signals: run-queue length rather than CPU percentage, pool acquisition wait rather than downstream latency, queue depth rather than throughput. Each of those distinguishes the two cases directly.
Explain Little's Law and give me a use for it that is not a calculation.
The average number of items in a stable system equals the average arrival rate times the average time each item spends in the system. It holds regardless of the distributions involved, which is what makes it usable when you know very little.
Its best use is as a consistency check. If someone reports a throughput, a latency and a concurrency and the three do not agree, one of them is measured wrong, and finding out which is usually more informative than the original question. The classic resolution is that the latency was measured only over fast successful responses, or that the concurrency limit quoted is not the binding one because a smaller pool sits behind it.
Its second-best use is deriving a ceiling without measuring it. A pool of some size in front of a call of some duration completes at most one over that duration per slot, so the maximum rate is the pool size divided by the call duration. When the downstream slows, that ceiling falls proportionally without anyone changing a setting, which is the mechanism of most pool-exhaustion incidents and a far more convincing explanation than "the pool was too small".
What is the difference between utilisation and saturation, and why does it matter?
Utilisation is the fraction of time a resource is busy; saturation is the extent to which work is queued waiting for it. A resource can be highly utilised with no queue and be perfectly healthy, and it can be moderately utilised with a growing queue because arrivals are bursty.
It matters because latency is caused by saturation, not by utilisation, and because the metrics most teams collect are utilisation metrics. CPU percentage is utilisation; run-queue length is saturation. Downstream latency is not your pool's saturation; acquisition wait is. Throughput is not a queue's saturation; depth and age are. A dashboard made entirely of utilisation metrics will show green while requests wait, which is why "the boxes look fine" is not evidence.
The related point is that utilisation is a poor autoscaling signal for the same reason. A service blocked on a downstream has low CPU exactly while it is failing.
You made three changes and latency halved. What is wrong with that?
You cannot attribute the improvement, so you have learned nothing transferable and you are now carrying three changes of unknown individual value. At least one of them is plausibly neutral and at least one may be harmful under a load shape you have not seen yet, and you will not find out until it matters.
There is a second problem: you cannot be confident the change caused the improvement at all. Latency moves for reasons unrelated to your work, including traffic mix, time of day, a neighbouring service's deploy and a dependency's own change. Attribution needs either a controlled comparison, such as rolling the change to a subset of instances and comparing, or a mechanism that predicted the specific magnitude and direction you observed.
The professional answer is that changing one thing at a time is slower and is the price of knowing anything, and that where you genuinely must ship several changes together, you should at least state which one you expect to dominate and what you would see if you were wrong.
Diagnosing a specific symptom
Your p99 jumped tenfold after a deploy and your p50 is unchanged. Talk me through it.
Start with what the flat median excludes. Anything happening on every request would move it: a new synchronous call, a heavier serialisation, a slower runtime, a smaller instance. So the cause is not on the common path.
That leaves two shapes. Either the tail is a distinct population that was always slower and has become much slower, which means large tenants, a particular endpoint, cache misses, big payloads. Or every request draws from the same distribution and a fraction of them are caught by a stall unrelated to what they asked for, which means queueing or contention: a GC pause, a saturated pool, a lock, a thread pool backlog. Both produce the same p50-versus-p99 picture, so the graph cannot distinguish them.
The next move is a slice, not a hypothesis. Split the histogram by instance, which tells you whether one machine is dragging the fleet; by endpoint, which points at the diff; and by tenant, which is the signature of a plan change that only hurts accounts above a certain size. Then match each surviving hypothesis to its confirming measurement: GC logs correlated in time, pool acquisition wait, thread dumps showing a common monitor, the actual plan for a large tenant's parameters.
Two candidates that fit "after a deploy" specifically and share a signature worth naming: cold cache and JIT warm-up both decay, so if the tail improves over minutes after each instance starts and does so again on the next restart, you are looking at warm-up rather than a defect. A retry storm has the opposite signature, being self-sustaining, and shows as inbound request count at the dependency exceeding what callers believe they sent. The full walkthrough covers the confirmation step for each.
CPU is at twenty per cent and requests are taking seconds. What is happening?
The service is waiting rather than working, and the job is to find out what for. Idle CPU with high latency is one of the most informative combinations there is, because it eliminates the entire class of computational hotspots.
The candidates, and the measurement for each. Waiting on a downstream service or database, visible in a trace as a long child span. Waiting for a pool slot, visible as time before the first child span begins and directly in an acquisition-wait timer. Waiting on a lock, visible in thread dumps as many stacks on one monitor, or in a contention profile. Waiting on storage, visible as IO wait and device queue depth. Waiting to be scheduled despite low average CPU, which happens under a container CPU quota where the process is throttled in bursts and average utilisation looks fine. Waiting on a semaphore, rate limiter or leader lock that nobody modelled as a resource. And the deceptive one: waiting in the accept queue before your handler was entered, which server-side handler duration cannot see at all, so the client's number and yours disagree.
The tool choice follows from the diagnosis rather than the reverse. A CPU profiler here shows an idle machine and tells you nothing; a wall-clock profile and a trace are the right instruments.
A single endpoint got slow but only for a few customers. Where do you look?
At what those customers have in common, and the first hypothesis is data volume, because a query that is fine for a small account and terrible for a large one is the most common shape by a wide margin. The mechanism is usually a plan that switches on estimated row counts, so the same statement uses an index for a small tenant and a scan for a large one, or a nested loop whose inner side is small for most and enormous for a few. Confirm by taking the executed plan with a large customer's parameters rather than a typical one.
The second hypothesis is an unbounded collection. Something that returns all of a customer's items works until a customer has a great many, and this frequently arrives as a gradual slowdown that suddenly becomes a complaint.
The third is per-tenant configuration: a feature enabled for those accounts only, a different pricing or permissions model that adds work, a rate limit they are hitting, or a shard they share with a noisy neighbour.
The fourth is the fan-out multiplier: an N+1 whose N is the customer's data size, which is invisible in query timings and obvious in a per-request query count.
The general lesson to state is that "slow for some customers" is nearly always the same underlying statement as "cost scales with something we did not bound", and finding the unbounded quantity is the diagnosis.
Latency has been creeping up for two months with no deploys. What is your hypothesis?
Growth in something, and the job is to find out what and to establish which growth curve the latency is following. Candidates: data volume, so scans read more and the working set stops fitting in memory; a table whose dead-tuple accumulation is outpacing cleanup, which makes every access read more pages; an index that has grown or whose selectivity has decayed as the data distribution shifted; a collection that was small when the pagination was designed; a queue whose backlog is growing slowly enough that nobody noticed; a leak in memory or in file descriptors or in connections; an accumulating audit or history table joined on a hot path; or simply traffic growth pushing the system towards the bend in the queueing curve, where a modest load increase produces a large latency increase.
The measurement that discriminates is to plot latency against a size metric rather than against time. If latency tracks row count, it is data volume. If it tracks concurrency, it is queueing and the fix is capacity or contention. If it tracks heap occupancy after collection, it is a leak. If it tracks nothing you are measuring, you have found a gap in what you measure, which is itself the finding.
Say also that gradual regressions are the ones nobody has a baseline for, which is an argument for tracking a small number of latency and size metrics over long windows deliberately.
Users say it is slow and every server-side metric looks fine. What now?
Believe the users and interrogate the instrument. Server-side handler duration typically begins once the framework has a fully read request and ends when it hands back a response, which structurally excludes several places time can go.
Enumerate them. DNS resolution and connection establishment, including a TLS handshake that costs a round trip or more on a new connection. Time in the accept queue or in a proxy, mesh sidecar or gateway before your process saw it. Response transfer, which for a large body over a slow or lossy link can dwarf the handler. Time in a downstream that your metric attributes elsewhere. And everything client-side after the bytes arrive, which for a browser can be the majority: parsing, rendering, blocking scripts, layout, and further requests the page issues.
The measurements to propose are real-user timing from the client, comparison of the client's total against the server's handler duration at the same trace identifier, and timing at each intermediate hop so the missing interval can be localised. For the network layer specifically, the shell-level approach in diagnosing an unreachable service from the shell is the same discipline applied to connectivity: establish at which hop the behaviour changes rather than reasoning about the whole path at once.
One instance out of twenty is slow. Does that matter?
Yes, and disproportionately. If a twentieth of traffic is served by a degraded instance, that is five per cent of requests, which is enough to move the p99 substantially while leaving the median untouched, and it is enough to be very visible to users because a user's session hits it repeatedly.
The diagnosis is per-instance metrics, which is the argument for exporting histograms with an instance label rather than pre-aggregated quantiles. Causes worth naming: that machine has a noisy neighbour or degraded hardware; it holds an unfair share of connections because a connection-level load balancer distributed connections evenly but the request rates behind them are not equal; it is the leader for something; it has a leaked resource such as a growing heap or a full connection pool; it missed a configuration change; it is running a different build after a partial deploy.
The important operational point is that the fastest resolution is usually to remove the instance rather than to diagnose it in place, and the important engineering point is that if you remove it without capturing a heap dump, a thread dump and a profile, you have destroyed the evidence and will meet the same problem next week.
Profiling
How do you decide which kind of profile to take?
From the utilisation picture. If the box is busy, take a CPU profile, because the time is being spent executing. If the box is idle and requests are slow, take a wall-clock or off-CPU profile, because a CPU profile will show you an empty machine. If garbage collection is implicated, take an allocation profile, because the CPU profile will show time inside the collector and never tell you which code created the garbage. If you suspect contention, take a lock profile or, failing that, several thread dumps a few seconds apart during the problem.
Then say the thing that demonstrates experience: state your expectation before you look. A profile examined without a prior hypothesis always contains something that looks optimisable, and the human default is to optimise it whether or not it was on the critical path. Deciding in advance what you expect to see makes the profile evidence rather than inspiration.
A CPU profile of a slow service shows nothing standing out. What does that tell you?
Most likely that the time is not being spent on CPU, so the instrument is blind to it, and the next step is a wall-clock profile or a trace rather than a deeper reading of the same profile.
But consider the alternatives before moving on. The profile may be flat because the work genuinely is spread across many call paths, which is a real finding and means there is no single fix, only a broad efficiency problem or an architectural one. The profiler may be suffering safepoint bias in a managed runtime, attributing samples to methods that happen to contain the points at which the runtime can pause rather than to where time is spent. The sampling window may not have covered the slow period, which is the commonest practical error, since profiling for a minute during which nothing was slow tells you about the fast path. Or the process you profiled may not be the one that is slow, which happens more often than anyone admits in a system with sidecars and proxies.
How would you find lock contention?
The direct route is a contention profile, which records where threads block acquiring monitors and how long they wait. The crude route, which is available everywhere and is often sufficient, is to take three or four thread dumps a few seconds apart while the problem is occurring and look for the same monitor at the top of many stacks, with the same owner. If a dozen threads are parked on one lock across successive dumps, you have your answer without any tooling at all.
The reason contention needs its own hunt is that it hides from everything else. The CPU profile shows an idle machine, since blocked threads are not running. Request metrics show elevated latency with no attribution. The database looks fine because the contention is in your process. And utilisation dashboards are green, because a blocked thread uses no resource.
The fixes are worth ranking rather than listing: hold the lock for less time, especially never across IO or a remote call; reduce the number of contenders by partitioning the protected state; replace the lock with an approach that does not need one, such as immutable data, per-thread state or an atomic operation; or accept the serialisation and bound the queue in front of it so it degrades predictably. The one to name as an error is making the critical section larger to "be safe", which is a common instinct.
What can a distributed trace not tell you?
Anything inside a span with no children, which is the practical limit that most disappoints. A request that shows as one four-second span means nobody instrumented what happened inside it, and reading that as "the time is in the application" is only true in the sense that you have not looked.
Also: it cannot tell you about work that happens outside the request path, such as a background job consuming the CPU your request needed. It rarely captures queueing in front of the traced process, since the span starts when the process starts handling. Sampling means the specific slow request you want may not have been kept, unless you use tail-based sampling that decides after seeing the duration. Clock differences between machines make cross-service timings approximate, so a child span appearing to start before its parent is usually clock skew rather than a mystery. And a trace shows what happened, not what was contended, so two spans that are slow because they were fighting over the same resource look identical to two spans that were independently slow.
The complement to name is exemplars, which link a slow histogram bucket to a specific trace identifier and turn an aggregate into an example you can read. That connection is the cheapest high-value observability improvement most services can make.
Databases
A database is saturated. What do you look at first?
What sessions are waiting on, before any SQL. The categories are roughly the same across engines: CPU inside execution, storage reads, commit durability, locks held by other sessions, internal latches, and waiting on the client. Each points at a different remedy and only one of them makes the query text the primary suspect.
Locks mean the statements are not slow, they are blocked, and the fix is transaction scope and access ordering. Storage reads mean the working set no longer fits in memory, which is a data-volume, caching or sizing problem before it is a query problem. Commit waits mean the write path, and the answer may be batching rather than anything about reads. Only CPU inside execution sends you to the plan.
Then rank statements by total time consumed over a window rather than by worst single execution, since the cheap statement running constantly usually outranks the slow report and is more often the cause of the saturation. Only then look at plans. Skipping the wait analysis is why so much query tuning produces no measurable improvement: the queries were tuned and the sessions were waiting on something else.
Walk me through reading an execution plan.
Read the tree from the innermost nodes outwards, because that is execution order, and prefer an executed plan to an estimated one, because the estimate is what you are trying to check.
The primary finding is the gap between estimated and actual rows at each node. A large discrepancy means the planner chose on a false premise, and every decision above that node inherits the error. A nested loop selected because the inner side was estimated small is disastrous when it is not, and the operator is not the fault; the estimate is.
Three reading habits matter. Multiply a node's per-loop figures by its loop count, because a cheap operation repeated many times is where the time usually is and it does not look expensive per loop. Subtract children's times from a node's cumulative time to get its own contribution. And remember the plan says what was done, not why, so a suspicious choice sends you to statistics, to whether the predicate can use an index at all, or to parameter sensitivity.
Then name the usual causes of a bad estimate: stale statistics after bulk changes, correlated predicates whose selectivities the planner multiplies as if independent, an expression or function wrapping a column so no statistics apply, a type mismatch forcing a cast, and a skewed parameter distribution where one plan suits common values and another suits rare ones. The mechanics in full go further into reading the operators.
How do you choose which query to tune?
By total time consumed across a measured window, which is executions multiplied by mean duration, not by the slowest observed execution. The statement running thousands of times a minute at a few milliseconds usually outranks the ten-second report, and it is also more likely to be causing the saturation that makes everything else slow.
Then apply a second filter that candidates usually miss: the query is only worth tuning if the resource it consumes is the one that is scarce. Halving the CPU cost of a statement on an instance bottlenecked by storage reads changes nothing measurable. Rank by consumption of the constrained resource, not by total time in the abstract.
Two measurement notes worth volunteering. Statement statistics normalise parameters, so a shape that is fast for most values and pathological for a few is reported as an average that describes neither, and finding the bad values needs per-execution logging or sampling. And the counters are cumulative, so comparing two periods means snapshotting at each end and subtracting. The full argument covers how to establish the window.
The same statement is fast from a console and slow from the application. Why?
First establish whether the extra time is inside the database at all, using database-side statement statistics and automatic plan logging rather than the client's timer, because the answer is frequently that the database is fast in both cases and the time is elsewhere: pool acquisition, transaction management, result-set materialisation, driver-side fetch size causing many round trips, or an object mapper turning one logical query into many.
If the time is inside the database, the usual mechanism is that the application sends bind parameters while your console run supplied literals, so the planner may be working without the values. Depending on the engine and the settings, it may produce a plan suited to a typical parameter that is very wrong for this one, or reuse a plan chosen for an earlier execution with different parameters. That is parameter sensitivity, and it produces exactly this symptom.
Other candidates: a different session configuration such as search path, isolation level, or a work-memory setting; a different user with different row-level security; caching, because your console run is the second execution and the data is now in memory; and a transaction context where the application's statement is competing with locks its own earlier statements took. The walkthrough separates the in-database and out-of-database cases properly.
When is adding an index the wrong answer?
When the query is not the constraint, when the index will not be used, and when the write cost outweighs the read benefit.
Not the constraint: if sessions are waiting on locks or commit durability, a faster read path does not help. If the endpoint is slow because it issues a thousand fast queries, indexing them makes a thousand fast queries slightly faster.
Will not be used: if the predicate wraps the column in a function or a cast, if the leading columns of a composite index do not match the query's usable predicates, or if the planner correctly judges that the fraction of rows involved makes a scan cheaper. Why an index is not being used enumerates the cases.
Write cost: every index is maintained on insert, update and delete, consumes memory that the buffer cache wanted, and adds to the volume that must be written, backed up and kept warm. A table with many indexes has a materially slower write path, and the effect compounds with update-heavy workloads. Fourteen indexes and slower writes is that trade in full, and column order in a composite index is the reason three well-chosen indexes often replace eight badly chosen ones.
The answer that scores is proposing the index with its cost attached and stating how you would verify it helped, rather than offering it as a free improvement.
Every query is fast and the endpoint takes four seconds. Explain.
Almost certainly the number of queries rather than the cost of any one. This is N+1: a collection is fetched, and then something is fetched per member, so the count scales with data rather than with code and the cost is dominated by per-round-trip overhead that no query-level monitoring will flag.
It is invisible in the places people look. The slow-query log has nothing in it. The statement statistics show a cheap shape with a high execution count, which looks like healthy traffic. The application trace shows a long span with many small children, which is only obvious if someone instrumented the queries.
The detection method to propose is a per-request query counter, logged or alerted above a threshold, because it catches the entire class permanently rather than this instance. The fixes are a join, a second batched query keyed on the collected identifiers, an explicit batch-fetch facility in the data layer, or bounding the collection through pagination. Mention that the batched-second-query approach is often better than a join, because a join to a one-to-many relation multiplies rows and can transfer far more data than it saves in round trips.
Why would you refuse to raise the connection pool size?
Because the pool is a rate limiter on your access to a shared resource, and raising it does not add capacity to that resource. If the database is the constraint, a larger pool moves the queue out of your application, where it is visible, bounded and cheap, and into the database, where it is less visible, competes with every other client, and consumes memory and scheduling on your most expensive machine. Worse, it multiplies across replicas: the database sees pool size times instance count, and an autoscaler adding instances under load also multiplies connection demand at the worst moment.
Then say when raising it is correct, because a flat refusal is not an answer. It is correct when the hold time is legitimately long, the resource has demonstrable headroom, and the wait is pure queueing at the pool rather than downstream saturation. The evidence for that is resource-side: the database is not saturated, and its own wait analysis shows sessions idle rather than contending.
The better first move is usually to reduce hold time rather than to raise the size, since capacity is size divided by hold time and hold time is often inflated by something structural such as a transaction held open across a remote call, or a connection acquired earlier in the request than it is needed.
Caching, GC and runtime behaviour
Should we put a cache in front of this?
Answer with the questions that determine it. What staleness is acceptable, and who has the authority to say. What happens on a miss, since the uncached path is what the system falls back to at the worst possible moments. What happens when the cache is unavailable, since the dependency behind it is now provisioned for the miss rate and not the full rate. How entries are invalidated, and whether the invalidation has a race with concurrent reads that can leave a stale entry persisting until expiry. And whether the cached computation is expensive enough to justify a network hop, because a remote cache in front of cheap work is a pessimisation.
Then name the general trade honestly: a cache does not make anything faster, it avoids work by reusing a previous result, and the exchange rate is staleness. Where the underlying operation is slow because of a defect, a cache hides the defect and leaves the system's stress behaviour governed by an unmeasured path.
Two specific mechanisms worth volunteering. Entries created together expire together, producing a periodic herd, so jitter the expiry. And duplicate concurrent misses for the same key multiply load exactly when you can least afford it, so coalescing in-flight fetches for the same key is a resilience mechanism rather than a nicety.
Why does a service with a high cache hit ratio collapse when it restarts?
Because the hit ratio is the thing protecting the dependency, and a restart removes it instantly. The load reaching the datastore multiplies by roughly the inverse of the miss rate, so a cache absorbing the great majority of reads means the uncached load is many times larger, arriving at a dependency provisioned for the served fraction.
Then describe the loop, because the multiplier alone is not the whole answer. The datastore slows under the multiplied load, so requests time out before they can populate the cache, so the next request for the same key is also a miss, so the cache never fills. It is self-sealing: filling it requires successful requests and there are none. Concurrent duplicate misses for the same key make it worse.
The remedies are warming before accepting traffic, which deliberately makes restarts slower; coalescing identical in-flight fetches; admitting traffic in stages with a concurrency limit rather than going from zero to full; and shedding load so that the requests you do serve can complete and populate the cache. The scenario in full works through the interaction with accumulated retries.
Your GC logs show short collections and the application sees long stalls. What is going on?
The pause is not the collection. Getting every application thread to a safe stopping point precedes the work and is not proportional to heap size or garbage volume; it is bounded by the slowest thread to arrive. A thread in a long loop the runtime cannot interrupt, a thread descheduled by the operating system, or a thread blocked in a system call can all extend that phase far beyond the collection itself.
The other explanations for the same symptom. The process was throttled by a CPU quota mid-pause, so wall-clock time greatly exceeds the work done, which is a container configuration issue rather than a collector one. The machine is swapping or under memory pressure, so touching pages is slow. Or the stall is not GC at all and merely coincides, which is why correlation of timestamps is a required step rather than an optional one.
The conclusion to state is that heap tuning is the wrong response here, and may make it worse by making collections larger. The right responses are finding the thread that will not stop, checking quota throttling, and only then considering collector configuration.
How do you reduce garbage collection impact without touching the collector settings?
By allocating less, which is what an allocation profile tells you how to do. The common sources are worth naming: intermediate collections created and discarded per request, string concatenation and formatting on hot paths, boxing of primitives in generic containers, defensive copies of data that is never mutated, and deserialising an entire payload when a small part of it is needed. Reducing allocation reduces collection frequency directly, and frequency is what determines how many requests land inside a pause.
The second lever is object lifetime. Objects that die young are cheap in a generational collector; objects that survive long enough to be promoted are expensive, and a cache or buffer sized just large enough to keep objects alive past promotion is a common and non-obvious cost. So sizing caches deliberately is a GC intervention as well as a memory one.
The third is not creating the work at all. A request that fetches ten times the data it needs allocates ten times the garbage, so pagination, projection of only required columns, and streaming rather than materialising are all garbage-collection improvements even though nobody files them that way.
What does a container CPU limit do to latency?
It throttles the process when it exceeds its quota within each scheduling period, which means the process is stopped entirely for the remainder of that period. The effect on latency is bursty and does not appear as high average CPU utilisation, so a dashboard showing modest CPU alongside terrible latency is exactly what throttling looks like.
The consequences worth naming. A runtime that sizes its thread pools from the machine's core count rather than the container's quota creates far more threads than it can run, which increases context switching and makes throttling more likely. A garbage collection that begins just before the quota is exhausted is suspended mid-pause. And a service that is mostly idle but bursty is the worst fit for a quota, because its bursts are exactly what gets cut.
The measurement to name is the throttling counter itself, which container runtimes expose directly, and the point to make is that it is a first-class latency metric that most teams never look at. What happens when a container hits its CPU and memory limits covers the memory side too, where the failure is a kill rather than a slowdown.
Distributed and tail behaviour
Why does one slow dependency dominate a page that calls six services?
Because the page's latency is the maximum of the six, not the average, and the maximum of several draws sits far out in the tail of the distribution. Fan-out converts a rare per-call event into a common per-request one, and it does so faster than intuition suggests: as illustrative arithmetic from assumed independence, a one-in-a-hundred per-call chance of exceeding a threshold gives roughly a one-in-twenty chance across six calls and around a quarter across thirty.
Real calls are worse than independent, because they frequently share the same slow instance, the same saturated pool, or the same degraded availability zone, so the correlated case is more common than the arithmetic suggests.
The remedies to offer as a set: reduce fan-out, which is the largest and most structural win; issue the calls concurrently so the cost is a maximum rather than a sum; propagate a deadline so a slow dependency is cut off rather than allowed to set the page's latency; degrade partially, returning the page without one section; hedge to a second replica after a short delay where the operation is safe to duplicate; and reduce the variance of the dependencies themselves, since tail behaviour is driven by variance rather than by the mean. One screen, three services and a 200ms budget works through the composition problem when a filter or sort spans owners, which is the case where none of these help and the design has to change.
What is head-of-line blocking and where have you seen it?
One item at the front of a queue preventing everything behind it from being served, even though the later items could have been served independently.
Give it at more than one layer, because the value is in recognising the shape. At the transport layer, an ordered byte stream stalls on a lost segment, so multiplexing several logical streams over one ordered connection does not remove the coupling. At the application layer, a single work queue mixing cheap and expensive items lets one expensive item block many cheap ones, which is why separating workloads with very different service times into different queues is such a reliable improvement. In a partitioned log, one slow or poison message blocks its partition, and you cannot simply skip it because ordering is the point, so the mitigations are bounded attempts with a dead-letter path and per-key rather than per-partition concurrency.
Then add the load-balancing variant, because it surprises people in production: a long-lived multiplexed connection is balanced once at connection time, so a connection-level balancer spreads connections evenly and requests unevenly, producing one hot instance and a load balancer reporting a perfect distribution, which is gRPC behind a load balancer hitting one pod.
A service collapses within a minute of every restart even though the original trigger is gone. Why?
Because it is in a state that sustains itself: its own behaviour now generates enough load to keep it there, so removing the trigger removes nothing. The load killing it is load the outage created.
Three amplifiers do the work. Accumulated retries mean the offered load at recovery is steady-state plus the whole outage window's demand, arriving at once because every client learned you were healthy simultaneously. The cold cache multiplies read load at the datastore by the inverse of the miss rate and the loop is self-sealing. And responses arriving after the caller's deadline mean full utilisation with zero goodput, since every unit of capacity produces a result nobody reads and provokes another retry.
The way out is to refuse work, which is the counter-intuitive part interviewers listen for. Serving fewer requests well beats serving all of them too late. That requires a bounded queue so overload becomes rejection rather than unbounded delay, a rejection path cheap enough to be worth taking, deadlines checked before each stage so nothing is half-completed for an absent caller, and a deliberate ranking so that what you shed is chosen. Then come back in stages with a concurrency limit rather than a rate limit, watching goodput rather than CPU. The full treatment includes the queue-discipline subtlety that first-in-first-out maximises wasted work under sustained overload.
Your database failed over in thirty seconds and the service returned errors for ten minutes. Where did the time go?
Not in the database. Attribute it in stages rather than tuning anything. Sockets to the old primary may hang rather than fail, because a machine that stopped answering produces a timeout rather than a reset, and the timeout is governed by settings nobody chose deliberately. A cached name resolution keeps pointing at the old address for as long as the client's cache holds it, which for some runtimes is effectively forever. The promoted node may accept connections before it accepts writes, so early attempts fail in a way the client may misclassify as permanent. And then the entire fleet reconnects at once, so the new primary is handed a connection storm on top of a cold cache.
The reason this is a performance question rather than an availability one is that each stage is a latency to be measured and reduced, and the fixes are different for each: connection-level timeouts and health checking for the hanging sockets, resolution caching policy for the stale address, error classification so a transient refusal is retried and a permanent one is not, and staggered reconnection with jitter for the storm. The database recovered, the service did not works through the attribution.
Consumer lag on a topic grows every morning and never clears. How do you approach it?
Start from the definition: lag is the accumulated difference between arrival rate and service rate, so establish which of the two moved. If arrivals grew, this is capacity. If service slowed, this is a consumer problem and you diagnose it like any other slow process. If neither moved and lag still grows, the consumer is stalling rather than being slow, which is a different investigation.
Then look per partition rather than per group, because an even-looking group lag can hide one partition doing nothing, which points at a stuck consumer, a poison message, or a key distribution so skewed that one partition carries most of the traffic. Rule out a rebalance loop, where consumers repeatedly leave and rejoin so nothing makes progress, typically because processing exceeds the interval the group coordinator expects between polls.
And know the ceiling: adding consumers beyond the partition count does nothing, because a partition is consumed by at most one member of a group. That single fact resolves a large fraction of real lag investigations, and it means capacity work here starts with partitioning rather than with instance count. Consumer lag that never catches up covers the rest.
Capacity, scaling and testing
We doubled the instance count and latency got worse. How?
Because you scaled something that was not the bottleneck, and adding instances added load to whatever was. The classic mechanism is connections: each instance carries its own pool, so the database sees pool size times instance count, and doubling instances doubles connection demand against a component that was already the constraint.
Other mechanisms worth naming. More consumers of a shared cache means more concurrent misses for the same keys. More instances means more of everything periodic, so scheduled jobs, cache refreshes and metric scrapes all double. In a partitioned system, adding consumers beyond the partition count adds rebalancing without adding throughput. And with a cold-start cost, adding instances during an incident means a fleet where a growing share of instances are warming up and serving slowly.
The general statement is the one to lead with: horizontal scaling relieves wait time at the layer being scaled and does nothing for service time or for a constraint below it, so every scaling decision should come with an explicit claim about what the bottleneck is and how you know.
Your autoscaler is configured correctly and the service still browns out under a spike. Explain.
Because scaling has a latency of its own that is frequently longer than the event. It is the sum of the metric window, the controller's evaluation interval and stabilisation period, provisioning time, and the time for a new instance to become genuinely useful, which includes runtime start, connection establishment, cache warming and JIT warm-up. A scale-out completing after the spike has cost money and helped nobody.
Then the signal problem. CPU is the default metric and is a poor proxy for a service that is slow because it is waiting, since a blocked service has low CPU exactly while it is failing. Scaling on a saturation signal such as in-flight concurrency, queue depth or pool acquisition wait tracks the constraint far better.
Then the ceiling nobody configured: quotas, address space, connection limits, licences and downstream rate limits all cap the policy in ways that are invisible until reached. And the oscillation problem, where scaling in after latency improves raises latency again, so scale-in should be slower and more conservative than scale-out.
The design conclusion is that autoscaling handles growth and diurnal shape well and handles spikes badly, so a spike needs headroom, shedding or a queue, not a policy.
How would you load test this properly?
Decide first which question you are asking, because the two kinds of test are different exercises. A validation test asks whether the system meets a target at a stated load and produces a pass or a fail. A characterisation test ramps until the response-time curve bends and past it, and produces a description of where the system breaks and how. Design and capacity work needs the second, and it is the one candidates rarely propose.
Then name the measurement hazards, because a test with these defects is worse than no test. Coordinated omission, where a generator that waits for each response before sending the next slows down with the system, so the offered load falls exactly when it should not and the requests a real user would have sent during the stall are never recorded; generate at a fixed rate and record against intended send time. Measuring inside the network, bypassing the proxy and TLS termination that real traffic traverses. Unrealistic data, where a small dataset produces a plan and a cache behaviour that production will not reproduce. A single-endpoint test on a system that serves a mix, which misses every interaction. And stubbed dependencies that never slow down, which specifically prevents the test from ever reproducing pool exhaustion or thread starvation, the two failures you most wanted to see.
Finish with the discipline that makes it evidence: state the prediction before you run it. "At this concurrency I expect the pool to saturate and to see it as acquisition wait rather than as query time" is refutable. A ramp with no prediction produces a chart.
What does it mean to have enough capacity?
Not a maximum throughput number. It is a statement about how much headroom exists before the response-time curve bends, measured against expected growth and against the failures you have committed to surviving. A system sized to today's peak has no room for a lost availability zone, a retry storm or a marketing event, and a system sized to its theoretical maximum throughput is sized to a point where latency is already unacceptable.
Add that capacity is per-resource and ordered. There is a first constraint and behind it a second, and relieving the first exposes the second, which may be much harder or much more expensive. Being able to name the next two constraints after the current one is what distinguishes a plan from a discovery.
And tie it to the reliability target, because that is what converts an argument into arithmetic. An error budget states how much unreliability may be spent, which makes "do we need more capacity" a question with an answer rather than a matter of nerve. Setting an SLO and using an error budget is the mechanism, and the useful complication is that a short total outage and a long partial degradation can consume the same budget while producing very different reactions, which a one-minute blip and a day degraded works through.
How would you make sure this diagnosis takes two minutes next time?
By naming instruments rather than intentions. Export latency as histograms with labels for instance, endpoint and tenant, so the tail can be sliced after the fact rather than only in the dimensions someone anticipated. Add exemplars so a slow bucket links to a trace you can read. Emit a saturation signal for every pool and queue, especially acquisition wait, since that is the metric that distinguishes queueing from downstream slowness in one number. Count queries and outbound calls per request, which catches the whole N+1 class permanently. Log GC pauses with timestamps that can be overlaid on latency. Record client-side timing so the server-versus-client gap is visible rather than inferred.
Then the process half: capture evidence before mitigating, because removing the bad instance without a thread dump and a profile destroys the only artefact that would have prevented a repeat. And keep the baseline, because a gradual regression is only detectable against a number somebody wrote down months ago.
The answer to avoid is "better observability", which is a sentiment, and the answer to avoid more strongly is a runbook, because a runbook helps only the person who already knows which one to open.
Reporting and judgement
You have found the cause. How do you report it?
Lead with the impact in the business's terms, not with the mechanism. What is affected, who experiences it, how often, and what it costs, whether that cost is revenue, capacity spend, or a reliability commitment. The mechanism goes second, compressed to the level the audience needs, with the detail available rather than delivered.
Then state your confidence honestly and say what would change it. "I am confident this accounts for most of the tail; I have not accounted for the remainder and here is how I would" is more credible than certainty, and it protects you when the fix improves things by less than promised.
Then give options with costs, because a single recommendation removes the decision and the decision is not yours. A cheap mitigation available today, a proper fix that takes longer, and the do-nothing option with its consequence stated. Say what you would choose and why, and let them choose otherwise.
Finally, say how the fix will be verified and when you will report back, in the same units as the original measurement. A finding that ends without a verification plan is a story rather than a piece of work.
The business wants a date and you do not have one. What do you say?
Separate what you know from what you do not, and give them something to plan with in the meantime. You can usually commit to a date for the next piece of information even when you cannot commit to a date for the fix, and that is a real commitment rather than a deflection.
Then offer the mitigation that is available now, with its cost, so there is a decision on the table rather than a wait. Reducing the fan-out on one page, disabling a feature, shedding a class of traffic, or accepting degraded behaviour for a subset are all things a business can weigh, and offering them demonstrates that you understand the pressure rather than resenting it.
What loses credibility is a date invented to end the conversation. It ends this conversation and destroys the next three.
When would you decide not to fix a performance problem?
When the cost of the fix exceeds its value, and you should be able to argue that rather than merely feel it. A slow internal report used monthly by two people is not worth a week. An endpoint whose latency is dominated by a dependency you do not own and cannot influence may be better handled by changing the product's expectations than by engineering. A tail affecting a small number of requests that are not on any path anyone cares about is a legitimate thing to leave alone.
Also when the fix has risk out of proportion to the benefit. Restructuring a payment path to save a modest amount of latency is a different proposition from doing the same to a recommendations widget, and treating them the same is a failure of judgement rather than of skill.
And when the problem is a symptom of something being retired. Optimising a system scheduled for replacement is a real cost with a short payback, and saying so protects the credibility you need for the occasions when you insist the work must happen.
The signal being graded here is whether you can prioritise against value rather than against how much the problem annoys you, which is the actual constraint on every performance backlog.
Somebody proposes rewriting the service in a faster language. What do you say?
Ask where the time currently goes, because the proposal implicitly claims it is in the language runtime and that claim is testable. If the service spends most of its time waiting on a database, a downstream or a lock, the language is close to irrelevant and the rewrite will produce a service that waits at the same speed in a new codebase.
Where the language genuinely matters is where the work is computational, where allocation and collection behaviour dominate, where memory footprint is a binding constraint on density, or where start-up time matters because the deployment model creates processes frequently. Those are real cases and it is worth naming them so the answer is not reflexively dismissive.
Then state the cost honestly. A rewrite discards accumulated correctness that nobody has written down, and it typically arrives with a period of worse reliability that outweighs the latency benefit for some months. The credible position is to profile first, quantify the fraction of time attributable to the runtime, and let that number decide, with the observation that if the profile shows the win is small then the conversation was about something other than performance.
What is the most surprising performance problem you have diagnosed?
Have a real one, told in the diagnostic order rather than the narrative order: what the symptom was, what you believed at first, what measurement changed your mind, what the cause turned out to be, and how you verified the fix. The structure is being graded as much as the content, and the moment where you were wrong is the most valuable part of the story rather than something to minimise.
Pick one where the cause was not the obvious candidate, because that is where the method shows. And include what you changed afterwards so the class of problem became detectable, since that converts an anecdote into evidence of judgement. If the honest answer is that you found it by luck, say so and say what instrument would have found it deliberately, which is a better answer than a tidier story that is not true.
How to prepare
Learn the diagnostic path well enough to walk it without prompting, because a large share of questions here are a probe into one point on it, and because under pressure the shape survives when a list does not. Practise saying what an observation excludes before saying what it suggests; that single habit changes how the whole round sounds.
Get properly comfortable with the measurement arguments, since they carry more weight than any individual mechanism. Why the average misleads, why percentiles do not average, why a percentile over requests is not a percentile over users, why a latency metric that omits timeouts improves as the system degrades, and why a load generator that waits for responses understates the tail. Those five points come up constantly and most candidates can state one of them.
Pick two or three mechanisms and go deep enough to derive rather than recite: the queueing argument for why latency bends, the pool-capacity derivation from hold time, the cache-multiplier arithmetic on a cold start, and the fan-out probability. Deriving a result in front of an interviewer is worth far more than asserting a memorised one, and it is the thing that cannot be faked.
Then bring one real investigation you can tell in diagnostic order, including the point at which you were wrong and the measurement that corrected you. Interviewers have heard a great many tidy stories about caches and indexes. They have heard very few honest accounts of a week spent narrowing, and that is the one that gets remembered when the panel meets.