Top 11 Latency Interview Questions
Browse 11 real-world technical and behavioral interview questions about Latency. Review scenarios, edge cases, and architectural best practices.
Your GC log shows a stop-the-world pause far longer than the collection work inside it. Where did the rest of the time go?
Most likely into reaching the safepoint. Every application thread has to arrive at a poll point before the operation can begin, and one late thread stalls all of them, so the pause is time-to-safepoint plus the actual work. Safepoint logging separates the two.
How do you decide whether a model should be served in batch, online or streaming?
Freshness decides it. If a prediction is still valid hours later, precompute it in a batch job and serve a lookup; if it depends on what the user did seconds ago, score it online; if it depends on an evolving event sequence, score it from a stream. Most mature systems end up hybrid.
Why is a recommender built as candidate generation followed by ranking rather than as one model?
Because you cannot score a million-item catalogue per request inside a few tens of milliseconds. Retrieval reduces millions to hundreds with a cheap model judged on recall, then a far more expensive ranker orders those hundreds and is judged on the ordering it produces at the top of the list.
Users say the LLM feature in your product feels slow. How do you work out what to fix?
Split the measurement into time-to-first-token and inter-token latency, because they come from different phases. The first is driven by prompt length and queueing and is fixed with shorter prompts, prefix caching and streaming; the second is the decode loop, fixed only by generating fewer tokens.
Models take a million tokens of context now. Does that remove the need for retrieval?
No. Cost and latency scale with what you send, accuracy over a long context is uneven rather than flat, and a corpus has no upper bound while a window does. Long context did change the job - it absorbs whole documents and forgives loose recall - but it replaces chunking, not retrieval.
You have 50 ms for a model call in a request path. How do you make that budget?
Break the budget into its parts first, because feature fetching usually costs more than the model does. Then parallelise and cache the fetches, shrink the model with quantisation or distillation, set a timeout shorter than the budget, and define a degraded path that answers without the model.
Your p99 latency jumped tenfold after a deploy while p50 is unchanged. Diagnose it out loud.
A flat p50 rules out a uniform slowdown, so the tail belongs either to a subset of requests or to a queueing effect. Partition the latency histogram by instance, endpoint and tenant before naming a cause, then confirm with traces, GC logs and pool-wait metrics.
How do you optimise Java Virtual Machine (JVM) garbage collection to eliminate tail latency spikes in a game server?
Analyse JVM tuning strategies to prevent Stop-The-World (STW) pauses and ensure predictable microsecond response times for game server logic. It also connects garbage collection to the point an interviewer is testing.
What does batching buy you when serving an LLM, and what does it do to your latency budget?
Batching amortises the memory traffic of reading model weights across many requests, so throughput rises far faster than per-step time does. Continuous batching lets requests join and leave each iteration, but the gain is paid for in tail latency, which is why interactive and bulk traffic want separate pools.
When is a neural network the wrong choice?
On small and medium tabular data, where gradient-boosted trees usually match it for a fraction of the effort; where a decision must be explained exactly rather than approximately; and where a tight CPU latency or memory budget makes it unservable. The simpler model then wins on every axis that matters.
Where does a cross-encoder reranker earn the latency it costs you?
It earns it when your gold set shows recall at 50 is much better than recall at 5, because that gap is what reranking converts into answer quality. The cost is a model pass per candidate with nothing precomputable, so the latency budget divided by measured per-batch cost sets the candidate count.