Loading...
Loading...
Browse 11 real-world technical and behavioral interview questions about Latency. Review scenarios, edge cases, and architectural best practices.
Most likely into reaching the safepoint. Every application thread has to arrive at a poll point before the operation can begin, and one late thread stalls all of them, so the pause is time-to-safepoint plus the actual work. Safepoint logging separates the two.
Freshness decides it. If a prediction is still valid hours later, precompute it in a batch job and serve a lookup; if it depends on what the user did seconds ago, score it online; if it depends on an evolving event sequence, score it from a stream. Most mature systems end up hybrid.
Because you cannot score a million-item catalogue per request inside a few tens of milliseconds. Retrieval reduces millions to hundreds with a cheap model judged on recall, then a far more expensive ranker orders those hundreds and is judged on the ordering it produces at the top of the list.
Split the measurement into time-to-first-token and inter-token latency, because they come from different phases. The first is driven by prompt length and queueing and is fixed with shorter prompts, prefix caching and streaming; the second is the decode loop, fixed only by generating fewer tokens.
No. Cost and latency scale with what you send, accuracy over a long context is uneven rather than flat, and a corpus has no upper bound while a window does. Long context did change the job - it absorbs whole documents and forgives loose recall - but it replaces chunking, not retrieval.
Break the budget into its parts first, because feature fetching usually costs more than the model does. Then parallelise and cache the fetches, shrink the model with quantisation or distillation, set a timeout shorter than the budget, and define a degraded path that answers without the model.
A flat p50 rules out a uniform slowdown, so the tail belongs either to a subset of requests or to a queueing effect. Partition the latency histogram by instance, endpoint and tenant before naming a cause, then confirm with traces, GC logs and pool-wait metrics.
Analyse JVM tuning strategies to prevent Stop-The-World (STW) pauses and ensure predictable microsecond response times for game server logic. It also connects garbage collection to the point an interviewer is testing.
Batching amortises the memory traffic of reading model weights across many requests, so throughput rises far faster than per-step time does. Continuous batching lets requests join and leave each iteration, but the gain is paid for in tail latency, which is why interactive and bulk traffic want separate pools.
On small and medium tabular data, where gradient-boosted trees usually match it for a fraction of the effort; where a decision must be explained exactly rather than approximately; and where a tight CPU latency or memory budget makes it unservable. The simpler model then wins on every axis that matters.
It earns it when your gold set shows recall at 50 is much better than recall at 5, because that gap is what reranking converts into answer quality. The cost is a model pass per candidate with nothing precomputable, so the latency budget divided by measured per-batch cost sets the candidate count.