Top 9 Inference Interview Questions
Browse 9 real-world technical and behavioral interview questions about Inference. Review scenarios, edge cases, and architectural best practices.
What is the KV cache, and why does it decide how many users one GPU can serve?
The KV cache stores the attention keys and values of every token a sequence has seen, so decoding need not recompute them. It grows with tokens per sequence and with concurrent sequences, and after the weights it fills GPU memory, so tokens in flight, not requests, set how many users one GPU serves.
How do you architect a serving system for a 70B parameter LLM to maximize GPU throughput without violating strict time-to-first-token (TTFT) latency SLAs?
An authoritative deep dive into LLM inference optimization, exploring continuous batching, paged attention for KV cache memory management, and tensor parallelism scheduling.
How would you autoscale a GPU inference service?
Scale on queue depth or in-flight concurrency rather than request rate, because a GPU replica that takes minutes to load weights cannot be added after the queue has already formed. Then decide deliberately how much warm capacity to keep, since that is what you are paying to avoid cold starts.
How do you decide whether a model should be served in batch, online or streaming?
Freshness decides it. If a prediction is still valid hours later, precompute it in a batch job and serve a lookup; if it depends on what the user did seconds ago, score it online; if it depends on an evolving event sequence, score it from a stream. Most mature systems end up hybrid.
Users say the LLM feature in your product feels slow. How do you work out what to fix?
Split the measurement into time-to-first-token and inter-token latency, because they come from different phases. The first is driven by prompt length and queueing and is fixed with shorter prompts, prefix caching and streaming; the second is the decode loop, fixed only by generating fewer tokens.
Your inference bill tripled last month and traffic was flat. How do you find out why?
Flat traffic and a tripled bill means cost per request tripled, so hunt what changed per request: a larger model version, more feature reads per prediction, retries multiplying calls, or warm capacity left running. Per-request cost attribution is what makes this answerable.
You have 50 ms for a model call in a request path. How do you make that budget?
Break the budget into its parts first, because feature fetching usually costs more than the model does. Then parallelise and cache the fetches, shrink the model with quantisation or distillation, set a timeout shorter than the budget, and define a degraded path that answers without the model.
What does one prediction cost you, and how would you work it out?
Divide the all-in hourly cost of the serving capacity by the predictions it delivers in that hour at your real utilisation, then add the per-request cost of feature reads. Utilisation dominates the result, and most of what people call cost per prediction is fixed cost that one more request does not change.
What does batching buy you when serving an LLM, and what does it do to your latency budget?
Batching amortises the memory traffic of reading model weights across many requests, so throughput rises far faster than per-step time does. Continuous batching lets requests join and leave each iteration, but the gain is paid for in tail latency, which is why interactive and bulk traffic want separate pools.