hardConceptDesignMidSeniorStaff
What is the KV cache, and why does it decide how many users one GPU can serve?
The KV cache stores the attention keys and values of every token a sequence has seen, so decoding need not recompute them. It grows with tokens per sequence and with concurrent sequences, and after the weights it fills GPU memory, so tokens in flight, not requests, set how many users one GPU serves.
4 min readUpdated 2026-10-10
Answer
Short answer
The KV cache holds the attention keys and values for every token a sequence has processed. Without it, each new token would recompute attention inputs for the whole history. It grows with every token of every concurrent conversation, and after the weights it is the main consumer of GPU memory. So the number of users one GPU serves is free memory divided by cache bytes per token, divided by tokens per conversation.
Was this answer helpful?
Likely follow-ups
- Product wants to raise the context limit from 8K to 32K tokens. What do you tell them about capacity, with numbers?
- What happens to running requests when the cache fills mid-generation, and which recovery is cheaper for a long prompt?
- How does a shared system prompt change the memory arithmetic when the engine can reuse cached prefixes?
- What would you measure before storing the cache at 8-bit precision instead of 16-bit?
Related questions
- What does one prediction cost you, and how would you work it out?mediumAlso on capacity-planning and inference5 min
- What does batching buy you when serving an LLM, and what does it do to your latency budget?hardAlso on kv-cache and inference5 min
- How would you autoscale a GPU inference service?hardAlso on inference and capacity-planning6 min
- How do you decide whether a model should be served in batch, online or streaming?mediumAlso on inference4 min