Token economics, streaming and time-to-first-token, batching and KV cache, quantisation trade-offs, and routing between models.
5 questions in Generative AI & Large Language Models.
Split the measurement into time-to-first-token and inter-token latency, because they come from different phases. The first is driven by prompt length and queueing and is fixed with shorter prompts, prefix caching and streaming; the second is the decode loop, fixed only by generating fewer tokens.
The KV cache stores the attention keys and values of every token a sequence has seen, so decoding need not recompute them. It grows with tokens per sequence and with concurrent sequences, and after the weights it fills GPU memory, so tokens in flight, not requests, set how many users one GPU serves.
Split LLM cost by feature and by line: uncached input, cached input, output, retries and hidden callers. Find which line moved before cutting anything. A collapsed prompt-cache hit rate or longer replies can double a bill with no change in traffic; fix those before you downgrade a model.
Decide from the task. For grounded work over your own corpus, quality is set by what reaches the context, so retrieval dominates model size and a smaller model usually suffices. Keep the larger model for long multi-step reasoning, and treat routing as a second system to evaluate and monitor.
Batching amortises the memory traffic of reading model weights across many requests, so throughput rises far faster than per-step time does. Continuous batching lets requests join and leave each iteration, but the gain is paid for in tail latency, which is why interactive and bulk traffic want separate pools.