Token economics, streaming and time-to-first-token, batching and KV cache, quantisation trade-offs, and routing between models.
3 questions in Generative AI & Large Language Models.
Split the measurement into time-to-first-token and inter-token latency, because they come from different phases. The first is driven by prompt length and queueing and is fixed with shorter prompts, prefix caching and streaming; the second is the decode loop, fixed only by generating fewer tokens.
Decide from the task. For grounded work over your own corpus, quality is set by what reaches the context, so retrieval dominates model size and a smaller model usually suffices. Keep the larger model for long multi-step reasoning, and treat routing as a second system to evaluate and monitor.
Batching amortises the memory traffic of reading model weights across many requests, so throughput rises far faster than per-step time does. Continuous batching lets requests join and leave each iteration, but the gain is paid for in tail latency, which is why interactive and bulk traffic want separate pools.