How do you architect a serving system for a 70B parameter LLM to maximize GPU throughput without violating strict time-to-first-token (TTFT) latency SLAs?
3 min readUpdated 2026-08-15
0
Answer
Short answer
An authoritative deep examine LLM inference optimization, exploring continuous batching, paged attention for KV cache memory management, and tensor parallelism scheduling.
Was this answer helpful?
Likely follow-ups
→How would prefix caching change your admission-control decisions when many requests share a system prompt?
→What signals would tell your scheduler it is time to preempt a running request rather than let it finish?
→How does the scheduler's behaviour need to change if you introduce a second model size onto the same GPU pool?
Serving a 70B parameter Large Language Model in production is fundamentally a memory-bandwidth problem masquerading as a compute problem. The primary objective is ruthless efficiency: maximising batch size to increase token throughput and reduce serving costs, while strictly capping the time-to-first-token (TTFT) to preserve user experience. Production workloads are inherently hostile, characterized by wildly heterogeneous prompt lengths and unpredictable output generation. Under these conditions, legacy serving architectures buckle under their own inefficiency.
Relying on traditional static batching for LLM inference guarantees resource exhaustion. In static batching, the entire batch must wait for the longest sequence to complete generation. Shorter sequences sit idle, wasting precious GPU cycles, and the key-value (KV) cache memory is severely fragmented by padding tokens. This static approach predictably results in catastrophic throughput degradation and unavoidable Out-Of-Memory (OOM) errors under peak load.
Continuous batching and paged attention
The definitive architecture requires continuous batching (iteration-level scheduling). The serving engine dynamically injects new requests into the compute pipeline at the exact iteration a running request completes its generation. This demands highly aggressive memory management. The KV cache, which stores the attention keys and values for previously generated tokens, grows dynamically and unpredictably, making contiguous memory allocation impossible.
To solve this, paged attention is mandatory. Borrowing the concept of virtual memory from operating systems, the KV cache is partitioned into fixed-size blocks. Sequences no longer require contiguous memory; their state is mapped via a block table to non-contiguous physical blocks allocated dynamically on the GPU.
Loading diagram…
Block size configuration is a delicate trade-off. Small blocks minimize internal fragmentation but explode the size of the block table and increase the overhead of memory mapping within CUDA kernels. Large blocks reduce metadata overhead but waste memory on internal fragmentation for short sequences. Profiling the specific production workload distribution is required to tune this parameter effectively.
Orchestrating distributed KV caches
For a 70B model, tensor parallelism is required to fit the weights and KV cache across multiple GPUs. This fundamentally complicates the continuous batching scheduler, which must now operate as a distributed coordinator. A request can only be admitted if sufficient KV cache blocks are available synchronously across all participating devices. The AllReduce operations required to synchronize activations during the forward pass become brutal bottlenecks if batch sizes drift or network topology is sub-optimal. InfiniBand or NVLink topologies are strictly required to minimize communication latency, alongside execution engines capable of overlapping computation with network transfers.
Furthermore, prefix caching must be integrated to eliminate redundant prefill computation for requests sharing system prompts or multi-turn contexts. By implementing robust reference counting, allocated memory blocks can be safely shared across different sequences matching the same prefix, bypassing the compute-heavy prefill phase entirely for those shared tokens.
The scheduling policy dictates the system's survival under load. A naive First-In-First-Out (FIFO) queue causes starvation. When the KV cache is exhausted during autoregressive generation, the scheduler must preempt active requests. It must dynamically calculate the penalty of swapping the preempted request's KV cache to CPU memory (incurring PCIe bandwidth latency) versus aggressively recomputing the prompt entirely when memory becomes available. This decision must be evaluated dynamically based on prompt length and current hardware utilization, relentlessly prioritizing the defined SLOs over local optimizations.
The successful deployment of a large language model requires a holistic architectural approach where dynamic memory management, distributed systems coordination, and hardware-aware scheduling converge to maximise hardware utilisation whilst rigorously maintaining latency objectives.