Users say the LLM feature in your product feels slow. How do you work out what to fix?

One number is not enough to act on

"The feature is slow" is unactionable because total wall-clock time is the sum of two phases with almost nothing in common. Before touching anything, instrument two metrics per request: time-to-first-token, the interval from your call leaving the application to the first content token arriving, and inter-token latency, the average gap between subsequent tokens. Record output token count alongside them, because total time is roughly time-to-first-token plus inter-token latency multiplied by tokens generated.

That decomposition immediately tells you which complaint you have. A long silence followed by fast text is a prefill problem. Text that starts promptly and then crawls out for twenty seconds is a decode problem, or simply a response that is far longer than anyone wanted. The two have disjoint fix lists, which is why guessing wastes a sprint.

Prefill and decode

Prefill is the forward pass over the whole prompt. Every token of the prompt is processed, and because they are all available at once the work parallelises well, so prefill is compute-bound and its cost grows with prompt length. Decode is the autoregressive loop: one token at a time, each step reading the model weights and the accumulated key-value cache out of memory to produce a single token. Decode is bound by memory bandwidth rather than arithmetic, which is why a step takes essentially as long whether you are generating for one user or several.

flowchart LR
    A[Request accepted] --> B[Queue wait]
    B --> C[Prefill<br/>whole prompt, compute bound]
    C --> D[First token emitted]
    D --> E[Decode step<br/>memory bandwidth bound]
    E --> E
    E --> F[Stop token or limit]

The self-loop is the point: everything before the first token happens once, and everything after it happens once per output token, so a fix aimed at one side of that boundary cannot move the other.

Fixing time-to-first-token

The levers are prompt size, queueing and caching. Prompt size is usually where the surprise lives: a retrieval step that grew from three chunks to fifteen, a conversation history replayed in full on every turn, a system prompt that accumulated instructions nobody has deleted. Measure input tokens per request over time and you will often find the regression predates the complaints.

Prefix caching is the strongest lever available without changing the prompt's content, and it has one rule you must state correctly. Providers cache the computed state of a prompt prefix, and a request reuses it only up to the first token that differs. That makes prompt layout load-bearing: the stable material — system instructions, tool definitions, long reference documents, few-shot examples — belongs at the front, and anything per-request, especially a timestamp, a session identifier or a randomised ordering, belongs at the end. A single injected "current time" line near the top silently disables caching for the entire prompt behind it. Check whether your provider reports cached input tokens per call and alert on the hit rate, because a prompt edit that reorders one block can take the rate to zero without any other symptom.

Streaming does not reduce time-to-first-token but it changes what the user experiences after it, and it is the difference between one long wait and a response that appears to be working. If the feature currently buffers the whole completion before rendering, streaming is usually the highest-value change available, and it costs you the ability to validate or post-process the output before showing it, which matters if you were relying on that.

Fixing inter-token latency

Here the honest answer is that you have few levers on a hosted model, because per-step time belongs to the provider's hardware and serving configuration. What you control is how many steps happen. Cap maximum output tokens deliberately rather than leaving a generous default. Ask for the shape you actually need: a structured object with three short fields instead of an explanatory essay containing them. Remove instructions that invite the model to restate its reasoning when you are not using that reasoning. If the model is one of the reasoning variants, the thinking tokens are decode steps too, and they are frequently the bulk of the wait.

Beyond that, the choices are architectural. A smaller or distilled model in the same family has a faster decode loop; whether it is good enough is an evaluation question, not a latency question, and you answer it with your eval set. Self-hosted deployments add quantisation and speculative decoding to the list, both of which trade something for speed and both of which need their own quality check before you rely on them.

Where the analysis usually goes wrong

The mistake that costs candidates the question is proposing a fix from the wrong side of the boundary and not noticing. Prompt caching does nothing for a user watching tokens dribble out, because prefill was already finished when the first one arrived. Trimming the output does nothing for a user staring at an empty box for six seconds. Switching to a smaller model helps both, which makes it tempting, and also changes your quality profile, which makes it the most expensive answer to give first.

Two structural causes hide behind both metrics and are worth checking early. Sequential chains multiply the whole cost: two calls where the second depends on the first means two prefills and two decode loops, and a chain of four is four silences even if each stage looks fine in isolation. And a bad p99 with an acceptable p50 is usually queueing rather than model speed, which points at concurrency limits, rate-limit retries with backoff, or your own thread pool, none of which get better by editing the prompt.