mediumConceptScenarioMidSeniorStaff
Users say the LLM feature in your product feels slow. How do you work out what to fix?
5 min readUpdated 2026-08-15
Answer
Short answer
Split the measurement into time-to-first-token and inter-token latency, because they come from different phases. The first is driven by prompt length and queueing and is fixed with shorter prompts, prefix caching and streaming; the second is the decode loop, fixed only by generating fewer tokens.
Was this answer helpful?
Likely follow-ups
- Your p50 time-to-first-token is fine but p99 is five times worse. Where do you look?
- How would you make a two-step chain feel faster without removing either step?
- You add a reranker in front of the model and total latency rises. How do you decide whether to keep it?
- What changes about this analysis when the feature is a background job rather than a chat box?
Related questions
- How do you decide whether a model should be served in batch, online or streaming?mediumAlso on inference and streaming4 min
- Your inference bill tripled last month and traffic was flat. How do you find out why?hardAlso on observability and inference6 min
- You have 50 ms for a model call in a request path. How do you make that budget?hardAlso on latency and inference5 min
- Your p99 latency jumped tenfold after a deploy while p50 is unchanged. Diagnose it out loud.hardAlso on latency and observability6 min