hardConceptDesignMidSeniorStaff
You stream the model's answer token by token to the browser. What does that change about error handling, guardrails and cost?
5 min readUpdated 2026-08-15Target archetype: Big Tech, Enterprise Captive, Product Startup
Answer
Short answer
Streaming commits you to the response before you have seen it: the status code is already sent, any check that needs the whole output now runs after the user has read part of it, a mid-stream failure has no clean retry, and an abandoned tab keeps generating tokens you pay for.
Was this answer helpful?
Likely follow-ups
- Your endpoint must return validated JSON. Do you stream it at all, and what do you show the user meanwhile?
- How would you report an error that happens after two hundred tokens have already been displayed?
- Where does a caching layer fit if responses are streamed, and what can you cache?
- What does streaming do to your latency measurements, and which number do you put in an SLO?
Related questions
- Design a chat service that streams model output to users and lets them press stop. What has to happen when they do?hardAlso on streaming and cancellation6 min
- What is the KV cache, and why does it decide how many users one GPU can serve?hardAlso on llm-serving4 min
- How do you decide whether a model should be served in batch, online or streaming?mediumAlso on streaming4 min
- The same domain has to be exposed to a mobile app, a partner integration and internal service-to-service traffic. Where does GraphQL fit, where does gRPC, and where does neither?hardAlso on streaming7 min