hardDesignMidSeniorStaff
Design a chat service that streams model output to users and lets them press stop. What has to happen when they do?
Stream tokens over server-sent events or a WebSocket, but treat stop as a server-side command routed to whichever worker owns the generation. That worker aborts the upstream model call, saves the partial reply with a stopped status, and sends a final event so the client and the stored history agree.
6 min readUpdated 2026-10-10
Answer
Short answer
Streaming is the easy half. Send tokens to the browser over server-sent events or a WebSocket, but treat the stop button as a command to the server, not a client-side switch. The worker that owns the generation must abort the upstream model call, save the partial reply with a stopped status, and emit a final event so the screen and the stored history agree.
Was this answer helpful?
Likely follow-ups
- The user refreshes the page halfway through a reply. How does the new tab pick up the same stream?
- How do you run moderation on output the user is already reading?
- A stopped reply is half a sentence. What goes into the context of the next turn: the fragment, nothing, or a marker?
- Your provider keeps generating for a moment after you close the connection. How would you find out what you are billed for?
Related questions
- You stream the model's answer token by token to the browser. What does that change about error handling, guardrails and cost?hardAlso on streaming and cancellation5 min
- Rate-limit an LLM gateway where one request can cost a hundred times another.hardAlso on streaming5 min
- How do you decide whether a model should be served in batch, online or streaming?mediumAlso on streaming4 min
- The same domain has to be exposed to a mobile app, a partner integration and internal service-to-service traffic. Where does GraphQL fit, where does gRPC, and where does neither?hardAlso on streaming7 min