You stream the model's answer token by token to the browser. What does that change about error handling, guardrails and cost?

Streaming trades a guarantee for a perception

Streaming exists because a generated answer arrives at a rate the reader can consume, so showing the first words after a few hundred milliseconds makes a response that takes eight seconds feel responsive. The gain is real and it is entirely perceptual: total generation time is unchanged, and in exchange you give up something concrete, which is the ability to look at the whole answer before deciding what to do with it.

That is the trade to lead with, because everything difficult about streaming follows from it. In a buffered design the server holds the complete output, inspects it, validates it, decides on a status code and then responds. In a streaming design you have already answered. The status line went out before the first token, the headers are committed, and the transport is now a body you can only append to.

Guardrails split into two kinds

Once you accept that, checks divide cleanly. Some can run incrementally on the tokens as they pass, because they are decidable on a prefix: a blocklist, a regular expression for something that looks like a credential, a detector for a leaked system prompt. Those can be applied in the stream, and the strongest form is to buffer a small sliding window rather than a single token so that a pattern spanning a chunk boundary is still caught.

The rest are not decidable on a prefix, and this is the category that matters. Whether the answer is grounded in the retrieved passages, whether it contradicts itself, whether it answered the question asked, and whether a classifier judges it safe are all properties of the complete text. Running them means either buffering until the end, which discards the entire reason you streamed, or running them after the user has already read most of the output, which means your remedy is a retraction rather than a prevention.

Neither option is wrong, but the choice has to be deliberate and it should differ by surface. A conversational assistant can reasonably stream and retract, replacing the message with a notice if the post-hoc check fails, because the cost of a briefly visible bad answer is low. A feature that produces a legal summary, a medical statement or anything a user might screenshot should not stream at all, and the honest way to keep it feeling fast is a progress indicator over a buffered call rather than partial content you may have to withdraw. Interviewers listen for whether you make that distinction or apply one policy everywhere.

Failures that arrive after success

The second consequence is that your error contract now has a shape most services never need. Consider what happens when generation fails two hundred tokens in: the upstream provider errors, a rate limit is hit mid-generation, a network hop drops, or your own guardrail trips.

You cannot send a 500, because you sent 200. Whatever you do has to be expressed inside the body, which means the stream needs a protocol rather than being a bare sequence of text fragments. In practice that means framed events with a type, so the client can distinguish a content chunk from a terminal error from a normal completion, and a defined end-of-stream marker so a truncated connection is distinguishable from a finished one. Without the marker, a client cannot tell a complete answer from a severed one, and the usual symptom is a message that silently stops mid-sentence and is stored as if it were the model's considered output.

Retry is the harder half. Reissuing the request produces a fresh generation that does not continue the text the user has already read, so appending it is incoherent and replacing it discards visible content. The defensible options are to fail the message visibly and offer a regeneration the user initiates, or to reserve automatic retry for failures that happen before the first token is emitted, where nothing has been shown and the request is still safely repeatable. Choosing where that boundary sits, rather than retrying blindly, is the design decision.

Cancellation is a cost control

A buffered request that the client abandons is wasted work you were going to pay for anyway. A streamed request that the client abandons is wasted work you continue to do, because generation runs to completion on the server unless something tells it to stop, and tokens are billed as they are produced whether or not anyone is reading them.

So client disconnection has to be detected and propagated to the upstream call rather than absorbed by an intermediate layer. This is the specific place where a naive proxy or a framework that reads the response into a buffer before forwarding it quietly destroys the property, since the cancellation signal never reaches the generation. It matters most on chat surfaces, where users routinely ask a question, read the first sentence, decide it is not what they wanted and navigate away — a pattern that generates a large volume of tokens with no reader at all.

Pair it with a hard output cap regardless of cancellation. A generation that fails to stop is bounded by the maximum output length you set, and that ceiling is the difference between a bad prompt costing a fraction of a penny and it costing whatever the context window permits.

Evaluating something that arrived in pieces

Streaming also breaks the observability you would otherwise get for free, because there is no single response object to log. You have to assemble one: record the concatenated output, the finish reason, whether a guardrail intervened and at which point, and whether the client disconnected before completion. Without that, your evaluation set is missing exactly the responses that failed interestingly, and your quality dashboards are computed over the subset that completed cleanly.

Latency needs the same treatment. One number no longer describes the request, and the two that do are time to first token, which is what users perceive as speed, and total generation time, which is what capacity planning depends on. A change that improves one can worsen the other — batching being the obvious case — so putting only the aggregate in an SLO hides the regression users will actually complain about.

Streaming means answering before you know what the answer is. Decide per surface whether partial content you may have to retract is acceptable, frame the stream so errors and completion are expressible inside the body, and make sure an abandoned request stops generating rather than quietly finishing on your bill.