Your code needs JSON back from the model and it has to parse every time. How do you make that happen?

An instruction is a request, the decoder is an enforcement point

Asking for JSON in the prompt is a probabilistic request. Most of the time you get JSON, and the rest of the time you get JSON wrapped in a markdown fence, or a sentence of preamble in front of it, or trailing commentary explaining the choice. Every one of those is a parse failure, and the usual response — a firmer instruction, capital letters, "respond with ONLY the raw JSON object" — reduces the rate without changing the fact that the output is sampled rather than constrained.

The mechanism that does change it is constrained decoding. At each step the model produces a distribution over the whole vocabulary; a constraint compiled from your schema masks out every token that cannot appear next in a valid document, and sampling happens over what remains. If the grammar says the next character must be a quote or a closing brace, no other token is reachable, whatever the underlying probabilities were. That is a structural guarantee about the shape of the output, and it is qualitatively different from a well-worded request. Where a provider exposes this — as a strict schema-constrained response mode, or by forcing a tool call whose input schema you define — use it, and treat plain instructions as the fallback for providers that do not.

Two caveats keep this honest. Constrained decoders generally implement a subset of JSON Schema, and the omissions are the interesting parts: recursive definitions, some composition keywords, and numeric range or string pattern constraints are often unsupported or silently ignored, so you cannot assume a value is in range just because it parsed. And a constraint on shape says nothing about content. A schema that requires a confidence number between zero and one will happily receive a fabricated one.

Truncation is arithmetic, and no wording fixes arithmetic

The most common source of unparseable output in production is not disobedience, it is running out of room. Generation stops when the model emits a stop token or when it reaches the output limit, and if it reaches the limit mid-document you get a prefix of valid JSON that ends inside a string. This is worse than a refusal, because the failure looks like a model quality problem when it is a budgeting problem, and it correlates with exactly the inputs you care about — the long documents, the results with many items, the free-text explanation that ran to three paragraphs.

So reserve the output. The context window is shared between everything you send and everything the model generates, and the reservation has to be subtracted before you decide how much input to include, not hoped for afterwards. Estimate the worst-case serialised size of your schema — every optional field present, arrays at their maximum length, free-text fields at whatever cap you have imposed — convert it to tokens with the provider's own tokeniser rather than a characters-per-token rule of thumb, and add headroom. JSON is token-expensive relative to prose because punctuation, quoted keys and escapes each cost tokens, and a deeply nested object repeats its keys on every element of every array.

Then detect the case explicitly. The response carries a reason for stopping, and a length-limited stop is a distinct signal from a completed one. Branch on it before you attempt to parse, because parsing a truncated document and reporting a syntax error throws away the one piece of information that tells you what actually happened.

Validate at the boundary and treat failure as an outcome

Whatever the decoder guarantees, validate the parsed object against the schema in your own code before anything downstream sees it. The provider may change behaviour, the constraint may not cover your range checks, and a fallback path on a different model may not be constrained at all. The validation belongs at the edge of the system, in the same place you would validate an untrusted HTTP body, and the result of the call should be a type your code cannot use without handling the failure branch.

# The stop reason is checked before parsing: a truncated body is a
# budgeting failure, not a syntax error, and must not be retried blindly.
if response.stopped_for_length:
    raise OutputBudgetExceeded(reserved=max_output_tokens)

try:
    extraction = Extraction.model_validate_json(response.text)
except ValidationError as err:
    extraction = repair_once(response.text, err.errors())  # one attempt, then give up

One retry, and it must quote the violation

A single repair attempt is worth having, because a near-miss is often recoverable. It is only worth having if it carries new information. Sending the original prompt again is a lottery ticket at full price; sending the model its own invalid output plus the specific validator message — which field, which constraint, what was received — turns the retry into a correction task, which is a much easier one.

Bound it at one attempt. An unbounded repair loop is the shape of every LLM cost incident: each iteration pays for the whole prompt again, latency compounds, and a systematically impossible schema means every attempt fails identically. If one repair does not succeed, fail the request to a deterministic fallback and record the input, because two failures on the same document is a signal about your schema rather than about that call.

Shape is enforceable by the decoder, size is enforceable only by arithmetic you do beforehand, and correctness is enforceable by neither.