How do you ship a change to a production prompt safely?

A prompt is code with an unusually bad diff

A prompt determines behaviour, has no type system, no compiler and no local test that runs in a second, and a two-word change can alter output across every request. That combination argues for more process than normal code, not less, and the process is mostly the ordinary one applied honestly.

Keep the prompt in the repository, in a file, with a version identifier. It goes through pull request review like anything else, and the diff is worth reading carefully because the reviewer's job is to ask what the removed sentence was compensating for. If a prompt-management platform is in use, treat it as the deployment mechanism rather than the source of truth: something that can be edited in a console by anyone, without review or history, will be, usually during an incident, and afterwards nobody can tell you what the prompt said last Tuesday.

Version the whole assembled unit, not just the instruction text. The system prompt, the tool definitions, the schema you constrain output to, the retrieval parameters and the model settings all shape the result together, so a change to any of them is a change to the artefact. A version identifier that covers only the prose leaves you unable to explain a behaviour shift caused by the retrieval count going from five to eight.

Pin the model, or you cannot attribute anything

This is the load-bearing detail. If your configuration names a floating alias that resolves to whatever the provider currently serves, then when quality moves you have two candidate causes — your edit and the endpoint changing underneath you — and no way to separate them. You will spend the day arguing about it and reach no conclusion, because the evidence to distinguish them was never collected.

Pin to a specific dated or numbered model identifier, and record that identifier on every request alongside the prompt version. Now a regression is either in the diff, in which case revert, or in something you did not change, in which case it is worth escalating to the provider or investigating as data drift. Moving to a new model version becomes its own deliberate change, evaluated on its own, rather than an event that arrives on a Thursday. Sampling parameters get the same treatment: a non-zero temperature is fine in production, but the eval run should hold it fixed and low so that run-to-run variance does not swamp the effect you are measuring.

The gate in CI

The eval set is a collection of real inputs with graded expectations, and running it on every prompt change is what turns "it looks better to me" into a decision. Its composition matters more than its size: cases drawn from actual traffic, every past regression preserved as a permanent case, and the awkward inputs — empty, hostile, ambiguous, in the wrong language, adversarial in the injection sense — represented deliberately.

Grade by the cheapest method that works per case. Exact assertions where the output is structured or the answer is closed. Programmatic checks for the properties you care about: valid JSON against the schema, a citation present, no forbidden claim, length within bounds. A judge model only for the genuinely open-ended cases, with its own agreement against human labels checked periodically, because an unvalidated judge is a metric you have not verified.

Report the result as a comparison against the current production version on the same set, not as an absolute score, and set a pass rule that includes both a floor and a no-regression clause. The regression clause is the one that catches real damage: an aggregate that improves by two points while three previously passing cases now fail is a change that will generate support tickets. Publish the per-case diff into the pull request so the reviewer can see which behaviours moved.

Canary and rollback

Evals are a proxy, and the honest reason to canary is that they miss the input distribution you have not thought of yet. Route a slice of production traffic to the new prompt version, keyed deterministically so a given user does not flip between behaviours mid-conversation, and compare the two arms on the signals you can measure live: refusal rate, retry and regeneration rate, tool-call error rate, output validation failures, latency, cost per request, and whatever implicit satisfaction signal your product provides such as edits, thumbs-down or abandonment. Ramp only when the arms are indistinguishable or the new one is better.

Rollback should be a configuration change that takes effect without a deploy, because you will want it at the moment when a deploy is slowest. That means the active prompt version is runtime state, and both versions remain loadable. The case where this is not enough is worth naming: if the change altered the output schema, or changed what gets written to a datastore downstream, then reverting the prompt leaves inconsistent records behind, so schema-shaped changes need the same forward-and-backward compatibility discipline as a database migration — ship the reader that tolerates both shapes first, then the prompt.

Keep the whole thing legible after the fact. A prompt version identifier on every request, a changelog entry per version saying what it was meant to change and which eval cases moved, and the eval run archived against that version. Six months later somebody will ask why an instruction is in the prompt, and the only defence against carrying dead instructions forever is a record of what each one was added for.