How do you write prompts that survive a model upgrade?

What a prompt is really made of

Pull apart any prompt that has been in production for a year and you find two kinds of text mixed together. There is the intent: what the task is, what the output must contain, what the boundaries are, what to do when the input is unusable. And there is the compensation: Do not begin your reply with "Certainly", Return the JSON with no markdown fences, You MUST include the currency code, Think step by step before answering. Each compensating line was added on a specific afternoon because a specific model did a specific annoying thing, and each one is invisible in the file as anything other than another instruction.

Intent survives an upgrade. Compensation is fitted to one model's behaviour, and after the upgrade it ranges from harmless through wasteful to actively harmful. Harmful is the case people miss: instructions that scaffolded a weaker model can suppress a stronger one. Forcing a rigid reasoning template on a model already trained to reason at length can truncate its own process; few-shot examples chosen to demonstrate a format to a model that struggled with it can anchor a better model onto the exact phrasing and shallowness of those three examples. So a prompt that grew by accretion does not just fail to benefit from the upgrade, it can lose to it.

The practical discipline is to keep the two kinds of text visibly separate, and to write each compensation with the date and the failure it fixes attached as a comment. That turns a wall of imperatives into a list of hypotheses you can test and delete, which is the only way anyone will ever dare delete them.

The schema and the eval set are the durable artefacts

Prompt wording is the least portable thing you own. The output schema is the most portable: it is a contract expressed in a form no model gets a vote on, and it is what the rest of your system was written against. If a change of model does not change the schema, the change is contained. A feature whose downstream code parses free prose has no such containment, and every upgrade becomes a diff against text nobody specified.

The eval set is what makes the upgrade a decision rather than a leap. Without one, "does the new model work for us" resolves to somebody trying five prompts by hand and forming an impression, which reliably rewards the model that writes more fluently rather than the one that is right more often. With one, you run both models over the same cases, compare on the metrics you chose in advance, and see the regressions as well as the wins. That is also the mechanism that lets you attempt the removal of compensating instructions: drop one, run the set, and if nothing moves, it was scaffolding for a model you no longer use.

Give the eval set enough breadth that it can register loss as well as gain. An upgrade that improves the headline task while degrading refusals, tone or non-English inputs is a net loss you will discover from support tickets if you did not include those cases.

Pin the version, because the alternative is silent

Production should name a specific model version, never a floating alias that resolves to whatever is current. A floating alias means the provider gets to change your system's behaviour at a time of their choosing with no deployment on your side, no diff, and no correlation to anything in your change log. When output quality shifts on a Tuesday and nothing was released, that is where you look — and if you were pinned, you can rule it out in a minute.

Pinning bounds the risk without eliminating it, so pair it with two other things. Watch deprecation notices as a scheduled obligation rather than an email you will read later, because a pinned version has a retirement date and discovering it during the retirement window is the worst time to run a migration. And keep the model identifier in configuration, not in code, so that switching it and switching back are both operations you can perform during an incident.

Making the switch

Treat the migration as a rollout, not an edit. Run the new model in shadow first, on a sample of live traffic, scoring both outputs offline where the comparison costs you nothing in user experience. Then move a small share of real traffic and watch the product signals the eval set cannot contain: retries, abandonment, escalation to a human, edit-distance between what the model produced and what the user shipped. Keep the old model reachable by configuration until the new one has held for long enough to be boring.

Then do the cleanup, because otherwise you have two models' worth of compensations in one file. Once the new model is carrying the traffic, work through the annotated workarounds and remove the ones whose failures no longer reproduce. A prompt that survives an upgrade untouched has usually not survived it well; it has merely postponed the reckoning to the next one.