Your provider announces the model you depend on retires in six weeks. Walk me through what you do.
6 min readUpdated 2026-08-15
0
Answer
Short answer
Inventory everywhere the model is coupled to behaviour, then use the eval set as your migration harness, shadow the candidate on live traffic to compare without user exposure, and re-measure cost and latency because both shift.
Was this answer helpful?
Likely follow-ups
→You have no eval set on the day of the announcement. What do you do in the first week?
→How would you migrate a fine-tuned model whose base is being retired?
→The replacement is better on your evals but 40% slower to first token. How do you decide?
→What would you change in the architecture so the next deprecation is cheaper?
The instinct is to swap the model identifier and see what happens, and it is the wrong first move because you do not yet know what depends on the old behaviour. Deprecation is a coupling audit with a deadline.
Find every call site. That includes the ones outside the main service: batch jobs, an internal tool somebody built, the summarisation step in a data pipeline, the judge model in your own eval harness. A judge on a retiring model is the trap inside the trap, because if it changes at the same time as the system under test, your measurements move for reasons you cannot untangle — migrate the judge separately and before the rest, and re-establish its agreement with human labels afterwards.
Then find the behavioural coupling, which is the expensive part. Anything downstream that parses the output is coupled to its shape: regexes over prose, code expecting a particular field ordering or a habitual preamble, a UI that assumes the model always answers in three paragraphs. Prompts containing few-shot examples chosen because this model responded to them. Tool-calling behaviour, since models differ in how eagerly they call tools and how they handle a tool returning an error. Token-budget assumptions, because a different tokeniser and a different context limit change what fits. Any stored artefacts that will now be compared against differently-generated ones, embeddings being the obvious case if the deprecation touches an embedding model as well.
Write that list down and rank it by how visible a break would be. It is also the scope estimate you owe whoever is asking whether six weeks is enough.
The eval set is the migration harness
If you have an eval set, this is the moment it pays for itself, and the migration needs no new tooling: run the current model and each candidate replacement over the same cases and read the per-case diff. Aggregate scores are nearly useless here because they hide the shape of the change; what you need is which specific cases flipped, in which direction, and whether the failures cluster around a task type.
If you do not have one, building it is the first week's real work, not an optional extra, and the fastest honest route is to sample production traffic, capture the current model's outputs while it still exists, and have someone grade a few hundred of them. Those captured outputs are perishable — once the model is retired you cannot regenerate the baseline — so capture before you evaluate. Include the cases you already know are hard, the ones behind past incidents, and a deliberate spread across the input distribution rather than the pleasant middle of it.
Expect the diff to be two-sided. A newer model is usually better on the aggregate and worse on something specific: verbosity, formatting discipline, willingness to answer a borderline request, exact adherence to an output template. Those specifics are what your product's behaviour was built on, so they are the work, and finding them in an eval run is much cheaper than finding them in support tickets.
Shadow before you switch
Evals cover the distribution you thought of. Shadow running covers the one you have. Send live requests to both models, serve the incumbent's response to the user, and record both for comparison.
Loading diagram…
The branch worth noticing is that the candidate's output never reaches a user, so a bad response costs nothing but a little money — which is the point, because it lets you sample real traffic at real volume before committing. Grade the pairs offline, with a judge for scale and humans on the disagreements, and pay attention to the cases where the two differ most rather than to the average similarity. Shadow running does cost double inference on the shadowed fraction, so sample rather than mirroring everything, and stratify the sample so rare request types are actually represented.
When you cut over, do it as a canary keyed per user, with the model identifier recorded on every request so production metrics can be split by arm. Keep both paths live until the retirement date genuinely forces the issue, and remember that "rollback" acquires an expiry: after the shutdown there is no reverting, which is the argument for finishing the migration well before the deadline rather than up against it.
Cost and latency are separate migrations
Do not let the quality result stand in for the whole decision. Per-token pricing changes, and so does the number of tokens: a different tokeniser gives a different count for the same text, a chattier model produces longer outputs, and a reasoning-style replacement can generate a large volume of tokens you pay for and never display. Re-derive cost per thousand requests from your own measured input and output lengths on the new model rather than reasoning from the headline price.
Latency needs the same treatment, split into time-to-first-token and inter-token latency, because a replacement can improve one and worsen the other. Check prompt caching too: cache behaviour, discounts and minimum cacheable prefix sizes are properties of the model and tier, so a migration can quietly turn a cached workload into an uncached one and multiply the bill without any change in traffic. Rate limits and quotas are usually per model as well, so confirm you have the headroom before cutover rather than discovering it under load.
Delete the scar tissue instead of porting it
A prompt that has been in production for a year is full of instructions that exist to compensate for the old model's specific failures. "Do not begin your answer with a summary." "Always output valid JSON with no surrounding text." "Do not apologise." Each was added after an incident, and each is now unexamined.
Carrying them forward wholesale is the most common way a migration underperforms, because those instructions are not neutral: they consume context, they constrain a model that no longer has the failing behaviour, and occasionally they actively cause a new problem — a rigid formatting instruction can suppress a capability the replacement has, and a warning against a habit the new model lacks can induce a strange compensation. Worse, they make it impossible to tell how good the replacement really is, since you are evaluating it while it wears a corrective lens ground for something else.
So use the eval set to test their removal. Take each compensating instruction, remove it, run the suite, and keep the removal if nothing regresses. Some will need to stay and now have evidence behind them. This is the only reliable opportunity to shrink a prompt that has only ever grown, and it usually pays for itself in cache-friendliness and clarity alone. Write down what you removed and why, so the next migration starts from a record rather than from archaeology.
The last deliverable is the one that makes the next deprecation cheap: a single place where the model identifier is configured, an eval suite that is a standing asset rather than a project artefact, downstream code that validates output against a schema rather than pattern-matching prose, and a note of which behaviours you had to defend. Deprecations recur, and the difference between a six-week scramble and a two-week task is entirely what you left behind last time.