hardScenarioDesignCase StudySeniorStaffLead
Your provider announces the model you depend on retires in six weeks. Walk me through what you do.
6 min readUpdated 2026-08-15
Answer
Short answer
Inventory everywhere the model is coupled to behaviour, then use the eval set as your migration harness, shadow the candidate on live traffic to compare without user exposure, and re-measure cost and latency because both shift.
Was this answer helpful?
Likely follow-ups
- You have no eval set on the day of the announcement. What do you do in the first week?
- How would you migrate a fine-tuned model whose base is being retired?
- The replacement is better on your evals but 40% slower to first token. How do you decide?
- What would you change in the architecture so the next deprecation is cheaper?
Related questions
- How do you write prompts that survive a model upgrade?mediumAlso on model-migration and evaluation4 min
- How do you ship a change to a production prompt safely?mediumAlso on llmops and evaluation4 min
- A wrong answer from the model would cost real money. How do you put it in the product anyway?hardAlso on evaluation5 min
- How would you test a model for bias?hardAlso on evaluation6 min