Versioning prompts as artefacts, canarying a model change, tracing and cost observability, and surviving a provider deprecation.
3 questions in Generative AI & Large Language Models.
Treat the prompt as a versioned artefact under review, run your eval set in CI against a pinned model version, canary on a slice of traffic with the version recorded per request, and keep rollback a config change rather than a deploy. Pinning is what makes a regression attributable to your edit.
Trace every request with prompt and model versions, token counts and cost, then add sampled groundedness grading, refusal and retry rates, validation failures and implicit user signals. Alert on countable rates, because an averaged quality score moves too little and too late to page anyone.
Inventory everywhere the model is coupled to behaviour, then use the eval set as your migration harness, shadow the candidate on live traffic to compare without user exposure, and re-measure cost and latency because both shift. Use the migration to delete instructions that only compensated for the old model.