A better embedding model has come out. What does moving to it cost you?

Two models produce two incompatible spaces

The first thing to establish is that this is not an upgrade in the sense that a library version bump is. An embedding model defines a space; distance is only meaningful between vectors the same model produced. Change the model and every stored vector becomes an artefact of a coordinate system nothing else in the system uses. Cosine similarity between an old document vector and a new query vector still returns a number, which is precisely the danger — nothing throws, the results are simply nonsense that looks like search results.

This is true even when the dimensionality matches. Two models producing 1,024-dimensional vectors have no reason to agree on what any dimension means, and there is no transformation you can apply to convert one to the other. So there is no incremental migration, no lazy re-embedding on read, and no mixing. Every chunk in the corpus has to be embedded again, and the index has to be built again from those vectors.

What the bill is made of

The compute or API spend on re-embedding the whole corpus is the visible line item and often the smaller one. Derive it in front of the interviewer rather than asserting a figure: chunk count times average tokens per chunk gives total tokens, and that times the unit price gives the cost. The useful part of doing it out loud is that it shows the size of corpus at which the answer changes, and it exposes throughput as the real constraint — a rate-limited hosted model over millions of chunks is measured in days, and that duration is what forces the plan below.

Then the costs that people forget. You need the index built again, which for a graph index is its own significant build. You need to store both sets of vectors for the overlap period, so peak storage and memory are roughly double, and if your index is memory-resident that may exceed what a node can hold and force a temporary scale-out. You need the evaluation work, because an upgrade you cannot measure is a change of unknown sign. And you need engineering time on the machinery for running two indexes at once, which is the part that does not exist yet in most systems and is the reason the second migration is far cheaper than the first.

Run the new index in the shadow of the old one

The safe shape is a shadow index: keep serving from the current index while you build the new one, then compare, then cut over behind a flag with the old index still intact.

flowchart LR
    A[Ingestion pipeline] --> B[Embed with model v1]
    A --> C[Embed with model v2]
    B --> D[Index v1 serving]
    C --> E[Index v2 shadow]
    F[Query] --> G{Router by index version}
    G --> D
    G --> E
    E --> H[Gold-set comparison and cutover gate]

What to look for in that shape is the router: it must pick a single index per request and use that index's matching query model, because a query embedded with v2 against documents embedded with v1 is the failure this whole arrangement exists to prevent. Pinning both sides of a request to one version is the invariant, and stamping the model version onto the index and refusing a query that does not match it is worth the few lines it takes.

Dual-writing new ingestion into both indexes during the backfill is what keeps the shadow current, otherwise the new index is stale by exactly the length of the backfill and your comparison is measuring the wrong thing. Once the shadow is complete and dual-written, you can send mirrored read traffic to it and compare results without exposing them, which gives you the comparison on real queries rather than on the gold set alone.

The cutover gate has to be a measurement, and the only honest one is your own gold set evaluated against both indexes: recall at k per index, and end-answer quality on the same questions. A newer model with a better published benchmark can be worse on your corpus, which is the outcome most worth catching before the old index is deleted. Keep the old index alive after the cutover for as long as it takes to be confident, because the rollback is only a router flip while it exists and a multi-day rebuild afterwards.

The constants that quietly become wrong

The part that is missed even by teams who plan the reindex properly is everything downstream that was calibrated against the old score distribution. Absolute similarity scores are not comparable between models, so a relevance threshold — the cut-off below which you decline to answer rather than pass weak context to the generator — means something different the moment the model changes, and a threshold that used to filter noise may now discard good passages or admit rubbish. The same applies to a minimum-score guard on a reranker's input, to any alerting on mean top-1 similarity, and to a deduplication rule expressed as a distance.

Two other places to check. Cached query embeddings are invalidated wholesale, so the cache must be keyed by model version or cleared, and a cache keyed only by query text will serve v1 vectors to a v2 index indefinitely. And if the new model has a different context limit or a different tokeniser, your chunk sizes are no longer what you tuned them to be, which turns a re-embed into a re-chunk and moves the whole exercise back a stage.

When to decline

The answer is not always yes. The upgrade is worth it when your evaluation shows the failing queries are ones a better representation would fix — vague phrasing, paraphrase, domain vocabulary. If your failure analysis says the misses are part numbers, dates and negations, no embedding model addresses those, and the same effort spent on hybrid retrieval, metadata filters or a reranker will move the number further for a fraction of the cost. Knowing that before spending days of compute is the judgement being scored.

There is no in-place swap. Plan for two complete indexes coexisting behind a version-pinned router, gate the cutover on your own gold set, and go through every tuned threshold before you delete the old one.