hardDesignScenarioSeniorStaffLead
You need to reindex a ten-million-chunk corpus with a new embedding model. How do you do it without downtime?
6 min readUpdated 2026-08-15
Answer
Short answer
Build a shadow index beside the live one, dual-write from a recorded change-feed watermark so it never falls behind, verify on reconciliation counts plus a gold set and shadowed production queries, then cut over by moving an alias and keep the old index writable long enough to roll back.
Was this answer helpful?
Likely follow-ups
- The new model has different dimensionality and a different score distribution. What breaks besides the index?
- Halfway through the backfill you find a parsing bug in the new pipeline. Restart or repair in place?
- How would you cut over one tenant at a time, and what does that require of the query layer?
- Your gold set says the new index is better and shadowed traffic says it is worse. How do you resolve that?
Related questions
- A better embedding model has come out. What does moving to it cost you?hardAlso on embeddings and reindexing5 min
- You are moving a sharded datastore to Cassandra with no maintenance window. How do you cut over, and how do you know both copies agree before you do?hardAlso on zero-downtime6 min
- You need to migrate the schema of a 400-million-row table on a live system. Talk me through the plan, the backup you would want first, and how the connection pool affects it.hardAlso on zero-downtime7 min
- Your RAG system cannot answer a question whose answer is definitely in the documents. Where is the fault?hardAlso on embeddings4 min