hardScenarioDesignSeniorStaffLead
A configuration change that passed review takes the entire fleet down in under a minute. What in your design should have limited the damage?
7 min readUpdated 2026-08-15Target archetype: Big Tech, Enterprise Captive, Product Startup
Answer
Short answer
No retry policy or circuit breaker helps here, because the change was valid and the defect was already deployed. What bounds the damage is treating configuration as a rollout with stages and bake time, a kill switch whose own dependencies cannot fail with it, and a measured rollback time.
Was this answer helpful?
Likely follow-ups
- Your kill switch is fetched from the same configuration service that just took the fleet down. What do you do instead?
- How would you stage a configuration change whose faulty path is only exercised by one customer's traffic?
- What is your rollback time for a change that is not a container image, and who has measured it end to end?
- How do cells or shuffle sharding change the arithmetic when one tenant's input can crash a shared process?
Related questions
- An upstream feed resets ten thousand of your prices to a penny and orders start arriving. What should have stopped it, and what do you do with the orders that got through?hardAlso on kill-switch6 min
- A Terraform plan in CI wants to destroy resources in production. What has to happen before anyone is allowed to approve it?hardAlso on blast-radius6 min
- You want to prove the system survives losing a database, in production, on a Tuesday afternoon. How do you run that without being fired?hardAlso on blast-radius7 min
- How would you put mandatory pre-trade risk checks on the order path when the desk will not accept the latency?hardAlso on kill-switch7 min