You want to prove the system survives losing a database, in production, on a Tuesday afternoon. How do you run that without being fired?
Turn it into an experiment: a steady state you can measure in under a minute, a blast radius you chose, an abort that is one command with a named person who can call it, and written approval from whoever owns the revenue. Tuesday afternoon is the right time because everyone who understands the system is awake, and the announcement goes to the channel your own alerts land in.
7 min readUpdated 2026-08-15
0
Answer
Short answer
A production database failover test should be run as a controlled experiment with a measurable steady state, written approval, a small blast radius, and a pre-tested abort command owned by one named person. Running it during staffed hours is safer than discovering the same untested path during a real overnight incident.
Was this answer helpful?
Likely follow-ups
→Your own on-call pages during the experiment for an alert you did not expect. Do you abort, and what does the alert tell you either way?
→The failover takes 20 seconds and users see errors for 50. Where did the extra 30 seconds come from?
→How do you run the same experiment for the first time on a system with no read replica and a four-hour restore?
→The error budget for the month is already spent. What do you do with the experiment you had scheduled?
You have a runbook. It says the replica is promoted in 30 seconds and the application reconnects. Somebody wrote that eighteen months ago and it has not been exercised since.
Ask what the odds are that it still holds. In eighteen months the schema changed, credentials rotated, the connection pool configuration was tuned by someone who has left, a new service started writing to the same database, and the promotion script's dependencies moved. The design document's availability figure was calculated assuming the failover works. Nothing has ever confirmed that assumption, so the number is a hope with arithmetic attached.
That is the argument to make to whoever has to approve this. You are not introducing risk. The risk is already there, sitting in an untested recovery path, and it will be discovered either on a Tuesday afternoon with everybody watching or at 3am during a real failure. Running the experiment does not create the exposure, it chooses when you find out.
An experiment needs a steady state with a number on it
The single thing that turns this from an outage into an experiment is a hypothesis you wrote down before touching anything, expressed as something you can read off a dashboard.
Not "the site stays up". That is unfalsifiable in the moment: everything is always partly up. Something like: checkout success rate stays above 99.5 per cent, p99 API latency stays below 800 milliseconds, and the queue of unprocessed orders stays under 200. Three numbers, all visible on one screen, all with a normal range you have looked at on an ordinary day.
Then apply the test that decides whether you are ready. Can you tell, within 60 seconds, whether those numbers have moved? If your dashboards aggregate on a five-minute window, you cannot, and you will spend the experiment guessing. That is a finding on its own, and it is worth fixing before you inject anything, because an experiment you cannot observe is a fault you cannot distinguish from a coincidence.
Write the hypothesis as a prediction with a direction: "we expect writes to fail for under 30 seconds, reads to continue uninterrupted from the replica, and checkout success rate to dip no lower than 97 per cent for no longer than a minute." Now the run has an outcome. If it matches, the claim is confirmed. If it does not, you have learned the specific thing you got wrong, which is the point.
Choose the blast radius, do not accept the one on offer
The question says "losing a database", and the temptation is to take that at face value on the first attempt. Do not. Pick the smallest version of the experiment that can still fail informatively, and escalate over several runs.
A reasonable ladder starts with a non-critical service's database in a staging-shaped copy of production traffic, then one shard of a sharded production store, then one availability zone's replica, then the production primary of something with a bounded audience, then the store you care about most. Each rung teaches you something. Each rung also costs more if it goes wrong.
The other dimension of blast radius is traffic. If you can route 1 per cent of sessions through a path that hits the failing dependency, you learn nearly everything for a hundredth of the exposure. That is worth building before the first experiment, because it converts every future experiment from a negotiation into a routine.
Loading diagram…
The transition worth arguing about is the one into Aborted, because an abort is a successful experiment rather than a failed one. If your process treats aborting as embarrassing, people will hesitate at the moment hesitation is most expensive.
The abort has to be one command and one named person
Write the abort condition before you start, in the same units as the steady state: if checkout success drops below 95 per cent, or if the dip lasts more than 90 seconds, we abort. Not "if it looks bad". Under stress, judgement is exactly the faculty that degrades, so the decision must have been made in advance by people who were calm.
Then make aborting cheap. One command, tested, with its output known. If reverting requires six steps across two systems, you will spend two minutes deciding whether to start, and those two minutes are the difference between a blip and an incident. Rehearse the abort itself before you rehearse the failure - the rollback path is part of the experiment, not a contingency outside it.
And name the person. One individual, present, with the authority to stop the experiment without consulting anyone, and everyone in the room knows who it is. Diffuse authority produces the specific failure where four engineers each assume another has already called it.
The announcement, and why 3am is the wrong answer
Tuesday afternoon is correct and the reasoning matters more than the conclusion. Everybody who understands this system is awake, at a keyboard, in the same channel, not on a plane and not asleep. If the experiment produces something unexpected, you have the full bench. Running it at 3am to reduce user impact optimises the wrong variable: it reduces the number of affected users and multiplies the consequence of anything going sideways, because you have one tired person with no context.
Announce it in advance, in the channel that receives your alerts, with a start time, an expected duration and the name of the person who can stop it. Unannounced chaos is indistinguishable from an incident, so your own on-call will page, escalate and convene a response over your test - which wastes their evening and teaches everyone to distrust the practice.
Suppress precisely the alerts you predict will fire, and leave everything else armed. This is not a detail. An alert you did not predict is the most valuable output the experiment can produce, and silencing the whole alerting stack for the window throws it away.
Above all of that sits the approval. That is what the phrase "without being fired" points at. The engineering risk is manageable with the controls above. The career risk is a function of one thing only: whether the person accountable for the revenue agreed in advance and in writing, with the blast radius and the abort condition stated. Get that, keep it, and the worst case becomes a decision that was made properly and turned out badly - which organisations survive. Do it informally and the same outcome becomes something you did to production.
The error budget is the other gate. If this month's budget is already consumed by real incidents, you do not run the experiment, and that is the budget working rather than caution winning. Deliberate risk-taking is affordable out of a surplus and not out of a deficit.
What the experiment usually finds, which is not the failover
Expect the promotion itself to work. It has a script, the script was written by someone competent, and databases are good at this now.
What breaks is adjacent, and it breaks because nobody modelled the interaction. A connection pool holds sockets to the demoted host and only validates them on a 30-second interval, so clients keep failing for up to half a minute after a 20-second failover, and the user-visible outage is 50 seconds against a documented 30. A cache warms itself from a hostname that still resolves to the old primary. A cron job with its own connection string writes happily to a demoted node for ten minutes. A service that was believed read-only turns out to write a heartbeat row.
Each of those is invisible in a design review and obvious within 90 seconds of a real failover. Which is the whole argument for doing this at all, and it is the answer to give when an interviewer asks what you expect to learn: not whether the database fails over, but what else was quietly depending on it not doing so.
An experiment is a fault with a hypothesis, a chosen blast radius, a rehearsed abort and a written approval. Remove any one of those four and what you are running is an outage you scheduled.