You have just joined and you are asked whether the company could restore production if the primary database were destroyed right now. How do you find out?

A backup is a claim; a restore is the evidence

The only honest answer to "can we recover" is the result of a recovery you have performed. Everything else - a green backup job, a configured snapshot schedule, a replica with low lag, a runbook someone wrote - is a statement of intent that has not been tested against reality. So the shape of the investigation is fixed from the outset: agree the targets, inventory the mechanisms, then restore from each one into somewhere you cannot damage, and time it.

Say that plainly and early, because a large part of what this question scores is whether you audit configuration or produce proof. An answer that lists the backup types available in the platform has not begun.

Agree the targets before judging the mechanisms

Two numbers turn the question from a vague worry into an engineering task. The recovery point objective is how much recent data the business can afford to lose, and it dictates the frequency of whatever you capture and whether you need continuous archiving rather than periodic dumps. The recovery time objective is how long you may take to be serving again, and it dictates the restore mechanism, because the fastest path back is rarely the same as the cheapest way to store data.

These have to come from the business rather than from you, and they are usually not uniform. Losing an hour of analytics events is an inconvenience; losing an hour of payments is a legal problem. Ask which datasets carry which target, and expect to discover that nobody has written it down, which is itself a finding worth reporting.

Inventory the mechanisms and test each one on its own

List everything that could be a source of recovery and treat each as a separate experiment, because they fail independently and a single working one is enough. Typically you will find some subset of logical dumps, physical or block-level snapshots of the storage, continuous archiving of the write-ahead log giving point-in-time recovery, streaming replicas, and object-storage copies of any of the above.

For each one, answer four questions with observations rather than beliefs. Where does the output actually land, and can you list the most recent object with its timestamp. Can you read it from a host that is not the one that produced it. Does restoring it produce a working system, and what version of the engine does the restore require. And how long does the whole path take from decision to serving traffic, including the parts nobody counts, like transferring hundreds of gigabytes out of object storage or rebuilding indexes and caches afterwards.

The most common thing this exercise uncovers is not an absent backup but a backup whose write path has been quietly failing. A dump command that exits non-zero into a scheduled job nobody reads, an object-storage prefix that changed, a credential that expired, a tool that no longer matches the engine version and refuses to run. All of these leave the schedule looking healthy, because the schedule is monitored for having run rather than for having produced a usable artefact.

Replication is not a backup, and the distinction is not pedantic

Replicas exist to survive a host failing. They do not protect you from anything that is faithfully replicated, which includes a destructive statement, a bad migration, an application bug that corrupts rows and an operator deleting a table. The replica applies your mistake with admirable speed. Delayed replicas partially cover this by holding a configurable lag behind the primary, which buys you a window in which the damage has not yet arrived, but the window is finite and it protects only the interval you chose.

Point-in-time recovery is the mechanism that genuinely covers logical damage, because it lets you restore a base copy and replay changes up to a moment just before the harm. It is also the mechanism most likely to be misconfigured without anyone noticing, since it only breaks when you need it. If you test one thing in your first week, test that a base copy plus archived logs can be replayed to an arbitrary timestamp you choose on the day.

flowchart TD
    A[Loss event] --> B{Host or storage failure}
    B -- yes --> C[Promote replica]
    B -- no --> D{Damage is logical and replicated}
    D -- yes --> E[Base copy plus log replay to a chosen time]
    D -- no --> F{Whole estate or one tenant}
    F -- one tenant --> G[Selective extract from a restored copy]
    F -- whole estate --> H[Full restore and timed rehearsal]

The branch that catches teams out is the third one, because most tooling is built for the bottom-right box and the request that arrives is the one above it.

When five mechanisms are each unusable

In January 2017 a GitLab.com engineer working a replication problem late at night ran a directory removal against the primary database host rather than the intended secondary. The recovery then revealed that five separate backup or replication mechanisms were each unusable, and the restore ultimately came from an older snapshot that had been taken for another purpose entirely. One of the dump paths had been failing because the tool no longer matched the database engine, and the notification that should have reported it never reached anyone.

Read that as a monitoring failure rather than a backup failure. Five mechanisms is not carelessness; it is more redundancy than most organisations have. What was missing was any signal that would go red while things were calm, because every check confirmed that a mechanism was configured and none confirmed that it had produced something restorable. The transferable rule is that a backup system needs a test that fails loudly in the absence of an incident, and the only test with that property is a restore.

The second reading is about the human in the story. An engineer with production credentials operating on the wrong host at the end of a long night is not a root cause you can fix by asking people to be careful. The fixes are structural: make the host you are on visibly distinguishable, remove the ability to destroy data from the accounts used for routine work, and require a second pair of eyes on anything irreversible during fatigue hours.

What to build once you know the answer

Turn the rehearsal into a scheduled job rather than an event. A restore pipeline that runs on a fixed cadence, provisions a throwaway environment, restores the most recent artefacts, replays logs to a random timestamp, and then runs assertions against the result gives you a continuously refreshed answer to the original question. The assertions matter as much as the restore: check row counts against expectations, check that recent known records exist, check referential integrity on the tables you care about, and publish the elapsed time so drift against the recovery time objective becomes visible before an incident reveals it.

Then protect the artefacts from the blast radius that would make you need them. Backups should live in a different account or subscription with different credentials, be immutable or versioned for a retention period so that a deletion cannot be propagated, and be verifiable by checksum. Anything reachable and destroyable by the same identity that runs production is sharing a failure mode with production.

Nobody can tell you whether recovery works; only a restore can. Establish the recovery point and time targets, test each mechanism into an isolated environment on a schedule, and remember that the usual defect is not a missing backup but a monitored job that reports success while producing nothing you could restore.