Backup and recovery, replication and failover, upgrades, capacity planning, and incident response.
4 questions in Databases & SQL.
A backup job that exits zero is a belief, not a control. Establishing recoverability means agreeing an RPO and RTO, then rehearsing a restore onto clean infrastructure, measuring how long it took and verifying the result — including everything the dump does not contain.
Replicate from the source's change log rather than dual-writing from application code, backfill with each row's original write timestamp so history can never overwrite live data, then run continuous range checksums and shadow reads until divergence is understood rather than merely small. The read flip and the write flip are separate steps, each reversible.
Changing a shard key moves rows between shards, so it is a copy-and-delete across nodes rather than an UPDATE. Put a bucket-to-shard directory in front of the data, move one bucket at a time behind a freeze measured in milliseconds, verify each bucket before flipping its ownership, and keep every bucket independently reversible.
Every step is chosen to hold weak locks briefly - add the constraint invalid and validate separately, index concurrently, backfill in committed batches - behind a restore you have rehearsed and timed, and a pool small enough that one blocked DDL statement does not exhaust it.