DevOps and cloud fundamentals

A container is a normal process on the host kernel, isolated by namespaces — which restrict what it can see — and limited by cgroups, which restrict what it can consume. A virtual machine runs its own kernel on virtualised hardware. That difference explains everything practical: containers start in milliseconds because there is no kernel to boot, they are far denser because they share one, and the isolation is weaker because a kernel vulnerability is shared by every container on the host.

Not a filesystem snapshot but a stack of read-only layers, each the diff produced by one build instruction.

FROM node:22-slim          # layer 1: the base
WORKDIR /app
COPY package*.json ./      # layer 2: two small files
RUN npm ci                 # layer 3: node_modules, ~200MB
COPY . .                   # layer 4: your source, ~2MB
CMD ["node", "server.js"]

Each layer is content-addressed, so identical layers are stored once and transferred once no matter how many images share them. A running container adds a thin writable layer on top, and everything written there disappears when the container does — which is why persistence needs a volume and why "it worked until we restarted" is a container-shaped bug.

The build cache follows the same structure: an instruction is only re-executed if it or anything before it changed. That is the reason the two COPY lines above are split. Copying package.json alone before npm ci means a source-only change invalidates layer 4 and reuses the cached layer 3, so the install is skipped entirely.

Reverse those lines — COPY . . before RUN npm ci — and every one-character source change invalidates the install layer, turning a fifteen-second build into a three-minute one. That single ordering decision is the most common Dockerfile review comment there is.

Build tools are needed to produce the artefact and are pure liability in the image that runs it.

FROM golang:1.23 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /app/server ./cmd/server

FROM gcr.io/distroless/static-debian12
COPY --from=build /app/server /server
USER nonroot:nonroot
ENTRYPOINT ["/server"]

The first stage has a compiler, a package manager, a shell and the full source tree. The second has one static binary and nothing else. Only the final stage becomes the image, so the result is a few megabytes rather than nine hundred.

Size is the least interesting benefit. The real one is attack surface: a distroless image has no shell, so an attacker who achieves execution inside it cannot run sh, cannot curl a second payload, and cannot use any of the standard tooling. Most container exploitation chains assume a shell exists.

It also cuts vulnerability noise dramatically. A scanner against a full base image reports hundreds of CVEs in packages your service never invokes, and that volume is what causes teams to stop reading scan results at all.

The trade is debuggability: you cannot exec into a shell that is not there. The answer is ephemeral debug containers that attach tooling to a running pod temporarily, rather than shipping the tooling permanently in production.

Because the container's root is the host's root in the relevant namespace unless user namespaces are configured, so a container escape starts from the strongest possible position. Running as an unprivileged user costs one line and removes an entire escalation step. The same reasoning covers dropping capabilities, mounting the root filesystem read-only, and refusing privileged mode — each removes a mechanism an attacker would otherwise have available for free.

Because a tag is a mutable pointer. myapp:1.4.2 can be repushed, and latest is repointed constantly, so the image you tested and the image that starts on a node an hour later may differ with nothing in your history recording it. A digest is the content hash: myapp@sha256:9f2c... is exactly one set of bytes forever. Tags remain useful for humans; deployments should pin the digest, which also makes a rollback a specific, reproducible artefact.

ENTRYPOINT declares the executable the container runs, and CMD supplies default arguments to it — so docker run image --flag appends to the entrypoint rather than replacing it. Using only CMD means any argument passed at run time replaces the whole command, which is convenient for a general-purpose image and wrong for a service, because a stray argument silently starts something else. The other detail is the exec form versus the shell form: the shell form wraps your process in /bin/sh -c, so your application is not PID 1 and does not receive SIGTERM, which quietly breaks graceful shutdown.

A bind mount maps a host path into the container, so it depends on that host's layout and is mostly a development convenience. A volume is managed by the runtime or the orchestrator, has a lifecycle independent of any container, and can be backed by network storage that follows a workload between nodes. In an orchestrated environment only the second is meaningful, because a pod rescheduled to another node has no relationship with the previous node's filesystem.

The image holds the application and its dependencies — everything identical across environments. Configuration, endpoints, credentials and feature settings come from the environment at start-up, because otherwise you build a different image per environment and have therefore tested none of the ones you ship. The practical test is whether the same digest can run in staging and production with only its environment differing; if it cannot, something environment-specific has been baked in.

Four layers, each solving one problem, and most confusion comes from attributing one layer's behaviour to another.

flowchart TD
    D[Deployment<br/>declares desired version and count] --> R[ReplicaSet<br/>one per version, keeps N pods alive]
    R --> P[Pod<br/>one or more containers sharing a network namespace]
    S[Service<br/>stable virtual IP and DNS name] --> P
    I[Ingress<br/>external HTTP routing by host and path] --> S

A Pod is the unit of scheduling, not the unit of deployment. It is mortal by design — rescheduled, replaced, given a new IP — which is why nothing should ever address a pod directly.

The Deployment owns the rollout, and it does so by creating a new ReplicaSet for the new version and scaling the old one down. That is why a rollback is fast: the previous ReplicaSet definition still exists and is simply scaled back up, with no rebuild involved.

The Service exists because pod IPs change. It provides a stable name and load balances across whichever pods currently match its label selector — and the selector is the coupling, so a label typo produces a Service pointing at nothing, which surfaces as connection refused rather than as a configuration error.

The Ingress is HTTP-layer routing on top, so one external address serves many services by host and path. Confusing Service and Ingress is common: a Service is L4 and cluster-internal by default, an Ingress is L7 and external.

They are separate numbers with completely different consequences, and setting them casually produces failures that look nothing like a configuration mistake.

resources:
  requests:            # what the scheduler reserves — decides placement
    cpu: "250m"
    memory: "256Mi"
  limits:              # the hard ceiling — decides what happens under pressure
    cpu: "1000m"
    memory: "512Mi"

Exceeding the CPU limit    -> throttled. The process is descheduled for part
                              of each period. Latency rises, nothing crashes,
                              nothing is logged. Hardest to diagnose.

Exceeding the memory limit -> OOMKilled immediately, exit code 137. No
                              graceful shutdown, no in-flight request drained,
                              and the restart looks like a crash bug.

Memory is incompressible, which is why the kernel's only option is to kill. CPU is compressible, so the response is to give you less of it — silently.

The scheduler only looks at requests, so requests far below actual usage cause overcommitted nodes: everything fits on paper, then real load arrives and pods are evicted. Requests far above usage waste most of the cluster.

Setting a CPU limit at all is genuinely contested. Throttling a service that would otherwise burst briefly and harmlessly can add tail latency for no protective benefit, so a common position is to set CPU requests and omit CPU limits, while always setting memory requests and limits to the same value. Whichever side you take, having a reason is what the question is for.

A ConfigMap holds non-confidential configuration as key-value pairs, injected as environment variables or mounted as files, which keeps configuration out of the image without a separate system. Secrets should not live in one because a ConfigMap is stored in plain text, is readable by anything with list permission on the namespace, and appears in full in kubectl describe output that people paste into tickets. The Secret type is barely better on its own — it is base64-encoded rather than encrypted — so the real answer is encryption at rest plus an external secret manager, with the cluster object holding only a reference.

Kubernetes creates a new ReplicaSet and shifts pods across it gradually, governed by maxSurge — how many extra pods may exist — and maxUnavailable — how many may be missing. During the overlap both versions serve traffic simultaneously, which is the fact that matters: your two versions must be mutually compatible, including their database schema and their API contract. A deploy that assumes only the new version is running will break for the duration of the rollout.

Because readiness controls traffic and liveness controls restarts, and getting traffic routing wrong affects users immediately. A pod that is running but not yet warm will receive requests and fail them unless readiness says otherwise. Liveness only matters for a process wedged beyond recovery, which is rarer than people assume — and a liveness probe checking dependencies restarts the entire fleet during any dependency blip, converting a wobble into an outage.

Workloads where instance identity matters: stable network names, storage bound to each ordinal, and ordered start-up and shutdown. Databases and consensus systems need it because instance three must come back as instance three, attached to its own volume, or the cluster cannot recover its membership. Everything stateless should be a Deployment, which is both simpler and faster to roll, because a StatefulSet updates pods one at a time in order by design. Reaching for one because a workload happens to write files usually means the files belonged in object storage, and the identity guarantee is being paid for without being used.

Mutual TLS, retries, timeouts, traffic splitting and per-call telemetry applied uniformly without changing application code — genuinely valuable across many services in many languages. The cost is a sidecar proxy per pod consuming memory and adding latency to every hop, plus a substantial new control plane to operate and debug. For a handful of services in one language, libraries deliver most of the benefit at a fraction of the operational weight.

Because it is a naming scope, and nothing more until you add three other things. Without network policies, pods in different namespaces reach each other freely, since the default in most clusters is a flat network. Without RBAC bound to the namespace, a service account is not constrained by living in it. And without resource quotas, one namespace can consume the cluster and evict everything else. Namespaces become a boundary only once those three are configured, and assuming the separation exists by default is a common and consequential mistake in multi-tenant clusters — particularly because the layout looks isolated in every diagram and dashboard.

The order is not arbitrary: each stage is placed to fail as early and as cheaply as possible.

1. Lint and typecheck        seconds     catches the trivial before anything
                                         expensive runs
2. Unit tests                1-2 min     the bulk of coverage, no I/O
3. Build once                2-5 min     ONE artefact, tagged by digest,
                                         reused by every later stage
4. Integration tests         5-10 min    real database, real broker, in
                                         containers
5. Security and dep scan     2 min       fails on new criticals only, or it
                                         is ignored within a week
6. Deploy to staging         2 min       the same digest from step 3
7. Smoke tests               1 min       a handful of real user journeys
8. Deploy to production      2 min       the same digest again, promoted
9. Post-deploy verification  5 min       error rate and latency against the
                                         previous version; auto-rollback

Step 3 is the pivot. Everything before it produces an artefact; everything after it promotes that exact artefact. Rebuilding for production means the tested bytes and the shipped bytes are different, and dependency resolution alone makes that a real difference rather than a theoretical one.

Step 9 is the one most pipelines lack, and it is what makes the rest safe. A deploy that completes is not a deploy that worked; comparing error rate and latency against the previous version for a few minutes, with an automatic rollback, is what turns a bad release into four minutes of degradation instead of however long it takes someone to notice.

The ordering principle throughout is cost against confidence. A thirty-second lint failure costs nothing; discovering the same class of error after a ten-minute integration suite has run costs a developer's context.

By computing which projects a commit actually affects and running only those, using the dependency graph rather than the changed paths alone — a change to a shared library must rebuild everything that imports it, which path filtering alone will miss. Content-addressed caching then skips any target whose inputs are unchanged, including across branches and machines. Without both, a monorepo's pipeline duration grows with the repository rather than with the change, and at that point people start batching commits to avoid waiting, which removes the fast feedback the pipeline existed to provide.

Because a suite that fails randomly trains everyone to re-run it, and once re-running is the reflex, a real failure gets re-run too. The signal is gone even though coverage looks unchanged. Quarantining the test immediately — removing it from the blocking path with a ticket and an owner — is better than leaving it in, because an unreliable gate is worse than an acknowledged gap.

Continuous delivery means every commit that passes the pipeline is deployable and reaching production is a decision someone makes. Continuous deployment removes the decision: passing the pipeline means it ships. The gap between them is entirely about confidence in the automated gates, so continuous deployment demands strong post-deploy verification and fast automatic rollback. Claiming the second while relying on a manual smoke test is the common misdescription.

Whatever you are prepared to enforce every time, which is a shorter list than most teams write down. Compilation, unit tests, and a security scan for new critical findings qualify. Coverage thresholds usually do not, because they get gamed with tests that assert nothing. The real question behind this is what happens when the gate fires at six in the evening on a release day — a gate routinely bypassed is not a gate, and it would be more honest to remove it.

Because merge pain grows superlinearly with branch age, and a branch alive for three weeks is a large, poorly understood change landing all at once. Small frequent merges keep every conflict trivial and keep the main branch continuously releasable, which is the precondition for continuous delivery. The objection — that unfinished work would ship — is answered by feature flags, which separate deploying code from releasing behaviour.

That it is deterministic, fast enough that people wait for it, and that a green result actually means something. A pipeline taking forty minutes gets bypassed; a pipeline that fails intermittently gets ignored; a pipeline whose tests do not exercise real integration points gives false confidence. Trust is the only property that matters, because every other benefit of automation depends on people acting on the result rather than routing around it.

Review, history and reproducibility for changes that were previously made by clicking. It means an infrastructure change goes through the same pull request as a code change, that you can see who altered a security group and why, and that a second environment can be built from the same definitions rather than approximated by hand. The discipline it demands is that nobody changes anything in the console, because the moment they do, the code no longer describes reality.

State is a file mapping your declared resources to the real identifiers the cloud provider assigned.

main.tf declares:            state records:
  aws_instance.api      ->     i-0abc123def456
  aws_db_instance.main  ->     prod-db-7f2a

Removing the resource block does not orphan it. It tells the next apply
"this exists and is no longer wanted", and the plan proposes destroying it.

Without state there is no way to know whether a declared resource needs creating or already exists, so state is what makes the tool declarative at all.

It is dangerous for three reasons. It contains secrets — database passwords and generated keys appear in plaintext — so it belongs in an encrypted remote backend, never in the repository. It must be locked, because two engineers applying simultaneously against the same state produce interleaved writes and a file that no longer matches reality. And losing it is worse than losing the infrastructure: the resources keep running and the tool no longer knows they exist, so the next apply tries to create duplicates of everything.

Drift is the related failure. Someone changes a rule in the console, the next plan proposes reverting it, and whoever runs that plan either destroys a fix applied during an incident or, more often, approves a change they did not read. Detecting drift on a schedule surfaces it while the reason is still remembered.

Because declaring the desired end state lets the tool compute the difference from whatever exists now, so running it twice is safe and running it against a partially built environment converges. An imperative script says "create this", which fails or duplicates on the second run, and encodes an assumption about the starting state that stops being true. Convergence is the property that makes infrastructure code re-runnable, and re-runnable is what makes it trustworthy.

Referenced, never embedded. The code names a secret in a manager and the value is resolved at apply or run time, so the repository holds a pointer and the audit trail lives in the secret store. Anything committed must be assumed permanently compromised, since rewriting history does not reach clones, forks or CI caches. The related habit is a pre-commit scan for credential patterns, because the common failure is accidental rather than deliberate.

That it makes the change reviewable before it is real. A plan showing three modifications is fine; the same plan showing a database being replaced is a different conversation, and replacement is exactly the kind of thing an innocuous attribute change can trigger. Posting the plan onto the pull request and requiring someone to read it is the control, and treating a plan output as noise to scroll past is how deletions get approved.

When the same pattern is deployed at least three times and its parameters are genuinely stable. Extracting after the first use produces an abstraction shaped by one case that immediately grows options for every subsequent one, and a module with twenty variables passing them straight through has not abstracted anything. The cost of a premature module is that every consumer is coupled to a shared change, which is precisely what makes infrastructure changes frightening.

Patching, backups, failover mechanics, minor-version upgrades and the on-call burden for the layer below your data. What it does not remove is your responsibility for schema design, query performance, capacity planning, cost, or the consequences of its failure modes. The trade is real: you pay more per unit of compute and get back engineering time, and the decision hinges on whether operating that component is something your team should be spending time on at all.

Serverless removes idle cost and introduces a latency distribution most people underestimate until it appears in their p99.

Warm invocation          5-30ms      the container is already running
Cold, small runtime      100-400ms   container start plus runtime init
Cold, JVM or .NET        1-6s        plus JIT warm-up and framework scan
Cold, inside a VPC       add 100ms+  network interface attachment

The distribution is what matters rather than the average. At steady traffic most invocations are warm and cold starts land in the tail — so the p50 looks excellent and the p99 is a second, and it is disproportionately the first user of a quiet period who experiences it.

It gets worse where you least want it: a traffic spike scales out by starting many new containers at once, so the cold-start rate peaks exactly when load does. And each concurrent request needs its own instance, since one instance handles one request at a time in most models.

The mitigations each cost something. Provisioned concurrency keeps instances warm and reintroduces idle cost, which is the thing serverless was chosen to avoid. Smaller runtimes and lazy initialisation help genuinely. Keeping functions out of a VPC where possible removes the network attachment penalty.

Where this decides architecture: serverless suits spiky, infrequent or event-driven work, and is a poor fit for a latency-sensitive synchronous path with steady traffic — where a always-running container is both faster and cheaper.

They protect against different failures, and conflating them produces a design that survives nothing it was meant to.

flowchart TD
    R1[Region eu-west-1] --> A[AZ a<br/>separate building and power]
    R1 --> B[AZ b]
    R1 --> C[AZ c]
    R2[Region us-east-1] --> D[AZ a]
    R2 --> E[AZ b]

Availability zones within a region are physically separate facilities connected by low-latency private links — typically low single-digit milliseconds. That latency is small enough for synchronous replication, which is why a multi-AZ database failover is automatic and loses nothing.

Regions are geographically distant. Cross-region latency is tens to hundreds of milliseconds, so replication is asynchronous, so a regional failover loses recent writes and requires a deliberate decision. Regions are also separate failure domains for control planes, which is what a regional outage actually takes down.

Multi-AZ is close to free and should be the default for anything that matters: spread instances across zones, run the database multi-AZ, and a building-level failure is a non-event.

Multi-region is expensive and demands answers about data residency, replication lag, split-brain and how you would actually route traffic during a failover. The question worth asking before proposing it is whether the business needs to survive a full regional outage, because most do not and the ones that do rarely have the budget they assumed.

Trading retrieval cost and latency against storage cost, based on how often data is read. Frequently accessed data sits in the standard class; data read a few times a year costs a fraction to store and more to retrieve; archival classes are cheaper again and may take hours to restore. Lifecycle rules move objects automatically as they age. The trap is putting frequently read data in an infrequent class, where retrieval charges exceed the storage saved.

Whatever most directly reflects the work queued rather than the resource consumed. CPU is the default and is often wrong: an I/O-bound service saturates on connections or concurrency while its CPU sits at fifteen per cent, so it never scales and merely gets slower. Request concurrency, queue depth or latency against a target track the real constraint. Whatever the signal, scaling out must be faster than the traffic ramp, or the policy is documentation rather than protection.

When traffic is steady rather than spiky, when latency is user-facing and tight, when the workload runs longer than the platform's limit, or when it needs persistent connections — a WebSocket server or a database connection pool sits awkwardly in a model with no durable process. It is also poor for anything needing local state between invocations. The clearest signal is a function that is really a long-running service with an awkward front door.

Everything the cloud actually offers. Moving a virtual machine unchanged gives you the same operational burden at a higher unit cost, because you now pay cloud prices for a self-managed database and a hand-built failover you still own. It is sometimes the right first step, to hit a data-centre deadline, but only if it is explicitly a step — the failure is treating the migration as finished once everything is running, which locks in the cost without the benefit.

A VPC is a private network you control, and subnets partition it — conventionally into public subnets with a route to an internet gateway and private subnets without one. The point is that databases and application servers live in private subnets and are unreachable from the internet regardless of any mistake made in a security group, while load balancers live in public subnets. That structural separation is worth more than any single rule, because it survives misconfiguration.

A security group is stateful and attached to an instance: allow inbound on 443 and the response leaves automatically, and there are no deny rules — everything not allowed is denied. A network ACL is stateless and attached to a subnet, so return traffic needs its own explicit rule, and it does support deny, which makes it the tool for blocking a specific address range. Security groups do the routine work; ACLs are a coarse second layer.

The difference between an intention and a policy is whether anything is actually constrained.

{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:PutObject"],
  "Resource": "arn:aws:s3:::acme-uploads/tenant/${aws:PrincipalTag/tenant}/*",
  "Condition": {
    "Bool": { "aws:SecureTransport": "true" }
  }
}

Compare it to what usually ships: "Action": "s3:*" on "Resource": "*". That grants deletion of every bucket in the account to a service whose job is to store a file, and it is written because it works immediately and narrowing it takes an afternoon.

Three things make the version above actually restrictive. The actions are enumerated, so a compromised service can read and write and cannot delete or change bucket policy. The resource is scoped to a path, and the tenant variable means one service's credentials cannot reach another tenant's data. And the condition rejects anything not over TLS.

The practical route to a policy like this is to start permissive in development with access logging on, then generate the policy from what was actually used — every major cloud has a tool for this now, and it produces a better result than reasoning about it.

The larger point is that the boundary is the credential, not the code. Reviewing application logic while the role attached to it can delete the production database means the review examined the wrong artefact.

Because a key is a bearer credential with no expiry: once leaked — into a repository, a log, a laptop backup — it works until someone notices and revokes it, which is typically months. A role issues short-lived credentials to an identity the platform verifies, so a leak has a lifetime measured in hours and rotation happens automatically. The same logic drives federated CI credentials, which replace a stored key in the build system with a token minted per run.

Ordinary TLS authenticates the server to the client and encrypts the channel. Mutual TLS additionally authenticates the client to the server with its own certificate, so a service can verify precisely which service is calling rather than accepting anything that reached the port. In a zero-trust internal network that is the mechanism that makes network position stop being an authorisation signal. The cost is certificate issuance, distribution and rotation for every workload, which is why it usually arrives with a mesh.

Because it is a deadline with no gradual warning: the certificate works perfectly until an exact timestamp and then everything fails at once, usually across every instance simultaneously. Manual renewal depends on somebody remembering a date up to a year away, and the person who set the reminder has often changed teams. Automated issuance and renewal removes the class of failure, and an alert on days-to-expiry catches the case where the automation itself has quietly stopped.

An SLI is the measurement — the proportion of requests served successfully under 300 milliseconds. An SLO is the target you hold yourselves to for that indicator, say 99.9% over thirty days. An SLA is a contract with a customer including a penalty, and it should always be looser than the SLO so that missing your internal target is a signal rather than a breach. Teams that only have an SLA discover problems when a customer invoices them.

The point of an SLO is not the number; it is that the remainder is a resource with an owner.

SLO 99.9% over 30 days  ->  budget 43m 12s of unavailability

Week 1  bad deploy, rolled back in 6m      spent  6m   remaining 37m
Week 2  dependency outage, 11m degraded    spent 17m   remaining 26m
Week 3  risky migration, 3m                spent 20m   remaining 23m
Week 4  budget largely intact              -> ship freely

While budget remains, spend it: ship the risky migration, run the load test against production, deploy on a Thursday. Reliability beyond the target is not free and is not being asked for.

When it is exhausted, feature work stops and reliability work starts until the window rolls. That rule is the entire value of the mechanism, because it settles the velocity-versus-stability argument with arithmetic instead of seniority.

The two failure modes are equally common. A budget consistently untouched means the target is too loose and you are over-investing in reliability nobody required. A budget exhausted every month means the target is unrealistic or the system genuinely needs work, and continuing to ship features through it means the SLO is decorative.

That it is specific and executable by someone who did not build the system. Symptom, the exact query or dashboard to check, the precise command to run, the expected output, and what to do when it is not that. Prose explaining architecture is not a runbook. The test is whether the newest person on the rota can follow it unaided, and the way to find out is to have them use it during a drill rather than during an incident.

Because the goal is an accurate account, and people do not volunteer accurate accounts when the outcome is attribution. The framing that works is that a system allowing one mistake to cause an outage has a systemic problem — the missing guardrail, the absent confirmation, the deploy that could not be rolled back — and those are fixable in a way that "be more careful" is not. Blameless does not mean consequence-free; it means analysing the mechanism rather than the person.

The role that matters most is incident commander, and it is deliberately not a technical role: that person coordinates, decides, and communicates, while resisting the urge to debug. Someone else investigates, someone else handles external communication, and someone keeps a timeline as events occur rather than reconstructing it later. The failure this structure prevents is five engineers independently changing things in production with no shared picture, which is how an incident acquires a second cause. On a small team one person may hold several roles, and saying aloud who is commanding is still worth the two seconds.

Alert volume high enough that the pager is ignored, which is the most dangerous because it looks like a working rotation. Too few people, so the same engineers are permanently tired. Alerts that are not actionable, training everyone to acknowledge and dismiss. And a rota without follow-up time, so the same incident recurs weekly because nobody was given the day to fix it. The health metric worth tracking is pages per shift, with a target low enough that each one is read.

Verifying that the resilience you designed exists, by injecting the failures you claim to survive — killing an instance, adding latency, severing a dependency — and observing whether the system behaves as documented. It is not breaking things randomly: it is a hypothesis, a bounded experiment, and a rollback. Its real value is usually organisational, because it converts "we have retries" from a belief into an observation, and the gap between the two is regularly large.

Two numbers define the requirement, and both are business decisions rather than technical ones.

RPO  Recovery Point Objective   how much data may be lost
     Set by backup frequency. Nightly backups mean an RPO of 24 hours.
     Continuous log shipping means seconds.

RTO  Recovery Time Objective    how long recovery may take
     Set by restore speed, which is almost never measured.

Worked: 4TB database, nightly snapshot, restore at 200MB/s
  restore time      4,000,000MB / 200MB/s  ~ 5.5 hours
  plus log replay                          ~ 1 hour
  plus verification and cutover            ~ 1 hour
  actual RTO                               ~ 7.5 hours

If the documented RTO is one hour, the plan does not exist — it is a wish.

The rehearsal is what turns this from an estimate into a fact, and it routinely surfaces problems no amount of planning would: the backup omitted a schema, the restore needs a credential nobody has, the target instance has insufficient disk, the runbook references a tool that was decommissioned.

The other thing rehearsing establishes is that the backup is readable. Silent corruption, a misconfigured retention policy that expired everything older than a week, and an encryption key nobody can find are all discovered at exactly the worst moment otherwise.

And replication is not a backup, however many replicas there are. A dropped table replicates in milliseconds. Point-in-time recovery is what protects against a mistake; replication protects against a machine.

The surprise is rarely compute, which is the only line most people estimate.

Typical mid-size workload, monthly

Compute, on-demand              $18,000    the line everyone plans for
Managed database                 $6,200    often oversized "for safety"
Data transfer out                $4,800    egress, per gigabyte
Cross-AZ traffic                 $2,100    chatty services in different zones
Storage, snapshots               $1,900    nobody deletes old snapshots
Load balancers, NAT gateways     $1,400    per-hour plus per-gigabyte
Logs and metrics ingestion       $3,600    verbose logging at debug level
Idle non-production              $2,700    staging running at 3am

Egress is the one that changes designs. Inbound data is generally free and outbound is charged per gigabyte, so a service that proxies large payloads through itself pays for every byte twice. Cross-AZ traffic is also charged in both directions, which is why two chatty services scattered across zones can cost more in transfer than in compute.

Log ingestion is the fastest-growing surprise, because the volume scales with traffic and with how much someone logged at debug level, and there is no natural brake until an invoice arrives.

The cheapest three interventions are almost always the same. Turn non-production off outside working hours, which is a scheduled job and roughly a two-thirds saving on those environments. Set retention on logs and snapshots. And right-size from actual utilisation rather than the number chosen at provisioning time, which was picked under uncertainty and never revisited.

When the commitment outlasts your confidence in the workload. A one-year commitment on a service you may re-architect in six months locks in spend for capacity you stop using, and the discount does not cover that. The reasonable approach is to commit only to your measured steady-state baseline and leave the variable portion on demand, which captures most of the saving with far less exposure to being wrong.

Interruptible work: batch processing, CI runners, rendering, data pipelines, anything that can checkpoint and resume. The discount is large and the capacity can be reclaimed with a couple of minutes' notice, so it suits workloads where an interruption costs time rather than correctness. Running a stateful production service on it is possible with enough engineering and is usually a way of spending more on complexity than the instances saved.

Because it brings a cluster to operate, upgrade and secure, a substantial learning curve, and a large surface of ways to be subtly misconfigured — costs that are justified by many services, several teams and genuine bin-packing needs. For three services and one team, a managed container platform delivers the same outcome with a fraction of the operational load. Naming the threshold at which the answer changes is what distinguishes a considered choice from a default.

Everything that assumed one source of truth. Writes must be routed to a single designated region or you need genuine conflict resolution, which is a data-model problem rather than an infrastructure one. Read-after-write stops holding, because replication lag is now tens of milliseconds at best and seconds under load. Failover needs a decision-maker, since automatic cross-region failover risks split-brain — two regions each believing they are primary. Data residency rules may prohibit replicating some records at all, which fragments the model. And the deployment pipeline must tolerate different versions running per region for the duration of a rollout, which multiplies the compatibility matrix you already had.

Because it names a category. Monitoring what, alerting on which threshold, paging whom, and with what runbook? A dashboard nobody watches during an incident is not monitoring, and an alert that fires nightly and is dismissed is worse than none — it has trained the team to ignore the channel. The specific version is "we alert on the p99 of the checkout endpoint exceeding 800ms for five minutes, which pages the owning team and links to this runbook."

"Walk me through the last time you rolled something back." It is impossible to answer convincingly from reading, because the interesting parts are the details: how the problem was noticed, how long the decision took, what the schema change did to the rollback, who was told, and what changed afterwards. A candidate who has done it talks about the awkward parts unprompted; one who has not describes the process as it appears in a diagram.