How would you design a deployment pipeline that can be rolled back safely?

Rollback is a property you design in, not a procedure you write down

The useful framing is that rollback is not an operation you perform after a bad release; it is a guarantee the pipeline either preserves or destroys at every stage. A pipeline preserves it when the previous version is still runnable, still has a live compatible datastore, and can be made to serve traffic without a build. Every design choice below either protects that or quietly removes it, and the choices that remove it usually look like conveniences at the time.

So the target is a testable claim: at any moment, restoring the previous release must be a traffic change or a config change, never a rebuild, never a data restore, and never a decision that needs an architect on the call.

Ordering the quality gates by cost and by what they can still catch

Gates should be ordered so the cheapest, fastest, most deterministic checks fail first, because the value of a gate is the feedback latency multiplied by the class of defect only it can catch. On commit, run compilation, unit tests, static analysis and dependency vulnerability scanning; these are seconds to low minutes and need no environment. Then build the artefact once and run integration and contract tests against it with real dependencies in containers.

After that the gates change character. Deployment to a staging environment lets you run migration rehearsal, smoke tests and any end-to-end suite you actually trust. Performance and security testing sit here too, but treat them as gates only where you have a stable baseline; a flaky load test that blocks deploys teaches the team to bypass gates, which is a worse outcome than not having the gate. Production gates are then progressive: deploy, observe error rate and latency against the previous version, and let a health check rather than a human decide whether to continue.

The judgement being scored is that you can defend the order: contract tests before end-to-end tests because they localise the failure, vulnerability scanning of the built image rather than the source tree because that is what ships, and manual approval — if it exists at all — immediately before the production traffic shift and nowhere earlier.

One artefact, promoted rather than rebuilt

Build the deployable exactly once, tag it with the commit SHA, and promote that identical, immutable artefact through every environment. Never rebuild per environment. A rebuild reintroduces every non-determinism you thought you had eliminated: a floating base image, a transitive dependency that resolved differently, a different builder version. If dev, staging and production each get their own build, then the thing you tested is provably not the thing you shipped, and worse, the previous version you want to roll back to may no longer be reproducible at all.

Environment-specific behaviour therefore has to come from outside the artefact: configuration injected at deploy time, secrets from a vault, endpoints from service discovery. The practical test is whether you can deploy the artefact currently in production to staging and have it work. If you cannot, something environment-specific is baked in, and your rollback target is a hypothesis rather than a known-good binary.

Blue-green, canary and rolling, and why the choice is about blast radius

Rolling replaces instances in batches. It is the Kubernetes default, needs no extra capacity, and is fine for low-risk changes, but both versions serve traffic during the roll and rollback means another roll, so recovery time is roughly deploy time. It gives the weakest control over blast radius.

Blue-green stands up a complete second environment, verifies it while it takes no production traffic, then switches the router. Its virtue is that rollback is a single routing change measured in seconds, which is exactly the guarantee we want. Its costs are doubled capacity for the duration and the fact that the switch is all-or-nothing, so a defect that only appears under real traffic hits every user at once.

Canary sends a small traffic slice to the new version, compares its error rate and latency against the old, and progresses or aborts on that evidence. It has the smallest blast radius and is the only one of the three that discovers production-only defects before full exposure, but it demands per-version metrics, enough traffic for the comparison to be meaningful, and tolerance for two versions running concurrently for a long time.

If I had to defend one choice: canary for user-facing services with high traffic, because bounded exposure plus automated abort beats fast recovery from full exposure. Blue-green for lower-traffic services and for anything where a canary comparison would be noise, because a seconds-long rollback is worth the capacity. Rolling only where the change is genuinely low risk. The trade-off to state out loud is that canary buys smaller blast radius by paying in operational complexity and version-skew tolerance, and version skew is precisely where the next section bites.

The trap: the database is what makes rollback impossible

Almost every candidate describes a clean deployment strategy and then treats the schema as if it moves with the code. It does not. Application instances are disposable and the database is not, so a migration that drops a column, renames it, tightens a constraint or rewrites data is a one-way door: the old binary can no longer run against the new schema, and reversing the migration either loses data or is simply not defined. At that point your elegant blue-green switch is decorative.

The answer is to make every schema change backward-compatible for at least one release, using expand-contract (also called parallel change). Expand: add the new structure additively while the old one still works. Migrate: have the application write to both and backfill existing rows, then move reads to the new structure. Contract: only once no running or rollback-target version depends on the old structure, remove it — a separate deployment, typically a release or more later.

-- Release N: EXPAND. Additive only, nullable, no constraint yet.
-- The previous binary is unaware of this column and keeps working.
ALTER TABLE customer ADD COLUMN email_normalised text;

-- Release N: app writes both `email` and `email_normalised`; reads `email`.
-- Release N+1: reads switch to `email_normalised`. Rollback to N is still safe.
-- Release N+2: CONTRACT, and only now, because no rollback target reads it.
ALTER TABLE customer DROP COLUMN email;

The second half of the answer is decoupling migration from deploy. Run migrations as their own pipeline step with its own approval, not as an init container or an application-startup hook that ties schema state to the rollout. That separation means a code rollback does not attempt a schema rollback, backfills of large tables can run in throttled batches without blocking a deploy, and a failed migration is diagnosed as a migration failure instead of presenting as a crash-looping pod.

Note that the migration branches away from the deploy rather than sitting inside it, so shifting traffic back never implies undoing a schema change.

flowchart LR
    A[Commit gates] --> B[Build artefact once]
    B --> C[Integration and contract tests]
    C --> D[Staging deploy and migration rehearsal]
    D --> E[Expand migration - own step and approval]
    D --> F[Deploy same artefact as canary]
    F -->|health signals good| G[Full rollout]
    F -->|health signals bad| H[Shift traffic to previous artefact]
    E -.-> I[Contract migration a release later]

Feature flags: the rollback that needs no redeploy

Flags separate deploying code from releasing behaviour, and that separation gives you the fastest recovery mechanism available. Ship the new path dark, enable it for internal users, then a percentage, and if it misbehaves flip the flag off. Recovery is a config propagation, seconds, with no build, no traffic shift and no capacity churn — and critically it is reversible per-feature rather than per-release, so you do not withdraw four good changes to undo one bad one.

Be honest about the costs. Every flag doubles a code path and neither branch is exercised by default, so tests must cover both, and flags accumulate into conditional sprawl unless each is created with an owner and a removal date. A flag also protects only the code it wraps: it cannot undo a schema contraction or an event already published, which is why expand-contract remains the foundation rather than an alternative.

Deployment strategy determines how fast you can roll back; schema compatibility determines whether you can roll back at all. Get expand-contract and migration decoupling right first, then pick between canary and blue-green on blast radius.