Your pipeline gates every code change. What else reaches production without passing through it?

The pipeline guards one input among several

Ask what gates a release and you get a confident answer: unit tests, integration tests, a browser suite, a security scan, a manual approval. Ask what can change production behaviour and the list is much longer, and most of it never touches that pipeline. The gap between those two lists is where a surprising share of serious incidents live, and the question is designed to find out whether you have ever gone looking for it.

The framing that makes the answer coherent is to stop thinking of the pipeline as protecting the codebase and start thinking of it as protecting production. Every route by which production behaviour changes is then a channel with its own speed and its own gates, and the interesting property is the mismatch: the channels with the weakest gates frequently have the widest and fastest reach, because that speed is why they were built.

The channels people forget

Configuration is the largest. Timeouts, thread and pool sizes, retry counts, rate limits, cache durations, connection strings, endpoint URLs, log levels: all of these change how the system behaves under load, and in many organisations they are edited in a console and applied within seconds. A pool size set to a wrong value can take a service down as thoroughly as a null dereference, and there is often no review, no test and no record.

Feature flags are configuration with the blast radius of a deployment. Enabling a flag activates code that was merged weeks ago and has not run in production, which means the flag flip is the real release and the deployment that carried the code was a non-event. Combinations make it worse, because the tested path is usually the default set and the interaction of two independently enabled flags is untested by construction.

Content and reference data are the channel most often overlooked, and the most consequential when the product interprets rather than merely displays what it is given. Pricing tables, tax rates, fraud rules, routing rules, email templates, entitlement definitions, translated strings, detection signatures. These are frequently published by a non-engineering team through a tool built specifically to avoid a release, and the assumption embedded in that design is that data cannot break software. That assumption holds only when the software treats the data defensively, and usually it does not.

Schema migrations run as part of a deployment but are not covered by the tests that gated it, because the tests ran against a database that had already been migrated in a fresh environment. The interesting risk is the migration executing against a large production table under live traffic, which no test in the suite simulated.

Infrastructure and platform changes — a load-balancer rule, a security group, a DNS record, a Kubernetes manifest, a node pool upgrade, a certificate rotation — change behaviour with no application build involved at all. Dependency updates arrive on somebody else's schedule and can change behaviour without changing your code. And a scheduled job, whose logic is code but whose trigger and parameters are configuration, will run unattended at 02:00 against production data with nobody watching.

Why the fastest channel is the least protected

These channels come to exist in a consistent way. Some class of change is needed urgently and the pipeline takes an hour, so a mechanism is built to move that class faster, and being faster is the point, so gates are omitted deliberately rather than forgotten. Then the mechanism is used for changes more consequential than the ones it was designed for, because it is the convenient path, and nobody revisits the original judgement.

The July 2024 CrowdStrike incident belongs in this conversation for exactly that reason. A content update crashed a Windows kernel driver and left machines around the world unable to boot, and the artefact that caused it was not application code: it travelled a rapid content channel that existed specifically to move faster than the release pipeline, so the gates the code would have passed did not apply to it. The question to turn on your own pipeline is which artefacts reach production without crossing a gate, and whether the ones that do are the ones you would least like to be wrong.

Gates proportionate to the channel

The wrong response is to force every change through the full pipeline, which destroys the speed the channel exists to provide and gets routed around within a month. Match the gate to what the channel can actually break.

For configuration, the highest-value gate is schema validation and range checking applied at the point of change rather than at the point of use, so a pool size of zero or a timeout of one millisecond is rejected before it is stored. Version the configuration, make changes reviewable and revertible, and record who changed what — most configuration incidents are diagnosed within minutes once you can see the last change, and take hours when you cannot.

For content and reference data, validate the structure and the plausibility of the values, then stage the rollout: a canary population first, with an automated check on the metrics that would move if it were wrong. None of this needs to take an hour, and it catches the error where the data is well-formed but wrong.

For feature flags, the useful gates are that the code behind the flag was exercised in a pre-production environment with the flag on, that enabling is progressive rather than global, and that flags have owners and expiry dates so the combination space does not grow without limit. Flags left on permanently are dead configuration that still costs you a branch in every test matrix.

For migrations, the gate is a rehearsal against a dataset of production scale, plus a stated rollback plan, plus the discipline of separating a schema change from the code change that depends on it so neither is blocked by the other's failure.

For infrastructure, definitions in version control with a plan step reviewed before it is applied gives you the review and revert properties the code already has.

Progressive rollout is the gate for what you cannot test

Some changes cannot be meaningfully tested before release, and honesty about that is worth more than a gate that pretends otherwise. A configuration value whose effect only appears under production load, a fraud rule whose accuracy depends on real traffic, a migration whose duration depends on real data volume: for these the answer is not a better pre-release test but a controlled exposure. Roll out to a small population, define in advance which metric would show it going wrong, and automate the abort.

The property that makes this work, and the one to state plainly, is that a change is safe in proportion to how quickly it can be undone. A gate reduces the chance of a bad change reaching production; a fast, rehearsed, always-available revert reduces the cost when one does. For channels moving faster than the pipeline, the second is usually the better investment, and any channel that can ship a change in seconds but takes an hour to reverse is the one to fix first.

How to find your own list

In an unfamiliar organisation, the way to build this inventory is not to ask people what they deploy. Take the last several incidents and identify what changed immediately before each one, because the answer is frequently something nobody would have described as a release. Then read the audit logs of every console with a write capability, and ask each team what they can change in production without raising a pull request. The list that comes back is longer than anyone expects, and it is the actual scope of quality assurance in that organisation.