Some of what you ship - rules files, config bundles, model updates - reaches production without passing through the deploy pipeline. What would you put around that path?
6 min readUpdated 2026-08-15Target archetype: Big Tech, Enterprise Captive, Product Startup
0
Answer
Short answer
Treat any payload the fleet loads as a release, because it changes behaviour in production exactly as a binary does.It needs schema and replay validation, a canary ring with its own health comparison, and a rollback that survives the change breaking whatever would apply the rollback.
Was this answer helpful?
Likely follow-ups
→Your config rollout is gated on error rate, but the payload is only read at process start and hosts restart at random. How do you get a clean comparison?
→The security team needs a detection rule live within the hour. What are you willing to skip, and what stays mandatory even at that speed?
→How would you roll back a payload that stops the agent responsible for fetching payloads from starting?
→A feature flag and a config file both change behaviour without a deploy. Should they share the same rollout machinery, and what differs?
A pipeline exists to stop untested change reaching production, and almost every organisation eventually grows a second channel that reaches production faster. It carries detection rules, pricing tables, feature-flag values, allow-lists, routing weights, machine-learning models, tax rates, localisation bundles. Nobody calls these releases, because they are not code and there is no build. But the definition of a release that matters operationally is anything that changes what production does, and every one of these does exactly that.
Once you accept that framing the design question becomes tractable. The fast channel does not need to be abolished; it needs the same four properties the code pipeline has, implemented in a form appropriate to a payload rather than to an artefact. Something must validate the change before it goes out. It must reach a small population first. There must be an observable signal saying whether that population is healthier or sicker than the rest. And there must be a way back that does not depend on the change having worked.
Why the fast path exists, and what it is entitled to skip
Argue for the fast path before you constrain it, because someone who wants every config change to go through a forty-minute build has not understood why the channel was created. The legitimate reason is that some changes are urgent in a way code is not: a detection rule for an exploit being used right now, a rate limit against a scraper, a kill switch on a partner integration that is corrupting data. If it takes an hour to disable a broken integration, the safety mechanism has become the outage.
So the honest split is that the fast channel is entitled to skip the build, the artefact promotion and the human approval queue, because those cost time and buy little for a payload nobody compiles. It is not entitled to skip validation, staged exposure or an abort path, because those are precisely what keep a fast change from becoming a fast global failure. Speed should come from removing queueing, not from removing verification.
Validating a payload no compiler will ever see
Code gets a type checker and a test suite for free. A config bundle gets nothing unless you build it, so the first thing to specify is a schema and a checker that runs inside the same channel the payload travels through. Schema validation catches the missing key and the wrong type; semantic validation catches the plausible-looking value that is wrong, such as a rate limit of zero or a rule whose match expression matches everything.
The check that earns its keep, though, is running the payload through the real consumer rather than through a validator that reimplements the consumer's parsing. A separate validator drifts from the code that actually loads the file, and the divergence is exactly where the interesting failures live. Load the candidate payload into an instance of the production binary in a sandbox, exercise it against recorded traffic, and assert that it neither rejects the payload nor behaves absurdly with it. That is the only check that tests the pair rather than the file.
Staging exposure with no artefact to canary
Progressive delivery for config needs a population you can name and a signal you can read. The population is easier than people expect: label a subset of hosts, regions or tenants as the early ring, publish to that ring first, and hold. Rings should be ordered by how much you can learn against how little you can lose, so internal fleets first, then a single zone, then a single region, then everything. The discipline that matters is that the boundaries are enforced by the distribution mechanism rather than by a human remembering the order at two in the morning.
The signal is the harder half, and it is where an answer separates. A config payload usually changes behaviour without changing request volume, so aggregate error rate may not move at all. You want the metric closest to what the payload governs: match counts for a detection rule, decision latency for a policy engine, parse and reload failures for anything read at startup, and process restarts as a blunt catch-all. Whatever you pick, the rollout controller has to compare the ring against the rest of the fleet at the same moment, because comparing against yesterday folds in every other change that happened since.
Loading diagram…
The edge worth looking at is that the revert arrow leaves the first ring rather than the last. Exposure is bounded by where the comparison happens, so a rollout with rings but no per-ring comparison is just a slow global push.
When the change breaks the thing that would undo it
The failure mode that catches good engineers is a change whose blast radius includes its own recovery path. Most rollbacks assume a running host, a running agent and a working network, because that is how a code rollback works: you shift traffic, or you tell the orchestrator to reconcile to the previous version, and something on the far end obeys. A payload that stops the host booting, stops the agent starting, or breaks the network configuration removes the obedient far end. Recovery becomes physical or manual, and its cost then scales with the number of machines rather than with the size of the change.
In July 2024 a CrowdStrike content update, delivered on the rapid content channel rather than as a staged sensor release, crashed a Windows kernel driver and left machines around the world failing to boot, each one needing hands-on intervention to recover. Two properties of that combination are what make it worth studying rather than merely deploring. The payload was consumed in a context where a fault was fatal to the whole host, and it travelled on the channel built for speed on the assumption that content could not do that. Anything loaded into the kernel, the boot path, the network stack or the agent that fetches updates belongs in a stricter class than the rest of your configuration, however much it looks like data.
For that class the design changes shape. Exposure has to be staged even when the change is urgent, because there is no remote undo to fall back on. The consumer should refuse a payload it cannot parse and carry on with the last known-good one rather than failing hard, which turns a fatal update into a merely stale one. And where the platform allows it, the fallback should be automatic and local: if the new payload has been loaded and the process has died twice, revert on the host without waiting to be told, because the instruction to revert may never arrive.
Pointers rather than pushes
Prefer a model where the fleet resolves a pointer to an immutable, digest-addressed payload over one where a publisher pushes bytes at hosts. Immutability means the thing you validated is the thing that loaded, and every consumer can report which digest it is running, so you can answer "what is live" without trusting a deployment log. Rollback is then moving the pointer back to a digest that is still present, which is fast, idempotent and safe to repeat. Pushes are harder to reason about because a partial push leaves the fleet in a state no version number describes. The corollary is retention: a rollback target that has been garbage-collected is not a rollback target.
Anything production loads and acts on is a release, whatever it is made of. Give the fast channel schema and replay validation, ordered rings with a per-ring comparison, and an immutable digest to fall back to. Then treat payloads that can stop a host or its agent as a class of their own, because for those the remote rollback you were counting on no longer exists.