You need to deploy a change to a system a running production line depends on. How is that different from a normal release?

Find out what an hour of downtime costs

The first difference is that the cost of failure is a specific number and somebody on site knows it. Ask for it. On an automotive line it can be tens of thousands per hour of lost throughput; in a continuous process a stop also scraps whatever is mid-process, and restarting may take longer than the stop itself because the process has to come back up to temperature or pressure. On a pharmaceutical line an interrupted batch may be unsaleable.

That number is not trivia. It determines whether the answer is "deploy carefully" or "do not deploy at all this quarter", and quoting it back shows you understand why the plant's release cadence is what it is rather than treating it as institutional slowness.

The window is the constraint, and change control is the gate

You do not choose when to deploy. Changes go in during planned maintenance windows, which may be weekly overnight, monthly, or tied to a shutdown that happens a few times a year. Some plants run continuously for months. The window is short, other teams are working in it, and if your change overruns you are the reason production did not restart on time.

Change control is the process that gets you into the window. Expect to submit what is changing, why, the risk assessment, the test evidence, the rollback plan, who is on site, and how you will verify success — and expect a review board including operations and controls to approve or defer it. Treating that as bureaucracy is the fastest way to lose the trust you need. It is also the mechanism that protects you, because a change that went through it and still caused a problem is a shared decision rather than your mistake.

In regulated production — pharmaceutical, medical devices, food, aerospace, some automotive — there is a further layer. Systems affecting product quality or the batch record operate in a validated state, and a change to them carries an assessment of what re-qualification or re-validation is required. That is measured in weeks of documented testing, not hours, which is why batching several changes into one validated release is the norm and why an urgent patch is genuinely hard rather than merely discouraged.

There is no staging environment that tells you the truth

You can stand up a copy of your services, a copy of the MES database, even a simulated PLC. What you cannot reproduce is the plant: the actual timing of the equipment, the specific firmware revisions on twelve stations, the operator behaviour, the shift pattern, and the physical process with its momentum and its thermal inertia. The dependency you cannot stub is reality.

So verification shifts shape. You test what you can offline, then you rely on techniques that limit exposure rather than on pre-deployment confidence. Deploy to one line or one cell first and leave it there for a full production cycle including a shift change, since shift changeover and start-of-shift are where most latent problems surface. Run the new path in shadow mode against live data, comparing its output to the incumbent's without acting on it. Put changes behind a flag so the code deploys in one window and activates in another, which decouples the risky moment from the constrained one. And write down beforehand what you will measure in the first hour to decide it is working — cycle time, reject rate, station wait time — because deciding that afterwards, at two in the morning with the line waiting, goes badly.

Rollback has to be genuinely instant, and that means a switch rather than a redeployment. If reverting involves rebuilding an artefact, re-running a migration or a twenty-minute restart, you do not have a rollback plan, you have an intention. Schema changes must be backwards compatible in both directions across a release, so the previous version can still read what the new one wrote. And you have to be able to say what happens to the units produced under the change if you revert: whether they are traceable, releasable, and correctly recorded.

The boundary you do not cross

Safety is not a stronger version of reliability and it is not your layer. Safety functions — emergency stops, guard and light-curtain interlocks, two-hand controls, overpressure and overtemperature protection — are implemented in dedicated safety systems: safety relays or a safety controller, wired and assessed under functional-safety practice, with a defined safe state and normally arranged to de-energise on loss of power or signal so that a failure stops the machine rather than leaving it running.

Those systems are deliberately separate from the control system, and both are separate from anything you build. Three rules follow, and volunteering them is worth real credit.

First, your software must never be the safety function. If a candidate proposes that an anomaly detector stop the machine when it sees a dangerous condition, that is the answer to reject: analytics may inform a human or adjust a setpoint, but the thing that guarantees the machine stops when a guard opens cannot depend on a service, a network, a broker or a model. Its guarantee comes from being simple, assessed and independent.

Second, your software must not be able to defeat a safety function. Any interface that could bypass an interlock, suppress a safety-related alarm, or write to a safety controller does not get built, and if one exists you raise it.

Third, your deployment must not perturb the safety chain, which in practice means understanding what is on which network and not assuming that because a change is "just software" it cannot reach the safety system. Someone on site owns that assessment. Ask them; do not conclude it yourself.

Availability, and what the target should actually be

The last difference is what availability means. In a web service the target is a success rate on requests, and redundancy is the standard answer. Here the number the plant cares about is line uptime, and your service's uptime only matters insofar as it affects it. Those come apart in both directions: forty minutes of downtime during a changeover between products may cost nothing, and the same forty minutes mid-batch scraps material and stops the line.

Which exposes the mistake most candidates make. Asked how to make this deployable, they design for their own service never being unavailable — active-active, health checks, multi-region — and never ask whether the line can run without them at all. But if a station synchronously calls your service before it may release a unit, then no amount of redundancy removes the fact that you are now in the critical path of production, and every future incident of yours is a stoppage.

The better goal is that your unavailability is not a line stoppage. That means the plant keeps a local ability to proceed: the station buffers its records and continues in a defined degraded mode, the PLC holds the last valid recipe and keeps running, the operator has a documented manual fallback, and your system reconciles the queued records when it returns. Where a genuine hard dependency is unavoidable — a quality gate that legally must block release — say so explicitly, make it as small and simple as possible, and give it the availability engineering it warrants rather than spreading that requirement across your whole stack.

Design so that your worst day costs the plant latency rather than production. That is a lower bar than never failing, and a much more useful one.