The MES is unreachable for four hours and the line keeps producing. What happens to the production record, and how do you reconcile afterwards?

Two questions that get answered as one

There are two separate problems in this outage and conflating them is the commonest way to get the answer wrong. The first is whether the line can keep making product, which is a question about what each station needs locally. The second is whether the resulting production record is valid, which is a question about what you can honestly assert after the fact. A design can succeed completely at the first and produce a record that quality will not accept, which means four hours of output sitting in a cage while somebody reconstructs it by hand.

The line itself is mostly indifferent. The controllers hold their logic and the recipe currently loaded, so the machines carry on. What the MES supplies is upstream of the machine and downstream of it: which work order this station is running, which unit is in front of the operator, whether that unit may proceed, and the record of what went into it. Only the third of those is a genuine hard dependency, and only sometimes.

flowchart TD
  A[Unit completes at station] --> B{MES reachable}
  B -- yes --> C[Event written and unit released]
  B -- no --> D[Event queued locally with source timestamp]
  D --> E{Release gate required}
  E -- no --> F[Unit released in degraded mode]
  E -- yes --> G[Unit held pending reconciliation]
  C --> H[Genealogy record complete]
  F --> H

The branch worth looking at is the one that does not rejoin. Everything queued eventually lands in the same record, except the units that needed a decision nobody could make, and those accumulate physically as well as logically.

The queue is the easy half

Store-and-forward at the station is not the interesting engineering. A durable local queue, timestamps applied where the event happened, a bounded size with a stated policy for overflow, and a rate-limited drain so nine stations reconnecting simultaneously do not flatten the MES. That part is ordinary.

What makes the outage expensive is that the queued events are not measurements, they are transitions in a state machine that the MES owns. A unit moves from queued to in-process to complete; a work order accumulates a produced quantity against a target; material is consumed against a lot balance. Replaying transitions into a state machine is nothing like appending points to a time series, because the machine's response depends on the order it receives them and on what it already believes.

So three properties have to be designed before the outage, not after. Each event needs a deterministic identity — station, unit, event type and source timestamp is usually enough — so that a retried drain is a no-op rather than a second unit. The MES side has to apply events by their source timestamp rather than by arrival, because otherwise a late-arriving "started" transition after an already-processed "completed" one either errors or, worse, resets the unit. And the local identifiers assigned during the outage need a defined relationship to the MES's own, since the station could not ask for a serial number from a system it could not reach. Pre-allocating blocks of identifiers to each station is the usual answer, and it is a decision you make months earlier.

The physical order is the authority

When the merged record and the plant disagree, the plant wins, and the plant's own evidence is the sequence in which units physically exist. Units sitting in a buffer between two stations are in an order that no amount of clock skew changes, and the parts bins tell you what was consumed whether or not anything recorded it.

This matters because your queued events almost certainly contain a contradiction. Two stations whose clocks differ by twenty seconds will produce interleaved timestamps that imply a unit reached station four before station three. Reconciliation that trusts the timestamps produces a routing violation; reconciliation that knows the routing can use it to order events whose timestamps disagree, and can flag the skew as the defect it is. Reconstructing counts against the physical inventory of finished units, work in progress and scrap is the check that catches the case where you silently lost events rather than delayed them.

What happened outside the software

Four hours of a plant not having its system does not mean four hours of nothing being recorded. It means operators wrote on paper, or on a whiteboard, or remembered. That data has to enter the record, and how you treat it is a judgement rather than a technical decision.

Keying it in as though the system had captured it is the option to refuse. The record then asserts a system-captured measurement that was in fact a manual entry, which is untrue on a document that may eventually be read by an auditor or a lawyer. The correct handling is that a manual entry is stored as a manual entry, with who entered it, when, and against what evidence, and remains distinguishable forever. That is more work and it is the difference between a defensible record and a tidy one.

The gate that cannot be deferred

Here is the thing that separates a considered answer. Most MES interactions can be made asynchronous, and candidates who spot that go on to make all of them asynchronous, including the release decision. But a gate exists precisely to stop a unit proceeding until a condition is confirmed. If the station cannot evaluate it, buffering the unit's "pass" and continuing does not defer the gate — it removes it, and it fabricates a confirmation that was never made.

So gates get classified deliberately, in advance, with quality rather than by you. Most are advisory and can be recorded late. A few are genuine blocks: a safety-critical torque, a leak test on a pressure part, a check a regulator requires before release. Those units keep being produced if the process cannot be stopped cleanly, and they are held — physically quarantined and logically flagged as unreleasable — until the check is performed or the unit is scrapped. The volume of held units per hour of outage is a number worth calculating during design, because it converts an availability target into cage space and rework hours, which is a language the plant already speaks.

An outage you handled is one where the line kept running and the record still says only true things about it. Those are two separate pieces of work and the second is the one that gets skipped.