The line has been stopped for twenty minutes and everyone in the room believes it is your system. How do you run the next hour?

The room is not hostile, it is expensive

Twenty minutes of a stopped line has a cost somebody in that room can quote, and it is accruing while you talk. The atmosphere follows from that arithmetic rather than from anyone's opinion of you. Understanding this changes how you behave, because the useful contribution in the first ten minutes is not an explanation, it is a reduction in uncertainty about what to do next.

It also changes who is in charge. You do not decide when the line restarts. A shift supervisor or production manager owns that, with the controls engineer advising on the equipment. Your role is to supply one input to their decision — whether your component is implicated and whether it can be taken out of the way — and to supply it fast enough to be useful.

Answer the attribution question first, from the caller's side

The single most valuable thing you can produce quickly is a defensible yes or no: is my component in the failing path. Not "it looks fine", which is worth nothing to a room that has already decided it is you, but a statement with evidence attached.

The mistake that wastes the first fifteen minutes is answering it from your own telemetry. Your service's dashboards describe requests that reached it. A station that cannot resolve your hostname, cannot get through a firewall change made overnight, is timing out below your latency threshold, or is holding a stale connection through a load balancer produces no requests at all, so your error rate is zero and your availability is perfect while the line stands still. Green dashboards during an outage are the expected observation, not an exoneration.

So the instrumentation that saves you has to exist on the station side: the client's own call count, error count and latency, and the timeout it is configured with. That last figure is the one people never know and it decides these incidents. A station configured to wait two seconds, calling a service whose slow tail sits close to two seconds, fails intermittently and then stops the line under exactly the conditions that make you slower — a changeover, a shift start, a backlog draining. Your service was never down. It was slower than a number nobody wrote down, and finding that out mid-incident is much harder than knowing it beforehand.

Restoring production outranks understanding it

The instinct trained by software incidents is to preserve the failing state so you can diagnose it. Here that instinct is subordinate, because every minute of preservation has a price and the decision is not yours. Expect to be asked to restart, and expect that to be the right call.

What you can do is make the restart cheap in evidence. Agree, in advance, a short capture list that takes under a minute: current configuration as actually loaded, queue depths and connection counts, a thread or stack snapshot, the tail of the logs pinned somewhere they will not roll away, and the wall-clock time you did it. That is the difference between a postmortem with data and one that concludes "we restarted it and it went away", which is a conclusion that guarantees a recurrence.

Then look for the fastest path back, which is usually not a fix. Take yourself out of the path. If the station can proceed in a defined degraded mode — buffering its records locally, running against a cached recipe, falling back to a documented manual procedure — invoking that converts a production stoppage into a data backlog, which is a category of problem you can work on calmly afterwards. For this to be available it has to have been built, documented and rehearsed, and the shift supervisor has to be able to invoke it without you, because the incident where they most need it is the one where you are unreachable.

The product built during the incident is a separate decision

A web incident ends when traffic is healthy. This one does not, because the line produced physical units while it was degraded and those units exist. Once production is running again there is a second question: what is the status of everything made in that window.

The parts of it you have to be able to answer are whether those units have complete records, whether any checks were skipped or recorded manually, and whether the genealogy links are intact. The answers determine whether the units can be released, need to be held, need re-inspection or need rework, and that determination belongs to quality rather than to you. Your job is to say precisely what the record does and does not contain for that interval, without softening it. There is no rollback for a unit that has already been built, and a record you patch up afterwards to look complete is worse than a gap you declare.

Being exonerated is not the objective

There is a strong pull, once the immediate pressure lifts, towards establishing that it was not your fault. It is understandable and it is a losing move on both directions of the argument.

Pushing back hard to be cleared costs you the standing you need. The plant remembers who helped get the line running and who spent the incident defending a boundary. If you turn out not to have been the cause, that becomes evident from the timeline without you campaigning for it, and it lands far better coming from the controls engineer than from you.

Accepting blame to defuse the room is the opposite failure and it is worse. A wrong cause means the real one is still present, and the next occurrence arrives with your credibility already spent on the previous explanation. The honest position during the incident is a narrow, evidenced statement of what your component did and did not do, with the unknowns named as unknowns.

What the postmortem needs that a software postmortem does not

Two things make plant postmortems different, and both are easy to get wrong from a desk.

The timeline crosses clocks. Your logs, the historian, the controller's own event buffer and the MES each stamp events with their own time source, and a few seconds of relative offset is enough to reverse cause and effect in the reconstruction. Reconciling the offsets is part of building the timeline, not a detail to skip, and if you cannot reconcile them you say so rather than presenting a sequence you do not trust.

And the trigger is frequently physical, so it is invisible in your telemetry entirely. A jam cleared by hand, a changeover to a product with a different cycle time, a tool replaced, a station bypassed, a new operator following the procedure slightly differently, a maintenance activity on an adjacent machine. Any of those can change the arrival pattern your software sees, and your software's response to it is then the visible symptom of an invisible cause. This is why a postmortem run only by the software team lands on the wrong conclusion more often than not: the operator who was standing there and the controls engineer who was on the radio hold the half of the timeline that no system recorded.