The line is running fine but the plant cannot ship, because a business system is down. How would you have found that dependency before it bit you?
Plants accumulate business-system dependencies nobody deliberately chose: a label printed from ERP, a recipe fetched over a link, an authentication call, a licence check. You find them by asking of each system what stops if it disappears, then give every answer a written degraded mode and rehearse it.
What the interviewer is scoring
- Whether the candidate distinguishes making product from being able to dispatch it
- Does the answer enumerate couplings by asking what stops rather than by reading a network diagram
- That the direction of each dependency is stated, since OT calling IT and IT calling OT fail differently
- Whether a degraded mode is expressed as a duration the plant can run for, not as an aspiration
- Any recognition that the degraded mode is worthless until somebody has rehearsed it on a shift
Answer
Making product and being able to ship it are separate abilities
The scenario is common enough to be a category rather than an incident. Controllers hold their logic, the machines run, quality is being made, and the plant still cannot dispatch, because something in the enterprise estate holds a fact the shipment needs. Usually it is a label. A carton cannot leave without a compliant label bearing a customer part number, a serial or lot reference and a barcode, and the data behind that label lives in ERP or in a customer-specific label service. Sometimes it is the goods-issue posting, sometimes the transport booking, sometimes a certificate of analysis that must accompany the batch.
An interviewer asking this wants to see you separate two claims that get merged into one reassuring sentence. "The line does not depend on IT" is often true about producing units and almost never true about turning units into revenue. The plant is a physical process wrapped in a set of business assertions, and the assertions are what a customer, a regulator or an auditor actually receives.
Where the couplings actually sit
The dependencies are rarely in the places a network diagram highlights, because a diagram shows links rather than obligations. They accumulate one convenience at a time, each individually reasonable.
| Coupling | Looks like | Fails as |
|---|---|---|
| Label and document generation | ERP or a label server owns the template and the data | Product is made and cannot be released |
| Work order release | Orders arrive from ERP into MES on a schedule | The line runs out of authorised work at shift change |
| Recipe or parameter download | Master recipe held centrally, sent to the line per batch | The next changeover cannot happen |
| Identity and authentication | Operators log into HMI or MES against a corporate directory | Nobody can sign a record, or everyone shares one account |
| Licence and activation checks | Vendor software validates entitlement over the internet | Software refuses to start after a restart |
| Time synchronisation | Plant clocks chase an enterprise time source | Timestamps drift and records stop lining up |
The two most easily missed are the last two. Authentication is missed because it works invisibly until it does not, and because the workaround people reach for under pressure is a shared login, which destroys the attribution that quality records depend on. Licence checks are missed because they only bite on restart, which is precisely what happens during a recovery, so the dependency reveals itself at the worst moment.
Map it by asking what stops, not by drawing the network
The method that produces a real list is a structured interview with the people who run the plant, conducted one system at a time, and the question is always the same: if this disappeared right now, what would stop, and how long could you carry on without it? Ask it of the plant, and separately of each business system, and record the answer as a duration. A number in hours is the useful output, because it converts an abstract dependency into a decision about whether to invest.
Then record the direction, because the two directions fail differently. When OT depends on IT, the plant is exposed to enterprise change control, to a corporate patch window, and to a ransomware event in an estate you do not run. When IT depends on OT, an unavailable historian or MES means the business cannot report, invoice or plan, which is painful but does not stop production. A single "integration" arrow on an architecture diagram hides which of these you have, and candidates who only draw the arrow tend not to notice that they have created both.
The third thing to record is the fan-in. A dependency that fifteen lines share, such as one label service or one directory, is a different risk from a dependency confined to one cell, and it is usually the shared one that has never been tested because it has never failed.
A degraded mode is a duration, not an aspiration
Once you have a list of couplings with directions and durations, the design work is to extend each duration to something the plant can live with, and to state what happens when it runs out.
Caching is the first lever and it is the one that gets over-applied. Holding work orders, recipes and label data locally means the line can continue through a link outage, and it means the local copy is authoritative for that period, which is a genuine risk rather than a technicality: a recipe superseded centrally while the plant was isolated will keep being used. So a cache needs an age, a maximum age beyond which it must not be used, and a visible indication on the line that the plant is running on a copy. Caching without those three is not resilience, it is a stale-data incident waiting for a trigger.
Local authority is the second lever and it is the honest one for identity. Plant systems should be able to authenticate operators without a round trip off site, whether that is a local directory replica, a cached credential with a stated lifetime, or a break-glass procedure that is auditable rather than a whiteboard password. The break-glass path is a design artefact, and it must record who used it and when.
The third lever is the one people resist writing down: the manual fallback. Pre-printed generic labels reconciled later, a paper batch record, a handwritten dispatch note. It is slower, it is worse, and it is much better than an idle plant. Its cost is reconciliation afterwards, and the entries it produces must remain distinguishable from system-captured ones forever, because a manual record keyed in as though the system had captured it turns a workaround into a false statement on a document an auditor may read.
The dependency you should keep, and the rehearsal
Not every coupling should be removed. The plant that genuinely cannot be reached by the business is a plant that plans badly, ships the wrong things and cannot be measured, so the goal is not isolation, it is deliberate coupling: a link you chose, through a controlled boundary between plant and enterprise networks, carrying a named set of interfaces in a stated direction, with a defined behaviour when it is unavailable. That is what the layered plant-network models and the DMZ between plant operations and enterprise systems are for, and the value of the model is that it makes an unlisted flow visible as an exception rather than as normality.
What makes the whole exercise real is the rehearsal. A degraded mode that has never been run on a live shift is a document, and documents do not know that the pre-printed labels expired, that the operator with the break-glass procedure left in March, or that the local cache was pointed at a decommissioned server during an upgrade. Running it deliberately, for an hour, on one line, with the people who would actually have to do it, is the cheapest way to discover which of your assumptions were wrong.
When the business system is the thing that stops the plant
In May 2021, a ransomware intrusion into Colonial Pipeline's corporate IT systems led the company to shut down pipeline operations, because billing and scheduling systems it could no longer trust sat between it and safe delivery. The control network itself was not the thing that was compromised. That is this question in one event: the plant could physically move product and the organisation could not responsibly let it, because the systems that account for what moves were unavailable and untrusted.
The lesson an interviewer is fishing for is not about ransomware. It is that the dependency which stopped operations was a commercial one, it was almost certainly not on anyone's list of safety-critical systems, and the decision to stop was correct given no defined way to operate without it. Having that way defined, bounded and rehearsed is the whole of the answer.
Likely follow-ups
- Which of those dependencies would you deliberately keep, and what do you buy by keeping it?
- How long can the plant run on cached master data before the cache itself becomes the hazard?
- The corporate network is quarantined and you cannot trust it. Do you cut the link or keep it?
- Who decides that the plant should switch to degraded operation, and how do they find out they must?
Related questions
- The MES is unreachable for four hours and the line keeps producing. What happens to the production record, and how do you reconcile afterwards?hardAlso on degraded-mode5 min
- The electronic record will be down for a four-hour upgrade, and it also fails without warning. How does a ward keep working?hardAlso on business-continuity5 min
- The line has been stopped for twenty minutes and everyone in the room believes it is your system. How do you run the next hour?hardAlso on degraded-mode5 min
- A maintenance script deletes rows it should not have touched, and nobody notices for six hours. Walk me through the recovery.hardSame kind of round: scenario6 min
- A customer ports their number away to another operator. What has to happen on your side, and what usually goes wrong?hardSame kind of round: scenario6 min
- A customer's API token turns up in a public repository. What do you do in the next hour, and what in your token design decides how bad this is?hardSame kind of round: scenario6 min
- A dependency that normally answers in 80ms starts taking eight seconds. What in your service reacts, and in what order?hardSame kind of round: scenario6 min
- A dividend is declared and half the client's holding is out on loan while part of the rest is pledged as collateral. Who receives the income, who votes, and what does your system show the client?hardSame kind of round: scenario5 min