Three days of your visibility events are unrecoverable and freight is still moving. How do you re-establish where everything is?

The freight does not pause for your recovery

The first thing to say out loud is that this is not a data recovery problem with an operational side effect. It is an operations problem in which data recovery is one workstream. Containers are being discharged, trailers are arriving at gates, drivers are completing rounds, and every hour of the outage adds new events on top of the missing ones. Any plan that starts with restoring the store and then resuming ingestion has already lost the events generated during the restore.

So the shape of the answer is two parallel efforts. One keeps operations running on the last known state plus whatever manual reporting can be stood up, on the assumption that the system will be blind for a while. The other rebuilds history. Keeping them separate matters, because the temptation is to hold operations until the picture is complete, and the picture will not be complete for days.

Almost all of those events still exist somewhere else

The word to resist is reconstruct. Reconstruction implies inference, and inference produces a plausible history that nobody can audit. What you should be doing is re-ingesting, because a visibility platform is downstream of everything, and the parties upstream kept their own records.

Carriers hold their own scan and milestone history and most expose it, either through a tracking interface that accepts a shipment reference or through a feed that can be requested for a date range. Terminal operating systems know which containers were discharged, gated in and gated out, with timestamps, because their own billing depends on it. EDI arrives through a mailbox or a value-added network, and those transports typically retain what they delivered for a period, which means the original messages can often be pulled again rather than recovered from your own store. Handheld devices and driver applications hold local queues, and depending on how they were designed, some of the events lost centrally are still sitting on a device that has not been wiped. Partners running visibility repositories of their own, particularly where an event-sharing standard is in use, hold their side of every exchange.

Working that list into a table of sources, with what each can supply, over what range, and how quickly, is the actual first deliverable. It also immediately identifies your worst dependencies: the carrier whose only interface is a per-shipment lookup, which turns a bulk replay into tens of thousands of individual calls, and the partner who sends a daily file and does not keep it.

Idempotent re-ingestion, or you will make it worse

Re-ingesting only helps if applying the same event twice is harmless, and this is the property that has to have been designed long before today.

An event needs a deterministic identity derived from its content rather than from an assigned sequence number, because the sequence numbers were in the store you lost. The natural key is close to what visibility standards already model: what object the event is about, when it happened, where it happened, and why, meaning which business step it represents. The GS1 EPCIS model organises event data along exactly those dimensions, and the discipline it imposes is useful even where the standard itself is not adopted, because it forces each event to carry enough context to be identified without reference to a database.

The second property is that events are applied by the time they occurred, not the time they arrived. During a rebuild, arrival order is meaningless: you will receive a delivery event from one carrier before an arrival event from another that happened two days earlier. A projection that advances a status only forward through a known sequence, and that recomputes from the event set rather than mutating in place, handles this. A projection that applies whatever arrives last does not, and will leave a delivered shipment showing as in transit for reasons nobody can trace.

The third is that re-ingestion must not fire the side effects again. Customer notifications, invoice triggers, carrier bookings and exception alerts all hang off event processing, and replaying three days of events with notifications enabled will send tens of thousands of messages about deliveries that happened last week. Suppressing side effects during replay is a switch you either have or spend a very bad afternoon building.

Physical observation closes the gap the feeds cannot

Some events will not come back. A scan taken on a device that was reimaged, an event from a partner with no retention, a manual status change made by a person who does not remember. For those, the only authority left is the freight itself, and the recovery plan needs a physical arm.

At a yard or a terminal, that means a sweep: read what is physically present, container by container or trailer by trailer, and treat the result as an observation event with a timestamp of now. That does not tell you the missing history, and it does tell you the current state, which is what operations needs. In a warehouse the equivalent is a count. On the road it is a driver debrief and the paperwork in the cab, since a signed delivery note is evidence even when no scan survived.

The important discipline is that these observations enter the store marked as what they are. An inferred or manually asserted position is not the same object as a captured scan, and conflating them means that six months later, when a claim is being argued, nobody can say whether the record reflects an observation or somebody's best reconstruction. Provenance on every event is cheap to add and impossible to backfill.

What you say while the picture is incomplete

Customers and partners will ask, and the answer that damages least is a specific one. A tracking page that shows a stale last-known event with an honest age, and says plainly that updates are delayed, is better than one that shows nothing and far better than one that guesses an arrival estimate. Suppressing arrival estimates during the outage is usually correct, because an estimate computed from an event history with holes in it is not an estimate, and a confidently wrong date generates more contact than an absent one.

Internally, the equivalent is to stop trusting your own exception detection. Rules that flag a shipment as quiet after a threshold will fire on everything, so exception queues need suppressing or rebasing, otherwise the genuine problems drown in the artefacts of the outage. Deciding that in advance, rather than after the alerting system has paged somebody four thousand times, is part of a considered answer.

The archive that makes this a day rather than a fortnight

The architectural point the interviewer is really testing is whether you separated the record from what you built out of it. A visibility platform typically holds two very different things: the raw inbound messages exactly as they were received, and the projections derived from them, which include shipment status, milestone histories, arrival estimates and customer-facing views.

The projections are expensive to hold and cheap to rebuild, provided the raw messages exist. The raw messages are cheap to hold, because they are append-only, compress well and are never updated, and they are impossible to rebuild once gone. So the raw inbound feed belongs in immutable, append-only storage, in a different system and ideally a different account or tenancy from the operational database, with its own retention and its own credentials. That is what turns this incident from an archaeology project into a replay, and it is a decision that costs very little on the day it is made.

Being explicit about the threat model matters here too, because the outage in the question is more likely to be malicious or a botched migration than a disk failure, and both of those can reach backups that share credentials with the thing they protect. A copy that the production service cannot delete is the property you want, not merely a copy that exists.

When visibility disappears before the cargo does

In June 2017, the NotPetya malware spread through a compromised update to Ukrainian accounting software and destroyed much of Maersk's IT estate, halting booking and container operations at terminals worldwide for weeks while systems were rebuilt from the few surviving copies. The containers were still physically there. Nobody could say where they were or whose they were, and the business ran for a period on manual processes, direct contact and whatever people knew.

What that event teaches for this question is not about ransomware defence. It is that the physical network kept functioning while the informational layer did not, that the recovery depended on which copies happened to survive rather than on a designed capability, and that the operational fallback was improvised by people under pressure. Both halves of a good answer follow from it: hold an immutable copy of the events you receive so a rebuild is a replay, and decide in advance how the operation runs on last known state while that replay is happening.