The tooling you would use to fix a network outage runs over the network that is down. How do you break that circularity before it costs you a day?

Ask which layer is broken before designing around it

Engineers collapse network failure into one idea and then design one mitigation for it, which is how circular dependencies survive review. There are at least four separable layers, and recovery can be blocked by any of them independently.

Routing decides whether packets can reach an address at all, and at the internet edge that is BGP: your prefixes are advertised, and if the advertisement stops, your addresses cease to exist as far as everyone else is concerned. Name resolution decides whether a client can turn a name into one of those addresses, and it fails separately - a name can resolve to an address nobody can route to, and a routable address can be unreachable because nothing will tell you what it is. Identity decides whether you can authenticate once you have a connection. And physical access decides whether you can stand in front of the equipment when the first three have all gone.

The useful discipline is to take each recovery action you would perform and ask which of those four it needs. Most teams discover that every action they have needs all four.

The health check that withdrew the only way in

In October 2021 a configuration change during maintenance on Facebook's backbone led their DNS servers to withdraw the BGP routes that made them reachable, so the authoritative name servers for their domains vanished from the internet. Internal tooling that resolved through the same names was unusable during the recovery, and the engineers who needed to reach hardware faced the physical-access problem as well.

The mechanism is the part worth understanding rather than the sequence of events. Those name servers advertised their own routes and were designed to withdraw the advertisement when they could no longer reach the data centre behind them, which is a sensible anycast health check: a name server that cannot serve correct answers should stop attracting queries so that another site takes them. The behaviour was correct in the case it was designed for. It became catastrophic in the case where every site lost that connectivity at once, because then the mechanism removed every site rather than shifting load between them.

This is a general shape, not a Facebook-specific bug. A health check that removes capacity is a load-shifting device that assumes healthy capacity exists elsewhere. When the fault is correlated across all instances, the same logic removes the service entirely and does it faster than a human can intervene. So any withdrawal mechanism needs a floor: keep advertising if withdrawing would leave nothing advertised, and prefer serving possibly-stale answers over serving none. That is the difference between failing closed and failing static, and it is the single most transferable idea here.

Mapping your own circular dependencies

Do this as an inventory rather than as an argument, because the answers are specific to your estate. Take the six things you would do in a serious network incident - reach a jump host, read dashboards, authenticate, look up which device is which, push a configuration change, talk to your colleagues - and write down what each one resolves, routes over and authenticates against.

flowchart TD
    A[Backbone change] --> B[Sites unreachable]
    B --> C[DNS servers withdraw their routes]
    C --> D[Names no longer resolve]
    D --> E[Bastion and dashboards unreachable]
    D --> F[Chat and paging unreachable]
    E --> G[Recovery needs physical access]

The interesting property is that only the first arrow is the fault; every arrow after it is a designed behaviour operating as specified, which is why review did not catch it.

The findings are usually uncomfortable. Chat and incident paging are hosted services reached by name. Dashboards are behind the same single sign-on as everything else. The inventory of which device has which address lives in a wiki behind that sign-on. The bastion is reached by name, and the name resolves through the resolvers that are down. And the address of the router you need is in a system you cannot open.

What out-of-band actually means

Out-of-band is not a slogan; it is a set of concrete properties. The access path should use different physical links, different addresses, different name resolution or none at all, and different credentials. In practice that means console access to network devices over serial or a management processor, a management network with its own transit that does not depend on the production backbone, and often a cellular link at each site precisely because a mobile network shares nothing with your fibre.

Then remove name resolution from the path deliberately. Keep the addresses of jump hosts, console servers and core devices as literal addresses, in a form that is available offline: a printed or exported runbook, a local hosts file on the laptops of the people on call, an encrypted copy on a phone. Static addressing for management interfaces is worth the inflexibility for the same reason. If reaching the recovery path requires a lookup, the recovery path is not out of band.

Identity needs the same treatment. Break-glass accounts should authenticate locally on the device or against a service that is not the one you are trying to reach, hold credentials in an offline-capable store, and be protected by a hardware token rather than a factor delivered over the network you have broken. Their use must be loud - alerted, logged, and reviewed afterwards - because an account that bypasses your controls is a standing risk that you accept only in exchange for visibility.

Physical access is the layer people never write down. Electronic locks, badge readers and lift controllers frequently depend on the corporate network, so the site you must reach can lock you out at exactly the wrong moment. The answer is not technical: it is knowing who holds a mechanical override, whether they are contactable without your normal comms, and whether anyone has ever tried.

Rehearsing the path you never use

An untested emergency path is a belief. Schedule the exercise: on a fixed cadence, have an on-call engineer reach a production network device using only the out-of-band path, authenticate with break-glass credentials, and read back a configuration. Time it. The exercise will find expired certificates, a console server nobody has patched, a cellular SIM that was cancelled to save money, and a runbook whose addresses changed two migrations ago. Every one of those is cheaper to find on a Tuesday afternoon.

Choose a secondary communication channel too, and make it genuinely independent - a different provider, reachable from a phone, with the group already created and populated. During an outage is the wrong time to be collecting mobile numbers, and a channel that has never been used is a channel nobody will find.

Routing, name resolution, identity and physical access fail independently, and a recovery plan is only real if it shares none of those with the fault. Prefer mechanisms that fail static over ones that fail closed, keep addresses and credentials available offline, and rehearse the out-of-band path on a schedule so that its defects are found while everything is still working.