hardDesignScenarioSeniorStaffLead
The tooling you would use to fix a network outage runs over the network that is down. How do you break that circularity before it costs you a day?
6 min readUpdated 2026-08-15Target archetype: Big Tech, Enterprise Captive, Product Startup
Answer
Short answer
Separate the layers that can fail together: routing reachability, name resolution, identity and physical access. Then build an access path that shares none of them, keep controls failing static rather than closed, and rehearse the emergency path often enough that it is known to work.
Was this answer helpful?
Likely follow-ups
- Your bastion is reached by name and name resolution is gone. What do you still have if nothing was prepared in advance?
- Your identity provider is a hosted service and its login page cannot be reached. How should break-glass authentication work instead?
- Should a server withdraw its route when its own health check fails? Argue both sides and say when you would choose which.
- The door controllers and badge readers depend on the corporate network. Whose problem is that, and what do you do about it?
Related questions
- Your application cannot reach a service that should be up. Walk me through diagnosing it from the shell.mediumAlso on networking and dns6 min
- Would you use iptables or eBPF for network policy enforcement in a massive multi-tenant Kubernetes cluster, and what are the operational trade-offs?hardAlso on networking2 min
- Why does the standard Linux network stack fail at MMO scale, and how do you bypass it using DPDK?hardAlso on networking2 min
- What cost does kernel-bypass networking actually remove, and what do you give up to get it?hardAlso on networking5 min