One managed service in your primary region is having a bad day and your product is down with it. What can you do during the outage, and what should you have designed before it?
6 min readUpdated 2026-08-15Target archetype: Big Tech, Enterprise Captive, Product Startup
0
Answer
Short answer
During the event your only levers are the ones already built: fail over, serve degraded from cache, or shed the dependent feature. Beforehand the work is finding your transitive dependencies on foundational services, making the running system statically stable, and choosing which features are allowed to fail alone.
Was this answer helpful?
Likely follow-ups
→Your failover requires launching instances in the second region and the provider's launch API is also degraded. What now?
→How would you find out, today, which of your services would keep serving if object storage in your primary region stopped accepting reads?
→The provider's status page says everything is fine and your dashboards say otherwise. How do you proceed, and what do you tell the business?
→You have budget for resilience in exactly one place. How do you choose where it goes?
During an outage of something you do not operate, your options are limited to mechanisms that already exist and are already exercised. You cannot build a failover, you cannot introduce a cache, and you cannot rewrite a code path so that it tolerates a missing dependency. So the useful answer has two halves that must not be blurred together: a short list of things you can pull on now, and a longer list of things whose absence is the reason the list is short.
The levers that are ever genuinely available are failing over to capacity that is already running elsewhere, serving stale data from a cache or replica that is already populated, turning off the feature that needs the failed dependency so the rest of the product survives, and communicating honestly while you do. Everything else is a design decision you either made or did not.
The dependencies that are not on the diagram
Ask which of your services depend on the failed one and you will get the direct callers. The interesting set is the transitive one. A foundational service like object storage, managed DNS, an identity provider or a secrets store tends to sit underneath other managed services, your own deployment tooling, your container registry, your observability stack and your authentication path. When it degrades, things fail that nobody had recorded as depending on it, including some of the tools you would reach for.
In February 2017 an engineer working through an established playbook in the S3 us-east-1 region mistyped a command and removed far more index-subsystem capacity than intended. Recovery ran for hours because subsystems of that size had not been fully restarted in a very long time, and enough other services in the region depended on S3 that one service's failure presented as a platform-wide one. AWS also reported that its own health dashboard could not be updated normally during the event because that dashboard's update path depended on the affected region. That last detail is the one to carry into your own estate. The question is not only what breaks, but whether the things you use to see and to fix breakage share a fate with it.
The practical exercise is a dependency inventory you can actually maintain: for each foundational service, list what stops working, what degrades, and what you would be unable to do. Reads and writes usually differ. Startup and steady state usually differ, which is why the next section matters more than any other single idea here.
Static stability, and why a healthy process is not the same as a working one
A system is statically stable when it keeps working during a dependency failure without needing to make any changes. The opposite pattern is a system that is fine while nothing moves but cannot take any action, because every action requires the degraded dependency.
The distinction that makes this concrete is control plane against data plane. The control plane is what changes the shape of the system: launching instances, updating routing, issuing credentials, registering endpoints, publishing config. The data plane is what serves requests with the shape it already has. Control planes are more complex, depend on more things and fail more often; data planes are simpler and more available by design. A design that reacts to failure by asking the control plane to do something is depending on the less reliable half at the worst moment.
That is why pre-provisioned standby capacity beats scaling up on demand during an event, why a load balancer that keeps sending traffic to the last known healthy set beats one that needs a fresh registration, and why cached credentials and cached service discovery results are worth their staleness. It is also why "we will fail over to the other region" deserves the follow-up question of what the failover itself needs in order to run.
Loading diagram…
Every branch that leads anywhere other than the bottom box had to be built in advance, and the gap worth noticing is that two of the three escape routes are cheap while the third is not.
Deciding what is allowed to fail on its own
Most products do not need every feature to be equally available, and the cheapest resilience available is the decision that a non-essential feature fails alone. Recommendations, avatars, analytics, export, search suggestions and audit views can usually be degraded or omitted while checkout, login and the core write path continue. This only works if the dependency is behind a bounded call with a short timeout and a defined fallback, and if the fallback path is exercised often enough to be trusted. A synchronous call with a generous timeout and no fallback converts an optional feature into a mandatory one, and thread-pool exhaustion then takes the essential path down with it.
Marking degraded data as degraded belongs in the same conversation. Serving a stale price without saying it is stale is sometimes worse than serving an error, and that judgement is the business's rather than yours. Being able to state which is which is a large part of what a senior answer looks like.
Pricing the second region honestly
Multi-region active-active is the strongest answer and the most expensive one, and it is not automatically correct. It costs duplicated capacity, cross-region data transfer, a genuine answer to write consistency across regions, and a continuous testing burden, because a failover path that is never exercised is a hypothesis. Multi-region for the data with a warm standby for the compute is often the better trade, and for many products a single region with strong degradation behaviour plus a credible recovery-time commitment to the business is the honest choice.
What is not defensible is asserting a resilience posture you have never tested, or claiming an availability target that exceeds the composed availability of the managed services underneath it. If your product needs to be more available than a dependency you cannot replace, either the dependency is not really required for the critical path or the target is fiction. Saying that out loud, with the numbers derived from the provider's own published figures rather than invented, is worth more in an interview than a diagram with two of everything.
The only mitigations available during a provider outage are the ones already running, so the design work is deciding in advance which features may fail alone, keeping the serving path independent of the control plane, and checking that the tooling you would use to recover does not depend on the thing that is broken.