You promise 99.9 percent availability and you call five services that each promise the same. What have you committed to?
Nines only mean something as a downtime budget over a stated window, and dependencies in series multiply: five hard dependencies at three nines each put your own ceiling near 99.4 percent, so the target is met by removing dependencies from the critical path rather than by asking each owner for more nines.
What the interviewer is scoring
- Does the candidate convert a nines figure into a downtime allowance over an explicit window
- Whether dependencies in series are multiplied, rather than the target being assumed independent of them
- That how availability is measured, by request success or by uptime, is raised as a question
- Whether redundancy is priced against the correlated failure that would defeat it
- Can they say what the team does differently once the budget is spent, instead of treating the figure as decoration
Answer
Convert the nines before you argue about them
A percentage is not a commitment until it has a window attached, so the first thing to do is turn it into time. Three nines allows 0.1 per cent of the window to be bad: about 43 minutes in a 30-day month, or roughly 8 hours 45 minutes across a year. Four nines allows about 4.3 minutes a month, and two nines allows about 7 hours 12 minutes.
Stating it that way changes the conversation immediately, because 43 minutes a month is a number people can reason about operationally and 99.9 per cent is not. It tells you that a single unlucky deploy that takes twenty minutes to roll back has spent half the month's allowance, which in turn tells you that rollback speed is an availability feature and not merely a convenience. It also exposes the difference between a monthly and an annual window: the same figure computed yearly lets you absorb one long outage that a monthly window would treat as a breach in the month it happened.
Series multiplies, and it multiplies fast
If your request cannot be served unless all five dependencies respond, availability is the product of their availabilities and your own. Six components at 0.999 gives 0.999 to the sixth power, which is about 0.9940 — near 99.4 per cent, or roughly 4 hours 18 minutes of unavailability a month. You have promised 43 minutes and your architecture allows six times that, before anything in your own code has failed.
The consequence is the one worth carrying into any design round: you cannot reach a target by asking each dependency owner to be better than you. Above a handful of hard dependencies the arithmetic makes that impossible, so the ceiling is raised by changing the topology rather than the promises. Three moves do that, and each is a design decision you can defend.
Take dependencies off the critical path. A recommendation service, an analytics call, a personalisation lookup — none of these needs to be able to fail your request. Give each a timeout and a fallback, and it stops appearing in the product entirely. This is usually the largest single improvement available, and it costs a decision about what the page looks like without that component rather than any infrastructure.
Make a critical dependency redundant, so it appears as a parallel term instead of a serial one. Two instances that fail independently at 1 per cent each are unavailable together only 0.01 per cent of the time, so a pair of two-nines components behaves like four nines. Cache or hold state locally so a dependency's outage degrades freshness instead of function; a stale price is usually better than no page.
The independence assumption is where redundancy is oversold
That parallel calculation is only true if the two failures are independent, and in practice they frequently are not. Two replicas in one availability zone share its power and network. Two instances of a service share a configuration store, a deployment pipeline, a certificate authority and a DNS zone. Two regions share the control plane you use to fail over between them, and share the release that was rolled out to both.
So the useful question about any redundant pair is not how likely each side is to fail but what they have in common, because that shared thing is the real availability of the pair. A design that claims four nines from redundancy while both halves take the same configuration push at the same moment has claimed a number the arithmetic does not support. Naming the correlated element out loud is what makes a redundancy argument credible.
Measured how, and against what
Two services can both report three nines and mean entirely different things, so define the indicator before the objective. Availability measured as uptime asks whether the process was running. Measured as the proportion of successful requests, it asks whether users got answers, which is what they care about and what makes partial failures visible: a service serving 40 per cent errors is fully up by the first definition.
Two refinements are worth having. Count slow responses as failures by defining success as a correct response within a latency threshold, since a request that takes thirty seconds has failed from the caller's point of view whatever status it eventually returns. And be careful whose requests you count, because a service where every user sees 1 per cent errors and one where 1 per cent of users see nothing at all produce identical numbers and very different support queues.
The number is only real if spending it changes behaviour
The reason to compute a budget rather than an aspiration is that the remainder is meant to be used. Forty-three minutes a month is permission: while the budget is intact you can ship faster, migrate the datastore, and run the failure experiment. When it is exhausted, something has to actually change — a freeze on non-essential releases, reliability work taking priority over features — and if nothing changes then the objective was never an engineering constraint.
That is also what protects you from the opposite failure, which is chasing nines nobody asked for. Each additional nine costs roughly an order of magnitude more, and a target set well above what the product needs quietly consumes the engineering capacity that would have gone into the product. A budget consistently coming in far under is evidence the target is too strict, not a triumph.
Multiply your hard dependencies before you promise anything, and if the product does not clear the target, remove something from the critical path — no dependency owner can be reliable enough to fix arithmetic you created by making their service mandatory.
Likely follow-ups
- Your dependency meets its three nines but its outages land inside your busiest hour. Why does the arithmetic understate the damage?
- How would you set an objective for a nightly batch pipeline, where there are no requests to succeed or fail?
- Which of the five dependencies could you make non-critical, and what does the degraded response return?
- Who is allowed to spend the error budget, and what changes in the sprint after it runs out?
Related questions
- The business says the system must be highly available. How do you turn that into a number, and what does that number cost?mediumAlso on availability and slo4 min
- How would you set an SLO for a service, and what is an error budget for?mediumAlso on slo and error-budget5 min
- Service A needs something from service B. When should that be a synchronous call and when should it be an event?mediumAlso on availability3 min
- You add one small npm package to a service. What have you just taken on?mediumAlso on dependencies5 min
- Checkout calls pricing, tax, fraud, inventory and a session store. Which of those are you willing to carry on without, and what does the shopper see?hardAlso on graceful-degradation6 min
- One managed service in your primary region is having a bad day and your product is down with it. What can you do during the outage, and what should you have designed before it?hardAlso on graceful-degradation5 min
- You have six teams building one product and they keep blocking each other. How do you coordinate the dependencies, and what do you make of frameworks like SAFe?hardAlso on dependencies6 min
- A stakeholder tells you the system must be scalable. How do you turn that into a requirement you can design and test against?hardAlso on slo6 min