How would you set an SLO for a service, and what is an error budget for?

Three words that are not synonyms

An SLI is a measurement: a ratio of good events to valid events, or a latency percentile, expressed as a number that moves over time. An SLO is a target for that number over a stated window, chosen by you and enforced internally. An SLA is a contract with a customer that attaches money or credits to missing a target, and it should always be looser than the SLO, because you want to find out you are in trouble before your customer's lawyer does.

Candidates collapse these constantly, usually by saying "our SLA is 99.9%" when they mean the internal target and there is no contract at all. Separating the three early is most of the signal here, because the rest only makes sense once the SLO is understood as self-imposed rather than externally imposed.

Measure it where the user is

An SLI that is not measured from the user's vantage point will report health that nobody experiences. Container uptime, CPU headroom and pod restart counts are not SLIs; they are diagnostics you consult after an SLI has already gone bad. If the load balancer is returning 503 because no backend is ready, every backend can be simultaneously "up" and the service entirely unavailable.

In practice this means measuring as far out as you can afford to. Request logs at the edge, an API gateway or a load balancer are the usual pragmatic choice: they see the request and its status code, and they are cheap. Real-user monitoring in the browser sees more truth but adds cost and its own gaps. Synthetic probes are useful for coverage during idle periods and poor as a primary availability SLI, because they exercise one path with one client from one place.

Getting the denominator right matters as much as the numerator. "Valid events" should exclude traffic you are not promising anything about: health checks, requests from a load-test rig, and client errors you did not cause. A 400 because the caller sent malformed JSON is not your outage, and counting it as one teaches the team to distrust the SLO.

The arithmetic

The error budget is simply 1 - SLO applied to the window. A 30-day month contains 30 x 24 x 60 = 43,200 minutes, so a 99.9% availability SLO permits 0.1% of that, which is 43.2 minutes. Tighten to 99.95% and you have 21.6 minutes; tighten again to 99.99% and you have 4.32 minutes, which is less time than most teams need to acknowledge a page, let alone fix anything. That is the number to say out loud when someone asks for four nines without asking what it costs.

Note that the window changes the answer and people quote it carelessly. The same 99.9% is 44.64 minutes across a 31-day month, 40.32 minutes across a 28-day February, and 4.38 hours across a year. If you use a rolling 30-day window rather than a calendar month, the budget never resets on the first of the month, which is generally what you want.

For a request-ratio SLI, "43.2 minutes" is shorthand for the worst case where every request fails. If you serve a steady rate and 5% of requests fail, you are spending budget at one twentieth of that rate, so the same 0.1% allowance stretches across 864 minutes, a little over fourteen hours, of degradation. A partial brownout and a hard outage are not equivalent, and the ratio form of the SLI captures that where a naive "minutes of downtime" count does not.

What the budget is for

The budget exists to make the reliability-versus-velocity trade-off explicit and to settle it with data instead of seniority. Reliability beyond what users notice is paid for in slower releases, more review gates, more replicas and more on-call, so unspent budget is release capacity you are declining to use.

Crucially, the budget triggers a policy agreed in advance, in writing, by both engineering and product. Something like: while budget remains, ship on the normal cadence and take the risk; once it is exhausted, feature deploys stop and reliability work takes priority until the window recovers. The value is entirely in having agreed that rule before the difficult week, so the conversation during the difficult week is about applying it rather than negotiating it.

It follows that the budget is not a target to hit. Consistently spending none of it means the SLO is too loose or you are over-investing; consistently blowing through it means either the SLO is fiction or there is real work to do. Neither reading is available if the team treats the budget as a score to keep at 100%.

The SLO nobody can afford to spend

The failure that quietly kills the whole practice is picking the SLO number from ambition rather than from user tolerance. A team writes down 99.99% because it looks serious, discovers the service has never done better than 99.9%, and is therefore permanently in budget deficit. The policy that was supposed to fire on exhaustion now fires every month, so it gets waived every month, and within a quarter the SLO is a dashboard nobody opens. Start from what your users actually notice and what the current data says you deliver, set the SLO just tight enough to be uncomfortable, and only then tighten it. An SLO with a credible policy attached beats a more impressive number with no teeth.

Burn rate, not blips

Alerting directly on the SLI threshold pages you for every transient spike, which is how on-call rotations learn to ignore alerts. Instead, alert on burn rate: the multiple of budget consumption relative to the rate that would exactly exhaust the budget by the end of the window. A burn rate of 1 means you finish the month at precisely the SLO. A burn rate of 14.4 sustained for one hour consumes 2% of a 30-day budget, because one hour is 1/720 of the window and 14.4/720 is 2%. Sustain 6x for six hours and you have spent 5%.

Fast burn deserves a page; slow burn deserves a ticket. The usual construction pairs a long window with a short one, so the alert both requires the burn to have lasted long enough to be real and stops firing promptly once recovery begins.

# Page on fast burn: 14.4x sustained over an hour (2% of a 30-day budget).
# The 5m window is the reset condition - without it the alert stays lit for
# an hour after the incident is over, because the 1h average is still high.
- alert: CheckoutErrorBudgetFastBurn
  expr: |
    slo:error_ratio:rate1h{service="checkout"} > 14.4 * 0.001
    and
    slo:error_ratio:rate5m{service="checkout"} > 14.4 * 0.001
  for: 2m
  labels:
    severity: page

Two numbers make an SLO real: the budget in minutes, and the pre-agreed thing that happens when it runs out. Without the second, you have a dashboard.