Your error budget burn alert pages every few hours, but half the time nobody outside the team has noticed anything. How do you tune it without simply making it quieter?
A single threshold on a single window cannot separate a real outage from a brief spike, so it either pages on noise or misses slow burns. Replace it with multi-window multi-burn-rate alerting: a fast rule that pages and a slow rule that files a ticket, each gated by a short window so a spike that already stopped does not keep paging.
Answer
Short answer
A single burn-rate threshold over a single window has to choose between catching fast outages and ignoring brief spikes, and it cannot do both. The standard fix is multi-window, multi-burn-rate alerting: a fast rule that pages a human and a slow rule that files a ticket, each paired with a short window so an incident that has already recovered stops alerting. Before touching thresholds, though, check the SLI — most spurious pages are a measurement problem wearing an alerting costume.
Was this answer helpful?
Likely follow-ups
- Write me the two alert rules you would actually deploy, with windows and burn rates.
- At 3am you serve forty requests an hour and one fails. What does your alert do, and what should it do?
- The SLO is met for the month but users complained loudly for twenty minutes. What went wrong with the SLI?
- Your budget is exhausted on day nine of a thirty-day window. What actually happens next?
- How many SLOs should one service have, and what goes wrong at either extreme?
Related questions
- How would you set an SLO for a service, and what is an error budget for?mediumAlso on slo and alerting5 min
- Pods are being evicted during node pressure even though your CPU dashboards look fine. Where do you look, and what would you change so it stops?hardAlso on sre4 min
- The business says the system must be highly available. How do you turn that into a number, and what does that number cost?mediumAlso on slo4 min
- Every dashboard is green and a customer tells you checkout has been broken for an hour. How does your monitoring tell nothing-is-wrong apart from nothing-is-reporting?hardAlso on alerting6 min