The primary metric and the guardrails do different jobs
The primary metric is the one the change is meant to improve, and the A/B test is sized to detect a lift in it. Guardrail metrics answer a different question: did the change break something we care about while it was improving the primary? You are not trying to show that a guardrail improved. You are trying to show it did not get meaningfully worse.
There are two families worth naming. Trust guardrails check that the experiment itself is valid: sample ratio, logging volume per arm, error rates in assignment. If one of those fails, the result is not interpretable at all, and the sample ratio mismatch page covers what to do. Business guardrails protect outcomes the organisation will not trade away: refund rate, support contacts, page latency, crash-free sessions, revenue. This answer is mostly about the second family.
A margin and a rule, fixed in advance
A guardrail without a number is a dashboard tile. A useful one has three parts written into the design: the metric, the direction that counts as harm, and the margin. "Refund rate may not rise by more than 3% relative" is a guardrail. "Watch refunds" is not.
The decision rule matters as much as the margin, and there are two common policies. The weaker one says stop if the guardrail is significantly worse than control. The stronger one says ship only if you can show the guardrail is not worse than the margin, which is a non-inferiority test: a one-sided test of whether the true increase in harm is smaller than the margin. The weaker rule is easy to pass by running an underpowered test. The stronger rule asks the data to prove safety, not just to fail to prove harm.
Worked through: conversion up, refunds wobbling
Suppose an A/B test lifts checkout conversion by 2.1% relative, with an interval that clearly excludes zero. Refund rate, a guardrail with a 3% margin, rose by 4% relative, with a 95% interval from -1% to +9%.
Under the weak rule, this passes. The interval contains zero, so refunds are "not significantly worse". Under the non-inferiority rule, it fails. The upper end of the interval, +9%, sits well past the 3% margin, so the data is consistent with a refund increase three times larger than you said you would accept. The honest reading is not "refunds are fine". It is "we do not know whether refunds are fine".
What you do next depends on what you pre-agreed. You can extend the test to shrink the interval, look for a mechanism that explains the refund move, or put the trade-off to whoever owns refunds with the numbers attached. Shipping and calling it safe is the one option the data does not support.
Guardrails need power too
Tests are almost always sized for the primary metric. Guardrail metrics ride along, and many of them are noisier. Refunds are rare events and revenue is heavy-tailed, so their intervals are wide at a sample size that is ample for conversion.
The arithmetic is simple to state. An interval's half-width shrinks with the square root of the sample. If the refund interval is about five percent either side of the estimate and you need it within about two and a half, you need roughly four times the sample. If that is not available, say so in the design: this test can detect a conversion lift but cannot rule out a relative refund increase smaller than about five percent. That sentence is far more useful than a guardrail that silently cannot fire.
Variance reduction helps here as it does for the primary. Use a pre-period covariate, or measure the guardrail only on users who reached the point where the change could affect it. Both narrow the interval without more traffic.
Many guardrails means false alarms
Each guardrail tested at a 5% significance level has a 5% chance of firing on noise. With ten roughly independent guardrails, the chance that at least one fires falsely is 1 - 0.95^10, about 40%. Real metrics are correlated, so the true figure is usually lower, but the direction holds: a long list of hard-stop guardrails will block good changes regularly.
The usual answer is tiers. Keep a short list of hard guardrails that stop a launch automatically, such as crash rate, severe latency regression and payment failure. Treat the rest as soft guardrails that trigger an investigation and a named owner's decision rather than an automatic stop. Writing down which tier each metric is in is part of the pre-registration.
Watching guardrails while the test runs
Monitoring guardrails daily for severe harm is legitimate. A crash spike on day one should stop the test without waiting for the planned horizon, because you are containing damage, not estimating an effect. But a rule that stops on any daily dip will fire on noise, for the same reason peeking at the primary does. If guardrails are monitored continuously, use a sequential method chosen in advance, as described on significant on day three of fourteen.
Reading "not significant" as "no harm"
The single error that separates a strong answer is treating a non-significant guardrail move as evidence of safety. A guardrail that did not reach significance may simply have had too little data to detect the harm you care about. That is absence of evidence, not evidence of absence.
The fix is to report guardrail metrics as intervals against their margins, not as p-values. An interval that sits entirely inside the margin is a pass. An interval that crosses the margin is unresolved, whatever its p-value. Candidates who reason this way, and who mention guardrail power without being asked, sound like people who have run experiments that shipped harm. Pairing rates with absolute counts, as covered in when a metric improves and the business gets worse, closes the remaining gap.
A guardrail passes when its interval sits inside the margin you declared beforehand, not when its p-value is large. If you cannot rule out the harm, you have not shown the change is safe.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).