Distributions, hypothesis testing, A/B design, and reading a result honestly.
4 questions in Data & AI Engineering.
Guardrail metrics are outcomes you are not trying to improve but will not let degrade beyond a margin declared before launch. They stop a launch when the data shows harm past that margin, or cannot rule it out, which means guardrails need their own power and their own pre-agreed decision rule.
Treat it as a broken experiment, not a rounding issue. At two million users a 53/47 split is astronomically unlikely by chance, so something removed users non-randomly - and that something almost certainly correlated with the outcome. The result cannot be trusted in either direction, so find the mechanism, fix it and rerun rather than adjusting the numbers.
Fix the hypothesis, one primary metric, the randomisation unit and the sample size before launch. Peeking at a fixed-horizon test inflates false positives because each look is another chance to cross the threshold, and an inconclusive result is an interval to interpret, not proof of no effect.
No, if the analysis was designed as a fixed-horizon test: checking repeatedly inflates the false-positive rate, and stopping at the first significant look also overstates the effect size. Either commit to the horizon, or adopt a sequential design that budgets error across looks.