Top 4 Statistical power Interview Questions
Browse 4 real-world technical and behavioral interview questions about Statistical power. Review scenarios, edge cases, and architectural best practices.
Your test lifts conversion, but something else moved the wrong way. What are guardrail metrics, and when do they stop a launch?
Guardrail metrics are outcomes you are not trying to improve but will not let degrade beyond a margin declared before launch. They stop a launch when the data shows harm past that margin, or cannot rule it out, which means guardrails need their own power and their own pre-agreed decision rule.
Walk me through designing an A/B test for a checkout change. What do you fix before any data exists?
Fix the hypothesis, one primary metric, the randomisation unit and the sample size before launch. Peeking at a fixed-horizon test inflates false positives because each look is another chance to cross the threshold, and an inconclusive result is an interval to interpret, not proof of no effect.
Your experiment dashboard says the variant is significant on day three of a fourteen day test. Can you ship it?
No, if the analysis was designed as a fixed-horizon test: checking repeatedly inflates the false-positive rate, and stopping at the first significant look also overstates the effect size. Either commit to the horizon, or adopt a sequential design that budgets error across looks.
Your A/B test came back not significant. What do you do next?
Establish whether the test could ever have detected the effect you cared about, because an underpowered null tells you nothing; then read the confidence interval rather than the verdict and decide whether to extend, test a larger change, or accept that the assumption was wrong.