Top 3 Experiment design Interview Questions
Browse 3 real-world technical and behavioral interview questions about Experiment design. Review scenarios, edge cases, and architectural best practices.
Your test lifts conversion, but something else moved the wrong way. What are guardrail metrics, and when do they stop a launch?
Guardrail metrics are outcomes you are not trying to improve but will not let degrade beyond a margin declared before launch. They stop a launch when the data shows harm past that margin, or cannot rule it out, which means guardrails need their own power and their own pre-agreed decision rule.
Your A/B test was configured for a 50/50 split but the data shows 53/47 across two million users. The result is significant and positive. What do you do?
Treat it as a broken experiment, not a rounding issue. At two million users a 53/47 split is astronomically unlikely by chance, so something removed users non-randomly - and that something almost certainly correlated with the outcome. The result cannot be trusted in either direction, so find the mechanism, fix it and rerun rather than adjusting the numbers.
Walk me through designing an A/B test for a checkout change. What do you fix before any data exists?
Fix the hypothesis, one primary metric, the randomisation unit and the sample size before launch. Peeking at a fixed-horizon test inflates false positives because each look is another chance to cross the threshold, and an inconclusive result is an interval to interpret, not proof of no effect.