Top 6 Ab testing Interview Questions
Browse 6 real-world technical and behavioral interview questions about Ab testing. Review scenarios, edge cases, and architectural best practices.
Design an experimentation and feature-flag platform: assignment, exposure logging and a kill switch.
Evaluate feature flags in an SDK from a cached ruleset, assign users by hashing a per-experiment salt with the user id so assignment is sticky, log exposure where the arms first diverge, gate readouts on an automated SRM check, and make the kill switch a top-precedence rule that works with the control plane down.
Your test lifts conversion, but something else moved the wrong way. What are guardrail metrics, and when do they stop a launch?
Guardrail metrics are outcomes you are not trying to improve but will not let degrade beyond a margin declared before launch. They stop a launch when the data shows harm past that margin, or cannot rule it out, which means guardrails need their own power and their own pre-agreed decision rule.
Your A/B test was configured for a 50/50 split but the data shows 53/47 across two million users. The result is significant and positive. What do you do?
Treat it as a broken experiment, not a rounding issue. At two million users a 53/47 split is astronomically unlikely by chance, so something removed users non-randomly - and that something almost certainly correlated with the outcome. The result cannot be trusted in either direction, so find the mechanism, fix it and rerun rather than adjusting the numbers.
Walk me through designing an A/B test for a checkout change. What do you fix before any data exists?
Fix the hypothesis, one primary metric, the randomisation unit and the sample size before launch. Peeking at a fixed-horizon test inflates false positives because each look is another chance to cross the threshold, and an inconclusive result is an interval to interpret, not proof of no effect.
You have changed the ranking. How do you decide whether search got better, rather than just different?
Offline you need graded relevance judgements and a rank-aware metric to compare orderings; online you need an experiment sensitive enough to detect the effect, with click data corrected for the fact that position drives clicks on its own, and a guardrail on zero-result and abandonment rates.
Your experiment dashboard says the variant is significant on day three of a fourteen day test. Can you ship it?
No, if the analysis was designed as a fixed-horizon test: checking repeatedly inflates the false-positive rate, and stopping at the first significant look also overstates the effect size. Either commit to the horizon, or adopt a sequential design that budgets error across looks.