Loading...
Loading...
Browse 4 real-world technical and behavioral interview questions about Ab testing. Review scenarios, edge cases, and architectural best practices.
Treat it as a broken experiment, not a rounding issue. At two million users a 53/47 split is astronomically unlikely by chance, so something removed users non-randomly - and that something almost certainly correlated with the outcome. The result cannot be trusted in either direction, so find the mechanism, fix it and rerun rather than adjusting the numbers.
Fix the hypothesis, one primary metric, the randomisation unit and the sample size before launch. Peeking at a fixed-horizon test inflates false positives because each look is another chance to cross the threshold, and an inconclusive result is an interval to interpret, not proof of no effect.
Offline you need graded relevance judgements and a rank-aware metric to compare orderings; online you need an experiment sensitive enough to detect the effect, with click data corrected for the fact that position drives clicks on its own, and a guardrail on zero-result and abandonment rates.
No, if the analysis was designed as a fixed-horizon test: checking repeatedly inflates the false-positive rate, and stopping at the first significant look also overstates the effect size. Either commit to the horizon, or adopt a sequential design that budgets error across looks.