Statistics & Experimentation interview questions

Distributions, hypothesis testing, A/B design, and reading a result honestly.

3 questions in Data & AI Engineering.

Data and AIA 22-minute read on how the topic fits together, before the questions.Data and AI engineering fundamentals58 short answers on one page, for revising rather than studying.

hard3

hardConceptScenarioCase Study

Your A/B test was configured for a 50/50 split but the data shows 53/47 across two million users. The result is significant and positive. What do you do?

Treat it as a broken experiment, not a rounding issue. At two million users a 53/47 split is astronomically unlikely by chance, so something removed users non-randomly - and that something almost certainly correlated with the outcome. The result cannot be trusted in either direction, so find the mechanism, fix it and rerun rather than adjusting the numbers.

5 minmid, senior, staff

hardDesignCase Study

Walk me through designing an A/B test for a checkout change. What do you fix before any data exists?

Fix the hypothesis, one primary metric, the randomisation unit and the sample size before launch. Peeking at a fixed-horizon test inflates false positives because each look is another chance to cross the threshold, and an inconclusive result is an interval to interpret, not proof of no effect.

6 minmid, senior, staff

hardConceptScenario

Your experiment dashboard says the variant is significant on day three of a fourteen day test. Can you ship it?

No, if the analysis was designed as a fixed-horizon test: checking repeatedly inflates the false-positive rate, and stopping at the first significant look also overstates the effect size. Either commit to the horizon, or adopt a sequential design that budgets error across looks.

5 minmid, senior, staff