Your A/B test was configured for a 50/50 split but the data shows 53/47 across two million users. The result is significant and positive. What do you do?
Treat it as a broken experiment, not a rounding issue. At two million users a 53/47 split is astronomically unlikely by chance, so something removed users non-randomly - and that something almost certainly correlated with the outcome. The result cannot be trusted in either direction, so find the mechanism, fix it and rerun rather than adjusting the numbers.
Answer
Short answer
Do not report the result. A 53/47 split on two million users is not noise — under a true 50/50 assignment the probability of a deviation that large is effectively zero. Something removed or misassigned users in a way that was not random, and whatever did that is very likely correlated with the metric you are measuring. The experiment is invalid, and the job is to find the mechanism rather than to explain the imbalance away.
Was this answer helpful?
Likely follow-ups
- Compute the chi-square for 1.06M versus 940k and tell me roughly what p-value you expect.
- The split is 50/50 at assignment but 53/47 at the first logged event. Where do you look?
- Would you ever ship a result that had an SRM? Under what circumstances?
- Your variant loads a heavier bundle. How could that alone produce an SRM?
- How would you detect this automatically for every experiment, and what threshold would you alert on?
Related questions
- Your test lifts conversion, but something else moved the wrong way. What are guardrail metrics, and when do they stop a launch?hardAlso on ab-testing and experiment-design5 min
- Walk me through designing an A/B test for a checkout change. What do you fix before any data exists?hardAlso on ab-testing and experiment-design6 min
- Engineers added a user_id label to a few metrics and the observability bill went up tenfold. Explain what actually happened, and how you would fix it without losing the ability to debug.hardAlso on metrics5 min
- Your headline metric went up and the business got worse. How does that happen, and how do you catch it?hardAlso on metrics4 min