Your A/B test was configured for a 50/50 split but the data shows 53/47 across two million users. The result is significant and positive. What do you do?

Short answer

Do not report the result. A 53/47 split on two million users is not noise — under a true 50/50 assignment the probability of a deviation that large is effectively zero. Something removed or misassigned users in a way that was not random, and whatever did that is very likely correlated with the metric you are measuring. The experiment is invalid, and the job is to find the mechanism rather than to explain the imbalance away.

Why the size of the sample settles it

Intuition fails here, so do the arithmetic out loud. Under a fair coin with n = 2,000,000, the count in each arm has a standard deviation of about sqrt(n × 0.5 × 0.5) ≈ 707. The observed split is 1,060,000 versus 940,000 — a deviation of 60,000 from the expected 1,000,000, which is roughly 85 standard deviations.

A chi-square goodness-of-fit test returns a p-value so small it underflows to zero in most tooling. There is no interpretation under which this is chance. This is worth being precise about because "53/47 sounds close to even" is exactly the reasoning that lets a broken experiment ship — the percentages look reassuring and the absolute numbers do not.

Why it invalidates the result rather than annoying you

The dangerous assumption is that losing users is like taking a smaller random sample, which would cost power but not correctness. That is almost never what happened. Users are missing because of a mechanism, and mechanisms select on user characteristics.

Suppose the variant ships a heavier JavaScript bundle and a fraction of users on slow connections abandon before the exposure event fires. Those users are now absent from the treatment arm — and they were also, systematically, your least engaged and lowest converting users. The treatment arm is now measuring a cleaner, faster, more committed population than the control arm. Conversion goes up, the test reports a win, and the effect is entirely an artefact of who is missing.

The direction is unpredictable, which is the point. The same test could just as easily lose your heaviest users and show a spurious loss. A significant result alongside an SRM tells you nothing about the feature, not even its sign.

Where the users actually go

Diagnosis is a matter of measuring the ratio at each stage and finding where 50/50 becomes 53/47.

Assignment. A hash bucketing function that is not uniform, a modulo over an id space with structure, a sticky assignment that falls back to control when a cookie is unreadable, or a variant that errors during initialisation and silently defaults users to control.

Exposure logging. The most common cause in practice. If the variant logs exposure after a component renders and the control logs it earlier, any difference in load time, crash rate or interactivity becomes a difference in logged users. Late assignment — bucketing at the point the feature is reached rather than at session start — has the same effect.

Filtering downstream. Bot and fraud filters applied after assignment are a frequent culprit, because bots do not distribute evenly across arms if the variant changes the page structure the bot was matching. Deduplication logic, session stitching, and "exclude internal traffic" rules all do this too.

Infrastructure. A redirect-based test where the variant carries an extra hop loses users to the latency of that hop. Different CDN or cache behaviour per arm produces the same asymmetry.

The trigger condition. If the experiment only counts users who reach a particular screen, and the variant changes how easily that screen is reached, the population is being selected by the treatment itself.

The practical method is a funnel: ratio at assignment, at first exposure event, after bot filtering, after dedup, in the final analysis table. The stage where it breaks names the cause, and this is far faster than reasoning about it in the abstract.

What not to do

Do not reweight. Scaling the smaller arm up to restore a 50/50 ratio assumes the missing users are exchangeable with the present ones, which is precisely the assumption the SRM tells you is false. It produces a number that looks corrected and is not.

Do not trim the larger arm. Randomly dropping 60,000 users from control restores the ratio and leaves the treatment arm just as biased as it was. You have made the symptom disappear.

Do not ship it with a footnote. An SRM is not a limitation to disclose alongside the result; it means there is no result. Presenting "the variant won, though we had an SRM" invites the reader to average the two, and readers reliably keep the win and discard the caveat.

The narrow exception is when you can prove the mechanism is orthogonal to the metric — for instance, an outage that took one arm's logging offline for a bounded window, where you can excise that window cleanly and confirm the ratio holds everywhere else. That is a high bar and needs the mechanism identified, not assumed.

Making it a gate rather than a discovery

SRM is cheap to detect and catastrophic to miss, so it belongs in the pipeline rather than in an analyst's judgement. Run a chi-square test against the intended allocation on every experiment readout and refuse to render the results if the p-value falls below a threshold — 0.001 is a common choice, deliberately strict because the cost of a false alarm is an investigation and the cost of a miss is a wrong product decision.

The strongest version also runs the check per segment: by platform, by browser, by country, by day. An experiment can be balanced overall while being badly skewed on iOS, which both localises the bug and catches cases where two opposing biases cancel out in the aggregate. Proposing the automated gate rather than only the manual diagnosis is usually what distinguishes someone who has been burned by this from someone who has read about it.

---

© 2026 Preptima. Originally published at preptima.com.