Walk me through designing an A/B test for a checkout change. What do you fix before any data exists?

Write the decision down while you still cannot see the answer

The reason to fix everything up front is not procedural tidiness. Once you have seen the data, every remaining choice - which metric, which window, which users to exclude, when to stop - can be made in the direction of the result you hoped for, usually without noticing. A pre-registered design removes that freedom, and it is the only version of the analysis whose stated error rate is the real one.

So the document written before launch names six things. The hypothesis, phrased as a mechanism and a direction, not a wish: removing the address-confirmation step reduces abandonment between payment entry and order placement, because the step is where the session is lost. The primary metric, singular, chosen to reflect that mechanism. The guardrail metrics you will not trade away regardless of the primary - checkout latency, payment failure rate, refund rate. The unit of randomisation. The analysis window and any exclusions, decided in the abstract. And the stopping rule.

The single-primary-metric discipline matters more than it looks. Five metrics each tested at the 5% level give you five chances to find something, and something usually is there; name one primary and demote the rest to hypothesis generation for a future test rather than evidence from this one.

Randomisation unit, and the variance mistake that follows

Randomise by user, not by session, when the change plausibly affects behaviour beyond one visit. A session-randomised checkout test shows the same person both variants, which contaminates the comparison and produces an experience nobody designed.

The consequence people miss is statistical rather than experiential. Randomise by user and then analyse at session or order level, and the rows are not independent - one heavy user contributes fifty correlated rows. Standard errors computed as though they were independent come out too small, intervals too narrow and p-values too optimistic, so the test manufactures significance from nothing. Either analyse at the unit you randomised, aggregating each user to one number, or use a variance estimator that accounts for clustering. Naming this is one of the clearest markers of someone who has run experiments rather than read about them.

Sizing it, with the arithmetic in the open

The design has to state, before launch, what effect it is powered to detect. For comparing two conversion rates the per-variant sample size is approximately

n ≈ 2 (z_1-α/2 + z_1-β)² p(1 - p) / δ²

and every term needs an assumption you supply. Take a two-sided α of 0.05, so z_1-α/2 = 1.96, and 80% power, so z_1-β = 0.84. Their sum is 2.80 and the square is 7.85. Now suppose - and this is an assumption about your product, not a fact about anyone's - a baseline conversion of p = 0.04, and that the smallest lift worth shipping for is 5% relative, meaning δ = 0.002 in absolute terms. Then p(1 - p) = 0.04 x 0.96 = 0.0384, and

n ≈ 2 x 7.85 x 0.0384 / 0.002² = 0.6029 / 0.000004 ≈ 150,700 per variant

so roughly 301,000 users in total. At 20,000 eligible users a day that is 301,000 / 20,000 ≈ 15 days, which is the number the design commits to. Note also what the δ² denominator does: halving the detectable effect to 0.001 multiplies the requirement by four, to about 600,000 per arm and two months of traffic. That is why "what is the smallest effect worth acting on" is a commercial question asked before the test, not a statistical one settled after it.

Round up to whole weeks rather than running the bare number of days, because weekday and weekend traffic behave differently and a 15-day window weights one part of the cycle twice. And if the primary metric is revenue per user rather than a proportion, the same structure holds but the variance must be estimated from history - and heavy-tailed revenue makes it large, which is why a conversion-primary test with revenue as a secondary is usually what gets run.

Why peeking costs you

A fixed-horizon test allocates its error budget to one decision, taken at the pre-declared sample size. That is what the 5% means: if the null is true, one such test wrongly rejects it 5% of the time.

Each interim look adds another opportunity for the statistic to be over the threshold at that moment. The looks are correlated, since day 8 contains day 7's data, so you cannot simply multiply - but the probability that at least one look in a series shows p < 0.05 is strictly greater than 5%, and it rises with the number of looks. Quoting a specific inflated figure without stating the look schedule is repeating a number rather than reasoning; the mechanism is what you should be able to explain. A test statistic wanders, an early sample wanders more because it has less data to steady it, and a rule that stops the moment it crosses a line will fire on noise.

There is a second cost that survives even when a real effect exists. Stopping early because the result looks good selects the moment the estimate happened to be high, so the reported effect size is biased upward - and the roadmap decision downstream is made on an inflated number even though the direction was right.

The legitimate answers are designs that budget for multiple looks rather than discipline alone. Group sequential designs with alpha-spending boundaries (O'Brien-Fleming, Pocock) allocate the error rate across a planned schedule of interim analyses, with a stricter threshold early on. Always-valid sequential methods let you monitor continuously by construction, at the cost of requiring more data for the same power than a fixed-horizon test that runs to completion. Both are fine; choosing one before launch is the requirement. What is not fine is a fixed-horizon design watched daily and stopped on a good day.

Checks that come before you read the result

Before interpreting anything, verify the assignment. Sample ratio mismatch compares the observed split against the intended one, and a significant deviation means randomisation or logging is broken - a bot filter applied to one variant, a redirect that drops slow connections, an SDK that fails to log for one arm. A failing SRM does not weaken the result, it makes it uninterpretable, and the action is to find the cause and rerun. Also confirm no other release landed mid-test and that the exclusion rules applied are the ones you wrote down.

When it comes back inconclusive

An inconclusive test does not say the change had no effect. It says the data cannot distinguish the effect from zero at this sample size, which is a statement about your test as much as about your change. So stop reporting the p-value alone and report the confidence interval against the minimum effect you declared, which gives three genuinely different situations.

If the interval is narrow and sits entirely inside the region you called too small to matter, you have learned something real: effects worth acting on have been ruled out. Ship whichever version is cheaper to maintain and move on with confidence. If the interval is wide and comfortably contains your minimum useful effect, the test was underpowered and you learned nothing about the hypothesis - the honest options are to extend it to the sample size the design called for, reduce variance (trigger analysis on users who actually reached checkout, or use a pre-period covariate), or drop the idea because it is not worth the traffic. If a guardrail interval excludes zero in the wrong direction, act on the guardrail regardless of the primary result; that is the point of having declared them.

What not to do is hunt through segments until one is significant. A post-hoc segment finding is a hypothesis at best - label it as one, and if it matters, design a confirmatory test in which that segment is the pre-registered population.

Everything that makes an experiment trustworthy is decided while the answer is still unknown: one metric, one randomisation unit, one sample size, one stopping rule. Analysis after the fact can only spend the credibility the design established.