Your A/B test came back not significant. What do you do next?

A null result is not a finding until the test could have found something

The first thing to establish is whether the experiment was ever capable of detecting the effect you were hoping for. A test that lacked the power to see the lift you cared about produces a non-significant result almost regardless of the truth, and treating that as evidence the change does nothing is the single most common error in the whole area.

Do the arithmetic out loud. The usual approximation for a two-arm test at 80 per cent power and 95 per cent confidence is that you need roughly sixteen times the outcome's variance divided by the square of the effect you want to detect, per arm. Take a baseline conversion of 5 per cent and suppose the change was worth running if it produced a 10 per cent relative lift, so 5 per cent to 5.5 per cent, an absolute difference of 0.5 percentage points. The variance of a 5 per cent binary outcome is 0.05 times 0.95, which is 0.0475. Sixteen times that is 0.76, divided by 0.005 squared, which is 0.000025, gives about 30,000 users per arm.

Now compare that with what you had. If the surface receives 20,000 visitors a week split evenly, each arm gained 10,000 a week, so the test needed about three weeks and was run for two. It was underpowered by roughly a third and the non-significant result carries almost no information. The correct next step there is not interpretation, it is finishing the test — and the lesson is that the sample-size calculation belonged before the launch, not after the disappointment.

Read the interval, not the verdict

Once the test was adequately powered, stop looking at whether the p-value crossed a threshold and look at the range of effects the data is consistent with. Two non-significant results can mean completely different things. An observed lift of 0.2 per cent with an interval from minus 0.4 to plus 0.8 per cent says the effect, if any, is too small to matter, which is a genuine and useful finding. An observed lift of 4 per cent with an interval from minus 1 to plus 9 per cent says you have learnt very little, because your original hypothesis of a 10 per cent relative lift sits comfortably inside the range you cannot rule out.

This is the distinction between absence of evidence and evidence of absence, and it decides what you do next. Only the tight interval licenses the sentence "this change does not work".

flowchart TD
  A[Non significant result] --> G[Check assignment and instrumentation]
  G --> B{Was the test powered for the target effect}
  B -- No --> C[Finish the test or test a bigger change]
  B -- Yes --> D{Is the interval tight around zero}
  D -- Yes --> E[Accept no effect and revisit the assumption]
  D -- No --> F[Inconclusive so decide on cost and strategy]

The interesting branch is the one on the right-hand side, where a properly run test still refuses to answer — that case is decided by judgement rather than by data, and pretending otherwise is where teams get into trouble.

The temptations, and why each one is a defect

Three moves feel productive and are not. Extending a test that has already reached its planned sample and peeking until it turns significant inflates the false-positive rate, because every additional look is another chance for noise to cross the line. Slicing into segments after the fact until one is significant does the same thing with more dimensions available; with enough cuts something always clears the threshold. And relaxing the threshold after seeing the result converts a decision rule into a rationalisation.

None of this means segments are forbidden. It means the segment has to have been named in advance as a hypothesis with its own power calculation, or else treated as a lead to be tested rather than a result to be shipped. Saying this unprompted is one of the clearest markers of someone who has run experiments rather than read about them.

Send it back to the assumption

The reason this question sits in discovery rather than analytics is that the experiment was testing a belief, and the belief is what needs updating. Work back through the chain: you assumed users were dropping out at this step because of some cause, that your change addressed that cause, and that the effect would be large enough to see. A null result could falsify any of the three. If the change was shown to the intended users, was noticed by them, and still moved nothing, the diagnosis of the cause is the most likely thing to be wrong, and the answer is more qualitative work rather than another variant of the same idea.

So the practical next step is often to check whether the intervention was even perceived. Look at whether the treatment group interacted with the changed element at all. A redesign that 8 per cent of the treatment group ever scrolled to has not been tested; it has been hidden, and that is an implementation finding rather than a hypothesis finding.

Where all of this actually goes wrong is the direction of the pressure. Nobody wants to hear that a quarter of work produced no measurable effect, and there is always a reading of the data that suggests otherwise. The candidate who earns the round is the one who says what the interval permits, names the cost already sunk as irrelevant to the decision, and puts the honest conclusion on record — including that a large fraction of well-designed experiments genuinely do nothing, which is why running them is worthwhile.

An underpowered null is silence, not a no; decide what the interval permits before you decide what to build.