hardScenarioConceptMidSeniorStaff
Your A/B test came back not significant. What do you do next?
5 min readUpdated 2026-08-15
Answer
Short answer
Establish whether the test could ever have detected the effect you cared about, because an underpowered null tells you nothing; then read the confidence interval rather than the verdict and decide whether to extend, test a larger change, or accept that the assumption was wrong.
Was this answer helpful?
Likely follow-ups
- The interval runs from minus one per cent to plus nine per cent. Do you ship?
- Your designer wants to check whether it worked for mobile users specifically. How do you handle that?
- You only have 3,000 weekly users. What would you do instead of an A/B test?
- How would you tell a team that has spent a quarter on this that the honest reading is no effect?
Related questions
- How do you know a problem is real before you build anything to solve it?mediumAlso on discovery and assumption-testing5 min
- Your test lifts conversion, but something else moved the wrong way. What are guardrail metrics, and when do they stop a launch?hardAlso on statistical-power5 min
- Your experiment dashboard says the variant is significant on day three of a fourteen day test. Can you ship it?hardAlso on statistical-power5 min
- Tell me about a decision you had to make in minutes, with information you knew was incomplete.mediumAlso on decision-making6 min