Your experiment dashboard says the variant is significant on day three of a fourteen day test. Can you ship it?

What a fixed-horizon test actually promises

A conventional test fixes the sample size in advance from the effect you want to detect, the baseline rate, and the error rates you will accept. The five percent significance threshold is then a promise about one specific procedure: collect exactly that sample, compute the statistic once, and reject the null if it crosses the threshold. Under that procedure, the chance of a false positive is the nominal one.

Looking at the result every morning is a different procedure. Each look is another opportunity for random fluctuation to cross the line, and because you stop the moment it does, the overall chance of stopping in error is the chance of crossing on look one, or look two, or any later look. The more often you look, the closer that combined probability creeps towards certainty, and in the limit a continuously monitored test with no fixed horizon will eventually show significance for two identical variants. The threshold is unchanged, but what it means has silently become something much weaker than five percent.

So the direct answer to the question is that the p-value on the dashboard is not the quantity you think it is, and it is not evidence that the variant works. It is evidence that the difference crossed a line at least once in three days.

The number you would ship is inflated, not just uncertain

The subtler half, and the part that separates a strong answer, is what early stopping does to the estimated effect rather than to the error rate. Stopping when the observed difference is large enough to be significant selects on the observed difference being large. Among all the tests you stop early, you have preferentially picked the ones where noise happened to run in the variant's favour, so the measured lift is systematically higher than the true one.

This has a practical consequence beyond statistical hygiene. The number from a test stopped on day three is the number that goes into the roadmap justification, the forecast and the promotion packet, and it will not be reproduced when the feature is fully rolled out. A team that ships early repeatedly ends up with a portfolio of features whose claimed gains sum to far more than the actual movement in the company's metrics, and no single experiment looks wrong.

Day three is specially untrustworthy, for reasons that are not statistical

Even if the arithmetic were sound, three days of data has structural problems that fourteen days is designed to average out.

Behaviour is weekly. Traffic mix, basket size, conversion rate and the ratio of new to returning users all differ between a Tuesday and a Saturday, and a test whose window does not contain whole weeks measures the variant plus whatever days it happened to cover. Second, a visible change produces a novelty effect: existing users click the new thing because it is new, and the effect decays. The mirror image, where users are briefly disrupted by a changed interface before adapting, is just as real and points the other way. Both are largest at the start, which is exactly the window an early stop uses. Third, in a test that assigns users on their first exposure, the population on day three is skewed towards the most frequent users, who are not the population you are inferring about.

None of these is fixed by more data on the same three days. They are fixed by duration, which is why the honest framing of a test length is not only "enough samples" but "enough time for the effect to stabilise and for the exposed population to be representative".

If you genuinely need to stop early, design for it

The mature version of this answer is not that peeking is forbidden. It is that continuous monitoring is a legitimate requirement and there are methods that support it, provided you commit to them before the test rather than after seeing a promising number.

A group-sequential design declares a small number of interim analyses in advance and spends the error budget across them, using a stricter boundary at early looks than at the final one, so an early stop requires a genuinely large effect and the overall error rate is preserved. Alpha-spending functions are the general form of that idea. Always-valid sequential methods go further and allow monitoring at any moment, at the cost of wider intervals for the same data, which is the price of the flexibility. Bayesian analyses avoid the specific problem of repeated significance testing, but they do not make early stopping free: the posterior at day three still reflects three days of a novelty effect and a skewed population, and stopping rules based on posterior thresholds still need their operating characteristics checked.

Whichever you pick, the invariant is that the stopping rule is part of the design, written down with the metric and the horizon before data exists, and not something the person who wants to ship gets to choose on the morning the dashboard looks good.

The looks that are always fine

It is worth distinguishing the peeking that corrupts an inference from the monitoring that is simply operational, because a blanket rule against looking is the wrong lesson and interviewers listen for that.

Checking that assignment is working is not an inference about the treatment effect. Sample-ratio mismatch, error rates, latency, crash rates and obvious breakage should be watched from the first hour, and a serious guardrail regression is a reason to stop the test regardless of what any significance calculation says, because you are not testing a hypothesis at that point, you are containing harm. The asymmetry is defensible: stopping early for harm risks abandoning a good variant, which costs you one experiment, while stopping early for success risks shipping a nonexistent gain, which costs you the credibility of every number the team publishes.

The line, then, is between looking to decide whether the experiment is valid and looking to decide whether the variant won. The first is required. The second is what the horizon and the pre-registered stopping rule exist to protect.

The p-value on day three is not the false-positive rate you agreed to, and the lift is biased upwards by the act of stopping. Either hold the horizon, or adopt a sequential design chosen before the test started rather than after the dashboard turned green.