Your test suite fails randomly about one run in five. How do you get it under control?

Stabilise the signal before you debug anything

The instinct is to start fixing tests. Resist it for a week, because you cannot debug reliably inside a pipeline that is already red for reasons you have not catalogued, and the team is making merge decisions today. Get the pipeline back to meaning something first; find out which tests are costing you second.

So record every test execution, not every run: identity, outcome, duration, worker, commit, and whether the same commit passed elsewhere. Parsing JUnit XML into a table is an afternoon's work if CI cannot emit it directly. Without that data you rely on people's memory of which tests "are always like that", which points at the loudest test rather than the most expensive one. With it you can rank by per-test flake rate: the share of executions that disagree with another execution of the same commit.

The arithmetic explains why that ranking is usually a shock. In a 500-test suite failing one run in five, evenly spread independent failures would mean a per-test pass rate p with p^500 = 0.8, so p ≈ 0.99955 — one failure in roughly 2,200 executions each. Real suites never look like that; five or ten tests produce most of the noise, so the problem is smaller than it feels.

Then quarantine, deliberately and visibly. Move the top offenders into a separate suite that still runs and still reports but cannot block a merge. Quarantine is neither deletion nor forgiveness: each test in it gets an owner, a ticket, and an expiry date after which it is fixed or deleted, and the list is capped so adding one forces a conversation. The blocking suite becomes trustworthy again immediately, which buys you room to do the real work.

The root-cause families, because they need different fixes

Waiting on the wrong thing is the largest family in any UI suite. A fixed sleep encodes an assumption about machine speed that stops holding the moment CI gets busier, and it asserts nothing about readiness. An implicit wait is subtler: it retries element location only, so it does nothing for the commoner case where the element exists but its contents have not been repainted. Mixing implicit and explicit waits in one Selenium session is documented as producing unpredictable timeouts and should never be done.

// Flaky: two seconds is tuned to a laptop, and nothing here waits for
// the state the assertion depends on.
driver.findElement(By.id("place-order")).click();
Thread.sleep(2000);
assertEquals("Order placed", driver.findElement(By.id("status")).getText());

// Deterministic: wait for the condition itself, with no implicit wait
// set on the driver. The timeout is a ceiling, not a delay.
new WebDriverWait(driver, Duration.ofSeconds(10))
    .until(ExpectedConditions.textToBe(By.id("status"), "Order placed"));

Frameworks with retrying assertions, such as Playwright's expect(locator) family, close this hole by retrying the whole assertion rather than the lookup, which is why migrations often drop the flake rate without anybody fixing a test.

Shared mutable test data produces the failures nobody can reproduce locally. Two tests use the account with email test@example.com, one updates its address, and in parallel they interleave. The fix is ownership rather than more locking: each test creates its own fixtures under a unique key and never asserts on a global count. Any assertion of the form "the list has three rows" is a shared-state bug waiting for a second worker.

Test interdependence and ordering is related but distinct, because here the data is fine and the sequence is load-bearing. Test A logs in, test B assumes a session, and B passes only because A ran first. That hides indefinitely while the runner is deterministic and detonates the day someone enables sharding. Randomising order on purpose, with the seed printed in the output, converts a lurking dependency into a reproducible failure — which is what you want, even though it looks for a fortnight like you made things worse.

Time and timezone failures arrive on a schedule. A test computing "yesterday" fails across a month boundary, one asserting a formatted timestamp fails when CI runs in UTC and the developer runs in IST, and anything near midnight fails for whoever is unlucky. The fix is not a tolerance window; it is that production code takes an injected clock instead of calling the system clock, so a test can pin the instant — java.time.Clock with Clock.fixed in Java, an equivalent in most stacks, and browser-level clock control in the newer runners.

Animation flakiness looks like a mis-click and is really a moving target: the element is located, the modal finishes sliding in, and the click lands on whatever is now under the pointer. Waiting for visibility does not help, because a transitioning element is visible throughout. Wait for a stable position or for the state that follows the animation, or disable transitions in the test environment with a CSS override, which removes the class of failure at a cost you can state.

Network and third parties mostly should not be in the suite at all. A payment sandbox, an analytics beacon and a map tile server will each be slow or down at some point in your test window, and none is the behaviour under test. Stub them at the network boundary and keep one separate, non-blocking job exercising the real integration on a schedule, so a provider outage tells you about a provider outage instead of failing forty unrelated tests.

What a retry policy may and may not do

A blanket "retry every failed test twice" is the most damaging option available, because it is indistinguishable from a fix while being the opposite of one. The failure mode is specific: a race condition in the product — a double-submit that occasionally creates two orders, a cache that occasionally serves stale data — presents exactly as an intermittently failing test. Retry until green and you have taught the pipeline to suppress the only evidence of a real defect, in the category of bug hardest to find any other way. Blanket retries also hide a steadily worsening test, which survives invisibly from 5% failure up to 40%.

A defensible policy is narrow, applying to named tests or a named infrastructure failure class rather than everything; logged as a first-class event, so a test that passes on retry is recorded as flaky and feeds the ranking that drives quarantine; budgeted, so exceeding a retry rate fails the build in its own right, because a suite that needed thirty reruns to go green has not gone green; and temporary by construction, attached to a ticket, because the retry is the mitigation and the fix is still owed. Retrying the whole suite is worse than retrying a test, since it multiplies runtime and destroys the per-test evidence.

The part that is not a technical problem

One-in-five matters more than the number suggests because it changes behaviour. Once a red build is more likely to be noise than a real regression, the rational move for any individual engineer is to press rerun without reading the output, and that habit spreads faster than the flakiness does. From then on the suite consumes CI spend, review latency and attention while protecting nothing, and the genuine regression it eventually catches gets rerun and merged like everything else. A suite nobody trusts is worse than no suite, because with no suite the team knows it has to test by hand. That is why the metric you commit to here is the pipeline's false-failure rate rather than coverage, and why quarantine is right even when it feels like cheating: you trade a little coverage, temporarily and in writing, to restore the property that makes the whole investment worth anything.

Where candidates lose this one

Almost everyone can list waits and sleeps, and most stop there, which reads as having fixed a few tests rather than run a campaign. Two omissions separate the answers. The first is failing to say what evidence you keep from every failure — trace, video, DOM snapshot, request log, order seed, worker ID — so a one-in-twenty failure is diagnosable from the artefacts of the run that produced it rather than by trying to reproduce it. The second is assuming every flaky test is a test defect. Before rewriting a test that intermittently sees the wrong balance, you owe the product one question: could it genuinely produce the wrong balance? A fair share of "flaky tests" on a mature system are tests correctly reporting a non-deterministic product, and those are the highest-value bugs the suite will ever find.

Measure and quarantine first so the pipeline means something again, then fix by family — and never let a retry policy answer the question of whether the non-determinism lives in your test or in your product.