Your test suite takes fifty minutes and every merge waits for it. How do you fix that?
Profile before cutting: suite time concentrates in a few tests, and the cause is usually repeated setup or real waiting rather than work. Parallelise what is independent, move the slow and rarely-failing out of the merge path, and treat deleting redundant tests as legitimate.
What the interviewer is scoring
- Whether the candidate profiles the suite before proposing changes
- That per-test setup cost is distinguished from the work the test does
- Does the candidate address test independence before proposing parallelisation
- Whether moving tests out of the merge path is paired with a stated risk accepted
- That flaky tests are treated as a throughput problem and not only a correctness one
Answer
Why fifty minutes is worse than it sounds
Before optimising, be clear about what the number costs, because it is not fifty minutes.
A developer who cannot merge for fifty minutes does not sit and wait. They start something else, and the context switch back costs more than the wait did. Reviews queue up behind it. Changes get batched together to amortise the wait, so when the pipeline does fail it is failing on three changes at once and someone has to work out which. And because the loop is long, people stop running tests locally and let CI find out, which pushes even more traffic through the slow path.
There is also a flakiness multiplier that candidates rarely mention. With a one-in-fifty per-test failure rate and a suite of a few hundred tests, a meaningful fraction of runs fail for no reason, and each of those costs another fifty minutes. Fixing the top few flaky tests can reclaim more wall-clock time than any amount of optimisation, and it is usually the cheapest available fix.
Profile first
Suite runtime is almost never spread evenly. The usual distribution has a handful of tests accounting for a large share of the total, and the reason they are slow is usually not the work they do.
Look for three things. Per-test setup: a suite that starts a container, migrates a schema, or boots the application context once per test class is paying that cost hundreds of times, and the fix is to share it — one container for the suite, transactional rollback between tests rather than a rebuild. Real waiting: sleep calls inserted to stabilise a timing-dependent test are pure cost, and replacing them with a wait on the actual condition often removes seconds each. Accidental scope: a test that boots the whole application to exercise one validation rule, usually because it was easier to copy the neighbouring test than to find the seam.
These three routinely account for most of the runtime, and none of them requires deciding which tests to sacrifice.
Parallelise, once the tests allow it
Splitting across N machines is the biggest single lever and it has one prerequisite: the tests must be independent. Most suites are not, and finding out the hard way produces failures that look like flakiness and are actually contention.
Shared mutable state is the blocker. Tests that write to the same database rows, that assume an empty table, or that depend on execution order will fail intermittently once they run concurrently. The options, roughly in order of cost: a schema or database per worker, which is clean and needs the setup to be scriptable; a transaction per test rolled back at the end, which is cheap but does not work for anything that commits; or namespacing every test's data so tests cannot see each other's rows, which is the most work and the most robust.
Sharding by historical duration rather than by file count matters more than it sounds. An even split of files gives you one worker finishing in three minutes and another in twenty, and the suite takes twenty.
Move things out of the merge path
Not every test needs to gate a merge. The question for each group is what it costs to find that failure an hour later instead of now.
The tests that stay are the ones that fail often enough to catch real mistakes and are fast: unit tests, contract tests, a thin slice of integration coverage over the paths most changes touch. The tests that can move are the ones that are slow and rarely fail — full browser journeys across every supported browser, long-running data migrations, exhaustive matrix runs across versions.
Be explicit that this is accepting risk rather than eliminating work. Something will get merged that a nightly run then catches, and the team needs an agreed response for that — who looks at the nightly result, and whether a failure reverts or rolls forward. A nightly suite nobody reads is worse than no suite, because it produces the appearance of coverage.
Deleting tests is allowed
The option teams resist. A suite that has grown for years contains tests that assert the same behaviour several times over, tests for features that no longer exist, and tests so coupled to implementation that they fail on every refactor while catching nothing.
The question is not whether a test passes but what it would tell you if it failed. A test whose failure always means "someone renamed a method" is a tax on refactoring, and removing it increases the amount of real testing the team can afford. Coverage percentage is a poor guide here — three tests through the same branch add coverage of zero and runtime of three.
Do this deliberately and in review, not unilaterally, and pair it with the profile so the argument is about a named test that costs ninety seconds and duplicates another.
What to aim for
Ten minutes is the usual target for a merge-gating pipeline, and it is chosen because it is roughly the longest a person will wait rather than switch tasks. Getting there is normally the same sequence: fix the flakiest tests, share the expensive setup, remove the sleeps, parallelise, then move the slow tail out of the path.
Whatever you achieve, track it. Pipeline duration and flake rate belong on a dashboard the team sees, because suites do not become slow in a single change — they gain forty seconds a month and nobody notices until it is fifty minutes again in a year.
Profile before you cut. The fifty minutes is usually a handful of tests paying setup costs over and over, and fixing that costs nothing in coverage.
Likely follow-ups
- You want to parallelise but tests share a database. What are your options?
- Which tests would you move to a nightly run, and what do you lose?
- How does a flaky test at one failure in fifty affect a fifty-minute pipeline?
- When is deleting a test the right answer?
Related questions
- What is the test pyramid, and is it still the right model for your system?mediumAlso on flakiness and feedback-loops4 min
- Which tests should run on every commit and which should run nightly, and how do you decide?mediumAlso on parallelisation6 min
- Two teams own two services that talk to each other. How do you stop one breaking the other without running both together in a shared environment?mediumAlso on ci4 min
- How do you decide what to automate and what to leave manual?mediumAlso on flakiness6 min
- The operations team says there is no rule for this, they just use judgement. How do you model that?hardSame kind of round: design6 min
- You have been handed a scope and a budget, and the budget cannot buy the scope. What do you put in front of the customer?hardSame kind of round: design5 min
- The business says the system must be highly available. How do you turn that into a number, and what does that number cost?mediumSame kind of round: design4 min
- How do you decide whether to use a managed service or self-host a component, and which cloud costs catch teams out?hardSame kind of round: design6 min