Which tests should run on every commit and which should run nightly, and how do you decide?

Start from a budget, not from a taxonomy

The wrong way to answer is to recite the pyramid back with pipeline stages attached. The decision is economic. Every commit pays the cost of the blocking stage in developer waiting time, multiplied by the number of commits per day, and past roughly ten minutes people stop waiting for the result. They context-switch, come back later, and the feedback loop the pipeline exists to provide is gone. So fix a budget first and then decide what fits inside it.

Three properties qualify a test for the blocking stage, and all three are required. It has to be fast enough that the whole stage stays inside the budget. It has to be deterministic, meaning a red result is caused by the commit and nothing else. And it has to be diagnosable by the developer who broke it without them needing access to an environment, because a failure they cannot understand gets retried rather than read.

That admits unit tests, component tests that spin their own dependencies in-process or in a container, contract tests against a recorded pact, static analysis and lint, a build and a container image build, and a very small number of end-to-end journeys — the ones where the value is proving the pieces are wired together at all. Somewhere between one and five, not fifty.

Everything else runs on a schedule or after deployment. The full cross-browser end-to-end pack, mobile device-cloud runs, soak and load tests, mutation testing, accessibility sweeps, dependency and licence scanning, data migration rehearsals against a production-shaped dataset, and long-running batch or reporting flows. Two extra categories are worth naming because candidates rarely do: post-deploy smoke tests, which run against each environment immediately after a deployment and are the fastest way to catch configuration that only exists there, and a small suite that runs against production continuously as synthetic monitoring.

The condition attached to the scheduled runs is the part that separates a real answer from a diagram. A nightly suite with no named owner and no rule about what happens when it fails will be red within a month and ignored within two. So it needs a triage rota, a rule that a failure is a ticket the same working day, and a distinction in the report between failures that are new and failures already known, because a wall of thirty reds nobody can decode gets closed.

Parallelism buys you the budget, and independence buys you parallelism

Once the budget binds, the only way to fit more in is to run more at once. Wall-clock time for a sharded stage is the duration of the slowest shard, which is why sharding well matters more than sharding at all: four shards of 3, 3, 3 and 14 minutes is a 14-minute stage, and you have bought almost nothing while quadrupling your CI spend.

So balance on measured history rather than on the file list. Record per-test durations from previous runs, sort descending, and assign greedily to whichever shard currently has the least work. Most modern runners support this directly, and where they do not it is thirty lines of build script. Report per-shard duration in the pipeline output, because a shard that has drifted to double the others is a leading indicator that a test has started sleeping somewhere.

The prerequisite is genuine test independence, and parallelism is what exposes its absence. A suite that passes serially and fails at eight-way concurrency was always broken; it was relying on execution order, on a shared database row, or on a fixed port. That is worth saying plainly in an interview, because the common instinct when parallelism goes red is to reduce concurrency, which hides a defect that will resurface as flakiness later.

Isolation is the mechanism. Give each shard its own database schema or its own containerised dependency, its own message-queue namespace or topic prefix, and randomised ports. Where a genuinely shared resource cannot be split, gate it behind an explicit lock rather than hoping, and accept that it caps your concurrency.

Test data has to be created by the test that needs it

Three approaches exist and only one of them survives contact with a pipeline.

Relying on data that already exists in the environment is the failure mode. It works for a week, then someone runs a suite that deletes the account your test logs in with, or the data ages out and the "expiring soon" test starts passing for the wrong reason, and now your suite fails for reasons unrelated to any commit. This is the single largest source of pipeline flakiness in most organisations.

Restoring a known snapshot before each run is better and is sometimes the only option for a legacy schema, but it serialises everything against one dataset, so it fights parallelism and takes minutes off your budget.

Building what you need at the start of the test, through the application's own API or a factory helper, is the approach to argue for. Each test creates its own user with a unique identifier, sets up exactly the state its assertion depends on, and does not care what else lives in the database. Combine that with unique keys derived from a run identifier so two concurrent shards cannot collide, and cleanup by dropping the whole namespace afterwards rather than by deleting rows one at a time, which is a teardown that will itself fail one day and leave the environment dirty.

Two details make this practical. Seed the reference data that is genuinely static — country lists, product categories, feature configuration — as part of environment provisioning, not per test. And never let a test depend on data another test created, even when it would be convenient, because that is the same order dependency as before wearing a different coat.

Reporting a developer will read

The measure of a pipeline's reporting is how long it takes a developer who has just been told their build is red to find the assertion that failed. If that is more than about two clicks, they will rerun the build instead, and every rerun is a vote of no confidence you cannot get back.

Emit machine-readable results — JUnit XML is the near-universal format — so the pipeline renders a test tab rather than a log file. Attach the artefacts that explain the failure to the failure itself: a Playwright trace or a screenshot and DOM snapshot for a UI test, the request and response bodies for an API test, the container logs for the run. Annotate the pull request with the failing test names so the information arrives where the review conversation is happening, rather than in an email.

Then track two things over time that individual runs cannot show you. Duration per stage and per shard, so growth is visible before it breaches the budget. And failure rate per test across runs on the main branch, which is the only reliable way to identify flaky tests — a test that fails four percent of the time is invisible in any single run and obvious in a fortnight of history. Publish that list, quarantine off the blocking path with an expiry date, and delete tests that nobody fixes.

A gate people do not trust is worse than no gate

The property that makes the blocking stage worth its cost is that red means "your change broke something". Once that stops being true, the cost stays and the benefit goes: developers rerun on red by reflex, and the genuine regression the gate eventually catches gets rerun too.

This is why the instinct to strengthen a gate by adding more tests to it is usually wrong. Adding a suite that depends on a shared environment, a third-party sandbox, or a nightly data load moves the gate's false-failure rate up, and the team's response to a false failure is not investigation, it is a retry policy — which then also swallows real intermittent product bugs. The right move when quality is escaping is almost always to add tests at a cheaper, more deterministic layer, or to add them to the scheduled run with an owner, and to keep the blocking stage small enough and clean enough that going red still means something.

The same argument disposes of coverage-percentage gates. A threshold that blocks a merge is trivially satisfiable by tests that execute code without asserting anything about it, so it changes behaviour without changing quality, and it will one day block an urgent fix for a reason nobody can defend.