QA and testing fundamentals

The ability to change the code. A system without tests is not one that has more bugs — it may have very few — it is one where every modification carries unknown risk, so the rational response is to stop modifying it, and that is how a codebase becomes something nobody will touch. Tests convert "I believe this still works" into "the machine checked", which is what makes refactoring, dependency upgrades and rewrites of internals affordable. That framing also tells you which tests are worth writing: the ones that will still pass after a change you intend to make, and fail after a change you did not intend.

Because tests mostly do not find bugs; they find regressions. A test written after the code passes on the first run almost by definition, and the defects that matter were usually never imagined by the person writing the assertions. What the suite does is fix the behaviour in place so that tomorrow's change cannot quietly alter it. Saying this out loud matters in an interview, because it leads to different decisions: you invest in tests around behaviour that is expensive to break, rather than chasing a percentage over code that nobody will ever edit again.

Each step exists to check something the other two cannot, which is why skipping one quietly removes the benefit.

flowchart LR
    R[Red<br/>write a failing test<br/>for behaviour you want] --> G[Green<br/>smallest change<br/>that makes it pass]
    G --> F[Refactor<br/>improve the structure<br/>while the test holds]
    F --> R
    R -.-> W[If it passes immediately<br/>the test proved nothing]

The red step is the only point at which you verify the test itself. A test that has never failed is an unverified instrument: it may assert on the wrong object, be silently skipped, or exercise a code path that was already correct. Watching it fail for the reason you expected is the whole reason the step is first.

The green step is deliberately ungenerous. Writing the smallest thing that passes keeps you from building machinery no test demands, and it exposes the next test you need, because the obviously-inadequate implementation tells you which case is still unspecified.

Refactor is the step teams drop, and dropping it converts TDD into a slow way of writing badly structured code with tests attached. The suite exists precisely so that this step is safe; not using it wastes the thing you paid for.

The dotted branch is the common failure in practice. If your new test is green first time, either the behaviour already existed — delete the test or keep it as documentation, knowingly — or the test is not wired to the code you think it is.

Not that it finds more bugs, which the evidence is equivocal about. The claim is about design: writing the test first forces you to use the interface before it exists, so awkward construction, hidden dependencies and untestable coupling show up as pain while they are still cheap to change. A class that needs six mocks to instantiate is telling you something about the class, and TDD makes you hear it in the first minute rather than the third week. The secondary claim is behavioural — it produces small steps and a working system at all times, which keeps debugging sessions short because the last change was tiny.

Which lines were executed while the tests ran. That is genuinely useful as a negative signal: a module at five per cent has no safety net, and a diff that adds forty uncovered lines is worth a comment in review. What coverage cannot tell you is whether anything was checked, because a test with no assertions covers every line it touches. So it is a floor detector, not a quality measure, and the moment it becomes a target it is gamed — by tests that call code and assert nothing, by excluding awkward files, and by testing getters to raise the average.

One function, one test, every line green, and a defect that reaches production.

int discountedPrice(int priceMinor, int percentOff) {
    if (percentOff > 100) percentOff = 100;      // covered
    return priceMinor - (priceMinor * percentOff / 100);   // covered
}

@Test
void appliesDiscount() {
    assertEquals(9000, discountedPrice(10000, 10));
}

lines   6/6    100%
branch  2/2    100%       <- the if was taken and not taken across the suite

The report is honest and useless. The test exercised both lines and one branch outcome, and the suite happens to cover the other elsewhere, so the tool has nothing left to complain about. Meanwhile percentOff = -50 returns a price higher than the input, and nothing in the code or the suite has an opinion about negative discounts.

The general shape of the gap: coverage measures execution, and correctness lives in the values. Line coverage cannot see the untested input, branch coverage cannot see the missing branch, and no coverage metric can see the assertion you did not write.

Two things do see it. Thinking in equivalence classes and boundaries — below zero, zero, one, ninety-nine, one hundred, above one hundred — would have produced the failing case in seconds. And mutation testing would have flipped the guard to < 100 or deleted it entirely, found the suite still green, and reported the line as covered but unprotected.

That is the distinction worth carrying into an interview: covered means visited, protected means a change here breaks a test. Only the second is what anyone actually wanted.

It deliberately introduces small faults into your code — flipping a comparison, negating a condition, replacing a return with a constant — and runs the suite against each mutant. A mutant the tests still pass is called survived, and it names a change to your code that no test would catch. That is the measurement coverage cannot give: not which lines ran, but which lines are actually defended. The cost is runtime, because it is your whole suite multiplied by the number of mutants, so it belongs on a nightly or weekly schedule over the modules where correctness matters rather than in the pull-request gate.

That the number of tests at each level should be inversely proportional to their cost and scope: many fast tests over small units, fewer over integrated components, very few driving the whole system. The reasoning is economic rather than aesthetic — a broad test is slower to run, slower to diagnose when it fails, and fails for more reasons unrelated to your change, so the same confidence bought lower down is cheaper to own. The pyramid is therefore a statement about feedback loops and maintenance cost, not a claim that unit tests find more defects.

The two columns move in opposite directions, and that opposition is the entire argument.

flowchart TD
    E[End-to-end<br/>a handful<br/>minutes per run<br/>proves the system works] --> I[Integration<br/>tens to hundreds<br/>seconds per run<br/>proves the seams work]
    I --> U[Unit<br/>hundreds to thousands<br/>milliseconds<br/>proves the logic works]
    U --> S[Static checks<br/>types, lint, compiler<br/>near free<br/>proves nothing about behaviour]

Read it as a budget. If the whole suite must finish inside ten minutes to stay in the pull-request gate, then every end-to-end test you add costs you dozens of integration tests you could have had instead, and the question is always whether this particular confidence is available more cheaply lower down.

The confidence column is why the pyramid cannot simply be flattened to its base. A thousand green unit tests are compatible with a completely broken application: every collaborator correct in isolation, wired together wrongly, with a misconfigured connection string that no unit test could reach. That is the argument for insisting on a thin top layer rather than none.

The cost column is why the top must stay thin. End-to-end tests fail for reasons that are not your bug — a slow environment, a third party, a stale fixture — and each such failure spends someone's afternoon and teaches the team to re-run rather than investigate, which is the point at which the suite stops being believed.

The bottom layer is the one candidates omit and it is the cheapest of all. A type system, a null-safety checker and a linter eliminate whole categories of test you would otherwise write by hand, at zero runtime cost, which is why "what does the compiler already guarantee" is a reasonable first question when deciding what to test.

That its base is defined by implementation structure, so a suite of heavily isolated unit tests ends up coupled to the shape of the code and resists exactly the refactoring tests were supposed to enable. The testing trophy and the testing honeycomb both respond by widening the middle: most tests should exercise a meaningful slice — a service with its real database, a component with real rendering — because that is where the cost-to-confidence ratio is genuinely best now that containers make real dependencies cheap. The pyramid was drawn when starting a database took minutes, and part of its shape is an artefact of that.

Because "unit" was never defined as "one class". The original sense is a unit of behaviour with an isolated test — no shared state with other tests — which may span several collaborating objects. The other school reads it as one class with every collaborator doubled. The two produce very different suites: the first survives internal refactoring and gives less precise failure locations, the second pinpoints failures and breaks whenever you move a method. Most arguments about the pyramid are actually this disagreement, so in an interview it is worth defining which you mean before defending a ratio.

The pyramid inverted: a large body of slow end-to-end and manual tests over a thin or absent unit layer. Teams arrive there honestly. Testing through the user interface requires no changes to the application, so it is what a separate QA function can do without developer cooperation, and each individual test is easy to justify. The result is a suite taking hours, failing intermittently, and too expensive to maintain, so coverage of new work quietly stops. The way out is not deleting it but pushing each existing scenario down: reproduce the same assertion at the lowest level that can hold it, then retire the broad one.

By what the system's risk actually is. A service whose complexity is business rules over simple storage wants a thick unit layer, because that is where the logic lives. A service that is mostly orchestration — five HTTP calls and a database write, with almost no branching — has nearly nothing to unit test and needs integration tests, because every interesting failure is at a seam. A user-interface-heavy product needs component tests with real rendering. So the honest answer to "what ratio should we have" is that the ratio is an output of where the logic and the risk sit, not an input you choose first.

Underneath everything, as the cheapest layer available. A non-nullable type removes every test you would have written for a missing value; an exhaustive switch over a sealed type makes the forgotten-case test unnecessary because the compiler refuses to build; a linter rule against a dangerous API is a test that runs in the editor. The practical consequence is that strengthening types is often a better response to a class of bug than adding tests, because it makes the defect unrepresentable rather than merely detected — and it costs nothing at runtime, which no test does.

Both tests check the same rule. Only one can be read in ten seconds.

// Ignores the shape: setup, action and checking are interleaved.
@Test
void test1() {
    var cart = new Cart();
    cart.add(new Item("A", 500));
    assertEquals(500, cart.total());
    cart.add(new Item("B", 300));
    cart.applyCoupon("SAVE10");
    assertEquals(720, cart.total());
    cart.remove("B");
    assertEquals(450, cart.total());
}

// Arrange, act, assert: one behaviour, three visible phases.
@Test
void couponReducesTotalByItsPercentage() {
    var cart = new Cart();                       // arrange
    cart.add(new Item("A", 500));
    cart.add(new Item("B", 300));

    cart.applyCoupon("SAVE10");                  // act

    assertEquals(720, cart.total());             // assert
}

The first test fails and tells you almost nothing. Which of the three assertions broke? Was the arithmetic wrong, the coupon, or the removal? You read the whole body and then step through it, and you do that every time anything in Cart changes, because this one test is coupled to add, total, coupons and removal at once.

It also stops early. When line three fails, the coupon and removal behaviour is never exercised at all, so one bug hides two others and the suite reports a single failure where there were three.

The shape fixes both by construction. One action per test means the test name can state the rule, the failure message identifies the rule, and the arrange block is visibly just context rather than something being checked. When the arrange block grows past a few lines, that is itself a signal — either the object is hard to construct, which is a design finding, or you need a builder.

The discipline worth naming is that there is exactly one act. If you need a second action to reach the state you care about, it belongs in arrange, and the mental test is whether a reader can point at the single line whose behaviour is under examination.

One concept. The strict single-assertion rule is a proxy for the thing that actually matters — a test should have one reason to fail — and taken literally it produces four tests that duplicate their setup to check four fields of the same returned object. Asserting on that whole object, or on a group of properties that together describe one outcome, keeps the reason-to-fail count at one. The rule does its real work in the other direction: if you cannot name the single concept your assertions describe, you have more than one test hiding in the body.

Coupling to anything other than the behaviour under examination. Asserting on the exact wording of a message, on the order of an unordered collection, on the number of calls made to a collaborator, on a formatted date, on internal fields reached through reflection, on the total row count of a shared database. Each of these breaks when someone makes a change that harms nobody, and the cost is not the fix — it is that the team learns failures are usually noise. A brittle suite is worse than a small one, because it spends the credibility that makes a red build stop a release.

Same requirement, two tests. The first fails on a rename; the second fails on a bug.

// Brittle: asserts how the work was done.
@Test
void registersUser() {
    var repo = mock(UserRepository.class);
    var mailer = mock(Mailer.class);
    var hasher = mock(PasswordHasher.class);
    when(hasher.hash("pw")).thenReturn("H");
    new Registration(repo, mailer, hasher).register("a@b.com", "pw");

    verify(hasher).hash("pw");
    verify(repo).save(argThat(u -> u.getPasswordHash().equals("H")));
    verify(mailer).send(eq("a@b.com"), eq("Welcome!"), anyString());
    verifyNoMoreInteractions(repo, mailer, hasher);
}

// Behavioural: asserts what is now true.
@Test
void registeredUserCanAuthenticateAndIsWelcomed() {
    var users = new InMemoryUserRepository();
    var mailer = new RecordingMailer();
    var registration = new Registration(users, mailer, new Argon2Hasher());

    registration.register("a@b.com", "pw");

    assertTrue(users.authenticate("a@b.com", "pw"));
    assertFalse(users.authenticate("a@b.com", "wrong"));
    assertEquals(1, mailer.messagesTo("a@b.com").size());
}

The first test breaks if you rename the subject line, switch hashing algorithm, add an audit write, or reorder the calls — none of which changes what a user experiences. It also passes if the stored hash is never actually usable for login, because it only checks that a particular string was handed to save.

The second breaks on precisely one thing: a user who registered cannot log in, or was not welcomed. It survives the algorithm change, the extra audit write and the copy edit, and it would fail on the real defect the first one missed.

The mechanism that makes it possible is replacing mocks with fakes. An in-memory repository that genuinely stores and retrieves lets you assert on the resulting state rather than on the calls, and a recording mailer lets you ask "was one message sent to this address" instead of pinning the exact arguments.

What you give up is honest. The behavioural test is slower, and when it fails it tells you less about where. That trade is usually right, and verifyNoMoreInteractions is the line to remove first in any suite — it turns every future addition into a test failure.

Asserting on anything the caller cannot observe. Private methods, internal field values, the sequence and count of calls to collaborators, the class names involved. The reason is not purity: those assertions make the test fail on refactoring, which inverts the suite's purpose from enabling change to taxing it. The practical rule is to write assertions using only the public interface a real consumer would use, and to treat a test that needs reflection or a package-private accessor as evidence that the behaviour you want to check has no proper way in yet.

The rule, in the language of the domain, so a failure report is readable without opening the file. transferFailsWhenSourceBalanceIsInsufficient tells a release manager something; testTransfer2 does not. The name is also a design check: if you cannot state the rule in a sentence without "and", the test is doing two things, and if you can only describe it in terms of the code — "returns null when the map lookup misses" — you are probably testing an implementation detail rather than a behaviour anyone cares about. The suite then doubles as a specification you can read top to bottom.

Twelve tests need an order, each cares about one field, and all twelve construct the whole thing.

// Before: every test knows every field, and adding one breaks all twelve.
var order = new Order("o-1", "c-9", List.of(new Line("sku-1", 2, 500)),
                      Status.PENDING, "GBP", Instant.parse("2026-01-01T00:00:00Z"),
                      null, false);

// After: a builder with valid defaults, overriding only what the test is about.
public class OrderBuilder {
    private Status status = Status.PENDING;
    private List<Line> lines = List.of(new Line("sku-1", 1, 500));
    private String currency = "GBP";

    public static OrderBuilder anOrder() { return new OrderBuilder(); }
    public OrderBuilder shipped() { this.status = Status.SHIPPED; return this; }
    public OrderBuilder withLines(Line... l) { this.lines = List.of(l); return this; }
    public Order build() { /* fills ids and timestamps with valid values */ }
}

// The test now states only what matters to it.
var order = anOrder().shipped().build();
var multiLine = anOrder().withLines(new Line("a", 1, 100), new Line("b", 3, 250)).build();

The duplicated constructor call has two costs. Adding a required field is a mechanical edit across every test file, which is the kind of change teams avoid making, so the model calcifies around the test suite. And the reader cannot tell which of those eight arguments the test depends on, so nobody dares change any of them.

The builder makes the relevant field the only visible one. anOrder().shipped() says the test is about shipped orders and nothing else, which is both shorter and more informative than the full construction, and a new required field is one edit in one place.

The detail that makes it work is that the defaults must be valid and boring. A builder that produces an order with a null currency pushes the same problem back into every test, and a builder whose defaults are interesting — a discount applied, a zero quantity — makes tests pass or fail for reasons their bodies do not mention.

Two habits keep it healthy. Name the methods after domain states rather than setters, so the test reads as a scenario. And resist adding assertions to the builder or letting it reach a database, at which point it stops being a fixture and becomes a second implementation of your application.

Instead of asserting an output for a chosen input, you state a property that must hold for all inputs and let the framework generate hundreds of them, including the awkward ones humans do not think of — empty, zero, negative, maximum, duplicated, unicode. When it finds a failure it shrinks the input to the smallest case that still fails, which usually hands you a two-element reproduction rather than the thousand-element one it found. It suits pure functions with algebraic properties: round-trip encode and decode, sorting, parsing, monetary arithmetic, anything with an inverse or an invariant.

A rounding helper for splitting a bill, with a thorough-looking example table.

// Example-based: five cases, all passing.
@ParameterizedTest
@CsvSource({"1000,2,500", "1000,4,250", "900,3,300", "100,1,100", "50,2,25"})
void splitsEvenly(int total, int ways, int each) {
    assertEquals(each, Money.split(total, ways).get(0));
}

// Property-based: the invariant that actually matters.
@Property
void splitAlwaysPreservesTheTotal(@ForAll @IntRange(min = 0, max = 1_000_000) int total,
                                  @ForAll @IntRange(min = 1, max = 50) int ways) {
    assertEquals(total, Money.split(total, ways).stream().mapToInt(i -> i).sum());
}

Property failed after 14 tries, shrunk to the minimal case:
  total = 1, ways = 2
  expected 1 but was 0        // split returned [0, 0]

Every example in the table divides exactly, which is the bias to notice: a human choosing cases picks numbers that make the arithmetic easy to state, and those are exactly the cases where integer division has nothing to go wrong with. Ten pence split three ways is where the money disappears.

The property is also a better specification than the table. "The parts sum to the total" is the actual business rule — a bill split must not lose or invent money — and stating it that way makes the missing requirement obvious: somebody must receive the remainder pennies, and the code must decide who.

Shrinking is what makes the technique usable. The first failure the generator found was probably some large unhelpful pair; the report gives you 1 and 2, which is a case you can reason about and paste straight into a regression test.

The limit is worth stating too. Properties need an invariant, and a lot of code has none beyond "does what the specification says", where examples remain the right tool. The strongest pattern is both: properties for the invariants, examples for the specific agreed cases including the one just found.

That the assertion is generated rather than written, so it encodes whatever the code did on the day it was recorded — including the bugs. When a change breaks forty snapshots, the affordance offered is a single command to accept them all, and that command is pressed under time pressure without reading the diff, at which point the tests are documenting the new behaviour rather than checking it. Snapshots are reasonable for large rendered output where hand-written assertions are impractical, provided they are small enough to review, and they are a poor substitute for an explicit assertion about the one thing you care about.

Any stand-in for a real collaborator in a test — the umbrella term, of which mock is one specific kind. The vocabulary matters because the words describe different trade-offs and interviewers use them as a filter: a candidate who calls everything a mock cannot express the difference between "returns canned data" and "asserts that it was called", which is precisely the distinction between a test that survives refactoring and one that does not. The five terms are dummy, stub, spy, mock and fake, and knowing which you are reaching for is usually the decision that determines whether the test is brittle.

Same dependency, five roles, and the difference is what the test does with it.

// DUMMY - passed to satisfy a signature, never used.
var auditLog = new AuditLog() { public void record(Event e) { } };
new Registration(users, dummyMailer, auditLog);   // this test is not about audit

// STUB - returns canned answers so the test can reach the case it wants.
var rates = mock(RateProvider.class);
when(rates.gbpToUsd()).thenReturn(new BigDecimal("1.25"));

// SPY - records what happened, and the test inspects it afterwards.
class RecordingMailer implements Mailer {
    final List<String> sentTo = new ArrayList<>();
    public void send(String to, String subject, String body) { sentTo.add(to); }
}
assertEquals(List.of("a@b.com"), mailer.sentTo);

// MOCK - has an expectation set in advance; failing to meet it fails the test.
var mailer = mock(Mailer.class);
subject.register("a@b.com", "pw");
verify(mailer, times(1)).send(eq("a@b.com"), anyString(), anyString());

// FAKE - a real, working, simplified implementation.
class InMemoryUserRepository implements UserRepository {
    private final Map<String, User> byEmail = new HashMap<>();
    public void save(User u) { byEmail.put(u.email(), u); }
    public Optional<User> findByEmail(String e) { return Optional.ofNullable(byEmail.get(e)); }
}

The line that matters runs between stub and mock. A stub is an input to the test — it supplies the state you need to reach an interesting case, and asserting on a stub is a category error. A mock is an assertion about an interaction, so every mock you set is one more implementation detail the test is now pinned to. Most tests want stubs and want them everywhere, and want at most one mock, for the one interaction that is genuinely the behaviour under examination.

A spy sits between them and is usually the better choice than a mock. It records and lets you assert afterwards in the assert block, which keeps arrange and assert separate and lets you assert loosely — "a message went to this address" — rather than pinning every argument.

Dummies look trivial and carry a real signal. Needing three of them says the constructor demands collaborators this behaviour does not use, which is an argument for splitting the class rather than for a tidier dummy.

Fakes are the most underused and the most valuable. Because a fake actually works, tests using it assert on resulting state instead of on calls, and the same fake serves dozens of tests. Its risk is drift from the real implementation, which is why a fake should be verified by running the same contract test suite against both it and the real thing.

When several tests need the collaborator to behave, not merely to answer — a repository whose contents you then query, a clock you advance, a queue you drain, a payment gateway that can be put into a declining state. Fakes let assertions be about outcomes, which is what makes them refactor-proof, and they remove the stub-setup noise from every test body. The cost is that a fake is code you own and it can diverge from reality, so the discipline is to run one shared contract test suite against both the fake and the real implementation, and to keep the fake simple enough that it cannot develop bugs of its own.

The test asserts every interaction, the code has a real defect, and the build is green.

// The code: a bug on line three - the balance check uses the wrong account.
void transfer(String fromId, String toId, int amountMinor) {
    Account from = accounts.find(fromId);
    Account to   = accounts.find(toId);
    if (to.balanceMinor() < amountMinor) throw new InsufficientFunds();  // BUG
    accounts.save(from.debit(amountMinor));
    accounts.save(to.credit(amountMinor));
    events.publish(new Transferred(fromId, toId, amountMinor));
}

// The test: entirely mocks, entirely green.
@Test
void transfersMoney() {
    when(accounts.find("A")).thenReturn(new Account("A", 10_000));
    when(accounts.find("B")).thenReturn(new Account("B", 10_000));

    subject.transfer("A", "B", 5_000);

    verify(accounts).save(argThat(a -> a.id().equals("A")));
    verify(accounts).save(argThat(a -> a.id().equals("B")));
    verify(events).publish(any(Transferred.class));
}

Both stubbed accounts hold the same balance, so the wrong-account check passes for the wrong reason, and the assertions only confirm that two saves and a publish happened — never what they contained. The test is thorough-looking and checks nothing about money.

The fix is to assert on state rather than on calls, which a fake makes possible:

var accounts = new InMemoryAccounts(new Account("A", 10_000), new Account("B", 200));
new Transfers(accounts, events).transfer("A", "B", 5_000);
assertEquals(5_000, accounts.find("A").balanceMinor());
assertEquals(5_200, accounts.find("B").balanceMinor());
// and the case the original could not express at all:
assertThrows(InsufficientFunds.class, () -> transfers.transfer("B", "A", 1_000));

Two diagnostics come out of this. Asymmetric fixtures find asymmetric bugs — the original test used identical balances, and any bug that swaps the two accounts is invisible under identical inputs. And a test whose assertions are all verify calls has not checked an outcome, so ask what state changed and assert on that instead.

The deeper reading is that mocking every collaborator tests only the wiring you already wrote, in the arrangement you already assumed. The suite becomes a mirror of the implementation, which is why it stays green through defects and goes red through refactoring — exactly backwards.

Because the amount of mocking a class needs is a measurement of its coupling. Six mocks in a constructor means six collaborators, which means the class is coordinating rather than doing, and the test is hard because the design is. The productive response is to extract the logic into something that takes values and returns values — trivially testable with no doubles at all — and leave a thin orchestration layer that an integration test covers. This is the whole classical-versus-mockist argument in practice: mockists isolate every class, classicists let real collaborators participate and double only at the process boundary.

Types you do not own, and anything whose real behaviour is the risk. Mocking a database driver, an HTTP client library or a cloud SDK produces a test that asserts your understanding of that library's semantics, which is precisely the thing most likely to be wrong — and it will keep passing after the vendor changes behaviour in a minor version. Wrap the third party in your own narrow interface, mock or fake that, and test the wrapper against the real dependency in an integration test. Value objects are the other case: mocking a Money or a LocalDate is always more work than constructing one.

There is no agreed line, which is why the useful move in an interview is to define yours before using the words. A workable set: a unit test runs in one process with no I/O; an integration test crosses one real boundary — a database, a broker, an HTTP call to a stubbed service; an end-to-end test drives the deployed system through its real entry point with real dependencies. What matters is not the taxonomy but the properties you are trading: process count, runtime, what a failure implicates, and whether the test can run on a laptop before a push.

One request, four participants, and the span each kind of test covers.

sequenceDiagram
    participant B as Browser
    participant A as API handler
    participant D as Domain logic
    participant P as Postgres
    B->>A: POST /orders
    A->>D: place the order
    D->>P: INSERT and read back
    P-->>B: response rendered
    Note over D: unit test - pure logic, no I/O
    Note over A,P: integration test - real database, no browser
    Note over B,P: end-to-end - the whole path

The unit test covers pricing, validity rules and state transitions inside D, because that is where branching lives. It runs in a millisecond, and when it fails you know the rule that broke without reading a stack trace.

The integration test spans the handler down to a real Postgres. It is the only layer that can catch a wrong column type, a constraint violation, a transaction boundary that commits too early, a serialisation mismatch, or a query that works in the fake and not in SQL. Those are the defects unit tests structurally cannot see, which is why widening this band is the strongest version of the testing-trophy argument.

The end-to-end test exists to answer one question: is the system actually wired up. Configuration, routing, authentication, migrations applied, the right build deployed. That question has one answer per deployment, not one per feature, which is why a handful of these is right and two hundred is a maintenance problem.

The overlap is deliberate and should be minimal. If a business rule is asserted at all three levels, two of the three will fail together on every change to it, tripling the diagnosis cost for no extra information. Assert each rule at the lowest level that can hold it, and let the levels above assert only what they uniquely can.

Because the substitute has different semantics, and the differences are exactly where the bugs are: SQL dialect, type coercion, constraint enforcement, isolation behaviour, index usage, ON CONFLICT, JSON operators, collation and sort order. A suite green against an embedded database and red in staging has cost you the whole value of the layer. Containers made this cheap — a real engine started per suite, migrations applied, torn down after — so the historic reason for the substitute has gone. The remaining cost is startup time and Docker availability in CI, which is a scheduling problem rather than a correctness one.

Two services deployed independently, where the consumer's expectations and the producer's behaviour can drift with nothing failing until production. The alternatives are both bad: integration-testing them together requires a shared environment and blocks independent release, while stubbing the producer in the consumer's tests just enshrines the consumer's possibly-wrong assumption. A contract test records what the consumer actually needs, and then verifies that recording against the real producer in the producer's own pipeline — so the producer learns at build time that a change breaks somebody, without either team running the other's suite.

The consumer states what it depends on; the producer's build proves it still holds.

// CONSUMER side: this generates the contract file.
await provider.addInteraction({
  state: 'user 42 exists',
  uponReceiving: 'a request for user 42',
  withRequest: { method: 'GET', path: '/users/42' },
  willRespondWith: {
    status: 200,
    body: { id: like(42), email: like('a@b.com'), plan: term({ matcher: 'free|pro' }) }
  }
});
// The consumer's own tests run against this stub, so they are honest about
// what they actually read: id, email, plan. Nothing else is in the contract.

PRODUCER side, in the producer's pipeline:

  Verifying contract "billing-ui -> user-service"
    given user 42 exists
      GET /users/42
        returns status 200                          PASS
        body has id                                 PASS
        body has email                              PASS
        body has plan matching free|pro             FAIL
          expected free|pro, got "trial"

  1 of 4 expectations failed. Build red in user-service.

The important property is where the failure appeared. The producer added a trial plan value, and the producer's build broke — not the consumer's, and not production. Nobody had to run a shared environment and neither team had to remember the other existed.

The second property is that the contract covers only what the consumer reads. The producer is free to add fields, rename ones nobody consumes, and change internals, because the contract is a statement about the consumed subset rather than the full schema. That is what makes it compatible with independent deployment.

Two operational details decide whether this works. The contracts must be published somewhere the producer's build fetches them, and the verification must run against every consumer version currently deployed — otherwise you verify against a contract from a consumer nobody is running. And deployment needs to consult the results, so a producer cannot ship a version that fails a live consumer's contract.

What it does not do is check that either side is correct. The consumer can record an expectation based on a misunderstanding, and both builds will happily agree about the wrong thing, which is why contract tests complement rather than replace a thin end-to-end layer.

Five sources, and naming them is most of the diagnosis. Time — sleeps, timeouts tuned to one machine's speed, tests that break at midnight or in another zone. Order dependence — a test that passes only after another has populated something. Shared mutable state — a static field, a singleton, a database row, a temp file, a port. Asynchrony — asserting before the work finished, which passes on a fast machine. And the network, including real third parties and DNS. All five have the same cure in common: make the dependency explicit and controllable rather than hoping it behaves.

The sleep is a guess about how long something takes, and both directions of the guess are wrong.

// Flaky. Passes on a laptop, fails in CI, and wastes 2 seconds when it passes.
@Test
void publishesOrderPlacedEvent() {
    orders.place(anOrder().build());
    Thread.sleep(2000);
    assertEquals(1, consumer.received().size());
}

Two failure modes from one line:

  too short   CI is loaded, the consumer polls at 2.1s -> intermittent red,
              the standard fix is to raise the sleep, and the suite gets slower
              every time somebody debugs it

  too long    every run pays 2s whether or not it needed it; 200 such tests
              is nearly seven minutes of the build spent sleeping

// Deterministic: poll for the condition with a generous ceiling, or better,
// remove the wait entirely by controlling the trigger.
@Test
void publishesOrderPlacedEvent() {
    orders.place(anOrder().build());
    await().atMost(Duration.ofSeconds(10))
           .pollInterval(Duration.ofMillis(20))
           .until(() -> consumer.received().size() == 1);
}

The polled version is fast when things are fast and patient when they are not, which is the property a fixed sleep cannot have. Note the asymmetry: the timeout is now a failure threshold rather than an expected duration, so it can be set generously without costing anything on the happy path.

The better fix, where the design allows it, is to remove the wait. Inject the executor so the test runs the task synchronously, or expose a hook that completes the pending work, and the asynchrony disappears from the test entirely. A test that has no wait cannot be flaky about timing.

The rule to state: a sleep in a test is a race condition with a comment. If you cannot express the condition you are waiting for, you do not yet know what the test is asserting.

Both pass together, in that order, on that machine. Neither passes alone.

class UserServiceTest {
    static final Database db = Database.shared();   // shared, not per-test

    @Test @Order(1)
    void createsUser() {
        service.register("a@b.com", "pw");
        assertEquals(1, db.count("users"));         // depends on empty start
    }

    @Test @Order(2)
    void findsUserByEmail() {
        assertTrue(service.findByEmail("a@b.com").isPresent());
        // no arrange block at all: relies on test 1 having run
    }
}

Run the class          both pass
Run findsUserByEmail   fails - no such user
Reverse the order      both fail
Run in parallel        intermittent, depends on scheduling
Add a third test that also registers a user   createsUser now fails, count is 2

Two distinct faults are tangled here. The second test has no arrange phase and borrows state from the first, so it is not a test of findByEmail but of the pair. And the first asserts on a global count rather than on its own row, so any other test touching users breaks it — which is why this class fails the day somebody adds an unrelated test.

The consequences are worse than an occasional red build. You cannot run one test to debug it, cannot parallelise the suite, and cannot trust a failure, because the test that fails is often not the test that is wrong.

The fixes are mechanical. Give every test its own arrange block, even at the cost of repetition, and let a builder absorb the noise. Assert on what this test created — findByEmail returning the row it inserted — never on aggregate counts of shared tables. And isolate the state: a transaction rolled back after each test, a fresh schema per class, or a container per suite.

The check that catches it permanently is running the suite in a randomised order in CI. Most frameworks support it in a line of configuration, and it converts a class of latent flakiness into a reliable, reproducible failure.

A deterministic failure reproduces on demand, so it is a bug report with a recipe: you can bisect it, attach a debugger, and know when it is fixed. A non-deterministic one appears in some fraction of runs, which means every part of the loop breaks — you cannot confirm a fix, only fail to reproduce it, and the temptation is to declare victory after three green runs. The practical consequence is that the first move on a flaky test is to make it deterministic: run it a thousand times, in random order, in isolation, under load, until you find the axis that controls it.

Quarantine it out of the blocking gate, with an owner and a date, and file it as a defect rather than a chore. Leaving it in place is the expensive option, because it trains the team to re-run red builds, and once that habit exists the suite has stopped gating anything — including the real failure that arrives next week. Deleting it silently is worse, since it was covering something. The reason to treat flakiness as a bug is that it often is one: a race the test exposed exists in production too, where it will surface as an intermittent customer-facing failure nobody can reproduce.

Load asks whether the system meets its latency targets at expected traffic, so it validates a stated requirement. Stress pushes past that until something breaks, to find the ceiling and — more usefully — to see how it breaks, since graceful shedding and a cascading collapse look identical on a capacity plan. Soak holds moderate load for hours or days to expose what accumulates: memory leaks, connection exhaustion, log disks, growing tables, slow-degrading caches. Spike applies a sudden step change to test autoscaling reaction time and cold-start behaviour, which a smooth ramp deliberately hides.

A ramp from 100 to 1,000 virtual users. The numbers say more than the pass or fail.

users     rps      p50      p95       p99      errors    cpu
  100     480     18ms     41ms      70ms      0.00%     22%
  300   1,430     21ms     58ms     110ms      0.00%     58%
  500   2,340     26ms     96ms     240ms      0.01%     81%
  700   2,510     94ms    610ms   1,900ms      0.42%     94%
  900   2,480    310ms  2,400ms   6,100ms      7.2%      96%
1,000   2,190    780ms  9,000ms  timeouts      23%       97%

The saturation point is between 500 and 700 users. Throughput stops rising at about 2,500 requests per second and then falls, which is the signature reading: past saturation, extra concurrency adds queueing rather than work, so latency grows roughly linearly with users while throughput is flat. The falling rps at 1,000 users means the system is now spending capacity on work that will time out.

Error onset is at 700, at 0.42%. That is the number to alarm on, because it is the first evidence of failure and it appears while the p50 still looks acceptable at 94ms — a dashboard showing medians would call this healthy.

So the honest capacity statement is 500 users and about 2,300 rps, not the 2,510 peak. You size to the last point that met the latency objective, and if the objective is a p99 under 300ms then 500 users is the answer and 700 is already a violation.

The p50-to-p99 spread is the other reading. At 100 users it is four times; at 700 it is twenty times. A widening spread means queueing somewhere with limited concurrency — a connection pool, a lock, a single-threaded consumer — and it usually identifies the bottleneck faster than the CPU column, which here reaches 94% and tells you only that something is busy.

What the table cannot tell you is whether the test was valid. Ask whether the data volume matched production, whether caches were warm, whether think time was realistic, and whether the load generator itself saturated — a flat rps is sometimes the test tool's ceiling rather than the system's.

Because it describes nobody and hides the shape. Ninety-five requests at 40ms and five at three seconds average to under 190ms, which reads as fine while one user in twenty had an unusable experience. Averages also cannot be aggregated safely across instances or time windows, and they move very little when a tail gets worse. Report percentiles instead, with the caveat that percentiles do not average either: a page making ten backend calls will hit somebody's p99 on most loads, so a per-service p99 is not the user's p99.

Static analysis reads the source or bytecode without running it, so it can reach every path and finds patterns — string-concatenated SQL, a disabled certificate check, a hardcoded secret — at the cost of a high false-positive rate and no knowledge of whether the path is reachable. Dynamic analysis attacks the running application from outside, so everything it reports is real and exploitable, but it only sees what it can reach through the interface and needs a deployed environment. They find different classes of defect, which is why mature pipelines run both, static on the pull request and dynamic against a deployed build.

Known vulnerabilities in the libraries you pulled in, matched against advisory databases, including the transitive ones you never chose — which is where most of the risk lives, since a direct dependency count of thirty routinely resolves to six hundred artefacts. What it does not catch is whether the vulnerable code path is reachable from your application, so the raw report over-reports badly and a team that treats every finding as urgent stops reading it. The useful practice is to gate on severity plus reachability where the tool supports it, and to keep upgrades continuous so that the emergency patch is a small version bump.

In three layers, because no one of them is sufficient. Automated checks — axe or similar, in the component and end-to-end suites — catch missing labels, contrast failures, invalid ARIA and heading-order problems, which is perhaps a third of the issues and worth gating on. Manual keyboard and screen-reader passes catch what automation structurally cannot: focus order, whether a modal traps focus, whether a dynamic update is announced, whether an error is reachable. And testing with actual assistive-technology users catches whether the flow is usable rather than merely conformant. Quoting WCAG level AA as the target is the expected vocabulary.

That it is a single mutable resource with many writers, so it is never in a known state. Someone's half-finished feature is deployed, someone's test deleted the data your test needed, two teams' migrations disagree, and a failure there is as likely to be environmental as real. The consequence is a queue for access and a suite nobody trusts, which is the worst combination — slow and uninformative. The direction of travel is ephemeral environments created per branch and destroyed after, with staging reserved for the small number of checks that genuinely need production-like scale or third-party sandboxes.

By the test that needs it, in the test, through a builder or factory — so that reading the test tells you what state it assumes. The alternative, a large shared seed fixture, starts convenient and becomes the thing nobody may change: tests depend on user 42 having three orders, and a new test that needs a fourth breaks eleven others. Where volume is required, generate it rather than committing it, and keep the generated data derived from a seed so failures reproduce. Production dumps are the tempting third option and bring a legal problem with them.

Do not put it there. A production copy in a non-production environment inherits every obligation — lawful basis, retention limits, subject access, breach notification — while shedding all of the controls, and a lower environment is where credentials are shared and logging is verbose. The workable options are synthetic generation, or anonymisation strong enough to be irreversible: pseudonymisation still counts as personal data under GDPR, so replacing names while keeping postcode, birth date and purchase history is a re-identifiable dataset with extra confidence. Where real data is genuinely necessary, the environment has to carry production-grade access control.

The gate is chosen by the cost of a false red and the value of fast feedback.

check                        when              budget     blocks merge
---------------------------  ----------------  ---------  ---------------
compile, types, lint         every push        under 1m   yes
unit tests                   every push        under 2m   yes
integration, real containers  every PR         under 10m  yes
consumer contract verify     every PR          under 5m   yes
smoke end-to-end, ~10 paths  every PR          under 8m   yes
full end-to-end suite        merge to main     under 40m  no, alerts owner
load and soak                nightly, weekly   hours      no, trend tracked
dependency and SAST scan     PR diff, nightly  minutes    critical only
mutation testing             weekly            hours      no, reviewed
accessibility automated      every PR          under 2m   yes

The organising rule is the pull-request budget. Somewhere around ten minutes, developers stop waiting for the build and start context-switching, so everything in the blocking gate competes for that budget and anything that cannot fit moves right. That is a scheduling decision, not a statement about importance.

The second rule is that a blocking check must be reliable. A flaky end-to-end suite in the merge gate does more damage than no suite, because it teaches people to re-run, so the ten smoke paths are chosen for stability as much as coverage and the rest run post-merge where a failure pages an owner instead of blocking everybody.

Note what changes after the merge gate rather than disappearing. The full end-to-end suite, load tests and mutation runs still fail loudly — they just fail to a person rather than to a branch, and their signal is a trend over days. Load-test results in particular are worthless as a single pass or fail and valuable as a line that moved.

The one thing that should block regardless of budget is a critical dependency advisory or a leaked secret, because the cost of merging it is not proportional to the time it takes to check.

An isolated deployment per branch or pull request, so tests run against a known state with no queue and no cross-contamination, and the environment definition gets exercised on every change rather than drifting. What it costs is that everything must be reproducible from code — infrastructure, migrations, seed data — and third-party dependencies need sandboxes or fakes, which is where the effort actually goes. There is also a real cloud bill and a lifecycle problem, since environments that are not destroyed accumulate. It is a strong pattern that requires the automation maturity to be in place first.

Decide the isolation mechanism deliberately rather than per test. Inside a process, that means no static mutable state and no singletons the tests can see; for a database, a transaction rolled back at the end of each test where the code allows it, or a schema or container per suite where it does not. Anything global — a fixed port, a temp file path, a system property, a clock — must be parameterised so parallel runs cannot collide. The verification is to run the suite in random order and in parallel in CI, because interference that is not tested for is interference you will meet on a release day.

When the property you need to verify only exists there: real traffic mix, real data volume and skew, real third-party behaviour, real infrastructure. Canary releases, synthetic monitoring on live journeys, feature flags with a small exposure and shadow traffic against a new implementation are all tests in production, and treating them as such is more honest than pretending staging covered it. The preconditions are the whole answer: fast rollback, per-cohort metrics, an error budget to spend, and no side effects escaping — a shadowed payment call must not charge anybody.

When the question is whether the behaviour is right, not whether it changed. An automated check compares against an expectation somebody already had, so it cannot notice that the flow is confusing, the empty state is alarming, the error message is unhelpful, or the whole feature solves the wrong problem. Exploratory testing — time-boxed, charter-driven, with notes — is the discipline that finds the defects nobody anticipated, which is the category automation structurally cannot cover. Saying so is a mark of seniority, because the reflex answer is "automate everything" and the honest position is that automation covers regression and humans cover discovery.

Severity describes the technical impact of the defect — data loss, crash, cosmetic — and is largely objective. Priority describes when it will be fixed, which is a business decision balancing impact, frequency, workaround availability and cost. The pairs that matter are the mismatched ones: a typo in the company name on the landing page is low severity and top priority, while a crash reachable only by an internal tool with two users is high severity and can wait. Candidates who conflate them produce triage where everything critical is urgent, and the queue stops carrying information.

As a floor on new code, sometimes; as a global number, almost never. "Coverage must not fall in this diff" is enforceable and points attention at untested new work. "The repository must reach eighty per cent" produces assertion-free tests over trivial code, exclusions added to configuration, and a metric that rises while protection does not. The better policies are qualitative and specific: every bug fix ships with a test that reproduces it, critical modules are named and held to a higher bar, and mutation scores are reviewed periodically where correctness genuinely matters.

First establish whether it is the test or the system, because the answers diverge completely. Re-run it in isolation and in a loop, read the failure rather than the status, and check whether the assertion is about timing, order or shared state — the three signatures of a bad test. If it is genuinely flaky, quarantine it with an owner and ship, having said so explicitly. If it might be a real race, do not ship: an intermittent test failure and an intermittent production failure have the same cause, and the test found it first. The wrong answer is re-running until green.

By working outwards in a fixed order rather than listing test types. Start with what the thing is supposed to do and what would be worst if it were wrong, since that decides where the effort goes. Then the inputs — equivalence classes and boundaries, including empty, zero, negative, maximum, duplicate and malformed. Then the state and sequence: what happens on retry, out of order, concurrently. Then the seams, naming what you would fake and what you would keep real. Then the non-functional angles that apply — load, permissions, accessibility. Then say what you would not test and why, because scoping aloud is what separates judgement from recitation.

"This test is failing intermittently — what do you do?" A strong answer treats it as a defect with a cause: it separates test flakiness from a real race, names the usual axes of time, order, shared state and asynchrony, describes making the failure deterministic before attempting a fix, and says what happens to the test in the meantime and who owns it. A weak answer proposes a retry, a longer sleep, or a rerun until green — each of which suppresses the signal and, if the race was real, ships the bug with the evidence deleted.