A test that has passed for a year starts failing after a refactor that was supposed to change nothing. Is it a bug or a bad test?

The move to avoid

The tempting first action is to look at the new output, decide it seems reasonable, and update the expected value. It makes the build green in a minute.

It is also the single most effective way to convert a test suite into decoration. Every time an assertion is edited to match current behaviour without establishing whether that behaviour is correct, the suite loses a little of its authority, and the end state is a suite that asserts the code does what the code does.

So the question has to be answered before the fix, and the answer is not found by looking at the diff. It is found by asking what the test was for.

What is the test pinning down?

Every assertion pins something. The useful distinction is whether the thing it pins is part of the observable contract — something a caller, a user, or another system relies on — or an implementation detail that happened to be true when the test was written.

Asserting that a function returns 42 for a given input is a contract. Asserting that it calls the repository exactly twice is not; it is a description of how the answer was arrived at, and the whole point of a refactor is to change that.

Some cases sit on the line and are worth thinking about rather than guessing. The order of a returned collection is a contract if the API documents an order or if consumers page through it, and an implementation detail if it is whatever the database happened to return. A rounded currency amount is a contract. The precise wording of a log line is not, unless something parses it, in which case it very much is.

If the failing assertion is on the contract side, the refactor changed behaviour and was not the pure refactor its author believed. That is a bug and the test just earned its keep. If it is on the detail side, the test was brittle from the day it was written, and this failure is the bill for that.

Confirm before concluding

Reasoning about which category you are in is not the same as verifying it, and the cost of getting it wrong is asymmetric — shipping a regression is much worse than spending twenty minutes.

So check independently. Exercise the behaviour by hand, or through a different test at a different level, and see whether the user-visible outcome changed. Read the original commit that introduced the test; if it references a ticket, that ticket usually says exactly what was being prevented, which is the most direct evidence available. Ask whoever performed the refactor what they expected to change — not accusingly, but because a genuine behaviour change is often already known to them and considered acceptable.

Reverting the refactor locally to confirm the test passes again is worth doing too. It sounds obvious and it is frequently skipped, and it distinguishes "this change broke it" from "this change happened to run first after something else broke it".

Both answers at once

The case people miss: the test was badly written and it caught something real.

An over-specified test can fail for the wrong reason while a genuine regression hides behind it. You look at an assertion on call counts, correctly conclude it is an implementation detail, relax it — and ship the behaviour change that was also in the diff. Having decided a test is brittle, still ask whether the behaviour changed. Brittleness explains why the test failed; it does not establish that nothing broke.

Fix the right thing

If it is a bug, fix the code and leave the test alone. Consider whether the test's failure message was clear enough to point at the cause, since a test that says only "expected true, got false" cost you an hour it need not have.

If it is a bad test, rewrite it rather than delete it. Something motivated it, and the motivation is usually still valid — someone wanted to know that a value gets cached, and expressed it by counting repository calls. The behavioural version asserts that a second call returns the same result without a second round trip being observable, which survives the refactor and catches the same regression. Deleting outright is right only when the behaviour is genuinely covered elsewhere, and it is worth checking rather than assuming.

Either way, note it. A test that has failed spuriously twice will do it again, and the pattern across a suite — which tests keep failing on refactors — tells you where the tests are coupled to structure rather than behaviour. That is a more useful map than any coverage report.

Ask what the test was protecting before you touch either side. Editing the expected value to match the current output is how a suite stops being able to tell you anything.