Every test passed and the release broke in production anyway. How do you work out what your suite was never going to catch?

Establish the defect before theorising about coverage

The instinct is to start listing what the suite lacks. Resist it, because you cannot reason about a coverage gap until you know precisely what escaped, and the initial description of a production problem is almost always wrong in some detail that turns out to matter.

So first reconstruct the mechanism: which input, from which kind of user, hitting which code path, in which state, producing which incorrect outcome. Get to the specific condition rather than the symptom. "Checkout failed" is a symptom. "Orders containing a discounted item and a gift card were charged the pre-discount total, because the two adjustments were applied to different copies of the basket" is a mechanism, and only the mechanism tells you where a test could have stood.

Then establish the timeline, because it frequently reframes everything. When was the defective code merged, when was it deployed, and when did the symptom start? If the code shipped three weeks ago and the failure began yesterday, the release is not the cause and your investigation is looking at the wrong change. If nothing was deployed at all, the change came through a different channel entirely — configuration, a flag, reference data, a dependency, an infrastructure change — and the suite was never in the path. That case is not a coverage gap and treating it as one produces a useless test.

The three ways a suite misses something

Once you have the mechanism, walk down from the bottom of the stack and ask at each layer whether a test there could have observed this defect, and if so, why none did. The answers fall into three groups that need entirely different responses.

The first is a case nobody thought of. The logic is unit-testable, a test at that layer would have failed, and the input combination simply never occurred to anyone. Two adjustments to one basket, a quantity of zero, a name that collides with a reserved word. This is the cheapest kind of escape to fix and the least interesting, and the useful question is not "why was this case missing" but "what technique would have generated it" — boundary analysis, a decision table, property-based testing over the whole input space rather than examples.

The second is a defect that lives between components, where each unit is correct and the assembly is not. Two services agreeing on a field name but not on its units, a caller retrying an operation the callee is not idempotent for, a rounding difference between two layers. No unit test can see these, because each unit passes on its own terms. The fix belongs at the seam: a contract test, an integration test that spans both sides, or a schema that makes the disagreement impossible to express.

The third is the one that matters most and gets the least attention. The defect depends on a condition your test environments do not have and largely cannot have. Production data volume and its distribution, including the malformed historical rows that predate current validation. Concurrency, where two real users act simultaneously and your tests act one at a time. Real elapsed time, so a token expires or a scheduled job fires mid-flow. Scale-dependent behaviour, where the system runs on many instances and your environment runs one. Third-party behaviour, where the sandbox returns clean responses and the live provider returns something the sandbox never produces. Traffic mix, load, and the failure of a dependency you never see fail.

Naming this third category correctly is the whole judgement in this question, because the honest conclusion is that no reasonable amount of pre-release testing would have caught it, and pretending otherwise sends the team on a long project that will not work.

Answer the layer question, not the volume question

The reflex after an escape is to add an end-to-end test that reproduces exactly this scenario. Sometimes that is right. Frequently it is the worst available option, because it adds a slow, fragile test that covers one specific instance of a class of bug and will never fail again, while the same class recurs in a different form next quarter.

Ask instead where the cheapest reliable observation point is. If a unit test would have caught it, that is where it goes, and the end-to-end test adds nothing but minutes. If the defect was a disagreement between two components, a contract test at the boundary catches every future instance of the same disagreement rather than this one. If the defect was a whole category of input, a property-based test or a generated case matrix covers the category. And if the defect can only exist in production, then the answer is not a test at all.

For that last case the useful investments are detection and containment rather than prevention. An alert or a data-integrity check that would have caught the wrong totals within minutes instead of a day. A canary release that would have limited exposure to a small population. A feature flag allowing the change to be disabled without a deployment. A reconciliation job comparing what was charged against what should have been. Any of these reduces the cost of the next unpreventable defect far more than another end-to-end test does, and offering them is what distinguishes a senior answer.

Fix the class, and write down what you decided not to do

Close the loop on the specific defect, then spend the remaining effort on the class. If the mechanism was two components disagreeing about a field's meaning, the question is which other fields have the same exposure. If the mechanism was historical data violating a current assumption, the question is what else assumes clean data. If it was a concurrency problem, the question is which other operations lack the same protection. One escape is usually a sample from a population, and the population is worth more attention than the sample.

Then record what you decided not to do and why. An escape generates pressure to add gates, and some of that pressure should be resisted: a test that cannot reliably reproduce the condition will be flaky and will be deleted or retried away within months, leaving the appearance of coverage without any. Writing down that a particular condition is not reproducible pre-release, and that the mitigation is detection instead, converts a gap into a stated and reviewable decision. It also gives you something concrete to answer with when a stakeholder concludes that the tests are worthless: the suite prevented the defects it was designed to prevent, this one was outside that design, and here is the specific change to what the design covers.

The number that tells you whether this is normal

One escape is an event, not a signal. What tells you whether your suite is doing its job is the rate over time and the pattern in the causes: how often defects reach production, how long they survive undetected, and which of the three categories they keep falling into. A steady trickle of category-three escapes in a system with high production variability is expected and argues for investment in observability. A steady trickle of category-one escapes argues that the team's test design technique is weak, which is a fixable and much cheaper problem. Being able to distinguish those two from a quarter of data is a more valuable answer than any individual root cause.