How would you design a test to find where the system breaks, rather than one that confirms it meets its target?

Two different questions, two different tests

A load test asks whether the system meets a stated target: at the expected peak throughput, does latency stay inside its objective and the error rate near zero. It has a pass and a fail, it belongs in a pipeline, and it answers a question somebody has already asked.

A breakpoint test asks where the system stops working and what it does when it gets there. It has no pass condition, its output is a number and a description rather than a verdict, and it answers the question you will be asked during an incident: how much headroom do we have, and what happens if we run out. The two tests share tooling and share almost nothing else, and conflating them is why so many teams have a green performance suite and no idea what their capacity is.

The reason the second one matters more than it appears is that a passing load test only proves the target is reachable. It says nothing about whether the target sits comfortably below the limit or a few per cent under it, and those two systems behave completely differently on a bad day.

Ramp so that a change can be attributed

The design is a staged ramp rather than a single jump. Hold each level long enough for the system to reach steady state, measure, then step up. Steady state matters because caches fill, connection pools grow, autoscalers react and garbage collection settles, and a measurement taken in the first minute of a new level describes a transient rather than the level. A step held for several minutes, with the first portion discarded, gives you a number you can compare against the previous step.

Keep every other variable fixed as you climb. Same request mix, same data volume, same environment, one thing changing. Otherwise a change in behaviour at step seven cannot be attributed to step seven's load rather than to the dataset having grown or the autoscaler having added a node.

Two properties of the ramp are worth designing explicitly. Arrival should be open-model where you can manage it: virtual users that wait for a response before sending the next request throttle themselves as the system slows, which hides the collapse you are trying to observe. And the load generator must be verified as not being the bottleneck, by checking its own CPU and its own connection limits, because a generator that saturates first produces a beautiful plateau that describes your test harness.

Read three points off the curve, not one

Plot throughput and latency against offered load and there are three interesting places, which people routinely collapse into one.

The knee is where latency starts rising faster than linearly while throughput still climbs. This is the onset of queueing, and it is usually where the useful capacity limit sits, because past it every increment of load costs disproportionate latency.

The saturation point is where throughput stops rising. Some resource is now fully utilised, and additional offered load turns entirely into queue depth and latency. The system is still doing its maximum work here, so it is not yet failing, but it has no headroom left.

The collapse point is where throughput actually falls as load increases. This is the dangerous one, because it is unstable: a system past collapse does less work under more load, so a traffic spike or a retry storm can push it into a state it cannot recover from without shedding traffic. Finding the gap between saturation and collapse tells you how much of a buffer your protection mechanisms have to work in.

The first hard limit is often not request rate

The most common defect in a stress test is that it varies only one thing, requests per second, and therefore finds only the limits that requests per second can reach. Real systems hit bounded resources first, and several of them have nothing to do with traffic volume.

Per-process limits are the classic: threads, file descriptors, sockets in a wait state. Pools are next — database connections, HTTP client connections, worker slots — where the limit produces a sharp latency cliff as callers begin queueing for a connection rather than for work. Memory produces a slow degradation through collection pressure before it produces an error. Downstream quotas and rate limits impose a ceiling that is not yours. And a class of limit exists that only appears as the fleet grows, where each instance opens connections to every other, or to a shared coordination service, so the total scales with the square of the node count or simply exceeds a limit nobody was tracking per host.

The November 2020 Kinesis event in us-east-1 is the reference case for that last category. Adding capacity to the front-end fleet pushed the number of threads on each server past an operating-system limit, so the trigger was a routine scale-up rather than a traffic spike. It makes the argument compactly: a test that only raises request rate can pass repeatedly while the real limit sits in a per-process ceiling that the test never approaches, and the event that finds it is an ordinary operational action rather than an unusual load.

So the test design should include deliberately raising the things that are not throughput. Hold request rate constant and increase payload size, or concurrent connections, or the number of distinct clients, or the dataset the queries run against. Each of these finds a different limit, and each has produced real outages.

What happens past the limit is the finding

Where the limit is matters less than how the system behaves once it is exceeded, because production will exceed it eventually. The behaviours to look for are whether latency degrades or requests are refused, whether the refusal is fast and cheap or a timeout that occupies a connection for thirty seconds, whether queues are bounded or grow until memory is exhausted, whether errors propagate to callers who then retry and amplify the load, and whether the system recovers on its own once load falls or needs a restart.

That last property is the one worth the most. A system that returns to normal when traffic subsides gives you an incident that ends by itself; a system that stays broken because its queues are full of expired work, or because every client is now retrying, gives you an incident that requires intervention. Testing recovery means ramping down as deliberately as you ramped up and measuring how long normal service takes to return.

Report all of it as a shape rather than a single number. The knee, the saturation point, the collapse point, the resource that ran out first, the behaviour past the limit, and the recovery time. Then set the autoscaling threshold, the rate limit and the alert against the knee rather than against the collapse, because by the time the collapse point is in sight there is no time left to react.