A service stops accepting new connections while CPU and memory look fine. Which operating-system limits would you suspect, and how would you tell them apart?

Bounded, not saturated

CPU and memory are the resources people watch because they degrade gradually. The limits in this question behave differently: they are hard ceilings, nothing gets slower as you approach one, and the moment you cross it the failure is total for new work while everything in flight continues normally. That signature — healthy machine, healthy existing connections, every new connection refused — should send you looking for a bound rather than a bottleneck.

The first discipline is to be precise about scope, because Linux enforces these at four different levels and confusing them wastes the outage. Some are per process, such as RLIMIT_NOFILE for open file descriptors. Some are per real user ID, and RLIMIT_NPROC is the one that catches people, because it counts threads and processes across every process that user is running, so a second copy of your service can exhaust it for the first. Some are per cgroup, which is how a container's pids.max bounds the whole container regardless of what the process thinks its own limit is. And some are system-wide, such as fs.file-max, with fs.nr_open capping how high a per-process descriptor limit can even be raised.

The arithmetic on one process

Descriptors are the usual culprit and the arithmetic is worth doing out loud. A soft limit of 1024 sounds generous until you count what consumes one: every accepted client socket, every outbound connection in the database pool, every open log file, every socket to a cache or a message broker, every epoll instance, plus a handful for the runtime's own use. A service holding a pool of 50 database connections, 20 to a broker and a dozen files has perhaps 940 left, so it will fail at somewhere near 900 concurrent clients — not at 1024 — and it will fail with no warning at all at 899.

The failure is legible if you know the mapping, which is why the errno matters more than the symptom.

Symptom	Error returned	Which ceiling
accept fails, existing connections fine	EMFILE	Per-process descriptor limit
accept or open fails across several processes at once	ENFILE	System-wide file table
Thread creation fails	EAGAIN from pthread_create	Per-user process and thread limit, or the cgroup pids limit
Outbound connect fails while inbound is fine	EADDRNOTAVAIL	Ephemeral port range exhausted for that destination
Clients time out with nothing logged server-side	none, the SYN is dropped	Listen queue overflowed

The last row is the one that produces the most confused debugging, because there is no error anywhere in your application. The kernel accepts the connection into a queue whose length is the smaller of the listen backlog you requested and net.core.somaxconn; when the application is not calling accept fast enough and that queue fills, further connection attempts are dropped, and the client sees a timeout. It looks like a network fault and it is a scheduling fault. The confirmation is a kernel counter rather than a log line: the listen-queue overflow and socket-drop counters visible through nstat or netstat -s will be climbing.

Ephemeral ports are similarly misattributed. A socket is identified by the four-tuple of local and remote address and port, so exhaustion is per destination rather than global, and the ceiling is the size of net.ipv4.ip_local_port_range — commonly around 28,000 usable ports on a default Linux configuration — minus everything sitting in TIME_WAIT after close. A service making many short-lived outbound connections to one upstream can fail to connect while the machine is almost idle, and the fix is connection reuse rather than a wider range.

Why raising the limit in your shell changes nothing

The reflex is to run ulimit -n 65536 and conclude the problem is solved. That changes the soft limit for that shell and its children only, and a service started by an init system is not one of its children. On a systemd-managed host the value that applies is LimitNOFILE in the unit, and nothing you type in a terminal affects the running service. In a container, the effective limits come from the runtime's configuration and the cgroup, not from anything in the image.

There is a second half to the trap. ulimit -n shows the soft limit, which a process may raise up to its hard limit itself, so a runtime that calls setrlimit on startup can differ from what your shell reports. Read the truth from the process instead: the limits as the kernel sees them for that PID, and the actual count of open descriptors for it. Confirm the specific ceiling before changing anything, because raising the wrong limit takes the same outage time as raising the right one and teaches you nothing.

When adding servers made each server fail

In November 2020, an AWS Kinesis front-end fleet in us-east-1 crossed the operating system's thread limit as capacity was added, so each server began to fail instead of the fleet getting larger. Resource limits are the operating-systems topic most often skipped in interviews and most often hit in production: threads, file descriptors and process counts are all bounded per process, and adding capacity can move you towards a bound rather than away from it. Being able to say where those limits live and what the failure looks like when one is reached is the concrete form of this knowledge.

The mechanism generalises to any fleet where each member holds a thread or a connection per peer. Adding peers then multiplies the per-node cost, so the scaling action itself pushes every existing node closer to its ceiling. The check is arithmetic you can do beforehand: express the per-node descriptor and thread count as a function of fleet size and compare it with the configured limit.

These are ceilings, not gradients, so nothing degrades before the failure and the machine looks healthy throughout. Identify which ceiling by the error the syscall returned, read the limit from the running process rather than from your shell, and check whether adding capacity moves you towards the bound rather than away from it.