Your .NET service keeps getting OOM-killed in its container while the GC reports a small heap. Where has the memory gone?

Two different accountants

Start by naming the mismatch, because everything else follows from it. The kernel enforces the container's memory limit against the whole control group: every resident page the process owns, plus its file page cache, plus anything else in the cgroup. When that total crosses the limit, the OOM killer sends SIGKILL. There is no exception, no stack trace, no chance to log, just a process that vanishes and a restart count that goes up.

The GC reports something much narrower. A heap size figure — from GC.GetTotalMemory, from a counter, from a dump — is about objects on the managed heap. It is not the process working set, it does not include memory the GC has committed but is not currently using, and it says nothing about the several other consumers in the same process. So "the heap is 200 MB and we were killed at 1 GB" is not a contradiction. It is the normal case, and the diagnosis is a matter of finding which of the other consumers is large.

flowchart TD
  L[Container memory limit] --> G[GC heap committed]
  L --> N[Native allocations]
  L --> S[Thread stacks]
  L --> J[JIT code and loaded assemblies]
  L --> F[File page cache in the cgroup]
  G --> H[GC hard limit defaults to 75 percent of the container limit]

The edge worth noticing is that the GC governs exactly one of those boxes, and its own budget is expressed as a fraction of the limit rather than of what is left after the others have taken their share.

Committed is not live

The single most common resolution is that nothing has leaked at all. The GC commits memory in order to allocate into it, and it does not hurry to give it back, because returning pages to the operating system and taking them again is expensive and the process is probably about to want them. So resident memory tracks the high-water mark of allocation rather than the current live set, and a service that briefly deserialised a large payload keeps the committed pages long after the objects are collected.

That is why a graph of heap size against resident memory is more informative than either alone. If the live heap sawtooths between 100 and 300 MB while resident memory climbs to 800 MB and plateaus, you have an allocation rate problem rather than a retention problem, and the fix is in the allocation path: pooled buffers, streaming instead of materialising, avoiding a large intermediate array. Objects at or above 85,000 bytes go on the large object heap, which is not compacted by default, so a workload that repeatedly allocates arrays a little over that threshold fragments its way to a large committed heap with very little live data in it.

Server GC in a small container

The default for the runtime is Workstation GC, but the Web SDK turns Server GC on for ASP.NET Core applications, so most services in production are running it whether or not anyone chose it. Server GC allocates a separate heap and a dedicated GC thread per logical core, which is exactly what you want on a dedicated machine and can be wrong in a container.

The problem is the number of cores it counts. If your orchestrator gives the pod a CPU quota rather than a set of pinned cores, the runtime can see the machine's full core count, and on a 64-core node that means a great many heaps and GC threads for a container that is only allowed half a core of CPU. Each heap carries its own allocation budget, so the total the process commits before triggering a collection scales with the heap count, and a container with a small memory limit can be killed long before any individual heap looks interesting.

There are three defensible responses and they are not equivalent. Constrain the heap count explicitly so it matches what the container is actually allowed to use. Switch to Workstation GC, which uses a single heap and accepts more frequent, shorter pauses in exchange for a much smaller footprint — a good trade for a small container running many replicas. Or let DATAS do it, which adapts the heap count and budget to the application's live data size at runtime and has been enabled by default since .NET 9; on .NET 8 it exists but is off unless you switch it on.

The GC's hard limit versus the kernel's

These produce different symptoms and confusing them wastes a lot of time. Since .NET Core 3.0 the runtime reads the cgroup memory limit and, if you set nothing, applies a heap hard limit of 75% of that limit, or 20 MB, whichever is larger. When the GC hits its hard limit it throws OutOfMemoryException — a managed exception, in your code, with a stack, which you can log and alert on.

Being SIGKILLed is the other case, and it means the total crossed the container limit while the GC still believed it was within budget. That is almost always the other 25% being oversubscribed by things the GC does not manage. So the two symptoms tell you where to look: a managed OutOfMemoryException points at the managed heap, and a silent kill points at everything else.

The GC also becomes more aggressive about full compacting collections when memory load crosses a high threshold, which defaults to 90% and is measured against the container limit rather than the host's memory when a limit is set. A service that plateaus just under that line pays for it in pause time, and the symptom is latency rather than a crash.

What lives outside the managed heap

Name the specific consumers rather than gesturing at native memory. Every thread has a stack, committed on use, so a library that creates a thread per connection or a thread pool that has grown to hundreds of threads under starvation costs real resident memory. Loaded assemblies, their metadata and the code the JIT emits all live outside the GC heap and grow with the size of your dependency graph. Native libraries allocate through the C allocator and are entirely invisible to managed diagnostics — a database driver's connection buffers, a compression or crypto library, an image codec.

Two more are worth calling out because they are easy to misread. Pooled buffers, from ArrayPool or a recycling stream, are managed objects deliberately kept alive; that is the whole point, and their memory is a fixed cost you chose rather than a leak, but a pool sized for peak throughput has to fit in the container. And memory-mapped files or heavy file I/O add page cache to the cgroup, which counts against the limit even though no allocation in your code corresponds to it.

Measure in this order

Take the working set and the managed heap side by side first, over time, from dotnet-counters, so you know which of them is growing. If it is the managed heap, take two heap snapshots minutes apart with dotnet-gcdump and diff them by type — a growing type with a growing retention path is a leak, and the path names the cache or the event handler holding it. If the managed heap is flat and the working set is not, the remainder is native or stack or code, and a full dump plus the thread count and the loaded-module list will tell you which.

Only then reach for a setting, and prefer the one that expresses your actual intent. Constraining the heap count or switching GC flavour says something about the shape of your deployment. Setting a heap hard limit lower than the default converts a silent kill into a catchable OutOfMemoryException, which is a genuine improvement in observability even when it does not fix the underlying growth. What none of them do is fix an allocation pattern, and a knob applied before a measurement usually moves the failure rather than removing it.

The kernel counts the whole control group and the GC reports one part of it, so the useful first question is never how big the heap is but which of committed heap, native memory, stacks or code is the part that grew.