Pods are being evicted during node pressure even though your CPU dashboards look fine. Where do you look, and what would you change so it stops?

Short answer

The CPU dashboards look fine because CPU is not an eviction trigger. It is a compressible resource — when a node runs out, the kernel throttles containers and everything gets slower. Eviction happens for incompressible resources: memory, ephemeral storage and inodes, where there is no way to give a process less without taking it away. And the pods chosen for eviction are selected by QoS class, not by which pod caused the pressure.

Compressible versus incompressible

This distinction explains the whole symptom and is worth leading with. A container exceeding its CPU allocation is throttled by the CFS scheduler: it gets fewer slices, its latency degrades, and it keeps running. Nothing is killed, so nothing is evicted, and your CPU graphs can sit at 100% all day producing slow responses and zero evictions.

Memory has no equivalent. A process cannot be given a fraction of a page it has already written to. When the node approaches its memory threshold the kubelet's only lever is to remove workloads, so it starts evicting.

Where to actually look

Node conditions tell you which resource is under pressure:

kubectl describe node <node> | grep -A6 Conditions
# MemoryPressure   True    kubelet has insufficient memory available
# DiskPressure     False
# PIDPressure      False

The eviction event names the threshold that was crossed:

kubectl get events --field-selector reason=Evicted -A
# The node was low on resource: ephemeral-storage.
# Container app was using 4Gi, which exceeds its request of 0.

That second line is the whole story in most incidents. ephemeral-storage pressure surprises people constantly, because nobody is monitoring it — and the usual cause is the application writing logs or temp files to the container filesystem rather than to stdout or a mounted volume. A pod that logs verbosely at debug level after a config change will fill the node's disk and get evicted while every memory and CPU graph looks healthy.

Inode exhaustion produces the same DiskPressure condition with plenty of free bytes, and comes from many small files — cached artifacts, session files, per-request temp files never cleaned up. df -i on the node is the check; df -h will look fine and mislead you.

Eviction is not an OOM kill

These get conflated and they behave differently, which matters for diagnosis.

Kubelet eviction is proactive. The kubelet watches node resources against its thresholds, and before the node is in real trouble it selects pods, terminates them gracefully, and records an Evicted status. The pod object remains in a failed state and the controller reschedules elsewhere.

An OOM kill is the kernel acting on a single cgroup. A container exceeding its own memory limit is killed immediately by the OOM killer, with OOMKilled in the container's lastState and a restart by the kubelet in place. No node-level pressure is required — one container exceeding its own limit is enough.

So Evicted points at the node, OOMKilled points at the container's own limit. Reading which one you have determines whether you are sizing a workload or a node.

Who gets evicted, and why it feels unfair

The kubelet ranks candidates by QoS class, then by how far each exceeds its request:

BestEffort — no requests, no limits set. Evicted first, always.

Burstable — requests set, limits higher or absent. Evicted next, worst offenders relative to request first.

Guaranteed — requests equal limits for every resource on every container. Evicted last, and only when there is nothing else to take.

The consequence is the one that catches teams out: the pod that caused the pressure is often not the pod that gets evicted. A memory-hungry batch job with generous requests is Burstable and well within its request; your critical API with no requests set is BestEffort and dies first. The batch job carries on. Nothing is malfunctioning — the scheduler is doing exactly what the manifests told it.

Fixing it

Set requests deliberately on everything that matters. A workload with no memory request is BestEffort and is volunteering to be evicted first. Base the request on observed usage — the p95 of the working set over a representative period — not on a guess or a copied value.

Make critical workloads Guaranteed. Setting requests equal to limits for CPU and memory on every container in the pod places it in the last group to be evicted. The cost is real: you reserve that capacity whether or not it is used, so cluster utilisation drops. That is the trade you are making and it is worth stating rather than presenting Guaranteed as free.

Get logs off the container filesystem. Write to stdout and let the node's log rotation handle it, or mount a volume with its own capacity. Also set container log rotation limits at the kubelet, since unbounded log files are the single most common ephemeral-storage eviction.

Set ephemeral-storage requests and limits, which most teams omit entirely. Without them the kubelet has no basis for ranking storage consumers, and the pod writing 40GB of temp files is indistinguishable from one writing nothing.

Reserve capacity for the system. --system-reserved and --kube-reserved keep the kubelet and container runtime from competing with workloads, so pressure produces orderly eviction rather than a node going NotReady and taking everything with it.

One thing to be clear about if asked: PodDisruptionBudgets do not help here. They constrain voluntary disruptions like drains and upgrades. Node-pressure eviction is involuntary, and the kubelet will evict regardless of what your PDB says — which is why the fix has to be QoS and sizing rather than a policy object.

---

© 2026 Preptima. Originally published at preptima.com.