A pod is stuck in CrashLoopBackOff. How do you debug it?
CrashLoopBackOff means the container starts and exits repeatedly, so read the previous container's logs and the exit code first, then work outward through probes, configuration, dependencies, and resource limits.
What the interviewer is scoring
- Whether you know to read the PREVIOUS container's logs, not the current one
- Whether you use the exit code to narrow the cause rather than guessing
- Whether you distinguish an application crash from a failing liveness probe
- Whether you check events and resource limits, not only logs
Answer
What the status actually means
CrashLoopBackOff is not an error in itself — it is Kubernetes telling you that the container has exited repeatedly and the kubelet is now backing off between restart attempts, with the delay doubling up to five minutes. The important implication is that the container did start. That rules out image pull problems and most admission or scheduling issues, and points you at either the application exiting or something killing it.
Read the previous container's logs
This is the detail that separates people who have debugged this from people who have read about it. By the time you run kubectl logs, the crashed container has already been replaced, so you are looking at a fresh instance that may have logged nothing yet.
# The one that actually matters - logs from the instance that died.
kubectl logs <pod> --previous
# Multi-container pods need the container named explicitly.
kubectl logs <pod> -c <container> --previous
Get the exit code
The exit code narrows the cause faster than anything else, and it lives in the pod description rather than the logs.
kubectl describe pod <pod>
# or, precisely:
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'
The codes worth recognising on sight:
- 0 — the process exited successfully. Almost always a container whose main process is not long-running: a script that completed, or an entrypoint that forks and returns. This should probably be a Job, not a Deployment.
- 1 or another small non-zero — the application itself failed. The logs will tell you why, and this is the most common case.
- 137 — SIGKILL, which is 128 + 9. Usually OOMKilled: the container exceeded its memory limit. Confirm in
describeunder Last State, where the reason is stated explicitly. Occasionally it is a liveness probe kill after the grace period. - 139 — SIGSEGV, a segmentation fault, pointing at a native library or binary issue.
- 143 — SIGTERM, 128 + 15, meaning something asked it to stop. Frequently a failing liveness probe.
Work outward through the causes
Application configuration. A missing or malformed environment variable, a ConfigMap or Secret key that does not exist, or a bad connection string. A referenced-but-absent Secret usually surfaces as CreateContainerConfigError rather than a crash loop, but a present but wrong value produces exactly this.
A liveness probe that is too aggressive. This one is worth calling out because it is the most misdiagnosed. If the application takes 40 seconds to start and the liveness probe begins checking at 10 seconds, Kubernetes kills a perfectly healthy container mid-startup, forever. The symptom looks identical to an application crash. The tell is a healthy-looking log that gets cut off partway through startup, plus Liveness probe failed in the events. The fix is a startup probe, which suspends liveness checking until the application reports ready, rather than inflating initialDelaySeconds and thereby delaying real failure detection for the whole life of the pod.
A dependency that is not reachable. A service that exits when it cannot reach its database on boot will crash-loop until the database is up. This is arguably correct behaviour, and the fix is usually retry-with-backoff on startup rather than changing the Kubernetes configuration.
Resource limits. Beyond outright OOM kills, a memory limit set close to the application's steady-state usage produces intermittent kills under load that look random. Compare the limit against actual usage rather than against the request, and remember that for JVM workloads the heap must be sized well below the container limit to leave room for metaspace, thread stacks, and native allocations.
Events, which people skip. kubectl describe pod lists them at the bottom, and kubectl get events --sort-by=.lastTimestamp gives cluster-wide context. Probe failures, evictions, and node pressure show up here and nowhere in the logs.
When there are no logs at all
If the container dies before producing output, the loop is too fast to inspect. Two approaches:
Override the entrypoint so the container stays up, then investigate inside it — run a copy with command: ["sleep", "3600"], exec in, and try starting the process by hand to see the failure directly. Or use kubectl debug to attach an ephemeral container that shares the target's namespaces, which lets you inspect the environment and filesystem without modifying the workload.
Preventing it
The follow-up about prevention is where senior candidates separate themselves. A readiness probe distinct from the liveness probe, so a slow start does not remove a healthy pod from service and does not get it killed. A startup probe for anything with a long initialisation. Configuration validated at boot with a clear fatal error rather than a stack trace, so the log immediately names the missing key. Resource limits derived from measured usage under load rather than copied from another service. And the same manifests exercised in a staging environment, since a crash loop that only appears in production usually means production has configuration that staging does not.
Likely follow-ups
- The logs are empty and the exit code is 137. What happened?
- How would you debug a container that crashes before it can log anything?
- How does this differ from ImagePullBackOff or CreateContainerConfigError?
- What would you change so this is detected before it reaches production?
Related questions
- What happens when a container hits its CPU limit, and what happens when it hits its memory limit?mediumAlso on oomkilled4 min
- Every rolling update drops a small number of requests. Where do they go?hardAlso on kubernetes4 min
- A dashboard has been showing the same temperature for two days and nobody noticed. Walk the path and tell me what would have caught it.hardSame kind of round: scenario6 min
- Everyone in the business calls it the fax queue, and nothing has been faxed since 2014. Do you keep that name in the code?mediumSame kind of round: concept4 min
- How do you split a story that is too big to fit in a Sprint?mediumSame kind of round: concept3 min
- The business insists on a requirement that nobody can test — "the system must be intuitive". What do you write instead?mediumSame kind of round: concept4 min
- How does your order placement change on a venue that allocates pro rata within a price level instead of first in, first out?hardSame kind of round: concept6 min
- The business says this story cannot be delivered in parts. How do you split it anyway?mediumSame kind of round: scenario4 min