Debugging CrashLoopBackOff: Why Pods Crash and How to Fix Them

CrashLoopBackOff means the container keeps exiting and kubelet is waiting before the next restart with exponential backoff from 10 seconds up to 5 minutes. The symptom is a loop; the root cause is one of a handful of things.

The diagnostic flow I use

• kubectl describe pod and read the Last State section — that gives the exit code and reason. 137 means OOMKilled, 1 is an app error, 143 is SIGTERM, 2 is an ENTRYPOINT issue.
• kubectl logs POD --previous — this is the critical step most people miss. The current container just started so kubectl logs returns almost nothing; --previous shows the last crashed container's output which usually has the stack trace.
• Check the Events section for liveness probe failures — if the probe is killing the pod during startup, the app didn't crash, it was SIGKILLed.

The big four root causes

• App crash on startup — bad DB credentials, missing config, port conflict. Fix via logs: "password authentication failed" or "Could not resolve placeholder".
• Liveness probe killing a slow-starting JVM — Spring Boot takes 30 to 60 seconds to warm up; liveness fires before then, kubelet SIGKILLs. Fix: add a startupProbe so liveness only starts after startup succeeds once.
• Init container failing — pod shows Init:CrashLoopBackOff and main container never starts. Check init logs with -c INIT_NAME --previous.
• OOMKilled (exit code 137) — cgroup memory limit hit. Raise memory limit and set -XX:MaxRAMPercentage=75 so JVM respects the container limit.

Probe configuration

• Use /actuator/health/liveness from Spring Boot 2.3+, not aggregate /actuator/health — the liveness endpoint returns UP even if DB is temporarily down, so transient DB issues don't cause pointless restarts.
• startupProbe plus livenessProbe plus readinessProbe is the production pattern — startup covers warm-up, liveness restarts on true hangs, readiness takes the pod out of Service rotation during transient issues.

Tools that help when logs aren't enough

• kubectl debug --image=busybox --target=CONTAINER attaches an ephemeral container sharing process and filesystem namespace — you can inspect /proc, environment, mounted configs without rebuilding the image.
• terminationMessagePath lets the container write its cause-of-death; kubelet surfaces it in describe pod, so root cause is visible even if logs were lost.

Escalation pattern

• One pod — app bug, config typo, probe misconfig.
• All pods of one Deployment — bad image, wrong Secret, env misconfig.
• Many Deployments — node issue, DNS failure cluster-wide, or platform-level problem.

Production hardening I apply

• Startup probe on every JVM service.
• -XX:+HeapDumpOnOutOfMemoryError writing to an emptyDir at /dumps so I can copy the dump out before the pod is replaced.
• Prometheus alert on restart rate per 10 minutes — catches CrashLoop before users notice.