Debugging OOMKilled Spring Boot in Kubernetes

Q: Debugging OOMKilled Spring Boot in Kubernetes

`OOMKilled` (exit code 137) means the cgroup limit was breached — the kernel SIGKILLed the container. First confirm kill reason with `kubectl describe pod`. Then classify: heap exhaustion (leak or undersized `-Xmx`), metaspace overflow (classloader leak), or thread stack accumulation. Use `jcmd VM.native_memory` and `jstat -gcutil` for live diagnosis; `-XX:+HeapDumpOnOutOfMemoryError` + `kubectl cp` for forensic heap dumps. Fix with `-XX:MaxRAMPercentage=75.0` (not `-Xmx = limit`) plus leak repair if confirmed.

Intermediate·3 min read·

AmazonGoogleNetflixPivotal

Fast recall

Quick revision

Core answer

OOMKilled (exit code 137) means the cgroup limit was breached — the kernel SIGKILLed the container. First confirm kill reason with kubectl describe pod. Then classify: heap exhaustion (leak or undersized -Xmx), metaspace overflow (classloader leak), or thread stack accumulation. Use jcmd VM.native_memory and jstat -gcutil for live diagnosis; -XX:+HeapDumpOnOutOfMemoryError + kubectl cp for forensic heap dumps. Fix with -XX:MaxRAMPercentage=75.0 (not -Xmx = limit) plus leak repair if confirmed.

Interview Answer3 min

OOMKilled means the kernel killed the container because it exceeded its memory limit — the container runtime reported exit code 137, not a JVM OutOfMemoryError. Distinguishing these two is the starting point for diagnosis.

Immediate diagnosis

kubectl describe pod under Last State shows OOMKilled with exitCode: 137 — this is kernel cgroup kill, not Java heap exhaustion; the JVM had no warning and no heap dump was written automatically.
kubectl get events --namespace --sort-by='.lastTimestamp' shows memory events and node pressure conditions around the kill.
Compare kubectl top pod (live RSS) against resources.limits.memory — if RSS is consistently at or above limit, the limit is too low for the actual footprint.

Root-cause classification

Heap too small for the workload: raise -XX:MaxRAMPercentage or the container memory limit; add -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heap.hprof so the next event produces a dump.
Non-heap exhaustion: thread stacks (1 MB default), metaspace, direct buffers live outside heap limits; a service with 100 active threads consumes 100 MB of stack alone — check with kubectl exec -- jcmd 1 Thread.print | grep -c 'java.lang.Thread.State'.
Memory leak: RSS grows continuously even at stable traffic; multiple heap dumps over time or JFR recording identifies the leaking object type in Eclipse MAT.

Heap dump extraction

Running pod: kubectl exec -- jcmd 1 GC.heap_dump /tmp/heap.hprof then kubectl cp :/tmp/heap.hprof ./heap.hprof — open in Eclipse MAT and use Leak Suspects.
After a kill: mount an emptyDir volume at /tmp and set HeapDumpPath=/tmp/ so the dump persists briefly after container exit; copy it out before the next start overwrites.

The fix, not just the bandaid

-XX:MaxRAMPercentage=75.0 with -XX:+UseContainerSupport keeps heap at 75% of the cgroup automatically — no hardcoded -Xmx to forget; the remaining 25% covers non-heap.
If jstat -gcutil shows Old Gen climbing after every full GC, the limit fix is a bandaid — you have a leak; do the heap dump and fix the retaining code before shipping a memory-limit bump.

Interview-ready guide

Interview answer

Immediate diagnosis

kubectl describe pod under Last State shows OOMKilled with exitCode: 137 — this is kernel cgroup kill, not Java heap exhaustion; the JVM had no warning and no heap dump was written automatically.
kubectl get events --namespace --sort-by='.lastTimestamp' shows memory events and node pressure conditions around the kill.
Compare kubectl top pod (live RSS) against resources.limits.memory — if RSS is consistently at or above limit, the limit is too low for the actual footprint.

Root-cause classification

Heap too small for the workload: raise -XX:MaxRAMPercentage or the container memory limit; add -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heap.hprof so the next event produces a dump.
Non-heap exhaustion: thread stacks (1 MB default), metaspace, direct buffers live outside heap limits; a service with 100 active threads consumes 100 MB of stack alone — check with kubectl exec -- jcmd 1 Thread.print | grep -c 'java.lang.Thread.State'.
Memory leak: RSS grows continuously even at stable traffic; multiple heap dumps over time or JFR recording identifies the leaking object type in Eclipse MAT.

Heap dump extraction

Running pod: kubectl exec -- jcmd 1 GC.heap_dump /tmp/heap.hprof then kubectl cp :/tmp/heap.hprof ./heap.hprof — open in Eclipse MAT and use Leak Suspects.
After a kill: mount an emptyDir volume at /tmp and set HeapDumpPath=/tmp/ so the dump persists briefly after container exit; copy it out before the next start overwrites.

The fix, not just the bandaid

-XX:MaxRAMPercentage=75.0 with -XX:+UseContainerSupport keeps heap at 75% of the cgroup automatically — no hardcoded -Xmx to forget; the remaining 25% covers non-heap.
If jstat -gcutil shows Old Gen climbing after every full GC, the limit fix is a bandaid — you have a leak; do the heap dump and fix the retaining code before shipping a memory-limit bump.

03 · Deep dive