JVM Memory in Docker: -Xmx vs Container Limits and OOMKilled

Intermediate·3 min read·

Fast recall

Quick revision

Core answer

Before JDK 11, the JVM didn't respect cgroup memory limits — it sized the heap based on total host RAM, causing OOMKilled when the container exceeded its limit. JDK 11+ with -XX:+UseContainerSupport (default) reads the cgroup limit correctly. Use -XX:MaxRAMPercentage=75.0 instead of fixed -Xmx to size heap as a percentage of container memory.

Interview Answer3 min

Running a JVM inside a Docker container without explicit configuration causes the JVM to read total host memory rather than the container's memory limit, sizing its heap to 25% of the host RAM — on a 32 GB host, the JVM claims 8 GB of heap inside a container limited to 512 MB, and the kernel OOMKills it.

Container-aware heap sizing

JDK 10+ respects cgroup memory limits with -XX:+UseContainerSupport enabled by default — the JVM reads the container limit, not host RAM, and sets initial heap to 25% of the limit
-XX:MaxRAMPercentage=75.0 allocates 75% of the container limit to heap — leaving 25% for non-heap memory (Metaspace, thread stacks, direct buffers, the JVM itself); using 90%+ causes frequent GC pauses as the JVM runs up against the limit before OOMKill occurs
Verify the effective heap with java -XX:+PrintFlagsFinal -version | grep MaxHeapSize inside the container — compare the value to your Kubernetes resource limit to confirm they are consistent

The direct memory trap

Non-heap memory is not managed by -Xmx — Metaspace, thread stacks (1 MB per thread by default), and direct byte buffers (NIO, gRPC, Netty) all live outside the heap limit
A Spring Boot app under load with 50 active threads consumes 50 MB of stack alone; a Netty-based WebFlux app allocates significant direct memory; set the container limit to at least twice the configured heap to leave room for non-heap allocation
-XX:MaxDirectMemorySize limits direct buffer allocation; without it, direct memory allocation is unbounded and the total process footprint can far exceed the heap limit

Diagnostic and monitoring

When a container is OOMKilled, kubectl describe pod shows OOMKilled in the last state with an exit code of 137 — the kill came from the container runtime, not a JVM OutOfMemoryError; the JVM had no warning
Monitor RSS (resident set size) of the Java process with kubectl top pod, not just heap usage — RSS includes heap plus all non-heap memory and is what the kernel counts against the cgroup limit

Interview-ready guide

Interview answer

Container-aware heap sizing

JDK 10+ respects cgroup memory limits with -XX:+UseContainerSupport enabled by default — the JVM reads the container limit, not host RAM, and sets initial heap to 25% of the limit
-XX:MaxRAMPercentage=75.0 allocates 75% of the container limit to heap — leaving 25% for non-heap memory (Metaspace, thread stacks, direct buffers, the JVM itself); using 90%+ causes frequent GC pauses as the JVM runs up against the limit before OOMKill occurs
Verify the effective heap with java -XX:+PrintFlagsFinal -version | grep MaxHeapSize inside the container — compare the value to your Kubernetes resource limit to confirm they are consistent

The direct memory trap

Non-heap memory is not managed by -Xmx — Metaspace, thread stacks (1 MB per thread by default), and direct byte buffers (NIO, gRPC, Netty) all live outside the heap limit
A Spring Boot app under load with 50 active threads consumes 50 MB of stack alone; a Netty-based WebFlux app allocates significant direct memory; set the container limit to at least twice the configured heap to leave room for non-heap allocation
-XX:MaxDirectMemorySize limits direct buffer allocation; without it, direct memory allocation is unbounded and the total process footprint can far exceed the heap limit

Diagnostic and monitoring

When a container is OOMKilled, kubectl describe pod shows OOMKilled in the last state with an exit code of 137 — the kill came from the container runtime, not a JVM OutOfMemoryError; the JVM had no warning
Monitor RSS (resident set size) of the Java process with kubectl top pod, not just heap usage — RSS includes heap plus all non-heap memory and is what the kernel counts against the cgroup limit

03 · Deep dive