When the heap is fine and memory still grows

Key points

• A JVM process uses far more than its heap. Metaspace, code cache, thread stacks, direct byte buffers, GC structures and the native allocator all sit outside -Xmx.
• This is why a container is killed with an OOM while heap dumps look perfectly healthy: the limit applies to the whole process, and the heap was never the problem.
• Metaspace grows with loaded classes. Repeated redeployment in one JVM, or heavy dynamic proxy generation, leaks it - and the classloader is kept alive by a single lingering reference.
• Direct byte buffers are freed only when their Java wrapper is collected, so native memory can be held long after it is logically unused. NIO and Netty allocate these heavily.
• Thread stacks are the quiet one: 1MB each by default, so a thread leak costs native memory rather than heap and shows up nowhere in a heap dump.
• Native Memory Tracking is the tool for all of this. Enable it, take a baseline, and diff - guessing at native memory is unproductive.

Example

# Native Memory Tracking - the only reliable way to see this.
$ java -XX:NativeMemoryTracking=detail -jar app.jar
$ jcmd <pid> VM.native_memory baseline
#   ... let it run ...
$ jcmd <pid> VM.native_memory summary.diff

#   Total: reserved=2.5GB, committed=1.8GB
#   -   Java Heap (reserved=1GB,   committed=1GB)
#   -     Class (reserved=200MB, committed=180MB  +40MB)  <- metaspace growing
#   -    Thread (reserved=520MB, committed=520MB  +60MB)  <- thread leak
#   -      Code (reserved=250MB, committed=120MB)
#   -        GC (reserved=100MB, committed=90MB)

# The arithmetic that explains a container OOM kill:
#
#   heap        1024 MB   (-Xmx1g)
#   metaspace    180 MB
#   code cache   120 MB
#   threads      520 MB   (520 threads x 1 MB stack)
#   direct        64 MB
#   GC + misc    190 MB
#   ---------------------
#   TOTAL       ~2.1 GB   against a 2 GB container limit -> killed
#
# And the heap dump would have shown 1 GB, looking entirely healthy.

# Bound the parts that are boundable:
-XX:MaxMetaspaceSize=256m
-XX:MaxDirectMemorySize=256m
-XX:ReservedCodeCacheSize=128m
-Xss512k                          # halves the cost of every thread
-XX:MaxRAMPercentage=60           # leave real room for the rest

A container OOM with a healthy heap means native memory - enable NMT and diff a baseline rather than guessing.