Native memory: why RSS beats -Xmx

Key points

• A Java process is the heap plus metaspace, code cache, GC structures, thread stacks, direct buffers and the JDK itself.
• Thread stacks are native memory. 400 threads at the default 1 MB -Xss is 400 MB that nobody put in the budget.
• Direct ByteBuffers and memory-mapped files sit outside the heap entirely. Netty and NIO allocate plenty of them.
• Switch on -XX:NativeMemoryTracking=summary and read it with jcmd <pid> VM.native_memory summary.
• A kernel OOM-kill is SIGKILL: exit code 137, no stack trace, no heap dump. It is not an OutOfMemoryError.
• Rule of thumb: leave a quarter of the container limit outside the heap, which is exactly what MaxRAMPercentage=75 does.

Example

import java.lang.management.BufferPoolMXBean;
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryPoolMXBean;
import java.lang.management.MemoryType;
import java.nio.ByteBuffer;

public class Main {
    public static void main(String[] args) {
        Runtime rt = Runtime.getRuntime();
        ByteBuffer direct = ByteBuffer.allocateDirect(16 * 1024 * 1024);   // 16 MB, off the heap
        direct.putInt(0, 42);

        System.out.println("the heap - the only part -Xmx controls");
        System.out.printf("  max %d MB, used %d MB%n",
                rt.maxMemory() >> 20, (rt.totalMemory() - rt.freeMemory()) >> 20);
        System.out.println();
        System.out.println("everything else the process is holding, none of it inside -Xmx:");
        for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) {
            if (pool.getUsage() == null || pool.getType() == MemoryType.HEAP) { continue; }
            System.out.printf("  %-38s %8d KB%n", pool.getName(), pool.getUsage().getUsed() >> 10);
        }
        for (BufferPoolMXBean pool : ManagementFactory.getPlatformMXBeans(BufferPoolMXBean.class)) {
            System.out.printf("  %-38s %8d KB   (%d live)%n",
                    "buffers: " + pool.getName(), pool.getMemoryUsed() >> 10, pool.getCount());
        }
        System.out.println();
        System.out.println("Thread stacks add -Xss for every thread on top of all of this,");
        System.out.println("and the container limit applies to the sum, not to the heap.");
        System.out.println("checksum " + direct.getInt(0));
    }
}

Size the container for the whole process, not for the heap.