Heap dumps and thread dumps

Key points

• A thread dump is a snapshot of every thread and its stack. Take three, ten seconds apart, and compare them.
• The same stack in all three means stuck. A BLOCKED thread names the lock and the thread that is holding it.
• jcmd <pid> Thread.print is the modern way and jstack still works. Neither one restarts anything.
• A heap dump is every object on the heap. It is roughly the size of the live heap, so check the disk first.
• jcmd <pid> GC.heap_dump app.hprof pauses the JVM while it writes. Do it on one instance, not the whole fleet.
• Open the dump in Eclipse MAT and read the dominator tree. It names the single object holding everything else alive.

Example

import java.lang.management.ManagementFactory;
import java.lang.management.ThreadInfo;
import java.lang.management.ThreadMXBean;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        Object lock = new Object();
        Thread waiter = new Thread(() -> {
            synchronized (lock) {
                try {
                    lock.wait(300);
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            }
        }, "waiter");
        waiter.start();
        Thread.sleep(100);

        ThreadMXBean bean = ManagementFactory.getThreadMXBean();
        System.out.println("live=" + bean.getThreadCount()
                + "  peak=" + bean.getPeakThreadCount()
                + "  daemon=" + bean.getDaemonThreadCount());
        System.out.println();
        for (ThreadInfo info : bean.dumpAllThreads(false, false)) {
            System.out.printf("  %-22s %-14s frames=%d%n",
                    info.getThreadName(), info.getThreadState(), info.getStackTrace().length);
        }
        System.out.println();
        long[] stuck = bean.findDeadlockedThreads();
        System.out.println("deadlocks found: " + (stuck == null ? "none" : String.valueOf(stuck.length)));
        System.out.println("This is what jcmd Thread.print prints, taken from the inside.");
        waiter.join();
    }
}

Thread dump for a hang, heap dump for a leak. Take them before you restart.