Diagnosing a hung application

Key points

• Take the dump with jcmd <pid> Thread.print. Take three, ten seconds apart, and compare them.
• The JVM finds lock cycles itself and prints "Found one Java-level deadlock" with both stacks, so that case needs no detective work.
• Many threads BLOCKED on one monitor with near-zero CPU means contention. High CPU with unchanged stacks means a livelock.
• Threads WAITING on a pool queue while the queue keeps growing means the pool is too small, or a task is blocking inside it.
• ThreadMXBean.findDeadlockedThreads() runs the same detection from inside the process, which suits a health check.

Example

import java.lang.management.ManagementFactory;
import java.lang.management.ThreadInfo;
import java.lang.management.ThreadMXBean;
import java.util.concurrent.CountDownLatch;

public class Main {
    static final Object LOCK = new Object();

    public static void main(String[] args) throws Exception {
        CountDownLatch release = new CountDownLatch(1);

        Thread holder = new Thread(() -> { synchronized (LOCK) { await(release); } }, "holder");
        holder.start();
        Thread.sleep(100);

        Thread waiter = new Thread(() -> { synchronized (LOCK) { System.out.println("waiter   : finally got in"); } }, "waiter");
        waiter.start();
        Thread.sleep(100);

        // This is what jcmd Thread.print shows, minus the stack frames.
        ThreadMXBean mx = ManagementFactory.getThreadMXBean();
        for (ThreadInfo info : mx.getThreadInfo(new long[] { holder.threadId(), waiter.threadId() }, 2)) {
            System.out.println(pad(info.getThreadName()) + ": " + info.getThreadState()
                    + (info.getLockName() == null ? "" : " on " + info.getLockName())
                    + (info.getLockOwnerName() == null ? "" : " owned by " + info.getLockOwnerName()));
        }

        long[] cycle = mx.findDeadlockedThreads();          // the JVM finds lock cycles for you
        System.out.println("deadlocks: " + (cycle == null ? "none - this is contention, not deadlock" : cycle.length + " threads"));
        System.out.println("live     : " + Thread.getAllStackTraces().size() + " threads in this JVM");

        release.countDown();
        holder.join();
        waiter.join();
        System.out.println("advice   : take three dumps ten seconds apart and compare them");
    }

    static String pad(String s) { return (s + "         ").substring(0, 9); }

    static void await(CountDownLatch l) { try { l.await(); } catch (InterruptedException e) { Thread.currentThread().interrupt(); } }
}

Three dumps, ten seconds apart. What changed between them names the hang.