Reading a thread dump properly

Key points

• Take three dumps, a few seconds apart. One dump shows a moment; three show whether anything is moving, which is the actual question.
• Thread states mean specific things. BLOCKED is waiting for a monitor. WAITING is waiting to be signalled. TIMED_WAITING is a sleep or a timed wait. RUNNABLE includes threads blocked on socket reads, which is a persistent source of confusion.
• The JVM detects and prints genuine deadlocks for you - a cycle of threads each holding what the next wants. It is one of the few problems the dump names outright.
• Pool exhaustion looks quite different: dozens of threads with identical stacks, all parked in the same place - typically waiting for a database connection.
• A live-lock is threads that are RUNNABLE across all three dumps but never progress - usually a retry or CAS loop that keeps failing.
• Look for the same stack repeated across many threads. In a real incident the pattern is nearly always a crowd doing one thing, not one exotic thread doing something strange.

Example

# Three dumps, five seconds apart. Comparison is the diagnostic.
$ for i in 1 2 3; do jstack <pid> > dump-$i.txt; sleep 5; done

# Which stacks are crowded - usually the whole answer.
$ grep -A1 "^\"" dump-1.txt | grep "at " | sort | uniq -c | sort -rn | head

#   47 at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:...)
#   ^^ 47 threads waiting for a database connection = pool exhaustion,
#      and every other symptom is downstream of it

---

# DEADLOCK - the JVM names it for you:
#
#   Found one Java-level deadlock:
#   "worker-1" waiting to lock <0x00000000d5a2> held by "worker-2"
#   "worker-2" waiting to lock <0x00000000d5b1> held by "worker-1"

# BLOCKED - contention on one monitor:
#   "http-nio-8080-exec-3" BLOCKED
#      at Cache.get(Cache.java:42)
#      - waiting to lock <0x00000000d5c3> (a java.lang.Object)

# RUNNABLE but doing nothing - a socket read. Not a CPU problem:
#   "http-nio-8080-exec-7" RUNNABLE
#      at java.net.SocketInputStream.socketRead0(Native Method)
#   -> this thread is WAITING on the network despite saying RUNNABLE

# Signatures, side by side:
#
#   deadlock          JVM prints it; nothing moves, ever
#   pool exhaustion   many identical stacks parked on a pool
#   live-lock         RUNNABLE in all 3 dumps, no progress, high CPU
#   slow downstream   many threads in socketRead0, low CPU

Take three dumps and count repeated stacks - the answer is nearly always a crowd of threads waiting on one thing.