Thread Dumps: Capture, Analyze, Diagnose

When a Java service hangs, runs slow, or mysteriously stops responding, a thread dump shows exactly what every thread is doing right now. It's one of the most valuable diagnostic tools for JVM production issues.

The capture-and-analyze flow

Capture 3 dumps, 10 seconds apart

• A single dump is misleading; doesn't distinguish 'genuinely stuck' from 'momentarily here'
• Three dumps separated by 10 seconds reveal which threads are actually stuck vs just doing work

Capture options

• jstack pid: most common; requires same JDK version as the JVM; SSH access
• kill -3 pid: writes to stdout; works without matching JDK; output goes to logs
• Spring Boot Actuator /threaddump: works without SSH if endpoint enabled; JSON format
• jcmd pid Thread.print: modern alternative; more flexible; use -l for lock info
• JDK Flight Recorder: continuous profiling for 60 seconds; heavier but richer

In Kubernetes

• kubectl exec into pod; jstack PID 1
• Or kubectl debug with ephemeral debug container if prod image lacks JDK tools
• Some images bake in jattach — tiny binary, no JDK needed

The analysis flow

Count threads per state

• grep 'Thread.State' dump.txt | sort | uniq -c | sort -rn
• Many BLOCKED means lock contention
• Many WAITING on a downstream pattern means downstream slow or pool exhausted
• Many RUNNABLE but wall-clock slow means CPU saturation
• Thread count above expected means leak

Group threads by stack pattern

• 120 threads stuck at PaymentService.callStripe means Stripe is the bottleneck
• Many threads at HikariCP.borrowObject means DB pool exhausted
• Many threads at the same synchronized block means lock contention

Look for BLOCKED threads

• 'BLOCKED on java.util.concurrent.locks.ReentrantLock owned by thread-7'
• Find thread 7; what is it doing?
• Often reveals cascading blocks — thread-7 itself waiting on something else

Look for WAITING threads on infrastructure

• At HikariPool.borrowObject means DB pool exhausted
• At BlockingQueue.take means queue starved or worker sleeping
• At HTTP client read means downstream service slow

Look for hot RUNNABLE threads

• Many RUNNABLE in same hot code path = CPU bottleneck
• Cross-reference with async-profiler to confirm which method eats CPU

Key technique: cross-dump comparison

• A thread that appears in same state and same stack across all 3 dumps is genuinely stuck
• A thread that changes state is doing its job
• Tools like FastThread.io automate this comparison

States and what they mean

• BLOCKED: waiting for intrinsic synchronized lock; another thread holds it
• WAITING: Object.wait() or LockSupport.park(); no timeout
• TIMED_WAITING: same with timeout; will auto-wake
• RUNNABLE: eligible to run; may still be blocked on socket I/O (JVM doesn't distinguish from CPU-bound)
• RUNNABLE doesn't mean CPU-bound; use top -H or jvmtop for actual CPU per thread

Common patterns

• Thread pool starvation: 200 worker threads all WAITING on same downstream; request queue backs up
• Lock contention: threads BLOCKED on same monitor
• Connection pool exhaustion: threads WAITING in HikariPool.borrowObject
• Deadlock: jstack detects and reports explicitly at end of output
• GC thrashing: application threads at 100% CPU but wall-clock slow; combine with GC logs

Virtual threads in JDK 21+

• Don't appear in traditional thread dumps the same way
• Use jcmd Thread.dump_to_file -format=json for JSON format
• Tooling is catching up; FastThread.io supports this

Complementary tools

• async-profiler: continuous CPU profiling with low overhead; shows where CPU goes over time
• Heap dumps: what objects are in memory; different diagnostic need
• GC logs: confirm if GC pressure is the issue
• Thread dump is a moment; profiler shows trends; combine for full picture

The strategic answer

• Thread dumps are cheap to capture and immensely diagnostic
• Always capture 3 dumps 10 seconds apart
• Group by stack, count by state, correlate with metrics
• FastThread.io or IBM TDA for complex cases; grep/awk for quick looks
• Protect the endpoint; don't expose /threaddump publicly