- 5 minutes to read

Thread Deadlock Detection

This troubleshooting guide explains thread deadlock detection, how to read a thread dump, and how to resolve thread contention in JVM applications monitored by the Nodinite JMX Monitoring Agent.

Understanding Thread Deadlocks

A thread deadlock occurs when two or more Java threads (also referred to as JVM threads) each hold a lock that another thread is waiting to acquire. None of the threads can proceed — they wait indefinitely, causing the application to stall or become unresponsive.

graph LR A[" Thread A\nHolds Lock 1\nWaiting for Lock 2"] B[" Thread B\nHolds Lock 2\nWaiting for Lock 1"] A -- "waiting for" --> B B -- "waiting for" --> A style A fill:#FF6B6B style B fill:#FF6B6B

Diagram: Thread deadlock — Thread A waits for Lock 2 held by Thread B, while Thread B waits for Lock 1 held by Thread A.

Common Causes of Thread Deadlocks

  • Acquiring multiple locks in inconsistent order across different code paths
  • Holding a database connection lock while waiting for an application-level lock
  • Synchronised blocks nested inside other synchronised blocks
  • Third-party libraries that use internal locking mechanisms conflicting with application locks
  • Boomi Atom connectors that perform synchronous operations while holding process locks

How the JMX Monitoring Agent Detects Deadlocks

The Nodinite JMX Monitoring Agent uses the standard JMX ThreadMXBean.findDeadlockedThreads() method to detect thread deadlock conditions. When a deadlock is detected:

  1. The agent records the blocked threads involved in the deadlock.
  2. A deadlock alert is raised in the Nodinite monitoring console.
  3. The alert includes thread IDs and lock ownership details from the JMX thread monitor.
  4. The alert severity is set according to the configured threshold in the JMX Monitoring Agent Configuration.

Important

A thread deadlock does not automatically resolve itself. The JVM application typically requires a restart to recover. Use the deadlock alert to trigger an immediate investigation.

Reading a Thread Dump

A thread dump captures the state of all JVM threads at a point in time. Thread dumps are the primary tool for diagnosing deadlock resolution and thread contention issues.

Generating a Thread Dump

Generate a thread dump using one of the following methods:

# Method 1: Using jstack (recommended)
jstack <PID> > thread-dump.txt

# Method 2: Using kill signal on Linux/macOS
kill -3 <PID>

# Method 3: Using PowerShell on Windows
# First find the PID with: jps -l
jstack.exe <PID>

Tip

Capture 2–3 thread dumps spaced 15 seconds apart to distinguish truly blocked threads from momentarily waiting threads. Threads that appear blocked in all dumps are involved in a deadlock or severe thread contention.

Identifying Deadlocked Threads in a Thread Dump

A thread stack trace section for a deadlocked thread looks like this:

"Thread-A" #42 prio=5 os_prio=0 tid=0x00007f tid=BLOCKED
   java.lang.Thread.State: BLOCKED (on object monitor)
   at com.example.ServiceA.process(ServiceA.java:57)
   - waiting to lock <0x00000000f8b3c420> (a java.lang.Object)
   - locked <0x00000000f8b3c500> (a java.lang.Object)

"Thread-B" #43 prio=5 os_prio=0 tid=0x00007f tid=BLOCKED
   java.lang.Thread.State: BLOCKED (on object monitor)
   at com.example.ServiceB.process(ServiceB.java:34)
   - waiting to lock <0x00000000f8b3c500> (a java.lang.Object)
   - locked <0x00000000f8b3c420> (a java.lang.Object)

Found one Java-level deadlock:
=============================
"Thread-A": waiting to lock monitor 0x00000000f8b3c420
  which is held by "Thread-B"
"Thread-B": waiting to lock monitor 0x00000000f8b3c500
  which is held by "Thread-A"

Key sections to look for:

  • BLOCKED (on object monitor) — The thread is waiting to acquire a lock.
  • waiting to lock <address> — The lock address the thread is waiting for.
  • locked <address> — The lock the thread currently holds.
  • Found one Java-level deadlock — The JVM has confirmed a thread deadlock.

Resolving Thread Deadlocks

Immediate Resolution

A thread deadlock requires a JVM restart to release the locked resources. Once the JVM is restarted:

  1. Confirm the deadlock alert clears in the Nodinite monitoring console.
  2. Capture a thread dump on startup to verify no deadlock is immediately re-created.
  3. Review application logs for any DeadlockException or InterruptedException entries.

Long-Term Fixes

To prevent recurring thread deadlocks, apply the following patterns:

  • Consistent lock ordering — Always acquire locks in the same order across all code paths. If ServiceA acquires Lock 1 then Lock 2, ensure no other code acquires Lock 2 then Lock 1.
  • Lock timeouts — Use java.util.concurrent.locks.ReentrantLock.tryLock(timeout) instead of synchronized blocks to implement a timeout that prevents infinite waiting.
  • Reduce lock scope — Minimize the amount of code inside synchronised blocks. Release locks as quickly as possible.
  • Thread-safe data structures — Use java.util.concurrent collections (ConcurrentHashMap, CopyOnWriteArrayList) to reduce the need for explicit locking.

Common Deadlock Questions

The deadlock alert fires but the application seems responsive

Cause: Some deadlocks affect only background threads (cleanup, housekeeping, batch processing) that do not impact request handling. The application may appear responsive while background tasks stall.

Resolution: Investigate the blocked threads identified in the thread dump. Even if the application is currently responsive, a background deadlock can cascade to user-facing threads over time.

Deadlock alerts fire on Boomi Atoms during high load

Cause: Boomi Atom connectors and process execution can use synchronized operations that create thread contention under high concurrency.

Resolution:

  1. Reduce the number of concurrent Boomi processes using the Boomi AtomSphere listener settings.
  2. Increase the Boomi Atom's JVM thread pool size.
  3. Review Boomi connector configuration for connection pool exhaustion, which can cause threads to block waiting for connections.

How do I configure the deadlock alert threshold?

The Nodinite JMX Monitoring Agent raises a deadlock alert when ThreadMXBean.findDeadlockedThreads() returns one or more threads. This is a binary state — the alert fires on any detected deadlock regardless of thread count.

Configure the alert severity and notification settings in the JMX Monitoring Agent Configuration.

Next Step

Supported Java Versions OS vs JVM Monitoring

Multi-JVM Monitoring Heap Used vs Committed JMX Monitoring Agent JMX Troubleshooting Overview