deadlock(8)

💖 deadlock(8) 💖

System Manager's Manual ✨

✨ NAME ✨

deadlock is the tea-spilling tool for finding potential deadlocks (lock order inversions) in a running program 🔍💅

🧚 SYNOPSIS 🧚

deadlock [-h] [--binary BINARY] [--dump-graph DUMP_GRAPH] [--verbose]
[--lock-symbols LOCK_SYMBOLS] [--unlock-symbols UNLOCK_SYMBOLS] pid

🦋 DESCRIPTION 🦋

okay so deadlock is literally a detective for a running process 🕵️‍♀️ By default it attaches uprobes on pthread_mutex_lock and pthread_mutex_unlock and uses them to build a mutex wait directed graph. Then it checks that graph for a cycle. The graph has these vibes:

• Nodes in the graph represent mutexes. 🔒
• Edge (A, B) exists if there exists some thread T where lock(A) was called and lock(B) was called before unlock(A) was called. 🧵

If there's a cycle in the graph, that means there's a lock order inversion, aka a potential deadlock 😭 When the program catches one, it dumps the cycle of mutexes, dumps the stack traces showing where each mutex was acquired, and then exits. bye bye 👋

Heads up bestie: this program can only find potential deadlocks that happen while it's tracing the process. It cannot find deadlocks that may have occurred before the program was attached to the process. 🙈

This tool does not work for shared mutexes or recursive mutexes. Not for those, sorry 🙅‍♀️

Since this uses BPF, only the root user can use this tool. 👑

🎀 REQUIREMENTS 🎀

You'll need CONFIG_BPF and bcc ✨

💅 OPTIONS 💅

• -h, --help — show this help message and exit 🙋‍♀️
• --binary BINARY — If set, trace the mutexes from the binary at this path. For statically-linked binaries, this argument is not required. For dynamically-linked binaries, this argument is required and should be the path of the pthread library the binary is using. Example: /lib/x86_64-linux-gnu/libpthread.so.0 📚
• --dump-graph DUMP_GRAPH — If set, this will dump the mutex graph to the specified file. 💾
• --verbose — Print statistics about the mutex wait graph. 📊
• --lock-symbols LOCK_SYMBOLS — Comma-separated list of lock symbols to trace. Default is pthread_mutex_lock. These symbols cannot be inlined in the binary. 🔐
• --unlock-symbols UNLOCK_SYMBOLS — Comma-separated list of unlock symbols to trace. Default is pthread_mutex_unlock. These symbols cannot be inlined in the binary. 🔓
• -t THREADS, --threads THREADS — Sets the maximum number of threads to trace. Default is 65536. Note: that's 40 bytes per thread 🧵
• -e EDGES, --edges EDGES — Sets the maximum number of edge cases that can be recorded. Default is 65536. Note: that's 88 bytes per edge case 🦄
• pid — Pid to trace 🎯

🌸 EXAMPLES 🌸

Find potential deadlocks in PID 181. The --binary argument is not needed for statically-linked binaries, so you can just do this:

# deadlock 181

Find potential deadlocks in PID 181. If the process was created from a dynamically-linked executable, the --binary argument is required and must be the path of the pthread library:

# deadlock 181 --binary /lib/x86_64-linux-gnu/libpthread.so.0

Find potential deadlocks in PID 181. If the process was created from a statically-linked executable, you can optionally pass the location of the binary. On older kernels without https://lkml.org/lkml/2017/1/13/585, binaries that contain : in the path cannot be attached with uprobes. The workaround is to make a symlink to the binary and pass the symlink name with the --binary option instead:

# deadlock 181 --binary /usr/local/bin/lockinversion

Find potential deadlocks in PID 181 and dump the mutex wait graph to a file:

# deadlock 181 --dump-graph graph.json

Find potential deadlocks in PID 181 and print mutex wait graph statistics:

# deadlock 181 --verbose

Find potential deadlocks in PID 181 with custom mutexes:

# deadlock 181 --lock-symbols
custom_mutex1_lock,custom_mutex2_lock --unlock_symbols
custom_mutex1_unlock,custom_mutex2_unlock

🔮 OUTPUT 🔮

This program doesn't output any fields, bestie. It just keeps running until it finds a potential deadlock or you hit Ctrl-C ⏳ If it does find one, it prints the stack traces and the lock order inversion in the format below, and then exits:

Potential Deadlock Detected!

Cycle in lock order graph: Mutex M0 => Mutex M1 => Mutex M0

Mutex M1 acquired here while holding Mutex M0 in Thread T:
        [stack trace]

Mutex M0 previously acquired by the same Thread T here:
        [stack trace]

Mutex M0 acquired here while holding Mutex M1 in Thread S:
        [stack trace]

Mutex M1 previously acquired by the same Thread S here:
        [stack trace]

Thread T created by Thread R here:
        [stack trace]

Thread S created by Thread Q here:
        [stack trace]

🐢 OVERHEAD 🐢

This traces all mutex lock and unlock events and all thread creation events on the traced process. It can get pretty laggy if the process has lots of threads and mutexes 😩 Only run this on a process where the slowdown is acceptable.

🌈 SOURCE 🌈

This is from bcc 🐝

https://github.com/iovisor/bcc

Also check out the bcc distribution for a companion _examples.txt file with example usage, output, and commentary for this tool 📖

🧁 OS 🧁

Linux 🐧

🫠 STABILITY 🫠

Unstable - in development. 🚧

👑 AUTHOR 👑

Kenny Yu 🫶

---

Bye bestie! 💜 That's the whole tea: this is a USER COMMANDS page, dated 2017-02-01, all about deadlock(8) 🦋✨