offwaketime(8)

💖 offwaketime(8) 💖

System Manager's Manual 🎀

---

✨ NAME ✨

okay so offwaketime is the tool that summarizes blocked time by off-CPU stack + waker stack 🧚. It uses Linux eBPF/bcc, so it's giving low-level and I'm obsessed 😭.

🌸 SYNOPSIS 🌸

offwaketime [-h] [-p PID | -t TID | -u | -k] [-U | -K] [-d] [-f]
[--stack-storage-size STACK_STORAGE_SIZE] [-m MIN_BLOCK_TIME] [-M
MAX_BLOCK_TIME] [--state STATE] [duration]

💅 DESCRIPTION 💅

bestie, this program shows kernel stack traces and task names for threads that got blocked and went "off-CPU" 🙈. It also shows the stack traces and task names of the threads that woke them up, plus the total elapsed time from when they blocked to when they got woken up ✨. It's basically the offcputime and wakeuptime tools having a baby 👶💜. The time measurement will be very similar to off-CPU time, but off-CPU time might include a little extra time spent waiting on a run queue to be scheduled 🫶.

The combined stacks, task names, and total time get summarized in kernel context for efficiency, using an eBPF map. Main character energy 👑.

The output summary helps you figure out why threads were blocking, and lets you measure the time from when they were blocked to when they were woken up 🔥. It covers every type of blocking activity: disk I/O, network I/O, locks, page faults, swapping, sleeping, involuntary context switches, and more 🦋.

This is the bestie of CPU profiling (like CPU flame graphs), which shows time spent on-CPU 💖. This one shows time spent blocked off-CPU instead. The output, especially the -f format, can be used to generate an "off-wake time flame graph" 🌈.

For more, check out http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html ✨

🎀 REQUIREMENTS 🎀

You need CONFIG_BPF and bcc. No cap, these are non-negotiable 💀.

🍓 OPTIONS 🍓

• -h — slay, this prints the usage message 💅
• -f — prints the output in folded stack format 🦋
• -p PID — traces only this process ID (filtered in-kernel) 🎯. It can also be a comma separated list of PIDS 💜.
• -t TID — traces only this thread ID (filtered in-kernel) 🧵. It can also be a comma separated list of TIDS 🫶.
• -u — only traces user threads, no kernel threads 🙋‍♀️
• -k — only traces kernel threads, no user threads 🤖
• -U — shows stacks from user space only, no kernel space stacks 🌙
• -K — shows stacks from kernel space only, no user space stacks ☀️
• -d, --delimited — inserts a delimiter between the kernel and user stacks ✨
• --stack-storage-size STACK_STORAGE_SIZE — changes how many unique stack traces can be stored and displayed 📚
• duration — how long to trace, in seconds ⏰
• -m MIN_BLOCK_TIME — the amount of time in microseconds over which we store traces (default 1) 🌱
• -M MAX_BLOCK_TIME — the amount of time in microseconds under which we store traces (default U64_MAX) 🌷
• --state — filters on this thread state bitmask (eg, 2 == TASK_UNINTERRUPTIBLE). Check include/linux/sched.h for the states 🧠

🧚 EXAMPLES 🧚

Trace all thread blocking events, then summarize (in-kernel) by user and kernel off-CPU stack trace, waker stack traces, task names, and total blocked time. Iconic 🔥:

# offwaketime

Trace for only 5 seconds ⏱️:

# offwaketime 5

Trace for 5 seconds and emit output in folded stack format (perfect for flame graphs), user-mode threads only 🌈:

# offwaketime -fu 5

Trace PID 185 only 🎯:

# offwaketime -p 185

🌊 OVERHEAD 🌊

Okay so this summarizes unique stack trace pairs in-kernel, which is super efficient 💖. That lets it trace a higher rate of events than methods that post-process in user space. The stack trace and time data only gets copied to user space once, when the output is printed ✨. Even with these techniques lowering overhead, scheduler events are still a high frequency event 😭. They can exceed 1 million events per second, so caution is still needed. Test before production use, periodt 🙈.

If the overhead is still a problem, take a look at the min block option 🚨. If you're chasing longer blocking events, you can increase it to filter out shorter blocking events, which lowers overhead even more 💅.

🦋 SOURCE 🦋

This is from bcc 🧚. You can find it at https://github.com/iovisor/bcc 💜.

Also look in the bcc distribution for a companion _examples.txt file with example usage, output, and commentary for this tool 📖.

🌈 OS 🌈

Linux 🐧

🫧 STABILITY 🫧

Unstable, it's still in development 🚧.

👑 AUTHOR 👑

Brendan Gregg 💖

⭐ SEE ALSO ⭐

offcputime(8), wakeuptime(8) 🫶

---

Bye bestie, you're officially a USER COMMANDS legend 💅✨ (2016-01-30, offwaketime(8)) 🎀💖