offcputime(8)

✨ offcputime(8) ✨

💖 NAME 💖

okay so the manual name is offcputime(8) 🎀 and it does this: summarize off-CPU time by kernel stack trace. It uses Linux eBPF/bcc 🔥

🧚 SYNOPSIS 🧚

offcputime [-h] [-p PID | -t TID | -u | -k] [-U | -K] [-d] [-f]
[--stack-storage-size STACK_STORAGE_SIZE] [-m MIN_BLOCK_TIME] [-M
MAX_BLOCK_TIME] [--state STATE] [duration]

🌸 DESCRIPTION 🌸

bestie, this program shows you the stack traces and task names of things that were blocked and "off-CPU" 💀, plus the total duration they were not running. That's their off-CPU time ✨. It works by tracing when threads block and when they get back on CPU, measuring how long they were off-CPU, along with the blocked stack trace and the task name. All of this gets summarized in the kernel using an eBPF map, by adding up the off-CPU time for each unique stack trace and task name 💅

The output summary helps you figure out why threads were blocking and how much time they lost being off-CPU 😭 It covers every kind of blocking activity: disk I/O, network I/O, locks, page faults, involuntary context switches, etc. The whole vibe is giving "everything that made my thread wait" 🫶

This is complementary to CPU profiling (e.g., CPU flame graphs), which shows time spent on-CPU 🔥. This one shows time spent off-CPU instead. The output, especially the -f format, can be used to generate an "off-CPU time flame graph" 🦋

See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html 💜

⚠️ This tool only works on Linux 4.6+. It uses the newer BPF_STACK_TRACE table APIs to generate the in-kernel stack traces. For kernels older than 4.6, see the version under tools/old.

Also, lowkey important: this tool only traces off-CPU times that both began and ended while tracing 🙈

---

🎀 REQUIREMENTS 🎀

You'll need CONFIG_BPF and bcc 🌈

👑 OPTIONS 👑

• -h — Prints the usage message 💅
• -p PID — Traces only this process ID (filtered in-kernel) 🍓
• -t TID — Traces only this thread ID (filtered in-kernel) 🍓
• -u — Only traces user threads, no kernel threads 🧚
• -k — Only traces kernel threads, no user threads 🧚
• -U — Shows stacks from user space only, no kernel space stacks ✨
• -K — Shows stacks from kernel space only, no user space stacks ✨
• -d — Inserts a delimiter between kernel and user stacks 🦋
• -f — Prints output in folded stack format 🔥
• --stack-storage-size STACK_STORAGE_SIZE — Changes how many unique stack traces can be stored and displayed 💖
• -m MIN_BLOCK_TIME — The minimum time in microseconds over which traces get stored (default 1) 🫶
• -M MAX_BLOCK_TIME — The maximum time in microseconds under which traces get stored (default U64_MAX) 🫶
• --state — Filters on this thread state bitmask (eg, 2 == TASK_UNINTERRUPTIBLE). Check include/linux/sched.h for the states 💜
• duration — How long to trace, in seconds ⭐

🧚 EXAMPLES 🧚

okay here are the vibes, bestie 💅

Trace all thread blocking events, and summarize (in-kernel) by kernel stack trace and total off-CPU time:

# offcputime

Trace for 5 seconds only:

# offcputime 5

Trace for 5 seconds, and emit output in folded stack format (suitable for flame graphs):

# offcputime -f 5

Trace PID 185 only:

# offcputime -p 185

🌈 OVERHEAD 🌈

This one summarizes unique stack traces in-kernel for efficiency 🧠, which means it can trace a higher rate of events than methods that post-process in user space. The stack trace and time data only gets copied to user space once, when the output is printed. Even so, scheduler events are still a high frequency event, since they can exceed 1 million events per second, so be careful 😭

⚠️ Caution: test before production use, bestie. Seriously 💀

If overhead is still a problem, take a look at the MINBLOCK_US tunable in the code 🔧. If you're chasing down longer blocking events, you can increase it to filter out the shorter blocking events, which lowers overhead even more 🫶

🦋 SOURCE 🦋

This is from bcc 💜

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool 🌸

🍓 OS 🍓

Linux 🐧

🙈 STABILITY 🙈

Unstable, still in development 🚧

💅 AUTHOR 💅

Brendan Gregg 👑

🔗 SEE ALSO 🔗

stackcount(8) ✨

---

Bye bestie, this was the USER COMMANDS manual page for offcputime(8), dated 2016-01-14 🎀🫶💖