wqlat(8)

💖 wqlat(8) 💖

Okay bestie, this is the manual page for wqlat, the System Manager's Manual entry for wqlat(8) 🦋

✨ NAME ✨

wqlat is a tool that summarizes kernel workqueue latency as a histogram 📊 🔥

🎀 SYNOPSIS 🎀

wqlat [-h] [-T] [-N] [-W] [-w WQNAME] [interval [count]]

🌸 DESCRIPTION 🌸

Okay so wqlat traces work's waiting on the workqueue and records the distribution of each work's queuing latency (time) 🫶 The result gets printed as a histogram either when you hit Ctrl-C or after every interval seconds you ask for ⏱️

This tool uses in-kernel eBPF maps for storing timestamps and the histogram, which is the efficient way to do it, so it's giving performance queen 👑

It uses the workqueue:workqueue_queue_work and workqueue:workqueue_execute_start kernel tracepoints. Those are a stable tracing mechanism, so you can trust them 💅 Just know that BPF programs can only attach to tracepoints from Linux 4.7 onward, which means this tool only supports kernel 4.7 or later ✨

🚨 Since this uses BPF, only the root user can use this tool. No cap, sudo or bust 🙈

📚 REQUIREMENTS 📚

You need CONFIG_BPF and bcc. That's the whole list, periodt 💜

✨ OPTIONS ✨

• -h — Prints the usage message 🧚
• -T — Adds timestamps to the output ⏰
• -N — Shows the histogram in nanoseconds instead of the default units 🌈
• -W — Prints a histogram for each workqueue 🎀
• -w WQNAME — Only traces the workqueue you name, so you get laser focus 🔥
• interval — How often to output, in seconds ⏱️
• count — How many outputs you want before it stops 🍓

💅 EXAMPLES 💅

Summarize kernel workqueue latency as a histogram:

# wqlat

Print 1 second summaries, 10 times:

# wqlat 1 10

Print 1 second summaries, using nanoseconds as units for the histogram, and include timestamps on output:

# wqlat -NT 1

Print 1 second summaries, 10 times per workqueue:

# wqlat -W 1 10

Print 1 second summaries for workqueue nvmet_tcp_wq:

# wqlat -w nvmet_tcp_wq 1

📊 FIELDS 📊

• usecs — The microsecond range 🦋
• nsecs — The nanosecond range 🌙
• count — How many works landed in this range 💖
• distribution — An ASCII bar chart that visualizes the distribution (the count column) 📈

🧚 OVERHEAD 🧚

This traces kernel functions and keeps in-kernel timestamps and a histogram, which then get asynchronously copied to user space. It's very efficient, and for most workqueue scheduling rates (< 100k) the overhead should be negligible 🫧 If you have a higher workqueue scheduling rate, please test and quantify the overhead before using it. Don't just vibe with it 🙈

🌈 SOURCE 🌈

This is from bcc 🫶

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file. It has example usage, output, and commentary for this tool, so it's basically your cheat sheet 📖✨

💻 OS 💻

Linux, obviously 🐧

🧪 STABILITY 🧪

Unstable, still in development 🚧 Handle with a little care, bestie 🥺

👑 AUTHOR 👑

Ping Gan 💜

🦋 SEE ALSO 🦋

biolatency

---

Okay that's the whole tea ☕ Manual page from the USER COMMANDS section, dated 2024-01-29, from the bcc source, rendered for you with love 💖🌸 Go trace those workqueues and slay the latency, bestie 💅✨