runqlat.bt(8)

📦 version 0.17.0, in debian bookworm

💖 runqlat.bt(8) — the CPU scheduler run queue latency tea 💖

okay bestie, welcome to runqlat.bt, straight out of the System Manager's Manual 🎀 this is a bpftrace/eBPF tool and it is literally the moment you find out how long your processes are sitting around waiting for a turn on the CPU. iconic behavior, no cap.

✨ NAME ✨

runqlat.bt — CPU scheduler run queue latency as a histogram. Uses bpftrace/eBPF. 🦋

🌸 SYNOPSIS 🌸

runqlat.bt

that's it. that's the whole vibe. no flags, no drama, just run it 💅

💜 DESCRIPTION 💜

so what this thing does is trace the time spent waiting in the CPU scheduler for a turn on-CPU. this metric is often called run queue latency, or scheduler latency — same energy, two names, iykyk ✨

the tool shows this latency as a power-of-2 histogram in nanoseconds, which is the whole point honestly, because it lets you study multimodal distributions and catch latency outliers. like, the weird ones. the little freaks in your data. obsessed 😭

this tool uses the sched tracepoints.

🚨 important: since this uses BPF, only the root user can use this tool. sorry bestie, permissions are permissions.

🎀 REQUIREMENTS 🎀

  • CONFIG_BPF
  • bpftrace

both of them. no shortcuts. 🌈

🔥 EXAMPLES 🔥

Trace CPU run queue latency system wide, printing a histogram on Ctrl-C:

# runqlat.bt

run it, watch the histogram, hit Ctrl-C when your heart is full 💖

👑 FIELDS 👑

  • 1st, 2nd — this is a range of latency, in microseconds (shown in "[...)" set notation).
  • 3rd — a column showing the count of scheduler events in this range.
  • 4th — this is an ASCII histogram representing the count column.

so columns one and two are the range, three is the count, four is the pretty picture. slay 📊

💀 OVERHEAD 💀

this traces scheduler functions, which can become very frequent. while eBPF has very low overhead, and this tool uses in-kernel maps for efficiency, the frequency of scheduler events for some workloads may be high enough that the overhead of this tool becomes significant.

⚠️ measure in a lab environment to quantify the overhead before use. fr fr, do not rawdog this on prod and then act surprised 🙈

🌟 SOURCE 🌟

this is from bpftrace.

https://github.com/iovisor/bpftrace

also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool. it's giving ✨ bonus content ✨

this is a bpftrace version of the bcc tool of the same name. the bcc tool may provide more options and customizations — so if you want extra toppings, that's where they live 🍓

https://github.com/iovisor/bcc

🐧 OS 🐧

Linux

🧚 STABILITY 🧚

Unstable — in development.

(it's a work in progress, be gentle with it, it's doing its best 🫶)

⭐ AUTHOR ⭐

Brendan Gregg

🦋 SEE ALSO 🦋

and that's the whole page bestie 💅 this was USER COMMANDS, dated 2018-09-17, from the runqlat.bt(8) era of the manual — go forth, run it as root, and may your histograms be multimodal in the most interesting way possible 🌸💖✨

📦 where this page comes from