runqlat.bt(8)

📦
version 0.27.0 · 3 versions
  • 0.27.0 · alpine 3.24, arch rolling, centos-stream 10, centos-stream 9, debian trixie, fedora 43, fedora 44, nixos 26.05, nixpkgs unstable, opensuse-leap 16.0, ubuntu questing, ubuntu resolute, ubuntu stonking (latest)
  • 0.14.0 · ubuntu jammy
  • 0.20.2 · ubuntu noble
, in alpine 3.24, arch rolling, centos-stream 10, centos-stream 9, debian trixie, fedora 43, fedora 44, nixos 26.05, nixpkgs unstable, opensuse-leap 16.0, ubuntu questing, ubuntu resolute, ubuntu stonking

🎀 runqlat.bt(8) 🎀

✨ NAME ✨

runqlat.bt — CPU scheduler run queue latency as a histogram. Uses bpftrace/eBPF. 🧚

🌸 SYNOPSIS 🌸

runqlat.bt

💖 DESCRIPTION 💖

okay so this traces the time a task spends waiting in the CPU scheduler for its turn on-CPU 😭 people call this run queue latency, or scheduler latency, and it is iconic 💅 This tool shows the latency as a power-of-2 histogram in nanoseconds, so you can study multimodal distributions and catch those latency outliers 👀

This tool uses the sched tracepoints. 🦋

Since this uses BPF, only the root user can use this tool. not me being sad about it 💀

🎀 REQUIREMENTS 🎀

you'll need CONFIG_BPF and bpftrace. ✨

🍓 EXAMPLES 🍓

Trace CPU run queue latency system wide, printing a histogram on Ctrl-C:

# runqlat.bt

💜 FIELDS 💜

  • 1st, 2nd — this is a range of latency, in microseconds (shown in [...) set notation) 🌈
  • 3rd — a column showing the count of scheduler events in that range 🔥
  • 4th — an ASCII histogram that visualizes the count column 📊

🌈 OVERHEAD 🌈

⚠️ bestie, listen up: this traces scheduler functions, and those can get very frequent 😭

eBPF has very low overhead and this tool uses in-kernel maps to stay efficient 🧚 but on some workloads the scheduler events can be frequent enough that the overhead becomes significant. so measure in a lab environment to quantify the overhead before you use it in the wild 🫶

🦋 SOURCE 🦋

this is from bpftrace 💖

https://github.com/bpftrace/bpftrace

also check the bpftrace distribution for a companion _examples.txt file with example usage, output, and commentary for this tool. it's giving study guide 📚

this is a bpftrace version of the bcc tool of the same name. the bcc tool may have more options and customizations ✨

https://github.com/iovisor/bcc

🌸 OS 🌸

Linux 🐧

⭐ STABILITY ⭐

Unstable - in development. it's a work in progress, so be gentle 🥺

👑 AUTHOR 👑

Brendan Gregg 🫶

💅 SEE ALSO 💅

runqlen.bt(8), mpstat(1), pidstat(1), uptime(1) 🍓

bestie that's the whole tea ✨ this page is from the USER COMMANDS section, dated 2018-09-17, and it's the runqlat.bt(8) manual 🌈💖 go trace those run queues and slay 💅🔥

📦 where this page comes from