runqslower(8)

🦋 NAME 🦋

bestie, this is runqslower (manual section 8) 💖 it's a System Manager's Manual page and the whole thing is about tracing long process scheduling delays ✨

runqslower - Trace long process scheduling delays.

💅 SYNOPSIS 💅

runqslower [-p PID] [-t TID] [-P] [min_us]

✨ DESCRIPTION ✨

okay so this tool measures how long a task sits waiting on a run queue (or whatever equivalent scheduler data structure is in play) for its turn on-CPU, and then it shows you every time that wait goes over the threshold you pass in 🫶 the wait should be small, but a task might have to wait its turn because the CPU is busy. the higher the CPU load, the longer a task will generally need to wait its turn 😭

this tool measures two types of run queue latency, and it's giving science class 🌈

1. The time from a task being enqueued on a run queue to its context switch and execution. This traces ttwu_do_wakeup(), wake_up_new_task() -> finish_task_switch() with either raw tracepoints (if supported) or kprobes, and it instruments the run queue latency after a voluntary context switch.
2. The time from when a task was involuntary context switched and still in the runnable state, to when it next executed. This is instrumented from finish_task_switch() alone.

⚠️ bestie, heads up: the overhead of this tool may become significant for some workloads. check the OVERHEAD section before you go wild with it 🙈

this works by tracing various kernel scheduler functions using dynamic tracing, so it will need updating to match any changes to those functions 🔧

also, since this uses BPF, only the root user can use this tool 👑 no sneaking in as a regular user, periodt.

🎀 REQUIREMENTS 🎀

you'll need CONFIG_BPF and bcc 🧚

✨ OPTIONS ✨

• -h — prints the usage message 💜
• -p PID — only shows this PID 🌸 (it gets filtered in the kernel, so it's efficient)
• -t TID — only shows this TID 🍓 (also filtered in the kernel for efficiency)
• min_us — the minimum scheduling delay, in microseconds, that gets output 🦋
• -P — also shows the previous task's comm and TID 👀

💖 EXAMPLES 💖

show scheduling delays longer than 10ms:

# runqslower

show scheduling delays longer than 1ms for the process with PID 123:

# runqslower -p 123 1000

🌈 FIELDS 🌈

• TIME — when the scheduling event happened ⏰
• COMM — the process name 🎀
• PID — the process ID 🪪
• LAT(us) — the scheduling latency, from the moment the task was ready to run until the moment it got assigned to a CPU to run 🔥

😭 OVERHEAD 😭

okay real talk bestie, this traces scheduler functions, and those can get very frequent 💀 eBPF has super low overhead and this tool uses in-kernel maps for efficiency, but for some workloads the frequency of scheduler events can be high enough that the overhead of this tool becomes significant. so measure in a lab environment to quantify the overhead before you use it for real 🧪

🫶 SOURCE 🫶

this one is from bcc 🦋

https://github.com/iovisor/bcc

also go look in the bcc distribution for a companion _examples.txt file, which has example usage, output, and commentary for this tool 📚

🌸 OS 🌸

Linux 🐧

🙈 STABILITY 🙈

Unstable - in development. it's still cooking, so don't be shocked if things shift 🍳

👑 AUTHOR 👑

Ivan Babrou made the original BCC Python version, and Andrii Nakryiko made the CO-RE version 💜

🔗 SEE ALSO 🔗

runqlen(8), runqlat(8), pidstat(1) ✨

---

bestie that's the whole tea 🫖💅 this page is for runqslower(8) from the USER COMMANDS section, dated 2016-02-07, and it's sourced from bcc. no cap, you're going to be a scheduling latency legend 🧚🌈