runqlat(8)

🧚 runqlat(8) 🧚

Okay bestie, this is the System Manager's Manual page for runqlat(8), and it's giving main character energy ✨

💅 NAME 💅

runqlat - Run queue (scheduler) latency as a histogram. 📊

🎀 SYNOPSIS 🎀

runqlat [-h] [-T] [-m] [-P] [--pidnss] [-L] [-p PID] [interval] [count]

🦋 DESCRIPTION 🦋

Okay so this tool measures how long a task sits waiting on a run queue (or the equivalent scheduler data structure) for its turn on-CPU, and then shows you that wait as a histogram 📊. This time should be small, but a task might have to wait its turn because the CPU is busy. The higher the CPU load, the longer a task will generally need to wait, so it's lowkey a vibe check for your scheduler 😭

This tool measures two types of run queue latency, and I'm obsessed with both:

1. The time from a task being enqueued on a run queue to its context switch and execution. This traces ttwu_do_wakeup(), wake_up_new_task() -> finish_task_switch() with either raw tracepoints (if supported) or kprobes, and it instruments the run queue latency after a voluntary context switch. 🌸
2. The time from when a task was involuntarily context switched while still in the runnable state, to when it next executed. This is instrumented from finish_task_switch() alone. 🫶

This tool uses in-kernel eBPF maps for storing timestamps and the histogram, which is so efficient 🔥. Despite that, the overhead can get significant for some workloads, so see the OVERHEAD section (seriously, read it 🙈).

It works by tracing various kernel scheduler functions using dynamic tracing, which means it will need updating to match any changes to those functions. Not gonna lie, that's a little bit of a commitment 💀

Since this uses BPF, only the root user can use this tool. No exceptions, bestie 🔒

🌈 REQUIREMENTS 🌈

You'll need CONFIG_BPF and bcc. ✨

💖 OPTIONS 💖

• -h — Prints the usage message. Your lifeline when you forget everything 🙈
• -T — Includes timestamps on the output. ⏰
• -m — Outputs the histogram in milliseconds instead of microseconds. 🍓
• -P — Prints a histogram for each PID. 🧚
• --pidnss — Prints a histogram for each PID namespace (short for PID namespaces). Perfect for container analysis 📦
• -L — Prints a histogram for each thread ID. 🧵
• -p PID — Only shows this PID. It's filtered in the kernel for efficiency, so it's giving smart 🧠
• interval — The output interval, in seconds. ⏱️
• count — The number of outputs you want. 🔢

✨ EXAMPLES ✨

Summarize run queue latency as a histogram, no cap, that's the whole move:

# runqlat

Print 1 second summaries, 10 times:

# runqlat 1 10

Print 1 second summaries, using milliseconds as units for the histogram, and include timestamps on the output:

# runqlat -mT 1

Trace PID 186 only, with 1 second summaries:

# runqlat -P 185 1

📊 FIELDS 📊

• usecs — The microsecond range 🌷
• msecs — The millisecond range 🌷
• count — How many times a task event fell into this range 💜
• distribution — An ASCII bar chart to visualize the distribution (count column) 📈

🙈 OVERHEAD 🙈

Be careful here, babe 😭 This traces scheduler functions, which can get super frequent. eBPF has very low overhead, and this tool uses in-kernel maps for efficiency, but for some workloads the frequency of scheduler events can be high enough that the overhead becomes significant. Measure in a lab environment to quantify the overhead before you use it in the wild 🧪

🌸 SOURCE 🌸

This is from bcc. 🐝

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool. It's like a bonus episode 📚

🌈 OS 🌈

Linux 🐧

🧪 STABILITY 🧪

Unstable - in development. Still cooking, so handle with care 🍳

👑 AUTHOR 👑

Brendan Gregg, the legend himself 🫡

🔗 SEE ALSO 🔗

runqlen(8), runqslower(8), pidstat(1) 💅

---

Okay that's a wrap, bestie! ✨ This page is from the USER COMMANDS section, dated 2016-02-07, and it's all about runqlat(8) 💖 Go forth and measure those run queues 🫶🦋