zfsslower(8)

🧚 zfsslower(8) 🧚

✨ NAME ✨

okay so the name is zfsslower 💖 and it's basically a tracer for slow ZFS file operations, with per-event details. literally the tool you want when your files are being sus and slow 🫶

💅 SYNOPSIS 💅

zfsslower [-h] [-j] [-p PID] [min_ms]

🌸 DESCRIPTION 🌸

okay bestie, this tool watches the common ZFS file ops: reads, writes, opens, and syncs 📂. It times each one and prints the details for any that go over a threshold. no cap, it's giving detective ✨

WARNING: See the OVERHEAD section. (seriously, read it 😭)

By default the threshold is 10 milliseconds 🎀. If you set it to 0, every single event gets printed. that's giving chaos and also a LOT of output, so be warned 💀

This uses kernel dynamic tracing of the ZPL interface (ZFS POSIX Layer) 🔥. Because it hooks into that interface, it'll need updates if the interface ever changes. not me being dramatic, it's just how it works 🙈

This is meant for the ZFS on Linux project: http://zfsonlinux.org 🌈

And since it runs on BPF, only the root user can use it 👑. sorry, not sorry 🫶

🎀 REQUIREMENTS 🎀

You'll need CONFIG_BPF and bcc 🧚 (no bcc, no tea)

✨ OPTIONS ✨

• -p PID — only trace this one PID 💜
• min_ms — the minimum I/O latency (duration) you want to trace, in milliseconds 🍓. It defaults to 10 ms if you don't set it.

🦋 EXAMPLES 🦋

Trace synchronous file reads and writes that are slower than 10 ms:

# zfsslower

Trace anything slower than 1 ms:

# zfsslower 1

Trace slower than 1 ms, and print just the fields in parsable format (csv) 📊:

# zfsslower -j 1

Trace all file reads and writes (warning: the output is going to be extremely verbose 😭):

# zfsslower 0

Trace slower than 1 ms, for PID 181 only 🎯:

# zfsslower -p 181 1

🔥 FIELDS 🔥

• TIME(s) — when the I/O finished, counted in seconds from the first I/O it saw ⏰
• COMM — the process name 🧑‍💻
• PID — the process ID 🪪
• T — the type of operation. R is read, W is write, O is open, and S is fsync 💅
• OFF_KB — the file offset for the I/O, in Kbytes 📍
• BYTES — the size of the I/O, in bytes 📦
• LAT(ms) — the latency (duration) of the I/O, measured from when VFS issued it to the filesystem until it completed 💀. This includes block device I/O, filesystem CPU cycles, filesystem locks, run queue latency, and so on. It's a more honest measure of what applications actually feel than timing things down at the block device layer, so it's the one to trust 🫶
• FILENAME — a cached kernel file name (it comes from dentry->d_name.name) 📝
• ENDTIME_us — the completion timestamp, in microseconds (-j only) ⏱️
• OFFSET_b — the file offset, in bytes (-j only) 🧮
• LATENCY_us — the latency (duration) of the I/O, in microseconds (-j only) 🌟

⚠️ OVERHEAD ⚠️

okay this section is important, so read it slowly 🙈

This adds low-overhead instrumentation to these ZFS operations, including reads and writes from the file system cache. Those reads and writes can be super frequent (it depends on the workload, eg, 1M/sec 😳). At that point, the tool's own overhead can start to get significant, even if it prints zero "slower" events. So measure and quantify before you use it 📉. If it's still a problem, consider switching to a tool that only prints in-kernel summaries 🧊.

The overhead here should be less than fileslower(8), since this tool only targets ZFS functions and not every file read/write path (which can include socket I/O) 🦋.

🌈 SOURCE 🌈

This comes from bcc 💜:

https://github.com/iovisor/bcc

There's also a companion _examples.txt file in the bcc distribution with example usage, output, and commentary for this tool 📖. go look, it's lowkey fire.

🍓 OS 🍓

Linux 🐧

🧪 STABILITY 🧪

Unstable (it's still in development) 🙈

👑 AUTHOR 👑

Brendan Gregg 🫶

🔗 SEE ALSO 🔗

biosnoop(8), funccount(8), fileslower(8) 💖

---

bestie, that's the whole thing 💅✨ this page is from the USER COMMANDS section, dated 2016-02-11, and the source is bcc 🌸. periodt 🫶💜