fileslower(8)

✨ NAME ✨

okay so the manual page for fileslower(8) (System Manager's Manual, bestie 💅) is basically here to tell you: fileslower is a tool that traces slow synchronous file reads and writes. 🦋

💜 SYNOPSIS 💜

fileslower [-h] [-p PID] [-a] [min_ms]

🌸 DESCRIPTION 🌸

okay this one is so good, no cap. It uses kernel dynamic tracing on synchronous reads and writes at the VFS interface, so it can catch slow file reads and writes on any file system. ✨

This version traces __vfs_read() and __vfs_write(), and it only shows synchronous I/O (the path through new_sync_read() and new_sync_write()) plus I/O that has filenames. So you're getting a view of just two file system request types: file reads and writes. There are typically a bunch of others too: asynchronous I/O, directory operations, file handle operations, file open()s, fflush(), etc. 🙈

💀 WARNING: See the OVERHEAD section. Seriously, read it before you do anything wild.

By default, the minimum millisecond threshold is 10. 🎀

Since this works by tracing various kernel __vfs_*() functions using dynamic tracing, it'll need updating to match any changes to those functions. A future version should switch to using FS tracepoints instead. 🧚

Since this uses BPF, only the root user can use this tool. 👑

📋 REQUIREMENTS 📋

CONFIG_BPF and bcc. that's it, that's the list. 💖

⭐ OPTIONS ⭐

• -p PID — trace this PID only. main character energy 👑
• -a — include non-regular file types in the output (sockets, FIFOs, etc). 🦋
• min_ms — the minimum I/O latency (duration) to trace, in milliseconds. It defaults to 10 ms. 🍓

🎀 EXAMPLES 🎀

Trace synchronous file reads and writes slower than 10 ms:

# fileslower

Trace slower than 1 ms:

# fileslower 1

Trace slower than 1 ms, for PID 181 only:

# fileslower -p 181 1

🌈 FIELDS 🌈

Here's what the columns mean, bestie:

• TIME(s) — the time of I/O completion, measured from the first I/O it saw, in seconds. ⏰
• COMM — the process name. 🧚
• PID — the process ID. 🆔
• D — the direction of the I/O. R == read, W == write. 💜
• BYTES — the size of the I/O, in bytes. 🫶
• LAT(ms) — the latency (duration) of the I/O, measured from when the application issued it to VFS until it completed. This time includes block device I/O, file system CPU cycles, file system locks, run queue latency, etc. It's a more accurate measure of the latency an application actually suffers when doing file system I/O, compared to measuring down at the block device interface. it's giving accuracy 🔥
• FILENAME — a cached kernel file name (comes from dentry->d_name.name). 📁

⚠️ OVERHEAD ⚠️

😭 Depending on how often the application reads and writes, the overhead can get severe, in the worst case slowing applications by 2x. In the best case, the overhead is negligible. Hopefully for real world workloads the overhead is often at the lower end of the spectrum, so test before use.

The reason for high overhead is that this traces VFS reads and writes, which includes FS cache reads and writes, and it can exceed one million events per second if the application is I/O heavy. The instrumentation is extremely lightweight and uses in-kernel eBPF maps for efficient timing and filtering, but multiply that cost by one million events per second and it becomes a million times worse. 💀

You can get an idea of the possible cost by just counting the instrumented events using the bcc funccount tool, eg:

# ./funccount.py -i 1 -r '^__vfs_(read|write)$'

This also costs overhead, but it's somewhat less than fileslower. 🌸

If the overhead is prohibitive for your workload, I'd recommend moving down-stack a little from VFS into the file system functions (ext4, xfs, etc). Look for updates to bcc for specific file system tools that do this. The advantage of a per-file-system approach is that we can trace post-cache, which greatly reduces events and overhead. The disadvantage is needing custom tracing approaches for each different file system, whereas VFS is generic. 🧠

🔗 SOURCE 🔗

This is from bcc:

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool. 📚

💻 OS 💻

Linux 🐧

🧪 STABILITY 🧪

Unstable - in development. 🙈

👩‍💻 AUTHOR 👩‍💻

Brendan Gregg 🫶

🔎 SEE ALSO 🔎

biosnoop(8), funccount(8) ✨

---

bye bestie, you're officially a fileslower expert now 🦋💖 this page came from the USER COMMANDS section, last updated 2016-02-07, and it's all from fileslower(8) 🌈💅