nfsslower(8)

✨ NFSSLOWER(8) ✨

Heyy bestie, welcome to the nfsslower(8) page from the System Manager's Manual 💖 It's the one for tracing slow NFS file ops 🦋

---

🔍 NAME 🔍

nfsslower - Trace slow NFS file operations, with per-event details 🧚

📝 SYNOPSIS 📝

nfsslower [-h] [-j] [-p PID] [min_ms]

💅 DESCRIPTION 💅

Okay so this tool traces the common NFSv3 & NFSv4 file operations: reads, writes, opens, and getattrs ✨ It measures how long each one takes and prints details for every single one that went over a threshold. Main character energy 👑

WARNING: Please go read the OVERHEAD section before you run this 🙈

By default the minimum millisecond threshold is 10, so anything 10 ms or slower gets printed. If you set the threshold to 0, every event gets printed. Warning: that is extremely verbose 😭

Real talk, this works by tracing the nfs_file_operations interface functions, so it'll need updating to match any changes to those functions 🔧

It uses kprobes to instrument the kernel for entry and exit info. In the future, the preferred way would be tracepoints 🌈 Right now though, there aren't any tracepoints for nfs_read_file, nfs_write_file and nfs_open_file. nfs_getattr does have entry and exit tracepoints, but we chose kprobes for consistency 🤷‍♀️

Since this uses BPF, only the root user can use this tool 🔐

🧰 REQUIREMENTS 🧰

CONFIG_BPF and bcc.

🎀 OPTIONS 🎀

• -p PID — Only trace this PID 🎯
• -j — Output the trace in CSV format, so it's easy to parse 📊
• min_ms — The minimum I/O latency (duration) to trace, in milliseconds. The default is 10 ms ⏱️

🌸 EXAMPLES 🌸

Trace synchronous file reads and writes slower than 10 ms:

# nfsslower

Trace slower than 1 ms:

# nfsslower 1

Trace slower than 1 ms, and output just the fields in parsable format (CSV):

# nfsslower -j 1

Trace all file reads and writes (warning: the output will be verbose):

# nfsslower 0

Trace slower than 1 ms, for PID 181 only:

# nfsslower -p 181 1

🍓 FIELDS 🍓

• TIME(s) — When the I/O completed, in seconds, counted from the first I/O seen ⏰
• COMM — The process name 🧑‍💻
• PID — The process ID 🆔
• T — The type of operation. R == read, W == write, O == open, G == getattr 🔤
• OFF_KB — The file offset for the I/O, in Kbytes 📍
• BYTES — The size of the I/O, in bytes 📦
• LAT(ms) — The latency (duration) of the I/O. It's measured from when VFS issued it to the filesystem, until it completed. This time includes RPC latency, network latency, cache lookup, remote fileserver processing latency, and so on. It's a more accurate measure of the latency apps actually suffer when doing NFS read/write calls to a fileserver 💯
• FILENAME — A cached kernel file name (it comes from dentry->d_name.name) 📁
• ENDTIME_us — The completion timestamp, in microseconds (-j only) 🕒
• OFFSET_b — The file offset, in bytes (-j only) 🧮
• LATENCY_us — The latency (duration) of the I/O, in microseconds (-j only) ⚡

🫧 OVERHEAD 🫧

Heads up, this adds low-overhead instrumentation to NFS operations, including reads and writes from the file system cache 🙈 Reads, writes, and especially getattrs can be super frequent (depending on the workload, e.g. 1M/sec). At that point, the overhead of this tool can start to get significant, even if it prints no "slower" events at all 😬 Measure and quantify before you use it. If this keeps being a problem, consider switching to a tool that only prints in-kernel summaries. This tool has been tested with NFSv3 & NFSv4, but it might work with NFSv{1,2}, since it traces the generic functions from nfs_file_operations.

Note: this tool's overhead should be lower than fileslower(8), because it only targets NFS functions and not all the file read/write paths 💜

🌍 SOURCE 🌍

This is from bcc 🐝

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion nfsslower_examples.txt file. It has example usage, output, and commentary for this tool 📖

💻 OS 💻

Linux 🐧

🚧 STABILITY 🚧

Unstable, it's still in development 🛠️

👩‍💻 AUTHOR 👩‍💻

Samuel Nair 🫶

🔗 SEE ALSO 🔗

biosnoop(8), funccount(8), fileslower(8) ✨

---

Bestie, that's the whole page 🎀 This one is from the USER COMMANDS section, last dated 2017-09-01, so go show your nfsslower(8) skills 💖🦋🌈