nfsslower(8)

📦
version 0.36.1 · 2 versions
, in centos-stream 10, centos-stream 9

💅 nfsslower(8) — System Manager's Manual, but make it cute 💅

okay so bestie, this is nfsslower(8): it traces slow NFS file operations and gives you per-event details. It’s giving “NFS detective with receipts” energy 🕵️‍♀️✨

🎀 NAME 🎀

nfsslower — traces slow NFS file operations, with per-event details.

🌸 SYNOPSIS 🌸

nfsslower [-h] [-j] [-p PID] [min_ms]

✨ DESCRIPTION ✨

This tool traces the common NFSv3 & NFSv4 file operations: reads, writes, opens, and getattrs. It measures how long those operations take, then prints the deets for every single one that exceeded your threshold. 💅

⚠️ WARNING: See the OVERHEAD section. Do not skip it, bestie.

By default, the minimum millisecond threshold is 10. If you use a threshold of 0, every event gets printed — warning: verbose. Like, very verbose. 😭

Since this works by tracing the nfs_file_operations interface functions, it will need updating to match any changes to those functions.

This tool uses kprobes to instrument the kernel for entry and exit information. In the future, a preferred way would be to use tracepoints. RN there aren’t any tracepoints available for nfs_read_file, nfs_write_file and nfs_open_file, while nfs_getattr does have entry and exit tracepoints — but we chose to use kprobes for consistency.

Since this uses BPF, only the root user can use this tool. No root, no vibes 🔒

🧚 REQUIREMENTS 🧚

CONFIG_BPF and bcc.

🎀 OPTIONS 🎀

  • -h — it appears in the synopsis; the original OPTIONS list doesn’t give it its own description, so I’m not out here inventing behavior 🫶
  • -p PID — trace this PID only. 🎯
  • -j — trace output in CSV format. CSV girlie era ✨
  • min_ms — minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms. ⏱️

💖 EXAMPLES 💖

  1. Trace synchronous file reads and writes slower than 10 ms:
# nfsslower
  1. Trace slower than 1 ms:
# nfsslower 1
  1. Trace slower than 1 ms, and output just the fields in parsable format (CSV):
# nfsslower -j 1
  1. Trace all file reads and writes — warning: the output will be verbose:
# nfsslower 0
  1. Trace slower than 1 ms, for PID 181 only:
# nfsslower -p 181 1

🌈 FIELDS 🌈

TIME(s)

Time of I/O completion since the first I/O seen, in seconds. ⏰

COMM

Process name.

PID

Process ID.

T

Type of operation. R == read, W == write, O == open, G == getattr.

OFF_KB

File offset for the I/O, in Kbytes.

BYTES

Size of I/O, in bytes.

LAT(ms)

Latency (duration) of I/O, measured from when it was issued by VFS to the filesystem, to when it completed. This time is inclusive of RPC latency, network latency, cache lookup, remote fileserver processing latency, etc. It’s a more accurate measure of the latency suffered by applications performing NFS read/write calls to a fileserver. 🫶

FILENAME

A cached kernel file name (comes from dentry->d_name.name).

ENDTIME_us

Completion timestamp, microseconds (-j only).

OFFSET_b

File offset, bytes (-j only).

LATENCY_us

Latency (duration) of the I/O, in microseconds (-j only).

💀 OVERHEAD 💀

This adds low-overhead instrumentation to NFS operations, including reads and writes from the file system cache. Those reads, writes, and especially getattrs can be very frequent depending on the workload — like, 1M/sec frequent — and at that point the overhead of this tool can begin to become significant, even if it prints no "slower" events. Measure and quantify before use. If this keeps being a problem, consider switching to a tool that prints in-kernel summaries only.

This tool has been tested with NFSv3 & NVSv4 — yes, that’s how it’s written — but it might work with NFSv{1,2}, since it is tracing the generic functions from nfs_file_operations.

Note that the overhead of this tool should be less than fileslower(8), as this tool targets NFS functions only, and not all file read/write paths.

🫶 SOURCE 🫶

This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion nfsslower_examples.txt file containing example usage, output, and commentary for this tool.

🐧 OS 🐧

Linux

💅 STABILITY 💅

Unstable - in development.

👑 AUTHOR 👑

Samuel Nair

🔥 SEE ALSO 🔥

biosnoop(8), funccount(8), fileslower(8)

okay that’s the whole page, bestie. It’s filed under USER COMMANDS, dated 2017-09-01, and it’s nfsslower(8). Slay responsibly 💖✨

📦 where this page comes from