iosnoop(8)

📦 version 2017-12, in nixos 26.05, nixpkgs unstable

✨ iosnoop(8) ✨

okay bestie, welcome to iosnoop from the System Manager's Manual section — it's a real one, and it's lowkey iconic. let's get into it 💅

🌸 NAME 🌸

iosnoop — trace block I/O events as they occur. Uses Linux ftrace.

🦋 SYNOPSIS 🦋

iosnoop [-hQst] [-d device] [-i iotype] [-p pid] [-n name] [duration]

💖 DESCRIPTION 💖

iosnoop prints block device I/O events as they happen, and it comes with all the useful little details: PID, device, I/O type, block number, I/O size, and latency. it's giving full picture.

This traces disk I/O at the block device interface, using the block: tracepoints. so it can help you characterize the I/O that's being requested for your storage devices and how they're actually performing. and you can also study I/O completions event-by-event for debugging disk and controller I/O scheduling issues. obsessed.

NOTE: using a duration buffers I/O, which reduces overheads, but it also introduces a limit to the number of I/O that will be captured. see the duration section in OPTIONS, bestie, don't skip it. 🫶

Since this uses ftrace, only the root user can use this tool. root only, no exceptions, that's the vibe. 👑

🎀 REQUIREMENTS 🎀

FTRACE CONFIG, and the tracepoints block:block_rq_insert, block:block_rq_issue, and block:block_rq_complete — which you may already have enabled and available on recent Linux kernels. And awk. yes, awk. we love her. 💜

✨ OPTIONS ✨

  • -d device — Only show I/O issued by this device. (eg, "202,1"). This matches the DEV column in the iosnoop output, and is filtered in-kernel.
  • -i iotype — Only show I/O issued that matches this I/O type. This matches the TYPE column in the iosnoop output, and wildcards ("*") can be used at the beginning or end (only). Eg, "*R*" matches all reads. This is filtered in-kernel.
  • -p PID — Only show I/O issued by this PID. This filters in-kernel. Note that I/O may be issued indirectly; for example, as the result of a memory allocation, causing dirty buffers (maybe from another PID) to be written to storage. 👀

With the -Q option, the identified PID is more accurate, however, LATms now includes queueing time (see the -Q option).

  • -n name — Only show I/O issued by processes with this name. Partial strings and regular expressions are allowed. This is a post-filter, so all I/O is traced and then filtered in user space. As with PID, this includes indirectly issued I/O, and -Q can be used to improve accuracy (see the -Q option).
  • -h — Print usage message. short, sweet, helpful. 🎀
  • -Q — Use block I/O queue insertion as the start tracepoint (block:block_rq_insert), instead of block I/O issue (block:block_rq_issue). This makes the following changes: COMM and PID are more likely to identify the origin process, as are -p PID and -n name; STARTs shows queue insert; and LATms shows I/O time including time spent on the block I/O queue. she's thorough. 💅
  • -s — Include a column for the start time (issue time) of the I/O, in seconds. If the -Q option is used, this is the time the I/O is inserted on the block I/O queue.
  • -t — Include a column for the completion time of the I/O, in seconds.
  • duration — Set the duration of tracing, in seconds. Trace output will be buffered and printed at the end. This also reduces overheads by buffering in-kernel, instead of printing events as they occur.

The ftrace buffer has a fixed size per-CPU (see /sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing, try increasing that size (the bufsize_kb setting in iosnoop). With the default setting (4 Mbytes), I'd expect this to happen around 50k I/O. 😭

🌈 EXAMPLES 🌈

Default output, print I/O activity as it occurs:

# iosnoop

Buffer for 5 seconds (lower overhead) and write to a file:

# iosnoop 5 > outfile

Trace based on block I/O queue insertion, showing queueing time:

# iosnoop -Q

Trace reads only:

# iosnoop -i '*R*'

Trace I/O issued to device 202,1 only:

# iosnoop -d 202,1

Include I/O start and completion timestamps:

# iosnoop -ts

Include I/O queueing and completion timestamps:

# iosnop -Qts

Trace I/O issued when PID 181 was on-CPU only:

# iosnoop -p 181

Trace I/O queued when PID 181 was on-CPU (more accurate), and include queue time:

# iosnoop -Qp 181

🧚 FIELDS 🧚

  • COMM — Process name (command) for the PID that was on-CPU when the I/O was issued, or inserted if -Q is used. See PID. This column is truncated to 12 characters.
  • PID — Process ID which was on-CPU when the I/O was issued, or inserted if -Q is used. This will usually be the process directly requesting I/O, however, it may also include indirect I/O. For example, a memory allocation by this PID which causes dirty memory from another PID to be flushed to disk. sneaky.
  • TYPE — Type of I/O. R=read, W=write, M=metadata, S=sync, A=readahead, F=flush or FUA (force unit access), D=discard, E=secure, N=null (not RWFD).
  • DEV — Storage device ID.
  • BLOCK — Disk block for the operation (location, relative to this device).
  • BYTES — Size of the I/O, in bytes.
  • LATms — Latency (time) for the I/O, in milliseconds.

🔥 OVERHEAD 🔥

By default, iosnoop works without buffering, printing I/O events as they happen (uses trace_pipe), context switching and consuming CPU to do so. This has a limit of about 10,000 IOPS (depending on your platform), at which point iosnoop will be consuming 1 CPU. The duration mode uses buffering, and can handle much higher IOPS rates, however, the buffer has a limit of about 50,000 I/O, after which events will be dropped. You can tune this with bufsize_kb, which is per-CPU. Also note that the "-n" option is currently post-filtered, so all events are traced. 😬

The overhead may be acceptable in many situations. If it isn't, this tool can be reimplemented in C, or using a different tracer (eg, perf_events, SystemTap, ktap.) you do you, bestie.

🍓 SOURCE 🍓

This is from the perf-tools collection.

https://github.com/brendangregg/perf-tools

Also look under the examples directory for a text file containing example usage, output, and commentary for this tool. it's got lore. 📖

⭐ OS ⭐

Linux

💀 STABILITY 💀

Unstable - in development.

👑 AUTHOR 👑

Brendan Gregg

💜 SEE ALSO 💜

iolatency(8), iostat(1), lsblk(8)

and that's the whole thing, no cap — this page comes to you from USER COMMANDS, dated 2014-07-12, and it's literally called iosnoop(8). go trace something slay, but like, as root. 🫶✨

📦 where this page comes from