iolatency(8)

💖 iolatency — the block device I/O latency histogram bestie 💖

okay so this is iolatency(8) from the System Manager's Manual, and she is serving you a whole histogram of block device I/O latency. Uses Linux ftrace. Let's get into it 🦋

✨ NAME ✨

iolatency — summarize block device I/O latency as a histogram. Uses Linux ftrace.

🌸 SYNOPSIS 🌸

iolatency [-hQT] [-d device] [-i iotype] [interval [count]]

👑 DESCRIPTION 👑

So basically this shows you the distribution of latency, which means you can spot the modes and the latency outliers and actually study them instead of just vibing in the dark. If you want more deets on block device I/O in general, go use iosnoop(8), she's got you.

This is a proof of concept tool using ftrace, and that means it involves user space processing and the related overheads. Peep the OVERHEAD section below, don't skip it.

NOTE: Because of the way trace buffers are switched per interval, there is the possibility of losing a small number of I/O — usually less than 1%. So the summary shows you the general distribution, but it may be slightly incomplete. If you literally need 100% of I/O studied, use iosnoop(8) and post-process. Also note that I/O may be missed when the trace buffer is full: see the interval section in OPTIONS. 😭

Since this uses ftrace, only the root user can use this tool. periodt.

🧚 REQUIREMENTS 🧚

FTRACE CONFIG, and the tracepoints block:block_rq_issue and block:block_rq_complete, which you may already have enabled and available on recent Linux kernels. And awk.

💅 OPTIONS 💅

• -d device — Only show I/O issued by this device. (eg, "202,1"). This matches the DEV column in the iolatency output, and is filtered in-kernel.
• -i iotype — Only show I/O issued that matches this I/O type. This matches the TYPE column in the iolatency output, and wildcards ("*") can be used at the beginning or end (only). Eg, "*R*" matches all reads. This is filtered in-kernel.
• -h — Print usage message.
• -Q — Include block I/O queueing time. This uses block I/O queue insertion as the start tracepoint (block:block_rq_insert), instead of block I/O issue (block:block_rq_issue).
• -T — Include timestamps with each summary output.
• interval — Interval between summary histograms, in seconds.

During the interval, trace output will be buffered in-kernel, which is then read and processed for the summary. This buffer has a fixed size per-CPU (see /sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing, try increasing that size (the bufsize_kb setting in iolatency). With the default setting (4 Mbytes), I'd expect this to happen around 50k I/O per summary.

• count — Number of summaries to print.

🔥 EXAMPLES 🔥

Default output, print a summary of block I/O latency every 1 second:

# iolatency

Include block I/O queue time:

iolatency -Q

Print 5 x 1 second summaries:

# iolatency 1 5

Trace reads only:

# iolatency -i '*R*'

Trace I/O issued to device 202,1 only:

# iolatency -d 202,1

🎀 FIELDS 🎀

• >=(ms) — Latency was greater than or equal-to this value, in milliseconds.
• <(ms) — Latency was less than this value, in milliseconds.
• I/O — Number of block device I/O in this latency range, during the interval.
• Distribution — ASCII histogram representation of the I/O column.

🫶 OVERHEAD 🫶

Block device I/O issue and completion events are traced and buffered in-kernel, then processed and summarized in user space. There may be measurable overhead with this approach, relative to the block device IOPS.

The overhead may be acceptable in many situations. If it isn't, this tool can be reimplemented in C, or using a different tracer (eg, perf_events, SystemTap, ktap.)

🌈 SOURCE 🌈

This is from the perf-tools collection.

https://github.com/brendangregg/perf-tools

Also look under the examples directory for a text file containing example usage, output, and commentary for this tool.

🐧 OS 🐧

Linux

⭐ STABILITY ⭐

Unstable - in development.

💜 AUTHOR 💜

Brendan Gregg

👀 SEE ALSO 👀

iosnoop(8), iostat(1)

okay that's the whole page bestie, you ate that 💅 this one came out of the USER COMMANDS corner of the manual, dated 2014-07-12, and that's iolatency(8) — go forth and find your outliers 🌸✨🦋