funcslower(8)

💖 funcslower — bestie's guide to catching slow kernel functions 💖

okay so real talk, this is a page from the System Manager's Manual (section 8, the big-girl admin section 💅), and it's all about funcslower. It traces kernel functions that are being slower than a threshold, measured in microseconds, and it does it with Linux ftrace. Let's get into it 🦋

---

✨ NAME ✨

funcslower — trace kernel functions slower than a threshold (microseconds). Uses Linux ftrace.

---

✨ SYNOPSIS ✨

funcslower [-aChHPt] [-p PID] [-L TID] [-d secs] funcstring latency_us

---

✨ DESCRIPTION ✨

So here's the vibe: this tool uses the Linux ftrace function graph profiler to time kernel functions, and then filters them by a latency threshold. Latency outliers? You can study them this way — confirming they exist, how long they take, and how often they show up. Lowkey iconic for debugging. This tool is a proof of concept using Linux ftrace capabilities on older kernels. (Yes, on older kernels. She's a throwback 👑)

The output format is based on the ftrace function graph trace format, which is described in the kernel source under Documentation/trace/ftrace.txt. Use the -H option to print column headings. And heads up bestie — the output may be shuffled when different CPU buffers are read. So check the CPU column for changes, or include timestamps (-t) and post sort. 🙈

⚠️ WARNING: This uses dynamic tracing of kernel functions, and could cause kernel panics or freezes. Test, and know what you are doing, before use. Like, actually know what you're doing. Not "I skimmed a blog post" know. 😭

Also: since this uses ftrace, only the root user can use this tool. periodt.

---

✨ REQUIREMENTS ✨

• FTRACE function graph, which you may already have enabled and available on recent kernels 🎀
• And awk. Yeah, awk. She's doing the heavy lifting as always.

---

✨ OPTIONS ✨

• -a — All info. Same as -HPt. Give me everything, no gatekeeping 💖
• -C — Function durations measure on-CPU time only (exclude sleep time). So like, only the time she was actually awake and working.
• -d seconds — Set the duration of tracing, in seconds. Trace output will be buffered and printed at the end. This also reduces overheads by buffering in-kernel, instead of printing events as they occur.
  • The ftrace buffer has a fixed size per-CPU (see /sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing, try increasing that size. fr fr, check it.
• -h — Print usage message.
• -H — Print column headers. Labels, bestie. We love labels ✨
• -p PID — Only trace kernel functions when this process ID is on-CPU.
• -L TID — Only trace kernel functions when this thread ID is on-CPU.
• -P — Show process names and process IDs with every line of output.
• -t — Show timestamps on every line of output.
• funcstring — A function name to trace, which may include file glob style wildcards ("*") at the beginning or ending of a string only. Eg, "vfs*" means match "vfs" followed by anything. Since the output is verbose, you probably only want to trace single functions, and not use wildcards. (Wildcards = chaos = a wall of text 💀)
• latency_us — Minimum function duration to trace, in units of microseconds. This is filtered in-kernel. slay.

---

✨ EXAMPLES ✨

Trace calls to vfs_read(), showing events slower than 10 ms:

# funcslower vfs_read 10000

Same as above, but include column headers, event timestamps, and process names:

# funcslower -HPt vfs_read 10000

Trace slow vfs_read()s for PID 198 only:

# funcslower -p 198 vfs_read 10000

---

✨ FIELDS ✨

The output format depends on the kernel version, and headings can be printed using -H. The format is the same as the ftrace function trace format, described in the kernel source under Documentation/trace/ftrace.txt.

Typical fields are:

• TIME — (Shown with -t.) Time of event, in seconds. ⏰
• CPU — The CPU this event occurred on.
• TASK/PID — (Shown with -P.) The process name (which could include dashes), a dash, and the process ID.
• DURATION — Elapsed time during the function call, inclusive of children. This is also inclusive of sleep time, unless -C is used.
• FUNCTION CALLS — Kernel function returns.

---

✨ OVERHEAD ✨

OVERHEADS: Timing and filtering is performed in-kernel context, costing lower overheads than post-processing in user space. If you trace frequent events (eg, pick a common function and a low threshold), you might want to try the "-d secs" option, which buffers events in-kernel instead of printing them live. 🧚

It's a good idea to start with a high threshold (eg, "100000" for 100 ms) then to decrease it. If you start low instead, you may start printing too many events. (Not the glow-up you wanted, bestie 😭)

---

✨ SOURCE ✨

This is from the perf-tools collection:

https://github.com/brendangregg/perf-tools

Also look under the examples directory for a text file containing example usage, output, and commentary for this tool. 🍓

---

✨ OS ✨

Linux 🐧

---

✨ STABILITY ✨

Unstable — in development. She's still growing, okay?? 🌱

---

✨ AUTHOR ✨

Brendan Gregg 👑

---

✨ SEE ALSO ✨

funccount(8), functrace(8), funcgraph(8), kprobe(8)

---

and that's the whole page bestie!! this one came out of the USER COMMANDS collection on 2014-07-30, filed under funcslower(8) — go forth and catch those slow functions 💅✨