funcslower(8)

✨ NAME ✨

Okay bestie, welcome to funcslower(8) from the System Manager's Manual 🎀 It's a tool for tracing slow kernel or user function calls 🐢💨

---

✨ SYNOPSIS ✨

funcslower [-hf] [-p PID] [-U | -K] [-m MIN_MS] [-u MIN_US] [-a ARGUMENTS] [-T] [-t] [-v] function [function ...]

---

✨ DESCRIPTION ✨

So this script watches a kernel or user function's entry and return points 👀 and yells at you with a message whenever the function's latency goes over the threshold you set. You can trace multiple functions at once, and you can even mix kernel functions with user functions from different libraries 🧚💖

WARNING: bestie, see the OVERHEAD section. Seriously, read it 😭

By default, the minimum threshold is 1 millisecond 💅 Also, recursive functions are not supported. Only the inner-most recursive invocation gets traced 🙈

Since this uses BPF, only the root user can run this tool. No cap, that's the rule 🔒👑

---

✨ REQUIREMENTS ✨

You need CONFIG_BPF and bcc. Non-negotiable 🫶

---

✨ OPTIONS ✨

• -p PID — Only trace this PID. Main character energy 👑
• -m MIN_NS — Minimum duration to trace, in milliseconds. The default is 1 ms ⏱️
• -u MIN_US — Minimum duration to trace, in microseconds. For when 1ms is too chunky 🍓
• -a ARGUMENTS — Print the function's arguments, up to 6 ✨
• -T — Print a HH:MM:SS timestamp with each entry 🕰️
• -t — Print a seconds timestamp with each entry, at microsecond resolution 🦋
• -f — Print the output in folded stack format 🌸
• -U — Show stacks from user space only. No kernel space stacks, bestie 🧚
• -K — Show stacks from kernel space only. No user space stacks at all 🌈
• -v — Print the resulting BPF program. It's for debugging, so it's giving detective 🕵️‍♀️
• function — The function to trace. You can pass multiple. If you just give a plain name, it's assumed to be a kernel function. For user functions, give the library name and the function name, like bash:readline or c:malloc 💜

---

✨ EXAMPLES ✨

Trace vfs_write calls slower than 1ms:

# funcslower vfs_write

Trace open() calls in libc slower than 10us:

# funcslower -u 10 c:open

Trace both malloc() and free() slower than 10us, in pid 135 only:

# funcslower -p 135 -u 10 c:malloc c:free

Trace the write syscall and print its first 4 arguments:

# funcslower -a 4 SyS_write

Trace opens from libc and print the user and kernel stack frames:

# funcslower -UK c:open

---

✨ FIELDS ✨

• TIME — The time of the event, either as a human-readable HH:MM:SS, or as a seconds timestamp at microsecond accuracy from the first event seen ⏰
• COMM — The process name 🎀
• PID — The process ID 🆔
• LAT — The latency of the operation, in either microseconds (us) or milliseconds (ms) 🐌
• RVAL — The return value from the function. Super useful for spotting a relationship between slow calls and failed ones 🔍
• FUNC — The function name, followed by its arguments if you asked for them 💅

---

✨ OVERHEAD ✨

Okay this section is the tea ☕ The overhead depends on which function(s) you trace, and it can get severe 💀

Tracing something common like malloc() can slow a C/C++ program down by a factor of 2 or more 😭 On the flip side, tracing a low-frequency event like the SyS_setreuid() function probably won't be a big deal. It's basically negligible for functions called up to 100-1000 times per second 🫧

Before you go wild, use the funclatency and argdist tools for investigation first. They summarize data in-kernel, so they have a way lower overhead than this tool 🧠 To get a vibe check on how often a function is called (and estimate the overhead), use the funccount tool, like this:

# funccount c:open

---

✨ SOURCE ✨

This is from bcc 🐝

https://github.com/iovisor/bcc

Also, look in the bcc distribution for a companion _examples.txt file. It has example usage, output, and commentary for this tool 📚

---

✨ OS ✨

Linux 🐧

---

✨ STABILITY ✨

Unstable, it's still in development 🚧 Be gentle with it 🥺

---

✨ AUTHOR ✨

Sasha Goldshtein 👑

---

✨ SEE ALSO ✨

funccount(8), funclatency(8), argdist(8), trace(8) 🦋

---

That's the whole tea, bestie 🫶 This page is from the USER COMMANDS section, dated 2017-03-30, and it's all about funcslower(8) 💖✨