funclatency(8)

funclatency(8), the System Manager's Manual, but make it cute 🎀

✨ NAME ✨

okay so funclatency is the tool that times functions and shows you their latency as a histogram 📊 literally the whole vibe 💖

🌸 SYNOPSIS 🌸

funclatency [-h] [-p PID] [-i INTERVAL] [-d DURATION] [-T] [-u] [-m]
[-F] [-r] [-v] pattern

💅 DESCRIPTION 💅

okay bestie, this tool traces function calls and times how long each one takes (that's the latency 🫶). it measures from the moment the function is called until it returns, and it counts both on-CPU time and time spent blocked. no cap, it's all included ✨

this tool uses in-kernel eBPF maps to store the timestamps and the histogram, which is so efficient it's giving main character 🦋

💀 heads up: nested or recursive functions are not supported properly right now. the timestamps get overwritten and you'll get dubious output 😭 try to match single functions, or groups of functions that run at the same stack layer, and that don't end up calling each other.

⚠️ WARNING: this uses dynamic tracing of (what can be many!) functions, and that has caused issues on some kernel versions (risk of panics or freezes 🙈). test it first, and know what you're doing before you use it, periodt.

since this uses BPF, only the root user can use this tool 👑

🎀 REQUIREMENTS 🎀

you'll need CONFIG_BPF and bcc 🧚

✨ OPTIONS ✨

• pattern — the function name or search pattern you want. it supports * wildcards (see EXAMPLES 👀). you can also use -r if you want regular expressions.
• -h — prints the usage message 🌷
• -p PID — only trace this process ID 🎯
• -i INTERVAL — print output every interval seconds ⏰
• -d DURATION — the total duration of the trace, in seconds ⏳
• -l LEVEL — sets the level of nested or recursive functions 🪆
• -T — adds timestamps to the output 🕰️
• -u — outputs the histogram in microseconds 🍓
• -m — outputs the histogram in milliseconds 🫧
• -F — prints a separate histogram for each function that matched 🌈
• -r — uses regular expressions for the search pattern 🧠
• -v — prints the BPF program (for debugging, bestie) 🔍

🦋 EXAMPLES 🦋

time the do_sys_open() kernel function, and print the distribution as a histogram:

# funclatency do_sys_open

time the read() function in libc across all processes on the system:

# funclatency c:read

time vfs_read(), and print the histogram in microseconds:

# funclatency -u vfs_read

time do_nanosleep(), and print the histogram in milliseconds:

# funclatency -m do_nanosleep

time libc open(), print output every 2 seconds, for a duration of 10 seconds:

# funclatency -i 2 -d 10 c:read

time vfs_read(), print output every 5 seconds, with timestamps:

# funclatency -mTi 5 vfs_read

time vfs_read() for process ID 181 only:

# funclatency -p 181 vfs_read

time both vfs_fstat() and vfs_fstatat() calls, using a wildcard:

# funclatency 'vfs_fstat*'

time both vfs_fstat* calls, and print a separate histogram for each one:

# funclatency -F 'vfs_fstat*'

📏 FIELDS 📏

• necs — the nanosecond range 🫧
• usecs — the microsecond range 🍓
• msecs — the millisecond range 🌸
• count — how many calls fell into this range 🔢
• distribution — an ASCII bar chart to visualize the distribution (the count column) 📊

💸 OVERHEAD 💸

this one traces kernel functions and keeps in-kernel timestamps and a histogram, which then get asynchronously copied to user-space. it's very efficient, but kernel functions can fire super often (>1M/sec), and at that point the overhead is expected to be measurable 🥺 measure in a test environment and understand the overheads before you use it. you can also use funccount to measure the rate of kernel functions over a short duration, so you know what to expect beforehand ✨

🫶 SOURCE 🫶

this is from bcc 💜

https://github.com/iovisor/bcc

also look in the bcc distribution for a companion _examples.txt file, which has example usage, output, and commentary for this tool 📚

🌍 OS 🌍

Linux 🐧

🧪 STABILITY 🧪

Unstable, it's still in development 🚧

👩‍💻 AUTHOR 👩‍💻

Brendan Gregg, Sasha Goldshtein 🌟

🔗 SEE ALSO 🔗

funccount(8) 🦋

---

that's a wrap, bestie 💖 this page is from the USER COMMANDS section, dated 2015-08-18, and it's all yours now 🫶💅