STAP-PROFILE-ANNOTATE(1)

💅 STAP-PROFILE-ANNOTATE — the General Commands Manual era, section 1 🎀

okay bestie, gather round, because this one is a whole vibe. In this house we call it stap-profile-annotate, and what it does is annotate source files of running programs. that's it. that's the whole thing. ✨

---

🦋 SYNOPSIS 🦋

strace-profile-annotate [ OPTIONS ] -d BINARY ...
strace-profile-annotate [ OPTIONS ] -x PID
strace-profile-annotate [ OPTIONS ] -c CMD

(yes, strace-profile-annotate — that's literally what it says up there, don't come for me, I'm just the messenger 💀)

---

🌸 DESCRIPTION 🌸

so here's the tea. the stap-profile-annotate command profiles the user-space processes you pick, based on the profiling events you pick, over the period of time you pick — and then it produces annotated source code listings from all the hits on their executables & shared libraries. the annotation tells you the number of profiling event hits on the source files' individual lines. it's giving detective work but make it source code. 🔍

picking which user-space processes and shared libraries? you have options, bestie:

• system-wide — in that case use the -d option to enumerate all potentially interesting binaries
• focused on a single pre-existing process — that's -x
• or a newly run process hierarchy — that's -c

SystemTap automatically adds dependent shared libraries for any binary explicitly given by the user. no extra effort required, periodt. 💅

for the profiling event, you can roll with the default kernel profiling timer (a few hundred Hz) or bring your own desired set of systemtap probe points with -e. your call, main character. ✨

the time selected for the profiling session can be:

• the lifetime of the targeted process
• a specified timeout (-T)
• or, honestly, you can just interrupt it by the user at any time. no thoughts, just vibes. 🙈

the stap-profile-annotate program uses debuginfod to fetch debuginfo and source files, and therefore it requires a configured debuginfod-find program. so like, if you're profiling locally built programs that aren't available in central debuginfod servers, consider running a local private server temporarily, possibly federated to upstream debuginfod servers. teamwork makes the dream work 🫶:

% export DEBUGINFOD_URLS=https://UPSTREAM.SERVER/
% debuginfod -p 8002 -d :memory: -F /BUILD/TREE1 /BUILD/TREE2 &

# export DEBUGINFOD_URLS=http://localhost:8002/
# stap-profile-annotate [...]

the resulting annotated source files may be written into subdirectories based on the buildid of the binaries, or printed to standard output (-p). either way, iconic.

⚠️ okay important one, don't skip this: a profiling session can end before the process it's targeting finishes. in this case, sending SIGINT to stap-profile-annotate will yield the script's desired output — if sent after "Stopped stap data collector" is in stdout. and you must still manually kill the targeted process after sending stap-profile-annotate SIGINT. do not forget the second part. I'm serious. 😭

when stap is run it adds user-space libraries into the kernel module by default. however — and this is the plot twist — when profiling a module which is not a user-space shared library, the -d option must be used to add that module's information into the SystemTap kernel module. noted. 📝

and fr fr, stap-profile-annotate relies heavily on the use of debuginfod. debuginfod allows the script to fetch debuginfo and translate virtual memory offsets into source line numbers. it is also used to fetch the actual source files to annotate. for these reasons, ensure your debuginfod client is properly configured for use. this is not a drill. 🚨

---

✨ OPTIONS ✨

• -h — Show help message. (the classic. the icon. the moment. 💖)
• -x, --pid PID — Pre-existing process PID for SystemTap to target. Its symbol information is automatically included. 🎯
• -c, --cmd CMD — Command for SystemTap to run and then target. Its symbol information is automatically included. 🏃♀️
• -d BINARY — Add symbol information for another binary (executable or shared library) and its referenced libraries. This option may be repeated. (spam it as much as you want, it's allowed 🔁)
• -e EVENTS, --events EVENTS — Use the given SystemTap probe points (comma-separated), instead of the default timer.profile to catch profiling hits. Consider perf.hw.branch_misses. 👀
• -T, --timeout TIMEOUT — stap-profile-annotate will exit after TIMEOUT seconds. Note: if -x or -c, any targeted processes will not be killed after this timeout. Targeted processes either need their own timeouts specified or to be killed manually. (yes, you're on your own for cleanup, bestie 💀)
• -p, --print — Print annotated source files to standard output instead of to individual files named like profile-BUILDID/source/PATH/FOO.c. 🖨️
• -w, --context-width WIDTH — This option limits the number of lines of context before and after each hit source line. Without this option, the default is to use unlimited context, i.e., print all lines. (unlimited context is a lot of lines, just so you know 😭)
• -s, --stap PATH — Override the path to the systemtap program. 🛣️
• -v, --verbose — Increase verbosity. May be repeated for more verbosity. (go off, repeat it, live your truth 🗣️)

---

🌈 EXAMPLES 🌈

💫 Command

export DEBUGINFOD_URLS=https://debuginfod.elfutils.org/ # if needed
stap-profile-annotate -w 1 -c '/usr/bin/stress -t 5 --cpu 2'

💫 Output

Starting stap data collector.
stress: info: [88582] dispatching hogs: 4 cpu, 0 io, 0 vm, 0 hdd
stress: info: [88582] successful run completed in 5s
Counted 19798 known userspace hits.
Ignored 12342 kernel hits.
Stopped stap data collector.
Consumed 321 profile records of 19798 hits across 3 buildids.
0001228 (6.20%) hits in profile-10da92de76b289c2e9cfb145ca1edc4e850ec4da/source/usr/src/debug/
    glibc-2.32-10.fc33.x86_64/stdlib/rand.c over 3 lines.
0004241 (21.42%) hits in profile-10da92de76b289c2e9cfb145ca1edc4e850ec4da/source/usr/src/debug/
    glibc-2.32-10.fc33.x86_64/stdlib/random_r.c over 18 lines.
0000094 (0.47%) hits in profile-10da92de76b289c2e9cfb145ca1edc4e850ec4da/source/usr/src/debug/
    glibc-2.32-10.fc33.x86_64/stdlib/../sysdeps/unix/sysv/linux/x86/lowlevellock.h over 1 lines.
0004769 (24.09%) hits in profile-10da92de76b289c2e9cfb145ca1edc4e850ec4da/source/usr/src/debug/
    glibc-2.32-10.fc33.x86_64/stdlib/random.c over 6 lines.
0001401 (7.08%) hits in profile-520e2b0352d6ac72cb3b58df4f44137e29a94250/source/usr/src/debug/
    stress-1.0.4-26.fc33.x86_64/src/stress.c over 1 lines.
00001038 (5.24%) hits in buildid 520e2b0352d6ac72cb3b58df4f44137e29a94250 with unknown source
0003109 (15.70%) hits in profile-07ae52cfc7f4eda1d13383c04564e3236e059993/source/usr/src/debug/
    glibc-2.32-10.fc33.x86_64/math/../sysdeps/ieee754/dbl-64/e_sqrt.c over 3 lines.
0003918 (19.79%) hits in profile-07ae52cfc7f4eda1d13383c04564e3236e059993/source/usr/src/debug/
    glibc-2.32-10.fc33.x86_64/math/w_sqrt_compat.c over 3 lines.

💫 Example Annotated Source File

Found in: profile-07ae52cfc7f4eda1d13383c04564e3236e059993/source/ usr/src/debug/glibc-2.32-10.fc33.x86_64/math/w_sqrt_compat.c

        {
0000730   if (__builtin_expect (isless (x, 0.0), 0) && _LIB_VERSION != _IEEE_)
0000353     return __kernel_standard (x, x, 26); /* sqrt(negative) */

0002799   return __ieee754_sqrt (x);
        }
        libm_alias_double (__sqrt, sqrt)

look at those hit counts. chef's kiss. 👑

---

🧚 SEE ALSO 🧚

• stap(1)
• stapprobes(3stap)
• debuginfod-find(1)

---

and that's the whole page, bestie!! 💜 it lives in the General Commands Manual, section 1, and yeah the name on the tin is STAP-PROFILE-ANNOTATE(1) — go annotate something, you're gonna slay. 🎀✨🦋