UFTRACE-LIVE(1)
✨ NAME ✨
okay so uftrace-live — trace functions in a command during live execution. That's the whole vibe. UFTRACE-LIVE(1) is in the house, and she's about to make your functions spill the tea 💅
🌸 SYNOPSIS 🌸
uftrace [live] [options] COMMAND [command-options]🦋 DESCRIPTION 🦋
This command runs COMMAND and prints its functions with time and thread info. Basically, it's the same as running the uftrace record and uftrace replay commands one after the other, but it does not save a data file. It accepts most options that are accepted by the record or replay commands. No cap.
💖 COMMON OPTIONS 💖
-F FUNC,--filter=FUNC— Set filter to trace selected functions and their children functions. This option can be used more than once. PeepFILTERSfor the full lore.-N FUNC,--notrace=FUNC— Set filter not to trace selected functions and their children functions. This option can be used more than once. SeeFILTERS.-C FUNC,--caller-filter=FUNC— Set filter to trace callers of selected functions only. This option can be used more than once. SeeFILTERS.-T TRG,--trigger=TRG— Set trigger on selected functions. This option can be used more than once. SeeTRIGGERS.-D DEPTH,--depth=DEPTH— Set global trace limit in nesting level. SeeFILTERS.-t TIME,--time-filter=TIME— Do not show functions which run under the time threshold. If some functions explicitly have thetracetrigger applied, those are always traced regardless of execution time. SeeFILTERS.-Z SIZE,--size-filter=SIZE— Do not show functions smaller than SIZE bytes. SeeFILTERS.-L LOCATION,--loc-filter=LOCATION— Set filter to trace selected source locations. This option can be used more than once. Applies toreplaycommand, notrecord. SeeFILTERS.--no-libcall— Do not record library function invocations. Library calls are normally traced by hooking calls to the resolver function of dynamic linker in the PLT. One can disable it with this option.--no-event— Disable event recording which is used by default. Note that explicit event tracing by--eventoption is not affected by this. Implies--no-sched.--no-sched— Disable schedule event recording which is used by default.--match=TYPE— Use pattern match using TYPE. Possible types areregexandglob. Default isregex.--disable— 🚨 DEPRECATED. Use--trace=offinstead.--trace=STATE— Set uftrace tracing STATE. Possible states areonandoff. Default ison. This is only meaningful when used with atrace_ontrigger or with the agent.--with-syms=DIR— Read symbol data from the.symfiles in DIR directory instead of the binary. This can be useful to deal with stripped binaries. The file name of the main binary should be the same when saved and used.
🎀 LIVE OPTIONS 🎀
--list-event— Show available events in the process.--report— Show live-report before replay.--record— Do not discard the recorded data.-p PID,--pid=PID— Switch to client mode. Forward the supported tracing options to a running target with given PID. SeeAGENT.
🔥 RECORD OPTIONS 🔥
-A SPEC,--argument=SPEC— Record function arguments. This option can be used more than once. SeeARGUMENTS.-R SPEC,--retval=SPEC— Record function return values. This option can be used more than once. SeeARGUMENTS.-P FUNC,--patch=FUNC— Patch FUNC dynamically. This is only applicable to binaries built bygccwith-pg -mfentry -mnop-mcountorclangwith-fxray-instrument. This option can be used more than once. SeeDYNAMIC TRACING.-U FUNC,--unpatch=FUNC— Do not apply dynamic patching for FUNC. This option can be used more than once. SeeDYNAMIC TRACING.-E EVENT,--event=EVENT— Enable event tracing. The event should be available on the system.-S SCRIPT_PATH,--script=SCRIPT_PATH— Run a given script to do additional work at the entry and exit of function during target program execution. The type of script is detected by the postfix such as.pyfor python. SeeSCRIPT EXECUTION.-W,--watch=POINT— Add watch point to display POINT if the value is changed. SeeWATCH POINT.-a,--auto-args— Automatically record arguments and return values of known functions. These are usually functions in standard (C language or system) libraries, but if debug info is available it includes functions in the user program.-l,--nest-libcall— Trace function calls between libraries. By default,uftraceonly records library calls from the main executable. Implies--force.-k,--kernel— Trace kernel functions as well as user functions. Only kernel entry/exit functions will be traced by default. Use the--kernel-depthoption to override this.-K DEPTH,--kernel-depth=DEPTH— Set kernel max function depth separately. Implies--kernel.--clock=CLOCK— Set clock source for timestamp recording. CLOCK can be one ofmono,mono_raw, orboot. Default ismono.--signal=TRG— Set trigger on selected signals rather than functions. But there are restrictions so only a few trigger actions are supported for signals. The available actions are:trace_on,trace_off,finish. This option can be used more than once. SeeTRIGGERS.--nop— Do not record and replay any functions. This is a no-op and only meaningful for performance comparisons.--force— Allow running uftrace even if some problems occur. Whenuftrace recordfinds nomcountsymbol (which is generated by compiler) in the executable, it quits with an error message since uftrace can not trace the program. However, it is possible that the user is only interested in functions within a dynamically-linked library, in which case this option can be used to cause uftrace to run the program regardless. Also, the-A/--argumentand-R/--retvaloptions work only for binaries built with-pg, so uftrace will normally exit when it tries to run binaries built without that option. This option ignores the warning and goes on tracing without the argument and/or return value.--time— Print running time of children intime(1)-style.-e,--estimate-return— Record only ENTRY data for each function. This option is useful when the target program deals with stack in some way. Normally uftrace modifies task’s execution stack frame to hook return from the function. However sometimes it makes troubles and it’s hard to handle all the cases properly. This option tells uftrace not to hook return address in order to prevent those problems. The return time is estimated as a half of execution time of two consecutive functions.
⭐ RECORD CONFIG OPTIONS ⭐
--libmcount-path=PATH— Loadlibmcountlibraries from this path. This is mostly for testing purposes.-b SIZE,--buffer=SIZE— Size of internal buffer in which trace data will be saved. Default size is128k.--kernel-buffer=SIZE— Set kernel tracing buffer size. The default value (in the kernel) is1408k.--no-pltbind— Do not bind dynamic symbol address. This option uses theLD_BIND_NOTenvironment variable to trace library function calls which might be missing due to concurrent (first) accesses. It is not meaningful to use this option with the--no-libcalloption.--max-stack=DEPTH— Set the max function stack depth for tracing. Default is1024.--num-thread=NUM— Use NUM threads to record trace data. Default is1/4of online CPUs (but when full kernel tracing is enabled, it will use the full number of CPUs).--libmcount-single— Use single thread version oflibmcountfor faster recording. This is ignored if the target program links with thepthreadlibrary.--rt-prio=PRIO— Boost priority of recording threads to real-time (FIFO) with priority of PRIO. This is particularly useful for high-volume data such as full kernel tracing.--keep-pid— Retain same pid for traced program. For some daemon processes, it is important to have same pid when forked. Running under uftrace normally changes pid as it callsfork()again internally. Note that it might corrupt terminal setting so it’d be better using it with--no-pageroption.--no-randomize-addr— Disable ASLR (Address Space Layout Randomization). It makes the target process fix its address space layout.-g,--agent— Spawn an agent thread in the target. At runtime, the agent receives external commands and can change supported tracing options. SeeAGENT.
💅 REPLAY OPTIONS 💅
-f FIELD,--output-fields=FIELD— Customize field in the output. Possible values are:duration,tid,addr,time,delta,elapsed,taskandmodule. Multiple fields can be set by using comma. Special field ofnonecan be used (solely) to hide all fields. Default isduration,tid. SeeFIELDS.--flat— Print flat format rather than C-like format. This is usually for debugging and testing purpose.--column-view— Show each task in separate column. This makes easy to distinguish functions in different tasks.--column-offset=DEPTH— When--column-viewoption is used, this option specifies the amount of offset between each task. Default is8.--task-newline— Interleave a new line when task is changed. This makes easy to distinguish functions in different tasks.--no-comment— Do not show comments of returned functions.--libname— Show library name along with function name.--srcline— Show source location of each function if available.--format=TYPE— Show format style output. Currently,normalandhtmlstyles are supported.--no-args— Do not show function arguments and return value.
✨ COMMON ANALYSIS OPTIONS ✨
These ones are for looking at the data, bestie. Same flags, whether you recorded five seconds ago or five years ago. 💅
-HFUNC,--hide=FUNC — sets a filter so uftrace does not trace the functions you picked. Super important: it does not touch their subtrees, it only hides those exact functions. You can stack this option as many times as you want. Peep theFILTERSsection for the whole story. 🙈--kernel-full— shows all kernel functions, even the ones called outside of user functions.--kernel-only— kernel functions only, no user functions at all. User code who? 💀--event-full— shows all (user) events, including the ones that happen outside of user functions.--demangle=TYPE — demangles C++ symbol names. The possible values arefull,simpleandno. Default issimple, which ignores function arguments and template parameters (so it's the chill, low-clutter one).-rRANGE,--time-range=RANGE — only shows functions executed inside the time RANGE. The RANGE can be<start>~<stop>(separated by a~) and one of <start> and <stop> can be omitted entirely. <start> and <stop> are a timestamp, or elapsed time if they have a<time_unit>postfix like100us. Real talk though: it is highly recommended to only use elapsed time, because there is literally no way to know the timestamp before you actually run the program. You can see the timestamp with-f time, or the elapsed time with-f elapsed. ⏳
🦋 FILTERS 🦋
Okay so, uftrace can filter out the uninteresting functions. And ngl, filtering is highly recommended — it helps you focus on the interesting functions and it shrinks the data size. Win-win. 🫶
When uftrace gets called, it receives two types of function filter:
- an opt-in filter, via
-F/--filter - an opt-out filter, via
-N/--notrace
These filters can be applied either at record time or at replay time. At record time, you can add and remove them at runtime from the client — see AGENT. To remove a filter, you specify the @clear suffix on the -F / --filter or -N / --notrace option. ✨
The first type of filter is opt-in. By default it doesn't trace anything. But the moment one of the specified functions executes, tracing starts. And when that function returns, tracing stops again.
For example, consider a simple program which calls a(), b() and c() in turn.
$ cat abc.c
void c(void) {
/* do nothing */
}
void b(void) {
c();
}
void a(void) {
b();
}
int main(void) {
a();
return 0;
}
$ gcc -pg -o abc abc.cNormally uftrace will trace all the functions from main() to c().
$ uftrace live ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
[ 1234] | a() {
[ 1234] | b() {
3.880 us [ 1234] | c();
5.475 us [ 1234] | } /* b */
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */In the above example the command name live is explicitly used, but it can totally be omitted because uftrace uses the live command by default. So that command can be reused as uftrace ./abc in short. ✨
But when the -F b filter option is used, it won't trace main() or a() — only b() and c().
$ uftrace -F b ./abc
# DURATION TID FUNCTION
[ 1234] | b() {
3.880 us [ 1234] | c();
5.475 us [ 1234] | } /* b */The second type of filter is opt-out. By default everything is traced, but when one of the specified functions is executed, tracing stops. When the excluded function returns, tracing starts again.
In the above example, you can omit the function b() and all the calls it makes with the -N option.
$ uftrace -N b ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
6.448 us [ 1234] | a();
8.631 us [ 1234] | } /* main */You can set a condition for these filters using the value of an argument. Currently it assumes the argument has an integer value and does the comparison for integers. The @if: suffix should be added after the function name, with the comparison expressions. 🧠
$ uftrace -F main@if:arg1==1 ./abc
# DURATION TID FUNCTION
[ 1234] | main() {
[ 1234] | a() {
[ 1234] | b() {
3.880 us [ 1234] | c();
5.475 us [ 1234] | } /* b */
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */It worked because the value of the first argument of main function was 1. If you change the condition, it won't enable the filter.
$ uftrace -F main@if:arg1==2 ./abc
WARN: cannot open record data: /tmp/uftrace-live-DEFJzZ: No data availableAnd of course, it only works if the function has arguments. 💀
You can hide the function b() only, without affecting the calls it makes in its subtree functions, with the -H option.
$ uftrace -H b ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
[ 1234] | a() {
3.880 us [ 1234] | c();
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */The above -H option is especially useful when hiding std namespace functions in C++ programs, by using the -H ^std:: option setting. 👑
If users only care about specific functions and want to know how they are called, one can use the caller filter. It makes the function a leaf and records the parent functions up to the function.
$ uftrace -C b ./abc
# DURATION TID FUNCTION
[ 1234] | main() {
[ 1234] | a() {
5.475 us [ 1234] | b();
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */In the above example, functions not in the calling path were not shown. Also the function c — which is a child of the function b — is hidden too.
Caller filters can be added and removed from the client at runtime, using the @clear suffix for the -C / --caller-filter option. 🔁
In addition, you can limit the nesting level of functions with the -D option.
$ uftrace -D 3 ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
[ 1234] | a() {
5.475 us [ 1234] | b();
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */In the above example, uftrace only prints functions up to a depth of 3, so leaf function c() was omitted. Note that the -D option works with -F. ⭐
Sometimes it's useful to see long-running functions only. That's a good move, because there are usually a ton of tiny functions that are simply not interesting. The -t / --time-filter option implements the time-based filter that only records functions which run longer than the given threshold. In the above example, the user might want to see functions running more than 5 micro-seconds, like below:
$ uftrace -t 5us ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
[ 1234] | a() {
5.475 us [ 1234] | b();
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */In addition, you can set a filter to record selected source locations with the -L option. 📍
$ uftrace -L s-libmain.c --srcline t-lib
# DURATION TID FUNCTION [SOURCE]
[ 5043] | main() { /* /home/uftrace/tests/s-libmain.c:16 */
6.998 us [ 5043] | foo(); /* /home/uftrace/tests/s-libmain.c:11 */
9.393 us [ 5043] | } /* main */And you can set a filter with the @hide suffix so selected source locations are not recorded.
$ uftrace -L s-libmain.c@hide --srcline t-lib
# DURATION TID FUNCTION [SOURCE]
[ 14688] | lib_a() { /* /home/uftrace/tests/s-lib.c:10 */
[ 14688] | lib_b() { /* /home/uftrace/tests/s-lib.c:15 */
1.505 us [ 14688] | lib_c(); /* /home/uftrace/tests/s-lib.c:20 */
2.816 us [ 14688] | } /* lib_b */
3.181 us [ 14688] | } /* lib_a */The -Z / --size-filter option is for filtering functions that have small sizes. It reads the ELF symbol size and compares it with the given value. The PLT functions may have no symbol size in the ELF format — in that case, the PLT entry size will be used as the size of the function. 🔍
$ uftrace -Z 100 t-arg
# DURATION TID FUNCTION
[162500] | main() {
12.486 us [162500] | foo();
0.505 us [162500] | many();
[162500] | pass() {
0.283 us [162500] | check();
1.449 us [162500] | } /* pass */
18.478 us [162500] | } /* main */You can also set triggers on filtered functions. See the TRIGGERS section below for details. 🌸
When kernel function tracing is enabled, you can also set the filters on kernel functions by marking the symbol with the @kernel modifier. The following example will show all user functions and the (kernel) page fault handler. 🔥
$ sudo uftrace -k -F '.*page_fault@kernel' ./abc
# DURATION TID FUNCTION
[14721] | main() {
7.713 us [14721] | __do_page_fault();
6.600 us [14721] | __do_page_fault();
6.544 us [14721] | __do_page_fault();
[14721] | a() {
[14721] | b() {
[14721] | c() {
0.860 us [14721] | getpid();
2.346 us [14721] | } /* c */
2.956 us [14721] | } /* b */
3.340 us [14721] | } /* a */
79.086 us [14721] | } /* main */✨ TRIGGERS ✨
okay bestie, this is the part where uftrace stops being a passenger and starts being a main character 🎀 The tool can fire off actions on specific function calls (with or without filters) and/or on signals. Here's the BNF so you know exactly what the grammar vibes are:
<trigger> := <symbol> "@" <actions>
<actions> := <action> | <action> "," <actions>
<action> := "depth="<num> | "backtrace" | "trace" | "trace_on" | "trace_off" |
"recover" | "color="<color> | "time="<time_spec> | "read="<read_spec> |
"finish" | "filter" | "notrace" | "hide" | "clear" [ "="<clear_spec> ] |
"if:"<cond_spec>
<time_spec> := <num> [ <time_unit> ]
<time_unit> := "ns" | "nsec" | "us" | "usec" | "ms" | "msec" | "s" | "sec" | "m" | "min"
<read_spec> := "proc/statm" | "page-fault" | "pmu-cycle" | "pmu-cache" | "pmu-branch"
<clear_spec> := <action> | <action> "+" <action>
<cond_spec> := "arg"<num> <cond_op> <num>
<cond_op> := "==" | "!=" | ">" | ">=" | "<" | "<="The depth trigger is literally just changing the filter depth while the function is running. So you can give different functions different filter depths, no big deal, slay 💅 And the backtrace trigger prints a stack backtrace at replay time.
The color trigger changes the color of the function in the replay output ✨ The supported colors are red, green, blue, yellow, magenta, cyan, bold, and gray. (a whole palette, we love to see it 🌈)
Here's how triggers actually do their thing — the global filter maximum depth is 5, but the second b() gets called it flips to 1, so everything below b() just… doesn't show up. poor things, they got ghosted 👻
$ uftrace -D 5 -T 'b@depth=1' ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
[ 1234] | a() {
5.475 us [ 1234] | b();
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */Quick note: the backtrace trigger is only meaningful in the replay command. Like, don't expect it to do cardio at record time 🫶
The trace_on and trace_off actions (and yes, you can drop the _ and just say traceon / traceoff) control whether uftrace records the specified functions or not. on/off, in/out, we love a toggle ✨
The recover trigger exists for some truly cursed corner cases where the process goes and touches the callstack directly 😭 During tracing of the v8 javascript engine, for example, it kept getting segfaults in the garbage collection stage. Why? Because v8 incorporates the return address into compiled code objects(?). The recover trigger restores the original return address at the function entry point and resets to the uftrace return hook address again at function exit. This was used to work around those segfaults by setting the recover trigger on the related function (specifically ExitFrame::Iterate). ngl that's a fight i'm glad i wasn't in 💀
The time trigger changes the time filter setting during execution of the function. So yes, different time filters for different functions. bespoke ✨
The read trigger reads some information at runtime 🔍 The result gets recorded as (builtin) events at the beginning and the end of a given function. As of now, these are the events supported:
"proc/statm"— process memory stat from/procfilesystem"page-fault"— number of page faults usinggetrusage(2)"pmu-cycle"— cpu cycles and instructions using Linux perf-event syscall"pmu-cache"— (cpu) cache-references and misses using Linux perf-event syscall"pmu-branch"— branch instructions and misses using Linux perf-event syscall
The results are printed as events (comments) like this, and honestly? iconic 📊
$ uftrace -T a@read=proc/statm ./abc
# DURATION TID FUNCTION
[ 1234] | main() {
[ 1234] | a() {
[ 1234] | /* read:proc/statm (size=6808KB, rss=776KB, shared=712KB) */
[ 1234] | b() {
[ 1234] | c() {
1.448 us [ 1234] | getpid();
10.270 us [ 1234] | } /* c */
11.250 us [ 1234] | } /* b */
[ 1234] | /* diff:proc/statm (size=+4KB, rss=+0KB, shared=+0KB) */
18.380 us [ 1234] | } /* a */
19.537 us [ 1234] | } /* main */The finish trigger ends recording 🛑 The process can still keep running, which is lowkey the whole point — it's useful for tracing non-terminating processes like a daemon that refuses to leave the party.
The filter and notrace triggers have the exact same effect as -F/--filter and -N/--notrace respectively. And they can carry a condition. and a little attitude if you want 💅
The hide trigger has the same effect as the -H/--hide option — it hides the given functions — but (and this is the key part bestie) it does not affect the functions in their subtree, unlike the notrace trigger. okay subtle but very real ✨
The clear trigger deletes existing actions and it's for agent use case. It can take optional trigger action names connected by "+" (for example, -T myfunc@clear=trace+read). If you give it names, it'll delete only those specified actions; otherwise it'll delete all actions. spring cleaning fr fr 🧹
Heads up: triggers only work for user-level functions for now.
Triggers can be used for signals too! That's done with a signal trigger via the --signal option. The syntax is similar to function trigger but only "trace_on", "trace_off" and "finish" trigger actions are supported. a smaller menu, still delicious 🍓
$ uftrace --signal 'SIGUSR1@finish' ./some-daemon💖 ARGUMENTS 💖
The uftrace tool supports recording function arguments and/or return values using the -A/--argument and -R/--retval options respectively. The syntax is very similar to that of triggers, which — honestly, we love a consistent grammar:
<argument> := <symbol> [ "@" <specs> ]
<specs> := <spec> | <spec> "," <spec>
<spec> := ( <int_spec> | <float_spec> | <ret_spec> )
<int_spec> := "arg" N [ "/" <format> [ <size> ] ] [ "%" ( <reg> | <stack> ) ]
<float_spec> := "fparg" N [ "/" ( <size> | "80" ) ] [ "%" ( <reg> | <stack> ) ]
<ret_spec> := "retval" [ "/" <format> [ <size> ] ]
<format> := "d" | "i" | "u" | "x" | "o" | "s" | "c" | "f" | "S" | "p"
<size> := "8" | "16" | "32" | "64"
<reg> := <arch-specific register name> # "rdi", "xmm0", "r0", ...
<stack> := "stack" [ "+" ] <offset>The -A/--argument option takes a symbol name pattern and its optional specs. The spec starts with argN where N is an index of the arguments 📇 The index starts from 1 and corresponds to the argument passing order of the calling convention on the system. Note that the indexes of arguments are separately counted for integer (or pointer) and floating-point type, and they can interfere depending on the calling convention. So argN is for the integer arguments and fpargN is for the floating-point arguments. two separate little numbered worlds 🦋
You can optionally specify a format and size for the arguments and/or return values ✨
- The
"d"format, or no format field at all, means uftrace treats them aslong inttype for integers anddoublefor floating-point numbers. - The
"i"format makes it a signed integer type, and"u"format is for unsigned type. Both are printed as decimal. - The
"x"format prints it as hexadecimal. - The
"o"format prints it as octal. - The
"s"format is for null-terminated string type. - The
"c"format is for character type. - The
"f"format is for floating-point type and is meaningful only for return value (generally). - Note that
fpargNdoesn't take the format field since it's always floating-point. it said what it said 💅 - The
"S"format is forstd::string, but it only supports libstdc++ library as of yet. - Finally, the
"p"format is for function pointer. Once the target address is recorded, it will be displayed as function name. cute ✨
Please beware when using string type arguments, because it can legitimately crash the program if the (pointer) value is invalid 😭 Actually uftrace tries to keep track of valid ranges of process address space but it might miss some corner cases. so like, careful bestie.
It's also possible to specify a certain register name or stack offset for arguments (but not for return value). Here are the register names you can use:
- x86:
rdi,rsi,rdx,rcx,r8,r9(for integer),xmm[0-7](for floating-point) - arm:
r[0-3](for integer),s[0-15]ord[0-7](for floating-point)
Examples, because seeing is believing 👑:
$ uftrace -A main@arg1/x -R main@retval/i32 ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main(0x1) {
[ 1234] | a() {
[ 1234] | b() {
3.880 us [ 1234] | c();
5.475 us [ 1234] | } /* b */
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } = 0; /* main */
$ uftrace -A puts@arg1/s -R puts@retval ./hello
Hello world
# DURATION TID FUNCTION
1.457 us [21534] | __monstartup();
0.997 us [21534] | __cxa_atexit();
[21534] | main() {
7.226 us [21534] | puts("Hello world") = 12;
8.708 us [21534] | } /* main */Note that these arguments and return value are recorded only if the executable was built with the -pg option 🫶 Executables built with -finstrument-functions will ignore it except for library calls. And recording of arguments and return values only works with user-level functions for now. (theme of this manpage tbh)
If the target program is built with debug info like DWARF, uftrace can identify the number of arguments and their types automatically (when built with libdw) ✨ Also, arguments and return value of some well-known library functions are provided even if the debug info is not available. In these cases you don't need to specify spec of the arguments and return value manually — just a function name (or pattern) is enough. In fact, manual argspec will suppress the automatic argspec. so pick a lane bestie 😌
For example, the above example can be written like below:
$ uftrace -A . -R main -F main ./hello
Hello world
# DURATION TID FUNCTION
[ 18948] | main(1, 0x7ffeeb7590b8) {
7.183 us [ 18948] | puts("Hello world");
9.832 us [ 18948] | } = 0; /* main */Note that the argument pattern (".") matches to any character, so it recorded all (supported) functions 💫 It shows two arguments for "main" and a single string argument for "puts". If you simply want to see all arguments and return values of every functions (if supported), use the -a/--auto-args option. effortless, no cap 🌸
🎀 FIELDS 🎀
okay so uftrace lets you customize the replay output with a bunch of fields 💅 and "field" here just means the info sitting on the left side of that little pipe (|) character. Default vibes are duration and tid, but bestie you can mix and match any fields, in any order you want, like:
$ uftrace -f time,delta,duration,tid,addr ./abc
# TIMESTAMP TIMEDELTA DURATION TID ADDRESS FUNCTION
75059.205379813 1.374 us [27804] 4004d0 | __monstartup();
75059.205384184 4.371 us 0.737 us [27804] 4004f0 | __cxa_atexit();
75059.205386655 2.471 us [27804] 4006b1 | main() {
75059.205386838 0.183 us [27804] 400656 | a() {
75059.205386961 0.123 us [27804] 400669 | b() {
75059.205387078 0.117 us [27804] 40067c | c() {
75059.205387264 0.186 us 0.643 us [27804] 4004b0 | getpid();
75059.205388501 1.237 us 1.423 us [27804] 40067c | } /* c */
75059.205388724 0.223 us 1.763 us [27804] 400669 | } /* b */
75059.205388878 0.154 us 2.040 us [27804] 400656 | } /* a */
75059.205389030 0.152 us 2.375 us [27804] 4006b1 | } /* main */Each field is giving its own little moment, here's what they all mean:
tid— task id (obtained bygettid(2))duration— function execution timetime— timestamp at the executiondelta— difference between two timestamps in a taskelapsed— elapsed time from the first timestampaddr— address of the functiontask— task name (comm)module— library or executable name of the function
The default value is duration,tid. Now here's the cute part ✨ — if a field name starts with "+", it gets appended to the default fields. So -f +time is exactly the same as -f duration,tid,time. And it also accepts a special field name none which disables the field display entirely and shows function output only. icon behavior honestly 👑
🌈 DYNAMIC TRACING 🌈
FULL DYNAMIC TRACING
The uftrace tool supports dynamic function tracing which can be enabled at runtime (load-time, to be precise) on x86_64 and AArch64. no cap. 🔥
So normally, before you can record functions, you gotta build the target program with -pg (or -finstrument-functions), and that incurs some performance impact because literally every function calls mcount(). kinda extra of it.
With dynamic tracing tho, you can trace only the specific functions given by the -P/--patch option, and you can also disable specific functions given by the -U/--unpatch option. And with the capstone disassembly engine? bestie you don't even need to (re)compile the target with that option at all 😌 Now uftrace can analyze the instructions and (if possible) it can copy them to a different place and rewrite it to call the mcount() function so that it can be traced by uftrace. After that, the control is passed to the copied instructions and then returned back to the remaining instructions.
The following example shows an error message when normally running uftrace. Because the binary doesn't call any instrumentation code (i.e. mcount). the audacity 💀
$ gcc -o abc tests/s-abc.c
$ uftrace abc
uftrace: /home/namhyung/project/uftrace/cmd-record.c:1305:check_binary
ERROR: Can't find 'mcount' symbol in the 'abc'.
It seems not to be compiled with -pg or -finstrument-functions flag
which generates traceable code. Please check your binary file.But when the -P a patch option is used, uftrace can dynamically trace a() like it's nothing 🦋
$ uftrace --no-libcall -P a abc
# DURATION TID FUNCTION
0.923 us [19379] | a();In addition, you can enable all functions using '.' (for glob, '*') that matches to any character in a regex pattern with P option. yes it ate 💅
$ uftrace --no-libcall -P . abc
# DURATION TID FUNCTION
[19387] | main() {
[19387] | a() {
[19387] | b() {
0.940 us [19387] | c();
2.030 us [19387] | } /* b */
2.451 us [19387] | } /* a */
3.289 us [19387] | } /* main */Note that -U option has the opposite effect of -P option so users can use both for fine-control. The option that comes later will override the formers. So for example, if you want to trace all functions but a in the above:
$ uftrace --no-libcall -P . -U a abc
# DURATION TID FUNCTION
[19390] | main() {
[19390] | b() {
0.983 us [19390] | c();
2.012 us [19390] | } /* b */
3.373 us [19390] | } /* main */The order of the options is important, bestie. If you change it like-U a -P .then it will trace all the functions since-P .will take precedence and match everything. order matters, periodt. 💯
GCC FENTRY
If the capstone is not available, you need to add some more compiler (gcc) options when building the target program. The gcc 5.1 or more recent versions provide -mfentry and -mnop-mcount options which add instrumentation code (i.e. calling mcount() function) at the very beginning of a function and convert the instruction to a NOP. Then it has almost zero performance overhead when running in a normal condition. chef's kiss 🧚 The uftrace can selectively convert it back to call mcount() using -P option.
$ gcc -pg -mfentry -mnop-mcount -o abc-fentry tests/s-abc.c
$ uftrace -P . --no-libcall abc-fentry
# DURATION TID FUNCTION
[ 18973] | main() {
[ 18973] | a() {
[ 18973] | b() {
0.852 us [ 18973] | c();
2.378 us [ 18973] | } /* b */
2.909 us [ 18973] | } /* a */
3.756 us [ 18973] | } /* main */CLANG XRAY
Clang/LLVM 4.0 provides a dynamic instrumentation technique called X-ray. It's similar to a combination of gcc -mfentry -mnop-mcount and -finstrument-functions. The uftrace also supports dynamic tracing on the executables built with the X-ray. slay ✨
For example, you can build the target program by clang with the below option and equally use -P option for dynamic tracing like below:
$ clang -fxray-instrument -fxray-instruction-threshold=1 -o abc-xray tests/s-abc.c
$ uftrace -P main abc-xray
# DURATION TID FUNCTION
[11093] | main() {
1.659 us [11093] | getpid();
5.963 us [11093] | } /* main */
$ uftrace -P . abc-xray
# DURATION TID FUNCTION
[11098] | main() {
[11098] | a() {
[11098] | b() {
[11098] | c() {
0.753 us [11098] | getpid();
1.430 us [11098] | } /* c */
1.915 us [11098] | } /* b */
2.405 us [11098] | } /* a */
3.005 us [11098] | } /* main */PATCHABLE FUNCTION ENTRY
Recent compilers in both gcc and clang support another useful option -fpatchable-function-entry=N[,M] that generates M NOPs before the function entry and N-M NOPs after the function entry. We can simply use the case when M is 0 so -fpatchable-function-entry=N is enough. The number of NOPs required for dynamic tracing depends on the architecture but x86_64 requires 5 NOPs and AArch64 requires 2 NOPs to dynamically patch a call instruction for uftrace recording. the more you know 💫
For example in x86_64, you can build the target program and trace as follows.
$ gcc -fpatchable-function-entry=5 -o abc-fpatchable tests/s-abc.c
$ uftrace -P . abc-fpatchable
# DURATION TID FUNCTION
[ 6818] | main() {
[ 6818] | a() {
[ 6818] | b() {
[ 6818] | c() {
0.926 us [ 6818] | getpid();
4.158 us [ 6818] | } /* c */
4.590 us [ 6818] | } /* b */
4.957 us [ 6818] | } /* a */
5.593 us [ 6818] | } /* main */This feature can also be used by explicitly adding compiler attribute to some specific functions with __attribute__ ((patchable_function_entry (N,M))). For example, the tests/s-abc.c program can be modified as follows.
static int c(void)
{
return 100000;
}
__attribute__((patchable_function_entry(5)))
static int b(void)
{
return c() + 1;
}
static int a(void)
{
return b() - 1;
}
__attribute__((patchable_function_entry(5)))
int main(void)
{
int ret = 0;
ret += a();
return ret ? 0 : 1;
}The attribute is added to function main and b only and this program can normally be compiled without any additional compiler options, but the compiler detects the attributes and adds 5 NOPs at the entry of main and b. lowkey genius 🎀
$ gcc -o abc tests/s-patchable-abc.c
$ uftrace -P . abc
# DURATION TID FUNCTION
[ 20803] | main() {
0.342 us [ 20803] | b();
1.608 us [ 20803] | } /* main */With this way, uftrace can selectively trace only the functions user wants by explicitly adding the attribute. This approach can collect trace records in a much less intrusive way compared to tracing the entire functions enabled by compiler flags. it's giving surgical precision honestly 💅
-fpatchable-function-entry=N[,M] option and its attribute are supported since gcc-8.1 and clang-10. This dynamic tracing feature can be used in both x86_64 and AArch64 as of now. 🌸
📜 SCRIPT EXECUTION 📜
okay so bestie, uftrace lets you run a script every time a function gets entered and exited 💖 and right now the script types it supports are Python 2.7, Python 3 and Lua 5.1. that's the whole squad as of now.
you get to write four functions, and they have very specific jobs:
uftrace_entryanduftrace_exitrun every single time a function is executed, at its entry and at its exit.uftrace_beginanduftrace_endare divas who only show up once — when the target program begins and when it ends.
here's the vibe:
$ cat scripts/simple.py
def uftrace_begin(ctx):
print("program begins...")
def uftrace_entry(ctx):
func = ctx["name"]
print("entry : " + func + "()")
def uftrace_exit(ctx):
func = ctx["name"]
print("exit : " + func + "()")
def uftrace_end():
print("program is finished")and you can run that script in record time like this, no cap:
$ uftrace -S scripts/simple.py -F main tests/t-abc
program begins...
entry : main()
entry : a()
entry : b()
entry : c()
entry : getpid()
exit : getpid()
exit : c()
exit : b()
exit : a()
exit : main()
program is finished
# DURATION TID FUNCTION
[10929] | main() {
[10929] | a() {
[10929] | b() {
[10929] | c() {
4.293 us [10929] | getpid();
19.017 us [10929] | } /* c */
27.710 us [10929] | } /* b */
37.007 us [10929] | } /* a */
55.260 us [10929] | } /* main */look at that output, it ate ✨
now the ctx variable is a dictionary type and it's carrying the info below 🧺
/* context information passed to uftrace_entry(ctx) and uftrace_exit(ctx) */
script_context = {
int tid;
int depth;
long timestamp;
long duration; # exit only
long address;
string name;
list args; # entry only (if available)
value retval; # exit only (if available)
};
/* context information passed to uftrace_begin(ctx) */
script_context = {
bool record; # True if it runs at record time, otherwise False
string version; # uftrace version info
list cmds; # execution commands
};every single field in script_context can be read from inside your script, periodt. and if you want the whole tea on scripting, go read uftrace-script(1) 💅
🧚 AGENT 🧚
bestie, uftrace can run an agent inside the traced target, and that agent can change the tracing config while things are running. main character energy fr fr.
- the agent is disabled by default
- you turn it on at start-up with the
-g/--agentoption - then you talk to it while the target is executing from a uftrace client instance, using the
-p/--pidoption
the client currently supports these features, and honestly obsessed:
- toggle tracing
- call depth filter
- time threshold filter
- opt-in and opt-out filters
- caller filters
let's look at a lil program that calls a() -> b() -> c() twice, and waits for external input in between 🦋
$ cat abc_abc.c
void c(void) {
/* do nothing */
}
void b(void) {
c();
}
void a(void) {
b();
}
int main(void) {
a();
wait_for_sigusr1();
a();
return 0;
}
$ gcc -pg -o abc_abc abc_abc.c⭐ toggling the tracing
tracing can be toggled anytime during execution, whenever you feel like it:
$ uftrace --agent --trace=off abc_abc &
$ uftrace --pid $(pidof abc_abc) --trace=on
$ kill -s SIGUSR1 $(pidof abc_abc)
# DURATION TID FUNCTION
10.508 us [ 30324] | } /* wait_for_sigusr1 */
[ 30324] | a() {
[ 30324] | b() {
0.138 us [ 30324] | c();
0.757 us [ 30324] | } /* b */
1.217 us [ 30324] | } /* a */
12.346 us [ 30324] | } /* main */🌸 the call depth filter
the call depth filter can be increased or decreased from the client, so flexible:
$ uftrace --agent --depth=2 abc_abc &
$ uftrace --pid $(pidof abc_abc) --depth=4
$ kill -s SIGUSR1 $(pidof abc_abc)
# DURATION TID FUNCTION
[ 32384] | main() {
0.324 us [ 32384] | a();
5.081 s [ 32384] | wait_for_sigusr1();
[ 32384] | a() {
[ 32384] | b() {
0.106 us [ 32384] | c();
0.552 us [ 32384] | } /* b */
0.862 us [ 32384] | } /* a */
5.081 s [ 32384] | } /* main */🎀 the time threshold
the time threshold can also go up or down from the client, iconic behaviour:
$ uftrace --agent --time-filter=0.8us abc_abc &
$ uftrace --pid $(pidof abc_abc) --time-filter=0.5us
$ kill -s SIGUSR1 $(pidof abc_abc)
# DURATION TID FUNCTION
[ 30196] | main() {
0.805 us [ 30196] | a();
6.859 s [ 30196] | wait_for_sigusr1();
[ 30196] | a() {
0.522 us [ 30196] | b();
0.802 us [ 30196] | } /* a */
6.859 s [ 30196] | } /* main */💜 opt-in / opt-out / caller filters
the agent can enforce opt-in and opt-out filters, as well as caller filters 💅
$ uftrace --agent --filter=c abc_abc &
$ uftrace --pid $(pidof abc_abc) --filter=a
$ kill -s SIGUSR1 $(pidof abc_abc)
# DURATION TID FUNCTION
0.398 us [ 3679] | c();
[ 3679] | a() {
[ 3679] | b() {
0.163 us [ 3679] | c();
2.099 us [ 3679] | } /* b */
3.655 us [ 3679] | } /* a */and bestie, filters can be removed using the @clear suffix. that's the cheat code:
$ uftrace --agent --caller-filter=a abc_abc &
$ uftrace --pid $(pidof abc_abc) --caller-filter=a@clear
$ kill -s SIGUSR1 $(pidof abc_abc)
# DURATION TID FUNCTION
[ 4956] | main() {
0.821 us [ 4956] | a();
10.525 s [ 4956] | wait_for_sigusr1();
[ 4956] | a() {
[ 4956] | b() {
0.116 us [ 4956] | c();
0.573 us [ 4956] | } /* b */
0.806 us [ 4956] | } /* a */
10.525 s [ 4956] | } /* main */👀 WATCH POINT 👀
the uftrace watch point is here to display a certain value only if it’s changed. lowkey it's conceptually the same as a debugger's watch point, BUT there's a catch you need to know: it only works at function entry and exit, so it might miss some updates 😭 don't let that surprise you.
as of now, the following watch points are supported:
- “cpu” : cpu number current task is running on
like read triggers, the result is displayed as an event (comment), like so:
$ uftrace -W cpu tests/t-abc
# DURATION TID FUNCTION
[ 19060] | main() {
[ 19060] | /* watch:cpu (cpu=8) */
[ 19060] | a() {
[ 19060] | b() {
[ 19060] | c() {
2.365 us [ 19060] | getpid();
8.002 us [ 19060] | } /* c */
8.690 us [ 19060] | } /* b */
9.350 us [ 19060] | } /* a */
12.479 us [ 19060] | } /* main */🔎 SEE ALSO 🔎
go give these a read, bestie, they're the rest of the lore:
👑 AUTHORS 👑
Namhyung Kim <namhyung@gmail.com>
and that's a wrap on the whole page 🫶 this came from the Uftrace User Manuals, version stamped Sep, 2018, filed under UFTRACE-LIVE(1) — go forth and trace, you absolute icon 💖✨
📦 where this page comes from
- nixos 26.05: uftrace 0.19
- ubuntu stonking: uftrace 0.19-1 · GPL-2.0-only
- 💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on : a language model's rewrite of the original, under its terms