LATRACE(1)

📦 version 0.5.11, in ubuntu jammy

💖 LATRACE(1) — the latrace manual, but make it make sense 💖

ok so bestie, this is latrace, and it's a whole vibe. it's the LD_AUDIT 2.4+ libc frontend. let's get into it 🦋

✨ NAME ✨

latrace — LD_AUDIT 2.4+ libc frontend 🫶

✨ SYNOPSIS ✨

latrace [-snltfvhiBdISbcCyYLpoaNADVTFERq] command [arg ... ]

✨ DESCRIPTION ✨

ok so the latrace tracer can run a command and show you its dynamic library calls, and it does this using the LD_AUDIT libc feature — which has been a thing since libc version 2.4 onward, so we're good. it can ALSO measure and display all kinds of statistics of dynamic calls. more deets in the section called “DISCUSSION” 👑

if you hand it the header file with functions' declarations, latrace will straight up display the function's arguments. the header file syntax is similar to the C language, with several exceptions — see the section called “HEADERS” for all that.

by default, latrace does all its work fully inside of the traced program. BUT ✋ there's another mode called "pipe mode" that moves the main work over to the tracer — that's the latrace binary itself. see the section called “PIPE mode” for more.

and the latrace use is similar to strace(1) and ltrace(1). iykyk 💅

✨ OPTIONS ✨

  • -l, --libs lib1[,lib2,...] — audit from and to lib1, lib2 ...
  • -t, --libs-to lib1[,lib2,...] — audit to lib1, lib2 ...
  • -f, --libs-from lib1[,lib2,...] — audit from lib1, lib2 ...
  • -s, --sym sym1[,sym2,...] — audit symbols sym1, sym2 ...
  • -n, --sym-omit sym1[,sym2,...] — omit symbols sym1, sym2 ...
  • -L, --lib-subst s1[,s2,...] — objsearch LD_AUDIT interface (see the section called “OBJSEARCH”)
  • -c, --counts — display statistics counts of symbols — implies pipe mode (see the section called “PIPE mode”) an no symbol output is displayed
  • -C, --sort-counts stat — implies -c, plus sort the statistics by stat with following values: time,per,call,ucall,lib,sym (default is call)
  • -p, --pipe — use pipe to latrace process to send audit data (see the section called “PIPE mode”)
  • -N, --conf — config file (see the section called “CONFIG”)
  • -A, --enable-args — enable arguments output (definitions from /etc/latrace.conf)
  • -D, --detail-args — display struct arguments in more detail
  • -a, --args file — specify arguments definition file, implies -A (without the default definition file of course)
  • -y, --framesize number — framesize for storing the stack before pltexit (default 100)
  • -Y, --no-framesize-check — disable framesize check
  • -F, --no-follow-fork — dont follow fork calls (childs). this is just supressing the latrace output from new childs. the nature of the LD_AUDIT feature prevents to disable it completely. ⚠️
  • -E, --no-follow-exec — dont follow exec calls
  • -S, --timestamp — display timestamp for each symbol
  • -b, --flow-below sym1[,sym2,...] — display flow for sym1, sym2 ...
  • -I, --no-indent-sym — do no indent symbols based on the their stack depth
  • -i, --indent-sym indent_size — indent size specification in indent_size
  • -B, --braces — allways display { } for the around the symbol body
  • -d, --demangle — C++ demangle symbols on the output
  • -T, --hide-tid — dont display thread id
  • -o, --output file — store output to file
  • -R, --ctl-config — controled config feature
  • -q, --disable — run with disabled auditing
  • -v, --verbose — verbose output
  • -V, --version — display version
  • -h, --help — display help

✨ EXAMPLES ✨

  • The simplest way to run latrace is like this:
latrace cat
  • To see the argument values specified by default config file run:
latrace -A cat
  • Same as above but using the pipe mode to get all the end symbols printed:
latrace -Ap cat
  • To see the argument values specified by specified config file run:
latrace -a latrace.conf cat
  • To get output only for specified symbol (eg. read and write) run:
latrace -A -s read,write cat
  • To get flow under the specified symbol (eg. sysconf) run:
latrace -b sysconf kill
  • To get output only for specified library (eg. libproc) run:
latrace -Al libproc w
  • To get symbol statistics run:
latrace -c ls
  • To get symbol statistics sorted by time run:
latrace -C time ls
  • To get output stored to the text file run:
latrace -o output.latrace ls
  • To change the libkrava1.so dependency to the libkrava2.so run one of these:
latrace -L krava1%krava2 ex
latrace -L krava1~libkrava2.so ex
latrace -L libkrava1.so=libkrava2.so ex

🌸 that's part 1 bestie, more parts incoming, don't go anywhere 🌸

💬 DISCUSSION 💬

🔍 NAMES CHECK 🔍

ok so for these options — -l, -t, -f, -s, -n, -b — you can drop a * in there and it flips over to substring search. no *? then it's a straight-up exact match on the name, no vibes, no wiggle room. 💅

Examples, bestie, look:

-s "*krava" checks for symbols with "krava" substring.

-s "krava"  checks only for "krava" symbol

🎧 LD_AUDIT 🎧

so this is just a lil brief-and-vague rundown of the LD_AUDIT situation, ok? for the real tea go read rtld-audit(7), that man page was done by Petr Baudis, or go study the glibc/latrace source code yourself. anyway, super brief explanation incoming. 👑

The libc dynamic linker audit feature lets you trace/audit a program's symbols and libraries. You switch it on with the LD_AUDIT environment variable. That variable has to hold the path to the audit shared library. This audit library has to follow a specific interface — the interface functions get called by the dynamic linker at the appropriate moments.

The audit library needs to export these symbols (and heads up, the "la_PLTENTER" and "la_PLTEXIT" names are architecture dependent, iykyk):

"la_activity"
"la_objsearch"
"la_objopen"
"la_preinit"
"la_symbind32"
"la_symbind64"
"la_PLTENTER"
"la_PLTEXIT"
"la_objclose"

And for the latrace package specifically, that audit shared library is called libltaudit.so. 🦋

🔀 OBJSEARCH 🔀

The objsearch LD_AUDIT interface gives you the means to change the traced program's shared object names and locations. The -L option argument should look like this:

-L s1[,s2,...] where sN is src [=%~] dst

Where src is the source pattern/name and dst is the destination name/pattern. Now the three flavours of matching, periodt:

  • = — compares src with the library name. If matched, it replaces the library name with dst.
library name         - /lib/krava1.so
src                  - /lib/krava1.so
dst                  - /lib/krava2.so

final library name   - /lib/krava2.so
  • % — looks for the src in the library name. If found, it replaces the src with the dst part.
library name         - /lib/krava1.so
src                  - krava1
dst                  - krava2

final library name   - /lib/krava2.so
  • ~ — looks for the src in the library name. If found, it replaces the library name with dst.
library name         - /lib/krava1.so
src                  - krava1
dst                  - /lib/krava2.so

final library name   - /lib/krava2.so

🚰 PIPE mode 🚰

latrace can work in two modes, bestie. The first one, native, does the output directly in the traced program process. The other one, pipe mode</>, uses the IPC fifo mechanism to send the data from the traced process over to the latrace process. Then the latrace process is the one responsible for the output. Heads up though: using pipe mode means you lose the traced program standard output context with the printed symbols. 😭

By using pipe mode, latrace isn't dependent on the trace program's usage/manipulation of the standard output descriptor. Also, the symbol statistics counts — that's the -c and -C options — use pipe mode to transfer symbol information to the latrace binary, and then the latrace binary does the counts at the end.

⚙️ CONFIG ⚙️

latrace lets you enable/disable some of the options through a configuration file. Some options are linked to the command line arguments, some of them are not. When latrace starts, the global configuration file gets read. This file is usually placed right here:

/etc/latrace.d/latrace.conf

Once the default values are read from this file, you can overload any of them with command line options, or by supplying another config file via the -N, --conf option. 🌸

The configuration file syntax is roughly:

INCLUDE <anotherconfigfile>

OPTIONS {
        OPTION1 = VALUE
        OPTION2 = YES|NO
        ...
        OPTIONN = VALUE
}

# comment

📝 Configuration file options 📝

  • HEADERS = FILE — -a, --args
  • INDENT_SYM = VALUE — -i, --indent-sym
  • PIPE = BOOL — -p, --pipe
  • TIMESTAMP = BOOL — -S, --timestamp
  • FRAMESIZE = VALUE — -y, --framesize
  • FRAMESIZE_CHECK = BOOL — -Y, --no-framesize-check
  • HIDE_TID = BOOL — -T, --hide-tid
  • FOLLOW_FORK = BOOL — -F, --no-follow-fork
  • FOLLOW_EXEC = BOOL — -E, --no-follow-exec
  • DEMANGLE = BOOL — -d, --demangle
  • BRACES = BOOL — -B, --braces
  • ENABLE_ARGS = BOOL — -A, --enable-args
  • DETAIL_ARGS = BOOL — -D, --detail-args
  • OUTPUT_TTY = FILE
    • stores tracee terminal output to the file
  • LIBS = LIB1[,LIB2,...] — -l, --libs
  • LIBS_TO = LIB1[,LIB2,...] — -t, --libs-to
  • LIBS_FROM = LIB1[,LIB2,...] — -f, --libs-from
  • SYM = SYM1[,SYM2,...] — -s, --sym
  • SYM_OMIT = SYM1[,SYM2,...] — -n, --sym-omit
  • SYM_BELOW = SYM1[,SYM2,...] — -b, --flow-below
  • SYM_NOEXIT = SYM1[,SYM2,...]
    • symbols which do no run exit callback (plt_exit)
  • ARGS_STRING_POINTER_LENGTH = BOOL
    • function arguments - display string length and pointer value

📚 HEADERS 📚

The latrace header file lets you define symbols as classic C functions with arguments. Argument names get displayed together with values in the latrace output. And real talk: the more arguments you define, the more performance and memory penalties you should expect. no cap.

The package ships with several predefined header files for the most commonly used functions. Here's the list of glibc header files used (this list mostly follows the ltrace header files list, and the author is willing to update it according to needs):

ctype.h dirent.h dlfcn.h fcntl.h getopt.h inet.h ioctl.h
libintl.h libio.h locale.h misc.h mman.h ncurses.h netdb.h
pthread.h pwd.h resource.h signal.h socket.h stat.h stdio.h
stdlib.h string.h syslog.h term.h termios.h time.h typedefs.h
unistd.h utmp.h wait.h

The latrace header files are usually stored under this directory:

/etc/latrace.d/headers/

You can specify a single header file using a command line option or the configuration file. That file can then include other needed headers. And like we already said, the latrace config file syntax lightly follows the C language syntax. The next bit describes the latrace config file language. ✨

  • Several POD types (plain old data) are hardcoded in latrace. The size of those arguments is determined by the sizeof macro. List follows.
void
char    u_char
short   u_short
int     u_int
long    u_long
llong   u_llong  # (long long)
float   double
  • The typedef keyword lets you specify a new type based on an already existing one (POD or typedefed). Eventhough there's a way for multiple pointer layers in the type definition (*), only one is taken.
typedef base_type new_type;
typedef base_type * new_type;
typedef base_type ** new_type;
  • Comments follow the C style /* */ logic.
/* comments */
  • The include keyword lets you include another config file.
#include "filename"
  • The struct keyword lets you define the structure. The syntax folows following grammar rules.
START::         struct NAME { STRUCT_DEF };
STRUCT_DEF::    DEF | EMPTY
DEF::           NAME NAME |
                NAME '*' NAME |
                struct NAME NAME |
                struct NAME '*' NAME
NAME::          [-0-9a-zA-Z_]+
  • The function definition follows following syntax (DEF and NAME are the same as for struct definition).
START::         DEF '(' ARGS ')' ';'
ARGS::          ARGS ',' DEF | DEF | EMPTY
  • The enum definition follows following syntax (NAME is same as for struct definition).
START::         ENUM NAME '{' ENUM_DEF '}' ';'
ENUM_DEF::      ENUM_DEF ',' ENUM_ELEM | ENUM_ELEM
ENUM_ELEM::     NAME '=' NAME | NAME
  • Example of a simple latrace config file.
---[ cut here ]-----------------------------
enum krava {
        krava1 = 1,
        krava2,
        krava3 = 100
};

#include "krava.conf"

typedef u_int pid_t;

struct ex_st {
        pid_t   p;
        int     cnt;
        char   *name;
};

int f1(pid_t p, struct ex_st *k);
int f2(char* name, struct ex_st k, int k = krava);
struct ex_st* f3(pid_t *p, struct ex_st k);
---[ cut here ]-----------------------------
  • > Arrays are not supported yet, so there's no way to define some structures. For such structures use void* type where the structure argument is passed by pointer. If it is passed by value, there's no workaround so far (aside from filling the structure body with POD types up to the actual length of the structure :). 💀
  • > Variable argument lists (va_list/...) are not supported yet. The function definition needs to stop before the first variable argument list argument.

🛠️ PORTS 🛠️

latrace should work on any glibc system with LD_AUDIT support. However, argument details are architecture specific and need special support inside latrace itself.

The author is willing to port latrace to any architecture, as long as he gets access to the corresponding system. Currently functional ports are:

  • x86 — ok ✅
  • x86_64 — ok ✅
  • arm — ok ✅

💀 BUGS 💀

MANY, plz report bugs to latrace@lists.fedorahosted.org or olsajiri@gmail.com.

You can also visit the http://people.redhat.com/jolsa/latrace/ page to see the latest release notes information.

LD_AUDIT related glibc bugs:

  • Bug 7055 (no longer reproducible)
  • Bug 9893 (FIXED in 2.10)
  • Bug 3924 (FIXED in 2.7-2)

👑 AUTHOR 👑

Jiri Olsa olsajiri@gmail.com

🫶 CONTRIBUTORS 🫶

  • Nix nix@esperi.org.uk
  • Akos Pasztory akos.pasztory@gmail.com
  • Artur Skawina art.08.09@gmail.com
  • Dr. David Alan Gilbert david.gilbert@linaro.org
  • Sebastian Pipping sping@gentoo.org

📜 LICENSE 📜

This is free software, distributed under the GPLv3 license.

👀 SEE ALSO 👀

strace(1), ltrace(1)

👑 AUTHOR 👑

Jiri Olsa olsajiri@gmail.com

  • Author.

📝 NOTES 📝

okay bestie, these are the little footnote linkies that got referenced all over the place up above 💫 here they are, every single one, nothing skipped, no cap:

  • 1. — latrace@lists.fedorahosted.org 🩷 — link target: mailto:latrace@lists.fedorahosted.org
  • 2. — olsajiri@gmail.com ✨ — link target: mailto:olsajiri@gmail.com
  • 3. — nix@esperi.org.uk 🦋 — link target: mailto:nix@esperi.org.uk
  • 4. — akos.pasztory@gmail.com 🌸 — link target: mailto:akos.pasztory@gmail.com
  • 5. — art.08.09@gmail.com 💅 — link target: mailto:art.08.09@gmail.com
  • 6. — david.gilbert@linaro.org 🌈 — link target: mailto:david.gilbert@linaro.org
  • 7. — sping@gentoo.org ⭐ — link target: mailto:sping@gentoo.org

and that's the whole entire page, bestie 🎀 you made it to the very end, I'm so proud of you fr fr 💖

little footer moment for the road: this manual page is latrace 0.5.11, dated October 2011, and it's filed under LATRACE(1). iconic. periodt. 💅✨🦋

📦 where this page comes from