LTTNG-UST(3)

Hii bestie!! πŸ’– Welcome to the LTTNG-UST(3) page from the LTTng Manual β€” aka the user-space tracing manual. This is part 1, and it is a lot, but we’re gonna get through it together, okay? 🫢

🌸 NAME 🌸

lttng-ust β€” LTTng user space tracing ✨

πŸ’… SYNOPSIS πŸ’…

#include <lttng/tracepoint.h>

#define LTTNG_UST_TP_ARGS(args...)
#define LTTNG_UST_TP_ENUM_VALUES(values...)
#define LTTNG_UST_TP_FIELDS(fields...)
#define LTTNG_UST_TRACEPOINT_ENUM(prov_name, enum_name, mappings)
#define LTTNG_UST_TRACEPOINT_EVENT(prov_name, t_name, args, fields)
#define LTTNG_UST_TRACEPOINT_EVENT_CLASS(cls_prov_name, cls_name,
                                         args, fields)
#define LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(cls_prov_name, cls_name,
                                            inst_prov_name, t_name, args)
#define LTTNG_UST_TRACEPOINT_LOGLEVEL(prov_name, t_name, level)
#define LTTNG_UST_TRACEPOINT_MODEL_EMF_URI(prov_name, t_name, uri)
#define lttng_ust_do_tracepoint(prov_name, t_name, ...)
#define lttng_ust_field_array(int_type, field_name, expr, count)
#define lttng_ust_field_array_nowrite(int_type, field_name, expr, count)
#define lttng_ust_field_array_hex(int_type, field_name, expr, count)
#define lttng_ust_field_array_nowrite_hex(int_type, field_name, expr,
                                          count)
#define lttng_ust_field_array_network(int_type, field_name, expr, count)
#define lttng_ust_field_array_network_nowrite(int_type, field_name,
                                              expr, count)
#define lttng_ust_field_array_network_hex(int_type, field_name, expr,
                                          count)
#define lttng_ust_field_array_network_nowrite_hex(int_type, field_name,
                                                  expr, count)
#define lttng_ust_field_array_text(char, field_name, expr, count)
#define lttng_ust_field_array_text_nowrite(char, field_name, expr,
                                           count)
#define lttng_ust_field_enum(prov_name, enum_name, int_type, field_name,
                             expr)
#define lttng_ust_field_enum_nowrite(prov_name, enum_name, int_type,
                                     field_name, expr)
#define lttng_ust_field_enum_value(label, value)
#define lttng_ust_field_enum_range(label, start, end)
#define lttng_ust_field_float(float_type, field_name, expr)
#define lttng_ust_field_float_nowrite(float_type, field_name, expr)
#define lttng_ust_field_integer(int_type, field_name, expr)
#define lttng_ust_field_integer_hex(int_type, field_name, expr)
#define lttng_ust_field_integer_network(int_type, field_name, expr)
#define lttng_ust_field_integer_network_hex(int_type, field_name, expr)
#define lttng_ust_field_integer_nowrite(int_type, field_name, expr)
#define lttng_ust_field_sequence(int_type, field_name, expr,
                                 len_type, len_expr)
#define lttng_ust_field_sequence_nowrite(int_type, field_name, expr,
                                         len_type, len_expr)
#define lttng_ust_field_sequence_hex(int_type, field_name, expr,
                                     len_type, len_expr)
#define lttng_ust_field_sequence_nowrite_hex(int_type, field_name, expr,
                                             len_type, len_expr)
#define lttng_ust_field_sequence_network(int_type, field_name, expr,
                                         len_type, len_expr)
#define lttng_ust_field_sequence_network_nowrite(int_type, field_name,
                                                 expr, len_type,
                                                 len_expr)
#define lttng_ust_field_sequence_network_hex(int_type, field_name, expr,
                                             len_type, len_expr)
#define lttng_ust_field_sequence_network_nowrite_hex(int_type,
                                                     field_name,
                                                     expr, len_type,
                                                     len_expr)
#define lttng_ust_field_sequence_text(char, field_name, expr, len_type,
                                      len_expr)
#define lttng_ust_field_sequence_text_nowrite(char, field_name, expr,
                                              len_type, len_expr)
#define lttng_ust_field_string(field_name, expr)
#define lttng_ust_field_string_nowrite(field_name, expr)
#define lttng_ust_field_fixed_length_blob(field_name, expr, count,
                                          media_type)
#define lttng_ust_field_fixed_length_blob_nowrite(field_name, expr,
                                                  count, media_type)
#define lttng_ust_field_variable_length_blob(field_name, expr, len_type,
                                             len_expr, media_type)
#define lttng_ust_field_variable_length_blob_nowrite(field_name, expr,
                                                     len_type, len_expr,
                                                     media_type)
#define lttng_ust_field_unused(expr)
#define lttng_ust_field_unused_nowrite(expr)
#define lttng_ust_tracepoint(prov_name, t_name, ...)
#define lttng_ust_tracepoint_enabled(prov_name, t_name)

Link with, following this manual page:

  • -llttng-ust -ldl
  • If you define _LGPL_SOURCE before including <lttng/tracepoint.h> (directly or indirectly): -llttng-ust-common

πŸ’– DESCRIPTION πŸ’–

Okay so the Linux Trace Toolkit: next generation (http://lttng.org/) is an open source software package used for correlated tracing of the Linux kernel, user applications, and user libraries. ✨

LTTng-UST is the user space tracing component of the LTTng project. It is a port to user space of the low-overhead tracing capabilities of the LTTng Linux kernel tracer. The liblttng-ust library is used to trace user applications and libraries.

Note β€” This man page is about the liblttng-ust library. The LTTng-UST project also provides Java and Python packages to trace applications written in those languages. How to instrument and trace Java and Python applications is documented in the online LTTng documentation (http://lttng.org/docs/).

There are three ways to use liblttng-ust:

🌈 Compatibility with previous APIs 🌈

Since LTTng-UST 2.13, the LTTNG_UST_COMPAT_API_VERSION definition controls which LTTng-UST APIs are available (compiled):

  • Undefined β€” all APIs are available.
  • N (0 or positive integer) β€” API version N, and all the following existing APIs, are available. Previous APIs are not available (not compiled).

Here’s the mapping from LTTng-UST versions (up to LTTng-UST 2.16.0) to available API versions:

  • LTTng-UST version 2.0 to 2.12 β†’ available API versions: 0
  • LTTng-UST version 2.13 β†’ available API versions: 0 and 1

This manual page only documents version 1 of the API.

If you wish to have access to version 0 of the API (for example, the tracepoint(), ctf_integer(), and TRACEPOINT_EVENT() macros), then either don’t define LTTNG_UST_COMPAT_API_VERSION, or define it to 0 before including any LTTng-UST header.

🧚 Creating a tracepoint provider 🧚

Creating a tracepoint provider is the first step of using liblttng-ust. The next steps are:

  • Instrumenting your application with lttng_ust_tracepoint() calls
  • Building your application with LTTng-UST support, either statically or dynamically.

A tracepoint provider is a compiled object containing the event probes corresponding to your custom tracepoint definitions. A tracepoint provider contains the code to get the size of an event and to serialize it, amongst other things.

To create a tracepoint provider, start with the following tracepoint provider header template:

#undef LTTNG_UST_TRACEPOINT_PROVIDER
#define LTTNG_UST_TRACEPOINT_PROVIDER my_provider

#undef LTTNG_UST_TRACEPOINT_INCLUDE
#define LTTNG_UST_TRACEPOINT_INCLUDE "./tp.h"

#if !defined(_TP_H) || \
    defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
#define _TP_H

#include <lttng/tracepoint.h>

/*
 * LTTNG_UST_TRACEPOINT_EVENT(), LTTNG_UST_TRACEPOINT_EVENT_CLASS(),
 * LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(),
 * LTTNG_UST_TRACEPOINT_LOGLEVEL(), and `LTTNG_UST_TRACEPOINT_ENUM()`
 * are used here.
 */

#endif /* _TP_H */

#include <lttng/tracepoint-event.h>

In this template, the tracepoint provider is named my_provider (LTTNG_UST_TRACEPOINT_PROVIDER definition). The file needs to bear the name of the LTTNG_UST_TRACEPOINT_INCLUDE definition (tp.h in this case). Between #include <lttng/tracepoint.h> and #endif go the invocations of the LTTNG_UST_TRACEPOINT_EVENT(), LTTNG_UST_TRACEPOINT_EVENT_CLASS(), LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(), LTTNG_UST_TRACEPOINT_LOGLEVEL(), and LTTNG_UST_TRACEPOINT_ENUM() macros.

Note β€” You can avoid writing the prologue and epilogue boilerplate in the template file above by using the lttng-gen-tp(1) tool shipped with LTTng-UST.

The tracepoint provider header file needs to be included in a source file which looks like this:

#define LTTNG_UST_TRACEPOINT_CREATE_PROBES

#include "tp.h"

Together, those two files (let’s call them tp.h and tp.c) form the tracepoint provider sources, ready to be compiled.

You can create multiple tracepoint providers to be used in a single application, but each one must have its own header file.

The LTTNG_UST_TRACEPOINT_EVENT() usage section below shows how to use the LTTNG_UST_TRACEPOINT_EVENT() macro to define the actual tracepoints in the tracepoint provider header file.

See the EXAMPLE section below for a complete example.

πŸ’… LTTNG_UST_TRACEPOINT_EVENT() usage πŸ’…

The LTTNG_UST_TRACEPOINT_EVENT() macro is used in a template provider header file (see the Creating a tracepoint provider section above) to define LTTng-UST tracepoints.

The LTTNG_UST_TRACEPOINT_EVENT() usage template is as follows:

           LTTNG_UST_TRACEPOINT_EVENT(
               /* Tracepoint provider name */
               my_provider,

               /* Tracepoint/event name */
               my_tracepoint,

               /* List of tracepoint arguments (input) */
               LTTNG_UST_TP_ARGS(
                   ...
               ),

               /* List of fields of eventual event (output) */
               LTTNG_UST_TP_FIELDS(
                   ...
               )
           )

So LTTNG_UST_TP_ARGS() is literally just the input side of the whole situation β€” the args your tracepoint takes in. And bestie, those args are exactly what you get to reference later inside the argument expressions of the output fields you define in LTTNG_UST_TP_FIELDS(). It's giving pipeline πŸ’…

Format of the LTTNG_UST_TP_ARGS() params? It's C type first, then argument name β€” and you repeat that pair as many times as you need, up to ten times max. Like so:

           LTTNG_UST_TP_ARGS(
               int, my_int,
               const char *, my_string,
               FILE *, my_file,
               double, my_float,
               struct my_data *, my_data
           )

Meanwhile LTTNG_UST_TP_FIELDS() is the output side β€” the actual data that can be recorded in the payload of an event emitted by this tracepoint. The receipts, basically ✨

LTTNG_UST_TP_FIELDS() holds a list of lttng_ust_field_*() macros with NO commas between them (yes, really, no commas β€” do not come for me about it, that's just how it is). All the available ones are documented in the Available lttng_ust_field_*() field type macros section right below 🫢


🌸 Available field macros 🌸

Okay so this section is the whole menu of lttng_ust_field_*() macros you can drop inside the LTTNG_UST_TP_FIELDS() macro of LTTNG_UST_TRACEPOINT_EVENT(). Buckle up, there's a lot, and they're all iconic in their own way πŸŽ€

Standard integer, displayed in base 10:

           lttng_ust_field_integer(int_type, field_name, expr)
           lttng_ust_field_integer_nowrite(int_type, field_name, expr)

Standard integer, displayed in base 16:

           lttng_ust_field_integer_hex(int_type, field_name, expr)

Integer in network byte order (big endian), displayed in base 10:

           lttng_ust_field_integer_network(int_type, field_name, expr)

Integer in network byte order, displayed in base 16:

           lttng_ust_field_integer_network_hex(int_type, field_name, expr)

Floating point number:

           lttng_ust_field_float(float_type, field_name, expr)
           lttng_ust_field_float_nowrite(float_type, field_name, expr)

Null-terminated string:

           lttng_ust_field_string(field_name, expr)
           lttng_ust_field_string_nowrite(field_name, expr)

Statically-sized array of integers (_hex versions displayed in hexadecimal, _network versions in network byte order):

           lttng_ust_field_array(int_type, field_name, expr, count)
           lttng_ust_field_array_nowrite(int_type, field_name, expr, count)
           lttng_ust_field_array_hex(int_type, field_name, expr, count)
           lttng_ust_field_array_nowrite_hex(int_type, field_name, expr, count)
           lttng_ust_field_array_network(int_type, field_name, expr, count)
           lttng_ust_field_array_network_nowrite(int_type, field_name, expr,
                                                 count)
           lttng_ust_field_array_network_hex(int_type, field_name, expr, count)
           lttng_ust_field_array_network_nowrite_hex(int_type, field_name,
                                                     expr, count)

Statically-sized array, printed as text β€” and nope, it does not need to be null-terminated:

           lttng_ust_field_array_text(char, field_name, expr, count)
           lttng_ust_field_array_text_nowrite(char, field_name, expr, count)

Dynamically-sized array of integers (_hex versions displayed in hexadecimal, _network versions in network byte order):

           lttng_ust_field_sequence(int_type, field_name, expr, len_type,
                                    len_expr)
           lttng_ust_field_sequence_nowrite(int_type, field_name, expr,
                                            len_type, len_expr)
           lttng_ust_field_sequence_hex(int_type, field_name, expr, len_type,
                                        len_expr)
           lttng_ust_field_sequence_nowrite_hex(int_type, field_name, expr,
                                                len_type, len_expr)
           lttng_ust_field_sequence_network(int_type, field_name, expr,
                                            len_type, len_expr)
           lttng_ust_field_sequence_network_nowrite(int_type, field_name, expr,
                                                    len_type, len_expr)
           lttng_ust_field_sequence_network_hex(int_type, field_name, expr,
                                                len_type, len_expr)
           lttng_ust_field_sequence_network_nowrite_hex(int_type, field_name,
                                                        expr, len_type,
                                                        len_expr)

Dynamically-sized array, displayed as text β€” again, no null-termination required, we love a low-maintenance queen:

           lttng_ust_field_sequence_text(char, field_name, expr, len_type,
                                         len_expr)
           lttng_ust_field_sequence_text_nowrite(char, field_name, expr,
                                                 len_type, len_expr)

Fixed-length BLOB β€” that's a fixed number of raw bytes tagged with a media type:

           lttng_ust_field_fixed_length_blob(field_name, expr, count, media_type)
           lttng_ust_field_fixed_length_blob_nowrite(field_name, expr, count,
                                                     media_type)

Variable-length BLOB β€” a dynamic number of raw bytes tagged with a media type:

           lttng_ust_field_variable_length_blob(field_name, expr, len_type,
                                                len_expr, media_type)
           lttng_ust_field_variable_length_blob_nowrite(field_name, expr, len_type,
                                                        len_expr, media_type)

So real talk on BLOBs: a BLOB field records raw bytes, as pointed to by expr, and it does not interpret them β€” no thoughts, just vibes 🫧 The number of bytes to record is count for the fixed-length version, or whatever len_expr evaluates to for the variable-length version. And the media_type parameter associates an IANA media type with the recorded data (more on that below).

Enumeration! Heads up though β€” the enumeration field must be defined before you use this macro, using the LTTNG_UST_TRACEPOINT_ENUM() macro. Peep the LTTNG_UST_TRACEPOINT_ENUM() usage section for the full tea πŸ“

           lttng_ust_field_enum(prov_name, enum_name, int_type, field_name,
                                expr)
           lttng_ust_field_enum_nowrite(prov_name, enum_name, int_type,
                                        field_name, expr)

πŸ’– The parameters, explained πŸ’–

  • count β€” number of elements/bytes in an array/sequence/BLOB. And this has to be known at compile time, no take-backs.
  • enum_name β€” name of an enumeration field you already defined earlier with the LTTNG_UST_TRACEPOINT_ENUM() macro. See the LTTNG_UST_TRACEPOINT_ENUM() usage section for more.
  • expr β€” the C expression that results in the field value. It can use one or more of the arguments passed to the tracepoint; those tracepoint arguments are defined in the LTTNG_UST_TP_ARGS() macro (see the Creating a tracepoint provider section above).
  • field_name β€” the event field name, in C identifier syntax. That is NOT a literal string, bestie, don't you dare put quotes on it πŸ’€
  • float_type β€” float C type, so float or double. The size of this type is what decides the size of the floating point number field.
  • int_type β€” integer C type. Same deal: the size of this type decides the size of the integer/enumeration field.
  • len_expr β€” C expression that results in the sequence length. Can use one or more of the tracepoint's arguments.
  • len_type β€” unsigned integer C type of the sequence length.
  • media_type β€” C expression of type const char * that results in the IANA media type (https://www.iana.org/assignments/media-types/) of a BLOB field's data, for example "image/png".
  • prov_name β€” tracepoint provider name. It must be the same as the tracepoint provider name used in a previous field definition. Consistency is the vibe check ✨
Okay important: the _nowrite versions omit themselves from the recorded trace, but they're otherwise identical. Why would you want that? Their main character purpose is to make some of the event context available to event filters without having to commit the data to sub-buffers. Go read lttng-enable-event(1) to learn more about dynamic event filtering πŸ¦‹

And now the chaotic-neutral icons of the family β€” the following macros record NO field whatsoever. Their whole entire job is to mark a tracepoint argument as deliberately unused, so the compiler doesn't come for you with a warning when that argument isn't used by any other field:

           lttng_ust_field_unused(expr)
           lttng_ust_field_unused_nowrite(expr)

Check the EXAMPLE section below for a complete example πŸ’œ


🧚 LTTNG_UST_TRACEPOINT_ENUM() usage 🧚

An enumeration field is a list of mappings between an integer β€” or a range of integers β€” and strings (sometimes called labels or enumerators, iykyk). Enum fields are the move when the possible values for a field are limited, because they give you a much more compact trace. Slay, efficiency πŸŽ€

You define an enumeration field with the LTTNG_UST_TRACEPOINT_ENUM() macro:

           LTTNG_UST_TRACEPOINT_ENUM(
               /* Tracepoint provider name */
               my_provider,

               /* Enumeration name (unique in the whole tracepoint provider) */
               my_enum,

               /* Enumeration mappings */
               LTTNG_UST_TP_ENUM_VALUES(
                   ...
               )
           )

LTTNG_UST_TP_ENUM_VALUES() holds a list of enumeration mappings, and just like the fields β€” NOT separated by commas, periodt. Two macros are allowed inside LTTNG_UST_TP_ENUM_VALUES(): lttng_ust_field_enum_value() and lttng_ust_field_enum_range().

lttng_ust_field_enum_value() is your single-value mapping:

           lttng_ust_field_enum_value(label, value)

It maps the given label string to the value value. Cute, simple.

lttng_ust_field_enum_range() is the range mapping:

           lttng_ust_field_enum_range(label, start, end)

It maps the given label string to the range of integers from start to end, inclusively. Now, plot twist: range mappings may overlap, but what happens then is implementation-defined β€” each trace reader handles overlapping ranges however it wants. So like, nobody's stopping you, but the results are not guaranteed to be your fantasy 🫠

See the EXAMPLE section below for a complete example.


πŸ‘‘ LTTNG_UST_TRACEPOINT_EVENT_CLASS() usage πŸ‘‘

A tracepoint class is a class of tracepoints that all share the same field types and names. A tracepoint instance is one instance of such a declared tracepoint class, with its own event name. Family energy: same genetics, different name πŸ’…

LTTng-UST creates one event serialization function per tracepoint class. So using LTTNG_UST_TRACEPOINT_EVENT() creates one tracepoint class per tracepoint definition, whereas using LTTNG_UST_TRACEPOINT_EVENT_CLASS() together with LTTNG_UST_TRACEPOINT_EVENT_INSTANCE() creates one tracepoint class plus one or more tracepoint instances of that class. In other words: many tracepoints can reuse the same serialization code. And reusing the same code when you can is highkey a glow-up β€” it can reduce cache pollution, which improves performance. We love a smart, efficient queen πŸ”₯

The LTTNG_UST_TRACEPOINT_EVENT_CLASS() macro accepts the same parameters as the LTTNG_UST_TRACEPOINT_EVENT() macro, except that its second parameter is the tracepoint class name instead of an event name:

           #define LTTNG_UST_TRACEPOINT_PROVIDER my_provider

           /* ... */

(more coming, don't go anywhere πŸ’–)

           LTTNG_UST_TRACEPOINT_EVENT_CLASS(
               /* Tracepoint class provider name */
               my_provider,

               /* Tracepoint class name */
               my_tracepoint_class,

               /* List of tracepoint arguments (input) */
               LTTNG_UST_TP_ARGS(
                   ...
               ),

               /* List of fields of eventual event (output) */
               LTTNG_UST_TP_FIELDS(
                   ...
               )
           )

So once your tracepoint class is defined, bestie, you can spin up as many tracepoint instances as you want. It's giving unlimited copies, no cap:

           #define LTTNG_UST_TRACEPOINT_PROVIDER natality

           /* ... */

           LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(
               /* Name of the tracepoint class provider */
               my_provider,

               /* Tracepoint class name */
               my_tracepoint_class,

               /* Name of the local (instance) tracepoint provider */
               natality,

               /* Tracepoint/event name */
               my_tracepoint,

               /* List of tracepoint arguments (input) */
               LTTNG_UST_TP_ARGS(
                   ...
               )
           )

See how LTTNG_UST_TRACEPOINT_EVENT_INSTANCE() literally does not contain the LTTNG_UST_TP_FIELDS() macro? That's on purpose β€” those fields already live up at the LTTNG_UST_TRACEPOINT_EVENT_CLASS() level. Zero duplication, we love a DRY queen ✨

And heads up: LTTNG_UST_TRACEPOINT_EVENT_INSTANCE() takes two provider names:

  • The name of the tracepoint class provider (my_provider in the example above).
    • This one is the same thing as the first argument of the LTTNG_UST_TRACEPOINT_EVENT_CLASS() expansion, so it knows which class you're referring to.
  • The name of the local, or instance, provider (natality in the example above).
    • This is the provider name that becomes the prefix part of the name of the events such a tracepoint creates.

The two provider names can totally be different β€” that's valid πŸ’… β€” if the tracepoint class and the tracepoint instance macros are sitting in two different translation units.

There's a full example in the EXAMPLE section below if you wanna see the whole thing together.

πŸͺ΅ LTTNG_UST_TRACEPOINT_LOGLEVEL() usage

Okay so optionally β€” like, not required, we're chill β€” you can assign a log level to a tracepoint you already defined. Why would you? Because giving your tracepoints different levels of severity is genuinely useful: when you're controlling tracing sessions you can decide to only enable events that fall into a specific log level range, using the --loglevel and --loglevel-only options of the lttng-enable-event(1) command. Selective queen behavior πŸ‘‘

Log levels get assigned to tracepoints that are already defined, using the LTTNG_UST_TRACEPOINT_LOGLEVEL() macro. And it must come after you've used LTTNG_UST_TRACEPOINT_EVENT() or LTTNG_UST_TRACEPOINT_EVENT_INSTANCE() for that tracepoint β€” order matters, bestie. It goes like this:

           LTTNG_UST_TRACEPOINT_LOGLEVEL(
               /* Tracepoint provider name */
               my_provider,

               /* Tracepoint/event name */
               my_tracepoint,

               /* Log level */
               LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO
           )

Here are all the log level definitions you can use πŸŽ€:

  • LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG β€” system is unusable.
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT β€” action must be taken immediately.
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT β€” critical conditions.
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR β€” error conditions.
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING β€” warning conditions.
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE β€” normal, but significant, condition.
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO β€” informational message.
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM β€” debug information with system-level scope (set of programs).
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM β€” debug information with program-level scope (set of processes).
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS β€” debug information with process-level scope (set of modules).
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE β€” debug information with module (executable/library) scope (set of units).
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT β€” debug information with compilation unit scope (set of functions).
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION β€” debug information with function-level scope.
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE β€” debug information with line-level scope (this one is the default log level, iykyk).
  • LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG β€” debug-level message.

Complete example lives in the EXAMPLE section below, go look 🫢

🧚 Associating an EMF model URI to a tracepoint

Optionally, you can also hand an Eclipse Modeling Framework (EMF) model URI to a tracepoint you already defined. Why? Because a trace reader can use that URI to go find a model describing the events your tracepoint produces. It's basically giving your tracepoint a little home address 🏠

Same energy as the LTTNG_UST_TRACEPOINT_LOGLEVEL() macro: you must use LTTNG_UST_TRACEPOINT_MODEL_EMF_URI() after having used LTTNG_UST_TRACEPOINT_EVENT() or LTTNG_UST_TRACEPOINT_EVENT_INSTANCE() for that tracepoint:

           LTTNG_UST_TRACEPOINT_MODEL_EMF_URI(
               /* Tracepoint provider name */
               my_provider,

               /* Tracepoint/event name */
               my_tracepoint,

               /* Model URI */
               "http://example.com/path/to/model"
           )

πŸ”₯ Instrumenting your application

Okay so once your tracepoint provider exists (peep the Creating a tracepoint provider section above), you can actually instrument your app with the tracepoints you defined β€” and you do it with the lttng_ust_tracepoint() macro, which is the main character here:

           #define lttng_ust_tracepoint(prov_name, t_name, ...)

Where:

  • prov_name β€” tracepoint provider name.
  • t_name β€” tracepoint/event name.
  • ... β€” the tracepoint arguments, if there are any.

And please, literally please, include the tracepoint provider header file anywhere you use lttng_ust_tracepoint() for that provider. No header, no vibes.

Note πŸ’€ Even though LTTng-UST does support lttng_ust_tracepoint() call site duplicates that have the same provider and tracepoint names, it's recommended to use a provider/tracepoint name pair only once within your application source code β€” that's what helps you map events back to their call sites when you're analyzing the trace. One pair, one place, periodt.

Sometimes the arguments you'd pass to a tracepoint are expensive to compute (taking a call stack, for example). And you do not wanna pay that cost when the tracepoint is disabled. So use the lttng_ust_tracepoint_enabled() and lttng_ust_do_tracepoint() macros:

           #define lttng_ust_tracepoint_enabled(prov_name, t_name)
           #define lttng_ust_do_tracepoint(prov_name, t_name, ...)

lttng_ust_tracepoint_enabled() returns a non-zero value if the tracepoint named t_name from the provider named prov_name is enabled at run time.

lttng_ust_do_tracepoint() is like lttng_ust_tracepoint(), except that it doesn't check whether the tracepoint is enabled.

⚠️ And now the serious bit, bestie, listen up: using lttng_ust_tracepoint() together with lttng_ust_tracepoint_enabled() is dangerous, because lttng_ust_tracepoint() also contains the lttng_ust_tracepoint_enabled() check. So a race condition is possible in exactly this situation:
           if (lttng_ust_tracepoint_enabled(my_provider, my_tracepoint)) {
               stuff = prepare_stuff();
           }

           lttng_ust_tracepoint(my_provider, my_tracepoint, stuff);

If the tracepoint becomes enabled after the condition is checked, then stuff never gets prepared: the emitted event will either contain wrong data, or the whole application could crash (segmentation fault, for example). Not iconic, do not do this 😭

Note 🌸 Neither lttng_ust_tracepoint_enabled() nor lttng_ust_do_tracepoint() contain a STAP_PROBEV() call, so if you need one, you should emit that call yourself.

πŸŽ€ Tracing C/C++ constructors and destructors

Okay so if any of the following is true:

  • You compile your C++ application with GCC ≀ 4.8.
  • You compile your C/C++ application with a C++ compiler and define LTTNG_UST_ALLOCATE_COMPOUND_LITERAL_ON_HEAP.

Then LTTng simply will not trace:

  • C constructors and destructors in the application itself, or in statically linked archives.
  • Some C++ constructors and destructors in the application itself, or in statically linked archives.
    • And in this case, which exact C++ constructors and destructors end up untraced depends on the initialization order within each translation unit β€” and across the entire program once all translation units are linked together. So yeah, it's giving "it depends" πŸ™ˆ

πŸ“¦ Statically linking the tracepoint provider

With the static linking method, your compiled tracepoint providers get copied into the target application. Like, physically moved in, they live there now 🏑

Define the LTTNG_UST_TRACEPOINT_DEFINE definition below the LTTNG_UST_TRACEPOINT_CREATE_PROBES definition in the tracepoint provider source:

           #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
           #define LTTNG_UST_TRACEPOINT_DEFINE

           #include "tp.h"

Then create the tracepoint provider object file:

           $ cc -c -I. tp.c
Note ✨ Even though an application instrumented with LTTng-UST tracepoints can totally be compiled with a C++ compiler, tracepoint probes should be compiled with a C compiler. C compiler for probes, C++ compiler for the app. Keep 'em separated πŸ’…

At this point you can archive that tracepoint provider object file β€” possibly along with other object files of your application, or with other tracepoint provider object files β€” as a static library:

           $ ar rc tp.a tp.o

And honestly? Using a static library has a real advantage: it centralises the tracepoint provider objects so they can be shared between multiple applications. That means when the tracepoint provider gets modified, the source code changes don't have to be patched into the source code tree of every single application. The applications do need to be relinked after each change, but they don't need to be otherwise recompiled (unless the API of the tracepoint provider changes). Relink only, no full rebuild, we love efficiency πŸ¦‹

Then link your application with that object file (or with the static library containing it) and with liblttng-ust and libdl (libc on a BSD system):

           $ cc -o app tp.o app.o -llttng-ust -ldl

🌈 Dynamically loading the tracepoint provider

The second approach to packaging the tracepoint provider is to use the dynamic loader: the library and its member functions are explicitly sought out and loaded at run time. Very "we'll find each other when the time is right" of you πŸ’«

In this scenario, the tracepoint provider is compiled as a shared object.

The process for creating the tracepoint provider shared object is pretty much the same as the static linking method β€” except for these two things:

  • Since the tracepoint provider is not part of the application, LTTNG_UST_TRACEPOINT_DEFINE must be defined, for each tracepoint provider, in exactly one source file of the application.
  • LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE must be defined next to LTTNG_UST_TRACEPOINT_DEFINE.

And about LTTNG_UST_TRACEPOINT_DEFINE and LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE β€” the recommended practice is to use a separate C source file in your application just to define them, then include the tracepoint provider header files afterwards. For example, as tp-define.c:

           #define LTTNG_UST_TRACEPOINT_DEFINE
           #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
#include "tp.h"

okay so the tracepoint provider object file that you use to make the shared library? it's built exactly like the static linking method, except you sprinkle in the -fpic option ✨

$ cc -c -fpic -I. tp.c

then it gets linked as a shared library like this, bestie:

$ cc -shared -Wl,--no-as-needed -o tp.so tp.o -llttng-ust

big important vibe check: this tracepoint provider shared object is not linked with your user application. nope. it has to be loaded manually. which is why the app gets built with zero mention of this tracepoint provider, but still needs libdl 🫢

$ cc -o app app.o tp-define.o -ldl

so there are like, two ways to dynamically load the tracepoint provider shared object, pick your fighter:

  • Load it manually from the application using dlopen(3)
  • Make the dynamic loader load it with the LD_PRELOAD environment variable (see ld.so(8))

and if the application doesn't dynamically load the tracepoint provider shared object using one of those methods? tracing is just disabled for that application, and the events don't even show up in the output of lttng-list(1) πŸ’€

ngl it is not safe to use dlclose(3) on a tracepoint provider shared object that's actively being used for tracing, because LTTng-UST doesn't do reference counting back to the shared object. no cap, just don't.

so for example: statically linking a tracepoint provider to a shared object that's gonna be dynamically loaded by an application (a plugin, for example) is not safe β€” because that shared object, which contains the tracepoint provider, could get dynamically closed (dlclose(3)) at any time by the application 😭

to instrument a shared object, you either:

  • statically link the tracepoint provider to the application, or
  • build the tracepoint provider as a shared object (following the procedure shown in this section), and preload it when tracing is needed using the LD_PRELOAD environment variable

🌸 Using LTTng-UST with daemons 🌸

okay so daemons need a little extra love πŸŽ€ some extra care is needed when you're using liblttng-ust with daemon applications that call fork(2), clone(2), or BSD's rfork(2) without a following exec(3) family system call. the library liblttng-ust-fork.so needs to be preloaded before you start the application, using the LD_PRELOAD environment variable (see ld.so(8)).

and if you're using liblttng-ust with a daemon application that closes file descriptors it didn't open itself? preload the liblttng-ust-fd.so library before you start the application 🫢 typical use cases include daemons closing all file descriptors after fork(2), and buggy applications doing "double-closes" (iykyk πŸ™ˆ).


✨ Context information ✨

so context information can be prepended by the LTTng-UST tracer before each event, or before specific events πŸ’…

context fields can be added to specific channels using lttng-add-context(1).

the following context fields are supported by LTTng-UST:

πŸŽ€ General context fields πŸŽ€

  • cpu_id β€” CPU ID.
Note β€” with the default per-CPU buffer allocation policy (see lttng-enable-channel(1)), this context field is always enabled, and it cannot be added with lttng-add-context(1): its main purpose is to be used for dynamic event filtering. See lttng-enable-event(1) for more information about event filtering.
  • ip β€” Instruction pointer: enables recording the exact address from which an event was emitted. this context field can be used to reverse-lookup the source location that caused the event to be emitted.
  • pthread_id β€” POSIX thread identifier. can be used on architectures where pthread_t maps nicely to an unsigned long type.

πŸ¦‹ Process context fields πŸ¦‹

  • procname β€” Thread name, as set by exec(3) or prctl(2). it is recommended that programs set their thread name with prctl(2) before hitting the first tracepoint for that thread.
  • vpid β€” Virtual process ID: process ID as seen from the point of view of the current process ID namespace (see pid_namespaces(7)).
  • vtid β€” Virtual thread ID: thread ID as seen from the point of view of the current process ID namespace (see pid_namespaces(7)).

πŸ”₯ perf context fields πŸ”₯

  • perf:thread:COUNTER β€” perf counter named COUNTER. use lttng add-context --list to list the available perf counters. only available on IA-32 and x86-64 architectures.
  • perf:thread:raw:rN:NAME β€” perf counter with raw ID N and custom name NAME. see lttng-add-context(1) for more details.

🌈 Namespace context fields (see namespaces(7)) 🌈

  • cgroup_ns β€” Inode number of the current control group namespace (see cgroup_namespaces(7)) in the proc file system.
  • ipc_ns β€” Inode number of the current IPC namespace (see ipc_namespaces(7)) in the proc file system.
  • mnt_ns β€” Inode number of the current mount point namespace (see mount_namespaces(7)) in the proc file system.
  • net_ns β€” Inode number of the current network namespace (see network_namespaces(7)) in the proc file system.
  • pid_ns β€” Inode number of the current process ID namespace (see pid_namespaces(7)) in the proc file system.
  • time_ns β€” Inode number of the current clock namespace (see time_namespaces(7)) in the proc file system.
  • user_ns β€” Inode number of the current user namespace (see user_namespaces(7)) in the proc file system.
  • uts_ns β€” Inode number of the current UTS namespace (see uts_namespaces(7)) in the proc file system.

πŸ’œ Credential context fields (see credentials(7)) πŸ’œ

  • vuid β€” Virtual real user ID: real user ID as seen from the point of view of the current user namespace (see user_namespaces(7)).
  • vgid β€” Virtual real group ID: real group ID as seen from the point of view of the current user namespace (see user_namespaces(7)).
  • veuid β€” Virtual effective user ID: effective user ID as seen from the point of view of the current user namespace (see user_namespaces(7)).
  • vegid β€” Virtual effective group ID: effective group ID as seen from the point of view of the current user namespace (see user_namespaces(7)).
  • vsuid β€” Virtual saved set-user ID: saved set-user ID as seen from the point of view of the current user namespace (see user_namespaces(7)).
  • vsgid β€” Virtual saved set-group ID: saved set-group ID as seen from the point of view of the current user namespace (see user_namespaces(7)).

πŸ‘‘ LTTng-UST state dump πŸ‘‘

if an application that uses liblttng-ust becomes part of a tracing session, info about its currently loaded shared objects, their build IDs, and their debug link information gets emitted as events by the tracer πŸ“‹

the following LTTng-UST state dump events exist and must be enabled to record application state dumps. and hey, heads up: during the state dump phase, LTTng-UST can also emit shared library load/unload events (see Shared library load/unload tracking below πŸ‘€).

lttng_ust_statedump:start

Emitted when the state dump begins. this event has no fields.

lttng_ust_statedump:end

Emitted when the state dump ends. once this event is emitted, it is guaranteed that, for a given process, the state dump is complete βœ… this event has no fields.

lttng_ust_statedump:bin_info

Emitted when information about a currently loaded executable or shared object is found 🎯

Fields:

  • baddr β€” Base address of loaded executable.
  • memsz β€” Size of loaded executable in memory.
  • path β€” Path to loaded executable file.
  • is_pic β€” Whether or not the executable is position-independent code.
  • has_build_id β€” Whether or not the executable has a build ID. if this field is 1, you can expect that an lttng_ust_statedump:build_id event record follows this one (not necessarily immediately after).
  • has_debug_link β€” Whether or not the executable has debug link information. if this field is 1, you can expect that an lttng_ust_statedump:debug_link event record follows this one (not necessarily immediately after).

lttng_ust_statedump:build_id

Emitted when a build ID is found in a currently loaded shared library. see Debugging Information in Separate Files (https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html) for more information about build IDs πŸ’…

Okay so we're mid-event here, still listing fields β€” same energy, keep scrolling bestie πŸ’–

Fields:

  • baddr β€” base address of the loaded library πŸŽ€
  • build_id β€” the Build ID ✨

Emitted when debug link information is found in a currently loaded shared library πŸ¦‹. Peep Debugging Information in Separate Files (https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html) if you wanna know more about debug links β€” it's giving documentation, respectfully.

Fields:

  • baddr β€” base address of the loaded library
  • crc β€” CRC of the debug link file
  • filename β€” debug link file name πŸŽ€

πŸ’œ lttng_ust_statedump:procname

The process procname, right at process start. Lowkey the first thing it says about itself πŸ‘‘

Fields:

  • procname β€” the process name

πŸ¦‹ Shared library load/unload tracking

So the LTTng-UST state dump and the LTTng-UST helper library that instruments the dynamic linker (see liblttng-ust-dl(3)) can emit shared library load/unload tracking events. Iconic of them honestly.

The following shared library load/unload tracking events exist, and ngl you must enable them if you actually want to track the loading and unloading of shared libraries πŸ”₯:

⭐ lttng_ust_lib:load

Emitted when a shared library (shared object) gets loaded πŸŽ€

Fields:

  • baddr β€” base address of the loaded library
  • memsz β€” size of the loaded library in memory
  • path β€” path to the loaded library file πŸ¦‹
  • has_build_id β€” whether or not the library has a build ID. If this field is 1, you can expect that an lttng_ust_lib:build_id event record follows this one (not necessarily immediately after πŸ‘€)
  • has_debug_link β€” whether or not the library has debug link information. If this field is 1, you can expect that an lttng_ust_lib:debug_link event record follows this one (again, not necessarily immediately after πŸ™ˆ)

πŸ’… lttng_ust_lib:unload

Emitted when a shared library (shared object) gets unloaded. bye bestie πŸ‘‹

Fields:

  • baddr β€” base address of the unloaded library

🌈 lttng_ust_lib:build_id

Emitted when a build ID is found in a loaded shared library (shared object). See Debugging Information in Separate Files (https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html) for more about build IDs ✨

Fields:

  • baddr β€” base address of the loaded library
  • build_id β€” the Build ID

Emitted when debug link information is found in a loaded shared library (shared object). See Debugging Information in Separate Files (https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html) for more about debug links 🫢

Fields:

  • baddr β€” base address of the loaded library
  • crc β€” CRC of the debug link file
  • filename β€” debug link file name

πŸ‘‘ Detect if LTTng-UST is loaded

Wanna know from inside your app whether liblttng-ust is even there? Bestie, it's two steps, you got this πŸ’ͺ

  1. Define the lttng_ust_loaded weak symbol globally:
int lttng_ust_loaded __attribute__((weak));

This weak symbol is set by the constructor of liblttng-ust β€” so like, it sets itself, you don't have to do anything πŸ’…

  1. Test lttng_ust_loaded wherever you need it:
/* ... */

if (lttng_ust_loaded) {
    /* LTTng-UST is loaded */
} else {
    /* LTTng-UST is NOT loaded */
}

/* ... */

And that's literally the whole vibe check β€” 0 means she's not there, and a full 1-ish truthy means she ate and left no crumbs πŸ”₯

🌈 EXAMPLE 🌈

πŸ’‘ Note: a few examples are available in the doc/examples <https://github.com/lttng/lttng-ust/tree/stable-2.16/doc/examples> directory of LTTng-UST's source tree. Go look, they're cute.

Okay so this next bit is the iconic one β€” the example that flexes every single feature we just talked about in the previous sections. And it picks the static linking method to link the app with the tracepoint provider, because that's the whole point of the demo. Slay.

You can compile the source files and link them together statically like this:

$ cc -c -I. tp.c
$ cc -c app.c
$ cc -o app tp.o app.o -llttng-ust -ldl

Then, using the lttng(1) tool, you create an LTTng tracing session, enable all the events of this tracepoint provider, and start tracing:

$ lttng create my-session
$ lttng enable-event --userspace 'my_provider:*'
$ lttng start

You may also enable specific events, if you're feeling picky (valid):

$ lttng enable-event --userspace my_provider:big_event
$ lttng enable-event --userspace my_provider:event_instance2

Run the application:

$ ./app some arguments

Then stop the current tracing session and inspect the recorded events:

$ lttng stop
$ lttng view

🌸 Tracepoint provider header file 🌸

tp.h:

#undef LTTNG_UST_TRACEPOINT_PROVIDER
#define LTTNG_UST_TRACEPOINT_PROVIDER my_provider

#undef LTTNG_USTTRACEPOINT_INCLUDE
#define LTTNG_USTTRACEPOINT_INCLUDE "./tp.h"

#if !defined(_TP_H) || \
    defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
#define _TP_H

#include <lttng/tracepoint.h>
#include <stdio.h>

#include "app.h"

LTTNG_UST_TRACEPOINT_EVENT(
    my_provider,
    simple_event,
    LTTNG_UST_TP_ARGS(
        int, my_integer_arg,
        const char *, my_string_arg
    ),
    LTTNG_UST_TP_FIELDS(
        lttng_ust_field_string(argc, my_string_arg)
        lttng_ust_field_integer(int, argv, my_integer_arg)
    )
)

LTTNG_UST_TRACEPOINT_ENUM(
    my_provider,
    my_enum,
    LTTNG_UST_TP_ENUM_VALUES(
        lttng_ust_field_enum_value("ZERO", 0)
        lttng_ust_field_enum_value("ONE", 1)
        lttng_ust_field_enum_value("TWO", 2)
        lttng_ust_field_enum_range("A RANGE", 52, 125)
        lttng_ust_field_enum_value("ONE THOUSAND", 1000)
    )
)

LTTNG_UST_TRACEPOINT_EVENT(
    my_provider,
    big_event,
    LTTNG_UST_TP_ARGS(
        int, my_integer_arg,
        const char *, my_string_arg,
        FILE *, stream,
        double, flt_arg,
        int *, array_arg
    ),
    LTTNG_UST_TP_FIELDS(
        lttng_ust_field_integer(int, int_field1, my_integer_arg * 2)
        lttng_ust_field_integer_hex(long int, stream_pos,
                                    ftell(stream))
        lttng_ust_field_float(double, float_field, flt_arg)
        lttng_ust_field_string(string_field, my_string_arg)
        lttng_ust_field_array(int, array_field, array_arg, 7)
        lttng_ust_field_array_text(char, array_text_field,
                                   array_arg, 5)
        lttng_ust_field_sequence(int, seq_field, array_arg, unsigned int,
                                 my_integer_arg / 10)
        lttng_ust_field_sequence_text(char, seq_text_field,
                                      array_arg, unsigned int,
                                      my_integer_arg / 5)
        lttng_ust_field_enum(my_provider, my_enum, int,
                             enum_field, array_arg[1])
    )
)

LTTNG_UST_TRACEPOINT_LOGLEVEL(my_provider, big_event,
                              LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING)

LTTNG_UST_TRACEPOINT_EVENT_CLASS(
    my_provider,
    my_tracepoint_class,
    LTTNG_UST_TP_ARGS(
        int, my_integer_arg,
        struct app_struct *, app_struct_arg
    ),
    LTTNG_UST_TP_FIELDS(
        lttng_ust_field_integer(int, a, my_integer_arg)
        lttng_ust_field_integer(unsigned long, b, app_struct_arg->b)
        lttng_ust_field_string(c, app_struct_arg->c)
    )
)

LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(
    my_provider,
    my_tracepoint_class,
    my_provider,
    event_instance1,
    LTTNG_UST_TP_ARGS(
        int, my_integer_arg,
        struct app_struct *, app_struct_arg
    )
)

LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(
    my_provider,
    my_tracepoint_class,
    my_provider,
    event_instance2,
    LTTNG_UST_TP_ARGS(
        int, my_integer_arg,
        struct app_struct *, app_struct_arg
    )
)

LTTNG_UST_TRACEPOINT_LOGLEVEL(my_provider, event_instance2,
                              LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO)

LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(
    my_provider,
    my_tracepoint_class,
    my_provider,
    event_instance3,
    LTTNG_UST_TP_ARGS(
        int, my_integer_arg,
        struct app_struct *, app_struct_arg
    )
)

#endif /* _TP_H */

#include <lttng/tracepoint-event.h>

Look at that β€” the #undef/#define pair up top sets the provider name and the include guard, then the #if !defined(_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ) dance, and the trailing #include <lttng/tracepoint-event.h> after the #endif. That ordering is not a vibe, it's a requirement, iykyk. 🧚


πŸ¦‹ Tracepoint provider source file πŸ¦‹

tp.c:

#define LTTNG_UST_TRACEPOINT_CREATE_PROBES
#define LTTNG_UST_TRACEPOINT_DEFINE

#include "tp.h"

Two defines, one include. That's it. Certified low-effort high-impact. ✨


πŸŽ€ Application header file πŸŽ€

app.h:

#ifndef _APP_H
#define _APP_H

struct app_struct {
    unsigned long b;
    const char *c;
    double d;
};

#endif /* _APP_H */

πŸ’œ Application source file πŸ’œ

app.c:

#include <stdlib.h>
#include <stdio.h>

#include "tp.h"
#include "app.h"

static int array_of_ints[] = {
    100, -35, 1, 23, 14, -6, 28, 1001, -3000,
};

int main(int argc, char* argv[])
{
    FILE *stream;
    struct app_struct app_struct;

    lttng_ust_tracepoint(my_provider, simple_event, argc, argv[0]);
    stream = fopen("/tmp/app.txt", "w");

    if (!stream) {
        fprintf(stderr,
                "Error: Cannot open /tmp/app.txt for writing\n");
        return EXIT_FAILURE;
    }

    if (fprintf(stream, "0123456789") != 10) {
        fclose(stream);
        fprintf(stderr, "Error: Cannot write to /tmp/app.txt\n");
        return EXIT_FAILURE;
    }

    lttng_ust_tracepoint(my_provider, big_event, 35,
                         "hello tracepoint", stream, -3.14,
                         array_of_ints);
    fclose(stream);
    app_struct.b = argc;
    app_struct.c = "[the string]";
    lttng_ust_tracepoint(my_provider, event_instance1, 23,
                         &app_struct);
    app_struct.b = argc * 5;
    app_struct.c = "[other string]";
    lttng_ust_tracepoint(my_provider, event_instance2, 17,
                         &app_struct);
    app_struct.b = 23;
    app_struct.c = "nothing";
    lttng_ust_tracepoint(my_provider, event_instance3, -52,
                         &app_struct);
    return EXIT_SUCCESS;
}

Notice how it calls lttng_ust_tracepoint(my_provider, simple_event, argc, argv[0]) first, then later fires big_event with 35, "hello tracepoint", stream, -3.14 and array_of_ints, and then the three event_instance* calls with their own little app_struct variations β€” each one a slightly different vibe check on the same tracepoint class. The /tmp/app.txt writes are there so stream_pos/ftell(stream) actually has something to report, bestie. πŸ”₯

And that's literally the entire example, start to finish 🫢 β€” more page coming, don't go anywhere πŸ’…

🌸 ENVIRONMENT VARIABLES 🌸

Okay bestie, this is the part where you get to boss liblttng-ust around before it even wakes up πŸ’… Set these in your shell/env and watch the vibes shift.

  • LTTNG_HOME β€” a whole alternate home directory for your user. This one's lowkey a lifesaver when the person running the instrumented app has a home directory that's not writable, because this path is literally where the Unix sockets live for talking to the per-user session daemon. No write access, no sockets, no slay 😭
  • LTTNG_UST_ABORT_ON_CRITICAL β€” set it and the instrumented application will straight-up abort when it hits a critical error message. Dramatic? Yes. Effective? Also yes πŸ’€
  • LTTNG_UST_ALLOW_BLOCKING β€” set it and you're giving the app permission to retry event tracing when there's no space left for the event record in the sub-buffer. Which basically means it blocks the app until space frees up or the configured timeout hits. Genuinely the "wait for your turn" energy 🫢

Heads up though β€” to actually let an app block during tracing you also need to specify a blocking timeout when you create a channel, using the --blocking-timeout option of the lttng-enable-channel(1) command. Both pieces, or it's giving nothing ✨

This one's genuinely useful for workloads generating very large trace data throughput, where blocking the application is an acceptable trade-off to prevent discarding event records.

⚠️ Setting this environment variable may significantly affect application timings. Not a drill, bestie β€” your timings will notice 😳
  • LTTNG_UST_APP_PATH β€” the path of the directory where you expect the LTTng session daemon to drop these files so instrumented applications can make connection requests:
    • the application registration Unix socket πŸ”Œ
    • the "wait" shared memory files, which wake up instrumented applications that are waiting for a session daemon to start ⏰
    • the agent port file
      • this file contains the TCP port the agents must connect to in order to enable LTTng tracing

This directory must exist when you start the application for LTTng tracing to be enabled. No directory, no tracing, that's the rule πŸŽ€

When LTTNG_UST_APP_PATH is set, liblttng-ust:

  • only considers this path to connect to a session daemon
  • won't connect to root and/or user session daemons as usual πŸ‘‘

See the corresponding LTTNG_UST_CTL_PATH environment variable of lttng-sessiond(8).

  • LTTNG_UST_CLOCK_PLUGIN β€” path to the shared object acting as the clock override plugin. There's an example of such a plugin in the LTTng-UST documentation under examples/clock-override <https://github.com/lttng/lttng-ust/tree/stable-2.16/doc/examples/clock-override>. Custom clock, main character energy ⭐
  • LTTNG_UST_DEBUG β€” set it and you get the debug and error output of liblttng-ust. Maximum tea β˜•
  • LTTNG_UST_GETCPU_PLUGIN β€” path to the shared object acting as the getcpu() override plugin. Example lives in the LTTng-UST documentation under examples/getcpu-override <https://github.com/lttng/lttng-ust/tree/stable-2.16/doc/examples/getcpu-override> πŸ¦‹
  • LTTNG_UST_MAP_POPULATE_POLICY β€” set it to override the policy used to populate shared memory pages within the application. The possible values are:
    • none (default) β€” do not pre-populate any page: take minor faults on first access while tracing
    • cpu_possible β€” pre-populate pages for all possible CPUs in the system, as shown in /sys/devices/system/cpu/possible
  • LTTNG_UST_REGISTER_TIMEOUT β€” the waiting time, in milliseconds, for the registration done session daemon command before proceeding to execute the main program ⏳

The value 0 means do not wait. The value -1 means wait forever. Setting this environment variable to 0 is recommended for applications with time constraints on the process startup time (aka the impatient girlies, we respect you).

Default: 3000.

  • LTTNG_UST_WITHOUT_BADDR_STATEDUMP β€” set it and liblttng-ust will not perform a base address state dump (peep the LTTng-UST state dump section above) πŸ™ˆ
  • LTTNG_UST_WITHOUT_PROCNAME_STATEDUMP β€” set it and liblttng-ust will not perform a procname state dump (again, see the LTTng-UST state dump section above) πŸ™ˆ

πŸ’€ BUGS πŸ’€

If you encounter any issue or usability problem, please report it on the LTTng bug tracker <https://bugs.lttng.org/projects/lttng-ust>. Seriously, tell them β€” they want to know, and honestly the glow-up starts with feedback πŸ’–

🌈 RESOURCES 🌈

Come hang out, iykyk 🫢

πŸ‘‘ COPYRIGHTS πŸ‘‘

This library is part of the LTTng-UST project.

This library is distributed under the GNU Lesser General Public License, version 2.1 <http://www.gnu.org/licenses/old-licenses/lgpl-2.1.en.html>. See the COPYING <https://github.com/lttng/lttng-ust/blob/v2.16/COPYING> file for more details. Respect the license, it's literally the law of the land πŸ“œβœ¨

🌸 THANKS 🌸

Thanks to Ericsson for funding this work, providing real-life use cases, and testing. Genuine icons, no cap πŸ’œ

Special thanks to Michel Dagenais and the DORSAL laboratory <http://www.dorsal.polymtl.ca/> at Γ‰cole Polytechnique de MontrΓ©al for the LTTng journey. What a ride 🎒

⭐ AUTHORS ⭐

LTTng-UST was originally written by Mathieu Desnoyers, with additional contributions from various other people. It is currently maintained by Mathieu Desnoyers <mailto:mathieu.desnoyers@efficios.com>.

πŸ¦‹ SEE ALSO πŸ¦‹

lttng_ust_tracef(3), lttng_ust_tracelog(3), lttng-gen-tp(1), lttng-ust-dl(3), lttng-ust-cyg-profile(3), lttng(1), lttng-enable-event(1), lttng-list(1), lttng-add-context(1), babeltrace(1), dlopen(3), ld.so(8)


And that's the whole page bestie, you made it to the end πŸŽ‰ This one's from LTTng 2.16.0, dated 08/27/2026 β€” go forth and trace, you absolute legend πŸ’…πŸ’–

πŸ“¦ where this page comes from
  • arch rolling: lttng-ust 2.16.0-1 Β· LGPL-2.1-only AND MIT AND GPL-2.0-only AND BSD-3-Clause AND BSD-2-Clause AND GPL-3.0-or-later
  • πŸ’– girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on : a language model's rewrite of the original, under its terms