LTTNG-CONCEPTS(7)

📦
version 2.15.1 · 8 versions
, in nixpkgs unstable, ubuntu stonking

💖 NAME 💖

lttng-concepts — the LTTng concepts, bestie. all of them. buckle up 🎀

(this one lives in the LTTng Manual, so like, yeah, that's the vibe ✨)

🌸 DESCRIPTION 🌸

okay so this page is the page. it documents the concepts of LTTng, like, the whole mental model, and honestly every other LTTng manpage points back here because they all assume you already know what the objects are and how they're all connected to each other 🫶

the concepts of LTTng 2.15.1 are:

  • Instrumentation point, event rule, and event
  • Trigger
  • Recording session
  • Tracing domain
  • Channel and ring buffer
  • Recording event rule and event record

periodt, that's the whole squad 💅

✨ INSTRUMENTATION POINT, EVENT RULE, AND EVENT ✨

so. an instrumentation point is a point, inside some piece of software, which — when it gets executed — creates an LTTng event. literally just a little spot in code that goes "hey, something happened" 🦋

LTTng offers various types of instrumentation; peep the "Instrumentation point types" section down below to learn about them.

an event rule is a set of conditions to match a set of events.

when LTTng creates an event E, an event rule ER is said to match E when E satisfies all the conditions of ER. this concept is similar to a regular expression which matches a set of strings. iykyk 💜

when an event rule matches an event, LTTng emits the event, therefore attempting to execute one or more actions. and that's the whole plot 😭

Important the event creation and emission processes are documentation concepts to help understand the journey from an instrumentation point to the execution of actions. like, they're a story we tell to make it make sense, not literal API calls. the actual creation of an event can be costly because LTTng needs to evaluate the arguments of the instrumentation point. so, ouch 💀 in practice, LTTng implements various optimizations for the Linux kernel and user space tracing domains (see the "TRACING DOMAIN" section below) to avoid actually creating an event when the tracer knows — thanks to properties which are independent from the event payload and current context — that it would never emit such an event. those properties are: - The instrumentation point type (see the "Instrumentation point types" section below). - The instrumentation point name. - The instrumentation point log level. - For a recording event rule (see the "RECORDING EVENT RULE AND EVENT RECORD" section below): - The status of the rule itself. - The status of the channel (see the "CHANNEL AND RING BUFFER" section below). - The activity of the recording session (started or stopped; see the "RECORDING SESSION" section below). - Whether or not the process for which LTTng would create the event is allowed to record events (see lttng-track(1)). in other words: if, for a given instrumentation point IP, the LTTng tracer knows that it would never emit an event, executing IP represents a simple boolean variable check and, for a Linux kernel recording event rule, a few process attribute checks. so cheap. iconically cheap 🔥

as of LTTng 2.15.1, there are two places where you can find an event rule:

  • Recording event rule — a specific type of event rule of which the action is to record the matched event as an event record.
    • see the "RECORDING EVENT RULE AND EVENT RECORD" section below.
    • create or enable a recording event rule with the lttng-enable-event(1) command. 👑
    • list the recording event rules of a specific recording session and/or channel with the lttng-list(1) and lttng-status(1) commands.
  • "Event rule matches" trigger condition (since LTTng 2.13) — when the event rule of the trigger condition matches an event, LTTng can execute user-defined actions such as sending an LTTng notification, starting a recording session, and more.

for LTTng to emit an event E, E must satisfy all the basic conditions of an event rule ER, that is:

  • The instrumentation point from which LTTng creates E has a specific type.
    • see the "Instrumentation point types" section below.
  • A pattern matches the name of E while another pattern doesn't.
  • The log level of the instrumentation point from which LTTng creates E is at least as severe as some value, or is exactly some value.
  • The fields of the payload of E and the current context fields satisfy a filter expression.

a recording event rule has additional, implicit conditions to satisfy. see the "RECORDING EVENT RULE AND EVENT RECORD" section below to learn more. sneak peek energy 🙈

🧚 Instrumentation point types 🧚

as of LTTng 2.15.1, the available instrumentation point types are, depending on the tracing domain (see the "TRACING DOMAIN" section below):

Linux kernel

  • LTTng tracepoint — a statically defined point in the source code of the kernel image or of a kernel module using the LTTng-modules macros.
    • list the available Linux kernel tracepoints with lttng list --kernel. see lttng-list(1) to learn more.
  • Linux kernel system call — entry, exit, or both of a Linux kernel system call.
    • list the available Linux kernel system call instrumentation points with lttng list --kernel --syscall. see lttng-list(1) to learn more.
  • Linux kprobe — a single probe dynamically placed in the compiled kernel code.
    • when you create such an instrumentation point, you set its memory address or symbol name.
  • Linux user space probe — a single probe dynamically placed at the entry of a compiled user space application/library function through the kernel.
    • when you create such an instrumentation point, you set:
      • With the ELF method — its application/library path and its symbol name.
      • With the USDT method — its application/library path, its provider name, and its probe name.
        • "USDT" stands for SystemTap User-level Statically Defined Tracing, a DTrace-style marker.
    • as of LTTng 2.15.1, LTTng only supports USDT probes which are NOT reference-counted. important!! ⚠️
  • Linux kretprobe — entry, exit, or both of a Linux kernel function.
    • when you create such an instrumentation point, you set the memory address or symbol name of its function.

User space

  • LTTng tracepoint — a statically defined point in the source code of a C/C++ application/library using the LTTng-UST macros.
    • list the available Linux kernel tracepoints with lttng list --userspace. see lttng-list(1) to learn more. (yes it says "Linux kernel tracepoints", that's what the page says, don't @ me 🙈)

java.util.logging, Apache log4j 1.x, Apache Log4j 2, and Python

  • Java or Python logging statement — a method call on a Java or Python logger attached to an LTTng-UST handler.
    • list the available Java and Python loggers with lttng list --jul, lttng list --log4j, lttng list --log4j2, and lttng list --python. see lttng-list(1) to learn more.

⭐ TRIGGER ⭐

a trigger associates a condition to one or more actions.

when the condition of a trigger is satisfied, LTTng attempts to execute its actions. no cap 🌈

as of LTTng 2.15.1, the available trigger conditions and actions are:

Conditions

  • The consumed buffer size of a given recording session (see the "RECORDING SESSION" section below) becomes greater than some value.
  • The buffer usage of a given channel (see the "CHANNEL AND RING BUFFER" section below) becomes greater than some value.
  • The buffer usage of a given channel becomes less than some value.
  • There's an ongoing recording session rotation (see the "Recording session rotation" section below).
  • A recording session rotation becomes completed.
  • An event rule matches an event.
    • as of LTTng 2.15.1, this is the only available condition when you add a trigger with the lttng-add-trigger(1) command. the other ones are available through the liblttng-ctl C API.

Actions

  • Send a notification to a user application.
  • Start a given recording session, like lttng-start(1) would do.
  • Stop a given recording session, like lttng-stop(1) would do.
  • Archive the current trace chunk of a given recording session (rotate), like lttng-rotate(1) would do.
  • Take a snapshot of a given recording session, like lttng-snapshot(1) would do.

a trigger belongs to a session daemon (see lttng-sessiond(8)), not to a specific recording session. for a given session daemon, each Unix user has its own, private triggers. preciate that 🫶

note, however, that the root Unix user may, for the root session daemon:

  • Add a trigger as another Unix user.
  • List all the triggers, regardless of their owner.
  • Remove a trigger which belongs to another Unix user.

for a given session daemon and Unix user, a trigger has a unique name.

that's triggers, bestie. we're not done though — more parts incoming 💖

💖 RECORDING SESSION 💖

A recording session (which was called a “tracing session” before LTTng 2.13, if you're reading old docs and got confused, that's why bestie) is a stateful dialogue between you and a session daemon (peep lttng-sessiond(8)) about literally everything event-recording related.

Every single thing you do when you boss your LTTng tracers around to record events? All of it happens inside a recording session. And like, specifically, a recording session:

  • Has its own name, unique for a given session daemon.
  • Has its own set of trace files, if any.
  • Has its own state of activity (started or stopped).

An active recording session is an implicit recording event rule condition (go look at the “RECORDING EVENT RULE AND EVENT RECORD” section below 👀).

  • Has its own mode (local, network streaming, snapshot, or live).

See the “Recording session modes” section below for the tea ☕.

  • Has its own channels (see the “CHANNEL AND RING BUFFER” section below) to which are attached their own recording event rules.
  • Has its own process attribute inclusion sets (see lttng-track(1)).

All those attributes and objects are completely isolated between different recording sessions. No crossover, no cross-contamination, nothing. ✨

Okay so think of a recording session like an ATM session: the stuff you do to the banking system through the ATM does not mess with anybody else's data on that same system. With an ATM, your session lives as long as your bank card is in the slot. With LTTng, a recording session lives from the lttng-create(1) command alllll the way to the lttng-destroy(1) command. 🏧

A recording session belongs to a session daemon (see lttng-sessiond(8)). For a given session daemon, each Unix user has their own private recording sessions. But — and this is important — the root Unix user may operate on or destroy another user's recording session. 👑

🫶 Current recording session

When you run the lttng-create(1) command, LTTng creates the $LTTNG_HOME/.lttngrc file if it doesn't already exist (and $LTTNG_HOME defaults to $HOME, no cap).

$LTTNG_HOME/.lttngrc contains the name of the current recording session.

When you create a new recording session with the create command, LTTng updates the current recording session. Fresh session, who dis. 💅

The following lttng(1) commands select the current recording session if you don't specify one — like they'll just assume you mean the current one, which is so them:

You can also set the current recording session manually with the lttng-set-session(1) command, so you don't have to go hand-editing the .lttngrc file like some kind of gremlin. 😌

🌈 Recording session modes

LTTng offers four recording session modes, and each one has a whole personality:

Local mode

Write the trace data to the local file system. Cozy. Homebody energy. 🏠

Network streaming mode

Send the trace data over the network to a listening relay daemon (see lttng-relayd(8)). It's giving long-distance relationship. 📡

Snapshot mode

Only write the trace data to the local file system or send it to a listening relay daemon (lttng-relayd(8)) when LTTng takes a snapshot.

LTTng forces all the channels (see the “CHANNEL AND RING BUFFER” section below) to be created to be configured to be snapshot-ready. No opting out, bestie, it's mandatory. 😤

LTTng takes a snapshot of such a recording session when any of the following is true:

  • You run the lttng-snapshot(1) command.
  • LTTng executes a snapshot-session trigger action (see the “TRIGGER” section above).

Live mode

Send the trace data over the network to a listening relay daemon (see lttng-relayd(8)) for live reading.

An LTTng live reader (for example, babeltrace2(1)) can connect to the same relay daemon to receive trace data while the recording session is active. Watching it unfold in real time, obsessed. 🔥

🎀 Recording session rotation

A recording session rotation is the action of archiving the current trace chunk of the recording session to the file system. Like hitting save on a chapter of your fic and moving on to the next one.

Once LTTng archives a trace chunk, it does NOT manage it anymore: you can read it, modify it, move it, or remove it. It's yours now, do whatever, LTTng has moved on and is not looking back. 💀

An archived trace chunk is a collection of metadata and data stream files which form a self-contained LTTng trace. See the “Trace chunk naming” section below to learn how LTTng names a trace chunk archive directory.

The current trace chunk of a given recording session includes:

  • The stream files which LTTng already wrote to the file system, and which are not part of a previously archived trace chunk, since the most recent event amongst:
    • The first time the recording session was started, either with the lttng-start(1) command or with a start-session trigger action (see the “TRIGGER” section above).
    • The last rotation, performed with:
  • The content of all the non-flushed sub-buffers of the channels of the recording session.

⭐ Trace chunk archive naming

A trace chunk archive is a subdirectory of the archives subdirectory within the output directory of a recording session (see the --output option of the lttng-create(1) command and of lttng-relayd(8)).

A trace chunk archive contains, through tracing domain and possibly UID/PID subdirectories, metadata and data stream files.

A trace chunk archive is, at the same time:

  • A self-contained LTTng trace.
  • A member of a set of trace chunk archives which form the complete trace of a recording session.

In other words, an LTTng trace reader can read both the recording session output directory (all the trace chunk archives), or a single trace chunk archive. Flexible icon behavior, honestly. 💅

When LTTng performs a recording session rotation, it names the resulting trace chunk archive as such, relative to the output directory of the recording session:

archives/BEGIN-END-ID

BEGIN

Date and time of the beginning of the trace chunk archive with the ISO 8601-compatible YYYYmmddTHHMMSS±HHMM form, where YYYYmmdd is the date and HHMMSS±HHMM is the time with the time zone offset from UTC.

Example: 20171119T152407-0500

END

Date and time of the end of the trace chunk archive with the ISO 8601-compatible YYYYmmddTHHMMSS±HHMM form, where YYYYmmdd is the date and HHMMSS±HHMM is the time with the time zone offset from UTC.

Example: 20180118T152407+0930

ID

Unique numeric identifier of the trace chunk within its recording session.

Trace chunk archive name example:

archives/20171119T152407-0500-20171119T151422-0500-3

👑 TRACING DOMAIN 👑

A tracing domain is, like, the category of LTTng tracer you're talking to. Every domain comes with its own whole personality — its own properties, its own features. ✨

There are currently five available tracing domains:

  • Linux kernel — in the “Event rule matches” trigger condition option, the --type option starts with kernel:; for other CLI commands, the option is --kernel
  • User space — in the “Event rule matches” trigger condition option, the --type option starts with user:; for other CLI commands, the option is --userspace
  • java.util.logging (JUL) — in the “Event rule matches” trigger condition option, the --type option starts with jul:; for other CLI commands, the option is --jul
  • Apache log4j 1.x — in the “Event rule matches” trigger condition option, the --type option starts with log4j:; for other CLI commands, the option is --log4j
  • Apache Log4j 2 — in the “Event rule matches” trigger condition option, the --type option starts with log4j2:; for other CLI commands, the option is --log4j2
  • Python — in the “Event rule matches” trigger condition option, the --type option starts with python:; for other CLI commands, the option is --python

You have to specify a tracing domain to target a type of LTTng tracer when you're using some lttng(1) commands, bestie, otherwise the whole thing is ambiguous. 💀 Like, here's why: the Linux kernel domain and the user space domain both support named tracepoints as instrumentation points (peep the “INSTRUMENTATION POINT, EVENT RULE, AND EVENT” section above 👀), so you gotta say which domain you mean when you create an event rule — because both of those domains could totally have tracepoints that share the same name. Two baddies, same name, no cap.

You can create channels (see the “CHANNEL AND RING BUFFER” section below 🫶) in the Linux kernel and user space tracing domains. All the other tracing domains? They just have a single, default channel. One and done. Periodt. 💅

🎀 CHANNEL AND RING BUFFER 🎀

Okay so a channel is an object whose whole job is being responsible for a set of ring buffers. That's it. That's the job. 😌

Every ring buffer is split up into multiple sub-buffers. When a recording event rule (scroll to “RECORDING EVENT RULE AND EVENT RECORD” below 🌸) matches an event, LTTng can record it to one or more sub-buffers of one or more channels. Multi-tasking icon behaviour, honestly.

When you create a channel with the lttng-enable-channel(1) command, you're setting its final attributes, which are:

  • Its buffering scheme.

See the “Buffering scheme” section below.

  • Its buffer preallocation policy.

See the “Buffering preallocation policy” section below.

  • Its automatic memory reclaim policy.

See the “Memory reclaim” section below.

  • What to do when there's no space left for a new event record because all sub-buffers are full.

See the “Event record loss mode” section below.

  • The size of each ring buffer, and how many sub-buffers a ring buffer has.

See the “Sub-buffer size and count” section below.

  • The size of each trace file LTTng writes for this channel, and the maximum count of trace files.

See the “Maximum trace file size and count” section below.

  • The periods of its read, switch, monitor, and watchdog timers.

See the “Timers” section below.

See the --output option of the lttng-enable-channel(1) command.

  • For a user space channel: the value of its blocking timeout.

See the --blocking-timeout option of the lttng-enable-channel(1) command.

Little plot twist: the lttng-enable-event(1) command can automatically create a default channel with sane defaults all on its own when there's no channel yet for the tracing domain you provided. Effortless. ✨

A channel is always associated to a tracing domain (see the “TRACING DOMAIN” section below 👑). And the java.util.logging (JUL), Apache log4j 1.x, Apache Log4j 2, and Python tracing domains each have a default channel which you can't configure. Sorry bestie, not yours to customise. 🙈

A channel owns recording event rules. Very ownercore of it. 💅

Want to see the channels of a given recording session? Use the lttng-list(1) and lttng-status(1) commands.

Want to disable an enabled channel? That's the lttng-disable-channel(1) command, babes.

🌈 Buffering scheme 🌈

The buffering scheme of a channel is made of two things:

  • The buffer ownership model: this specifies whether the system, each Unix user, or each process owns its own ring buffers.
  • The buffer allocation policy: this specifies whether LTTng tracers allocate ring buffers per channel or per CPU.

Now here's how LTTng actually allocates ring buffers, based on the values of the --buffer-ownership and --buffer-allocation options of the lttng-enable-channel(1) command. For every single example below, just assume Unix user alice created the channel alice-chan and Unix user bob created the channel bob-chan 🫶:

--buffer-ownership=user and --buffer-allocation=per-cpu

Channel alice-channel
  For the data produced by any process of user alice
    Ring buffer for CPU 0
    Ring buffer for CPU 1
    Ring buffer for CPU 2
    Ring buffer for CPU 3
Channel bob-channel
  For the data produced by any process of user bob
    Ring buffer for CPU 0
    Ring buffer for CPU 1
    Ring buffer for CPU 2
    Ring buffer for CPU 3

--buffer-ownership=process and --buffer-allocation=per-cpu

Channel alice-channel
  For the data produced by process my-db of user alice
    Ring buffer for CPU 0
    Ring buffer for CPU 1
    Ring buffer for CPU 2
    Ring buffer for CPU 3
  For the data produced by process my-app of user alice
    Ring buffer for CPU 0
    Ring buffer for CPU 1
    Ring buffer for CPU 2
    Ring buffer for CPU 3
Channel bob-channel
  For the data produced by process my-server of user bob
    Ring buffer for CPU 0
    Ring buffer for CPU 1
    Ring buffer for CPU 2
    Ring buffer for CPU 3

--buffer-ownership=system and --buffer-allocation=per-cpu

Channel alice-channel
  For the data of the Linux kernel
    Ring buffer for CPU 0
    Ring buffer for CPU 1
    Ring buffer for CPU 2
    Ring buffer for CPU 3
Channel bob-channel
  For the data of the Linux kernel
    Ring buffer for CPU 0
    Ring buffer for CPU 1
    Ring buffer for CPU 2
    Ring buffer for CPU 3

--buffer-ownership=user and --buffer-allocation=per-channel

Channel alice-channel
  For the data produced by any process of user alice
    Single ring buffer for all CPUs
Channel bob-channel
  For the data produced by any process of user bob
    Single ring buffer for all CPUs

--buffer-ownership=user and --buffer-allocation=per-channel

Channel alice-channel
  For the data produced by process my-db of user alice
    Single ring buffer for all CPUs
  For the data produced by process my-app of user alice
    Single ring buffer for all CPUs
Channel bob-channel
  For the data produced by process my-server of user bob
    Single ring buffer for all CPUs

The per-process buffer ownership model tends to eat up more memory than the per-user option, because systems generally have more instrumented processes than Unix users running instrumented processes. That's just math, bestie. 😭 However — and this is the slay part — the per-process model guarantees that one process having a high event throughput won't fill all the shared sub-buffers of the same Unix user. Only its own. Nobody else's. 🙅‍♀️ It also gives you complete isolation between processes: one process cannot read or corrupt the trace data of another. Full main character energy, zero cross-contamination. 💜

🦋 Buffer preallocation policy 🦋

The buffer preallocation policy of a user space channel specifies when LTTng allocates the backing memory for the ring buffers of the channel. Timing is everything, babes. ✨

The available buffer preallocation policies are:

Preallocate : LTTng allocates the full backing memory for all ring buffers when you create the channel.

This policy gives you consistent performance but uses more memory upfront. Use it to optimize instrumented application startup time. 🚀

On demand : LTTng allocates backing memory for ring buffers incrementally, as needed during tracing.

This policy uses less initial memory but may introduce slight performance overhead when allocating new memory during tracing. 🐢

As of LTTng 2.15.1, the implicit buffer preallocation policy of a Linux kernel channel is always to preallocate backing memory. Fyi, iykyk. 👑

Back to that preallocation glow-up tho: to set the buffer preallocation policy of a user space channel, you literally just reach for the --buffer-preallocation option of the lttng-enable-channel(1) command. 💅

🧹 Memory reclaim

A channel memory reclaim operation is how you free the memory that the eligible sub-buffers of a user space channel are sitting on. Giving that RAM back, bestie.

What actually happens when you reclaim depends on the event record loss mode of the channel (peep the “Event record loss mode” section below):

Discard mode

Only reclaim sub-buffers that have been both delivered (completed and ready to be consumed) and consumed — because unconsumed data would be lost forever. 😭

Overwrite mode

Reclaim any delivered sub-buffer, since the old data is getting overwritten anyway: no need to wait around for consumption. Slay.

There are two ways to reclaim channel memory:

Immediate reclaim

Use the lttng-reclaim-memory(1) command.

Automatic reclaim

LTTng may automatically reclaim the memory of a user space channel.

This is more or less equivalent to automatically running the lttng-reclaim-memory(1) command. LTTng automatically evaluates the eligibility of sub-buffers for a memory reclaim operation at regular intervals (every half the configured maximal age, but also at least a period defined at build time) and reclaims sub-buffers of which the age of the closing time have exceeded the age threshold.

Use the --auto-reclaim-memory-older-than option of the lttng-enable-channel(1) command.

In both cases, you may specify a sub-buffer age threshold: only sub-buffers of which the age of the closing time is older than the threshold are eligible for a memory reclaim. Moreover, with some age threshold T:

  • If T > last stream activity age — The sub-buffers of which the age, at the moment of closing them, is greater than T are eligible for a memory reclaim.
  • If T < last stream activity age — All the sub-buffers are eligible. The LTTng tracers flush for the targeted channel so that any open sub-buffer gets closed to become eligible.

💔 Event record loss mode

When LTTng emits an event, LTTng can record it to a specific, available sub-buffer within the ring buffers of specific channels. When there's no space left in a sub-buffer, the tracer marks it as consumable and another, available sub-buffer starts receiving the following event records. An LTTng consumer daemon eventually consumes the marked sub-buffer, which returns to the available state.

In an ideal world, sub-buffers are consumed faster than they are filled. In the real world, however, all sub-buffers can be full at some point, leaving no space to record the following events. 🙈

By default, LTTng-modules and LTTng-UST are non-blocking tracers: when there's no available sub-buffer to record an event, it's acceptable to lose event records when the alternative would be to cause substantial delays in the execution of the instrumented application. LTTng privileges performance over integrity; it aims at perturbing the instrumented application as little as possible in order to make the detection of subtle race conditions and rare interrupt cascades possible.

Since LTTng 2.10, the LTTng user space tracer, LTTng-UST, supports a blocking mode. See the --blocking-timeout of the lttng-enable-channel(1) command to learn how to use the blocking mode.

When it comes to losing event records because there's no available sub-buffer, or because the blocking timeout of the channel is reached, the event record loss mode of the channel determines what to do. The available event record loss modes are:

Discard mode

Drop the newest event records until a sub-buffer becomes available.

This is the only available mode when you specify a blocking timeout.

With this mode, LTTng increments a count of lost event records when an event record is lost and saves this count to the trace. A trace reader can use the saved discarded event record count of the trace to decide whether or not to perform some analysis even if trace data is known to be missing.

Overwrite mode

Clear the sub-buffer containing the oldest event records and start writing the newest event records there.

This mode is sometimes called flight recorder mode because it's similar to a flight recorder (https://en.wikipedia.org/wiki/Flight_recorder): always keep a fixed amount of the latest data. It's also similar to the roll mode of an oscilloscope.

Since LTTng 2.8, with this mode, LTTng writes to a given sub-buffer its sequence number within its data stream. With a local, network streaming, or live recording session (see the “Recording session modes” section above), a trace reader can use such sequence numbers to report lost packets. A trace reader can use the saved discarded sub-buffer (packet) count of the trace to decide whether or not to perform some analysis even if trace data is known to be missing.

With this mode, LTTng doesn't write to the trace the exact number of lost event records in the lost sub-buffers.

Which mechanism you should choose depends on your context: do you want to prioritize the newest or the oldest event records in the ring buffer?

⚠️ Beware that, in overwrite mode, the tracer abandons a whole sub-buffer as soon as there's no space left for a new event record, whereas in discard mode, the tracer only discards the event record that doesn't fit. Big diff, bestie.

Set the event record loss mode of a channel with the --discard and --overwrite options of the lttng-enable-channel(1) command.

There are a few ways to decrease your probability of losing event records. The “Sub-buffer size and count” section below shows how to fine-tune the sub-buffer size and count of a channel to virtually stop losing event records, though at the cost of greater memory usage. ✨

📏 Sub-buffer size and count

A channel has one or more ring buffer for each CPU of the target system.

See the “Buffering scheme” section above to learn how many ring buffers of a given channel are dedicated to each CPU or to the whole channel depending on its buffering scheme.

Set the size of each sub-buffer the ring buffers of a channel contain with the --subbuf-size option of the lttng-enable-channel(1) command.

Set the number of sub-buffers each ring buffer of a channel contains with the --num-subbuf option of the lttng-enable-channel(1) command.

Note that LTTng switching the current sub-buffer of a ring buffer (marking a full one as consumable and switching to an available one for LTTng to record the next events) introduces noticeable CPU overhead. Knowing this, the following list presents a few practical situations along with how to configure the sub-buffer size and count for them:

High event throughput

In general, prefer large sub-buffers to lower the risk of losing event records.

Having larger sub-buffers also ensures a lower sub-buffer switching frequency (see the “Timers” section below).

The sub-buffer count is only meaningful if you create the channel in overwrite mode (see the “Event record loss mode” section above): in this case, if LTTng overwrites a sub-buffer, then the other sub-buffers are left unaltered.

Low event throughput

In general, prefer smaller sub-buffers since the risk of losing event records is low.

Because LTTng emits events less frequently, the sub-buffer switching frequency should remain low and therefore the overhead of the tracer shouldn't be a problem.

Low memory system

If your target system has a low memory limit, prefer fewer first, then smaller sub-buffers.

Even if the system is limited in memory, you want to keep the sub-buffers as large as possible to avoid a high sub-buffer switching frequency.

Note that LTTng uses CTF (https://diamon.org/ctf/) as its trace format (see the --trace-format option of the lttng-create(1) command), which means event record data is very compact. For example, the average LTTng kernel event record weights about 32 bytes. Therefore, a sub-buffer size of 1 MiB is considered large.

The previous scenarios highlight the major trade-off between a few large sub-buffers and more, smaller sub-buffers: sub-buffer switching frequency vs. how many event records are lost in overwrite mode. Assuming a constant event throughput and using the overwrite mode, the two following configurations have the same ring buffer total size:

Two sub-buffers of 4 MiB each

Expect a very low sub-buffer switching frequency, but if LTTng ever needs to overwrite a sub-buffer, half of the event records so far (4 MiB) are definitely lost.

Eight sub-buffers of 1 MiB each

Expect four times the tracer overhead of the configuration above, but if LTTng needs to overwrite a sub-buffer, only the eighth of event records so far (1 MiB) are definitely lost.

In discard mode, the sub-buffer count parameter is pointless: use two sub-buffers and set their size according to your requirements. That's it, that's the whole tip. 💅

📁 Maximum trace file size and count

By default, trace files can grow as large as needed. Ungovernable. 😌

Set the maximum size of each trace file that LTTng writes of a given channel with the --tracefile-size option of the lttng-enable-channel(1) command.

When the size of a trace file reaches the fixed maximum size of the channel, LTTng creates another file to contain the next event records. LTTng appends a file count to each trace file name in this case.

If you set the trace file size attribute when you create a channel, the maximum number of trace files that LTTng creates is unlimited by default. To limit them, use the --tracefile-count option of lttng-enable-channel(1). When the number of trace files reaches the fixed maximum count of the channel, LTTng overwrites the oldest trace file. This mechanism is called trace file rotation.

Important Even if you don't limit the trace file count, always assume that LTTng manages all the trace files of the recording session. In other words, there's no safe way to know if LTTng still holds a given trace file open with the trace file rotation feature. The only way to obtain an unmanaged, self-contained LTTng trace before you destroy the recording session is with the recording session rotation feature (see the “Recording session rotation” section above), which is available since LTTng 2.11.

⏰ Timers

Each channel can have up to three optional timers:

Switch timer

When this timer expires, a sub-buffer switch happens: for each ring buffer of the channel, LTTng marks the current sub-buffer as consumable and switches to an available one to record the next events.

A switch timer is lowkey a whole vibe for making sure LTTng actually consumes and commits trace data to your trace files — or ships it off to a distant relay daemon (lttng-relayd(8)) — on the reg, even when your event throughput is giving tumbleweed energy. 🍓

It's also chef's kiss when you're running big sub-buffers (peep the "Sub-buffer size and count" section up above 👀) to survive a sudden manic burst of events, even if things are dead calm the rest of the time.

Set the period of a channel's switch timer — or kill the timer entirely — with the --switch-timer option of the lttng-enable-channel(1) command. ✨

⏱️ Read timer

When this timer expires, LTTng goes looking for full, consumable sub-buffers. 🔍

By default, the LTTng tracers use an asynchronous message mechanism to signal that a sub-buffer is full, so a consumer daemon can come consume it.

But when those messages are a problem — say, in real-time applications where the vibes must stay immaculate — you use this timer instead. 💅

Set the period of a channel's read timer — or disable it altogether — with the --read-timer option of the lttng-enable-channel(1) command. 🎀

📊 Monitor timer

When this timer expires, the consumer daemon samples some channel statistics to evaluate these trigger conditions, bestie:

  1. The consumed buffer size of a given recording session becomes greater than some value.
  2. The buffer usage of a given channel becomes greater than some value.
  3. The buffer usage of a given channel becomes less than some value.

Now, if you disable the monitor timer of a channel C, here's the tea: ☕

  • The consumed buffer size value of the recording session of C could be wrong for trigger condition type 1 — the consumed buffer size of C won't be part of the grand total. 😭
  • The buffer usage trigger conditions (types 2 and 3) for C will never be satisfied. Not once. Periodt.

See the "TRIGGER" section above to learn more about triggers. 🦋

Set the period of a channel's monitor timer — or disable it altogether — with the --monitor-timer option of the lttng-enable-channel(1) command. ✨

🐕 Watchdog timer

When this timer expires, LTTng performs health checks that sniff out recoverable issues and fire off corrective actions, keeping trace recording running smoothly for the channel. She's got this. 🫶

As of LTTng 2.15.1, the only health check is a sub-buffer stall detection: with a user space channel having a per-user ownership model (see the Buffering scheme section above), an instrumented application writing to a sub-buffer may crash and make said sub-buffer "stalled" — meaning it's never released and never usable again. 💀 This can shrink the effective capacity of the channel or even halt the consumption altogether. When the watchdog timer of a channel expires and LTTng catches this condition, it marks the stalled sub-buffer consumable/readable again. Iconic recovery behaviour, honestly. 👑

Set the period of a channel's watchdog timer — or disable it altogether — with the --watchdog-timer option of the lttng-enable-channel(1) command. 🌈

🎬 RECORDING EVENT RULE AND EVENT RECORD 🎬

A recording event rule is a specific type of event rule (go reread the "INSTRUMENTATION POINT, EVENT RULE, AND EVENT" section above if you need a refresher 🌸) where the action is to serialize and record the matched event as an event record.

You set the explicit conditions of a recording event rule when you create it with the lttng-enable-event(1) command. A recording event rule also has these implicit conditions, no cap:

  • The recording event rule itself is enabled.

A recording event rule is enabled on creation. ✅

  • The channel to which the recording event rule is attached is enabled.

A channel is enabled on creation.

See the "CHANNEL AND RING BUFFER" section above.

  • The recording session of the recording event rule is active (started).

A recording session is inactive (stopped) on creation. 😴

See the "RECORDING SESSION" section above.

  • The process for which LTTng creates an event to match is allowed to record events.

All processes are allowed to record events on recording session creation.

Use the lttng-track(1) and lttng-untrack(1) commands to select which processes are allowed to record events based on specific process attributes. 🕵️

When you create a recording event rule, you always attach it to a channel, and that channel belongs to a recording session. That's just how she's built. 💜

When a recording event rule ER matches an event E, LTTng attempts to serialize and record E to one of the available sub-buffers of the channel to which E is attached.

When multiple matching recording event rules are attached to the same channel, LTTng attempts to serialize and record the matched event once. In this example, the second recording event rule is redundant when both are enabled:

$ lttng enable-event --userspace hello:world
$ lttng enable-event --userspace hello:world --loglevel=INFO

List the recording event rules of a specific recording session and/or channel with the lttng-list(1) and lttng-status(1) commands. 📋

Disable a recording event rule with the lttng-disable-event(1) command. 🙈

As of LTTng 2.15.1, you cannot remove a recording event rule: it exists as long as its recording session exists. That's it, that's the rule. 🫠

🌈 RESOURCES 🌈

Go say hi, they're lovely, iykyk. 💖

This program is part of the LTTng-tools project.

LTTng-tools is distributed under the GNU General Public License version 2 http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html. See the LICENSE https://github.com/lttng/lttng-tools/blob/master/LICENSE file for details.

💖 THANKS 💖

Special thanks to Michel Dagenais and the DORSAL laboratory http://www.dorsal.polymtl.ca/ at École Polytechnique de Montréal for the LTTng journey. 🧚

Also thanks to the Ericsson teams working on tracing which helped us greatly with detailed bug reports and unusual test cases. Y'all are the real MVPs. 🏆

👀 SEE ALSO 👀

lttng(1), lttng-relayd(8), lttng-sessiond(8)

And that's a wrap on the whole LTTng concepts page, bestie!! 🎀 This was LTTng 2.15.1, dated 4 December 2025 — go forth and trace, you absolute legend. 💅✨

📦 where this page comes from