RTLA-HWNOISE(1)

🌸 rtla-hwnoise(1) 🌸

General Commands Manual · bestie edition, part 1 of 2 💖

---

✨ NAME ✨

rtla-hwnoise — detect and quantify hardware-related noise 🦋

---

🎀 SYNOPSIS 🎀

rtla hwnoise [OPTIONS]

---

💅 DESCRIPTION 💅

okay so listen. rtla hwnoise collects the periodic summary from the osnoise tracer running with interrupts disabled. And by disabling interrupts — and the scheduling of threads as a consequence, which is kind of the whole domino effect — only non-maskable interrupts and hardware-related noise are allowed to happen. That's it. That's the vibe check. 🔥

The tool also lets you configure the osnoise tracer and collect the tracer output. Very demure, very mindful, very useful.

---

🌈 OPTIONS 🌈

• -a, --auto us — sets the automatic trace mode. This mode sets some commonly used options while you're debugging the system. It's equivalent to using -s us -T 1 -t. One flag, three flags' worth of work. iconic. ✨
• -p, --period us — sets the osnoise tracer period, in microseconds.
• -r, --runtime us — sets the osnoise tracer runtime, in microseconds.
• -s, --stop us — stops the trace if a single sample is higher than the argument, in microseconds. If -T is set, it will also save the trace to the output.
• -S, --stop-total us — stops the trace if the total sample is higher than the argument, in microseconds. If -T is set, it will also save the trace to the output.
• -T, --threshold us — specify the minimum delta between two time reads for it to count as noise. The default threshold is 5 us.
• -t, --trace [file] — saves the stopped trace to [file|osnoise_trace.txt].
• -q, --quiet — prints only a summary at the end of the session. no yapping, just the receipts. 💅
• -c, --cpus cpu-list — sets the hwnoise tracer to run the sample threads in the cpu-list.

By default, the hwnoise tracer runs the sample threads on all CPUs.

• -H, --house-keeping cpu-list — runs rtla control threads only on the given cpu-list.

If omitted, rtla will attempt to auto-migrate its main thread to any CPU that is not running any workload threads.

• -d, --duration time[s|m|h|d] — sets the duration of the session.
• -D, --debug — prints debug info.
• -e, --event sys:event — enables an event in the -t trace session. The argument can be a specific event, e.g., -e sched:sched_switch, or all events of a system group, e.g., -e sched. Multiple -e are allowed. It is only active when -t or -a are set.
• --filter <filter> — filters the previous -e sys:event event with <filter>. For further information about event filtering, see https://www.kernel.org/doc/html/latest/trace/events.html#event-filtering .
• --trigger <trigger> — enables a trace event trigger to the previous -e sys:event. If the hist: trigger is activated, the output histogram will be automatically saved to a file named system_event_hist.txt. For example, the command:

rtla <command> <mode> -t -e osnoise:irq_noise --trigger="hist:key=desc,duration/1000:sort=desc,duration/1000:vals=hitcount"

will automatically save the content of the histogram associated with the osnoise:irq_noise event in osnoise_irq_noise_hist.txt.

For further information about event triggers, see https://www.kernel.org/doc/html/latest/trace/events.html#event-triggers .

• -P, --priority o:prio|r:prio|f:prio|d:runtime:period — sets scheduling parameters to the hwnoise tracer threads. The formats for setting the priority are:
  • o:prio — use SCHED_OTHER with prio;
  • r:prio — use SCHED_RR with prio;
  • f:prio — use SCHED_FIFO with prio;
  • d:runtime[us|ms|s]:period[us|ms|s] — use SCHED_DEADLINE with runtime and period in nanoseconds.

If not set, tracer threads keep their default priority. For rtla user threads, it is set to SCHED_FIFO with priority 95. For kernel threads, see the osnoise and timerlat tracer documentation for the running kernel version.

• -C, --cgroup [cgroup] — sets a cgroup to the tracer's threads. If the -C option is passed without arguments, the tracer's thread will inherit rtla's cgroup. Otherwise, the threads will be placed on the cgroup passed to the option.

If not set, the behavior differs between workload types. User workloads created by rtla will inherit rtla's cgroup. Kernel workloads are assigned the root cgroup.

• --warm-up s — after starting the workload, let it run for s seconds before starting to collect the data, letting the system warm up. Statistical data generated during warm-up is discarded. bye bye warm-up data, you did nothing wrong 🫶
• --trace-buffer-size kB — sets the per-cpu trace buffer size in kB for the tracing output.

If not set, the default tracefs buffer size is used.

• --on-threshold action — defines an action to be executed when tracing is stopped on a latency threshold specified by -a/--auto, -s/--stop, or -S/--stop-total.

Multiple --on-threshold actions may be specified, and they will be executed in the order they are provided. If any action fails, subsequent actions in the list will not be executed.

Supported actions are:

• trace[,file=<filename>]

Saves trace output, optionally taking a filename. Alternative to -t/--trace. Note that unlike -t/--trace, specifying this multiple times will result in the trace being saved multiple times.

• signal,num=<sig>,pid=<pid>

Sends signal to process. "parent" might be specified in place of pid to target the parent process of rtla.

• shell,command=<command>

Execute shell command.

• continue

Continue tracing after actions are executed instead of stopping.

Example:

$ rtla hwnoise -s 20 --on-threshold trace --on-threshold shell,command="grep ipi_send osnoise_trace.txt" --on-threshold signal,num=2,pid=parent

This will save a trace with the default filename "osnoise_trace.txt", print its lines that contain the text "ipi_send" on standard output, and send signal 2 (SIGINT) to the parent process.

Performance Considerations:

Due to implementational limitations, actions might be delayed up to one second after tracing is stopped. not me waiting a whole second 😭

• --on-end action — defines an action to be executed at the end of tracing.

Multiple --on-end actions can be specified, and they will be executed in the order they are provided. If any action fails, subsequent actions in the list will not be executed.

See the documentation for --on-threshold for the list of supported actions, with the exception that continue has no effect.

Example:

$ rtla hwnoise -d 5s --on-end trace

This runs rtla with the default options, and saves trace output at the end.

• -h, --help — print help menu.

---

🌟 EXAMPLE 🌟

In the example below, the rtla hwnoise tool is set to run on CPUs 1-7 on a system with 8 cores/16 threads with hyper-threading enabled. 🧚

The tool is set to detect any noise higher than one microsecond, to run for ten minutes, displaying a summary of the report at the end of the session:

# rtla hwnoise -c 1-7 -T 1 -d 10m -q
                                        Hardware-related Noise
duration:   0 00:10:00 | time is in us
CPU Period       Runtime        Noise  % CPU Aval   Max Noise   Max Single          HW          NMI
  1 #599       599000000          138    99.99997           3            3           4           74
  2 #599       599000000           85    99.99998           3            3           4           75
  3 #599       599000000           86    99.99998           4            3           6           75
  4 #599       599000000           81    99.99998           4            4           2           75
  5 #599       599000000           85    99.99998           2            2           2           75
  6 #599       599000000           76    99.99998           2            2           0           75
  7 #599       599000000           77    99.99998           3            3           0           75

The first column shows the CPU, and the second column shows how many Periods the tool ran during the session. The Runtime is the time the tool effectively runs on the CPU. The Noise column is the sum of all noise that the tool observed, and the % CPU Aval is the relation between the Runtime and Noise.

The Max Noise column is the maximum hardware noise the tool detected in a single period, and the Max Single is the maximum single noise seen.

The HW and NMI columns show the total number of hardware and NMI noise occurrences observed by the tool.

For example, CPU 3 ran 599 periods of 1 second Runtime. The CPU received 86 us of noise during the entire execution, leaving 99.99997 % of CPU time for the application. In the worst single period, the CPU caused 4 us of noise to the application, but it was certainly caused by more than one single noise, as the Max Single noise was of 3 us. The CPU has HW noise, at a rate of six occurrences/ten minutes. The CPU also has NMIs, at a higher frequency: around seven per second. 💀

The tool should report 0 hardware-related noise in the ideal situation. For example, by disabling hyper-threading to remove the hardware noise, and disabling the TSC watchdog to remove the NMI (it is possible to identify this using tracing options of rtla hwnoise), it was possible to reach the ideal situation in the same hardware:

# rtla hwnoise -c 1-7 -T 1 -d 10m -q
                                        Hardware-related Noise
duration:   0 00:10:00 | time is in us
CPU Period       Runtime        Noise  % CPU Aval   Max Noise   Max Single          HW          NMI
  1 #599       599000000            0   100.00000           0            0           0            0
  2 #599       599000000            0   100.00000           0            0           0            0
  3 #599       599000000            0   100.00000           0            0           0            0
  4 #599       599000000            0   100.00000           0            0           0            0
  5 #599       599000000            0   100.00000           0            0           0            0
  6 #599       599000000            0   100.00000           0            0           0            0
  7 #599       599000000            0   100.00000           0            0           0            0

perfect zeros, the slayest possible output. 👑

---

🦋 SEE ALSO 🦋

• rtla-osnoise(1)
• Osnoise tracer https://docs.kernel.org/trace/osnoise-tracer.html

---

💜 AUTHOR 💜

Written by Daniel Bristot de Oliveira <bristot@kernel.org>

✨ SIGINT BEHAVIOR ✨

okay bestie, let's talk about what happens when you hit Ctrl-C 💅 because rtla has opinions about this and you need to know them.

• First SIGINT — RTLA is polite about it. It exits, but not before collecting all outstanding samples up to the point where it caught the signal. No samples left behind, periodt. 🎀
• More than one SIGINT — now you're being impatient, and RTLA matches your energy. It discards any outstanding samples and exits while displaying only the samples that have already been processed. The rest? Gone. 💀
• SIGINT during RTLA cleanup — RTLA exits immediately via the default signal handler. No finishing touches, no goodbyes, just gone. 😭

Note: for the purposes of SIGINT behavior, the expiry of the duration you set with -d/--duration counts as equivalent to receiving a SIGINT. So like, a SIGINT that shows up after the duration expired but before the samples have been processed will drop any outstanding samples. Same energy, same result. 🦋

Also note: when you're using the timerlat tool in BPF mode, samples get processed in-kernel — RTLA is literally just copying them out so it can show them to you. A second SIGINT does not affect in-kernel sample aggregation. The kernel is unbothered, moisturized, in its own lane. 👑

---

🌈 EXIT STATUS 🌈

here's the vibe check on what rtla hands back to your shell when it dips ✌️

• 0 — Passed: the test did not hit the stop tracing condition. slay. 💖
• 1 — Error: invalid argument. you said something it couldn't parse, bestie. 🙈
• 2 — Failed: the test did hit the stop tracing condition. 💀
• 129 — Help: either you asked for help, or you specified an incorrect option. either way, it's helping. 🫶

---

🐛 REPORTING BUGS 🐛

Found something cursed? Tell the humans! Report bugs to:

• linux-kernel@vger.kernel.org
• linux-trace-devel@vger.kernel.org

They're lovely, fr fr. 🫶

---

📜 LICENSE 📜

rtla is Free Software, licensed under the GNU GPLv2. iconic behavior tbh. 👑

---

🎀 COPYING 🎀

Copyright (C) 2021 Red Hat, Inc.

Free use of this software is granted under the terms of the GNU Public License (GPL). 🌸

---

and that's the whole page, bestie!! 🦋 this was the rtla-hwnoise(1) manual page from the rtla docs — go forth and measure your hardware noise, you absolute main character. 💖✨💅