RTLA-HWNOISE(1)

📦 version 6.12.115, in debian trixie

💖 RTLA-HWNOISE 💖

okay bestie, welcome to rtla-hwnoise from the General Commands Manual — this is the page for detecting and quantifying hardware-related noise, and we're gonna get into it. 🦋

✨ NAME ✨

rtla-hwnoise — Detect and quantify hardware-related noise

🌸 SYNOPSIS 🌸

rtla hwnoise [OPTIONS]

💅 DESCRIPTION 💅

So like, rtla hwnoise grabs the periodic summary from the osnoise tracer while it's running with interrupts disabled. And the way that works is: once interrupts are off, thread scheduling gets turned off too as a consequence, so literally the only things that can still interrupt you are non-maskable interrupts and hardware-related noise. That's it. That's the whole vibe. 👑

The tool also lets you configure the osnoise tracer, and it collects the tracer output for you. Two-for-one, no cap. ✨

🎀 OPTIONS 🎀

  • -a, --auto us — turns on automatic trace mode. This mode just sets a bunch of commonly used options for you while you're debugging the system. It's equivalent to using -s us -T 1 -t. slay, less typing. 💅
  • -p, --period us — sets the osnoise tracer period, in microseconds.
  • -r, --runtime us — sets the osnoise tracer runtime, in microseconds.
  • -s, --stop us — stops the trace if a single sample goes higher than the argument (in microseconds). And if -T is set, it'll also save the trace to the output. 🫶
  • -S, --stop-total us — stops the trace if the total sample goes higher than the argument (in microseconds). If -T is set, it also saves the trace to the output.
  • -T, --threshold us — the minimum delta between two time reads that counts as noise. Default threshold is 5 us.
  • -t, --trace [file] — saves the stopped trace to [file|osnoise_trace.txt].
  • -q, --quiet — only prints a summary at the end of the session. minimal. iconic. 😌
  • -c, --cpus cpu-list — tells the osnoise tracer to run the sample threads on the cpu-list.
  • -H, --house-keeping cpu-list — runs rtla's control threads only on the given cpu-list.
  • -d, --duration time[s|m|h|d] — sets how long the session runs for.
  • -D, --debug — prints debug info.
  • -e, --event sys:event — enables an event in the trace (-t) session. The argument can be a specific event, e.g. -e sched:sched_switch, or all events of a system group, e.g. -e sched. Multiple -e are allowed. Heads up: it's only active when -t or -a are set. ⭐
  • --filter <filter> — filters the previous -e sys:event event with <filter>. For more info about event filtering, see https://www.kernel.org/doc/html/latest/trace/events.html#event-filtering.
  • --trigger <trigger> — enables a trace event trigger to the previous -e sys:event. If the hist: trigger is activated, the output histogram gets automatically saved to a file named system_event_hist.txt. For example, this command:
rtla <command> <mode> -t -e osnoise:irq_noise
--trigger="hist:key=desc,duration/1000:sort=desc,duration/1000:vals=hitcount"

will automatically save the content of the histogram associated to the osnoise:irq_noise event in osnoise_irq_noise_hist.txt.

For more info about event triggers, see https://www.kernel.org/doc/html/latest/trace/events.html#event-triggers.

  • -P, --priority o:prio|r:prio|f:prio|d:runtime:period — sets scheduling parameters for the osnoise tracer threads. The formats for setting the priority are:
    • o:prio — use SCHED_OTHER with prio;
    • r:prio — use SCHED_RR with prio;
    • f:prio — use SCHED_FIFO with prio;
    • d:runtime[us|ms|s]:period[us|ms|s] — use SCHED_DEADLINE with runtime and period in nanoseconds.
  • -C, --cgroup[=cgroup] — sets a cgroup for the tracer's threads. If you pass -C with no arguments, the tracer's thread inherits rtla's cgroup. Otherwise, the threads get placed on the cgroup you passed to the option.
  • --warm-up s — after starting the workload, let it run for s seconds before you start collecting data, so the system can warm up. Any statistical data generated during warm-up is discarded. 🧚
  • --trace-buffer-size kB — sets the per-cpu trace buffer size, in kB, for the tracing output.
  • -h, --help — prints the help menu. 🫶

🔥 EXAMPLE 🔥

In the example below, the rtla hwnoise tool is set to run on CPUs 1-7, on a system with 8 cores / 16 threads and hyper-threading enabled. 👑

The tool is set to detect any noise higher than one microsecond, run for ten minutes, and display a summary of the report at the end of the session:

# rtla hwnoise -c 1-7 -T 1 -d 10m -q
                                        Hardware-related Noise
duration:   0 00:10:00 | time is in us
CPU Period       Runtime        Noise  % CPU Aval   Max Noise   Max Single          HW          NMI
  1 #599       599000000          138    99.99997           3            3           4           74
  2 #599       599000000           85    99.99998           3            3           4           75
  3 #599       599000000           86    99.99998           4            3           6           75
  4 #599       599000000           81    99.99998           4            4           2           75
  5 #599       599000000           85    99.99998           2            2           2           75
  6 #599       599000000           76    99.99998           2            2           0           75
  7 #599       599000000           77    99.99998           3            3           0           75

The first column shows the CPU, and the second column shows how many Periods the tool ran during the session. Runtime is the time the tool effectively runs on the CPU. The Noise column is the sum of all the noise the tool observed, and % CPU Aval is the relationship between the Runtime and the Noise. ✨

The Max Noise column is the maximum hardware noise the tool detected in a single period, and Max Single is the maximum single noise seen.

The HW and NMI columns show the total number of hardware and NMI noise occurrences observed by the tool. 💜

For example, CPU 3 ran 599 periods of 1 second Runtime. The CPU received 86 us of noise during the entire execution, leaving 99.99997 % of CPU time for the application. In the worst single period, the CPU caused 4 us of noise to the application, but it was certainly caused by more than one single noise, since the Max Single noise was 3 us. The CPU has HW noise, at a rate of six occurrences/ten minutes. The CPU also has NMIs, at a higher frequency: around seven per second.

The tool should report 0 hardware-related noise in the ideal situation. For example, by disabling hyper-threading to remove the hardware noise, and disabling the TSC watchdog to remove the NMI (you can identify this using the tracing options of rtla hwnoise), it was possible to reach the ideal situation on the same hardware:

# rtla hwnoise -c 1-7 -T 1 -d 10m -q
                                        Hardware-related Noise
duration:   0 00:10:00 | time is in us
CPU Period       Runtime        Noise  % CPU Aval   Max Noise   Max Single          HW          NMI
  1 #599       599000000            0   100.00000           0            0           0            0
  2 #599       599000000            0   100.00000           0            0           0            0
  3 #599       599000000            0   100.00000           0            0           0            0
  4 #599       599000000            0   100.00000           0            0           0            0
  5 #599       599000000            0   100.00000           0            0           0            0
  6 #599       599000000            0   100.00000           0            0           0            0
  7 #599       599000000            0   100.00000           0            0           0            0

🌈 SEE ALSO 🌈

💜 AUTHOR 💜

Written by Daniel Bristot de Oliveira <bristot@kernel.org>

🙈 REPORTING BUGS 🙈

Report bugs to <linux-kernel@vger.kernel.org> and <linux-trace-devel@vger.kernel.org>

👑 LICENSE 👑

rtla is Free Software licensed under the GNU GPLv2.

🍓 COPYING 🍓

Copyright (C) 2021 Red Hat, Inc. Free use of this software is granted under the terms of the GNU Public License (GPL).

and that's a wrap bestie 🎀 this was RTLA-HWNOISE(1) from the General Commands Manual — go forth and find literally zero hardware noise, you absolute icon. 💖✨🦋

📦 where this page comes from