RTLA-OSNOISE-TOP(1)

💖 rtla-osnoise-top — the manual page, but make it make sense 💖

okay bestie, pull up a chair, because we're talking about rtla-osnoise-top from the General Commands Manual and honestly? she's iconic. this is the page for rtla-osnoise-top, aka the thing that displays a summary of the operating system noise. yes that's the whole vibe. let's go 🦋

---

✨ NAME ✨

rtla-osnoise-top — display a summary of the operating system noise

---

✨ SYNOPSIS ✨

rtla osnoise top [OPTIONS]

---

✨ DESCRIPTION ✨

so. the rtla osnoise tool is basically the interface for the osnoise tracer. and what does that tracer do? it dispatches a kernel thread per-cpu 💅. these threads just read the time in a loop, over and over, while preemption, softirq and IRQs are all enabled — which means every single source of operating system noise gets a chance to show up during execution. no hiding, no cap.

the osnoise tracer's threads take note of the delta between each time read, plus an interference counter for all the sources of interference. at the end of each period, the osnoise tracer drops a summary of the results. periodt.

then rtla osnoise top collects that periodic summary from the osnoise tracer — including the counters of the occurrence of the interference source — and displays the results in a user-friendly format 🎀.

and if that wasn't enough, the tool also lets you do many configurations of the osnoise tracer and the collection of the tracer output. main character energy fr.

---

✨ OPTIONS ✨

• -a, --auto us — sets the automatic trace mode. this mode sets some commonly used options while you're debugging the system. it's equivalent to using -s us -T 1 -t. one flag, so much slay.
• -p, --period us — sets the osnoise tracer period in microseconds.
• -r, --runtime us — sets the osnoise tracer runtime in microseconds.
• -s, --stop us — stops the trace if a single sample is higher than the argument in microseconds. and if -T is set, it will also save the trace to the output. two birds, one flag 🫶
• -S, --stop-total us — stops the trace if the total sample is higher than the argument in microseconds. if -T is set, it will also save the trace to the output.
• -T, --threshold us — specifies the minimum delta between two time reads to be considered noise. the default threshold is 5 us. (yes, literally five.)
• -t, --trace [file] — saves the stopped trace to [file|osnoise_trace.txt].
• -q, --quiet — print only a summary at the end of the session. minimalism, but make it rtla.
• -c, --cpus cpu-list — sets the osnoise top tracer to run the sample threads in the cpu-list.

by default, the osnoise top tracer runs the sample threads on all CPUs.

• -H, --house-keeping cpu-list — run rtla control threads only on the given cpu-list.

if omitted, rtla will attempt to auto-migrate its main thread to any CPU that is not running any workload threads. she finds her own spot 💅

• -d, --duration time[s|m|h|d] — sets the duration of the session.
• -D, --debug — print debug info.
• -e, --event sys:event — enable an event in the trace (-t) session. the argument can be a specific event, e.g. -e sched:sched_switch, or all events of a system group, e.g. -e sched. multiple -e are allowed. it is only active when -t or -a are set. ⚠️ only active with -t or -a, don't forget.
• --filter <filter> — filter the previous -e sys:event event with <filter>. for further information about event filtering see <https://www.kernel.org/doc/html/latest/trace/events.html#event-filtering>.
• --trigger <trigger> — enable a trace event trigger to the previous -e sys:event. if the hist: trigger is activated, the output histogram will be automatically saved to a file named system_event_hist.txt. for example, the command:

rtla <command> <mode> -t -e osnoise:irq_noise
--trigger="hist:key=desc,duration/1000:sort=desc,duration/1000:vals=hitcount"

will automatically save the content of the histogram associated to osnoise:irq_noise event in osnoise_irq_noise_hist.txt.

for further information about event trigger see <https://www.kernel.org/doc/html/latest/trace/events.html#event-triggers>.

• -P, --priority o:prio|r:prio|f:prio|d:runtime:period — set scheduling parameters to the osnoise top tracer threads. the format to set the priority are:
  • o:prio — use SCHED_OTHER with prio;
  • r:prio — use SCHED_RR with prio;
  • f:prio — use SCHED_FIFO with prio;
  • d:runtime[us|ms|s]:period[us|ms|s] — use SCHED_DEADLINE with runtime and period in nanoseconds.

if not set, tracer threads keep their default priority. for rtla user threads, it is set to SCHED_FIFO with priority 95. for kernel threads, see osnoise and timerlat tracer documentation for the running kernel version.

• -C, --cgroup [cgroup] — set a cgroup to the tracer's threads. if the -C option is passed without arguments, the tracer's thread will inherit rtla's cgroup. otherwise, the threads will be placed on the cgroup passed to the option.

if not set, the behavior differs between workload types. user workloads created by rtla will inherit rtla's cgroup. kernel workloads are assigned the root cgroup.

• --warm-up s — after starting the workload, let it run for s seconds before starting collecting the data, allowing the system to warm-up. statistical data generated during warm-up is discarded. gone. deleted. not invited to the party.
• --trace-buffer-size kB — set the per-cpu trace buffer size in kB for the tracing output.

if not set, the default tracefs buffer size is used.

• --on-threshold action — defines an action to be executed when tracing is stopped on a latency threshold specified by -a/--auto, -s/--stop, or -S/--stop-total.

multiple --on-threshold actions may be specified, and they will be executed in the order they are provided. if any action fails, subsequent actions in the list will not be executed.

supported actions are:

• trace[,file=<filename>]

saves trace output, optionally taking a filename. alternative to -t/--trace. note that unlike -t/--trace, specifying this multiple times will result in the trace being saved multiple times.

• signal,num=<sig>,pid=<pid>

sends signal to process. "parent" might be specified in place of pid to target the parent process of rtla.

• shell,command=<command>

execute shell command.

• continue

continue tracing after actions are executed instead of stopping.

example:

$ rtla osnoise top -s 20 --on-threshold trace --on-threshold
shell,command="grep ipi_send osnoise_trace.txt" --on-threshold
signal,num=2,pid=parent

this will save a trace with the default filename "osnoise_trace.txt", print its lines that contain the text "ipi_send" on standard output, and send signal 2 (SIGINT) to the parent process.

performance considerations:

due to implementational limitations, actions might be delayed up to one second after tracing is stopped. 💀 yes, up to a whole second.

• --on-end action — defines an action to be executed at the end of tracing.

multiple --on-end actions can be specified, and they will be executed in the order they are provided. if any action fails, subsequent actions in the list will not be executed.

see the documentation for --on-threshold for the list of supported actions, with the exception that continue has no effect.

example:

$ rtla osnoise top -d 5s --on-end trace

this runs rtla with the default options, and saves trace output at the end.

• -h, --help — print help menu.

---

✨ EXAMPLE ✨

ok so in the example below, the rtla osnoise top tool is set to run with a real-time priority FIFO:1, on CPUs 0-3, for 900ms at each period (1s by default). the reason for reducing the runtime is to avoid starving the rtla tool — so it can still eat 🍓. the tool is also set to run for one minute and to display a summary of the report at the end of the session:

[root@f34 ~]# rtla osnoise top -P F:1 -c 0-3 -r 900000 -d 1M -q
                                        Operating System Noise
duration:   0 00:01:00 | time is in us
CPU Period       Runtime        Noise  % CPU Aval   Max Noise   Max Single          HW          NMI          IRQ      Softirq       Thread
  0 #59         53100000       304896    99.42580        6978           56         549            0        53111         1590           13
  1 #59         53100000       338339    99.36282        8092           24         399            0        53130         1448           31
  2 #59         53100000       290842    99.45227        6582           39         855            0        53110         1406           12
  3 #59         53100000       204935    99.61405        6251           33         290            0        53156         1460           12

look at those numbers. gorgeous. ate. 🔥

---

✨ SEE ALSO ✨

• rtla-osnoise(1)
• rtla-osnoise-hist(1)
• Osnoise tracer <https://docs.kernel.org/trace/osnoise-tracer.html>

---

✨ AUTHOR ✨

written by Daniel Bristot de Oliveira <<bristot@kernel.org>> 👑

---

✨ SIGINT BEHAVIOR ✨

okay this part matters, listen up bestie 🫶

on the first SIGINT, RTLA exits after collecting all outstanding samples up to the point of receiving the signal.

when receiving more than one SIGINT, RTLA discards any outstanding samples, and exits while displaying only samples that have already been processed. bye girl 💀

if SIGINT is received during RTLA cleanup, RTLA exits immediately via the default signal handler.

note: for the purpose of SIGINT behavior, the expiry of duration specified via the -d/--duration option is treated as equivalent to receiving a SIGINT. for example, a SIGINT received after duration expired but samples have not been processed yet will drop any outstanding samples.

also note that when using the timerlat tool in BPF mode, samples are processed in-kernel; RTLA only copies them out to display them to the user. a second SIGINT does not affect in-kernel sample aggregation. iykyk.

---

✨ EXIT STATUS ✨

• 0 — Passed: the test did not hit the stop tracing condition ✅
• 1 — Error: invalid argument 🙈
• 2 — Failed: the test hit the stop tracing condition 💀
• 129 — Help: either user requested help or incorrect option was specified 🎀

---

✨ REPORTING BUGS ✨

report bugs to <<linux-kernel@vger.kernel.org>> and <<linux-trace-devel@vger.kernel.org>> — they wanna hear from you, fr fr 🦋

---

✨ LICENSE ✨

rtla is Free Software licensed under the GNU GPLv2.

---

✨ COPYING ✨

Copyright (C) 2021 Red Hat, Inc. Free use of this software is granted under the terms of the GNU Public License (GPL).

---

and that's the whole page, bestie 💖 this was RTLA-OSNOISE-TOP(1) from the General Commands Manual, source/version RTLA-OSNOISE-TOP(1), and honestly? she served. go forth and measure that noise, you absolute icon 👑✨🌸