SPAUSEDD(8)

🌸 SPAUSEDD 🌸 (section 8, from the System Manager's Manual)

okay so bestie, gather round, because we're doing spausedd(8) today — a section 8 admin utility, from the System Manager's Manual, and honestly it's giving "quietly keeping your whole machine honest" 💅

---

✨ NAME ✨

spausedd — the utility that detects and logs scheduler pause 👑

---

🦋 SYNOPSIS 🦋

spausedd [-dDfhp] [-m steal_threshold] [-P mode] [-t timeout]

---

💖 DESCRIPTION 💖

so what does spausedd actually do? bestie, it detects and logs scheduler pause — meaning those moments when a process should have been scheduled but literally wasn't. not me getting ghosted by the CPU scheduler 😭

it can also look at steal time (that's the time spent in other operating systems when you're running in a virtualized environment), so it's to some extend able to tell you whether the problem is on the VM side or the host side. iconic diagnostic energy fr fr.

spausedd can read steal time info either from the kernel or, if it was compiled in, also use VMGuestLib. ✨

internally, spausedd works like this pseudocode (copying it exactly because code is sacred):

repeat:
    store current monotonic time
    store current steal time
    sleep for (timeout / 3)
    set time_diff to (current monotonic time - stored monotonic time)
    if time_diff > timeout:
        display error
        set steal_time_diff to (current steal time - stored steal time)
        if (steal_time_diff / time_diff) * 100 > steal_threshold:
            display steal time error

and the spausedd arguments are as follows (one bullet each, no skipping, that's the rule 🫶):

• -d — display debug messages. specify it twice to also display trace messages 👀
• -D — run on background (daemonize) 🌙
• -f — run on foreground (do not demonize — this is the default) ☀️
• -h — show help 🙋
• -p — do not set RR scheduler 🚫
• -m steal_threshold — set steal threshold percent. default is 10 if kernel information is used, and 100 if VMGuestLib is used 📊
• -P mode — set mode of moving process to root cgroup. default is auto, which first checks if setting of RR scheduler is enabled. if so, it tries to set RR scheduler. if this fails, the process is moved to root cgroup and the set of RR scheduler is retried. another options are on, when process is always moved to root cgroup, and off, which makes spausedd never try to move pid into root cgroup.

it's worth noting that currently (May 3 2021) cgroup v2 doesn't yet support control of realtime processes and, for systems with CONFIG_RT_GROUP_SCHED kernel option enabled, the cpu controller can only be enabled when all RT processes are in the root cgroup. so when moving to root cgroup is disabled and used together with systemd, it may be impossible to make systemd options like CPUQuota working correctly until spausedd is stopped. also when moving to root cgroup is used together with cgroup v2 and systemd, it makes impossible (most of the time) for journald to add systemd specific metadata (most importantly _SYSTEMD_UNIT) properly, because spausedd is moved out of cgroup created by systemd. this means it is not possible to filter spausedd logged messages based on these metadata (for example using -u or _SYSTEMD_UNIT=UNIT pattern) and also running systemctl status doesn't display (all) spausedd log messages. problem is even worse because journald caches pid for some time (approx. 5 sec) so initial spausedd messages have correct metadata 💀

• -t timeout — set timeout value in milliseconds (default 200) ⏱️

and one more lil thing, because it matters: if spausedd receives a SIGUSR1 signal, the current statistics are show 📈

---

🎀 EXAMPLES 🎀

to generate CPU load, yes(1) together with chrt(1) is used in the following examples:

chrt -r 99 yes >/dev/null &

if chrt fails it may help to use cgexec(1) like:

cgexec -g cpu:/ chrt -r 99 yes >/dev/null &

first example is a physical or virtual machine with 4 CPU threads, so yes(1) was executed 4 times. in a while spausedd should start logging messages similar to:

Mar 20 15:01:54 spausedd: Running main poll loop with maximum
timeout 200 and steal threshold 10%
Mar 20 15:02:15 spausedd: Not scheduled for 0.2089s (threshold is
0.2000s), steal time is 0.0000s (0.00%)
Mar 20 15:02:16 spausedd: Not scheduled for 0.2258s (threshold is
0.2000s), steal time is 0.0000s (0.00%)
...

this means that spausedd didn't got time to run for longer time than default timeout. it's also visible that steal time was 0% so spausedd is running ether on physical machine or VM where host machine is not overloaded (VM was scheduled on time) ✨

second example is a host machine with 2 CPU threads running one VM. the VM is running an instance of spausedd. two instancies of yes(1) was executed on the host machine. after a while spausedd should start logging messages similar to:

Mar 20 15:08:20 spausedd: Not scheduled for 0.9598s (threshold is
0.2000s), steal time is 0.7900s (82.31%)
Mar 20 15:08:20 spausedd: Steal time is > 10.0%, this is usually
because of overloaded host machine
...

this means that spausedd didn't got the time to run for almost one second 😳 also because steal time is high, it means that spausedd was not scheduled because the VM wasn't scheduled by the host machine.

---

🔥 DIAGNOSTICS 🔥

the spausedd utility exits 0 on success, and >0 if an error occurs 🫡

---

👑 AUTHORS 👑

the spausedd utility was written by Jan Friesse <jfriesse@redhat.com> 💜

---

💀 BUGS 💀

• OS is not updating steal time as often as monotonic clock. this means that steal time difference can be (and very often is) bigger than monotonic clock difference, so steal time percentage can be bigger than 100%. it's happening very often for highly overloaded host machine when spausedd is called with small timeout. this problem is even bigger when VMGuestLib is used. 🙈
• VMGuestLib seems to randomly block. 🧊

---

okay that's the whole page bestie, you ate it 🍓 this one comes from Debian and it's dated Jul 15, 2021 — go forth and catch those scheduler pauses, main character energy only 💅✨🌈