SYSTEMTAP(8)

💖 SYSTEMTAP(8) 💖

okay bestie, gather round, we're doing the SystemTap initscript page from the System Manager's Manual and it's a whole journey. sit down, grab a strawberry 🍓, and let's get into it.

---

✨ NAME ✨

systemtap-service — the SystemTap initscript and systemd service, aka the thing that runs your scripts as a service and lets you boss them around individually. slay.

---

✨ SYNOPSIS ✨

systemtap-service COMMAND [OPTIONS] [SCRIPT...]

service systemtap COMMAND [OPTIONS] [SCRIPT...]

two whole ways to summon it, periodt.

---

✨ DESCRIPTION ✨

so the SystemTap initscript exists to give you a way to run scripts as a service and control them one by one, easily. like, individually. no cap. scripts can be configured to start when you manually ask them to, or during system startup. and on dracut-based systems?? bestie. you can even integrate scripts into the initramfs and have them start during early-boot. it's giving overachiever 💅

you can invoke the initscript by hand with the systemtap-service command. and on systemd-based systems, it's controlled by systemctl with the service file systemtap.service. iconic behavior.

there are various parameters and options available to modify global behaviour, plus script behaviour. you can also set up dependencies between scripts so that starting one starts others. the way that works is honestly so satisfying.

the config file of the initscript lives at /etc/systemtap/config. what parameters are acceptable there? all detailed in the GLOBAL PARAMETERS section below, babes.

scripts go in the /etc/systemtap/script.d directory and they must have a .stp extension. but when you're referring to them on the command-line, there is no need to include the .stp extension. ngl that's a nice little quality of life thing. script names can only contain alphanumeric characters (and '_') and must not start with a number. rules are rules 🎀 the scripts directory can be changed by setting the SCRIPT_PATH parameter in the configuration file.

---

✨ COMMANDS ✨

one of the commands below must be specified. pick one, bestie, you can't have all of them:

• start — start *SCRIPT*s. if you don't specify any scripts, it starts the scripts specified by the DEFAULT_START configuration. and if DEFAULT_START isn't set, it starts all scripts in the script directory. for scripts that are already started, the command is just ignored (no drama). the command will fail if the scripts fail to start — see also the PASSALL configuration.

if the AUTOCOMPILE configuration is on, the command will try to compile or update the specified scripts when one of the below conditions is true:

• the compiled cache file does not exist.
• the mtime (modified timestamp) of the original script file is newer than the time of the compiled script cache.
• the specified stap options used to compile the script have been changed (see also the SCRIPT PARAMETERS section).
• the result of uname -a has been changed.

• stop — stop *SCRIPT*s. if you don't specify any scripts, it stops all running scripts. for scripts already stopped, the command is ignored. the command will fail if the scripts fail to stop — see also the PASSALL configuration.
• restart — stop and start *SCRIPT*s. the classic.
• status — show the state of *SCRIPT*s and their dependencies. a vibe check, essentially 🦋
• compile — compile *SCRIPT*s but do not start them. if the scripts have already been compiled, it prompts for confirmation before overwriting the cache. compiles for the current kernel, or the kernel release specified by the -r option.
• onboot — make *SCRIPT*s part of the initramfs so they get started earlier during the boot process. this command is only available on dracut-based systems. if you don't specify any scripts, it creates a normal initramfs devoid of any SystemTap files. (clean slate energy.)

the initramfs is created for the current kernel, or the kernel release specified by the -r option. the path of the created initramfs defaults to /boot/initramfs-KVER.img, where KVER is the output of uname -r. the bootloader is also updated (using new-kernel-pkg(8)) to make the kernel entry use the new initramfs file. use the -o option to specify a different path — and heads up, the bootloader will not be updated in that case.

if the output file already exists, it is overwritten, unless the -b switch is given, in which case the file is appended .bak rather than overwritten. however, if there's already a .bak version of the file, the backup will not be overwritten. (the backup has seniority, respectfully.)

WARNING: do not use the -o option of stap(1) with onboot scripts, because the script is started before the root filesystem is even mounted. increase the buffer size if more space is needed. this one is important bestie, i'm not playing 😭

• cleanup — delete the compiled *SCRIPT*s from cache. if you don't specify any scripts, then all compiled scripts are deleted. only the cache for the current kernel is deleted, or the kernel release specified by the -r option. it prompts for confirmation before deleting. consent first, always 🫶

---

✨ OPTIONS ✨

many of the commands can also take options. however — since users can't pass these options on boot, they're only meant for managing scripts after boot and for testing. so don't get any big ideas about boot-time flag passing. available options are:

• -c CONFIG_FILE — specify a different configuration file in place of the default one.
• -R — when using the start and stop commands, also include the scripts' dependencies (recursively). we love a recursive moment.
• -r KERNEL_RELEASE — when using the compile, onboot, and cleanup commands, specify the target kernel version rather than using the current one. must be in the same format as uname -r.
• -y — answer yes for all prompts. auto-consent, no cap.
• -o PATH.IMG — when using the onboot command, specify the output path of the created initramfs. when specified, the bootloader configuration is not updated.
• -b — when using the onboot command, backup an existing initramfs image by adding a .bak extension rather than overwriting it. without this option, the initramfs is overwritten. choose wisely ✨

---

✨ GLOBAL PARAMETERS ✨

these parameters affect the general behaviour of the SystemTap initscript service. they can be specified in the configuration file. this is the big-picture section, bestie:

• SCRIPT_PATH — specify the absolute path of the script directory. these are the scripts on which the initscript can operate. scripts must have the .stp extension. the default path is /etc/systemtap/script.d.
• CONFIG_PATH — specify the absolute path of the script configuration directory. these configuration files contain options for specific scripts. they must have the .conf extension. the default path is /etc/systemtap/conf.d.
• CACHE_PATH — specify the absolute path of the cache directory. the default path is /var/cache/systemtap.
• TEMP_PATH — specify the absolute path of the temporary directory in which SystemTap makes temporary directories to compile scripts. the default path is /tmp.
• STAT_PATH — specify the absolute path of the directory containing PID files used to track the status of SystemTap scripts. the default path is /var/run/systemtap.
• LOG_FILE — specify the absolute path of the log file. all messages are sent to this file, including compilation and runtime errors. the default path is /var/log/systemtap.log.
• PASSALL — if this is set yes, initscript commands that operate on multiple scripts will report as failed when the action could not be performed on at least one script. if set to no, only a warning is emitted. the default is yes.
• RECURSIVE — if this is set yes, the initscript will always follow script dependencies recursively. this means there's no need to specify the -R option. this flag is effective only if you specify script(s) from the command-line. the default is no.
• AUTOCOMPILE — if this is set yes, the initscript automatically tries to compile specified scripts when needed if there is no valid cache. otherwise, the related command simply fails. the default is yes.
• DEFAULT_START — specify scripts which will be started by default. if omitted (or empty), all scripts in the script directory will be started. the default is "".
• ALLOW_CACHEONLY — if this is set yes, the initscript will also allow operating on scripts that are located in the cache directory, but not in the script directory. the default is no.

WARNING: the initscript may load unexpected obsolete caches with this option. the cache directory should be checked before enabling this option. like, actually check it. please. 🙈

• LOG_BOOT_ERR — because boot-time scripts are run before the root filesystem is mounted, staprun's stderr cannot be logged to the LOG_FILE as usual. however, the log can instead be output to /run/systemtap/$script.log by setting LOG_BOOT_ERR to yes. if STAT_PATH is different from the default, the log files will be moved there upon executing any of the initscript commands. the default is no.

here's a global configuration file example, since seeing it is way better than me explaining it:

SCRIPT_PATH=/var/systemtap/script.d/
PASSALL=yes
RECURSIVE=no

---

✨ SCRIPT PARAMETERS ✨

these parameters affect the compilation or runtime behaviour of specific SystemTap scripts. they must be placed in config files located in the CONFIG_PATH directory. this is where it gets personal, per-script 🎀

• <SCRIPT>_OPT — specify options passed to the stap(1) command for the SCRIPT. here, SCRIPT is the name of the script file without the .stp extension. note that the -F option is always added. (it's not optional, bestie, it's just there.)

the following options are ignored when compiling scripts: -p, -m, -r, -c, -x, -e, -s, -o, -h, -V, -k.

the following options are ignored when running starting scripts: -h, -V, -v, -t, -p, -I, -e, -R, -r, -m, -k, -g, -P, -D, -b, -u, -q, -w, -l, -d, -L, -F, and all long options.

• <SCRIPT>_REQ — specify script dependencies (i.e. which script this script requires). for example, if foo.stp requires (or needs to run after) bar.stp, set:

foo_REQ="bar"

specify multiple scripts by separating their names by spaces.

and here's a script configuration file example, because examples are the moment:

script1_OPT="-o /var/log/script1.out"
script2_OPT="-o /var/log/script2.out"
script2_REQ="script1"

okay wait, that's where part 1 wraps up — there's more coming, don't go anywhere 💜

✨ EXAMPLES ✨

okay bestie this is the fun part, this is where we actually do stuff 🎀 every single command is copied exactly, don't you dare tweak them.

🌸 INSTALLING SCRIPTS 🌸

first things first, we drop a SystemTap script (let's say script1.stp) into the script directory:

# cp script1.stp /etc/systemtap/script.d/

then we can set whatever script options we want, like:

# vi /etc/systemtap/conf.d/group1.conf
script1_OPT="-o /var/log/group1.out"

now say we install another script (call it script2.stp) that needs to run after script1. easy, we just do this:

# cp script2.stp /etc/systemtap/script.d/
# vi /etc/systemtap/conf.d/group1.conf
script2_OPT="-o /var/log/group2.out"
script2_REQ="script1"

and the way this works is iconic actually — if stap(1) fails to run script1, the initscript won't even try to run script2. it just knows. 💅

---

🦋 TESTING 🦋

once the scripts are installed, we vibe-check that they actually work by literally just doing:

# systemtap-service start
# systemtap-service stop

we could also get more specific, like:

# systemtap-service start script1
# systemtap-service stop script1

no errors? slay. we're ready to go ✨

---

🎀 ENABLING SERVICE 🎀

once we're happy with the scripts and their tests, we enable the SystemTap initscript service:

# chkconfig systemtap on

---

💅 DELETING SCRIPTS 💅

deleting scripts is literally just removing them from the script directory and removing any config lines that were specific to them:

# rm /etc/systemtap/script.d/script2.stp
# vi /etc/systemtap/conf.d/group1.conf

but bestie, if the script is still running, we gotta stop it too:

# systemtap-service stop script2

and then we can also yeet the cache associated with the script:

# systemtap-service cleanup script2

---

🌈 PREPARING FOR KERNEL UPDATES 🌈

usually there's nothing to do when you boot into a new kernel — the initscript will notice the kernel version is different and will compile the scripts for you. but you can do the compilation beforehand too, so you don't have to sit there compiling during boot. that's what the -r option is for:

# systemtap-service compile myscript -r <NEW_KERNEL_VERSION>

---

⭐ IMPORTING COMPILED SCRIPTS ⭐

for environments that are missing the whole compilation infrastructure (no compilers, no debuginfo — think a production system, rip 😭), you can compile the scripts on another (development) machine and then transfer them over to the production system:

# systemtap-service compile myscript -r \
>   <KERNEL_VERSION_OF_TARGET_MACHINE>
# tar czf stap-scripts-<kernel-version>.tar.gz \
>   /var/cache/systemtap/<kernel-version> \
>   /etc/systemtap/conf.d/<configfile>

and then you copy this package to the target machine and extract it. that's it, that's the whole move 🫶

---

🧚 STARTING SCRIPTS DURING EARLY-BOOT 🧚

the initscript also lets us start scripts earlier in the boot process by creating an initramfs that contains the script's module. heads up though: the system must be dracut-based for this to work. starting a script at this stage gives you access to information that's otherwise very hard to obtain — main character energy tbh 👑

so first we install the script by copying it into the script directory like usual, and set whatever options we'd like:

# cp myscript.stp /etc/systemtap/script.d
# vi /etc/systemtap/conf.d/myscript.conf

to add the script to the initramfs, we use the onboot command:

# systemtap-service onboot myscript

if the script isn't already compiled and cached, it gets done at this point. a new initramfs then gets created at the default location. we can use the -b option to make sure the existing initramfs gets backed up. and then? we restart the system. 😌

---

💜 USING A DIFFERENT INITRAMFS 💜

if you'd rather only start the script for one boot and not the others, it might be easier to instead use the -o option to specify a different initramfs output file:

# systemtap-service onboot myscript \
>   -o /boot/special_initramfs.img

once that initramfs is created, it's simply a matter of changing the command-line options at boot-time so the new image gets used instead of the usual one.

---

🌸 CREATING AN INITRAMFS FOR A DIFFERENT KERNEL 🌸

just like with the compile command, we can use the -r option to specify the kernel we want to create the initramfs for. super useful when you're about to upgrade and wanna prepare in advance. for example:

# systemtap-service onboot myscript \
>   -r 3.12.6-200.fc19.x86_64

---

🦋 REMOVING SCRIPTS FROM THE INITRAMFS 🦋

finally, to remove all scripts from the initramfs, we simply run the onboot command without specifying any scripts:

# systemtap-service onboot

this will simply create a standard initramfs without any SystemTap modules inserted. clean slate, we love to see it ✨

---

🔥 TROUBLESHOOTING EARLY-BOOT ISSUES 🔥

there can be so many reasons why the module didn't insert or didn't work like you expected. it may be useful to turn on dracut debugging by adding 'rdinitdebug' to the kernel command-line and checking dmesg/journalctl -ae. also, the stderr output of staprun can be captured by setting the LOG_BOOT_ERR option to yes.

ngl this is the part where you read the logs. the logs always know. 🙈

---

👑 SEE ALSO 👑

stap(1) dracut(8) new-kernel-pkg(8)

---

💀 BUGS 💀

use the Bugzilla link of the project web page or our mailing list. http://sourceware.org/systemtap/, <systemtap@sourceware.org>.

---

and that's the whole page bestie!! this was SYSTEMTAP(8) from the SystemTap source/version set, dated as the footer says 💖 go forth and trace some syscalls, you absolute legend 🎀✨