rig(1)

💖 rig(1) — General Commands Manual 🎀

🌸 NAME 🌸

rig — monitor a system for events and trigger specific actions. That's the whole vibe: watch stuff, then do stuff when the watched stuff hits the condition. 👑

💅 USAGE 💅

rig <RESOURCE OR SUBCOMMAND> [OPTIONS] <ACTIONS> [ACTION OPTIONS]

This is the shape of every rig call, bestie. Resource or subcommand first, options next, then the actions and their action options. ✨

✨ DESCRIPTION ✨

rig is a tool to assist in troubleshooting seemingly randomly occurring events or events that occur at times that make active monitoring by a sysadmin difficult. It's literally built for the “why did it break at 3am while I was asleep” moments. 😭

rig sets-up detached processes, known as 'rigs', that watch a given resource for a trigger condition, and once that trigger condition is met takes actions defined by the user. So you tell it what to stalk, and what to do when the stalked thing hits the condition. Periodt. 💅

👑 GLOBAL OPTIONS 👑

The following are options available to all rigs (resources). They're the universal little accessories every rig can wear. 🎀

• --delay DELAY — Specify the number of seconds to wait after a rig is triggered before running the configured actions. Note that the rig will still trigger and stop all watcher threads immediately - this delay comes after thread termination but before action execution in order to avoid a possible race condition where multiple watcher threads could conceivably trigger during a sufficiently high delay time.

Default 0 seconds, meaning execute actions immediately upon rig trigger condition being met.

• --debug — Set logging level to debug instead of the default info level.
• --foreground — Run the rig in the foreground, keeping stdout attached.
• --interval SECONDS — Specify the amount of time to wait between a rig's polling cycles. Most rigs monitor their resources in a flow of update -> compare -> wait, where wait is simply sleeping until the next needed update. Use this option to set how long a rig should wait/sleep before updating their monitors again.

Default: 1, meaning update and compare once every second.

• --name NAME — Give the rig a name, rather than generating a random one at deployment.

By default, rigs are given a randomly generated string as a name, which will appear in rig info output and in the rig's socket name. Using this option will use the provided name instead, and may be useful in distinguishing rigs when several are deployed at one time

• --no-archive — Do not create a tar archive of the collected data after a rig has been triggered.

Normally, once all data has been collected, rig will create a gzip'd tar archive under /var/tmp containing all the files created from the rig's actions - after which, the temp directory at /var/tmp/rig/<id>/ is deleted.

Using this option skips creating the archive and preserves the temp directory.

• --repeat COUNT — Repeat certain actions COUNT number of times after the initial execution of the action.

Actions will, unless otherwise specified by this option, only execute once. Using this option actions that support repetition will be repeated an additional COUNT number of times. For example, using --repeat 2 will result in repeatable actions being executed three (3) total times.

Not every action supports repetition - in fact most do not. See specific action's sections for information on if it can be repeated or not.

• --repeat-delay SECONDS — Number of seconds to wait between repetitive executions of the same action.

This can be useful when using an action like gcore when you want to get coredumps over a certain time period. For example, using --repeat 1 --repeat-delay 60 will give you two (2) coredumps taken one minute apart.

Defaults to one second.

• --restart COUNT — Restart a configured rig up to COUNT number of times after being triggered.

By default, a rig will trigger once and then terminate. Using this option, an individual rig may restart itself up to COUNT number of times, producing an additional archive of the requested data after the triggering event happens again.

Note that this is the number of times to restart, not the total number of times to run. Using a restart value of '2' means that there will be 3 total archives generated for a rig.

By default, this is set to 0, meaning terminate after the first trigger event. Use a value of '-1' to have a rig perpetually restart itself without limit.

🦋 SUBCOMMANDS 🦋

📋 rig list 📋

Show a list of known existing rigs and their status. Status information is obtained by querying the socket created for that particular rig. It's giving “who's here and what are they doing” energy. 👀

💥 rig destroy -i [ID or 'all'] 💥

Destroy a deployed rig with id ID. If ID is 'all', destroy all known rigs. Note that if another entity kills the pid for the running rig, destroy will fail as the socket is no longer connected to the (now killed) process. In this case use the --force option to cleanup the lingering socket.

Any data the rig has generated will be lost when invoking destroy. So bestie, be sure you don't need that data anymore. 💀

🔍 rig info -i [ID] 🔍

Get detailed information on a rig. This information will include configuration options, the entire cmdline string given to launch the rig, as well as information on each action the rig is configured to take and what the expected result from those action(s) are.

Currently, this data is written to stdout in JSON format. Slay. ✨

⚡ rig trigger -i [ID] ⚡

Manually trigger rig with id ID. This will cause the specified rig to begin executing the actions configured for it, as if the trigger condition had been met.

Note that this is only effective on a single rig basis, so using a value of 'all' for the ID will not work. Don't try it, bestie. 🙈

🌈 RESOURCES 🌈

These are the system resources that rig can monitor. There may be additional manpages for specific resources. Where applicable this will be noted below.

Note that 'resources', 'monitors', and referencing 'a rig' as a distinct entity all refer to the same thing. Same vibe, different name. 💖

When creating a rig, if successful the rig's ID will be printed to console.

📜 logs 📜

Watch a single or multiple log files and/or journald units for a specified message. When that message is matched to any watched file or journal, the trigger condition is met and configured actions are initiated. It's basically a little spy for your logs. 🕵️‍♀️

The following options are available for the logs rig:

• -m|--message STRING — Define the string that serves as the trigger condition for the rig. This can be a regex string or an exact message. Be very careful in using the '*' regex character as this may cause unintended behavior such as the rig immediately triggering on the first message seen.

Note that a small amount of transformation and testing is done on the provided STRING. First, '*' characters are converted to the python-style regex match of .*. After which, rig performs a test on if the provided message will regex-match itself, and if that fails the rig aborts the creation process.

Aside from the conversion noted above, regexes provided in this option must be python-style and not shell-style.

• --logfile FILE — A comma-delimited list of files to watch. Each FILE specified will be monitored from the current end of the file, so old entries will not set off the rig's actions.

Default: /var/log/messages

• --no-files — Do not monitor any log files.
• --journal UNIT — A comma-delimited list of journal units to watch. The journal is watched as a singular entity, and will be filtered to only read from the provided UNIT(s). If no UNIT is specified, the whole system journal will be monitored.

Default: 'system'

• --no-journal — Do not monitor the journal.
• --count COUNT — The number of times the --message string should be matched before the rig is triggered. Default 1 - meaning match on the first occurence.

🏓 ping 🏓

Perform a simple ongoing ping test against a specified host. Pings are sent one at a time at a defined interval, and the response is evaluated. Ping-type rigs may monitor for number of lost packets and/or packets exceeding a specified RTT in milliseconds.

Packets are first evaluated for loss (including timeouts), then for RTT time. Order matters, bestie. 💅

The following options are available for the ping rig:

• --host ADDRESS — The target IP or hostname to ping. This is a required option in order for a ping rig to be created.

During rig creation, a 'sanity check' ping is sent to the ADDRESS to ensure that it is an address that is reachable on the network and that it will respond to ICMP packets. If this sanity check fails, rig creation is aborted.

• --ping-timeout SECONDS — Specify the number of SECONDS to allow for a ping response. If a ping encounters a timeout, then it is considered both a lost packet and a packet exceeding the RTT threshold (see --ping-ms-max and --ping-ms-count).
• --lost-count PACKETS — Specify the number of PACKETS to accept being lost or timed-out, before triggering the rig.

Default: 1 (trigger on the first lost packet)

• --ping-interval SECONDS — Specify the number of SECONDS to wait between ping requests sent to the target host.

Default: 1

• --ping-ms-max MILLISECONDS — Specify the RTT threshold to allow for a returned ping request. If the RTT reported by the ping command is above this value in milliseconds, it is counted against the threshold of packets exceeding this value specified by --ping-ms-count.

By default, this form of checking is disabled. Any integer value passed to this option will enable RTT monitoring.

• --ping-ms-count PACKETS — Specify the number of PACKETS that may exceed the defined --ping-ms-max RTT value before triggering the rig.

Default: 5

🧍 process — watch one lil guy (or a whole squad) 👀

So bestie, this rig is for stalking — lovingly — a single process or a list of processes and watching for state changes or resource consumption thresholds. The way it works: when the process enters the state you asked for, or hits the resource threshold you asked for, boom — trigger condition met. ✨

The following options are available for the process rig:

• --proc — A PID or a process name of processes to watch. If you hand it a process name, rig will try to convert it into a PID during rig creation. If it finds multiple PIDs, the default behavior is to fail creation and exit 😭. Want rig to monitor all the processes found for a process name instead? Use the --all option, periodt.
• --state STATE — The state a process needs to be in for the rig to trigger. Here's the full menu of supported states:
  • dead — Dead - should never be seen — shorthand 'X'
  • disk-sleep — Uninterruptible sleep — shorthand 'D' or 'UN'
  • running — Currently running — shorthand 'R' or 'run'
  • sleeping — Interruptible sleep — shorthand 'S' or 'sleep'
  • stopped — Stopped — shorthand 'T' or 'stop'
  • zombie — Exited, still in proc table — shorthand 'Z' or 'zomb'

You can use either the full status name or the shorthand, no cap. Both the names and the shorthand values are case sensitive though, so don't get cute with the capitalization. 🙈

You can also flip it to a "not" value by putting an exclamation mark (!) in front of any of those state strings — so '!sleeping' matches any non-sleep (S) state status for the process(es). Most shells will make you quote the state string when you use that '!' character.

Heads up: using '!running' will make rig not trigger against a state of 'sleeping', because generally speaking 'running' processes spend a lot of their time in S state, and it's assumed you don't actually want to trigger against a process like that. 💅

Process status is polled once every second. ⏱️

• --rss INTEGER — The amount of rss (resident set size) memory usage to use as a threshold for triggering the rig. If the process' RSS usage goes above this value → trigger. 🔥 The value you give here may be suffixed with K, M, or G to denote the IEC unit. Rig will convert the value plus suffix into a value in bytes.
• --vms INTEGER — The same as --rss but it's monitoring Virtual Memory Size instead. 🫶
• --memperc PERCENT — The percentage of total system memory a process is consuming, used as the threshold for triggering the rig. If the process' %mem meets or exceeds this value → trigger.

PERCENT may be a whole integer or a float. When you use a float, the process rig respects up to two (2) decimal points of precision. So for example, --memperc 10.25 is the same as using --memperc 10.25678. 🎀

• --cpuperc PERCENT — The percentage of CPU usage a process is consuming, used as the threshold for triggering the rig. If the process' %cpu meets or exceeds this value → trigger.

PERCENT may be a whole integer or a float. When using a float and monitoring for CPU usage, rig respects one (1) decimal point of precision, because of how CPU usage is reported. 😌

Also PERCENT may be above 100 — because CPU usage can exceed 100 when a process is running on multiple CPUs. Slay, multi-core icon. 👑

---

🖥️ system — the whole vibe check of your machine

This rig watches the system's utilization of resources as a whole, like total CPU or memory usage. When the utilization of a given resource is either exceeded or falls below the given threshold (whichever is appropriate for that resource), the trigger condition is met. 🌈

The following options are available for the system rig:

• --iowait PERCENT — The amount of %iowait as reported by the kernel, used as a threshold value. If exceeded → trigger the rig.
• --steal PERCENT — The amount of %steal as reported by the kernel, used as a threshold value. If exceeded → trigger the rig.
• --nice PERCENT — The amount of %nice as reported by the kernel, used as a threshold value. If exceeded → trigger the rig.
• --guest PERCENT — The amount of %guest as reported by the kernel, used as a threshold value. If exceeded → trigger the rig.
• --user — The amount of %user as reported by the kernel, used as a threshold value. If exceeded → trigger the rig.
• --available INTEGER — The amount of available memory in MiB as reported by the kernel, used as a threshold value. If the amount of available memory falls below this threshold → trigger the rig. 🍓
• --free INTEGER — The amount of free memory in MiB as reported by the kernel, used as a threshold value. If the amount of free memory falls below this threshold → trigger the rig.
• --used INTEGER — The amount of used memory in MiB as reported by the kernel, used as a threshold value. If the amount of used memory exceeds this threshold → trigger the rig.
• --slab INTEGER — The amount of slab memory in MiB as reported by the kernel, used as a threshold value. If the amount of slab memory exceeds this threshold → trigger the rig.
• --cpuperc PERCENT — The amount of total CPU usage as reported by the kernel, as a percentage, used as a threshold value. If exceeded → trigger the rig. This value may be a whole integer or a float. Floats are precise out to one (1) decimal point.
• --memperc PERCENT — The amount of total memory usage as reported by the kernel, as a percentage, used as a threshold value. If exceeded → trigger the rig. This value may be a whole integer or a float. Floats are precise out to one (1) decimal point.
• --loadavg FLOAT — System load average as reported by the OS, used as a threshold value. If the reported loadavg exceeds this value → trigger the rig. This option can accept either an integer (1) or a float (1.0).

Linux returns loadavg data for the past 1, 5, and 15 minutes. The system rig will monitor only one (1) of these intervals at a time, as controlled by the --loadavg-interval option. 🦋

• --loadavg-interval [1, 5, 15] — Which time interval the rig should monitor when watching the system's loadavg. Only 1, 5, and 15 are accepted values for this option, because that's what the Linux kernel returns loadavg data for. Periodt.

Default: 1

• --temp INTEGER — The temperature in Celsius rig should monitor the CPU for meeting or exceeding. 🌡️ This option takes an integer value, though temperature data is single decimal point sensitive — so a temperature of 50.9 degrees will not trigger a rig that sets this option to 51.

By default rig will monitor the first physical CPU package installed on the system. This may be changed via the --cpu-id option. Note that rig will only monitor whole packages and not individual cores, and that package temperatures reported are the highest reported temperature for any core in that package.

• --cpu-id ID — If specified, monitor this physical CPU package. By default, rig will monitor physical CPU package 0 — meaning the first physically installed CPU.

When specifying an ID here, remember that in Linux CPU IDs are zero-indexed, so the first CPU will be ID 0, the second ID 1, and so forth. 💅

Default: 0

---

📁 Filesystem — where did all my disk space go

Watch a filesystem, directory, or file for utilization changes. Currently this rig is focused on space consumption, and it'll trigger when the specified path or backing filesystem exceeds the defined threshold for space utilization. 🎀

The following options are available for the filesystem rig:

• --path PATH — Specify the filesystem, directory, or file path for the rig to monitor. The location you provide must exist when the rig initializes for monitoring to be supported. No ghost paths bestie. 👻
• --size SIZE — Specify the size threshold to trigger on for the provided --path. The size you give must be an integer suffixed with either K, M, G, or T. The value you provide will be converted to bytes.
• --fs-size SIZE — Use this option instead of --size if you want to monitor the space usage of the backing filesystem for --path rather than the size of the path alone. Similar to --size, this value must be suffixed with either K, M, G, or T.
• --fs-used PERCENT — Similar to --fs-size but instead you provide a percentage value to trigger on, when the filesystem's %used exceeds this value. Note that using this option is ultimately the same as --fs-size, since rig will convert the specified percentage into a raw bytes value to use for comparisons. ✨

🌸 ACTIONS 🌸

okay bestie, so these are all the things a rig can do when it gets triggered 💖 You can chain multiple actions onto one single rig, which means you don't have to go deploy like five separate rigs with the same trigger conditions just to get different actions out of them. Efficiency queen behavior, honestly 👑

Here's the tea on how the order works: actions get executed based on a priority weighting system, and lower values = higher priority. So the ones with lower values run before the ones with higher values. Why? Because that way the time-sensitive stuff gets done first, before the actions that either take forever to run or genuinely do not care if something else ran before them. ✨ Action priority values are set by the actions themselves, and rn users can't modify them. That's just how it is, periodt.

---

💀 gcore

Collect a coredump of a given process (or processes) using GDB's gcore utility. Slay.

Heads up bestie: this does not interrupt the running process(es). No cap. 🙈

Cores get saved to /tmp and they're named either core.$pid or core.$proc_name.$pid, depending on whether you handed over a PID or a process name. This action runs first when a rig is triggered and you've specified multiple actions. First in line, main character energy 👑

This action supports repetition via the --repeat option.

The gcore action supports these options:

• --gcore PROCESS — Enables this action and takes either a PID or a process name as a value. If you give it a process name, the PID gets determined at rig creation. If multiple PIDs turn up for the same process name, the default behavior is to fail rig creation 😭 — so use the --all-pids option instead if you want all the PIDs discovered for a process name. This option can be specified multiple times. E.G. --gcore 12345 --gcore myprocess will generate a coredump for PID 12345 and a process matching the name myprocess.
• --all-pids — Tells this action to collect a coredump for all PIDs found for a provided process name.
• --freeze — Freeze the process(es) that are gonna be core dumped by sending a SIGSTOP before calling gcore on the discovered pid(s). If it works, rig will send a SIGCONT after the gcore execution finishes, so the process gets thawed out. 🧊➡️🔥

---

💥 kdump

Generate a vmcore by triggering a kernel crash via sysrq.

⚠️ BESTIE. LISTEN. This action WILL cause node disruption by triggering a kernel panic to generate the vmcore. Which means your system will reboot when this action is triggered. This is not a drill. 💀

The kdump action does not perform any configuration checks on the system's kdump installation. It just assumes kdump has been properly configured and tested before you ever touched this action. So like… go test it first, fr fr. 🙏

The kdump action supports these options:

• --kdump — Enables this action.
• --sysrq INTEGER — When the rig is deployed, if this option is set, rig will set the system's /proc/sys/kernel/sysrq to the value you provide. Check the sysrq kernel documentation for info on what values are supported. 📖

---

🎀 sosreport

Run an sos report after the rig has been triggered. Plugin enablement options, plus the --plugin-option from sos report, are all supported by this rig. This action should run after any time-sensitive actions you otherwise specified for a given rig. Patient queen, she waits her turn 💅

The sosreport action supports these options:

• --sosreport — Enables this action.
• --enable-plugins PLUGINS — Specifically force the specified comma-delimited list of PLUGINS to be enabled.
• --plugin-option PLUGOPT — Modify a specific plugin's runtime options. This gets passed directly to sos report as the same --plugin-option value, which should take the form name.option=value. For example, to increase the podman plugin timeout use --plugin-option podman.timeout=600. If you need to pass multiple sos report plugin options, use a comma-delimited list here instead of specifying this option multiple times. One and done, bestie ✨
• --skip-plugins PLUGINS — Do not run these specified plugins. Use a comma-delimited list to skip multiple plugins.
• --only-plugins PLUGINS — Only enable these specific plugins, disable all others. Use a comma-delimited list to specify multiple plugins.

---

🦋 tcpdump

Start collecting a tcpdump when the rig is initialized, and stop the collection when the rig triggers. This action gets triggered before most other actions — but after the gcore action. So: gcore first, then tcpdump, then everybody else 🌈

Heads up: there's gonna be a slight delay in configuring any rig that uses the tcpdump action, because rig has to verify that the tcpdump process started successfully during the initialization process. Worth it though 💖

The tcpdump action supports these options:

• --tcpdump — Enables this action.
• --iface INTERFACE — Starts the tcpdump to monitor the provided INTERFACE. In almost all situations this should probably be set to a specific interface on the system, however the value any is accepted by the tcpdump command in order to listen on all interfaces.

Be wary of using any though, because using it means it'll be impossible to determine which interface a particular packet came in on in the resulting packet capture. 🚩

Default: eth0

• --filter FILTER — Provide a filter to use with tcpdump in order to reduce the amount of traffic recorded in the packet capture. This value gets passed directly to the tcpdump utility, so it can be any valid filter accepted by tcpdump. For most shells you must quote the filter string so rig passes it correctly. Don't skip the quotes bestie, I'm begging 🙈
• --snaplen LENGTH, --snapshot-length LENGTH — Set the snapshot length for the packet capture. This truncates captured packets to LENGTH bytes, which defaults to 262144 bytes. Using a value of 0 (also the default) will imply a LENGTH of 262144 bytes.
• --dump-size SIZE — Limit the size of the packet capture file(s) to SIZE in MB. Default: 10
• --captures CAPTURES — Specify the number of packet capture files to keep. If more than one (1), then tcpdump will rotate the packet capture file when it reaches the --size value and keep CAPTURES number of files. E.G. using a CAPTURES of 2 and a DUMP-SIZE of 5, then when the rig terminates you'll have up to 2 5MB packet captures. Iconic 💅 Default: 1 (packet capture file is replaced upon reaching SIZE limit)

---

👀 monitor

While a rig is running, monitor various system statistics and record them for later review. These statistics can be file contents or command outputs. It's giving surveillance but make it helpful 🕵️♀️

This action starts collecting info when the rig is started, and stops when the rig is triggered.

By default, networking-centric information is monitored via commands such as netstat, ss, top, ps, and more. Similarly, several networking-related files under /proc/ are monitored.

The rate at which these collections happen is controlled by the --interval option.

The monitor action supports these options:

• --monitor — Enables this action.
• --disable-monitor-defaults — Do not monitor or collect any of the default items. This implies that all collections will be specified via --monitor-files and/or --monitor-commands. You're on your own now, bestie, but in a cool way 💪
• --monitor-files FILES — A comma-delimited list of files to monitor. Monitored files have their contents copied to a file within the rig's archive of the same name. The contents will be separated by a timestamp header taken at the time of collection. ⏰
• --monitor-commands COMMANDS — A comma-delimited list of commands to execute every --interval seconds, and have that output saved to a file within the rig's archive. Output collections are separated by a timestamp header taken at the time of collection. Note that commands will need to be properly quoted if there are spaces (or other quotes) in the command string. For example, to run ps auxwww the proper invocation would be --monitor-commands='ps auxwww'.

In-line shell scripting is not supported. While it may be possible for such values to function, there are no guarantees as to those executions working properly — at all — not causing unintended side-effects or harm, et cetera. Dragons ahead, and so forth. 🐉💀

---

🍓 noop

Does nothing — this action runs a no-op. Ideally used for when you need to test a rig's configuration to make sure a rig's trigger condition is set properly — e.g. a regex string for the logs' rig message option. It's the "am I doing this right?" action, and honestly? We love a safe little test run 🫶

The noop action supports these options:

• --noop — Enables this action.

---

🌟 MAINTAINER 🌟

Jake Hunsaker <jhunsake@redhat.com>

---

okay that's the whole entire page bestie, we did it 💖 this one's from January 2019, it's the rig(1) manual page, and you read the WHOLE thing which is genuinely so iconic of you. go forth and deploy responsibly, mwah 😘✨👑