SOS_COLLECT(1)

🌸 SOS_COLLECT(1) β€” the sos_collect manual, but make it make sense πŸ’…

okay bestie, welcome to the General Commands Manual glow-up of SOS_COLLECT(1). this is the sos collect page, and it's giving main character of your cluster energy. let's go πŸ¦‹


✨ NAME ✨

sos_collect β€” Collect sos reports from multiple (cluster) nodes 🫢


🌈 SYNOPSIS 🌈

no cap, here's the whole invocation, every bracket exactly where it lives:

sos collect
    [-a|--all-options]
    [-b|--become]
    [--batch]
    [--clean|--cleaner|--mask]
    [--cmd-timeout TIMEOUT]
    [-c CLUSTER_OPTIONS]
    [--chroot CHROOT]
    [--case-id CASE_ID]
    [--cluster-type CLUSTER_TYPE]
    [--container-runtime RUNTIME]
    [-e ENABLE_PLUGINS]
    [-i|--ssh-key SSH_KEY]
    [--inherit-config-file]
    [--encrypt-key KEY]
    [--encrypt-pass PASS]
    [--group GROUP]
    [-j|--jobs JOBS]
    [-l|--list-options]
    [--namespaces NAMESPACES]
    [--save-group GROUP]
    [--no-env-vars]
    [--nopasswd-sudo]
    [-k PLUGIN_OPTION]
    [--label LABEL]
    [--log-size SIZE]
    [-n SKIP_PLUGINS]
    [--nodes NODES]
    [--no-pkg-check]
    [--no-local]
    [--node-config-file FILE]
    [--plugin-timeout TIMEOUT]
    [--primary PRIMARY]
    [--image IMAGE]
    [--force-pull-image TOGGLE, --pull TOGGLE]
    [--registry-user USER]
    [--registry-password PASSWORD]
    [--registry-authfile FILE]
    [-o ONLY_PLUGINS]
    [-p SSH_PORT]
    [--password]
    [--password-per-node]
    [--preset PRESET]
    [--skip-commands COMMANDS]
    [--skip-files FILES]
    [-s|--sysroot SYSROOT]
    [--ssh-user SSH_USER]
    [--sudo-binary BINARY]
    [-t|--threads THREADS]
    [--upload]
    [--timeout TIMEOUT]
    [--transport TRANSPORT]
    [--tmp-dir TMP_DIR]
    [-v|--verbose]
    [--verify]
    [-z|--compression-type COMPRESSION_TYPE]

πŸ’– DESCRIPTION πŸ’–

collect is an sos subcommand whose whole job is to run around and grab sos reports from multiple nodes, then pack them all into one single, useful tar archive. one archive to rule them all, periodt. πŸ“¦βœ¨

you can run sos collect in one of two ways, and both are valid, slay:

  • from a workstation that already has SSH key authentication set up for the nodes in a given cluster, or
  • from a "primary" node inside a cluster that has SSH keys configured for the other nodes.

some sos report options are supported by sos-collect, and those get passed directly to the sos report command that runs on each node. so like, they travel. πŸ’Œ

⚠️ Note: the sos-collector command has been deprecated and will be removed in sos-4.9. use the new sos collect syntax instead, bestie. this is not a drill, this is a glow-up with a deadline. πŸ’€

πŸ‘‘ OPTIONS πŸ‘‘

one bullet per flag, nothing skipped, we do not gatekeep here 🫢

  • -a, --alloptions β€” Enables all sos report options.

ok but real talk: this does NOT enable all sos collect options. it's giving "all" but not all all, iykyk. πŸ™ˆ

  • -b, --become β€” Become the root user on the remote node when connecting as a non-root user.
  • --batch β€” Run in non-interactive mode. This will skip prompts for user input.
  • --clean, --cleaner, --mask β€” Enable inline obfuscation/cleaning of the collected archives.

when this option is used, the final collected archive (the one containing all node reports) will be processed to obfuscate sensitive information such as hostnames, IP addresses, MAC addresses, and user-specified keywords.

additional cleaning/masking options (such as --domains, --keywords, --map-file, --usernames, etc.) control the cleaning behavior. for documentation of all cleaning options, see sos-clean(1). 🧼

  • --all-logs β€” sos report option. Collects all logs regardless of size.
Default: no
  • -c CLUSTER_OPTIONS β€” Specify options used by cluster profiles. The format is 'profile.option_name=value'.

for example, for the ovirt plugin, if you wanted to restrict node enumeration to a specific cluster you would use -c ovirt.cluster=example_cluster.

available cluster options can be listed by running sos collect -l. πŸŽ€

  • --cmd-timeout TIMEOUT β€” sos report option. Specify a timeout limit in seconds for command execution on each node.

this sets the command timeout for all plugins on the remote nodes. a value of 0 means no timeout will be set. a value of -1 is used to indicate the default timeout of 300 seconds.

this option sets the command timeout for all plugins. for plugin-specific command timeouts, use the 'cmd-timeout' plugin option, e.g. '-k logs.cmd-timeout=600'.

note that setting a high cmd-timeout may require increasing --plugin-timeout as well. πŸ•°οΈ

  • --chroot CHROOT β€” sos report option. Set the chroot mode. When --sysroot is used, commands default to executing with SYSROOT as the root directory. This can be overridden by setting --chroot to "always" (always chroot) or "never" (always run in the host namespace).
  • --case-id CASE_ID β€” sos report option. Specifies a case number identifier. πŸ”’
  • --cluster-type CLUSTER_TYPE β€” When run by itself, sos collect will attempt to identify the type of cluster at play. This is done by checking package or configuration information against the localhost, or the primary node if "--primary" is supplied.

setting --cluster-type skips this step and forcibly sets a particular profile.

using a value of none or jbon (just a bunch of nodes) will effectively disable all cluster-specific checks, and cause sos collect to only use the nodes specified by the --nodes option. note that in this scenario, regex string(s) for node names will be ignored.

Example: sos collect --cluster-type=kubernetes will force the kubernetes profile to be run, and thus set sos report options and attempt to determine a list of nodes using that profile. πŸ”

  • --container-runtime RUNTIME β€” sos report option. Using this with collect will pass this option thru to nodes with sos version 4.3 or later. This option controls the default container runtime plugins will use for collections. See man sos-report.
  • -e ENABLE_PLUGINS, --enable-plugins ENABLE_PLUGINS β€” sos report option. Use this to enable a plugin that would otherwise not be run.

this option supports providing a comma-delimited list of plugins. 🧚

  • --encrypt-key KEY β€” Encrypts the resulting archive that sos collect produces using GPG. KEY must be an existing key in the user's keyring as GPG does not allow for keyfiles. KEY can be any value accepted by gpg's 'recipient' option.

note that the user running sos collect must match the user owning the keyring from which keys will be obtained. in particular this means that if sudo is used to run sos collect, the keyring must also be set up using sudo (or direct shell access to the account). πŸ”

users should be aware that encrypting the final archive will result in sos using double the amount of temporary disk space β€” the encrypted archive must be written as a separate, rather than replacement, file within the temp directory that sos writes the archive to. however, since the encrypted archive will be the same size as the original archive, there is no additional space consumption once the temporary directory is removed at the end of execution.

this means that only the encrypted archive is present on disk after sos finishes running.

if encryption fails for any reason, the original unencrypted archive is preserved instead.

IMPORTANT: As of this version of sos collect, only the final archive on the local machine running sos collect will be encrypted. The individual sos reports that are collected on the nodes will be collected unencrypted. 😭
  • --encrypt-pass PASS β€” The same as --encrypt-key, but use the provided PASS for symmetric encryption rather than key-pair encryption.
  • --group GROUP β€” Specify an existing host group definition to use.

host groups are pre-defined settings for the cluster-type, primary node, and nodes options saved in JSON-formatted files under /etc/sos/groups.d/<GROUP>.

if cluster_type and/or primary are set in the group, sos collect behaves as if these values were specified on the command-line.

if nodes is defined, sos collect extends the --nodes option, if set, with the nodes or regexes listed in the group.

note that sos collect will only write group definitions to /etc/sos/groups.d/ however the GROUP value may be a filename for any group definitions that exist outside of the default location. if you are manually writing these files, use the value null when a python NoneType is expected. Caveat: use string 'none' if setting cluster_type to none. πŸ—‚οΈ

  • --save-group GROUP β€” Save the results of this run of sos collect to a host group definition.

sos-collector will write a JSON-formatted file with name GROUP to /etc/sos/groups.d/ with the settings for cluster-type, primary, and the node list as discovered by cluster enumeration. note that this means regexes are not directly saved to host groups, but the results of matching against those regexes are. πŸ’Ύ

  • --inherit-config-file β€” Use the configuration file from the collector host for all nodes when running sos report.

when this option is specified, sos collect will use its own sos.conf (or other configuration file if specified via --config-file) and pass it to all nodes to use when collecting their individual sos reports. this ensures consistent configuration across all nodes in the cluster.

this option is mutually exclusive with --node-config-file. 🚫

  • -i SSH_KEY, --ssh-key SSH_KEY β€” Specify an SSH private key file to use for authentication when connecting to nodes.

this is useful when SSH key-based authentication is configured but the key is not in the default location (~/.ssh/id_rsa), or when you need to use a specific key for cluster node authentication.

the key file must be readable by the user running sos collect and must not be password-protected (or the ssh-agent must have the key loaded) as sos collect does not support interactive passphrase prompts for SSH keys. πŸ”‘βœ¨

  • -j JOBS --jobs JOBS β€” Specify the number of concurrent node collections that should be run.

if the number of nodes enumerated exceeds the number of JOBS, then sos collect will start collecting from the first X number of nodes and then continue to iterate through the remaining nodes as sos report collection finishes.

Defaults to 4.
  • -l, --list-options β€” Display available options for cluster profiles. πŸ“‹

...it shows you the cluster-specific options that work with the -c option for whatever cluster type got detected (or that you straight up told it about). Cluster profiles can support things like options to restrict node enumeration, or options that change up how collection behaves. πŸŽ€

Pair it with --cluster-type when you wanna see the options for one specific cluster profile β€” otherwise sos collect is out here trying to auto-detect the cluster type on its own.

For example:

sos collect --cluster-type=kubernetes --list-options
  • --nopasswd-sudo β€” use this when you're connecting as a non-root user and that user has passwordless sudo set up. ✨

Now listen: if you leave this off and you feed it a bogus sudo password, collection of sos reports may act all kinds of unexpected and/or just fully fail. πŸ’€ Do not skip it if it applies to you.

  • -k PLUGIN_OPTION, --plugin-option PLUGIN_OPTION β€” report option, bestie. Sets a plugin option to a particular value, and the format is plugin_name.option_name=value.

Example: if you wanna turn on the kubernetes "all" option in sos report, you do -k kubernetes.all=on. Easy. 🫢

  • --label LABEL β€” drop a label onto the archive names. This label lands on both the sos collect archive and the sos report archives.

And if a cluster already sets a default label, your user-provided label gets appended to that cluster default, not replacing it. 🌸

  • --log-size SIZE β€” caps how big collected logs and output can get, measured in MiB. Heads up though: this makes sos capture the last X amount of the file or command output it collects. So it's the tail end you're keeping, not the beginning. πŸ‘‘

By default this sits at 25 MiB, and it applies to all files and command output collected β€” the one exception being journal collections, which are limited to 100 MiB instead.

Set it to 0 and every size limitation just evaporates β€” files and commands get collected in their entirety. Which sounds iconic until you realize it may drastically increase the size of the final sos report tarball and the memory usage of sos during collection of commands, like very large journals that may be several GiB in size. 😭 Choose wisely.

  • --namespaces NAMESPACES β€” report option. For plugins that iterate their collections over namespaces that exist on the system, this limits how many namespaces get collected on each node.

Use '0' (that's the default) for no limit β€” all namespaces get used for collections.

Note that specific plugins may offer a similar 'namespaces' plugin option, and that one will override this option. πŸ’…

  • --no-env-vars β€” report option. Don't collect environment variables in the reports from the nodes.

By default, sos report does collect environment variables from the system, and ngl those may contain sensitive information. So use this to exclude environment variable collection on all nodes for privacy or security reasons. Just know that this may limit how useful the reports are for debugging environment-related issues. πŸ“

  • -n SKIP_PLUGINS, --skip-plugins SKIP_PLUGINS β€” report option. Disables (skips) a particular plugin that would otherwise run. Super handy when one plugin is prone to hanging for one reason or another. πŸ™ˆ

This one supports a comma-delimited list of plugins, so you can skip a whole little crew at once.

  • --nodes NODES β€” hand it either a comma-delimited list of nodes to collect sos reports from, or a regex string to compare against discovered node names. If you go the regex route, only nodes that match get used β€” think of it as a whitelist, never a blacklist.

You can also pass multiple regex strings separated by commas. And yes, you can mix and match β€” whole node names/addresses and regex strings can be provided at the same time. 🌈

  • --no-pkg-check β€” skips package checks entirely. Most cluster profiles check against installed packages to figure out whether the cluster profile should be applied or not.

Use this alongside --cluster-type if there are rpm or apt issues on the primary/local node. πŸ”₯

  • --no-local β€” don't collect a sos report from the local system.

If --primary isn't supplied, sos assumes the host running sos report is part of the cluster that's being collected. So use this to skip collecting a local sos report.

You do NOT need this option if --primary is provided.

  • --primary PRIMARY β€” specify a primary node IP address or hostname for the cluster.

If you provide it, sos collect checks the primary node instead of localhost when determining which type of cluster is in use. ⭐

  • --image IMAGE β€” specify an image to use for the temporary container that gets created for collections on a containerized host, in case you don't vibe with the default image that the host's policy specifies. Note that this should include the registry. πŸ’–
  • --node-config-file FILE β€” path to an existing configuration file on the nodes to use with sos report.

When you specify it, sos collect tells each node to use the config file at that given path (and it must exist on each node) when running sos report. This lets nodes use a pre-deployed config file instead of their default configuration or the collector's configuration.

Two things: the path must be absolute, and the file must exist on all nodes being collected from. Also this option is mutually exclusive with --inherit-config-file. No having both, bestie. 🚫

  • --force-pull-image TOGGLE, --pull TOGGLE β€” when collecting a sos report from a containerized host, this forces the host to always pull the specified image, even if that image already exists on the host. Which is useful to make sure the latest version of that image is always in use. If you disable it, the node uses whatever version of the image is present, and only attempts a pull if there's no copy of the image present at all.

Enable it with true/on/yes, or disable it with false/off/no.

Default: true

  • --registry-user USER β€” specify the username to authenticate to the registry with, so the container image can be pulled. πŸ”‘
  • --registry-password PASSWORD β€” specify the password to authenticate to the registry with so the container image can be pulled. If no password is required, leave this blank. πŸŽ€
  • --registry-authfile FILE β€” specify the filename to use for providing authentication credentials to the registry for pulling the container image.

Note that this file must exist on the node(s) performing the pull operations β€” not on the node from which sos collect was run. Location matters, okay? πŸ’…

  • -o ONLY_PLUGINS, --only-plugins ONLY_PLUGINS β€” report option. Runs ONLY the plugins listed. That's it, that's the whole list.

Note that a cluster profile will NOT override this option. Which means the sos reports generated may end up not containing the relevant output for a given type of cluster. So be intentional. ⚠️

This option supports providing a comma-delimited list of plugins.

  • --password β€” specifying this makes sos collect prompt the user for an SSH password, which gets used to connect to all the nodes.

If you've got differing passwords for the same user across cluster nodes, ideally you should deploy SSH keys instead. But the --password-per-node option is also out here available. πŸ¦‹

  • --password-per-node β€” with this one, sos collect prompts the user for the SSH password for each node that'll have a sos report collected from it, individually, before it attempts to connect to the nodes. One by one, no shortcuts. πŸ™ˆ
  • --preset PRESET β€” specify a sos preset to use. Note that this requires sos-3.6 or later to be installed on the node. The given preset must also exist on the remote node β€” local presets are not used. No sneaking your local ones in. πŸ’…

If --preset is specified and a given node either doesn't have that preset defined, or is running a version of sos prior to 3.6, then this option is simply ignored for that node.

  • --plugin-timeout TIMEOUT β€” report option. Specify a timeout in seconds to allow each plugin to run for, on each node.

A value of 0 means no timeout will be set. A value of -1 is used to indicate the default timeout of 300 seconds.

This option sets the timeout for all plugins. For plugin-specific timeouts, use the

The plugin-specific timeout option will override this option.

  • -p SSH_PORT, --ssh-port SSH_PORT β€” specify the SSH port for all nodes. Use this if SSH runs on any port other than 22. 🫢
  • --skip-commands COMMANDS β€” a comma delimited list of commands to skip executing, while still letting the rest of the plugin that calls the command run. This will generally need to be some form of UNIX shell-style wildcard matching. For example, using a value of hostname will skip only that single command, while using hostname* will skip all commands with names that begin with the string "hostname". ✨
  • --skip-files FILES β€” a comma delimited list of files or filepath wildcard matches to skip collection of. Values can be either exact filepaths or paths using UNIX shell-style wildcards, for example /etc/sos/*. 🌸
  • --upload β€” uploads the collected archive to a vendor-defined or user-specified location after collection.

Additional upload options (like --upload-url, --upload-user, --upload-s3-*, etc.) control the upload destination and authentication. For documentation of all the upload options, go look at sos-upload(1) or also sos-report(1).

Note: the upload applies to the final collected archive, not to the individual node reports. πŸ‘‘

  • --ssh-user SSH_USER β€” specify an SSH user for sos collect to connect to nodes with. Default is root.

And heads up: sos collect will prompt for a sudo password for non-root users. πŸ”

  • --sudo-binary BINARY β€” specify the binary to use for privilege escalation on remote hosts when connecting as a non-root user. Note that this binary MUST be in the connecting user's PATH on the remote host. No PATH, no escalation. πŸ’€
  • -s SYSROOT, --sysroot SYSROOT β€” report option. Specify an alternate root file system path. πŸŽ€
  • -t THREADS, --threads THREADS β€” report option. Specify the number of collection threads to run.

The report process on each node will run THREADS number of plugins concurrently during the collection process. Parallelize, bestie. ⚑

Defaults to 4.

  • --timeout TIMEOUT β€” timeout for sos report generation on each node, in seconds. ⏰

...and just so you know, sos reports get collected in parallel, meaning the total runtime of sos collect is roughly timeout*(number of nodes/jobs). Math queen behavior πŸ’…

Default is 180 seconds.

  • --transport TRANSPORT β€” pick what kind of remote transport we're using to manage the connections to the remote nodes.

sos collect doesn't rawdog the connections itself β€” it uses the binaries you already have installed locally to connect to and interact with the remote nodes. By default it loves OpenSSH's ControlPersist feature, but some cluster types have their own opinions about how remote sessions should be set up. Not all clusters agree, bestie, it's giving "group project" πŸ™ˆ

The transports we currently support are, as follows:

  • auto β€” let the cluster type decide which transport gets used ✨
  • control_persist β€” use OpenSSH's ControlPersist feature. This is the default behavior πŸ‘‘
  • oc β€” use a locally configured oc binary to deploy collection pods on OCP nodes πŸ¦‹
  • --tmp-dir TMP_DIR β€” pick a temporary directory to save sos archives to. If you don't, one gets created in /tmp and then gets removed after sos collect has finished running, so don't get attached 🫢
Heads up! This is NOT the same as specifying a temporary directory for sos report on the remote nodes. Two totally different things, don't mix them up, iykyk.
  • -v --verbose β€” print debug information to screen. Feral amounts of output, but it's honest work πŸ“œ
  • --verify β€” this one's a report option. It passes the "--verify" option to sos report on the nodes, which makes sos report validate plugin-specific data during collection. Very thorough, very main character ✨
Note that this option may considerably extend the time it takes sos report to run on the nodes. Consider increasing --timeout when using this option. Like, seriously, plan your afternoon around it 😭
  • -z COMPRESSION, --compression-type COMPRESSION β€” also a report option. Override the default compression type. You get to pick. Iconic πŸŽ€

🌈 SEE ALSO 🌈

sos(1) sos-report(1) sos-clean(1) sos.conf(5)


πŸ‘‘ MAINTAINER πŸ‘‘

Maintained on GitHub at https://github.com/sosreport/sos β€” go say hi, star it, it's free πŸ’–


πŸ’œ AUTHORS & CONTRIBUTORS πŸ’œ

See AUTHORS file in the package documentation. The whole squad is in there, no cap 🌸


nd that's the whole page bestie, we ate and left no crumbs πŸ“ This was SOS_COLLECT(1), the sos collect manual, dated April 2020 β€” yes it's giving vintage but she still slays πŸ’… Go collect something, periodt πŸ¦‹

πŸ“¦ where this page comes from