pdsh(1)

📦
version 2.36 · 3 versions
, in debian trixie, ubuntu questing, ubuntu resolute, ubuntu stonking

okay bestie, grab a snack 🍓 because this is pdsh(1) from the General Commands Manual and it's genuinely iconic. it's giving "run one command on a hundred hosts at once and don't even break a sweat." let's go 💅

✨ NAME ✨

pdsh — issue commands to groups of hosts in parallel 🫶

✨ SYNOPSIS ✨

pdsh [options]... command

✨ DESCRIPTION ✨

so pdsh is basically a variant of the rsh(1) command. but here's the glow-up: rsh(1) runs a command on one remote host, and pdsh runs multiple remote commands in parallel. main character energy fr.

pdsh uses a "sliding window" (or fanout) of threads to conserve resources on the initiating host while still allowing some connections to time out. resource girlie behavior, we love to see it 💖

when pdsh receives SIGINT (that's ctrl-C), it lists the status of current threads. a second SIGINT within one second terminates the program. and if you issue ctrl-Z within one second of ctrl-C, pending threads may be canceled. "pending threads" = the ones that haven't been initiated yet, or are still in the process of connecting to the remote host. no cap.

if a remote command is not specified on the command line, pdsh runs interactively — prompting for commands and executing them when terminated with a carriage return. in interactive mode, target nodes that time out on the first command are not contacted for subsequent commands, and commands prefixed with an exclamation point will be executed on the local system. cute, right?

the core functionality of pdsh may be supplemented by dynamically loadable modules. these modules may provide:

  • a new connection protocol (replacing the standard rcmd(3) protocol used by rsh(1)),
  • filtering options (e.g. removing hosts that are "down" from the target list),
  • and/or host selection options (e.g., -a selects all hosts from a configuration file).

by default, pdsh must have at least one "rcmd" module loaded. see the RCMD MODULES section for more info 👑

✨ RCMD MODULES ✨

the method by which pdsh runs commands on remote hosts may be selected at runtime using the -R option (see OPTIONS below). this functionality is ultimately implemented via dynamically loadable modules, so the list of available options may be different from installation to installation. a list of currently available rcmd modules is printed when using any of the -h, -V, or -L options. the default rcmd module will also be displayed with the -h and -V options. 💅

a list of rcmd modules currently distributed with pdsh follows.

  • rsh — uses an internal, thread-safe implementation of BSD rcmd(3) to run commands using the standard rsh(1) protocol.
  • exec — executes an arbitrary command for each target host. the first of the pdsh remote arguments is the local command to execute, followed by any further arguments. some simple parameters are substitued on the command line, including %h for the target hostname, %u for the remote username, and %n for the remote rank [0-n] (to get a literal % use %%). for example, the following would duplicate using the ssh module to run hostname(1) across the hosts foo[0-10]:
pdsh -R exec -w foo[0-10] ssh -x -l %u %h hostname

and this command line would run grep(1) in parallel across the files console.foo[0-10]:

pdsh -R exec -w foo[0-10] grep BUG console.%h
  • ssh — uses a variant of popen(3) to run multiple copies of the ssh(1) command.
  • mrsh — this module uses the mrsh(1) protocol to execute jobs on remote hosts. the mrsh protocol uses a credential based authentication, forgoing the need to allocate reserved ports. in other aspects, it acts just like rsh. remote nodes must be running mrshd(8) in order for the mrsh module to work.
  • krb4 — the krb4 module allows users to execute remote commands after authenticating with kerberos. of course, the remote rshd daemons must be kerberized.
  • xcpu — the xcpu module uses the xcpu service to execute remote commands.

✨ OPTIONS ✨

the list of available options is determined at runtime by supplementing the list of standard pdsh options with any options provided by loaded rcmd and misc modules. in some cases, options provided by modules may conflict with each other. in these cases, the modules are incompatible and the first module loaded wins. 💀

Standard target nodelist options

  • -w TARGETS,... — target and or filter the specified list of hosts.

do not use with any other node selection options (e.g. -a, -g, if they are available). no spaces are allowed in the comma-separated list. arguments in the TARGETS list may include normal host names, a range of hosts in hostlist format (see HOSTLIST EXPRESSIONS), or a single - character to read the list of hosts on stdin.

if a host or hostlist is preceded by a - character, this causes those hosts to be explicitly excluded. if the argument is preceded by a single ^ character, it is taken to be the path to file containing a list of hosts, one per line. if the item begins with a / character, it is taken as a regular expression on which to filter the list of hosts (a regex argument may also be optionally trailed by another /, e.g. /node.*/). a regex or file name argument may also be preceeded by a minus - to exclude instead of include thoses hosts.

a list of hosts may also be preceded by "user@" to specify a remote username other than the default, or "rcmd_type:" to specify an alternate rcmd connection type for these hosts. when used together, the rcmd type must be specified first, e.g. "ssh:user1@host0" would use ssh to connect to host0 as user "user1." 🦋

  • -x host,host,... — exclude the specified hosts.

may be specified in conjunction with other target node list options such as -a and -g (when available). hostlists may also be specified to the -x option (see the HOSTLIST EXPRESSIONS section below). arguments to -x may also be preceeded by the filename (^) and regex (/) characters as described above, in which case the resulting hosts are excluded as if they had been given to -w and preceeded with the minus - character.

Standard pdsh options

  • -S — return the largest of the remote command return values.
  • -k — fail fast on connect failure or non-zero return code.
  • -h — output usage menu and quit. a list of available rcmd modules will also be printed at the end of the usage message.
  • -s — only on AIX, separate remote command stderr and stdout into two sockets.
  • -q — list option values and the target nodelist and exit without action.
  • -b — disable ctrl-C status feature so that a single ctrl-C kills parallel job. (batch mode) 😭
  • -l user — this option may be used to run remote commands as another user, subject to authorization. for BSD rcmd, this means the invoking user and system must be listed in the user's .rhosts file (even for root).
  • -t seconds — set the connect timeout. default is 10 seconds. this option may also be set via the PDSH_CONNECT_TIMEOUT environment variable.
  • -u seconds — set a limit on the amount of time a remote command is allowed to execute. default is no limit. see note in LIMITATIONS if using -u with ssh. this option may also be set via the PDSH_COMMAND_TIMEOUT environment variable.
  • -f number — set the maximum number of simultaneous remote commands to number. the default is 32.
  • -R name — set rcmd module to name. this option may also be set via the PDSH_RCMD_TYPE environment variable. a list of available rcmd modules may be obtained via the -h, -V, or -L options. the default will be listed with -h or -V.
  • -M name,... — when multiple misc modules provide the same options to pdsh, the first module initialized "wins" and subsequent modules are not loaded. the -M option allows a list of modules to be specified that will be force-initialized before all others, in-effect ensuring that they load without conflict (unless they conflict with eachother). this option may also be set via the PDSH_MISC_MODULES environment variable.
  • -L — list info on all loaded pdsh modules and quit.
  • -N — disable hostname: prefix on lines of output.
  • -d — include more complete thread status when SIGINT is received, and display connect and command time statistics on stderr when done. also, display any module loading errors to stderr, including option conflicts.
  • -V — output pdsh version information, along with list of currently loaded modules, and exit.

machines module options

  • -a — target all nodes from machines file.

🌸 genders module options 🌸

ok so first, a lil extra beyond the flags themselves: the genders database can also carry an attribute called pdsh_rcmd_type, and if a host has it, that attribute sets an alternate rcmd connect type for that host instead of whatever pdsh's default is. Like, say your genders file has this line:

host0 pdsh_rcmd_type=ssh

then pdsh would use ssh to connect to host0 even if rsh was the default. 💅 You can still override it from the commandline with the "rcmd_type:host0" syntax. icon behavior, fr.

  • -A — targets every single node in the genders database. All of them. If you want some hosts skipped by default, go look at -a below bestie.
  • -a — targets all nodes in the genders database except the ones carrying the "pdsh_all_skip" attribute. It's literally shorthand for running pdsh -A -X pdsh_all_skip ... ✨
  • -g attr[=val][,attr[=val],...] — targets nodes matching any of the genders attributes you name (values optional). ⚠️ Conflicts with the -a option. If you pair it with other node selection options like -w, then -g selects from the node list you supplied instead of from the genders file as a whole. Otherwise? It targets the alternate hostnames in the genders database by default — and the -i option from the genders module can translate those into canonical genders hostnames. If your installed genders supports it, the attributes you hand to -g can also be genders queries. Those query the genders database for the union, intersection, difference, or complement of attributes and values: union is two pipes ('||'), intersection is two ampersands ('&&'), difference is two minus signs ('--'), and complement is a tilde ('~'). Parentheses can change the order of operations. Peep the nodeattr(1) manpage for query examples. 🔥
  • -X attr[=val][,attr[=val],...] — excludes nodes matching any of the specified genders attributes (values optional). Works in combination with any of the other node selection options (e.g. -w, -g, -a). -X can also take the form of genders queries — see the docs for the genders -g option for the deets on those.
  • -i — requests translation between canonical and alternate hostnames. 🦋
  • -F filename — reads genders info from filename instead of the system default genders file. If filename isn't an absolute path, it's taken relative to the directory given by the PDSH_GENDERS_DIR environment variable (/etc by default). You can also point at an alternate genders file with the PDSH_GENDERS_FILE environment variable. 💖

🌈 nodeupdown module options 🌈

  • -v — kicks out target nodes that libnodeupdown considers "down". bye felicia 👋

⭐ slurm module options ⭐

The slurm module lets pdsh target nodes based on currently running SLURM jobs. It's typically called after all the other node selection options have been processed, and if nothing has been selected, the module tries to read a running jobid from the SLURM_JOBID environment variable (which gets set when you're running under a SLURM allocation). If SLURM_JOBID points at an invalid job, it's silently ignored. 💀

  • -j jobid[,jobid,...] — targets the list of nodes allocated to the SLURM job jobid. Can be used multiple times to target multiple SLURM jobs. The special argument "all" targets every node running SLURM jobs, like -j all. 👑
  • -P partition[,partition,...] — targets the list of nodes contained in the SLURM partition partition. Use it multiple times for multiple partitions, and/or hand it partitions in a comma-delimited list.
  • -C feature[,feature,...] — restricts the selected nodes to SLURM nodes that have one of the specified features active. Should be used together with other node selectors, and it filters the resulting list. Any node that doesn't have one of the specified features (or isn't a SLURM node at all) gets excluded. 🚫

🎀 torque module options 🎀

The torque module lets pdsh target nodes based on currently running Torque/PBS jobs. Same energy as slurm: it's typically called after all other node selection options have been processed, and if no nodes have been selected, the module tries to read a running jobid from the PBS_JOBID environment variable (set when you're running under a Torque allocation).

  • -j jobid[,jobid,...] — targets the list of nodes allocated to the Torque job jobid. Use it multiple times to target multiple Torque jobs. 💜

🧚 dshgroup module options 🧚

The dshgroup module lets pdsh use dsh (or Dancer's shell) style group files from /etc/dsh/group/ or ~/.dsh/group/. The default search path can be overridden with the DSHGROUP_PATH environment variable — a colon-separated list of directories to search. Default value for DSHGROUP_PATH is /etc/dsh/group.

  • -g groupname,... — targets nodes in the dsh group file "groupname", found in either ~/.dsh/group/groupname or /etc/dsh/group/groupname. 🌸
  • -X groupname,... — excludes nodes in the dsh group file "groupname."

And here's a lil pdsh glow-up: dshgroup files may optionally include other dshgroup files via a special #include STRING syntax. The argument to #include can be either a file path or a group name — if it's a group name, the path used to search for the group file is the same as if the group had been given to -g. 🫶

💫 netgroup module options 💫

The netgroup module lets pdsh use standard netgroup entries to build lists of target hosts (/etc/netgroup or NIS).

  • -g groupname,... — targets nodes in netgroup "groupname."
  • -X groupname,... — excludes nodes in netgroup "groupname."

✨ ENVIRONMENT VARIABLES ✨

  • PDSH_RCMD_TYPE — equivalent to the -R option. Its value sets the default rcmd module pdsh uses (e.g. ssh, rsh).
  • PDSH_SSH_ARGS — overrides the standard arguments pdsh passes to the ssh(1) command ("-2 -a -x -l%u %h"). Using the parameters %u, %h, and %n (as documented in the rcmd/exec section above) is optional. If those parameters are missing, pdsh appends them to the ssh commandline, because it assumes they're mandatory. no cap.
  • PDSH_SSH_ARGS_APPEND — appends extra options to the ssh(1) command pdsh invokes. So PDSH_SSH_ARGS_APPEND="-q" runs ssh in quiet mode, and "-v" bumps up ssh's verbosity. (Note: these arguments are actually prepended to the ssh commandline so they land before any target hostname argument to ssh.)
  • WCOLL — if no other node selection option is used, you can set this to a filename that pdsh reads a list of target hosts from. The file should have a list of hosts, one per line — though each line may contain a hostlist expression, see HOSTLIST EXPRESSIONS below. 💅
  • DSHPATH — if set, the path in DSHPATH is used as the PATH for the remote processes.
  • FANOUT — sets the pdsh fanout (see the description of -f above).

🌈 HOSTLIST EXPRESSIONS 🌈

Like the sections above mentioned, pdsh accepts lists of hosts in the general form: prefix[n-m,l-k,...], where n < m and l < k, etc., as an alternative to spelling out explicit host lists. Don't get it twisted with regular expression character classes (also written with []). For example, foo[19] is not an expression matching foo1 or foo9 — it's the degenerate hostlist foo19. 🙈

The hostlist syntax is meant purely as a convenience on clusters with a "prefixNNN" naming convention, and specifying ranges should not be considered necessary — the list foo1,foo9 could be written as such, or as the hostlist foo[1,9].

Some examples of usage follow:

Run command on foo01,foo02,...,foo05

pdsh -w foo[01-05] command

Run command on foo7,foo9,foo10

pdsh -w foo[7,9-10] command

Run command on foo0,foo4,foo5

pdsh -w foo[0-5] -x foo[1-3] command

A suffix on the hostname is also supported:

Run command on foo0-eth0,foo1-eth0,foo2-eth0,foo3-eth0

pdsh -w foo[0-3]-eth0 command

Friendly reminder bestie: some shells interpret brackets ('[' and ']') for pattern matching. Depending on your shell, you may need to wrap ranged lists in quotes. For example, in tcsh, that first example above should be run as:

pdsh -w "foo[01-05]" command

💜 ORIGIN 💜

Originally a rewrite of IBM dsh(1) by Jim Garlick <garlick@llnl.gov> on LLNL's ASCI Blue-Pacific IBM SP system. It is now used on Linux clusters at LLNL. 🏆

😭 LIMITATIONS 😭

When you're using ssh for remote execution, expect the stderr of ssh to be folded in with that of the remote command. And when pdsh invokes it, ssh can't prompt for passwords if RSA/DSA keys are configured properly, etc. For ssh implementations that support a connect timeout option, pdsh tries to use that option to enforce the timeout (e.g. -oConnectTimeout=T for OpenSSH) — otherwise, connect timeouts are just not supported when using ssh. 😔 And finally, there is no reliable way for pdsh to ensure remote commands are actually terminated when you use a command timeout. So if -u is used with ssh, commands may be left running on remote hosts even after the timeout has killed the local ssh processes. 💀

The number of nodes pdsh can simultaneously execute remote jobs on is limited by the max number of threads that can be created concurrently, plus the availability of reserved ports in the rsh module. On systems implementing Posix threads, that limit is typically defined by the constant PTHREADS_THREADS_MAX.

⭐ FILES ⭐

ok bestie, this section is literally empty in the original. nothing here. we move. 🎀

👑 SEE ALSO 👑

rsh(1), ssh(1), dshbak(1), pdcp(1)

and that's the whole page, you're officially a pdsh girlie now 🫶 this was pdsh(1) from the linux-gnu crew — go forth and fan out responsibly, mwah 💅✨

📦 where this page comes from