CLUSH(1)

📦
version 1.10.1 · 5 versions
, in arch rolling, epel 10, epel 9, fedora 43, fedora 44, ubuntu stonking

💖 CLUSH(1) 💖 — ClusterShell User Manual (bestie edition)

okay so you found the clush manpage and you're like "what is this" — don't worry, I got you. this is clush from the ClusterShell User Manual, and it's genuinely one of my faves. let's go 🫶

✨ NAME ✨

clush — execute shell commands on a cluster 🖥️🖥️🖥️

🌸 SYNOPSIS 🌸

clush -a | -g group | -w nodes  [OPTIONS]

clush -a | -g group | -w nodes  [OPTIONS] command

clush -a | -g group | -w nodes  [OPTIONS] --copy file | dir [ file | dir ...] [ --dest path ]

clush -a | -g group | -w nodes  [OPTIONS] --rcopy file | dir [ file | dir ...] [ --dest path ]

like, pick your target (-a or -g group or -w nodes), then either give it a command, or go into --copy / --rcopy mode. that's the whole vibe. 💅

🦋 DESCRIPTION 🦋

so clush is a program for executing commands in parallel on a cluster and for gathering their results. it can run commands interactively, or it can be used inside shell scripts and other applications — both, she contains multitudes. ✨

she's a partial front-end to the ClusterShell library, and the whole point of that is a light, unified and robust parallel command execution framework. translation: your traditional shell scripts get to benefit from some of the library's features without you having to rewrite everything. iconic behaviour honestly.

and ngl, clush currently makes use of the Ssh worker of ClusterShell, by default, which only requires ssh(1) (the OpenSSH SSH client). that's it. that's the whole dependency. we love a low-maintenance queen. 👑

🌈 INVOCATION 🌈

so you can start clush non-interactively to run a shell command, or you can invoke it as an interactive shell. wanna start a clush interactive session? just invoke the clush command without providing command. easy.

💜 Non-interactive mode

when clush is started non-interactively, the command is executed on the specified remote hosts in parallel. parallel!! the way she does all of them at once.

if option -b or --dshbak is specified, clush waits for command completion and then displays gathered output results. so it's like, hold on, let me collect everything and THEN show you the receipts. 📋

the -w option lets you specify remote hosts by using ClusterShell NodeSet syntax, and that includes the node groups @group special syntax AND the Extended Patterns syntax so you can benefit from NodeSet basic arithmetic (like @Agroup&@Bgroup). go read EXTENDED PATTERNS in nodeset(1) and also groups.conf(5) for more info, iykyk. 📚

now listen, this part matters: unless the option --nostdin (or -n) is specified, clush detects when its standard input is connected to a terminal (as determined by isatty(3)).

  • if it is actually connected to a terminal, clush listens to standard input while commands are running, waiting for an Enter key press. doing that will display the status of current nodes. so press Enter, get status. cute.
  • if standard input is not connected to a terminal, and unless the option --nostdin is specified, clush binds the standard input of the remote commands to its own standard input — which is what makes scripting methods like this possible:
# echo foo | clush -w node[40-42] -b cat
---------------
node[40-42]
---------------
foo

and there are some other great examples waiting for you in the EXAMPLES section below. go look, fr fr. 👀

🎀 Interactive session

if a command is not specified, and its standard input is connected to a terminal, clush runs interactively. main character mode, activated.

in this mode, clush uses the GNU readline library to read command lines. Readline provides commands for searching through the command history for lines containing a specified string. for instance, type Control-R to search in the history for the next entry matching the search string typed so far. chef's kiss. 🔍

clush also recognizes special single-character prefixes that allow the user to see and modify the current nodeset — that's the nodes where the commands are executed, btw.

Single-character interactive commands are:

  • clush> ? — show current nodeset
  • clush> @<NODESET> — set current nodeset
  • clush> +<NODESET> — add nodes to current nodeset
  • clush> -<NODESET> — remove nodes from current nodeset
  • clush> !COMMAND — execute COMMAND on the local system
  • clush> = — toggle the output format (gathered or standard mode)

to leave an interactive session, type quit or Control-D. bye bestie 👋

🧚 Local execution ( --worker=exec or -R exec )

instead of running the provided command on remote nodes, clush can use the dedicated exec worker to launch the command locally, for each node. yes, locally. plot twist.

some parameters could be used in the command line to make a different command for each node:

  • %h or %host will be replaced by the node name
  • %n or %rank will be replaced by the remote rank [0-N]
  • to get a literal % use %%
  • %hosts will be replaced by the full target nodeset, in which case the command is executed only once

🍓 File copying mode ( --copy )

when clush is started with the -c or --copy option, it will attempt to copy specified files and/or directories to the provided cluster nodes.

the --dest option can be used to specify a single path where all the file(s) should be copied to on the target nodes.

in the absence of --dest, clush will attempt to copy each file or directory found in the command line to their same location on the target nodes. so, like, it keeps them where they were. 📁✨

💫 Reverse file copying mode ( --rcopy )

when clush is started with the --rcopy option, it will attempt to retrieve specified file and/or dir from the provided cluster nodes. bringing it home, as it should. 🏡

if the --dest option is specified, it must be a directory path where the files will be stored with their hostname appended. note that. it matters.

if the destination path is not specified, it will take each file or directory's parent directory as the local destination.

✨ OPTIONS ✨

okay bestie, this is the big one — every single flag, no skips, no shortcuts 💅

  • --version — spits out the clush version number and dips out immediately. that's it, that's the whole job.
  • -s GROUPSOURCE, --groupsource=GROUPSOURCE — if you wanna point at a specific group source from groups.conf(5), this is the optional little dial for that.
  • -n, --nostdin — tells clush to not sit there watching for input from stdin. use this when you're running clush in the background (or in scripts), because otherwise it's just standing there waiting like 👀 for a stdin that's never coming.
  • --groupsconf=FILE — swap in a different config file for groups.conf(5).
  • --conf=FILE — same energy but for clush.conf(5).
  • -O <KEY=VALUE>, --option=<KEY=VALUE> — override literally any key=value option from clush.conf(5). repeat it as many times as your heart desires.

🎯 Selecting target nodes

  • -w NODES — the nodes where you actually wanna run the command. the main character of node selection.
  • -x NODES — nah, exclude these ones from the node list.
  • -a, --all — run it on all the nodes. every single one. no cap.
  • -g GROUP, --group=GROUP — run it on a whole group of nodes instead of listing them out.
  • -X GROUP — exclude nodes from this group. bye ✋
  • --hostfile=FILE, --machinefile=FILE — path to a file that has a list of single hosts, node sets or node groups, separated by spaces and lines. you can pass it multiple times, one per file, if you're feeling extra.
  • --topology=FILE — the topology config file to use for tree mode 🌳
  • --pick=N — randomly pick N node(s) out of the nodeset. rng but make it infrastructure.

🌈 Output behavior

  • -q, --quiet — be quiet. only the essential output, nothing more. lowkey mysterious.
  • -v, --verbose — be verbose, print the informative messages.
  • -d, --debug — even more messages, for debugging purposes. chaotic but useful.
  • -G, --groupbase — don't show the group source prefix (it's always @groupname).
  • -L — disables the header block and orders output by nodes. if -b/-B isn't specified, clush will wait for all commands to finish and then display aggregated output of commands with the same return codes, ordered by node name. alternatively, when you pair it with -b/-B (like -bL), clush enables a "live gathering" of results by line, so the next line shows up as soon as possible (e.g. when all nodes have sent the line). it's giving real-time 💖
  • -N — disables labeling of command output.
  • -P, --progress — show progress while the command runs. and if writing is happening to standard input, the live progress indicator will show the global bandwidth of data written to the target nodes. bandwidth moment ✨
  • -b, --dshbak — display gathered results in a dshbak-like way. (note: it will only try to aggregate the output of commands with the same return codes.)
  • -B — same as -b but it includes standard error too. the fuller picture.
  • -r, --regroup — fold the nodeset using node groups.
  • --axis=RANGESET — for nD nodesets, fold the displayed nodeset (e.g. in gathered output headers) along the provided axis only. axes are indexed from 1 to n and you can specify them either with the rangeset syntax, e.g. '1', '1-2', '1,3', or with a single negative number, which means the index is counted from the end. because some nodesets may have several different dimensions, axis indices get silently truncated to fall in the allowed range. this is the per-invocation equivalent of the fold_axis library default.
  • -S, --maxrc — return the largest of the command return codes. biggest number wins 👑
  • --color=WHENCOLOR — clush can use the NO_COLOR, CLICOLOR and CLICOLOR_FORCE environment variables. NO_COLOR takes precedence over CLICOLOR_FORCE, which takes precedence over CLICOLOR. when the --color option is used, those environment variables are not taken into account. --color tells whether to use ANSI colors to surround the node or nodeset prefix/header with escape sequences so they show up in color on the terminal. WHENCOLOR is never, always or auto (which uses color if standard output/error refer to a terminal). colors are set to [34m (blue foreground text) for stdout and [31m (red foreground text) for stderr, and they cannot be modified. periodt.
  • --diff — show the diff between common outputs (it finds the best reference output by focusing on the largest nodeset and also the smaller command return code).
  • --outdir=OUTDIR — output directory for stdout files (OPTIONAL).
  • --errdir=ERRDIR — output directory for stderr files (OPTIONAL).

📁 File copying

  • -c, --copy — copy a local file or directory over to the remote nodes.
  • --rcopy — copy a file or directory from the remote nodes. reverse uno 🦋
  • --dest=DEST_PATH — destination file or directory on the nodes. optional: when you don't specify it, it uses the first source directory path.
  • -p — preserve modification times and modes.

🔌 Connection options

  • -f FANOUT, --fanout=FANOUT — don't execute more than FANOUT commands at the same time. super useful for limiting resource usage. in tree mode, the same fanout value is used on the head node and on each gateway (the fanout value is propagated). so like, if the fanout is 16, each gateway will initiate up to 16 connections to their target nodes at the same time. the default fanout value is defined in clush.conf(5).
  • -l USER, --user=USER — execute the remote command as this user.
  • -o OPTIONS, --options=OPTIONS — for giving ssh options, e.g. -o "-p 2022 -i ~/.ssh/myidrsa". these options are added first to ssh and they override the default ones. main character energy, they go in front 💅
  • -t CONNECT_TIMEOUT, --connect_timeout=CONNECT_TIMEOUT — limit the time allowed to connect to a node.
  • -u COMMAND_TIMEOUT, --command_timeout=COMMAND_TIMEOUT — limit the time allowed for the command to run on the node.
  • -m MODE, --mode=MODE — run mode; define MODEs in <confdir>/*.conf.
  • -R WORKER, --worker=WORKER — the worker name to use for the connection (exec, ssh, rsh, pdsh, or the name of a Python worker module). default is ssh.
  • --remote=REMOTE — whether to enable remote execution. in tree mode, 'yes' forces connections to the leaf nodes for execution, while 'no' establishes connections up to the leaf parent nodes for execution (default is 'yes').

for a short explanation of these options, see -h, --help.

💀 EXIT STATUS

by default, an exit status of zero means the clush command succeeded — but it gives you zero information about the remote commands' exit status. rude, honestly.

however, when you specify the -S option, the exit status of clush becomes the largest value of the remote commands' return codes.

for failed remote commands whose exit status is non-zero — and unless you've got the combination of options -qS going — clush shows you messages kinda like this:

clush: node[40-42]: exited with exit code 1

🌸 EXAMPLES

Remote parallel execution

# clush -w node[3-5,62] uname -r

run the command uname -r in parallel on nodes: node3, node4, node5 and node62.

Local parallel execution

# clush -w node[1-3] --worker=exec ping -c1 %host

run locally, in parallel, a ping command for nodes: node1, node2 and node3. you can also use -R exec as the shorter and pdsh-compatible option. 🫶

Display features

# clush -w node[3-5,62] -b uname -r

run uname -r on nodes[3-5,62] and display the gathered output results (integrated dshbak-like).

# clush -w node[3-5,62] -bL uname -r

line mode: run uname -r on nodes[3-5,62] and display gathered output results without the default header block.

# ssh node32 find /etc/yum.repos.d -type f | clush -w node[40-42] -b
xargs ls -l

search some files on node32 in /etc/yum.repos.d and use clush to list the matching ones on node[40-42], then use -b to display the gathered results.

# clush -w node[3-5,62] --diff dmidecode -s bios-version

run this Linux command to get the BIOS version on nodes[3-5,62] and show version differences (if any). 👀

# clush -w dc[1-2]n[1-2] --axis=1 -b uname -r

gather output and fold the displayed nodeset header along axis 1 only, e.g. show dc[1-2]n1,dc[1-2]n2 rather than dc[1-2]n[1-2] in the header block. see also fold_axis in the Library Defaults configuration if you wanna change it permanently.

All nodes

# clush -a uname -r

run uname -r on all cluster nodes — check groups.conf(5) to set up all cluster nodes (the all: field).

# clush -a -x node[5,7] uname -r

run uname -r on all cluster nodes except node5 and node7. not them. 🙈

# clush -a --diff cat /some/file

run cat /some/file on all cluster nodes and show differences (if any), line by line, between the common outputs.

Node groups

# clush -w @oss modprobe lustre

run modprobe lustre on the nodes from the node group named oss — see groups.conf(5) to set up node groups (the map: field).

# clush -g oss modprobe lustre

same as the previous example, but using -g so you don't have to deal with the @ group prefix.

# clush -w @mds,@oss modprobe lustre

you can specify several node groups by separating them with commas (please see EXTENDED PATTERNS in nodeset(1) and also groups.conf(5) for more info). comma-separated, iconic. 🎀

Copy files

# clush -w node[3-5,62] --copy /etc/motd

copy the local file /etc/motd to the remote nodes node[3-5,62].

# clush -w node[3-5,62] --copy /etc/motd --dest /tmp/motd2

copy the local file /etc/motd to the remote nodes node[3-5,62] at path /tmp/motd2.

# clush -w node[3-5,62] -c /usr/share/doc/clustershell

recursively copy the local directory /usr/share/doc/clustershell to the same path on the remote nodes node[3-5,62].

# clush -w node[3-5,62] --rcopy /etc/motd --dest /tmp

copy /etc/motd from the remote nodes node[3-5,62] to the local /tmp directory — each file gets its remote hostname appended, e.g. /tmp/motd.node3. cute little naming convention, we love to see it 💜

📁 FILES (the lore of where everything lives) 📁

okay bestie, here's the map of every config file and history file clush touches, in the order it looks at them:

  • $CLUSTERSHELL_CFGDIR/clush.conf — the global clush configuration file, the big one for the whole system. but if $CLUSTERSHELL_CFGDIR isn't defined? then it falls back to /etc/clustershell/clush.conf instead. no cap. 💅
  • $XDG_CONFIG_HOME/clustershell/clush.conf — your user config file for clush. and if $XDG_CONFIG_HOME isn't defined, it uses $HOME/.config/clustershell/clush.conf instead. it's giving "we have options, and backups for our options." ✨
  • {sys.prefix}/etc/clustershell/clush.conf — the config file used when ClusterShell is installed inside a Python virtual environment. venv girlies, this one's for you 🧚
  • $HOME/.local/etc/clustershell/clush.conf — the local user configuration file for clush, aka the default installation spot for pip --user. very "I do my own thing, locally" of it 🎀
  • ~/.clush.conf — the deprecated per-user clush config file. and when we say deprecated we mean it, bestie: it is no longer read since version 1.9. gone, retired, it did its time 💀
  • ~/.clush_history — the file where your interactive clush command history gets saved. every iconic command you ever typed, preserved forever like a scrapbook 📖💖

👀 SEE ALSO (aka the friend group) 👀

and the docs, which are literally right here:

https://clustershell.readthedocs.io/

🐛 BUG REPORTS (tell them, don't suffer in silence) 🐛

found something cursed? got feedback? use this URL to submit a bug report or just say your piece:

https://github.com/clustershell/clustershell/issues
bestie, bugs are not a vibe check you pass alone. report it, the maintainers want to know 🫶

👑 AUTHOR 👑

Stephane Thiell

GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)

and that's the whole page, babe 🦋 this was version 1.10.1, dated 2026-07-16, and it's the CLUSH(1) manual — read it, love it, clush responsibly 💜✨

📦 where this page comes from