paexec(1)

📦
version 1.1.6 · 3 versions
, in opensuse-leap 16.0

🌸 NAME 🌸

okay bestie, welcome to paexec(1) — the parallel executor that literally distributes tasks over your network or CPUs. no cap, it's giving main character energy for your cluster 💅✨

🦋 SYNOPSIS 🦋

paexec [options]

paexec -C [options] command [args...]

✨ DESCRIPTION ✨

okay so imagine you have a long list of tasks that need to be done — like, say, you want to convert thousands of .wav audio files to .ogg format 🎧. also imagine you have multiple CPUs available, like a multi-CPU system or a cluster made of individual computers connected to the network or internet. paexec can efficiently do this work. fr fr, it efficiently distributes different tasks to different processors (or computers), receives the results of processing from them, and sends these results to stdout. iconic behavior, honestly 💖

there are several notions that should be defined: task, command, transport, node. let's get into it 🫶

  • task — tasks are read by paexec from stdin and are represented as one line of text, i.e. one input line = one task. simple as that ✨
  • node identifier — the remote computer or CPU identifier, for example a CPU ordinal number or a computer's DNS name like node12.cluster.company.com 🖥️
  • command — the user's program that reads a one-line task from stdin and sends a multiline result to stdout ending with an end of task (EOT) marker. the EOT marker means “Task is done. I am ready for the next one”. after sending EOT to stdout, stdout MUST be flushed. remember that the EOT marker MUST NOT appear in the general result — otherwise, paexec may hang due to deadlock 😭💀 the default EOT is an empty line. the command may use the environment variable PAEXEC_EOT for the end-of-task marker.
  • transport — a special program that helps to run the command on a node. it takes the node identifier as its first argument and the command with its arguments as the rest. good examples for transport are ssh and rsh. both transport and command may be specified with their arguments, e.g. /usr/bin/ssh -x is allowed as a transport program 🌈

How paexec works. commands are run on each node with a help of the transport program. then tasks are read from stdin one-by-one and sent to a free node (exactly one task per node at a time). at the same time, result lines are read from the command's stdout and are output to paexec's stdout. when the EOT marker is obtained from the node, it is marked as free and becomes ready for the next task. these steps repeat until the end of stdin is reached and all nodes finish their job. periodt 🎀

More formally (to better understand how paexec works):

          run_command_on_each_node
          mark_all_nodes_as_free
          while not(end_of_stdin) or not(all_nodes_are_free)
             while there_is_free_node/i and not(end_of_stdin)
                task = read_task_from_stdin
                send_task_to_node(task, i)
                mark_node_as_busy(i)
             end
             while result_line_from_node_is_available/i
                result = read_result_line_from_node(i)
                send_line_to_stdout(result)
                if is_EOT(result)
                   mark_node_as_free(i)
                end
             end
          end
          close_command_on_each_node

note that commands are run once per node — it is not restarted for every task 👑.

also note that the output contains result lines (obtained from different nodes) in the mixed order. that is, the first line of the output may contain a result line obtained from the first node, the second line of output — from the second node, but the third output line may contain a result line from the first node again 🙈. it is also not guaranteed that the first line of output will be from the first node or from the first task. all result lines are output as soon as they are read by paexec, i.e. as soon as they are ready. paexec works this way for the efficiency reasons. you can play with -l, -r and -p options to see what happens 👀. for reordering output line you can use the paexec_reorder utility 🦋

✨ OPTIONS ✨

Okay bestie, this is the part where paexec shows you every little knob and dial it has. There are a LOT. Grab a snack. 🍓

  • -h — shows help info. That's it, that's the whole vibe. 💖
  • -V — shows version info. Gotta know what you're working with, periodt.
  • -c command — the command plus all its arguments. Like, the whole thing.
  • -C — says "actually, the command and its arguments are in the free arguments," so you don't need -c at all. Iconic shortcut energy.
  • -t transport — the transport program. The little courier that carries your tasks around. 🚚
  • -n +number — how many commands to run in parallel. Math! 🔥
  • -n nodes — a list of nodes separated by space or tab characters. Heads up: the first character must be alphanumeric, _ or /. All the other characters are reserved for future extensions (mysterious, iykyk). Leading and trailing spaces and tabs get ignored.
  • -n :filename — a filename that contains the list of nodes, one per line. Very organized of you. 🎀
  • -x — runs the command specified by -c for each task. Its stdout gets sent to paexec. And ngl, if you specify both -x and -g, the task is considered failed if the command's exit status is non-zero. Noted.
  • -X — implies -x and just… ignores the command's stdout. Like, "I don't wanna hear it." 🙈
  • -r — includes the node identifier or node number (0-based) in the output, i.e. the id/number of the node that produced that particular output line. If -l is also applied, this identifier or number shows up before the line number. Space character is the separator, always.
  • -l — includes a 0-based task number (that's the input line number) in the output, i.e. the line number that this particular output line came from. If -p is also applied, it shows up before the pid. Space character = separator. Again.
  • -p — includes the pid of paexec's subprocess that communicates with node+command in the output. The pid prepends the actual result line. Space character, separator, you know the drill by now.
  • -e — when the end-of-task marker comes back from a node, the EOT marker gets output to stdout. Honestly super useful together with -l and/or -r. ✨
  • -E — implies -e AND flushes stdout. Fancy.
  • -d — turns on debugging mode. For debugging purposes only, bestie, don't make it your whole personality. 💀
  • -i — copies input lines (i.e. the tasks) to stdout.
  • -I — implies -i and flushes stdout. Same energy as -E.
  • -s | -g — an orgraph of tasks (a partially ordered set, no big deal) is read from stdin.

So instead of autonomous tasks, the graph of tasks is read from stdin. In this mode every task can either FAIL or SUCCEED. As always, an empty line output by the command means end of task. The line before it shows an EXIT STATUS of the task. The word "failure" means failure, "success" means success, and "fatal" means the current task gets reassigned to another node (and restarted, of course) — see option -z. There's a sample over at examples/1_div_x/cmd if you wanna peek. An input line (paexec's stdin) should contain either a single task with no spaces inside, or two tasks separated by a single space character, e.g. task1<SPC>task2. A task1<SPC>task2 line means task1 must be done before task2, and it is mandatory — that is, if task1 fails, all dependent tasks (including task2) are also failed recursively. Tasks that have dependencies only start after all their dependencies have succeeded. When a task succeeds, paexec outputs the word "success" just before the end_of_task marker (see -e or -E); otherwise the word "failure" is output, followed by a list of tasks failed because of it.

Samples, bestie, look at these:

tasks (examples/make_package/tasks file)

textproc/dictem
devel/autoconf wip/libmaa
devel/gmake wip/libmaa
wip/libmaa wip/dict-server
wip/libmaa wip/dict-client
devel/m4 wip/dict-server
devel/byacc wip/dict-server
devel/byacc wip/dict-client
devel/flex wip/dict-server
devel/flex wip/dict-client
devel/glib2
devel/libjudy

command (examples/make_package/cmd__flex)

#!/usr/bin/awk -f
{
   print $0
   if ($0 == "devel/flex")
      print "failure"
   else
      print "success"

   print ""       # end of task marker
   fflush()
}

output of paexec -s -l -c cmd__flex -n +10 < tasks

3 devel/autoconf
3 success
4 devel/gmake
4 success
7 devel/m4
7 success
8 devel/byacc
8 success
9 devel/flex
9 failure
9 devel/flex wip/dict-server wip/dict-client
10 devel/glib2
10 success
11 devel/libjudy
11 success
1 textproc/dictem
1 success
2 wip/libmaa
2 success
  • -z — if applied, read/write(2) operations from/to nodes become not critical. In case paexec loses connection to the node, it reassigns the failed task to another node and, if -s is applied, outputs the "fatal" string to stdout ("success" + "failure" + "fatal"). This makes paexec resistant to I/O errors, which means you can create paexec clusters even over a network made of unreliable hosts (the Internet?! 😭). Failed hosts get marked as such and won't be used during the current run of paexec. Resilient queen behavior. 👑
  • -Z timeout — when -z is applied, if a command fails, the appropriate node gets marked as broken and is excluded from the following task distribution. But if -Z is applied, every timeout seconds an attempt is made to rerun a command on a failed node. -Z implies -z. This option makes it possible to organize clusters over unreliable networks/hardware. Trying again is self care, fr.
  • -w — if the -Z option was applied, paexec exits with an error if ALL nodes failed. With -w, it will not exit — it will wait for nodes to be restored. Patient. 🧚
  • -m s=success — sets an alternative string for the 'success' message.
  • -m f=failure — sets an alternative string for the 'failure' message.
  • -m F=fatal — sets an alternative string for the 'fatal' message. An empty string for 'fatal' means it will not be output to stdout in case of fatal error. Silence is golden. 💅
  • -m t=eot — sets an alternative string for the EOT message.
  • -m d=delimiter — sets an alternative string for the tasks delimiter (for -g mode). Heads up: the delimiter should be at most one character. No delimiter means an entire input line is treated as a task.
  • -m w=weight — sets an alternative string for the "weight:" message.
  • -W num — when multiple machines or CPUs are used for task processing, it makes sense to start "heavier" tasks as soon as possible to minimize total calculation time. If -W is specified, a special weight is assigned to each task, which is used for reordering tasks. If num is 0, the weights themselves are used for reordering tasks — and the bigger the weight, the more priority the task has. If num is 1, the total weight of a task is the sum of its own weight (specified on input) and the weights of all tasks depending on it directly or indirectly. If num is 2, the total weight of a task is the maximum value of the task's own weight and the weights of all tasks depending on it directly or indirectly. Weights are specified with the help of the "weight:" keyword (unless -m w= was specified). If weight is not specified, it defaults to 1. Here's the example for an input graph of tasks with weights:
weight: gtk2 30
weight: glib2 20
gtk2 firefox
weight: firefox 200
glib2 gtk2
weight: qt4 200
weight: kcachegrind 2
qt4 kcachegrind
qt4 djview4
tiff djview4
png djview4
weight: twm 1
weight: gqview 4
  • -y — if applied, the magic string is used as an end-of-task marker instead of an empty line. It's unlikely that this line appears in the command's output. This option has higher priority than the PAEXEC_EOT environment variable. Priority, bestie. ⭐
  • -0 — changes paexec to expect a NUL character as the line separator instead of newline. This is expected to be used in concert with the -print0 function in find(1). Classic combo.
  • -J replstr — executes the command for each task, replacing one or more occurrences of replstr with the entire task. Only a 2-character replstr is allowed. Take a note that such replacement works only if shell variable expansion is allowed in the appropriate part of the command (if -c is applied), or if the free argument is exactly a replstr (if -C is applied). This option implies -x. Sneaky but slay.

🌈 EXAMPLES 🌈

  1. The whole cluster situation:
paexec -t '/usr/bin/ssh -x' -n 'host1 host2 host3' \
       -le -g -c calculate-me < tasks.txt |
paexec_reorder -Mf -Sl
  1. Encoding your wavs in parallel, no cap:
ls -1 *.wav | paexec -x -n +4 -c 'oggenc -Q'
  1. Same but flac-y and with all the little output flags:
ls -1 *.wav | paexec -xCil -n+4 flac -f --silent
  1. Banner-ing your uname info, because why not:
{ uname -s; uname -r; uname -m; } |
paexec -x -lp -n+2 -c banner |
paexec_reorder -l
  1. NUL-separated find output going straight into scp, chef's kiss:
find . -name '*.dat' -print0 |
paexec -0 -n+10 -C -J// scp // remoteserver:/remote/path
  1. Uppercasing your filenames with awk magic:
ls -1 *.txt | paexec -n+10 -J%% -c 'awk "BEGIN {print toupper(\"%%\")}"'

For more examples see paexec.pdf and the examples/ subdirectory in the distribution. 💜

🌸 NOTES 🌸

The select(2) system call and non-blocking read(2) are used to read result lines from nodes. Very async of them.

At the moment, blocking write(2) is used to send tasks to the node. This may slow down an entire processing if tasks are too big. So it's recommended to use shorter tasks — for example, a filename or URI (several tens of bytes in size) instead of multi-megabyte content. Though this may be fixed in the future. Fingers crossed. 🤞

The original paexec tarball contains a number of samples of use in the presentation/paexec.pdf file. After installation you can find this file under share/doc/paexec/paexec.pdf or nearby. Go read it, it's giving documentation glamour.

🦋 ENVIRONMENT 🦋

  • PAEXEC_BUFSIZE — overrides the compile time initial size for internal buffers used to store tasks and the result lines. Versions of paexec prior to 0.9.0 used this value as a maximum buffer size. Now internal buffers are resized automatically. If unsure, do not set the PAEXEC_BUFSIZE variable. See the default value in Makefile. Honestly? Leave it alone unless you know what you're doing. 🙈
  • PAEXEC_ENV — a list of variables passed to the command.
  • PAEXEC_EOT — this variable sets the end-of-task marker, which is an empty line by default. Also, through this variable an end-of-task marker is passed to all commands.
  • PAEXEC_NODES — unless option -n was applied, this variable specifies the nodes.
  • PAEXEC_SH — this variable sets the shell interpreter used inside paexec. By default it is /bin/sh. 🐚
  • PAEXEC_TRANSPORT — unless option -t was applied, this variable specifies the transport.

💌 BUGS/FEEDBACK

okay so if you find a bug — or honestly if you just have thoughts — please tell me!! 🫶 You can send any comments, questions, bug reports etc. straight to my inbox by e-mail, or (even better) register them at the sourceforge project home. Like, the tracker exists for a reason bestie, use it 💅

Feature requests are also welcomed. fr fr, if there's something you wish paexec did, say it out loud — I wanna hear it ✨

🏠 HOME

The mothership, the main character, the place to be:

<http://sourceforge.net/projects/paexec>

go click it, iykyk 🌈

👀 SEE ALSO

your required reading for the group chat, no skipping:

like literally go read them, they're the whole vibe 🦋

and that's the whole page, bestie!! you made it to the end, iconic behaviour honestly 👑 this little beauty is paexec(1), dated 2024-06-07 — screenshot it, cite it, tattoo it on your forearm, whatever feels right 💖✨🎀

📦 where this page comes from