pbsrun(8B)

🎀 pbsrun(8B) PBS Professional 🎀

✨ NAME ✨

pbsrun — general-purpose wrapper script for mpirun. She’s the go-between, the bestie who translates, okay? 💅

📜 SYNOPSIS 📜

pbsrun

pbsrun --version

💖 DESCRIPTION 💖

Okay so pbsrun is a wrapper script for any of several versions of mpirun. The whole point? It gives PBS a user-transparent way to control jobs that call mpirun in their jobscripts. No cap. ✨

The pbsrun_wrap script instantiates pbsrun so that the wrapper script for the specific version of mpirun being used has the same name as that version of mpirun. Cute, right?

If the mpirun wrapper script is run inside a PBS job, it translates any mpirun call of this form:

mpirun [<options>] <executable> [<args>]

into:

mpirun [<options>] pbs_attach [<special options to pbs_attach>] \
        <executable> [<args>]

where special options to pbs_attach means any options pbs_attach needs to do its job (e.g. -j $PBS_JOBID). Slay. 🔥

If the wrapper script is executed outside of PBS, a warning is issued about "not running under PBS", but it proceeds as if the actual program had been called in standalone fashion. Lowkey polite of it. 🫶

The pbsrun wrapper script is not meant to be executed directly. Instead it is instantiated by pbsrun_wrap. It gets copied to the target directory and renamed pbsrun.<mpirun version/flavor>, where mpirun version/flavor is a string that identifies the mpirun version being wrapped (e.g. ch_gm).

The pbsrun script, if executed inside a PBS job, runs an initialization script named $PBS_EXEC/lib/MPI/pbsrun.<mpirun version/flavor>.init, then parses mpirun-like arguments from the command line. It sorts which options and option values to retain, to ignore, or to transform before calling the actual mpirun script with a pbs_attach prefixed to the executable. The actual mpirun to call? Found by tracing the link pointed to by $PBS_EXEC/lib/MPI/pbsrun.<mpirun version/flavor>.link.

For all of the wrapped MPIs, the maximum number of ranks that can be launched is the number of entries in $PBS_NODEFILE. Periodt. 👑

The wrapped MPIs are:

• MPICH-GM's mpirun (mpirun.ch_gm) with rsh/ssh ( deprecated as of 14.2.1)
• MPICH-MX's mpirun (mpirun.ch_mx) with rsh/ssh ( deprecated as of 14.2.1)
• MPICH-GM's mpirun (mpirun.mpd) with MPD ( deprecated as of 14.2.1)
• MPICH-MX's mpirun (mpirun.mpd) with MPD ( deprecated as of 14.2.1)
• MPICH2's mpirun
• Intel MPI's mpirun ( deprecated as of 13.0)
• MVAPICH1's mpirun ( deprecated as of 14.2.1)
• MVAPICH2's mpiexec

⚙️ OPTIONS ⚙️

• --version — pbsrun returns its PBS version information and exits. This option can only be used alone. So don’t try to combo it, bestie. 💀

🧚 INITIALIZATION SCRIPT 🧚

The initialization script is called $PBS_EXEC/lib/MPI/pbsrun.<mpirun version/flavor>.init, where mpirun version/flavor reflects the mpirun flavor/version being wrapped. An administrator can modify it to customize against the local flavor/version of mpirun being wrapped.

Inside this sourced init script, 8 variables are set:

options_to_retain="-optA -optB <val> -optC <val1> val2> ..."
options_to_ignore="-optD -optE <n> -optF <val1> val2> ..."
options_to_transform="-optG -optH <val> -optI <val1> val2> ..."
options_to_fail="-optY -optZ ..."
options_to_configfile="-optX <val> ..."
options_with_another_form="-optW <val> ..."
pbs_attach=pbs_attach
options_to_pbs_attach="-J $PBS_JOBID"

🛠️ Initialization Script Options 🛠️

• options_to_retain — Space-separated list of options and values that pbsrun.<mpirun version/flavor> passes on to the actual mpirun call. Options must begin with - or --, and option arguments must be specified by some arbitrary name with left and right arrows, as in "<val1>".
• options_to_ignore — Space-separated list of options and values that pbsrun.<mpirun version/flavor> does not pass on to the actual mpirun call. Options must begin with - or --, and option arguments must be specified by arbitrary names with left and right arrows, as in "<n>".
• options_to_transform — Space-separated list of options and values that pbsrun modifies before passing on to the actual mpirun call.
• option_to_fail — Space-separated list of options that will cause pbsrun to exit upon encountering a match. (Yep, the original says option_to_fail here while the variable list has options_to_fail — keep that energy. 💀)
• options_to_configfile — Single option and value that refers to the name of the "configfile" containing command line segments found in certain versions of mpirun.
• options_with_another_form — Space-separated list of options and values that can be found in options_to_retain, options_to_ignore, or options_to_transform, whose syntax has an alternate, unsupported form.
• pbs_attach — Path to pbs_attach, which is called before the <executable> argument of mpirun.
• options_to_pbs_attach — Special options to pass to the pbs_attach call. You may pass variable references (e.g. $PBS_JOBID) and they are substituted by pbsrun to actual values.

If pbsrun encounters any option not found in options_to_retain, options_to_ignore, and options_to_transform, then it is flagged as an error. No silent yikes. 😭

🧚 Functions Created 🧚

These functions are created inside the init script. These can be modified by the PBS administrator.

            transform_action () {
                      # passed actual values of $options_to_transform
                      args=$*
                 }

            boot_action () {
                      mpirun_location=$1
                 }

            evaluate_options_action () {
                      # passed actual values of transformed options
                      args=$*
                 }

            configfile_cmdline_action () {
                      args=$*
                 }

            end_action () {
                      mpirun_location=$1
                 }

• transform_action() — The pbsrun.<mpirun version/flavor> wrapper script invokes transform_action() (called once on each matched item and value) with actual options and values received matching one of the "options_to_transform". The function returns a string to pass on to the actual mpirun call.
• boot_action() — Performs any initialization tasks needed before running the actual mpirun call. For instance, GM's MPD requires the MPD daemons to be user-started first. This function is called by the pbsrun.<mpirun version/flavor> script with the location of actual mpirun passed as the first argument. Also, the pbsrun.<mpirun version/flavor> checks for the exit value of this function to determine whether or not to progress to the next step.
• evaluate_options_action() — Called with the actual options and values that resulted after consulting options_to_retain, options_to_ignore, options_to_transform, and executing transform_action(). This provides one more chance for the script writer to evaluate all the options and values in general, and make any necessary adjustments, before passing them on to the actual mpirun call. For instance, this function can specify what the default value is for a missing -np option.
• configfile_cmdline_action() — Returns the actual options and values to be put in before the option_to_configfile parameter.
• configfile_firstline_action() — Returns the item that is put in the first line of the configuration file specified in the option_to_configfile parameter.
• end_action() — Called by pbsrun.<mpirun version/flavor> at the end of execution. It undoes any action done by transform_action(), like cleanup of temporary files. It is also called when pbsrun.<mpirun version/flavor> is prematurely killed. This function is called with the location of actual mpirun passed as first argument.

The actual mpirun program to call is the path pointed to by $PBS_EXEC/lib/MPI/pbsrun.<mpirun version/flavor>.link.

🌸 Modifying *.init scripts 🌸

So administrators can modify *.init scripts without breaking package verification in RPM, master copies of the initialization scripts are named *.init.in. pbsrun_wrap instantiates the *.init.in files as *.init. For instance, $PBS_EXEC/lib/MPI/pbsrun.mpich2.init.in is the master copy, and pbsrun_wrap instantiates it as $PBS_EXEC/lib/MPI/pbsrun.mpich2.init. pbsrun_unwrap takes care of removing the *.init files. Iconic. 💅

🌈 MPIRUN VERSIONS/FLAVORS 🌈

🧵 MPICH-GM's mpirun (mpirun.ch_gm) with rsh/ssh: pbsrun.ch_gm 🧵

SYNTAX

pbsrun.ch_gm <options> <executable> <arg1> <arg2> ... <argn>

Deprecated. This is the PBS wrapper script to MPICH-GM's mpirun (mpirun.ch_gm) with rsh/ssh process startup method.

If executed inside a PBS job, this allows for PBS to track all MPICH-GM processes started by rsh/ssh so that PBS can perform accounting and have complete job control.

If executed outside of a PBS job, it behaves exactly as if standard mpirun.ch_gm was used.

OPTIONS HANDLING

If executed inside a PBS job script, all mpirun.ch_gm options given are passed on to the actual mpirun call with these exceptions:

• -machinefile <file> — The file argument contents are ignored and replaced by the contents of the $PBS_NODEFILE.
• -np — If not specified, the number of entries found in the $PBS_NODEFILE is used.
• -pg — The use of the -pg option, for having multiple executables on multiple hosts, is allowed but it is up to user to make sure only PBS hosts are specified in the process group file; MPI processes spawned are not guaranteed to be under the control of PBS. So yeah, that part is on you. 🙈

WRAP/UNWRAP

To wrap MPICH-GM's mpirun script:

# pbsrun_wrap [MPICH-GM_BIN_PATH]/mpirun.ch_gm pbsrun.ch_gm

To unwrap MPICH-GM's mpirun script:

# pbsrun_unwrap pbsrun.ch_gm

🧵 MPICH-MX's mpirun (mpirun.ch_mx) with rsh/ssh: pbsrun.ch_mx 🧵

SYNTAX

pbsrun.ch_mx <options> <executable> <arg1> <arg2> ... <argn>

Deprecated. This is the PBS wrapper script to MPICH-MX's mpirun (mpirun.ch_mx) with rsh/ssh process startup method.

If executed inside a PBS job, this allows for PBS to track all MPICH-MX processes started by rsh/ssh so that PBS can perform accounting and has complete job control. 💖

If it's run outside of a PBS job, bestie, it behaves exactly as if plain ol' standard mpirun.ch_mx was used — no PBS magic, no extras. 💫

OPTIONS HANDLING 💅

Inside a PBS job script? Every mpirun.ch_gm option you give gets passed on to the actual mpirun call — with a few iconic exceptions:

• -machinefile <file> — the contents of the file argument are ignored, bestie, and swapped out for whatever's in $PBS_NODEFILE. ✨
• -np — not specified? Then the number of entries found in $PBS_NODEFILE is used. Slay. ⭐
• -pg — using -pg (for having multiple executables on multiple hosts) is allowed, but it's on you to make sure only PBS hosts are listed in the process group file. The MPI processes spawned are not guaranteed to be under the control of PBS. No cap. ⚠️

WRAP/UNWRAP 🎀

To wrap MPICH-MX's mpirun script:

# pbsrun_wrap [MPICH-MX_BIN_PATH]/mpirun.ch_mx pbsrun.ch_mx

To unwrap MPICH-MX's mpirun script:

# pbsrun_unwrap pbsrun.ch_mx

---

🌈 MPICH-GM's mpirun (mpirun.mpd) with MPD: pbsrun.gm_mpd

SYNTAX

pbsrun.gm_mpd <options> <executable> <arg1> <arg2> ... <argn>

Deprecated, bestie 😭 — this is the PBS wrapper script for MPICH-GM's mpirun (mpirun.mpd) with the MPD process startup method.

Run it inside a PBS job and PBS gets to track all the MPICH-GM processes started by the MPD daemons — so PBS can do its accounting thing and have complete job control. Iconic behavior, honestly. 👑

Outside of a PBS job? Behaves exactly as if standard mpirun.ch_gm with MPD was used. 💖

OPTIONS HANDLING

Inside a PBS job script, all mpirun.ch_gm-with-MPD options get passed on to the actual mpirun call, except these:

• -m <file> — the file's contents are ignored and replaced by the contents of $PBS_NODEFILE. 🫶
• -np — if you don't specify it, the number of entries found in $PBS_NODEFILE is used. ✨
• -pg — using -pg (multiple executables on multiple hosts) is allowed, but you have to make sure only PBS hosts are listed in the process group file; MPI processes spawned are not guaranteed to be under the control of PBS. 💀

STARTUP/SHUTDOWN 🌸

The script starts MPD daemons on each of the unique hosts listed in $PBS_NODEFILE, using either rsh or ssh based on the value of the environment variable RSHCOMMAND. The default is rsh. 🎀

It also takes care of shutting those MPD daemons down at the end of a run, so you don't have to. Considerate queen behavior. 🧚

WRAP/UNWRAP

To wrap MPICH-GM's mpirun script with MPD:

# pbsrun_wrap [MPICH-GM_BIN_PATH]/mpirun.mpd pbsrun.gm_mpd

To unwrap MPICH-GM's mpirun script with MPD:

# pbsrun_unwrap pbsrun.gm_mpd

---

✨ MPICH-MX's mpirun (mpirun.mpd) with MPD: pbsrun.mx_mpd

SYNTAX

pbsrun.mx_mpd <options> <executable> <arg1> <arg2> ... <argn>

Deprecated again, sorry babe 😭 — this is the PBS wrapper script for MPICH-MX's mpirun (mpirun.mpd) with the MPD process startup method.

Inside a PBS job, PBS can track all the MPICH-MX processes started by the MPD daemons, so it can do its accounting and have complete job control. 💅

Outside of a PBS job? Behaves exactly as if standard mpirun.ch_mx with MPD was used. ✨

OPTIONS HANDLING

Inside a PBS job script, all mpirun.ch_gm-with-MPD options get passed on to the actual mpirun call, with these exceptions:

• -m <file> — the file's contents are ignored and replaced by the contents of $PBS_NODEFILE. 🫶
• -np — if not specified, the number of entries found in $PBS_NODEFILE is used. ⭐
• -pg — allowed (multiple executables on multiple hosts), but it's up to you to make sure only PBS hosts are in the process group file; spawned MPI processes are not guaranteed to be under the control of PBS. 💀

STARTUP/SHUTDOWN 🌸

The script starts MPD daemons on each of the unique hosts listed in $PBS_NODEFILE, using either the rsh or ssh method, based on the value of the environment variable RSHCOMMAND — rsh is the default. 🎀

It also handles shutting down the MPD daemons at the end of a run. Very demure, very mindful. 🧚

WRAP/UNWRAP

To wrap MPICH-MX's mpirun script with MPD:

# pbsrun_wrap [MPICH-MX_BIN_PATH]/mpirun.mpd pbsrun.mx_mpd

To unwrap MPICH-MX's mpirun script with MPD:

# pbsrun_unwrap pbsrun.mx_mpd

---

🦋 MPICH2's mpirun: pbsrun.mpich2

SYNTAX

pbsrun.mpich2 [<global args>] [<local args>] <executable>
              [<args>] [: [<local args>] <executable> [<args>]]
- or -
pbsrun.mpich2 -configfile <configfile>

where configfile contains command line segments as lines:

     [local args] executable1 [args]
     [local args] executable2 [args]
     [local args] executable3 [args]

This is the PBS wrapper script for MPICH2's mpirun. 🎀

Inside a PBS job, PBS gets to track all the MPICH2 processes so it can do its accounting and have complete job control. 💅

Outside of a PBS job? Behaves exactly as standard MPICH2's mpirun would. ✨

OPTIONS HANDLING

Inside a PBS job script, all of MPICH2's mpirun options you give get passed on to the actual mpirun call, except:

• -host and -ghost — for specifying the execution host to run on. Not passed on to the actual mpirun call. 🚫
• -machinefile <file> — the file's contents are ignored and replaced by the contents of $PBS_NODEFILE. 🫶
• MPICH2's mpirun -localonly <x> — for specifying the <x> number of processes to run locally. Not supported, bestie 😭. The user is advised instead to use the equivalent arguments: -np <x> -localonly. Why? Because the pbsrun wrapper script cannot handle a variable number of arguments to an option (like, -localonly has 1 argument and -localonly <x> has 2 arguments). 💀
• -np — if the user did not specify a -np option, then no default value is provided by the PBS wrapper scripts. It's up to the local mpirun to decide what the reasonable default should be — which is usually 1. ⭐

STARTUP/SHUTDOWN 🌸

The script makes sure the MPD daemons on each of the hosts listed in $PBS_NODEFILE are started, and it also makes sure those MPD daemons have been shut down at the end of MPI job execution. So thoughtful. 🧚

WRAP/UNWRAP

To wrap MPICH2's mpirun script:

# pbsrun_wrap [MPICH2_BIN_PATH]/mpirun pbsrun.mpich2

To unwrap MPICH2's mpirun script:

# pbsrun_unwrap pbsrun.mpich2

---

👑 Intel MPI's mpirun: pbsrun.intelmpi

Heads up bestie: wrapping Intel MPI, and support for mpdboot, are deprecated. 😭

SYNTAX

pbsrun.intelmpi [<mpdboot options>] [<mpiexec options>]
                <executable> [<prog args>]
                [: [<mpiexec options>] <executable> [<prog args>]]
- or -
pbsrun.intelmpi [<mpdboot options>] -f <configfile>

where mpdboot options are any options to pass to the mpdboot program — which Intel MPI's mpirun automatically calls to start MPDs — and configfile contains command line segments as lines. 💅

This is the PBS wrapper script for Intel MPI's mpirun. ✨

Inside a PBS job, PBS gets to track all the Intel MPI processes so it can do accounting and have complete job control. 🎀

Outside of a PBS job? Behaves exactly as standard Intel MPI's mpirun would. 💖

Using pbsrun, pbsrun_wrap, and pbsrun_unwrap with Intel MPI is deprecated as of 13.0. 🙈

OPTIONS HANDLING

Inside a PBS job script, all of the options to the PBS interface to MPI's mpirun get passed to the actual mpirun call, except:

• -host and -ghost — for specifying the execution host to run on. Not passed on. 🚫
• -machinefile <file> — the file's contents are ignored and replaced by the contents of $PBS_NODEFILE. 🫶
• mpdboot options --totalnum=* and --file=* — ignored and replaced by the number of unique entries in $PBS_NODEFILE and the name of $PBS_NODEFILE respectively. ⭐
• arguments to mpdboot options --file=* and -f <mpd_hosts_file> — replaced by $PBS_NODEFILE. 💫
• -s — if pbsrun.intelmpi is called inside a PBS job, Intel MPI's mpirun -s argument to mpdboot is not supported, because it closely matches the mpirun option -s <spec>. 😭 The user can simply run a separate mpdboot -s before calling mpirun. A warning message is issued by pbsrun.intelmpi when it hits a -s option, telling users about the supported form. ⚠️
• -np — if the user does not specify a -np option, then no default value is provided by the PBS wrap scripts. It's up to the local mpirun to pick a reasonable default, which is usually 1. 💅

STARTUP/SHUTDOWN 🌸

Intel MPI's mpirun itself takes care of starting/stopping the MPD daemons. 🔥 pbsrun.intelmpi always passes the arguments -totalnum=<number of mpds to start> and -file=<mpd_hosts_file> to the actual mpirun, taking its input from the unique entries in $PBS_NODEFILE. ✨

WRAP/UNWRAP

To wrap Intel MPI's mpirun script:

# pbsrun_wrap [INTEL_MPI_BIN_PATH]/mpirun pbsrun.intelmpi

To unwrap Intel MPI's mpirun script:

# pbsrun_unwrap pbsrun.intelmpi

---

🍓 MVAPICH1's mpirun: pbsrun.mvapich1

SYNTAX

pbsrun.mvapich1 <mpirun options> <executable> <options>

Deprecated 😭. Only one executable can be specified. This is the PBS wrapper script for MVAPICH1's mpirun.

okay so real talk bestie — if you run this inside a PBS job, it lets PBS clock every single MVAPICH1 rank and keep tabs on their resources, so PBS can do its accounting thing and have full job control. 👑

and if you run it outside of a PBS job? it just acts exactly like regular mpirun would. no notes, same vibe, same behaviour. ✨

---

🎀 OPTIONS HANDLING 🎀

when you run this inside a PBS job script, every mpirun option you pass gets handed straight to the real mpirun call — except these chaotic little exceptions:

• -map <list> — this one gets flat-out ignored. not today. 🙈
• -exclude <list> — also ignored, sorry babe. 💅
• -machinefile <file> — ignored too, PBS is not taking notes from you on this one.
• -np — if you don't specify it, the number of entries in $PBS_NODEFILE gets used instead. ty PBS 💖

🦋 WRAP/UNWRAP 🦋

wanna wrap MVAPICH1's mpirun script? say less:

# pbsrun_wrap <path-to-actual-mpirun> pbsrun.mvapich1

and to unwrap it (undo era, we're healing):

# pbsrun_unwrap pbsrun.mvapich1

---

🌈 MVAPICH2's mpiexec: pbsrun.mvapich2 🌈

SYNTAX

pbsrun.mvapich2 <mpiexec args> <executable> <executable's
                args> [: <mpiexec args> <executable> <executable's
                args>]

so you can specify multiple executables using the colon notation — iconic, honestly. this is the PBS wrapper script for MVAPICH2's mpiexec, which has the same format as the real thing.

if you run it inside a PBS job, PBS gets to see all the MVAPICH2 ranks and track their resources, so it can do accounting and keep complete job control. 🔥

if you run it outside of a PBS job, it behaves exactly like the standard mpiexec. same energy. ✨

OPTIONS HANDLING

inside a PBS job script, all your mpiexec options get passed through to the actual mpiexec call, with these two exceptions, bestie:

• -host <host> — the contents of the host argument are ignored. we don't know her. 💀
• -machinefile <file> — the file argument's contents are ignored and swapped out for the contents of $PBS_NODEFILE. PBS said I'll handle this one. 🫶

WRAP/UNWRAP

wrapping MVAPICH2's mpiexec script:

# pbsrun_wrap <path-to-actual-mpiexec> pbsrun.mvapich2

unwrapping it:

# pbsrun_unwrap pbsrun.mvapich2

---

🌸 REQUIREMENTS 🌸

the mpirun you're wrapping has to be installed and working on every single node in the PBS cluster. all of them. no half-slay allowed. 😭

---

💜 ERRORS 💜

if pbsrun runs into any option that isn't found in options_to_retain, options_to_ignore, or options_to_transform, it gets flagged as an error. periodt. 🚨

---

⭐ SEE ALSO ⭐

• pbs_attach(8B)
• pbsrun_wrap(8B)
• pbsrun_unwrap(8B)

go read those next, iykyk 🧚

---

that's the whole page bestie, you ate that 🍓 this one's the Local flavoured pbsrun(8B) from 6 May 2020 — mwah, and good luck out there 💅✨