MPIRUN(1)

📦 version 7.1.4, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking

MPIRUN(1) — LAM COMMANDS 💅

okay bestie, sit down, hydrate, because this is THE page. this is the one you're gonna be coming back to like it's your comfort show. it's mpirun, it lives in the LAM command set, and it's how you actually get your MPI program out into the world. let's go 🦋

🌸 NAME 🌸

mpirun — run MPI programs on LAM nodes. that's it, that's the whole job, and she does it flawlessly 💖

✨ SYNOPSIS ✨

here's the first form — the one for when you're launching an actual program:

mpirun [-fhvO] [-c # | -np #] [-D | -wd dir] [-ger | -nger] [-sigs |
-nsigs] [-ssi key value] [-nw | -w] [-nx] [-pty | -npty] [-s node] [-t |
-toff | -ton] [-tv] [-x VAR1[=VALUE1][,VAR2[=VALUE2],...]]  [[-p
prefix_str] [-sa | -sf]] [where] program [-- args]
heads up queen: even though every single one of those is individually optional, you still gotta give at least one of where, -np, or -c in the form above. like, when a schema isn't being used. that's non-negotiable bestie 😭

and here's the second form — the one where you hand it an application schema instead:

mpirun [-fhvO] [-D | -wd dir] [-ger | -nger] [-sigs | -nsigs] [-ssi key
       value] [-nw | -w] [-nx] [-pty | -npty] [-t | -toff | -ton] [-tv]
       [-x VAR1[=VALUE1][,VAR2[=VALUE2],...]]  schema
⚠️ PSA ⚠️ the -c2c and -lamd options are obsolete now. do not use them. use -ssi instead. see the "SSI" section below, iykyk 👑

⭐ QUICK SUMMARY ⭐

if you're just here like "bestie i literally only want to know how to run my MPI app," this is your section. no cap, this is probably the command line you want:

% mpirun C my_mpi_application

this runs one copy of my_mpi_application on every CPU in the current LAM universe. periodt. 💅

alternatively, N can go where C is, which means "one copy of my_mpi_application on every node" (as opposed to every CPU) in the current LAM universe. subtle difference, big vibes.

and finally:

% mpirun -np 4 my_mpi_application

this one tells LAM to explicitly run four copies of my_mpi_application, scheduling in a round-robin fashion by CPU across the LAM universe. 🌀

the rest of this page has more details, especially the "Location Nomenclature" section — that one's the main character, don't skip her. 👑

🎀 OPTIONS 🎀

okay so there are two forms of the mpirun command — one for programs (i.e., SPMD-style applications), and one for application schemas (see appschema(5)). 🫶

both forms of mpirun use these options by default: -nger -w. each of those can be overridden by their counterpart options, described below.

additionally, mpirun will send the name of the directory where it was invoked on the local node to each of the remote nodes, and attempt to change to that directory. see the "Current Working Directory" section, below. ✨

  • -c # — synonym for -np (see below). same energy, different letter 💫
  • -D — use the executable program location as the current working directory for created processes. the cwd of the created processes gets set before the user's program is invoked. this option is mutually exclusive with -wd.
  • -f — do not configure standard I/O file descriptors — just use the defaults.
  • -h — print useful information on this command. your little help button, love her 🥰
  • -ger — enable GER (Guaranteed Envelope Resources) communication protocol and error reporting. see MPI(7) for a description of GER. this option is mutually exclusive with -nger.
  • -nger — disable GER (Guaranteed Envelope Resources). this option is mutually exclusive with -ger.
  • -nsigs — do not have LAM catch signals in the user application. this is the default, and it's mutually exclusive with -sigs.
  • -np # — run this many copies of the program on the given nodes. this option says "the specified file is an executable program, not an application schema." if no nodes are specified, all LAM nodes are considered for scheduling; LAM will schedule the programs in a round-robin fashion, "wrapping around" (and scheduling multiple copies on a single node) if necessary. 🔄
  • -npty — disable pseudo-tty support. unless you're actively having problems with pseudo-tty support, you probably do not need this option. mutually exclusive with -pty.
  • -nw — do not wait for all processes to complete before exiting mpirun. this option is mutually exclusive with -w.
  • -nx — do not automatically export LAM_MPI_*, LAM_IMPI_*, or IMPI_* environment variables to the remote nodes.
  • -O — multicomputer is homogeneous. do no data conversion when passing messages. THIS FLAG IS NOW OBSOLETE. 💀
  • -pty — enable pseudo-tty support. among other things, this enables line-buffered output (which is probably what you want). this is the default. mutually exclusive with -npty. ✨
  • -s node — load the program from this node. this option is not valid on the command line if an application schema is specified.
  • -sigs — have LAM catch signals in the user process. this option is mutually exclusive with -nsigs.
  • -ssi key value — send arguments to various SSI modules. see the "SSI" section, below. 👀
  • -t, -ton — enable execution trace generation for all processes. trace generation will proceed with no further action. these options are mutually exclusive with -toff.
  • -toff — enable execution trace generation for all processes. trace generation for message passing traffic will begin after processes collectively call MPIL_Trace_on(2). note that trace generation for datatypes and communicators will proceed regardless of whether trace generation is enabled for messages or not. this option is mutually exclusive with -t and -ton.
  • -tv — launch processes under the TotalView Debugger. 🐛
  • -v — be verbose; report on important steps as they are done. she's chatty and we love that 💬
  • -w — wait for all applications to exit before mpirun exits.
  • -wd dir — change to the directory dir before the user's program executes. note that if the -wd option appears both on the command line and in an application schema, the schema takes precedence over the command line. this option is mutually exclusive with -D. 👑
  • -x — export the specified environment variables to the remote nodes before executing the program. existing environment variables can be specified (see the Examples section, below), or new variable names specified with corresponding values. lowkey important: the parser for the -x option is not very sophisticated; it does not even understand quoted values. users are advised to set variables in the environment, and then use -x to export (not define) them. 🙈
  • -sa — display the exit status of all MPI processes, regardless of whether they fail or run successfully.
  • -sf — display the exit status of all processes only if one of them fails.
  • -p prefix_str — prefixes each process status line displayed by [-sa] and [-sf] by the prefix_str. 🎀
  • where — a set of node and/or CPU identifiers indicating where to start program. see bhost(5) for a description of the node and CPU identifiers. mpirun will schedule adjoining ranks in MPI_COMM_WORLD on the same node when CPU identifiers are used. for example, if LAM was booted with a CPU count of 4 on n0 and a CPU count of 2 on n1 and where is C, ranks 0 through 3 will be placed on n0, and ranks 4 and 5 will be placed on n1. 🦋
  • args — pass these runtime arguments to every new process. these must always be the last arguments to mpirun. this option is not valid on the command line if an application schema is specified.

💜 DESCRIPTION 💜

one invocation of mpirun starts an MPI application running under LAM. 🚀

if the application is simply SPMD, the application can be specified right there on the mpirun command line.

if the application is MIMD, comprising multiple programs, an application schema is required in a separate file. see appschema(5) for a description of the application schema syntax — but essentially it contains multiple mpirun command lines, minus the command name itself. the ability to specify different options for different instantiations of a program is another reason to use an application schema. 💅

🌈 Location Nomenclature 🌈

as described above, mpirun can specify arbitrary locations in the current LAM universe. locations can be specified either by CPU or by node (noted by the "where" in the SYNTAX section, above). note that LAM does not bind processes to CPUs — specifying a location "by CPU" is really a convenience mechanism for SMPs that ultimately maps down to a specific node. 🧚

note that LAM effectively numbers MPI_COMM_WORLD ranks from left-to-right in the where, regardless of which nomenclature is used. this can be important because typical MPI programs tend to communicate more with their immediate neighbors (i.e., myrank +/- X) than distant neighbors. when neighbors end up on the same node, the shmem RPIs can be used for communication rather than the network RPIs, which can result in faster MPI performance. 🔥

specifying locations by node will launch one copy of an executable per specified node. using a capital "N" tells LAM to use all available nodes that were lambooted (see lamboot(1)). ranges of specific nodes can also be specified in the form "nR[,R]*", where R specifies either a single node number or a valid range of node numbers in the range of [0, num_nodes). for example:

  • mpirun N a.out — runs one copy of the executable a.out on all available nodes in the LAM universe. MPI_COMM_WORLD rank 0 will be on n0, rank 1 will be on n1, etc. ✨
  • mpirun n0-3 a.out — runs one copy of the executable a.out on nodes 0 through 3. MPI_COMM_WORLD rank 0 will be on n0, rank 1 will be on n1, etc.
  • mpirun n0-3,8-11,15 a.out — runs one copy of the executable a.out on nodes 0 through 3, 8 through 11, and 15. MPI_COMM_WORLD ranks will be ordered as follows: (0, n0), (1, n1), (2, n2), (3, n3), (4, n8), (5, n9), (6, n10), (7, n11), (8, n15). 🌟

specifying by CPU is the preferred method of launching MPI jobs. the intent is that the boot schema used with lamboot(1) will indicate how many CPUs are available on each node, and then a single, simple mpirun command can be used to launch across all of them. as noted above, specifying CPUs does not actually bind processes to CPUs — it is only a convenience mechanism for launching on SMPs. otherwise, the by-CPU notation is the same as the by-node notation, except that "C" and "c" are used instead of "N" and "n". 💅

assume in the following example that the LAM universe consists of four 4-way SMPs. so c0-3 are on n0, c4-7 are on n1, c8-11 are on n2, and 13-15 are on n3. 🦋

  • mpirun C a.out — runs one copy of the executable a.out on every single available CPU in the LAM universe. This is basically the simplest (and honestly the preferred) way to launch all your MPI jobs — even if it ends up resolving to one process per node. So: MPI_COMM_WORLD ranks 0-3 land on n0, ranks 4-7 on n1, ranks 8-11 on n2, and ranks 13-15 on n3. 💖
  • mpirun c0-3 a.out — runs one copy of the executable a.out on CPUs 0 through 3. All four ranks of MPI_COMM_WORLD will be on MPI_COMM_WORLD. (yes bestie, that's literally what it says, we don't make the rules 😭)
  • mpirun c0-3,8-11,15 a.out — runs one copy of the executable a.out on CPUs 0 through 3, 8 through 11, and 15. So MPI_COMM_WORLD ranks 0-3 are on n0, ranks 4-7 are on n2, and rank 8 is on n3. slay ✨

Okay so, the reason by-CPU naming is preferred over by-node naming is best shown with an example, so let's go. Say you wanna run that first CPU example (same MPI_COMM_WORLD mapping) but with the by-node nomenclature — one copy of a.out for every available CPU, maximizing local neighbors to potentially max out MPI performance. One solution:

mpirun n0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3 a.out

It works! But it's definitely klunky to type. By-CPU notation is just easier. Now, you might think this next one is equivalent:

mpirun N -np 16 a.out

Nope, not equivalent, because the MPI_COMM_WORLD rank mappings get assigned by node instead of by CPU. So rank 0 is on n0, rank 1 on n1, etc. But this one is equivalent, because LAM interprets a missing where as "C":

mpirun -np 16 a.out

That said, a "C" can tend to be more convenient, especially for batch-queuing scripts, because the exact number of processes may vary between queue submissions. The batch system decides the final number of CPUs available, so having a generic script that effectively says "run on everything you gave me" can lead to more portable / re-usable scripts. 🫶

Finally, lowkey important: specifying multiple where clauses is perfectly acceptable. So mixing by-node and by-CPU syntax is also valid, albeit typically not useful. Like:

mpirun C N a.out

But in some cases multiple where clauses are actually useful. Picture a parallel app where MPI_COMM_WORLD rank 0 is a "manager" and therefore burns barely any CPU cycles because it's usually just waiting for "worker" processes to return results. So it's probably desirable to run one "worker" process on all available CPUs, plus one extra process that'll be the "manager":

mpirun c0 C manager-worker-program

🧚 Application Schema or Executable Program? 🧚

To tell the two forms apart, mpirun looks on the command line for where or the -c option.

  • If neither is specified, the file named on the command line is assumed to be an application schema.
  • If either one or both are specified, the file is assumed to be an executable program.

If where and -c are both specified, then copies of the program get started on the specified nodes/CPUs according to an internal LAM scheduling policy. Specifying just one node effectively forces LAM to run all copies of the program in one place. If -c is given but where isn't, then all available CPUs on all LAM nodes get used. And if where is given but -c isn't, then one copy of the program runs on each node. 💅

🎀 Program Transfer 🎀

By default, LAM searches for executable programs on the target node where a particular instantiation is gonna run. But if the file system isn't shared, the target nodes are homogeneous, and the program gets frequently recompiled, it can be convenient to have LAM transfer the program from a source node (usually the local node) to each target node. The -s option specifies this behavior and identifies the single source node. ✨

🌸 Locating Files 🌸

LAM looks for an executable program by searching the directories in the user's PATH environment variable, as defined on the source node(s). This is consistent with logging into the source node and executing the program from the shell. On remote nodes, the "." path is the home directory.

For an application schema, LAM looks in three directories: the local directory, the value of the LAMAPPLDIR environment variable, and laminstalldir/boot, where "laminstalldir" is the directory where LAM/MPI was installed. 🦋

💖 Standard I/O 💖

LAM directs UNIX standard input to /dev/null on all remote nodes. On the local node that invoked mpirun, standard input is inherited from mpirun. That default is what used to be the -w option, and it's there to prevent conflicting access to the terminal.

LAM directs UNIX standard output and error to the LAM daemon on all remote nodes. LAM ships all that captured output/error to the node that invoked mpirun and prints it on the standard output/error of mpirun. Local processes inherit the standard output/error of mpirun and transfer to it directly.

So yes, you can redirect standard I/O for LAM applications using the typical shell redirection procedure on mpirun:

% mpirun C my_app  my_input  my_output
Heads up bestie: in this example, only the local node (the node where mpirun was invoked from) will receive the stream from my_input on stdin. The stdin on all the other nodes will be tied to /dev/null. But the stdout from all nodes will be collected into the my_output file. 🌈

The -f option avoids all the setup required to support the standard I/O described above. Remote processes are completely directed to /dev/null, and local processes inherit file descriptors from lamboot(1). 👑

⭐ Pseudo-tty support ⭐

The -pty option enables pseudo-tty support for process output (it's also enabled by default). This allows, among other things, for line buffered output from remote nodes — which is probably what you want, fr fr. This option can be disabled with the -npty switch.

💀 Process Termination / Signal Handling 💀

During the run of an MPI application, if any rank dies abnormally (either exiting before invoking MPI_FINALIZE, or dying as the result of a signal), mpirun will print out an error message and kill the rest of the MPI application. no cap.

By default, LAM/MPI only installs a signal handler for one signal in user programs (SIGUSR2 by default, but this can be overridden when LAM is configured and built). Therefore it's safe for users to install their own signal handlers in LAM/MPI programs — LAM notices death-by-signal cases by examining the process' return status provided by the operating system. 🫶

User signal handlers should probably avoid trying to cleanup MPI state — LAM is neither thread-safe nor async-signal-safe. For example, if a seg fault occurs in MPI_SEND (maybe because a bad buffer was passed in) and a user signal handler is invoked, and that user handler attempts to invoke MPI_FINALIZE, Bad Things could happen since LAM/MPI was already "in" MPI when the error occurred. Since mpirun will notice that the process died due to a signal, it's probably not necessary (and it's safest) for the user to only clean up non-MPI state. 🙈

If the -sigs option is used with mpirun, LAM/MPI will install several signal handlers locally on each rank to catch signals, print out error messages, and kill the rest of the MPI application. This is somewhat redundant behavior since this is now all handled by mpirun, but it has been left for backwards compatability.

🍓 Process Exit Statuses 🍓

The -sa, -sf, and -p parameters can be used to display the exit statuses of the individual MPI processes as they terminate.

  • -sa forces the exit statuses to be displayed for all processes.
  • -sf only displays the exit statuses if at least one process terminates either by a signal or a non-zero exit status (and note: exiting before invoking MPI_FINALIZE will cause a non-zero exit status).

The status of each process is printed out, one per line, in the following format:

prefix_string node pid killed status

If killed is 1, then status is the signal number. If killed is 0, then status is the exit status of the process.

The default prefix_string is "mpirun:", but the -p option can be used to override this string. 💅

🦋 Current Working Directory 🦋

The default behavior of mpirun has changed with respect to the directory that processes will be started in. 👀

The -wd option to mpirun lets the user change to an arbitrary directory before their program is invoked. It can also be used in application schema files to specify working directories on specific nodes and/or for specific applications.

If the -wd option appears both in a schema file and on the command line, the schema file directory will override the command line value.

The -D option will change the current working directory to the directory where the executable resides. It cannot be used in application schema files. And -wd is mutually exclusive with -D.

If neither -wd nor -D are specified, the local node will send the directory name where mpirun was invoked from to each of the remote nodes. Those remote nodes will then try to change to that directory. If they fail (e.g., if the directory does not exists on that node), they will start with from the user's home directory.

All directory changing occurs before the user's program is invoked; it does not wait until MPI_INIT is called. periodt. ✨

💜 Process Environment 💜

Processes in the MPI application inherit their environment from the LAM daemon upon the node on which they are running. The environment of a LAM daemon is fixed upon booting of the LAM with lamboot(1) and is typically inherited from the user's shell. On the origin node, this will be the shell from which lamboot(1) was invoked; on remote nodes, the exact environment is determined by the boot SSI module used by lamboot(1). The rsh boot module, for example, uses either rsh/ssh to launch the LAM daemon on remote nodes, and typically executes one or more of the user's shell-setup files before launching the LAM daemon. When running dynamically linked applications which require the LD_LIBRARY_PATH environment variable to be set, care must be taken to ensure that it is correctly set when booting the LAM. 🫶

🌈 Exported Environment Variables 🌈

All environment variables that are named in the form LAM_MPI_*, LAM_IMPI_*, or IMPI_* will automatically be exported to new processes on the local and remote nodes. This exporting may be inhibited with the -nx option. 😌

Additionally, the -x option to mpirun can be used to export specific environment variables to the new processes. Quick heads up tho bestie — while the syntax of the -x option allows the definition of new variables, the parser for this option is currently not very sophisticated — it does not even understand quoted values. 💀 So the advice is: set your variables in the environment and use -x to export them; do not try to define them here. Periodt.

🦋 Trace Generation 🦋

Two switches control trace generation from processes running under LAM, and both of them have to be in the on position for traces to actually be generated. No half-sends, no vibes-only.

  • The first switch is controlled by mpirun.
  • The second switch is initially set by mpirun but can be toggled at runtime with MPIL_Trace_on(2) and MPIL_Trace_off(2).

The -t (-ton is equivalent) and -toff options all turn on the first switch. Otherwise the first switch is off, and calls to MPIL_Trace_on(2) in the application program are ineffective — like yelling into the void, fr fr.

The -t option also turns on the second switch. The -toff option turns off the second switch.

See MPIL_Trace_on(2) and lamtrace(1) for more details. 📖

🌈 MPI Data Conversion 🌈

LAM's MPI library converts MPI messages from local representation to LAM representation upon sending them, and then back to local representation upon receiving them. In the case of a LAM consisting of a homogeneous network of machines where the local representation differs from the LAM representation, this can result in unnecessary conversions.

The -O switch used to be necessary to indicate to LAM whether the multi-computer was homogeneous or not. LAM now automatically determines whether a given MPI job is homogeneous or not. The -O flag will silently be accepted for backwards compatibility, but it is ignored. It's giving legacy icon who just stands there and does nothing. 💅

✨ SSI (System Services Interface) ✨

The -ssi switch allows the passing of parameters to various SSI modules. LAM's SSI modules are described in detail in lamssi(7). SSI modules have direct impact on MPI programs because they allow tunable parameters to be set at run time — such as which RPI communication device driver to use, what parameters to pass to that RPI, etc. 👑

The -ssi switch takes two arguments: key and value. The key argument generally specifies which SSI module will receive the value. For example, the key "rpi" is used to select which RPI to be used for transporting MPI messages. The value argument is the value that is passed. For example:

  • mpirun -ssi rpi lamd N foo — Tells LAM to use the "lamd" RPI and to run a single copy of "foo" on every node.
  • mpirun -ssi rpi tcp N foo — Tells LAM to use the "tcp" RPI.
  • mpirun -ssi rpi sysv N foo — Tells LAM to use the "sysv" RPI.

And so on. LAM's RPI SSI modules are described in lamssi_rpi(7).

The -ssi switch can be used multiple times to specify different key and/or value arguments. If the same key is specified more than once, the values are concatenated with a comma (, ) separating them. 🎀

Note that the -ssi switch is simply a shortcut for setting environment variables. The same effect may be accomplished by setting corresponding environment variables before running mpirun. The form of the environment variables that LAM sets are: LAM_MPI_SSI_key=value.
Note that the -ssi switch overrides any previously set environment variables. Also note that unknown key arguments are still set as environment variable — they are not checked (by mpirun) for correctness. Illegal or incorrect value arguments may or may not be reported — it depends on the specific SSI module. 🙈

The -ssi switch obsoletes the old -c2c and -lamd switches. These switches used to be relevant because LAM could only have two RPI's available at a time: the lamd RPI and one of the C2C RPIs. This is no longer true — all RPI's are now available and choosable at run-time. Selecting the lamd RPI is shown in the examples above. The -c2c switch has no direct translation, since "C2C" used to refer to all other RPI's that were not the lamd RPI. As such, -ssi rpi value must be used to select the specific desired RPI (whether it is "lamd" or one of the other RPI's). No shortcuts, bestie. 😭

🫶 Guaranteed Envelope Resources 🫶

By default, LAM will guarantee a minimum amount of message envelope buffering to each MPI process pair, and will impede or report an error to a process that attempts to overflow this system resource. This robustness and debugging feature is implemented in a machine specific manner when direct communication is used. For normal LAM communication via the LAM daemon, a protocol is used.

The -nger option disables GER and the measures taken to support it. The minimum GER is configured by the system administrator when LAM is installed. See MPI(7) for more details.

🌸 EXAMPLES 🌸

Be sure to also see the examples in the "Location Nomenclature" section, above. 👀

  • mpirun N prog1 — Load and execute prog1 on all nodes. Search the user's $PATH for the executable file on each node.
  • mpirun -c 8 prog1 — Run 8 copies of prog1 wherever LAM wants to run them.
  • mpirun n8-10 -v -nw -s n3 prog1 -q — Load and execute prog1 on nodes 8, 9, and 10. Search for prog1 on node 3 and transfer it to the three target nodes. Report as each process is created. Give "-q" as a command line to each new process. Do not wait for the processes to complete before exiting mpirun.
  • mpirun -v myapp — Parse the application schema, myapp, and start all processes specified in it. Report as each process is created.
  • mpirun -npty -wd /work/output -x DISPLAY C my_application — Start one copy of "my_application" on each available CPU. The number of available CPUs on each node was previously specified when LAM was booted with lamboot(1). As noted above, mpirun will schedule adjoining rank in MPI_COMM_WORLD on the same node where possible. For example, if n0 has a CPU count of 8, and n1 has a CPU count of 4, mpirun will place MPI_COMM_WORLD ranks 0 through 7 on n0, and 8 through 11 on n1. This tends to maximize on-node communication for many parallel applications; when used in conjunction with the multi-protocol network/shared memory RPIs in LAM (see the RELEASE_NOTES and INSTALL files with the LAM distribution), overall communication performance can be quite good. Also disable pseudo-tty support, change directory to /work/output, and export the DISPLAY variable to the new processes (perhaps my_application will invoke an X application such as xv to display output).

💀 DIAGNOSTICS 💀

  • mpirun: Exec format error — This usually means that either a number of processes or an appropriate where clause was not specified, indicating that LAM does not know how many processes to run. See the EXAMPLES and "Location Nomenclature" sections, above, for examples on how to specify how many processes to run, and/or where to run them. However, it can also mean that a non-ASCII character was detected in the application schema. This is usually a command line usage error where mpirun is expecting an application schema and an executable file was given. 😭
  • mpirun: syntax error in application schema, line XXX — The application schema cannot be parsed because of a usage or syntax error on the given line in the file.
  • filename: No such file or directory — This error can occur in two cases. Either the named file cannot be located, or it has been found but the user does not have sufficient permissions to execute the program or read the application schema.

⭐ RETURN VALUE ⭐

mpirun returns 0 if all ranks started by mpirun exit after calling MPI_FINALIZE.

A non-zero value is returned if:

  • an internal error occurred in mpirun, or
  • one or more ranks exited before calling MPI_FINALIZE.

If an internal error occurred in mpirun, the corresponding error code is returned. In the event that one or more ranks exit before calling MPI_FINALIZE, the return value of the rank of the process that mpirun first notices died before calling MPI_FINALIZE will be returned. Note that, in general, this will be the first rank that died but is not guaranteed to be so. 🙃

However, note that if the -nw switch is used, the return value from mpirun does not indicate the exit status of the ranks.

🧚 SEE ALSO 🧚

bhost(5), lamexec(1), lamssi(7), lamssi_rpi(7), lamtrace(1), loadgo(1), MPIL_Trace_on(2), mpimsg(1), mpitask(1)

and that's a wrap bestie!! 🎀 this whole page is straight from LAM 7.1.4, dated July, 2007 — vintage, iconic, still serving. go forth and launch some processes, you absolute main character. 👑💖🦋

📦 where this page comes from