ORTE_HOSTS(7)

💖 ORTE_HOSTS 💖

ok so this is the manual page for ORTE_HOSTS (section 7 of the Open MPI docs), and it's literally all about how OpenRTE handles hostfiles and host lists — like, who gets to run where. bestie, if you've ever been like "why is my job on that node", this is the page. 🦋

---

✨ NAME ✨

ORTE_HOSTS — OpenRTE Hostfile and HOST Behavior: the whole rundown of OpenRTE's support for user-supplied hostfiles and comma-delimited lists of hosts. no cap.

---

🌸 DESCRIPTION 🌸

OpenRTE lets you be the main character and decide which hosts get used, and it supports several levels of user-specified host lists that follow an established precedence order. here's the tea:

• you can give a default hostfile — a list of nodes available to all app_contexts given on the command line. only one default hostfile can be provided for any job. periodt.
• you can also give a hostfile that lists the nodes to be used for a specific app_context.
• or you can just hand over a comma-delimited list of nodes for that app_context via the -host command line option.

now, the precedence order these get applied in depends a lil bit on your local environment. so when there's no resource manager (RM) around, here's how host and hostfile directives vibe together to define the set of hosts a job runs on:

• default hostfile unset, host unset, hostfile unset — the job is co-located with mpirun 🫶
• default hostfile unset, host set, hostfile unset — host defines the resource list for the job
• default hostfile unset, host unset, hostfile set — hostfile defines the resource list for the job
• default hostfile unset, host set, hostfile set — hostfile defines the resource list for the job, then host filters the list to define the final set of nodes available to each application within the job
• default hostfile set, host unset, hostfile unset — the default hostfile defines the resource list for the job
• default hostfile set, host set, hostfile unset — the default hostfile defines the resource list for the job, then host filters the list to define the final set of nodes available to each application within the job
• default hostfile set, host set, hostfile set — the default hostfile defines the resource list for the job, then hostfile filters the list, and then host filters the list to define the final set of nodes available to each application within the job

and yes, that last one is three layers of filtering. it's giving onion. 🧅

now — this changes somewhat in the presence of an RM, because that entity specifies the initial allocation of nodes. in that case, the default hostfile, hostfile and host directives are all used to filter the RM's specification, so you can utilize different portions of the allocation for different jobs. it's done according to the same precedence order as in the table above, except the RM provides the initial pool of nodes. 💅

---

🎀 RELATIVE INDEXING 🎀

once an initial allocation has been specified — whether by an RM, default hostfile, or hostfile — any subsequent hostfile and -host specifications can be made using relative indexing. this is the slay part: it lets you stipulate which hosts are to be used for a given app_context without specifying the particular host name, and instead using its relative position in the allocation.

lowkey the best way to get it is to look at a few examples. so imagine an RM has allocated a set of nodes to the user named "foo1, foo2, foo3, foo4". the user wants the first app_context to have exclusive use of the first two nodes, and a second app_context to use the last two nodes. sure, the user could printout the allocation to find the names of the nodes allocated to them and then use -host to specify this layout — but that's cumbersome and would require hand-manipulation for every single invocation. 😭

a simpler method is to utilize OpenRTE's relative indexing capability to specify the desired layout. in this case, a command line of:

mpirun -pernode -host +n1,+n2 ./app1 : -host +n3,+n4 ./app2

would provide the desired pattern. the "+" syntax indicates that the information is being provided as a relative index to the existing allocation. two methods of relative indexing are supported:

• +n<#> — a relative index into the allocation referencing the <#> node. OpenRTE will substitute the <#> node in the allocation.
• +e[:<#>] — a request for <#> empty nodes, i.e. OpenRTE is to substitute this reference with <#> nodes that have not yet been used by any other app_context. if the ":<#>" is not provided, OpenRTE will substitute the reference with all empty nodes. note that OpenRTE does track the empty nodes that have been assigned in this manner, so multiple uses of this option will result in assignment of unique nodes up to the limit of the available empty nodes. requests for more empty nodes than are available will generate an error. 🙈

relative indexing can be combined with absolute naming of hosts in any arbitrary manner, and can be used in hostfiles as well as with the -host command line option. in addition, any slot specification provided in hostfiles will be respected — so you can specify that only a certain number of slots from a relative indexed host are to be used for a given app_context. obsessed. ✨

another example may help illustrate this point. consider the case where a user has a default hostfile containing:

dummy1 slots=4
dummy2 slots=4
dummy3 slots=4
dummy4 slots=4
dummy5 slots=4

this may, for example, be a hostfile that describes a set of commonly-used resources that the user wishes to execute applications against. for this particular application, the user plans to map byslot, and wants:

• the first two ranks to be on the second node of any allocation,
• the next ranks to land on an empty node,
• one rank specifically on dummy4,
• the next rank to be on the second node of the allocation again,
• and finally any remaining ranks to be on whatever empty nodes are left.

to accomplish this, the user provides a hostfile of:

+n2 slots=2
+e:1
dummy4 slots=1
+n2
+e

the user can now use this information in combination with OpenRTE's sequential mapper to obtain their specific layout:

mpirun --default-hostfile dummyhosts -hostfile mylayout -mca rmaps seq ./my_app

which will result in:

• rank0 being mapped to dummy3
• rank1 to dummy1 as the first empty node
• rank2 to dummy4
• rank3 to dummy3
• rank4 to dummy2 and rank5 to dummy5 as the last remaining unused nodes

heads up bestie: the sequential mapper ignores the number of slots arguments, because it only maps one rank at a time to each node in the list. 🫠

if the default round-robin mapper had been used instead, then the mapping would have resulted in:

• ranks 0 and 1 being mapped to dummy3 since two slots were specified
• ranks 2-5 on dummy1 as the first empty node, which has four slots
• rank6 on dummy4 since the hostfile specifies only a single slot from that node is to be used
• ranks 7 and 8 on dummy3 since only two slots remain available
• ranks 9-12 on dummy2 since it is the next available empty node and has four slots
• ranks 13-16 on dummy5 since it is the last remaining unused node and has four slots

so yeah — relative indexing can allow for complex mappings to be ported across allocations, including those obtained from automated resource managers, with zero need for manual manipulation of scripts and/or command lines. that's the whole point and it ate. 🔥

---

👀 SEE ALSO 👀

• orterun(1)

---

and that's the page!! you made it to the end, icon 👑 this was Open MPI version 4.1.6, dated Sep 30, 2023 — go forth and map your ranks, bestie 💖🦋🌸