fi_psm(7)

💖 fi_psm — the PSM Fabric Provider 💖

okay bestie, welcome to fi_psm(7), the manual page for the psm fabric provider. pull up a chair, grab a boba, and let's get into it ✨

---

🌸 OVERVIEW 🌸

so the psm provider runs over the PSM 1.x interface — the one that's currently supported by the Intel TrueScale Fabric 👑. PSM hands you tag-matching message queue functions that are optimized for MPI implementations (yes, optimized, we love a specialist). PSM also has limited Active Message support, which is not officially published, but it's honestly quite stable and well documented in the source code (it's part of the OFED release, iykyk).

the psm provider eats both of those things — the tag-matching message queue functions and the Active Message functions — so it can support a whole variety of libfabric data transfer APIs, including:

• tagged message queue
• message queue
• RMA
• atomic operations

and guess what, the psm provider can also work with the psm2-compat library, which exposes a PSM 1.x interface over the Intel Omni-Path Fabric. versatile queen 💅

---

🚧 LIMITATIONS 🚧

ngl, the psm provider does not support all the features defined in the libfabric API. here's the rundown of what's not it:

Endpoint types

only supports non-connection based types FI_DGRAM and FI_RDM. that's it, that's the list.

Endpoint capabilities

endpoints can support any combination of data transfer capabilities FI_TAGGED, FI_MSG, FI_ATOMICS, and FI_RMA. these capabilities can be further refined by FI_SEND, FI_RECV, FI_READ, FI_WRITE, FI_REMOTE_READ, and FI_REMOTE_WRITE to limit the direction of operations.

the limitation is that no two endpoints can have overlapping receive or RMA target capabilities in any of the above categories. periodt.

for example, it is fine to have:

• two endpoints with FI_TAGGED | FI_SEND
• one endpoint with FI_TAGGED | FI_RECV
• one endpoint with FI_MSG
• one endpoint with FI_RMA | FI_ATOMICS

but it is not allowed to have two endpoints with FI_TAGGED, or two endpoints with FI_RMA. don't do it, bestie 😭

FI_MULTI_RECV

FI_MULTI_RECV is supported for non-tagged message queue only.

other supported capabilities include FI_TRIGGER 🎀

Modes

FI_CONTEXT is required for the FI_TAGGED and FI_MSG capabilities. that means any request belonging to these two categories that generates a completion must pass as the operation context a valid pointer to type struct fi_context, and the space referenced by the pointer must remain untouched until the request has completed. if none of FI_TAGGED and FI_MSG is asked for, the FI_CONTEXT mode is not required. so like, it's giving conditional 💅

Progress

the psm provider requires manual progress. the application is expected to call fi_cq_read or fi_cntr_read from time to time when no other libfabric function is called, to ensure progress is made in a timely manner. the provider does support auto progress mode — but the performance can be significantly impacted if the application purely depends on the provider to make auto progress. so like, do your part fr fr 🙈

Unsupported features

these features are unsupported, full stop:

• connection management
• scalable endpoint
• passive endpoint
• shared receive context
• send/inject with immediate data

---

⭐ RUNTIME PARAMETERS ⭐

the psm provider checks for the following environment variables:

• FI_PSM_UUID — PSM requires that each job has a unique ID (UUID). all the processes in the same job need to use the same UUID in order to be able to talk to each other. the PSM reference manual advises to keep UUID unique to each job. in practice, it generally works fine to reuse a UUID as long as (1) no two jobs with the same UUID are running at the same time, and (2) previous jobs with the same UUID have exited normally. if you're running into "resource busy" or "connection failure" issues with unknown reason, it is advisable to manually set the UUID to a value different from the default.

the default UUID is 0FFF0FFF-0000-0000-0000-0FFF0FFF0FFF. 💜

• FI_PSM_NAME_SERVER — the psm provider has a simple built-in name server that can be used to resolve an IP address or host name into a transport address needed by the fi_av_insert call. the main purpose of this name server is to allow simple client-server type applications (such as those in fabtests) to be written purely with libfabric, without using any out-of-band communication mechanism. for such applications, the server would run first to allow endpoints be created and registered with the name server, and then the client would call fi_getinfo with the node parameter set to the IP address or host name of the server. the resulting fi_info structure would have the transport address of the endpoint created by the server in the dest_addr field. optionally the service parameter can be used in addition to node. notice that the service number is interpreted by the provider and is not a TCP/IP port number.

the name server is on by default. it can be turned off by setting the variable to 0. this may save a small amount of resource since a separate thread is created when the name server is on.

the provider detects OpenMPI and MPICH runs and changes the default setting to off. she's adaptive like that ✨

• FI_PSM_TAGGED_RMA — the RMA functions are implemented on top of the PSM Active Message functions. the Active Message functions have a limit on the size of data that can be transferred in a single message. large transfers can be divided into small chunks and be pipe-lined — however, the bandwidth is sub-optimal by doing this way 😭

the psm provider uses PSM tag-matching message queue functions to achieve higher bandwidth for large size RMA. for this purpose, a bit is reserved from the tag space to separate the RMA traffic from the regular tagged message queue.

the option is on by default. to turn it off, set the variable to 0.

• FI_PSM_AM_MSG — the psm provider implements the non-tagged message queue over the PSM tag-matching message queue. one tag bit is reserved for this purpose. alternatively, the non-tagged message queue can be implemented over Active Message. this experimental feature has slightly larger latency.

this option is off by default. to turn it on, set the variable to 1.

• FI_PSM_DELAY — time (seconds) to sleep before closing PSM endpoints. this is a workaround for a bug in some versions of the PSM library. 👀

the default setting is 1.

• FI_PSM_TIMEOUT — timeout (seconds) for gracefully closing PSM endpoints. a forced closing will be issued if the timeout expires.

the default setting is 5.

• FI_PSM_PROG_INTERVAL — when auto progress is enabled (asked via the hints to fi_getinfo), a progress thread is created to make progress calls from time to time. this option sets the interval (microseconds) between progress calls.

the default setting is 1 if affinity is set, or 1000 if not. see FI_PSM_PROG_AFFINITY.

• FI_PSM_PROG_AFFINITY — when set, specify the set of CPU cores to set the progress thread affinity to. the format is:

<start>[:<end>[:<stride>]][,<start>[:<end>[:<stride>]]]*

where each triplet <start>:<end>:<stride> defines a block of core_ids. both <start> and <end> can be either the core_id (when >=0) or core_id - num_cores (when <0).

by default, affinity is not set.

---

🦋 SEE ALSO 🦋

fabric(7), fi_provider(7), fi_psm2(7), fi_psm3(7)

---

👑 AUTHORS 👑

OpenFabrics. (icons only, honestly.)

---

okay that's the whole page bestie, you're officially psm-pilled 💖 this was the Libfabric Programmer's Manual, version #VERSION#, dated 2022-12-11 — and that's fi_psm(7), signing off. slay, go make some progress calls 🌈✨🫶