fi_psm2(7)

📦
version 2.3.1 · 6 versions
, in centos-stream 9, fedora 44

💖 fi_psm2 — the PSM2 Fabric Provider 💖

okay bestie, so this is the manual for fi_psm2, aka The PSM2 Fabric Provider, and it ships with Libfabric v2.3.1. the page you're reading is fi_psm2(7) and it's giving ✨ extremely specific ✨ so grab a strawberry drink and let's get into it 🍓

🌸 OVERVIEW 🌸

so the psm2 provider runs over the PSM 2.x interface, which is supported by the Intel Omni-Path Fabric. PSM 2.x has literally all the PSM 1.x features plus a whole set of new functions with enhanced capabilities 🦋

but here's the thing — PSM 1.x and PSM 2.x are not ABI compatible, so the psm2 provider only works with PSM 2.x and does not support Intel TrueScale Fabric. no cap. 💅

👑 LIMITATIONS 👑

ngl, the psm2 provider doesn't support all the features defined in the libfabric API. here's some of the limitations, bestie:

🎀 Endpoint types

only support non-connection based types FI_DGRAM and FI_RDM. that's it, that's the list.

🎀 Endpoint capabilities

endpoints can support any combination of data transfer capabilities FI_TAGGED, FI_MSG, FI_ATOMICS, and FI_RMA. these capabilities can be further refined by FI_SEND, FI_RECV, FI_READ, FI_WRITE, FI_REMOTE_READ, and FI_REMOTE_WRITE to limit the direction of operations. so you can really customize the vibe here 🔥

🎀 FI_MULTI_RECV

FI_MULTI_RECV is supported for non-tagged message queue only. that's the whole rule, don't try it anywhere else.

🎀 Scalable endpoints (the availability check)

scalable endpoints are supported if the underlying PSM2 library supports multiple endpoints. this condition must be satisfied both when the provider is built and when the provider is used. see the Scalable endpoints section for more info 👇

🎀 Other supported capabilities

other supported capabilities include FI_TRIGGER, FI_REMOTE_CQ_DATA, FI_RMA_EVENT, FI_SOURCE, and FI_SOURCE_ERR. furthermore, FI_NAMED_RX_CTX is supported when scalable endpoints are enabled. slay 💅

🎀 Modes

FI_CONTEXT is required for the FI_TAGGED and FI_MSG capabilities. that means any request belonging to these two categories that generates a completion must pass as the operation context a valid pointer to type struct fi_context, and the space referenced by the pointer must remain untouched until the request has completed. like, do not touch it. 🙈 if none of FI_TAGGED and FI_MSG is asked for, the FI_CONTEXT mode is not required.

🎀 Progress

the psm2 provider requires manual progress. the application is expected to call fi_cq_read or fi_cntr_read from time to time when no other libfabric function is called, to ensure progress is made in a timely manner.

the provider does support auto progress mode. however, the performance can be significantly impacted if the application purely depends on the provider to make auto progress. so like, don't do that. 😭

🎀 Scalable endpoints (the real section)

scalable endpoints support depends on the multi-EP feature of the PSM2 library. if the PSM2 library supports this feature, the availability is further controlled by an environment variable PSM2_MULTI_EP. the psm2 provider automatically sets this variable to 1 if it is not set. the feature can be disabled explicitly by setting PSM2_MULTI_EP to 0.

when creating a scalable endpoint, the exact number of contexts requested should be set in the fi_info structure passed to the fi_scalable_ep function. this number should be set in fi_info->ep_attr->tx_ctx_cnt or fi_info->ep_attr->rx_ctx_cnt or both — whichever greater is used. the psm2 provider allocates all requested contexts upfront when the scalable endpoint is created. the same context is used for both Tx and Rx.

for optimal performance, it is advised to avoid having multiple threads accessing the same context, either directly by posting send/recv/read/write request, or indirectly by polling associated completion queues or counters. one context, one thread, main character energy only 👑

using the scalable endpoint as a whole in communication functions is not supported. instead, individual tx context or rx context of the scalable endpoint should be used. similarly, using the address of the scalable endpoint as the source address or destination address doesn't collectively address all the tx/rx contexts — it addresses only the first tx/rx context instead. it's giving "close enough" but it is not close enough. 💀

🎀 Shared Tx contexts

in order to achieve the purpose of saving PSM context by using shared Tx context, the endpoints bound to the shared Tx contexts need to be Tx only. the reason is that Rx capability always requires a PSM context, which can also be automatically used for Tx. as the result, allocating a shared Tx context for Rx capable endpoints actually consumes one extra context instead of saving some. the math is not mathing, so just don't 🙈

🎀 Unsupported features

these features are unsupported: connection management, passive endpoint, and shared receive context. periodt. ✨

✨ RUNTIME PARAMETERS ✨

So the psm2 provider is out here checking a whole lil roster of environment variables, bestie. Here's the full list, no skips, no cap 💅

  • FI_PSM2_UUID — PSM is very particular: every job needs its own unique ID (a UUID), and every process in the same job has to use the same UUID or they literally cannot talk to each other. Rude but true 🫶

The PSM reference manual says keep the UUID unique per job. In practice tho? Reusing a UUID generally works fine as long as (1) no two jobs with the same UUID are running at the same time, and (2) previous jobs with the same UUID exited normally. If you're hitting "resource busy" or "connection failure" errors for absolutely no discernible reason, the move is to manually set the UUID to something different from the default. iykyk.

The default UUID is 00FF00FF-0000-0000-0000-00FF0F0F00FF.

Also, plot twist: you can create endpoints with a UUID that's different from the one set here. To do that, set info->ep_attr->auth_key to the uuid value and info->ep_attr->auth_key_size to its size (16 bytes) when you call fi_endpoint() or fi_scalable_ep(). Still tho — an endpoint can only communicate with endpoints that have the same UUID. That part never changes, periodt.

  • FI_PSM2_NAME_SERVER — the psm2 provider has a lil built-in name server (so cute, so handy) that can resolve an IP address or host name into a transport address, which is exactly what the fi_av_insert call needs. The whole point of it is letting simple client-server apps (like the ones in fabtests) be written purely with libfabric, no out-of-band communication mechanism required ✨

For those apps, the server runs first so endpoints get created and registered with the name server, then the client calls fi_getinfo with the node parameter set to the IP address or host name of the server. The resulting fi_info structure would have the transport address of the endpoint created by the server sitting in the dest_addr field. Optionally, the service parameter can be used in addition to node. Heads up: the service number is interpreted by the provider and is not a TCP/IP port number 👑

The name server is on by default. Set the variable to 0 to turn it off. Doing that may save a small amount of resource, since a separate thread gets created when the name server is on.

The provider also detects OpenMPI and MPICH runs and flips the default setting to off. It's giving observant.

  • FI_PSM2_TAGGED_RMA — the RMA functions are implemented on top of the PSM Active Message functions. The thing is, Active Message functions have a limit on how much data can be transferred in a single message. Big transfers can be chopped into small chunks and pipe-lined… but the bandwidth is sub-optimal that way 😭

So the psm2 provider uses PSM tag-matching message queue functions to get higher bandwidth for large-size RMA. It takes advantage of the extra tag bits available in PSM2 to separate the RMA traffic from the regular tagged message queue. Iconic behavior honestly.

This option is on by default. To turn it off, set the variable to 0.

  • FI_PSM2_DELAY — time (in seconds) to sleep before closing PSM endpoints. This one is a workaround for a bug in some versions of the PSM library. Not glamorous, but necessary 🙈

The default setting is 0.

  • FI_PSM2_TIMEOUT — timeout (in seconds) for gracefully closing PSM endpoints. If the timeout expires, a forced closing gets issued. So yeah, it's a countdown ⏳

The default setting is 5.

  • FI_PSM2_CONN_TIMEOUT — timeout (in seconds) for establishing a connection between two PSM endpoints.

The default setting is 5.

  • FI_PSM2_PROG_INTERVAL — when auto progress is enabled (which you ask for via the hints to fi_getinfo), a progress thread gets created to make progress calls from time to time. This option sets the interval (in microseconds) between those progress calls.

The default setting is 1 if affinity is set, or 1000 if not. See FI_PSM2_PROG_AFFINITY below, bestie 💖

  • FI_PSM2_PROG_AFFINITY — when set, this specifies the set of CPU cores to set the progress thread affinity to. The format is:
<start>[:<end>[:<stride>]][,<start>[:<end>[:<stride>]]]*

where each triplet <start>:<end>:<stride> defines a block of core_ids. Both <start> and <end> can be either the core_id (when >=0) or core_id - num_cores (when <0).

By default, affinity is not set.

  • FI_PSM2_INJECT_SIZE — maximum message size allowed for fi_inject and fi_tinject calls. This is an experimental feature to let some applications override the default inject size limitation. Warning tho: when the inject size is larger than the default value, some inject calls might block 😳

The default setting is 64.

  • FI_PSM2_LOCK_LEVEL — when set, this dictates the level of locking being used by the provider. Level 2 means all locks are enabled. Level 1 disables some locks and is suitable for runs that limit the access to each PSM2 context to a single thread. Level 0 disables all locks and is thus only suitable for single threaded runs.
To use level 0 or level 1, wait object and auto progress mode cannot be used, because they introduce internal threads that may break the conditions needed for these levels. Don't do it, bestie.

The default setting is 2.

  • FI_PSM2_LAZY_CONN — okay so there are two strategies for when to establish connections between the PSM2 endpoints that OFI endpoints are built on top of. In eager connection mode, connections are established when addresses are inserted into the address vector. In lazy connection mode, connections are established when addresses are used the first time in communication. Eager connection mode has slightly lower critical path overhead, but lazy connection mode scales better. Trade-offs, trade-offs 💅

This option controls how the two connection modes are used. When set to 1, lazy connection mode is always used. When set to 0, eager connection mode is used when all the required conditions are met, and lazy connection mode is used otherwise. The conditions for eager connection mode are: (1) multiple endpoint (and scalable endpoint) support is disabled by explicitly setting PSM2_MULTI_EP=0; and (2) the address vector type is FI_AV_MAP.

The default setting is 0.

  • FI_PSM2_DISCONNECT — the provider has a mechanism to automatically send disconnection notifications to all connected peers before the local endpoint is closed. As the response, the peers call psm2_ep_disconnect to clean up the connection state on their side. This lets the same PSM2 epid be used by different dynamically started processes (clients) to communicate with the same peer (server) 🦋

This mechanism, however, introduces extra overhead to the finalization phase. For applications that never reuse epids within the same session, that overhead is unnecessary.

So this option controls whether the automatic disconnection notification mechanism should be enabled. For the client-server application mentioned above, the client side should set this option to 1, but the server should set it to 0. Matching your settings to your role? That's called communication, periodt.

The default setting is 0.

  • FI_PSM2_TAG_LAYOUT — select how the 96-bit PSM2 tag bits are organized. Currently three choices are available:
    • tag60 means a 32-4-60 partitioning for CQ data, internal protocol flags, and application tag.
    • tag64 means a 4-28-64 partitioning for internal protocol flags, CQ data, and application tag.
    • auto means to choose either tag60 or tag64 based on the hints passed to fi_getinfo — tag60 is used if remote CQ data support is requested explicitly, either by passing a non-zero value via hints->domain_attr->cq_data_size or by including FI_REMOTE_CQ_DATA in hints->caps; otherwise tag64 is used. And if tag64 is the result of automatic selection, fi_getinfo also returns a second instance of the provider with tag60 layout. Two for the price of one, slay ✨

The default setting is auto.

Notice that if the provider is compiled with the macro PSMX2_TAG_LAYOUT defined to 1 (means tag60) or 2 (means tag64), the choice is fixed at compile time and this runtime option will be disabled. Compile time said no takebacks 💀

🌸 PSM2 EXTENSIONS 🌸

The psm2 provider supports limited low-level parameter setting through the fi_set_val() and fi_get_val() functions. Currently the following parameters can be set via the domain fid:

  • FI_PSM2_DISCONNECT — overwrite the global runtime parameter FI_PSM2_DISCONNECT for this domain. See the RUNTIME PARAMETERS section for details. (Yes, that's a typo in the original — it says "Overwite" — we keep the facts, not the spelling bees 😭)

Valid parameter names are defined in the header file rdma/fi_ext_psm2.h.

👀 SEE ALSO 👀

fabric(7), fi_provider(7), fi_psm3(7)

🫶 AUTHORS 🫶

OpenFabrics.

And that's the whole page, bestie — you read all of it, which is main character energy fr fr. 💖

This lil masterpiece came from the Libfabric Programmer's Manual, dated 2024-12-10, and it's the fi_psm2(7) page. Go forth and configure your UUIDs responsibly, mwah 💅✨🎀

📦 where this page comes from