fi_lnx(7)

💖 fi_lnx(7) — The LINKx (LNX) Provider 💖

okay so bestie, welcome to the LINKx man page, from the Libfabric crew (v2.5.1 era, iykyk) ✨

---

🌸 OVERVIEW 🌸

okay so the LNX provider is literally designed to link two or more providers together so your app can seamlessly use multiple providers or NICs at the same time. it's giving main character energy for your network stack 💅

it uses the libfabric peer infrastructure to help with all the underlying providers, and this version can link any libfabric provider that supports the FI_PEER capability. no cap, that's the vibe check. 🦋

---

✨ SUPPORTED FEATURES ✨

this release is an initial implementation, so here's what it's serving:

🎀 Endpoint types

only one endpoint type allowed bestie: FI_EP_RDM. that's it. periodt.

💜 Endpoint capabilities

LNX is a passthrough layer on the send path. on the receive path it uses the peer infrastructure to create shared receive queues (SRQ). receive requests go on the SRQ instead of on the core provider receive queue — the way that works is when the provider gets a message it queries the SRQ for a match. found one? the receive request completes. no match? the message goes on the LNX shared unexpected queue (SUQ) and future receive requests query the SUQ for matches. iconic behavior tbh 🔥

⚠️ heads up bestie: this first release only supports tagged operations. other message types are coming in future releases, slay.

🌈 Modes

no mode bits required. zero. nada. the provider doesn't need any of them 🫶

⭐ Progress

LNX uses the peer infrastructure for a shared completion queue. each linked provider still has to handle its own progress, but completion events get placed on the shared completion queue which gets passed to your application for access. easy peasy 💅

🧚 Address Format

LNX wraps the linked providers' addresses into one common binary blob — but it does not alter or change the linked providers' address format. it wraps them into an LNX structure, flattens it, and returns it to the application, then passes it between different nodes. LNX can parse the flattened format and operate on the different links. 🌸

🚨 important vibe check: this assumes nodes in the same group are all using the same version of the provider with the exact same links. IE: you can't have one node linking shm+cxi while another linking shm+rxm. consistency is key bestie, no exceptions 😭

🦋 Message Operations

LNX is designed to intercept message operations like fi_tsenddata and forward them to the appropriate provider based on specific criteria. it can link any provider supporting FI_PEER. the SHM provider is special cased — it's always used for intra-node communication if it's part of the link. otherwise, all other providers in the link are used in round robin. so iconic 🌈

👑 Using the Provider

to actually use it, you gotta set the FI_LNX_PROV_LINKS environment variable to the linked providers (syntax below in the variable description, keep scrolling bestie 💖). this lets you define multiple links — each link defines the provider to add plus an explicit list of domains. if you only specify the provider name? then all the domains get added to the link.

the provider returns the list of links to your app in the fi_getinfo() call. since there could be multiple providers in the link, the first domain of the first non-shm provider's fi_info is used as the link info. so if your link is defined like this:

- shm+cxi:cxi0,cxi1

then the cxi0 fi_info is used as the link fi_info, just with the provider and domain names modified. why does this matter? because the PCI information is part of the fi_info and some applications might use it to decide which link to use — a common selection criteria is the interface nearest the core the process is bound to. to make that determination, the app needs the interface's PCI information. so yeah, it matters fr fr 💯

---

💀 LIMITATIONS AND FUTURE WORK 💀

🔥 Hardware Support

LNX doesn't support hardware offload — like, ex hardware tag matching. this is an inherent limitation of the peer infrastructure. because of the shared receive queue (which linked providers have to query when a message arrives), any hardware offload that needs to send receive buffers straight to the hardware just won't work with it. 🙈

the shared receive queue gives two advantages, bestie:

1. reduces memory usage 🍓
2. coordinates the receive operations

and #2 is actually needed when receiving from FI_ADDR_UNSPEC — in that case both providers in the link can race to grab the receive buffer. figuring out how to use hardware tag matching and other hardware offload with LNX is a future effort. we'll get there 💜

🎀 Memory Registration

so, as part of memory registration, varying hardware can do hardware-specific steps like memory pinning. and because memory registration APIs don't specify the source or destination addresses, LNX literally cannot determine which provider to forward the registration to. so it does what it has to: registers the memory with all linked providers. 😭

this might not be efficient and might have unforeseen side effects. a better method is needed — one option is having a memory registration cache in lnx to avoid the expensive operations. up next on the glow-up list ✨

🌸 Operation Types

again bestie: this release only supports tagged operations. future releases will expand to other operation types. patience is a virtue 💅

⭐ Multi-Rail

LNX implements a multi-rail feature that lets messages be transmitted across multiple interfaces, driven by link config. it supports configuring multiple provider domains within a single link, like:

FI_LNX_PROV_LINKS=“cxi:cxi0,cxi1”

this defines a link with the cxi0 and cxi1 domains. when your app uses this link, messages are distributed round-robin across both domains. similarly, all available peer addresses get used round-robin too, ensuring balanced communication. 🔥

exception, as always: if the shm provider is in the link, all intra-node traffic gets routed through it. specialest of special cases 🫶

for this config:

FI_LNX_PROV_LINKS=“tcp;ofi_rxm+cxi:cxi0”

the link has all the tcp domains associated with the tcp;ofi_rxm provider, plus cxi0. messages are evenly distributed across all domain endpoints and peer addresses. the provider makes sure local endpoint types match the corresponding remote addresses — like, if the tcp provider is used, messages get directed to the peer's tcp address. matching vibes only 💖

---

🦋 RUNTIME PARAMETERS 🦋

LNX checks for these environment variables:

• FI_LNX_PROV_LINKS — this one specifies which providers to link. must be set for LNX to return a list of fi_info blocks in the fi_getinfo() call. the format is:

link ::= group [`|' group]
group ::= provider [`+' provider [`:' provider_list]]
provider_list ::= provider [`,' provider]
provider ::= [a-zA-Z0-9_]+

examples, bestie:

- shm+cxi:cxi0
- shm+cxi:cxi0,cxi1
- shm+cxi:cxi0|shm+cxi:cxi1
- shm+cxi

on a system with four cxi domains, that last example is equivalent to:

- shm+cxi:cxi0,cxi1,cxi2,cxi3

• FI_LNX_MULTI_RAIL_SELECTION — specifies the multi-rail selection policy LNX uses. LNX selects a local endpoint and a remote peer address when sending messages, and there are currently two supported methods:
  1. PER_MSG: round-robins over the local endpoint and remote peer addresses per message sent. bestie, this policy doesn't support send after send capability, because sent messages can arrive in a different order. 😬
  2. PER_PEER: round-robins per peer. on the first message to a peer, a local endpoint and peer address get selected and henceforth used for all messages to that peer. local endpoints are round-robined per peer. this policy ensures messages get received in the same order. 💖

the variable can be set to one of PER_MSG or PER_PEER. if it's not set, the policy defaults to PER_PEER. slay.

• FI_LNX_DISABLE_SHM — by default this is set to 0. but you can set it to one and then the SHM provider won't be used. useful for debugging and performance analysis. the SHM provider will naturally be used for all intra-node operations, so to test SHM in isolation with LNX, you can limit processes to the same node only. 🔍

---

👀 SEE ALSO 👀

fabric(7), fi_provider(7), fi_getinfo(3)

---

💜 AUTHORS 💜

OpenFabrics. — that's the crew behind this whole thing, give them their flowers 🌸

---

Libfabric Programmer's Manual · 2026-02-27 · fi_lnx(7)

k bye bestie, go link some providers 💅✨🦋