fi_endpoint(3)

📦
version 2.5.1 · 10 versions
, in centos-stream 10

💅 fi_endpoint(3) — Fabric endpoint operations ✨

Libfabric v2.5.1 says hi, and this is the endpoint page, bestie. It's a whole group chat of calls: allocate, bind, enable, cancel, alias, control, get/set options, open contexts, convert traffic classes, and check size-left stuff that is DEPRECATED. Let's go 🫶

✨ NAME ✨

  • fi_endpoint / fi_endpoint2 / fi_scalable_ep / fi_passive_ep / fi_close — Allocate or close an endpoint. Slay.
  • fi_ep_bind — Associate an endpoint with hardware resources, such as event queues, completion queues, counters, address vectors, or shared transmit/receive contexts. It's giving "let's connect the vibes."
  • fi_scalable_ep_bind — Associate a scalable endpoint with an address vector.
  • fi_pep_bind — Associate a passive endpoint with an event queue.
  • fi_enable — Transitions an active endpoint into an enabled state. The glow-up moment.
  • fi_cancel — Cancel a pending asynchronous data transfer.
  • fi_ep_alias — Create an alias to the endpoint.
  • fi_control — Control endpoint operation.
  • fi_getopt / fi_setopt — Get or set endpoint options.
  • fi_rx_context / fi_tx_context / fi_srx_context / fi_stx_context — Open a transmit or receive context.
  • fi_tc_dscp_set / fi_tc_dscp_get — Convert between a DSCP value and a network traffic class.
  • fi_rx_size_left / fi_tx_size_left (DEPRECATED) — Query the lower bound on how many RX/TX operations may be posted without an operation returning -FI_EAGAIN. These functions have been deprecated and will be removed in a future version of the library. Yes, they're serving deprecated realness 😭

🦋 SYNOPSIS 🦋

#include <rdma/fabric.h>

#include <rdma/fi_endpoint.h>

int fi_endpoint(struct fid_domain *domain, struct fi_info *info,
    struct fid_ep **ep, void *context);

int fi_endpoint2(struct fid_domain *domain, struct fi_info *info,
    struct fid_ep **ep, uint64_t flags, void *context);

int fi_scalable_ep(struct fid_domain *domain, struct fi_info *info,
    struct fid_ep **sep, void *context);

int fi_passive_ep(struct fi_fabric *fabric, struct fi_info *info,
    struct fid_pep **pep, void *context);

int fi_tx_context(struct fid_ep *sep, int index,
    struct fi_tx_attr *attr, struct fid_ep **tx_ep,
    void *context);

int fi_rx_context(struct fid_ep *sep, int index,
    struct fi_rx_attr *attr, struct fid_ep **rx_ep,
    void *context);

int fi_stx_context(struct fid_domain *domain,
    struct fi_tx_attr *attr, struct fid_stx **stx,
    void *context);

int fi_srx_context(struct fid_domain *domain,
    struct fi_rx_attr *attr, struct fid_ep **rx_ep,
    void *context);

int fi_close(struct fid *ep);

int fi_ep_bind(struct fid_ep *ep, struct fid *fid, uint64_t flags);

int fi_scalable_ep_bind(struct fid_ep *sep, struct fid *fid, uint64_t flags);

int fi_pep_bind(struct fid_pep *pep, struct fid *fid, uint64_t flags);

int fi_enable(struct fid_ep *ep);

int fi_cancel(struct fid_ep *ep, void *context);

int fi_ep_alias(struct fid_ep *ep, struct fid_ep **alias_ep, uint64_t flags);

int fi_control(struct fid *ep, int command, void *arg);

int fi_getopt(struct fid *ep, int level, int optname,
    void *optval, size_t *optlen);

int fi_setopt(struct fid *ep, int level, int optname,
    const void *optval, size_t optlen);

uint32_t fi_tc_dscp_set(uint8_t dscp);

uint8_t fi_tc_dscp_get(uint32_t tclass);

DEPRECATED ssize_t fi_rx_size_left(struct fid_ep *ep);

DEPRECATED ssize_t fi_tx_size_left(struct fid_ep *ep);

🎀 ARGUMENTS 🎀

  • fid — On creation, specifies a fabric or access domain. On bind, identifies the event queue, completion queue, counter, or address vector to bind to the endpoint. In other cases, it’s a fabric identifier of an associated resource. Iconic multitasking.
  • info — Details about the fabric interface endpoint to be opened. The struct fi_info must have been obtained using either fi_getinfo() or fi_dupinfo(). No homemade fi_info, bestie 💅
  • ep — A fabric endpoint.
  • sep — A scalable fabric endpoint.
  • pep — A passive fabric endpoint.
  • context — Context associated with the endpoint or asynchronous operation.
  • index — Index to retrieve a specific transmit/receive context.
  • attr — Transmit or receive context attributes.
  • flags — Additional flags to apply to the operation.
  • command — Command of control operation to perform on endpoint.
  • arg — Optional control argument.
  • level — Protocol level at which the desired option resides.
  • optname — The protocol option to read or set.
  • optval — The option value that was read or to set.
  • optlen — The size of the optval buffer.

🌸 DESCRIPTION 🌸

Okay so endpoints are transport level communication portals. There are two types: active and passive. Passive endpoints belong to a fabric domain and are most often used to listen for incoming connection requests. But a passive endpoint may also be used to reserve a fabric address that can be granted to an active endpoint. Active endpoints belong to access domains and can perform data transfers. Main character energy only for active ones, lowkey.

Active endpoints may be connection-oriented or connectionless, and may provide data reliability. The data transfer interfaces — messages (fi_msg), tagged messages (fi_tagged), RMA (fi_rma), and atomics (fi_atomic) — are associated with active endpoints. In basic configurations, an active endpoint has transmit and receive queues. In general, operations that generate traffic on the fabric are posted to the transmit queue. This includes all RMA and atomic operations, along with sent messages and sent tagged messages. Operations that post buffers for receiving incoming data are submitted to the receive queue. Transmit queue = posting the tea; receive queue = holding space for incoming tea 🫶

Active endpoints are created in the disabled state. They must transition into an enabled state before accepting data transfer operations, including posting of receive buffers. The fi_enable call is used to transition an active endpoint into an enabled state. The fi_connect and fi_accept calls will also transition an endpoint into the enabled state, if it is not already active. So yes, there are multiple paths to the enabled era ✨

In order to transition an endpoint into an enabled state, it must be bound to one or more fabric resources. An endpoint that will generate asynchronous completions, either through data transfer operations or communication establishment events, must be bound to the appropriate completion queues or event queues, respectively, before being enabled. Additionally, endpoints that use manual progress must be associated with relevant completion queues or event queues in order to drive progress. For endpoints that are only used as the target of RMA or atomic operations, this means binding the endpoint to a completion queue associated with receive processing. Connectionless endpoints must be bound to an address vector. Bind before you shine, periodt.

Once an endpoint has been activated, it may be associated with an address vector. Receive buffers may be posted to it and calls may be made to connection establishment routines. Connectionless endpoints may also perform data transfers.

The behavior of an endpoint may be adjusted by setting its control data and protocol options. This allows the underlying provider to redirect function calls to implementations optimized to meet the desired application behavior. Customization? We love to see it 💅

If an endpoint experiences a critical error, it will transition back into a disabled state. Critical errors are reported through the event queue associated with the EP. In certain cases, a disabled endpoint may be re-enabled. The ability to transition back into an enabled state is provider specific and depends on the type of error that the endpoint experienced. When an endpoint is disabled as a result of a critical error, all pending operations are discarded. That's the dramatic part 😭

💖 fi_endpoint / fi_passive_ep / fi_scalable_ep

fi_endpoint allocates a new active endpoint. fi_passive_ep allocates a new passive endpoint. fi_scalable_ep allocates a scalable endpoint. The properties and behavior of the endpoint are defined based on the provided struct fi_info. See fi_getinfo for additional details on fi_info. fi_info flags that control the operation of an endpoint are defined below. See section SCALABLE ENDPOINTS. It's giving "read the whole thing, bestie."

If an active endpoint is allocated in order to accept a connection request, the fi_info parameter must be the same as the fi_info structure provided with the connection request (FI_CONNREQ) event. Same fi_info, no substitutions 🔥

An active endpoint may acquire the properties of a passive endpoint by setting the fi_info handle field to the passive endpoint fabric descriptor. This is useful for applications that need to reserve the fabric address of an endpoint prior to knowing if the endpoint will be used on the active or passive side of a connection. For example, this feature is useful for simulating socket semantics. Once an active endpoint acquires the properties of a passive endpoint, the passive endpoint is no longer bound to any fabric resources and must no longer be used. The user is expected to close the passive endpoint after opening the active endpoint in order to free up any lingering resources that had been used. So close the passive one after the active one is born — cleanup is self-care 🧚

🦋 fi_endpoint2

Similar to fi_endpoint, it accepts an extra parameter flags. Mainly used for opening endpoints that use peer transfer feature. See fi_peer(3). Extra flag energy, iykyk.

💀 fi_close

Closes an endpoint and releases all resources associated with it.

When closing a scalable endpoint, there must be no opened transmit contexts, or receive contexts associated with the scalable endpoint. If resources are still associated with the scalable endpoint when attempting to close, the call will return -FI_EBUSY. The scalable endpoint said "not yet, bestie" 😭

Outstanding operations posted to the endpoint when fi_close is called will be discarded. Discarded operations will silently be dropped, with no completions reported. Memory buffers that the user associated with the endpoint by calling libfabric’s transmission interfaces must not be released, deregistered, or reused until either a completion is generated or fi_close on the respective endpoint returns. Additionally, a provider may discard previously completed operations from the associated completion queue(s). The behavior to discard completed operations is provider specific.

That's a serious one — buffers stay put until the completion or fi_close return happens. No early reuse, no cap 🫶

🌈 fi_ep_bind

fi_ep_bind is used to associate an endpoint with other allocated resources, such as completion queues, counters, address vectors, event queues, shared contexts, and memory regions. The type of objects that must be bound with an endpoint depend on the endpoint type and its configuration. Basically: know your endpoint type, then bind the right iconique resources ✨

Lowkey the binding rules are where people get cooked, so let's go slow 💅

Passive endpoints must be bound with an EQ that supports connection management events. Connectionless endpoints must be bound to a single address vector. And if an endpoint is using a shared transmit and/or receive context, those shared contexts must be bound to the endpoint too. CQs, counters, AV, and shared contexts all have to be bound to endpoints before they get enabled — whether that enabling is explicit or implicit. No skipping the line, bestie 🎀

An endpoint must be bound with CQs capable of reporting completions for any asynchronous operation initiated on the endpoint. Like — for example — if the endpoint supports any outbound transfers (sends, RMA, atomics, etc.), then it must be bound to a completion queue that can report transmit completions. And this is true even if the endpoint is configured to suppress successful completions, so that operations that complete in error may still be reported to the user. The error path always eats, ok? 🔥

An active endpoint may direct asynchronous completions to different CQs, based on the type of operation. You specify that using fi_ep_bind flags. The following flags may be OR'ed together when binding an endpoint to a completion domain CQ. 👇

  • FI_RECV — Directs the notification of inbound data transfers to the specified completion queue. This includes received messages. This binding automatically includes FI_REMOTE_WRITE, if applicable to the endpoint. ✨
  • FI_SELECTIVE_COMPLETION — By default, data transfer operations write CQ completion entries into the associated completion queue after they have successfully completed. Applications can use this bind flag to selectively enable when completions are generated. If FI_SELECTIVE_COMPLETION is specified, data transfer operations will not generate CQ entries for successful completions unless FI_COMPLETION is set as an operational flag for the given operation. Operations that fail asynchronously will still generate completions, even if a completion is not requested. FI_SELECTIVE_COMPLETION must be OR'ed with FI_TRANSMIT and/or FI_RECV flags. 🦋

When FI_SELECTIVE_COMPLETION is set, the user must determine when a request that does NOT have FI_COMPLETION set has completed indirectly — usually based on the completion of a subsequent operation, or by using completion counters. Use of this flag may improve performance by allowing the provider to avoid writing a CQ completion entry for every operation. Performance slay, but you gotta keep track of your own business, iykyk 😭

See the Notes section below for additional information on how this flag interacts with the FI_CONTEXT and FI_CONTEXT2 mode bits. Don't skip it, it matters fr fr.
  • FI_TRANSMIT — Directs the completion of outbound data transfer requests to the specified completion queue. This includes send message, RMA, and atomic operations. 👑

An endpoint may optionally be bound to a completion counter. Associating an endpoint with a counter is in addition to binding the EP with a CQ. When binding an endpoint to a counter, the following flags may be specified. 🧮

  • FI_READ — Increments the specified counter whenever an RMA read, atomic fetch, or atomic compare operation initiated from the endpoint has completed successfully or in error.
  • FI_RECV — Increments the specified counter whenever a message is received over the endpoint. Received messages include both tagged and normal message operations.
  • FI_REMOTE_READ — Increments the specified counter whenever an RMA read, atomic fetch, or atomic compare operation is initiated from a remote endpoint that targets the given endpoint. Use of this flag requires that the endpoint be created using FI_RMA_EVENT. 🌸
  • FI_REMOTE_WRITE — Increments the specified counter whenever an RMA write or base atomic operation is initiated from a remote endpoint that targets the given endpoint. Use of this flag requires that the endpoint be created using FI_RMA_EVENT. (yes, same requirement, no cap 🫶)
  • FI_SEND — Increments the specified counter whenever a message transfer initiated over the endpoint has completed successfully or in error. Sent messages include both tagged and normal message operations.
  • FI_WRITE — Increments the specified counter whenever an RMA write or base atomic operation initiated from the endpoint has completed successfully or in error.

A vibe check on loyalty: an endpoint may only be bound to a single CQ or counter for a given type of operation. For example, a EP may not bind to two counters both using FI_WRITE. Furthermore, providers may limit CQ and counter bindings to endpoints of the same endpoint type (DGRAM, MSG, RDM, etc.). So no double-dipping 💀

🧚 fi_scalable_ep_bind

fi_scalable_ep_bind is used to associate a scalable endpoint with an address vector. See the section on SCALABLE ENDPOINTS. A scalable endpoint has a single transport level address and can support multiple transmit and receive contexts. The transmit and receive contexts share the transport-level address. Address vectors that are bound to scalable endpoints are implicitly bound to any transmit or receive contexts created using the scalable endpoint. One address, many contexts — it's giving main character energy ✨

💖 fi_enable

This call transitions the endpoint into an enabled state. An endpoint must be enabled before it may be used to perform data transfers. Enabling an endpoint typically results in hardware resources being assigned to it. Endpoints making use of completion queues, counters, event queues, and/or address vectors must be bound to them before being enabled. 🎀

Calling connect or accept on an endpoint will implicitly enable an endpoint if it has not already been enabled. Surprise enabling, but make it convenient.

fi_enable may also be used to re-enable an endpoint that has been disabled as a result of experiencing a critical error. Applications should check the return value from fi_enable to see if a disabled endpoint has successfully be re-enabled. Please actually check it, bestie, don't just assume 🙈

🔥 fi_cancel

fi_cancel attempts to cancel an outstanding asynchronous operation. Canceling an operation causes the fabric provider to search for the operation and, if it is still pending, complete it as having been canceled. An error queue entry will be available in the associated error queue with error code FI_ECANCELED. On the other hand, if the operation completed before the call to fi_cancel, then the completion status of that operation will be available in the associated completion queue. No specific entry related to fi_cancel itself will be posted. Rude but fair. 💅

Cancel uses the context parameter associated with an operation to identify the request to cancel. Operations posted without a valid context parameter — either no context parameter is specified, or the context value was ignored by the provider — cannot be canceled. If multiple outstanding operations match the context parameter, only one will be canceled. In this case, the operation which is canceled is provider specific. The cancel operation is asynchronous, but will complete within a bounded period of time. So like… be patient, it's coming ⏳

🦋 fi_ep_alias

This call creates an alias to the specified endpoint. Conceptually, an endpoint alias provides an alternate software path from the application to the underlying provider hardware. An alias EP differs from its parent endpoint only by its default data transfer flags. For example, an alias EP may be configured to use a different completion mode. By default, an alias EP inherits the same data transfer flags as the parent endpoint. An application can use fi_control to modify the alias EP operational flags. Iconic little copy with a twist 👑

When allocating an alias, an application may configure either the transmit or receive operational flags. This avoids needing a separate call to fi_control to set those flags. The flags passed to fi_ep_alias must include FI_TRANSMIT or FI_RECV (not both) with other operational flags OR'ed in. This will override the transmit or receive flags, respectively, for operations posted through the alias endpoint. All allocated aliases must be closed for the underlying endpoint to be released. Yes — all of them. Every last one. 😤

🎀 fi_control

The control operation is used to adjust the default behavior of an endpoint. It allows the underlying provider to redirect function calls to implementations optimized to meet the desired application behavior. As a result, calls to fi_ep_control must be serialized against all other calls to an endpoint. One at a time, diva 🚦

The base operation of an endpoint is selected during creation using struct fi_info. The following control commands and arguments may be assigned to an endpoint.

  • FI_BACKLOG - int *value — This option only applies to passive endpoints. It is used to set the connection request backlog for listening endpoints.
  • FI_GETOPSFLAG – uint64_t *flags — Used to retrieve the current value of flags associated with the data transfer operations initiated on the endpoint. The control argument must include FI_TRANSMIT or FI_RECV (not both) flags to indicate the type of data transfer flags to be returned. See below for a list of control flags.
  • FI_GETWAIT – void ** — This command allows the user to retrieve the file descriptor associated with a socket endpoint. The fi_control arg parameter should be an address where a pointer to the returned file descriptor will be written. See fi_eq.3 for addition details using fi_control with FI_GETWAIT. The file descriptor may be used for notification that the endpoint is ready to send or receive data. ✨
  • FI_SETOPSFLAG – uint64_t *flags — Used to change the data transfer operation flags associated with an endpoint. The control argument must include FI_TRANSMIT or FI_RECV (not both) to indicate the type of data transfer that the flags should apply to, with other flags OR'ed in. The given flags will override the previous transmit and receive attributes that were set when the endpoint was created. Valid control flags are defined below.

🌈 fi_getopt / fi_setopt

Endpoint protocol operations may be retrieved using fi_getopt or set using fi_setopt. Applications specify the level that a desired option exists, identify the option, and provide input/output buffers to get or set the option. fi_setopt provides an application a way to adjust low-level protocol and implementation specific details of an endpoint, and must be called before the endpoint is enabled (fi_enable). Timing is everything, bestie ⏰

The following option levels and option names and parameters are defined.

FI_OPT_ENDPOINT •

  • FI_OPT_CM_DATA_SIZE (size_t) — this one tells you how much room there is inside CM messages for your custom data, like the little notes you can pass between peers. It literally caps the amount of data apps can swap between peer endpoints via fi_connect, fi_accept, and fi_reject. The size you get back depends on the endpoint's properties — except for passive endpoints, where the size reflects the maximum size of data that may show up as part of a connection request event. 💌 Oh and heads up: this option is read only. No touching. 🙅‍♀️
  • FI_OPT_MIN_MULTI_RECV (size_t) — the minimum receive buffer space out there, and once you dip below it the provider releases the receive buffer (peep FI_MULTI_RECV for the lore). Modifying this value is only guaranteed to set the minimum buffer space needed on receives posted after you changed it. ⏰ And the recommendation, bestie: if you wanna override the default MIN_MULTI_RECV value, set this option before enabling the corresponding endpoint. Timing is everything. ✨
  • FI_OPT_FI_HMEM_P2P (int) — decides how the provider handles peer-to-peer FI_HMEM transfers for this endpoint. By default the provider picks whether to use peer-to-peer based on the type of transfer, and that default is FI_HMEM_P2P_ENABLED. The valid values, defined in fi_endpoint.h, are:
    • FI_HMEM_P2P_ENABLED: peer-to-peer support may be used by the provider to handle FI_HMEM transfers, and which transfers actually get initiated using peer-to-peer is entirely up to the provider implementation. 🤷‍♀️
    • FI_HMEM_P2P_REQUIRED: peer-to-peer support must be used for transfers — any transfer that can't be done using p2p gets reported as failing. 💀
    • FI_HMEM_P2P_PREFERRED: peer-to-peer support should be used by the provider for all transfers if it's available, but if p2p is unavailable the provider may choose to copy the data to kick off the transfer. 🎀
    • FI_HMEM_P2P_DISABLED: peer-to-peer support should not be used. Periodt.

If the mode you asked for can't be supported by the provider, fi_setopt() returns -FI_EOPNOTSUPP. And important 🚨: the FI_HMEM_DISABLE_P2P environment variable discussed over in fi_mr(3) takes precedence over this setopt option.

  • FI_OPT_CUDA_API_PERMITTED (bool) — this one only applies to the fi_setopt call. It controls the endpoint's behavior when it comes to making calls to the CUDA API. By default, an endpoint is permitted to call the CUDA API. If you want to prohibit an endpoint from making such calls, just set this option to false. 🚫 Now, if an endpoint's support of CUDA memory relies on making CUDA API calls, it'll return -FI_EOPNOTSUPP for the call to fi_setopt. And if either the CUDA library or the CUDA device isn't available, the endpoint returns -FI_EINVAL. Every provider that supports the FI_HMEM capability implements this option. 💅
  • FI_OPT_SHARED_MEMORY_PERMITTED (bool) — again, fi_setopt call only. This one controls whether shared memory gets used for intra-node communication. Setting it to true allows the use of shared memory. Set it to false and shared memory won't be used — and how intra-node communication is implemented at that point is provider dependent. 🫶
  • FI_OPT_MAX_MSG_SIZE (size_t) — sets the maximum message size that can be transferred by the endpoint in a single untagged message. It's limited by the endpoint's configuration and the provider's capabilities, and it must be less than or equal to ep_attr->max_msg_size. Providers that don't support this option return -FI_ENOPROTOOPT — in that case, just use ep_attr->max_msg_size. 😌
  • FI_OPT_MAX_TAGGED_SIZE (size_t) — same deal but for a single tagged message. Limited by the endpoint's configuration and the provider's capabilities, and must be <= ep_attr->max_msg_size. Unsupported providers return -FI_ENOPROTOOPT, and then ep_attr->max_msg_size is your fallback. 🏷️
  • FI_OPT_MAX_RMA_SIZE (size_t) — the maximum message size that can be transferred by the endpoint via a single RMA operation. Limited by the endpoint's configuration and the provider's capabilities, must be <= ep_attr->max_msg_size. Providers that don't support it return -FI_ENOPROTOOPT; use ep_attr->max_msg_size instead. 🔥
  • FI_OPT_MAX_ATOMIC_SIZE (size_t) — the maximum data size that can be transferred by the endpoint via a single atomic operation. Limited by the endpoint's configuration and the provider's capabilities, must be <= ep_attr->max_msg_size. No support? -FI_ENOPROTOOPT, then fall back to ep_attr->max_msg_size. ⚛️
  • FI_OPT_INJECT_MSG_SIZE (size_t) — the maximum message size that can be injected by the endpoint in a single untagged message. Limited by the endpoint's configuration and the provider's capabilities, and must be <= tx_attr->inject_size. Providers that don't support it return -FI_ENOPROTOOPT, in which case use tx_attr->inject_size. 💉
  • FI_OPT_INJECT_TAGGED_SIZE (size_t) — the max message size that can be injected in a single tagged message. Must be <= tx_attr->inject_size, limited by endpoint config + provider capabilities. Unsupported → -FI_ENOPROTOOPT, fall back to tx_attr->inject_size. 🎯
  • FI_OPT_INJECT_RMA_SIZE (size_t) — the maximum data size that can be injected by the endpoint in a single RMA operation. Must be <= tx_attr->inject_size, limited by endpoint config + provider capabilities. Same -FI_ENOPROTOOPT story, same tx_attr->inject_size fallback. 🌈
  • FI_OPT_INJECT_ATOMIC_SIZE (size_t) — the maximum data size that can be injected by the endpoint in a single atomic operation. Must be <= tx_attr->inject_size, limited by endpoint config + provider capabilities. If the provider doesn't support it you get -FI_ENOPROTOOPT, and then tx_attr->inject_size is what you use. ⭐

🦋 fi_tc_dscp_set

This call converts a DSCP defined value into a libfabric traffic class value. Use it when you're assigning a DSCP value while setting the tclass field in either domain or endpoint attributes. 🎀

🦋 fi_tc_dscp_get

This call returns the DSCP value associated with the tclass field for the domain or endpoint attributes. The matching getter, we love symmetry. 💖

💀 fi_rx_size_left (DEPRECATED)

This function has been deprecated and will be removed in a future version of the library. It may not be supported by all providers. 😭

The fi_rx_size_left call returns a lower bound on the number of receive operations that may be posted to the given endpoint without that operation returning -FI_EAGAIN. Depending on the specific details of the subsequently posted receive operations (e.g., number of iov entries, which receive function is called, etc.), it may be possible to post more receive operations than originally indicated by fi_rx_size_left. So it's a floor, not a promise. ✨

💀 fi_tx_size_left (DEPRECATED)

Also deprecated, also going away in a future version of the library, also maybe not supported by all providers. 🙈

The fi_tx_size_left call returns a lower bound on the number of transmit operations that may be posted to the given endpoint without that operation returning -FI_EAGAIN. Depending on the specific details of the subsequently posted transmit operations (e.g., number of iov entries, which transmit function is called, etc.), it may be possible to post more transmit operations than originally indicated by fi_tx_size_left. Same vibes, other direction. 🫶

👑 ENDPOINT ATTRIBUTES 👑

The fi_ep_attr structure defines the set of attributes associated with an endpoint. Endpoint attributes may be further refined using the transmit and receive context attributes as shown below. 🧚

struct fi_ep_attr {
    enum fi_ep_type type;
    uint32_t        protocol;
    uint32_t        protocol_version;
    size_t          max_msg_size;
    size_t          msg_prefix_size;
    size_t          max_order_raw_size;
    size_t          max_order_war_size;
    size_t          max_order_waw_size;
    uint64_t        mem_tag_format;
    size_t          tx_ctx_cnt;
    size_t          rx_ctx_cnt;
    size_t          auth_key_size;
    uint8_t         *auth_key;
};

🌸 type - Endpoint Type

If specified, this indicates the type of fabric interface communication you want. The supported types are:

  • FI_EP_DGRAM — supports a connectionless, unreliable datagram communication. Message boundaries are maintained, but the maximum message size may be limited to the fabric MTU. Flow control is not guaranteed. 🎲
  • FI_EP_MSG — provides a reliable, connection-oriented data transfer service with flow control that maintains message boundaries. 🎀
  • FI_EP_RDM — reliable datagram message. Provides a reliable, connectionless data transfer service with flow control that maintains message boundaries. 👑
  • FI_EP_UNSPEC — the type of endpoint is not specified. This is usually provided as input, with other attributes of the endpoint or the provider selecting the type. 🤷‍♀️

🌈 Protocol

Specifies the low-level end-to-end protocol employed by the provider. A matching protocol must be used by communicating endpoints to ensure interoperability. 🔗 The following protocol values are defined. Provider specific protocols are also allowed. Provider specific protocols will be indicated by having the upper bit of the protocol value set to one. ⭐

  • FI_PROTO_EFA — the proprietary protocol on Elastic Fabric Adapter fabric ✨ and yes it supports both DGRAM and RDM endpoints, versatile queen 💅
  • FI_PROTO_IB_RDM — reliable-datagram protocol, implemented on top of InfiniBand reliable-connected queue pairs 🧵
  • FI_PROTO_IB_UD — this one runs over Infiniband unreliable datagram queue pairs 👀
  • FI_PROTO_IWARP — runs over the Internet wide area RDMA protocol transport 🌐
  • FI_PROTO_IWARP_RDM — reliable-datagram protocol implemented over iWarp reliable-connected queue pairs 🫶
  • FI_PROTO_NETWORKDIRECT — runs over the Microsoft NetworkDirect service provider interface. It adds reliable-datagram semantics on top of the NetworkDirect connection-oriented endpoint semantics 🔥
  • FI_PROTO_PSMX2 — based on an Intel proprietary protocol known as PSM2, a.k.a. performance scaled messaging version 2. PSMX2 is an extended version of the PSM2 protocol made to support the libfabric interfaces 💜
  • FI_PROTO_PSMX3 — Intel's protocol known as PSM3, performance scaled messaging version 3, and it's implemented over RoCEv2 and verbs ⚡
  • FI_PROTO_RDMA_CM_IB_RC — runs over Infiniband reliable-connected queue pairs, using the RDMA CM protocol for connection establishment 🤝
  • FI_PROTO_RXD — reliable-datagram protocol implemented over datagram endpoints. RXD is a libfabric utility component that adds RDM endpoint semantics over DGRAM endpoint semantics 🎀
  • FI_PROTO_RXM — reliable-datagram protocol implemented over message endpoints. RXM is a libfabric utility component that adds RDM endpoint semantics over MSG endpoint semantics 🌸
  • FI_PROTO_SOCK_TCP — the protocol is layered over TCP packets, classic behavior honestly 💀
  • FI_PROTO_UDP — sends and receives UDP datagrams. For example, an endpoint using FI_PROTO_UDP will be able to communicate with a remote peer that is using Berkeley SOCK_DGRAM sockets using IPPROTO_UDP 📮
  • FI_PROTO_SHM — protocol for intra-node communication using shared memory segments, used by the shm provider 🧠
  • FI_PROTO_SM2 — protocol for intra-node communication using shared memory segments, used by the sm2 provider 🧠✨
  • FI_PROTO_CXI — reliable-datagram protocol optimized for HPC applications, used by the cxi provider 🚀
  • FI_PROTO_CXI_RNR — a version of the FI_PROTO_CXI protocol that implements an RNR protocol which can be used when messaging is primarily expected and FI_ORDER_SAS ordering is not required 🦋
  • FI_PROTO_UNSPEC — the protocol is not specified. This is usually provided as input, with other attributes of the socket or the provider selecting the actual protocol 🙈

protocol_version — Protocol Version 🏷️

Identifies which version of the protocol the provider is employing. The protocol version lets providers extend an existing protocol — say, by adding support for extra features or functionality — in a backward compatible way 💖

Providers that support different versions of the same protocol should inter-operate, but only when using the capabilities defined for the lesser version. iykyk 👑

max_msg_size — Max Message Size 📏

Defines the maximum size for an application data transfer as a single operation. One transfer, one size cap, periodt.

msg_prefix_size — Message Prefix Size 🎀

Specifies the size of any required message prefix buffer space. This field will be 0 unless the FI_MSG_PREFIX mode is enabled. If msg_prefix_size is > 0 the specified value will be a multiple of 8-bytes ✨

Max RMA Ordered Size 🔥

The maximum ordered size specifies the delivery order of transport data into target memory for RMA and atomic operations. Data ordering is separate, but dependent on message ordering (defined below). Data ordering is unspecified where message order is not defined ⚠️

Data ordering refers to the access of the same target memory by subsequent operations. When back to back RMA read or write operations access the same registered memory location, data ordering indicates whether the second operation reads or writes the target memory after the first operation has completed. Like, will an RMA read that follows an RMA write read back the data that was written? 👀 Similarly, will an RMA write that follows an RMA read update the target buffer after the read has transferred the original data? Data ordering answers these questions, even in the presence of errors — such as when data needs to be resent because of lost or corrupted network traffic 😭

RMA ordering applies between two operations, and not within a single data transfer. So ordering is defined per byte-addressable memory location. I.e. ordering specifies whether location X is accessed by the second operation after the first operation. Nothing is implied about the completion of the first operation before the second operation is initiated 🫶 For example, if the first operation updates locations X and Y, but the second operation only accesses location X, there are no guarantees defined relative to location Y and the second operation. yeah. iconique but strict 💅

In order to support large data transfers being broken into multiple packets and sent using multiple paths through the fabric, data ordering may be limited to transfers of a specific size or less. Providers specify when data ordering is maintained through the following values. Note that even if data ordering is not maintained, message ordering may be ⭐

  • max_order_raw_size — Read after write size. If set, an RMA or atomic read operation issued after an RMA or atomic write operation, both of which are smaller than the size, will be ordered. Where the target memory locations overlap, the RMA or atomic read operation will see the results of the previous RMA or atomic write 💖
  • max_order_war_size — Write after read size. If set, an RMA or atomic write operation issued after an RMA or atomic read operation, both of which are smaller than the size, will be ordered. The RMA or atomic read operation will see the initial value of the target memory location before a subsequent RMA or atomic write updates the value 🌸
  • max_order_waw_size — Write after write size. If set, an RMA or atomic write operation issued after an RMA or atomic write operation, both of which are smaller than the size, will be ordered. The target memory location will reflect the results of the second RMA or atomic write ✨

An order size value of 0 indicates that ordering is not guaranteed. A value of -1 guarantees ordering for any data size 🎯 (yes, all of it, no cap)

mem_tag_format — Memory Tag Format 🏷️💖

The memory tag format field is used to convey information on the use of the tag and ignore parameters in the fi_tagged API calls, plus matching criteria. The provider uses this info to optimize tag matching support, including alignment with wire protocols 🧵 The following tag formats are defined:

  • FI_TAG_BITS — If specified on input to fi_getinfo, this indicates that tags contain up to 64-bits of data, and the receiver must apply ignore_bits to tags when matching receive buffers with sends. The output of fi_getinfo will set 0 or more upper bits of mem_tag_format to 0 to indicate those tag bits which are ignored or reserved by the provider. Applications must check the number of upper bits which are 0 and set them to 0 on all tag and ignore bits ⚠️

The value of FI_TAG_BITS is 0, making this the default behavior if the hints are left uninialized after being allocated by fi_allocinfo(). This format provides the most flexibility to applications, but limits provider optimization options. FI_TAG_BITS aligns with the behavior defined for libfabric versions 1.x 🎀

  • FI_TAG_MPI — FI_TAG_MPI is a constrained usage of FI_TAG_BITS. When selected, applications treat the tag as fields of data, rather than bits, with the ability to wildcard each field. The MPI tag format specifically targets MPI based implementations and applications. An MPI formatted tag consists of 2 fields: a message tag and a payload identier. The message tag is a 32-bit searchable tag. Matching on a message tag requires searching through a list of posted buffers at the receiver, which we refer to as a searchable tag. The integer tag in MPI point-to-point messages can map directly to the libfabric message tag field 💅

The second field is an identifier that corresponds to the operation or data being carried in the message payload. For example, this field may be used to identify the type of collective operation associated with a message payload. Note that only the size and behavior for the MPI tag formats are defined. Described use of the fields are only suggestions 🌈

Applications that use the MPI format should initialize their tags using the fi_tag_mpi() function. Ignore bits should be specified as FI_MPI_IGNORE_TAG, FI_MPI_IGNORE_PAYLOAD, or their bitwise OR'ing 🫶

  • FI_TAG_CCL — The FI_TAG_CCL format further restricts the FI_TAG_MPI format. When used, only a single tag field may be set, which must match exactly at the target. The field may not be wild carded. The CCL tag format targets collective communication libraries and applications. The CCL format consists of a single field: a payload identifier. The identifier corresponds to the operation or data being carried in the message payload. For example, this field may be used to identify whether a message is for point-to-point communication or part of a collective operation, and in the latter case, the type of collective operation 👑

The CCL tag format does not require searching for matching receive buffers, only directing the message to the correct virtual message queue based on to the payload identifier ⚡

Applications that use the CCL format pass in the payload identifier directly as the tag and set ignore bits to 0 💖

  • FI_TAG_MAX_FORMAT — If the value of mem_tag_format is >= FI_TAG_MAX_FORMAT, the tag format is treated as a set of bit fields. The behavior is functionally the same as FI_TAG_BITS. The following description is for backwards compatibility and describes how the provider may interpret the mem_tag_format field if the value is >= FI_TAG_MAX_FORMAT 🦋

Okay so the memory tag format thing — it's literally a way to slice your bit array into separate fields, bestie. The mem_tag_format can optionally start with a run of bits set to 0, and those are the bits your provider just plain ignores, like they're on do-not-disturb. After that little ignored prefix, the array is a bunch of alternating groups — some all 1s, some all 0s — and each group is a tagged field. 🎀

The tea on tagged fields: once you define one, applying a mask to the tagged bit array means every bit in one field flips to 1 or 0 together, as a unit. No lone wolves.

Example time, because examples ate: a mem_tag_format of 0x30FF supports 14 tagged bits split into 3 fields — first field is 2 bits, second field is 4 bits, and the final field is 8 bits. Valid masks for a tagged field like that would be a bitwise OR'ing of zero or more of 0x3000, 0x0F00, and 0x00FF. And ngl, the provider may not even validate the mask you hand it, for performance reasons. So like… don't be messy. 🙈

Why bother identifying fields inside a tag? Because a provider may be able to optimize its search routines. An app requesting tag fields has to provide tag masks that either set all mask bits for a field to all 0 or all 1 — pick a lane. When negotiating tag fields, an app can request a specific number of fields of a given size, and the provider must return a tag format that supports the requested number of fields, with each field at least the size requested, or else fail the request. A provider may increase field sizes though (generous king 👑). Also heads up: when reporting completions (see FI_CQ_FORMAT_TAGGED), it is not guaranteed that the provider clears out any unsupported tag bits in the completion entry's tag field. 💀

It's recommended that field sizes be ordered smallest to largest. And if you want a generic, unstructured tag and mask, just request a bit array of alternating 1s and 0s. Slay, simple.

🌸 tx_ctx_cnt — Transmit Context Count 🌸

Number of transmit contexts to associate with the endpoint. If you don't specify it (0), you get 1 context assigned if the endpoint supports outbound transfers. Transmit contexts are independent transmit queues that may be separately configured — each one can be bound to a separate CQ, and no ordering is defined between contexts. Bonus: no synchronization needed when accessing contexts in parallel, so go off. ✨

If the count is set to the value FI_SHARED_CONTEXT, the endpoint gets configured to use a shared transmit context, if the provider supports it. Providers that don't support shared transmit contexts will fail the request. Periodt.

See the scalable endpoint and shared contexts sections for the extra details.

🦋 rx_ctx_cnt — Receive Context Count 🦋

Number of receive contexts to associate with the endpoint. If you don't specify it, 1 context will be assigned if the endpoint supports inbound transfers. Receive contexts are independent processing queues that may be separately configured — each may be bound to a separate CQ, no ordering is defined between contexts, and no synchronization is needed when accessing contexts in parallel. Same energy as the tx side, fr fr.

If the count is set to the value FI_SHARED_CONTEXT, the endpoint will be configured to use a shared receive context, if supported by the provider. Providers that do not support shared receive contexts will fail the request. No exceptions, bestie.

See the scalable endpoint and shared contexts sections for additional details.

🎀 auth_key_size — Authorization Key Length 🎀

The length of the authorization key in bytes. This field will be 0 if authorization keys are not available or not used. It's ignored unless the fabric is opened with API version 1.5 or greater — so check your version, iykyk.

Careful: if the domain is opened with FI_AV_AUTH_KEY, auth_key_size must be 0.

💜 auth_key — Authorization Key 💜

If the fabric supports it, an authorization key (a.k.a. job key) to associate with the endpoint. An authorization key limits communication between endpoints — only peer endpoints programmed with the same authorization key may talk to each other. Authorization keys are often used to implement job keys, so that processes running in different jobs don't accidentally cross traffic (the absolute worst). 😭

The domain authorization key will be used if auth_key_size is set to 0. This field is ignored unless the fabric is opened with API version 1.5 or greater.

Careful: if the domain is opened with FI_AV_AUTH_KEY, auth_key must be NULL.

✨ TRANSMIT CONTEXT ATTRIBUTES ✨

Attributes specific to the transmit capabilities of an endpoint get specified using struct fi_tx_attr. Here's the whole thing, verbatim, no cap:

struct fi_tx_attr {
    uint64_t  caps;
    uint64_t  mode;
    uint64_t  op_flags;
    uint64_t  msg_order;
    uint64_t  comp_order;
    size_t    inject_size;
    size_t    size;
    size_t    iov_limit;
    size_t    rma_iov_limit;
    uint32_t  tclass;
};

⭐ caps — Capabilities ⭐

The requested capabilities of the context. These must be a subset of the ones you requested of the associated endpoint in fi_info->caps. See the CAPABILITIES section of fi_getinfo(3) for the full capability details. If the caps field is 0 on input to fi_getinfo(3), the applicable capability bits from the fi_info structure will be used instead. Free real estate.

The following capabilities apply to the transmit attributes: FI_MSG, FI_RMA, FI_TAGGED, FI_ATOMIC, FI_READ, FI_WRITE, FI_SEND, FI_HMEM, FI_TRIGGER, FI_FENCE, FI_MULTICAST, FI_RMA_PMEM, FI_NAMED_RX_CTX, FI_COLLECTIVE, and FI_XPU.

Many applications can straight up ignore this field and rely solely on the fi_info::caps field. Using it gives you fine grained control over the transmit capabilities tied to an endpoint — super useful when you're handling scalable endpoints with multiple transmit contexts, for example, since it lets you configure a specific transmit context with fewer capabilities than the endpoint or other transmit contexts support. Main character customization energy. 👑

🌈 mode 🌈

The operational mode bits of the context. These will be a subset of the ones associated with the endpoint. See the MODE section of fi_getinfo(3) for details. A mode value of 0 gets ignored on input to fi_getinfo(3), and the mode value of the fi_info structure is used instead. On return from fi_getinfo(3), mode is set only to those constraints specific to transmit operations.

💅 op_flags — Default transmit operation flags 💅

Flags that control how operations submitted against the context behave. The applicable flags are listed in the Operation Flags section.

🫶 msg_order — Message Ordering 🫶

Message ordering is about the order in which transport layer headers (as viewed by the application) are identified and processed. Relaxed message order lets data transfers be sent and received out of order, which may improve performance by utilizing multiple paths through the fabric from the initiating endpoint to a target endpoint. Message order applies only between a single source and destination endpoint pair — ordering between different target endpoints is not defined. Not a group thing, it's a duo thing.

Message order is determined using a set of ordering bits. Each set bit means ordering is maintained between data transfers of the specified type. Message order is defined for [read | write | send] operations submitted by an application after [read | write | send] operations. Value 0 means no ordering is specified, and value 0 may be used as input to obtain the default message order supported by the provider.

Real talk though: message ordering only applies to the end to end transmission of transport headers. It's necessary, but it does not guarantee the order in which message data is sent or received by the transport layer. And it requires matching ordering semantics on the receiving side of a data transfer operation to actually guarantee that ordering is met. Both sides gotta be in on it, bestie. 🧚

  • FI_ORDER_NONE (deprecated) — this is just an alias for value 0. It's deprecated and should not be used. Like, retire it. 💀
  • FI_ORDER_ATOMIC_RAR — Atomic read after read. If set, atomic fetch operations are transmitted in the order submitted relative to other atomic fetch operations. If not set, atomic fetches may be transmitted out of order from their submission.
  • FI_ORDER_ATOMIC_RAW — Atomic read after write. If set, atomic fetch operations are transmitted in the order submitted relative to atomic update operations. If not set, atomic fetches may be transmitted ahead of atomic updates.
  • FI_ORDER_ATOMIC_WAR — RMA write after read. If set, atomic update operations are transmitted in the order submitted relative to atomic fetch operations. If not set, atomic updates may be transmitted ahead of atomic fetches.
  • FI_ORDER_ATOMIC_WAW — RMA write after write. If set, atomic update operations are transmitted in the order submitted relative to other atomic update operations. If not — atomic updates may be transmitted out of order from their submission.
  • FI_ORDER_RAR — Read after read. If set, RMA and atomic read operations are transmitted in the order submitted relative to other RMA and atomic read operations. If not set, RMA and atomic reads may be transmitted out of order from their submission.
  • FI_ORDER_RAS — Read after send. If set, RMA and atomic read operations are transmitted in the order submitted relative to message send operations, including tagged sends. If not set, RMA and atomic reads may be transmitted ahead of sends.
  • FI_ORDER_RAW — Read after write. If set, RMA and atomic read operations are transmitted in the order submitted relative to RMA and atomic write operations. If not set, RMA and atomic reads may be transmitted ahead of RMA and atomic writes.
  • FI_ORDER_RMA_RAR — RMA read after read. If set, RMA read operations are transmitted in the order submitted relative to other RMA read operations. If not set, RMA reads may be transmitted out of order from their submission.
  • FI_ORDER_RMA_RAW — RMA read after write. If set, RMA read operations are transmitted in the order submitted relative to RMA write operations. If not set, RMA reads may be transmitted ahead of RMA writes.
  • FI_ORDER_RMA_WAR — RMA write after read. If set, RMA write operations are transmitted in the order submitted relative to RMA read operations. If not set, RMA writes may be transmitted ahead of RMA reads.
  • FI_ORDER_RMA_WAW — RMA write after write, bestie. If this is set, your RMA write ops get transmitted in the order you submitted them relative to other RMA write ops. If it’s not set, those RMA writes can totally be transmitted out of order from their submission. 💅
  • FI_ORDER_SAR — Send after read. If set, message send operations (including tagged sends) are transmitted in the order submitted relative to RMA and atomic read operations. If not set, message sends may be transmitted ahead of RMA and atomic reads. 👀
  • FI_ORDER_SAS — Send after send. If set, message send operations (including tagged sends) are transmitted in the order submitted relative to other message sends. If not set, message sends may be transmitted out of order from their submission. no cap.
  • FI_ORDER_SAW — Send after write. If set, message send operations (including tagged sends) are transmitted in order submitted relative to RMA and atomic write operations. If not set, message sends may be transmitted ahead of RMA and atomic writes. ✨
  • FI_ORDER_WAR — Write after read. If set, RMA and atomic write operations are transmitted in the order submitted relative to RMA and atomic read operations. If not set, RMA and atomic writes may be transmitted ahead of RMA and atomic reads. 🦋
  • FI_ORDER_WAS — Write after send. If set, RMA and atomic write operations are transmitted in the order submitted relative to message send operations, including tagged sends. If not set, RMA and atomic writes may be transmitted ahead of sends. 💀
  • FI_ORDER_WAW — Write after write. If set, RMA and atomic write operations are transmitted in the order submitted relative to other RMA and atomic write operations. If not set, RMA and atomic writes may be transmitted out of order from their submission. periodt.

🎀 comp_order — Completion Ordering 🎀

This field is provided for version 1 compatibility and should be set to 0. It’s deprecated, so like… only touch it if you’re literally stuck in a version 1 situation. 🙈

Completion ordering refers to the order in which completed requests are written into the completion queue. Supported completion order values are:

  • FI_ORDER_NONE (deprecated) — No ordering is defined for completed operations. Requests submitted to the transmit context may complete in any order. chaos, but make it legal. 💅
  • FI_ORDER_STRICT (deprecated) — Requests complete in the order in which they are submitted to the transmit context. very type A of them. 🎀

🧸 inject_size 🧸

The requested inject operation size (see the FI_INJECT flag) that the context will support. This is the maximum size data transfer that can be associated with an inject operation (such as fi_inject) or may be used with the FI_INJECT data transfer flag. so if you’re trying to sneak a lil payload in, this is your limit. 💌

📏 size 📏

The size of the transmit context. The mapping of the size value to resources is provider specific, but it is directly related to the number of command entries allocated for the endpoint. A smaller size value consumes fewer hardware and software resources, while a larger size allows queuing more transmit requests. lowkey a tradeoff, highkey important. ⚖️

While the size attribute guides the size of underlying endpoint transmit queue, there is not necessarily a one-to-one mapping between a transmit operation and a queue entry. A single transmit operation may consume multiple queue entries; for example, one per scatter-gather entry. Additionally, the size field is intended to guide the allocation of the endpoint’s transmit context. Specifically, for connectionless endpoints, there may be lower-level queues used to track communication on a per peer basis. The sizes of any lower-level queues may only be significantly smaller than the endpoint’s transmit size, in order to reduce resource utilization. the way that resource management is giving main character energy. 👑

🧚 iov_limit 🧚

This is the maximum number of IO vectors (scatter-gather elements) that a single posted operation may reference. don’t get greedy with your vectors, bestie. 🫶

🌸 rma_iov_limit 🌸

This is the maximum number of RMA IO vectors (scatter-gather elements) that an RMA or atomic operation may reference. The rma_iov_limit corresponds to the rma_iov_count values in RMA and atomic operations. See struct fi_msg_rma and struct fi_msg_atomic in fi_rma.3 and fi_atomic.3, for additional details. This limit applies to both the number of RMA IO vectors that may be specified when initiating an operation from the local endpoint, as well as the maximum number of IO vectors that may be carried in a single request from a remote endpoint. so it’s a two-way limit, iykyk. 🔥

🚦 Traffic Class (tclass) 🚦

Traffic classes can be a differentiated services code point (DSCP) value, one of the following defined labels, or a provider-specific definition. If tclass is unset or set to FI_TC_UNSPEC, the endpoint will use the default traffic class associated with the domain. let’s get this traffic sorted, shall we? 🌈

  • FI_TC_BEST_EFFORT — This is the default in the absence of any other local or fabric configuration. This class carries the traffic for a number of applications executing concurrently over the same network infrastructure. Even though it is shared, network capacity and resource allocation are distributed fairly across the applications. the communal vibe, but make it fair. 🤝
  • FI_TC_BULK_DATA — This class is intended for large data transfers associated with I/O and is present to separate sustained I/O transfers from other application inter-process communications. big data energy, but like, for actual big data. 📦
  • FI_TC_DEDICATED_ACCESS — This class operates at the highest priority, except the management class. It carries a high bandwidth allocation, minimum latency targets, and the highest scheduling and arbitration priority. VIP access only, no cap. 👑
  • FI_TC_LOW_LATENCY — This class supports low latency, low jitter data patterns typically caused by transactional data exchanges, barrier synchronizations, and collective operations that are typical of HPC applications. This class often requires maximum tolerable latencies that data transfers must achieve for correct or performance operations. Fulfillment of such requests in this class will typically require accompanying bandwidth and message size limitations so as not to consume excessive bandwidth at high priority. it’s giving “fast but please don’t hog the bandwidth.” 🏎️
  • FI_TC_NETWORK_CTRL — This class is intended for traffic directly related to fabric (network) management, which is critical to the correct operation of the network. Its use is typically restricted to privileged network management applications. admin business only, bestie. 🔐
  • FI_TC_SCAVENGER — This class is used for data that is desired but does not have strict delivery requirements, such as in-band network or application level monitoring data. Use of this class indicates that the traffic is considered lower priority and should not interfere with higher priority workflows. it’s the “nice to have, not need to have” lane. 🦋
  • fi_tc_dscp_set / fi_tc_dscp_get — DSCP values are supported via the DSCP get and set functions. The definitions for DSCP values are outside the scope of libfabric. See the fi_tc_dscp_set and fi_tc_dscp_get function definitions for details on their use. so like, go read those if you wanna get into the DSCP weeds. 🌿

💖 RECEIVE CONTEXT ATTRIBUTES 💖

okay so the receive side is lowkey its own whole vibe, and the receive-specific capabilities of an endpoint are specified using struct fi_rx_attr. Here’s the exact shape, no cap:

struct fi_rx_attr {
    uint64_t  caps;
    uint64_t  mode;
    uint64_t  op_flags;
    uint64_t  msg_order;
    uint64_t  comp_order;
    size_t    size;
    size_t    iov_limit;
};

caps - Capabilities

The requested capabilities of the context. These must be a subset of the capabilities requested for the associated endpoint in fi_info->caps. See the CAPABILITIES section of fi_getinfo(3) for the full capability details. If the caps field is 0 on input to fi_getinfo(3), the applicable capability bits from the fi_info structure will be used instead ✨

The following capabilities apply to the receive attributes: FI_MSG, FI_RMA, FI_TAGGED, FI_ATOMIC, FI_REMOTE_READ, FI_REMOTE_WRITE, FI_RECV, FI_HMEM, FI_TRIGGER, FI_RMA_PMEM, FI_DIRECTED_RECV, FI_TAGGED_DIRECTED_RECV, FI_TAGGED_MULTI_RECV, FI_MULTI_RECV, FI_SOURCE, FI_RMA_EVENT, FI_SOURCE_ERR, FI_COLLECTIVE, and FI_XPU 🫶

Many applications can totally ignore this field and rely solely on the fi_info::caps field. Using this field gives you fine-grained control over the receive capabilities associated with an endpoint. It’s useful when you’re handling scalable endpoints with multiple receive contexts, for example, and it lets you configure a specific receive context with fewer capabilities than the endpoint or the other receive contexts support 💅

mode

The operational mode bits of the context. The mode bits will be a subset of those associated with the endpoint. See the MODE section of fi_getinfo(3) for details. A mode value of 0 will be ignored on input to fi_getinfo(3), and the mode value from the fi_info structure will be used instead. On return from fi_getinfo(3), the mode will be set only to those constraints specific to receive operations ✨

op_flags - Default receive operation flags

Flags that control the operation of operations submitted against the context. The applicable flags are listed in the Operation Flags section 👑

msg_order - Message Ordering

For a description of message ordering, see the msg_order field in the Transmit Context Attribute section. Receive context message ordering defines the order in which received transport message headers are processed when they’re received by an endpoint. When ordering is set, it means message headers will be processed in order based on how the transmit side identified the messages. Typically, that means messages are handled in order based on a message-level sequence number 🦋

The following ordering flags, as defined for transmit ordering, also apply to the processing of received operations: FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, and FI_ORDER_ATOMIC_WAW ⭐

comp_order - Completion Ordering

This field is provided for version 1 compatibility and should be set to 0.

Deprecated 💀

Completion ordering refers to the order in which completed requests are written into the completion queue. Supported completion order values are:

  • FI_ORDER_DATA (deprecated) — When set, this bit indicates that received data is written into memory in order. Data ordering applies to memory accessed as part of a single operation and between operations if message ordering is guaranteed.
  • FI_ORDER_NONE (deprecated) — No ordering is defined for completed operations. Requests submitted to the transmit context may complete in any order.
  • FI_ORDER_STRICT (deprecated) — Requests complete in the order in which they are submitted to the transmit context.

total_buffered_recv

This field is provided for version 1 compatibility and should be set to 0 🙈

size

The size of the receive context. The mapping of the size value to resources is provider specific, but it’s directly related to the number of command entries allocated for the endpoint. A smaller size value consumes fewer hardware and software resources, while a larger size allows queuing more transmit requests 👑

While the size attribute guides the size of the underlying endpoint receive queue, there isn’t necessarily a one-to-one mapping between a receive operation and a queue entry. A single receive operation may consume multiple queue entries; for example, one per scatter-gather entry. Additionally, the size field is intended to guide the allocation of the endpoint’s receive context. Specifically, for connectionless endpoints, there may be lower-level queues used to track communication on a per-peer basis. The sizes of any lower-level queues may only be significantly smaller than the endpoint’s receive size, in order to reduce resource utilization 🎀

iov_limit

This is the maximum number of IO vectors (scatter-gather elements) that a single posted operation may reference 🌸

🦋 SCALABLE ENDPOINTS 🦋

A scalable endpoint is a communication portal that supports multiple transmit and receive contexts. Scalable endpoints are loosely modeled after the networking concept of transmit/receive side scaling, also known as multi-queue. Support for scalable endpoints is domain specific. Scalable endpoints may improve the performance of multi-threaded and parallel applications by allowing threads to access independent transmit and receive queues. A scalable endpoint has a single transport-level address, which can reduce the memory requirements needed to store remote addressing data, versus using standard endpoints. Scalable endpoints cannot be used directly for communication operations, and they require the application to explicitly create transmit and receive contexts as described below 💖

fi_tx_context

Transmit contexts are independent transmit queues. Ordering and synchronization between contexts are not defined. Conceptually, a transmit context behaves similar to a send-only endpoint. A transmit context may be configured with fewer capabilities than the base endpoint and with different attributes (such as ordering requirements and inject size) than other contexts associated with the same scalable endpoint. Each transmit context has its own completion queue. The number of transmit contexts associated with an endpoint is specified during endpoint creation 🎀

The fi_tx_context call is used to retrieve a specific context, identified by an index (see above for details on transmit context attributes). Providers may dynamically allocate contexts when fi_tx_context is called, or may statically create all contexts when fi_endpoint is invoked. By default, a transmit context inherits the properties of its associated endpoint. However, applications may request context-specific attributes through the attr parameter. Support for per-transmit-context attributes is provider specific and not guaranteed. Providers will return the actual attributes assigned to the context through the attr parameter, if provided ✨

fi_rx_context

Receive contexts are independent receive queues for receiving incoming data. Ordering and synchronization between contexts are not guaranteed. Conceptually, a receive context behaves similar to a receive-only endpoint. A receive context may be configured with fewer capabilities than the base endpoint and with different attributes (such as ordering requirements and inject size) than other contexts associated with the same scalable endpoint. Each receive context has its own completion queue. The number of receive contexts associated with an endpoint is specified during endpoint creation 🦋

Receive contexts are often associated with steering flows, which specify which incoming packets targeting a scalable endpoint to process. However, receive contexts may be targeted directly by the initiator, if supported by the underlying protocol. Such contexts are referred to as named. Support for named contexts must be indicated by setting the caps FI_NAMED_RX_CTX capability when the corresponding endpoint is created. Support for named receive contexts is coordinated with address vectors. See fi_av(3) and fi_rx_addr(3) 👑

The fi_rx_context call is used to retrieve a specific context, identified by an index (see above for details on receive context attributes). Providers may dynamically allocate contexts when fi_rx_context is called, or may statically create all contexts when fi_endpoint is invoked. By default, a receive context inherits the properties of its associated endpoint. However, applications may request context-specific attributes through the attr parameter. Support for per-receive-context attributes is provider specific and not guaranteed. Providers will return the actual attributes assigned to the context through the attr parameter, if provided 💜

🌈 SHARED CONTEXTS 🌈

Okay so bestie, shared contexts are exactly what they sound like: transmit and receive contexts that are explicitly shared among one or more endpoints. 💖 A shareable context lets an app use a single dedicated provider resource among multiple transport addressable endpoints. This can greatly reduce the resources needed to manage communication over multiple endpoints by multiplexing transmit and/or receive processing — with the potential cost of serializing access across multiple endpoints. Support for shareable contexts is domain specific.

Conceptually, shareable transmit contexts are transmit queues that may be accessed by many endpoints. The use of a shared transmit context is mostly opaque to an application. Applications must allocate and bind shared transmit contexts to endpoints, but operations are posted directly to the endpoint. Shared transmit contexts are not associated with completion queues or counters. Completed operations are posted to the CQs bound to the endpoint. An endpoint may only be associated with a single shared transmit context.

Unlike shared transmit contexts, applications interact directly with shared receive contexts. Users post receive buffers directly to a shared receive context, with the buffers usable by any endpoint bound to the shared receive context. Shared receive contexts are not associated with completion queues or counters. Completed receive operations are posted to the CQs bound to the endpoint. An endpoint may only be associated with a single receive context, and all connectionless endpoints associated with a shared receive context must also share the same address vector.

Endpoints associated with a shared transmit context may use dedicated receive contexts, and vice-versa. Or an endpoint may use shared transmit and receive contexts. And there is no requirement that the same group of endpoints sharing a context of one type also share the context of an alternate type. Furthermore, an endpoint may use a shared context of one type, but a scalable set of contexts of the alternate type. 🦋

🎀 fi_stx_context 🎀

This call is used to open a shareable transmit context (see above for details on the transmit context attributes). Endpoints associated with a shared transmit context must use a subset of the transmit context’s attributes. Note that this is the reverse of the requirement for transmit contexts for scalable endpoints. 💅

🦋 fi_srx_context 🦋

This allocates a shareable receive context (see above for details on the receive context attributes). Endpoints associated with a shared receive context must use a subset of the receive context’s attributes. Note that this is the reverse of the requirement for receive contexts for scalable endpoints. ✨

🔌 SOCKET ENDPOINTS 🔌

⚠️ Experimental alert: the following feature and description should be considered experimental. Until the experimental tag is removed, the interfaces, semantics, and data structures associated with socket endpoints may change between library versions. No cap, bestie — proceed with that in mind. 💀

This section applies to endpoints of type FI_EP_SOCK_STREAM and FI_EP_SOCK_DGRAM, commonly referred to as socket endpoints.

Socket endpoints are defined with semantics that allow them to more easily be adopted by developers familiar with the UNIX socket API, or by middleware that exposes the socket API, while still taking advantage of high-performance hardware features.

The key difference between socket endpoints and other active endpoints are socket endpoints use synchronous data transfers. Buffers passed into send and receive operations revert to the control of the application upon returning from the function call. As a result, no data transfer completions are reported to the application, and socket endpoints are not associated with completion queues or counters.

Socket endpoints support a subset of message operations: fi_send, fi_sendv, fi_sendmsg, fi_recv, fi_recvv, fi_recvmsg, and fi_inject. Because data transfers are synchronous, the return value from send and receive operations indicate the number of bytes transferred on success, or a negative value on error, including -FI_EAGAIN if the endpoint cannot send or receive any data because of full or empty queues, respectively. 😭

Socket endpoints are associated with event queues and address vectors, and process connection management events asynchronously, similar to other endpoints. Unlike UNIX sockets, socket endpoint must still be declared as either active or passive.

Socket endpoints behave like non-blocking sockets. In order to support select and poll semantics, active socket endpoints are associated with a file descriptor that is signaled whenever the endpoint is ready to send and/or receive data. The file descriptor may be retrieved using fi_control. 👑

🚩 OPERATION FLAGS 🚩

Operation flags are obtained by OR-ing the following flags together. Operation flags define the default flags applied to an endpoint’s data transfer operations, where a flags parameter is not available. Data transfer operations that take flags as input override the op_flags value of transmit or receive context attributes of an endpoint.

  • FI_COMMIT_COMPLETE — Indicates that a completion should not be generated (locally or at the peer) until the result of an operation have been made persistent. See fi_cq(3) for additional details on completion semantics.
  • FI_COMPLETION — Indicates that a completion queue entry should be written for data transfer operations. This flag only applies to operations issued on an endpoint that was bound to a completion queue with the FI_SELECTIVE_COMPLETION flag set, otherwise, it is ignored. See the fi_ep_bind section above for more detail.
  • FI_DELIVERY_COMPLETE — Indicates that a completion should be generated when the operation has been processed by the destination endpoint(s). See fi_cq(3) for additional details on completion semantics.
  • FI_INJECT — Indicates that all outbound data buffers should be returned to the user’s control immediately after a data transfer call returns, even if the operation is handled asynchronously. This may require that the provider copy the data into a local buffer and transfer out of that buffer. A provider can limit the total amount of send data that may be buffered and/or the size of a single send that can use this flag. This limit is indicated using inject_size (see inject_size above).
  • FI_INJECT_COMPLETE — Indicates that a completion should be generated when the source buffer(s) may be reused. See fi_cq(3) for additional details on completion semantics.
  • FI_MULTICAST — Indicates that data transfers will target multicast addresses by default. Any fi_addr_t passed into a data transfer operation will be treated as a multicast address.
  • FI_MULTI_RECV — Applies to posted receive operations. This flag allows the user to post a single buffer that will receive multiple incoming messages. Received messages will be packed into the receive buffer until the buffer has been consumed. Use of this flag may cause a single posted receive operation to generate multiple completions as messages are placed into the buffer. The placement of received data into the buffer may be subjected to provider specific alignment restrictions. The buffer will be released by the provider when the available buffer space falls below the specified minimum (see FI_OPT_MIN_MULTI_RECV).
  • FI_TRANSMIT_COMPLETE — Indicates that a completion should be generated when the transmit operation has completed relative to the local provider. See fi_cq(3) for additional details on completion semantics.

📝 NOTES 📝

Users should call fi_close to release all resources allocated to the fabric endpoint. 🧚

Endpoints allocated with the FI_CONTEXT or FI_CONTEXT2 mode bits set must typically provide struct fi_context(2) as their per operation context parameter. (See fi_getinfo.3 for details.) However, when FI_SELECTIVE_COMPLETION is enabled to suppress CQ completion entries, and an operation is initiated without the FI_COMPLETION flag set, then the context parameter is ignored. An application does not need to pass in a valid struct fi_context(2) into such data transfers.

Operations that complete in error that are not associated with valid operational context will use the endpoint context in any error reporting structures.

Although applications typically associate individual completions with either completion queues or counters, an endpoint can be attached to both a counter and completion queue. When combined with using selective completions, this allows an application to use counters to track successful completions, with a CQ used to report errors. Operations that complete with an error increment the error counter and generate a CQ completion event. 💖

As mentioned in fi_getinfo(3), the ep_attr structure can be used to query providers that support various endpoint attributes. fi_getinfo can return provider info structures that can support the minimal set of requirements (such that the application maintains correctness). However, it can also return provider info structures that exceed application requirements. As an example, consider an application requesting no msg_order. The resulting output from fi_getinfo may have all the ordering bits set. The application can reset the ordering bits it does not require before creating the endpoint. The provider is free to implement a stricter ordering than is required by the application. ✨

💖 RETURN VALUES 💖

Returns 0 on success. On error, a negative value corresponding to fabric errno is returned. For fi_cancel, a return value of 0 indicates that the cancel request was submitted for processing, a return value of -FI_EAGAIN indicates that the request could not be submitted and that it should be retried once progress has been made. For fi_setopt/fi_getopt, a return value of -FI_ENOPROTOOPT indicates the provider does not support the requested option.

Fabric errno values are defined in rdma/fi_errno.h.

💀 ERRORS 💀

  • -FI_EDOMAIN — A resource domain was not bound to the endpoint or an attempt was made to bind multiple domains.
  • -FI_ENOCQ — The endpoint has not been configured with necessary completion queue.
  • -FI_EOPBADSTATE — The endpoint’s state does not permit the requested operation.

👀 SEE ALSO 👀

The exact reference squad is:

fi_getinfo(3), fi_domain(3), fi_cq(3) fi_msg(3), fi_tagged(3), fi_rma(3) fi_peer(3)

🫶 AUTHORS 🫶

OpenFabrics.

This iconic little page is from the Libfabric Programmer’s Manual, dated 2025-06-20, and it’s fi_endpoint(3) — periodt. 💅✨

📦 where this page comes from