RDMA_CM(7)

📦
version 65.0 · 3 versions
, in alpine 3.24, arch rolling, centos-stream 10, centos-stream 9, fedora 44, nixos 26.05, nixpkgs unstable, ubuntu resolute, ubuntu stonking

💖 RDMA_CM 💖

okay bestie, gather round, because we're about to talk about the rdma_cm — aka the RDMA communication manager, straight outta the Librdmacm Programmer's Manual. yes this is a real manpage, yes it's a lot, yes we're doing it together. no cap. 🫶

✨ NAME ✨

rdma_cm — RDMA communication manager. that's the whole name. short, sweet, iconic. 👑

✨ SYNOPSIS ✨

#include <rdma/rdma_cma.h>

literally one include and you're in. 💅

✨ DESCRIPTION ✨

it's used to establish communication over RDMA transports. that's it, that's the description. the manpage said it in one line and honestly? ate. 🔥

✨ NOTES ✨

so the RDMA CM is a communication manager, and its whole job is setting up reliable, connected and unreliable datagram data transfers. it gives you an RDMA transport neutral interface for establishing connections, which is a fancy way of saying "you don't have to care which transport, we got you."

the API concepts are based on sockets — yes, like your fave socket() energy — but they've been adapted for queue pair (QP) based semantics. what does that mean? two things:

  • communication must be over a specific RDMA device, and
  • data transfers are message based.

the RDMA CM can control both the QP and the communication management (that's connection setup / teardown) portions of an RDMA API — or it can do only the communication management piece. either way, it works in conjunction with the verbs API defined by the libibverbs library, because libibverbs is the one providing the underlying interfaces needed to actually send and receive data. it's giving collab. 🤝

now, the RDMA CM can operate asynchronously or synchronously, and you control which mode via the rdma_cm event channel parameter in specific calls:

  • if you provide an event channel, an rdma_cm identifier will report its event data (results of connecting, for example) on that channel. ✨
  • if you don't provide a channel, then all rdma_cm operations for the selected rdma_cm identifier will block until they complete. so like, know what you're signing up for. 😭

and finally, the RDMA CM gives an option to different libibverbs providers to advertise and use various QP configuration options that are specific to that provider. this functionality is called ECE (enhanced connection establishment). iykyk. 🌈

✨ RDMA VERBS ✨

the rdma_cm supports the full range of verbs available through the libibverbs library and interfaces. but wait — it also provides wrapper functions for some of the more commonly used verbs funcationality. (that's a typo in the original and we're keeping it, character for character, because accuracy is the vibe. 💀)

the full set of abstracted verb calls is:

  • rdma_reg_msgs — register an array of buffers for sending and receiving
  • rdma_reg_read — registers a buffer for RDMA read operations
  • rdma_reg_write — registers a buffer for RDMA write operations
  • rdma_dereg_mr — deregisters a memory region
  • rdma_post_recv — post a buffer to receive a message
  • rdma_post_send — post a buffer to send a message
  • rdma_post_read — post an RDMA to read data into a buffer
  • rdma_post_write — post an RDMA to send data from a buffer
  • rdma_post_recvv — post a vector of buffers to receive a message
  • rdma_post_sendv — post a vector of buffers to send a message
  • rdma_post_readv — post a vector of buffers to receive an RDMA read
  • rdma_post_writev — post a vector of buffers to send an RDMA write
  • rdma_post_ud_send — post a buffer to send a message on a UD QP
  • rdma_get_send_comp — get completion status for a send or RDMA operation
  • rdma_get_recv_comp — get information about a completed receive

slay. that's fifteen wrappers, all accounted for, none skipped. 🎀

🌸 CLIENT OPERATION 🌸

this section is the general overview of the basic operation for the active, or client, side of communication. bestie listen — this flow assumes asynchronous operation with low level call details shown. if you want synchronous operation, then calls to rdma_create_event_channel, rdma_get_cm_event, rdma_ack_cm_event, and rdma_destroy_event_channel would be eliminated. gone. not invited to the party. 🙈

also, abstracted calls — such as rdma_create_ep — encapsulate several of these calls under a single API. and you can always go peek at the example applications for code samples. 💖

a general connection flow would be:

  • rdma_getaddrinfo — retrieve address information of the destination
  • rdma_create_event_channel — create channel to receive events
  • rdma_create_id — allocate an rdma_cm_id, this is conceptually similar to a socket
  • rdma_resolve_addr — obtain a local RDMA device to reach the remote address
  • rdma_get_cm_event — wait for RDMA_CM_EVENT_ADDR_RESOLVED event
  • rdma_ack_cm_event — ack event
  • rdma_create_qp — allocate a QP for the communication
  • rdma_resolve_route — determine the route to the remote address
  • rdma_get_cm_event — wait for RDMA_CM_EVENT_ROUTE_RESOLVED event
  • rdma_ack_cm_event — ack event
  • rdma_connect — connect to the remote server
  • rdma_get_cm_event — wait for RDMA_CM_EVENT_ESTABLISHED event
  • rdma_ack_cm_event — ack event
  • Perform data transfers over connection
  • rdma_disconnect — tear-down connection
  • rdma_get_cm_event — wait for RDMA_CM_EVENT_DISCONNECTED event
  • rdma_ack_cm_event — ack event
  • rdma_destroy_qp — destroy the QP
  • rdma_destroy_id — release the rdma_cm_id
  • rdma_destroy_event_channel — release the event channel
  • rdma_freeaddrinfo — release the list of rdma_addrinfo structures
  • rdma_set_local_ece — set desired ECE options

an almost identical process is used to setup unreliable datagram (UD) communication between nodes. however, no actual connection is formed between QPs, so disconnection is not needed. (no connection, no breakup text. iconic.) 💅

although this example shows the client initiating the disconnect, either side of a connection may initiate the disconnect. periodt.

🌸 SERVER OPERATION 🌸

and now for the passive, or server, side of communication. same energy, different vibes. a general connection flow would be:

  • rdma_create_event_channel — create channel to receive events
  • rdma_create_id — allocate an rdma_cm_id, this is conceptually similar to a socket
  • rdma_bind_addr — set the local port number to listen on
  • rdma_listen — begin listening for connection requests
  • rdma_get_cm_event — wait for RDMA_CM_EVENT_CONNECT_REQUEST event with a new rdma_cm_id
  • rdma_create_qp — allocate a QP for the communication on the new rdma_cm_id
  • rdma_accept — accept the connection request
  • rdma_ack_cm_event — ack event
  • rdma_get_cm_event — wait for RDMA_CM_EVENT_ESTABLISHED event
  • rdma_ack_cm_event — ack event
  • Perform data transfers over connection
  • rdma_get_cm_event — wait for RDMA_CM_EVENT_DISCONNECTED event
  • rdma_ack_cm_event — ack event
  • rdma_disconnect — tear-down connection
  • rdma_destroy_qp — destroy the QP
  • rdma_destroy_id — release the connected rdma_cm_id
  • rdma_destroy_id — release the listening rdma_cm_id
  • rdma_destroy_event_channel — release the event channel
  • rdma_get_remote_ece — get ECe options sent by the client
  • rdma_set_local_ece — set desired ECE options

yes, rdma_destroy_id shows up twice, and no, that's not a glitch — you're releasing two different ids, bestie. one connected, one listening. two icons, two farewells. 👑

⭐ RETURN CODES ⭐

=  0   success

= -1   error - see errno for more details

most librdmacm functions return 0 to indicate success, and a -1 return value to indicate failure.

if a function operates asynchronously, a return value of 0 means the operation was successfully started. but heads up — the operation could still complete in error. users should check the status of the related event. do not skip the vibe check on your events, fr fr. 🫶

if the return value is -1, then errno will contain additional information regarding the reason for the failure.

⚠️ okay this next part is important and slightly cursed. prior versions of the library would return -errno and not set errno for some cases related to ENOMEM, ENODEV, ENODATA, EINVAL, and EADDRNOTAVAIL codes. applications that want to check these codes and have compatibility with prior library versions must manually set errno to the negative of the return code if it is < -1. so if you're dealing with old code, that's the homework. good luck babe. 💀

💜 SEE ALSO 💜

the whole friend group, all of them, no one left out:

(yes, rdma_get_remote_ece(3) is in there twice. it's just that popular. and yes, rdma_post_read(3) is missing its comma in the original — we kept it exactly as-is because that's the whole point. 💖)

and that's the whole page, bestie!! 🎀✨ this one came from librdmacm, dated 2010-07-19 — vintage but still serving. go forth and resolve some routes. you got this. 💜🦋🌸

📦 where this page comes from