RXE(7)

✨ NAME ✨

rxe — software RDMA over Ethernet, bestie 💖

So like, the manual page is RXE(7) from the Miscellaneous Information Manual, and this whole thing is about rxe. Yes queen, we're doing RDMA over plain old Ethernet, no fancy hardware required.

🎀 SYNOPSIS 🎀

modprobe rdma_rxe

This is usually performed by a configuration utility (see rdma link(8)).

🦋 DESCRIPTION 🦋

The rdma_rxe kernel module provides a software implementation of the RoCEv2 protocol. And what is RoCEv2, you ask? It's an RDMA transport protocol that exists on top of UDP/IPv4 or UDP/IPv6, no cap. The InfiniBand (IB) Base Transport Header (BTH) is encapsulated in the UDP packet. Iconic behaviour honestly.

Once a RXE instance has been created, communicating via RXE is the same as communicating via any OFED compatible Infiniband HCA — literally the same vibe — albeit in some cases with addressing implications. 🙈

In particular: while the use of a GRH header is optional within IB subnets, it is mandatory with RoCE. Verbs applications written over IB verbs should work seamlessly, but they require provisioning of GRH information when creating address vectors. The library and driver are modified to provide for mapping from GID to MAC addresses required by the hardware. So like, the plumbing is handled, but you do have to give it the GRH info. Slay, but mind the details. 💅

🧚 FILES 🧚

• /sys/class/infiniband/rxe[0,1,...] — Directory that holds RDMA device information. The format is the same as other RDMA devices.
• /sys/module/rdma_rxe_net/parameters/mtu — Write only file used to configure RoCE and Ethernet MTU values.
• /sys/module/rdma_rxe/parameters/max_ucontext — Read/Write file that sets a limit on the number of UCs allowed per RXE device.
• /sys/module/rdma_rxe/parameters/max_qp — Read/Write file that sets a limit on the number of QPs allowed per RXE device.
• /sys/module/rdma_rxe/parameters/max_qp_wr — Read/Write file that sets a limit on the number of WRs per QP allowed per RXE device.
• /sys/module/rdma_rxe/parameters/max_mr — Read/Write file that sets a limit on the number of MRs allowed per RXE device.
• /sys/module/rdma_rxe/parameters/max_fmr — Read/Write file that sets a limit on the number of FMRs allowed per RXE device.
• /sys/module/rdma_rxe/parameters/max_cq — Read/Write file that sets a limit on the number of CQs allowed per RXE device.
• /sys/module/rdma_rxe/parameters/max_log_cqe — Read/Write file that sets a limit on the log base 2 of the number of CQEs per CQ allowed per RXE device.
• /sys/module/rdma_rxe/parameters/max_inline_data — Read/Write file that sets a limit on the maximum amount of inline data per WR allowed per RXE device.

⚠️ Okay bestie, listen up: the above configuration parameters only affect a new RXE instance when it is created, not afterwards. Not after. Once it's born, that's it, those knobs are decorative. 😭

• /sys/module/rdma_rxe/parameters/crc_disable — Read/Write file that controls the disabling of ICRC computation. Set to a nonzero value for TRUE. Zero for FALSE.
• /sys/module/rdma_rxe/parameters/fast_comp|req|resp|arb — Read/Write file that enables calling kernel tasklets as subroutines to reduce latency.
• /sys/module/rdma_rxe/parameters/nsec_per_packet|kbyte — Read/Write file that controls static rate pacing for output packets. If set to nonzero values the minimum delay to the next packet is set to nsec_per_kbyte * sizeof(current packet in KBytes) or nsec_per_packet, which ever is less.
• /sys/module/rdma_rxe/parameters/max_packet_per_ack — Read/Write file that controls the issuing of acks by the responder during a long message. If set, additional acks will be generated every max_pkt_per_ack packets.
• /sys/module/rdma_rxe/parameters/max_skb_per_qp — Read/Write file that controls the number of skbs (packets) that a requester can queue for sending internally.
• /sys/module/rdma_rxe/parameters/max_req_comp_gap — Read/Write file that controls the maximum gap between the PSN of request packets send and ack packets received.
• /sys/module/rdma_rxe/parameters/default_mtu — Read/Write file that controls the default mtu used for UD packets.

💜 SEE ALSO 💜

rdma(8), verbs(7),

👑 AUTHORS 👑

Written by John Groves, Frank Zago and Bob Pearson at System Fabric Works. Legends, fr fr. 🫶

Okay that's the whole page bestie, version 1.0.0, dated 2011-06-29, filed under RXE(7) in the Miscellaneous Information Manual. Go configure something cute. Byeee 💖✨🎀