IRDMA(4)

🌸 IRDMA(4) — the RDMA era, Intel E810 edition 🌸

okay bestie, this is the irdma(4) manual page and we're gonna make the kernel networking lore actually make sense. ✨

💖 NAME 💖

irdma — RDMA FreeBSD driver for Intel(R) Ethernet Controller E810. That's the whole vibe: it's the RDMA driver for that Intel E810 life. 💅

✨ SYNOPSIS ✨

This module relies on ice(4). no cap, that's the dependency. 🫶

The following kernel options should be included in the configuration:

options OFED
options OFED_DEBUG_INIT
options COMPAT_LINUXKPI
options SDP
options IPOIB_CM

🦋 DESCRIPTION 🦋

✨ Features ✨

The irdma driver provides RDMA protocol support on RDMA-capable Intel Ethernet 800 Series NICs which are supported by ice(4). It supports both iWARP and RoCEv2 protocols. Both. Not one, not the other — both. 💖

🌈 CONFIGURATION 🌈

🎀 TUNABLES 🎀

Tunables can be set at the loader(8) prompt before booting the kernel or stored in loader.conf(5). so you can set them early or save them for later. 🫶

• dev.irdma<interface_number>.roce_enable — enables RoCEv2 protocol usage on the <interface_numer> interface. By default RoCEv2 protocol is used.
• dev.irdma<interface_number>.dcqcn_cc_cfg_valid — indicates that all DCQCN parameters are valid and should be updated in registers or QP context. Setting this parameter to 1 means that settings in dcqcn_min_dec_factor, dcqcn_min_rate_MBps, dcqcn_F, dcqcn_T, dcqcn_B, dcqcn_rai_factor, dcqcn_hai_factor, dcqcn_rreduce_mperiod are taken into account. Otherwise default values are used. Note: roce_enable must also be set for this tunable to take effect.
• dev.irdma<interface_number>.dcqcn_min_dec_factor — the minimum factor by which the current transmit rate can be changed when processing a CNP. Value is given as a percentage (1-100). Note: roce_enable and dcqcn_cc_cfg_valid must also be set for this tunable to take effect.
• dev.irdma<interface_number>.dcqcn_min_rate_MBps — the minimum value, in Mbits per second, for rate to limit. Note: roce_enable and dcqcn_cc_cfg_valid must also be set for this tunable to take effect.
• dev.irdma<interface_number>.dcqcn_F — the number of times to stay in each stage of bandwidth recovery. Note: roce_enable and dcqcn_cc_cfg_valid must also be set for this tunable to take effect.
• dev.irdma<interface_number>.dcqcn_T — the number of microseconds that should elapse before increasing the CWND in DCQCN mode. Note: roce_enable and dcqcn_cc_cfg_valid must also be set for this tunable to take effect.
• dev.irdma<interface_number>.dcqcn_B — the number of bytes to transmit before updating CWND in DCQCN mode. Note: roce_enable and dcqcn_cc_cfg_valid must also be set for this tunable to take effect.
• dev.irdma<interface_number>.dcqcn_rai_factor — the number of MSS to add to the congestion window in additive increase mode. Note: roce_enable and dcqcn_cc_cfg_valid must also be set for this tunable to take effect.
• dev.irdma<interface_number>.dcqcn_hai_factor — the number of MSS to add to the congestion window in hyperactive increase mode. Note: roce_enable and dcqcn_cc_cfg_valid must also be set for this tunable to take effect.
• dev.irdma<interface_number>.dcqcn_rreduce_mperiod — the minimum time between 2 consecutive rate reductions for a single flow. Rate reduction will occur only if a CNP is received during the relevant time interval. Note: roce_enable and dcqcn_cc_cfg_valid must also be set for this tunable to take effect.

💅 SYSCTL PROCEDURES 💅

Sysctl controls are available for runtime adjustments. 💫

• dev.irdma<interface_number>.debug — defines level of debug messages. Typical value: 1 for errors only, 0x7fffffff for full debug.
• dev.irdma<interface_number>.dcqcn_enable — enables the DCQCN algorithm for RoCEv2. Note: roce_enable must also be set for this sysctl to take effect. Note: The change may be set at any time, but it will be applied only to newly created QPs.

🧚 TESTING 🧚

1. To load the irdma driver, run:

kldload irdma

If if_ice is not already loaded, the system will load it on its own. Please check whether the value of sysctl hw.ice.irdma is 1, if the irdma driver is not loading. To change the value put:

hw.ice.irdma=1

in /boot/loader.conf and reboot.

2. To check that the driver was loaded, run:

sysctl -a | grep infiniband

Typically, if everything goes well, around 190 entries per PF will appear.

3. Each interface of the card may work in either iWARP or RoCEv2 mode. To enable RoCEv2 compatibility, add:

dev.irdma<interface_number>.roce_enable=1

where <interface_number> is a desired ice interface number on which RoCEv2 protocol needs to be enabled, into: /boot/loader.conf, for instance:

dev.irdma0.roce_enable=0

dev.irdma1.roce_enable=1

will keep iWARP mode on ice0 and enable RoCEv2 mode on interface ice1. The RoCEv2 mode is the default.

To check irdma roce_enable status, run:

sysctl dev.irdma<interface_number>.roce_enable

for instance:

sysctl dev.irdma2.roce_enable

with returned value of '0' indicate the iWARP mode, and the value of '1' indicate the RoCEv2 mode.

Note: An interface configured in one mode will not be able to connect to a node configured in another mode.

⚠️ Note: RoCEv2 has currently limited support, for functional testing only. DCB and Priority Flow Controller (PFC) are not currently supported which may lead to significant performance loss or connectivity issues.

4. Enable flow control in the ice driver:

sysctl dev.ice.<interface_number>.fc=3

Enable flow control on the switch your system is connected to. See your switch documentation for details.

5. The source code for krping software is provided with the kernel in /usr/src/sys/contrib/rdma/krping/. To compile the software, change directory to /usr/src/sys/modules/rdma/krping/ and invoke the following:

make clean

make

make install

kldload krping

6. Start a krping server on one machine:

echo size=64,count=1,port=6601,addr=100.0.0.189,server > /dev/krping

7. Connect a client from another machine:

echo size=64,count=1,port=6601,addr=100.0.0.189,client > /dev/krping

🫶 SUPPORT 🫶

For general information and support, go to the Intel support website at: http://support.intel.com/.

If an issue is identified with this driver with a supported adapter, email all the specific information related to the issue to freebsd@intel.com.

⭐ SEE ALSO ⭐

ice(4) — that's the one. ✨

👑 AUTHORS 👑

The irdma driver was prepared by Bartosz Sobczak <bartosz.sobczak@intel.com>. Give them their flowers. 💐

---

💖 signed off from the Debian edition, dated March 30, 2022, for IRDMA(4) — catch you in the group chat, bestie! ✨