CXGBE(4)

💖 CXGBE(4) 💖

okay bestie, welcome to cxgbe(4), straight out of the Kernel Interfaces Manual — this is the whole entire lore on the Chelsio T4-, T5-, and T6-based 100Gb, 40Gb, 25Gb, 10Gb, and 1Gb Ethernet adapter driver. grab a snack, we're going in 🍓

---

✨ NAME ✨

cxgbe — the Chelsio T4-, T5-, and T6-based 100Gb, 40Gb, 25Gb, 10Gb, and 1Gb Ethernet adapter driver. that's it, that's the name, she's iconic 💅

---

🌸 SYNOPSIS 🌸

so you wanna compile this driver into the kernel? okay queen, drop these lines into your kernel configuration file:

device cxgbe

orrr if you'd rather load the driver as a module at boot time (honestly, valid), put these lines in loader.conf(5):

t4fw_cfg_load="YES"
t5fw_cfg_load="YES"
t6fw_cfg_load="YES"
if_cxgbe_load="YES"

no cap, that's the whole setup 🫶

---

🦋 DESCRIPTION 🦋

okay so the cxgbe driver provides support for PCI Express Ethernet adapters based on the Chelsio Terminator 4, Terminator 5, and Terminator 6 ASICs (aka T4, T5, and T6, we're on nickname basis here). and the feature list?? she ate, fr fr:

• Jumbo Frames
• Transmit/Receive checksum offload
• TCP segmentation offload (TSO)
• Large Receive Offload (LRO)
• VLAN tag insertion/extraction
• VLAN checksum offload
• VLAN TSO
• Receive Side Steering (RSS)

want further hardware info, or got questions about hardware requirements? go look at http://www.chelsio.com/ ✨

---

💫 the naming situation (iykyk)

the cxgbe driver uses different names for devices depending on which ASIC is involved, which is a lot but we roll with it:

• T4 — Port Name: cxgbe, Parent Device: t4nex, Virtual Interface: vcxgbe
• T5 — Port Name: cxl, Parent Device: t5nex, Virtual Interface: vcxl
• T6 — Port Name: cc, Parent Device: t6nex, Virtual Interface: vcc

Loader tunables with the hw.cxgbe prefix apply to all cards, periodt. the driver provides sysctl MIBs for both ports and parent devices using the names above. for example, a T5 adapter provides port MIBs under dev.cxl and adapter-wide MIBs under dev.t5nex. and for the rest of this page, whenever we mention sysctl MIBs:

• dev.<port> = port MIBs
• dev.<nexus> = adapter-wide MIBs

want more on configuring this device? see ifconfig(8) 💜

---

👑 HARDWARE 👑

🔥 T6 ASIC — 100Gb and 25Gb Ethernet adapters

the cxgbe driver supports these, bestie:

• Chelsio T6225-CR
• Chelsio T6225-SO-CR
• Chelsio T62100-LP-CR
• Chelsio T62100-SO-CR
• Chelsio T62100-CR

---

⭐ T5 ASIC — 40Gb, 10Gb and 1Gb Ethernet adapters

the cxgbe driver supports these:

• Chelsio T580-CR
• Chelsio T580-LP-CR
• Chelsio T580-LP-SO-CR
• Chelsio T560-CR
• Chelsio T540-CR
• Chelsio T540-LP-CR
• Chelsio T522-CR
• Chelsio T520-LL-CR
• Chelsio T520-CR
• Chelsio T520-SO
• Chelsio T520-BT
• Chelsio T504-BT

---

🌈 T4 ASIC — 10Gb and 1Gb Ethernet adapters

and the cxgbe driver supports these too:

• Chelsio T420-CR
• Chelsio T422-CR
• Chelsio T440-CR
• Chelsio T420-BCH
• Chelsio T440-BCH
• Chelsio T440-CH
• Chelsio T420-SO
• Chelsio T420-CX
• Chelsio T420-BT
• Chelsio T404-BT

---

okay that's part 1 done, don't go anywhere, there's literally more 🎀

🎛️ LOADER TUNABLES 🎛️

okay bestie, this is where you get to boss the driver around before the kernel even wakes up 💅. You can set these at the loader(8) prompt before booting the kernel, or you can stash them in loader.conf(5). There are a bunch of tunables controlling how many queues of various types get made. And here's the cute trick: if you give one of these a negative value, it tells the driver to create up to that many queues, as long as there are enough CPU cores lying around. slay.

---

hw.cxgbe.ntxq

How many NIC tx queues a port gets. Default is 16 or the number of CPU cores in the system — whichever is less.

hw.cxgbe.nrxq

How many NIC rx queues a port gets. Default is 8 or the number of CPU cores in the system, whichever is less.

hw.cxgbe.nofldtxq

How many TOE tx queues a port gets. Default is 8 or the number of CPU cores, whichever is less.

hw.cxgbe.nofldrxq

How many TOE rx queues a port gets. Default is 2 or the number of CPU cores, whichever is less.

hw.cxgbe.num_vis

Number of virtual interfaces (VIs) created for each port. Every virtual interface spawns its own separate network interface, which is very main character of it ✨. The first virtual interface on each port is required — it's the primary network interface on the port. Any additional virtual interfaces on a port get named using the Virtual Interface name from the table above. Those extra virtual interfaces use a single pair of queues for rx and tx, plus an additional pair of queues for TOE rx and tx. The default is 1.

hw.cxgbe.holdoff_timer_idx

hw.cxgbe.holdoff_timer_idx_ofld

Timer index value used to delay interrupts. The holdoff timer list defaults to the values 1, 5, 10, 50, 100, and 200 (all in microseconds, iykyk) and the index picks one out of that list. holdoff_timer_idx_ofld applies to queues used for TOE rx. The default value is 1, which means the timer value is 5us. Different interfaces can be assigned different values at any time through the dev.<port>.X.holdoff_tmr_idx and dev.<port>.X.holdoff_tmr_idx_ofld sysctls.

hw.cxgbe.holdoff_pktc_idx

hw.cxgbe.holdoff_pktc_idx_ofld

Packet-count index value used to delay interrupts. The packet-count list defaults to 1, 8, 16, and 32, and the index selects a value from that list. holdoff_pktc_idx_ofld applies to queues used for TOE rx. The default value is -1, which means packet counting is disabled and interrupts are generated based solely on the holdoff timer value. Different interfaces can be assigned different values via the dev.<port>.X.holdoff_pktc_idx and dev.<port>.X.holdoff_pktc_idx_ofld sysctls.

Heads up bestie 🫶 — those sysctls work only when the interface has never been marked up (as done by ifconfig up). Once it's up, that door is closed, fr fr.

hw.cxgbe.qsize_txq

Number of entries in a transmit queue's descriptor ring. A buf_ring of the same size also gets allocated for additional software queuing — see ifnet(9) if you want the lore. The default value is 1024. Different interfaces can be assigned different values via the dev.<port>.X.qsize_txq sysctl.

Again: this sysctl works only when the interface has never been marked up (as done by ifconfig up).

hw.cxgbe.qsize_rxq

Number of entries in a receive queue's descriptor ring. The default value is 1024. Different interfaces can be assigned different values via the dev.<port>.X.qsize_rxq sysctl.

And yes: this sysctl works only when the interface has never been marked up (as done by ifconfig up). the way that these three all share the same rule 💀

hw.cxgbe.interrupt_types

Permitted interrupt types. Bit 0 represents INTx (line interrupts), bit 1 is MSI, and bit 2 is MSI-X. The default is 7 (all of them allowed 🎀). The driver then selects the best possible type out of the allowed ones.

hw.cxgbe.pcie_relaxed_ordering

PCIe Relaxed Ordering. -1 means the driver should figure out whether to enable or disable PCIe RO. 0 disables PCIe RO. 1 enables PCIe RO. 2 means the driver should not modify the PCIe RO setting at all (hands off 🙈). The default is -1.

hw.cxgbe.fw_install

0 prohibits the driver from installing a firmware on the card. 1 allows the driver to install a new firmware if internal driver heuristics say the new firmware is preferable to the one already on the card. 2 tells the driver to always install the new firmware on the card, as long as it's compatible with the driver and is a different version than the one already there. The default is 1.

hw.cxgbe.fl_pktshift

Number of padding bytes inserted before the beginning of an Ethernet frame in the receive buffer. The default value is 0. A value of 2 would ensure the Ethernet payload (usually the IP header) sits at a 4 byte aligned address. 0–7 are all valid values. cute little range, no notes ✨

hw.cxgbe.fl_pad

A non-zero value ensures that writes from the hardware to a receive buffer are padded up to the specified boundary. The default is -1, which lets the driver pick a pad boundary. 0 disables trailer padding completely.

hw.cxgbe.cong_drop

Controls the hardware response to congestion. -1 disables congestion feedback and is not recommended. 0 tells the hardware to backpressure its pipeline on congestion — which usually results in the port emitting PAUSE frames. 1 tells the hardware to drop frames destined for congested queues.

hw.cxgbe.pause_settings

PAUSE frame settings. Bit 0 is rx_pause, bit 1 is tx_pause, bit 2 is pause_autoneg. So: rx_pause = 1 tells the hardware to heed incoming PAUSE frames, 0 tells it to ignore them. tx_pause = 1 lets the hardware emit PAUSE frames when its receive FIFO hits a high threshold, 0 stops it from emitting PAUSE frames. pause_autoneg = 1 overrides the rx_pause and tx_pause bits and tells the hardware to negotiate PAUSE settings with the link peer. The default is 7 (all three = 1). This tunable sets the default PAUSE settings for all ports. Settings can be displayed and controlled per-port via the dev.<port>.X.pause_settings sysctl.

hw.cxgbe.fec

Forward Error Correction settings. -1 (the default) means the driver should automatically pick a value. 0 disables FEC. Want finer grained control? Set individual bits: bit 0 enables RS FEC, bit 1 enables BASE-R FEC (aka Firecode FEC), bit 2 enables NO FEC, and bit 6 enables the FEC that's recommended by the transceiver/cable that's plugged in 🔌. These bits can be set together in any combination. This tunable establishes the default FEC settings for all ports, and per-port settings can be displayed and controlled via the dev.<port>.X.fec sysctl.

hw.cxgbe.autoneg

Link autonegotiation settings. This tunable establishes the default autonegotiation settings for all ports. Settings can be displayed and controlled per-port via the dev.<port>.X.autoneg sysctl. 0 disables autonegotiation. 1 enables autonegotiation. The default is -1, which lets the driver pick a value. And psst — dev.<port>.X.autoneg is -1 for port and module combinations that don't support autonegotiation at all.

hw.cxgbe.buffer_packing

Lets the hardware opportunistically deliver multiple frames in the same receive buffer (efficiency icon 👑). The default is -1, which lets the driver decide. 0 or 1 explicitly disable or enable the feature.

hw.cxgbe.largest_rx_cluster

hw.cxgbe.safest_rx_cluster

Sizes of rx clusters. Each of these must be set to one of the available sizes (usually 2048, 4096, 9216, and 16384) and largest_rx_cluster must be greater than or equal to safest_rx_cluster. The defaults are 16384 and 4096 respectively. The driver never tries to allocate a receive buffer larger than largest_rx_cluster, and falls back to allocating buffers of safest_rx_cluster size if an allocation larger than safest_rx_cluster fails. Note that largest_rx_cluster is merely a ceiling — the driver is still allowed to allocate buffers of smaller sizes.

hw.cxgbe.config_file

Select a pre-packaged device configuration file. A configuration file contains a recipe for partitioning and configuring the hardware resources on the card 🍓. This tunable is for specialized applications only and should not be used in normal operation. The configuration profile currently in use is available in the dev.<nexus>.X.cf and dev.<nexus>.X.cfcsum sysctls.

hw.cxgbe.linkcaps_allowed

hw.cxgbe.niccaps_allowed

hw.cxgbe.toecaps_allowed

hw.cxgbe.rdmacaps_allowed

hw.cxgbe.iscsicaps_allowed

hw.cxgbe.fcoecaps_allowed

Disallowing capabilities gives the driver and firmware a hint to not reserve hardware resources for that feature. Each of these is a bit field with a bit for each sub-capability within the capability. Like the one above, this tunable is for specialized applications only and should not be used in normal operation. The capabilities for which hardware resources have been reserved are listed in the dev.<nexus>.X.*caps sysctls.

---

🫶 SUPPORT 🫶

For general info and support, go hang out at the Chelsio support website: http://www.chelsio.com/.

If you find an issue with this driver on a supported adapter, email all the specific information related to the issue to support@chelsio.com. Give them the deets, bestie — all of them 💌

---

👀 SEE ALSO 👀

altq(4), arp(4), ccr(4), cxgb(4), cxgbev(4), netintro(4), ng_ether(4), ifconfig(8)

the crossover episode lineup ✨

---

📜 HISTORY 📜

The cxgbe device driver first appeared in FreeBSD 9.0. Support for T5 cards first appeared in FreeBSD 9.2 and FreeBSD 10.0. Support for T6 cards first appeared in FreeBSD 11.1 and FreeBSD 12.0. a whole glow-up arc 🌈

---

💖 AUTHORS 💖

The cxgbe driver was written by Navdeep Parhar <np@FreeBSD.org>. king behavior, honestly 👑

---

and that's the whole page, bestie 🎀 this version came from the Debian packaging, dated Dec 10, 2019, and it's page CXGBE(4) — now go forth and tune responsibly, mwah 💅✨