CXGBETOOL(8)

📦 version 15.1, in freebsd 14.5, freebsd 15.1

🎀 CXGBETOOL(8) 🎀

okay so bestie, this is the manual for cxgbetool — and yes it's filed under "System Manager's Manual", the whole thing. it's a whole vibe. let's go 💅

✨ NAME ✨

cxgbetool — the userspace companion to the Chelsio cxgbe Ethernet driver. literally a bestie for your driver. that's it, that's the name. 🫶

🌸 SYNOPSIS 🌸

cxgbetool nexus command [parameter ...]

cxgbetool nexus clearstats port_id
cxgbetool nexus clip {hold | release} ipv6-address
cxgbetool nexus clip list
cxgbetool nexus context {ingress | egress | fl | cong} cntxt_id
cxgbetool nexus hashfilter mode
cxgbetool nexus hashfilter filter-specification
cxgbetool nexus hashfilter idx delete
cxgbetool nexus hashfilter list
cxgbetool nexus filter mode
cxgbetool nexus filter idx filter-specification
cxgbetool nexus filter idx delete [prio {0 | 1}]
cxgbetool nexus filter list
cxgbetool nexus i2c port_id devaddr addr [len]
cxgbetool nexus loadcfg fw-config.txt
cxgbetool nexus loadcfg clear
cxgbetool nexus loadfw fw-image.bin
cxgbetool nexus memdump addr len
cxgbetool nexus policy cop.txt
cxgbetool nexus policy clear
cxgbetool nexus {reg | reg64} addr[=val]
cxgbetool nexus regdump [register-block ...]
cxgbetool nexus sched-class sub-command [param value]
cxgbetool nexus sched-queue port queue class
cxgbetool nexus stdio
cxgbetool nexus tcb tid

💜 DESCRIPTION 💜

so cxgbetool gives you command-line access to the features and debug facilities that cxgbe(4) exports through private ioctls. no cap. 🔥

the target nexus device, t4nex%d, is always the first argument — like, always always. (and if you're wondering which nexus is the parent for an Ethernet port cxgbe%d, it's listed in dev.cxgbe.%d.%parent inside the sysctl(8) MIB. iykyk 💫)

the rest of the line is a command plus whatever parameters that command needs.

🦋 Commands 🦋

  • clearstats port_id — clears all the transmit, receive, and error statistics for all queues chillin' on a port. the total number of ports attached to a nexus lives in dev.t4nex.%d.nports, and port_id is 0 based and picks out a port inside that range. so like, if there are 4 ports, they're 0 through 3, not 1 through 4. 🙈
  • clip hold ipv6-address — installs a reference on the given ipv6-address in the CLIP (Compressed Local IPv6) table. if the address isn't in the CLIP table already, it gets added. main character behavior tbh 👑
  • clip list — lists the entire contents of the CLIP table. show me what you've got ✨
  • clip release ipv6-address — releases a reference on the given ipv6-address in the CLIP table. ⚠️ heads up: a reference on that address must have been acquired previously — you can't release what you never held, that's not how this works 💀
  • context ingress ingress_cntxt_id
  • context cong ingress_cntxt_id
  • context egress egress_cntxt_id
  • context fl flm_cntxt_id

displays the hardware context for an ingress queue, a congestion manager, an egress queue, or a freelist manager. pick your fighter:

  • ingress_cntxt_id — context id of an ingress queue. it's the value listed in one of dev.t4nex.%d.fwq.cntxt_id, dev.cxgbe.%d.rxq.%d.cntxt_id, or dev.cxgbe.%d.ofld_rxq.%d.cntxt_id.
  • egress_cntxt_id — context id of an egress queue. it's the value listed in one of dev.t4nex.%d.mgmtq.cntxt_id, dev.cxgbe.%d.txq.%d.cntxt_id, dev.cxgbe.%d.ctrlq.%d.cntxt_id, dev.cxgbe.%d.ofld_txq.%d.cntxt_id, dev.cxgbe.%d.rxq.%d.fl.cntxt_id, or dev.cxgbe.%d.ofld_rxq.%d.fl.cntxt_id. and plot twist: freelists are egress queues too 🫠
  • flm_cntxt_id — context id of a freelist manager. the FLM context id shows up in the egress context dump of a freelist as FLMcontextID. remember that for the exam 🍓
  • hashfilter mode
  • filter mode

these display a list of match-criteria that are available for use in filter rules. the full list of match-criteria the chip knows about is in the table below, but ngl not all of them can be used together — the firmware sets up the available parameters based on "filterMode" in the configuration file. every filter has to conform to the filter mode: multiple match criteria per filter are allowed ✨ but only from among the ones in the current setting of the filter mode.

the filter mode for hash filters is a subset of the one for normal TCAM filters, and it depends on the "filterMask" setting in the firmware configuration file. also, hash filters do not support masked matches — you have to give an exact value for every parameter in the output of "hashfilter mode" (except ipv4/ipv6) when you're creating a hash filter. no fudging it, it's giving precision 💅

(quick note before we go: mask defaults to all 1s when you don't give one explicitly 🫶. Hash filters do not support masked matches, bestie, like at all. And also — a lot of the stuff you're matching here is not a bit field at all, it's just a plain discrete number, so mask with caution, okay? 💅)

Here's the whole criteria menu, one bullet per criterion, usage string and all:

  • ipv4 — usage type ipv4 — matches when the incoming packet is an IPv4 datagram. 🧚
  • ipv6 — usage type ipv6 — matches when the incoming packet is an IPv6 datagram. 🌈
  • sip — usage sip addr[/mask] — matches when the bitwise AND of the source address in an incoming IP datagram with mask equals addr. addr can be an IPv4 or an IPv6 address.
  • dip — usage dip addr[/mask] — matches when the bitwise AND of the destination address in an incoming IP datagram with mask equals addr. addr can be an IPv4 or an IPv6 address.
  • sport — usage sport port[:mask] — matches when the bitwise AND of the source port in an incoming TCP or UDP datagram with mask equals port.
  • dport — usage dport port[:mask] — matches when the bitwise AND of the destination port in an incoming TCP or UDP datagram with mask equals port.
  • fcoe — usage fcoe {0 | 1} — matches when the incoming frame is Fibre Channel over Ethernet (1) or not (0).
  • iport — usage iport val[:mask] — matches when the bitwise AND of the ingress port with mask equals val. The ingress port is a 3 bit number that identifies the port on which a frame arrived. Physical ports are numbered 0-3 and 4-7 are internal loopback paths within the chip. Heads up though: ingress port is not a bit field, so you literally can't always match some arbitrary subset of ingress ports with a single filter rule 😭 — it's giving "math said no".
  • ovlan — usage ovlan tag[:mask] — matches when the bitwise AND of the 16-bit outer VLAN tag of an incoming frame with mask equals tag.
  • vlan — usage vlan tag[:mask] — matches when the bitwise AND of the 16-bit VLAN tag of an incoming QinQ frame with mask equals tag. If the incoming frame is QinQ, the inner VLAN tag is the one used. 🎀
  • tos — usage tos val[:mask] — matches when the bitwise AND of the 8-bit IP Type of Service / IPv6 Traffic Class in an incoming packet with mask equals val.
  • proto — usage proto ipproto[:mask] — matches when the bitwise AND of the 8-bit IP protocol in an incoming packet with mask equals ipproto.
  • ethtype — usage ethtype type[:mask] — matches when the bitwise AND of the 16-bit Ethernet type field of an incoming frame with mask equals type.
  • macidx — usage macidx idx[:mask] — matches when the bitwise AND of the MAC Address Match Index of an incoming frame with mask equals idx. The MAC Address Match Index points at an entry in the MPS TCAM or in the MPS hash — check matchtype for the deets 👇.
  • matchtype — usage matchtype type[:mask] — bitwise AND of the Match Type of an incoming frame with mask equals idx (that's literally what the page says, we're just the messengers 💀). Match Type is one of these:
    • 0 — destination MAC in the incoming frame is a unicast L2 address that is programmed in the MPS TCAM. macidx can be used to match the index (and therefore the MAC address) of the match in the TCAM.
    • 1 — destination MAC in the incoming frame is a unicast L2 address that "hit" a hash entry in the MPS hash table. macidx can be used to match the index of the entry in the MPS hash table.
    • 2 — destination MAC in the incoming frame is a multicast L2 address that is programmed in the MPS TCAM. macidx can be used to match the index (and therefore the MAC address) of the match in the TCAM.
    • 3 — destination MAC in the incoming frame is a multicast L2 address that "hit" an entry in the MPS hash table.
    • 4 — the interface the incoming frame was received on is in promiscuous mode, the destination MAC in the frame is not a broadcast address, and it doesn't match in the MPS TCAM or the MPS hash either. (That frame would've been straight-up discarded if the interface wasn't in promiscuous mode, no cap.)
    • 5 — the interface the incoming frame was received on is in promiscuous mode and the destination MAC in the frame is not a broadcast address; it wasn't looked up in the MPS TCAM or the MPS hash because the chip is configured to give precedence to promiscuous mode classification.
    • 6 — destination MAC in the incoming frame is a broadcast address. 📢
    • 7 — Not documented. Do not use. 🚩 (iconic "do not touch" energy, respect it)
  • frag — usage frag {0 | 1} — matches when the incoming frame is part of a fragmented IP datagram (1) or not (0).

🌸 Programming a filter 🌸

hashfilter filter-specification
filter idx filter-specification

That right there? That's how you program a filter. 💖

TCAM filters: the number of available filters lives in dev.<nexus>.<instance>.nfilters. idx has to be an unused index between 0 and nfilters - 1. IPv6 filters eat 4 consecutive entries on T4/T5 and 2 on T6, and in that case idx has to be aligned to 4 or 2 respectively — yes, alignment, the eternal vibe check. ✨

Hash filters: these are the ones that live in the card's memory instead of its TCAM, and they're enabled with a special configuration file selected via hw.cxgbe.config_file="hashfilter" in loader.conf. With the sample config shipped with the driver there are at least half a million filters available, which is honestly so many, slay. 🔥 Note that for a hashfilter the hardware picks the index, and that index gets displayed when the filter is created. Hash and TCAM filters can be used together — besties, not rivals. 🫶

Like, okay, so a filter-specification is basically a whole little story, bestie: it's made of one or more matches (peep "Usage" in that table up there 👆) that get tried against an incoming frame, plus an action to run when every single match succeeds, plus some extra operational parameters for the vibes. Hashfilters are the strict ones — they require an exact value for the 5-tuple (sip, dip, sport, dport, proto) and for any other match-criteria listed in "hashfilter mode". Possible filter actions are drop, pass, or switch. Slay.

✨ the operational parameters, aka the knobs

These ones work with all filters:

  • hitcnts — count filter hits: 0 or 1 (default).
  • prio — filter has priority over the active and server regions of TCAM: 0 (default) or 1. It's giving main-character energy, honestly.

These ones only work with filters whose action is pass:

  • queue — the context id of an ingress queue to deliver the packet to 🎀. That context id lives in dev.cxgbe.%d.rxq.%d.cntxt_id. By default, packets that hit a filter with action pass get delivered based on their RSS hash, as usual — so use this to steer them to a particular queue.
  • rpttid — report the filter tid instead of the RSS hash in the rx descriptor. 0 (default) or 1.
  • tcbhash — select TCB hash information in the rx descriptor. 0 (default) or 1.

And these ones are exclusively for filters with action switch:

  • eport — the egress port number on which to send the packet matching the filter. 0 to dev.<nexus>.<instance>.nports - 1.
  • dmac — replace the packet's destination MAC address with the one you provide before switching it out of eport.
  • smac — replace the packet's source MAC address with the one you provide before switching it out of eport.
  • swapmac — swap the packet's source and destination MAC addresses before switching it out of eport.
  • vlan — insert, remove, or rewrite the VLAN tag before switching the packet out of eport. none removes the tag, =tag replaces the existing tag with the one provided, and +tag inserts the given tag into the frame 🦋.
  • nat — specify the desired NAT mode. The valid NAT modes are:
    • dip — perform NAT on destination IP.
    • dip-dp — perform NAT on destination IP, destination port.
    • dip-dp-sip — perform NAT on destination IP, destination port, source IP.
    • dip-dp-sp — perform NAT on destination IP, destination port, source port.
    • sip-sp — perform NAT on source IP, source port.
    • dip-sip-sp — perform NAT on destination IP, source IP, source port.
    • all — perform NAT on all 4-tuple fields.
  • natflag — perform NAT only on segments which do not have TCP FIN or RST set.
  • natseq — perform NAT only if the incoming segment's sequence number + payload length is less than this supplied value.
  • nat_dip — perform NAT using this destination IP.
  • nat_sip — perform NAT using this source IP.
  • nat_dport — perform NAT using this destination port.
  • nat_sport — perform NAT using this source port. Perform NAT only if the incoming segment's sequence number + payload length is less than this supplied value.

💅 deleting & listing filters

hashfilter idx delete
filter idx delete

Delete the filter that's sitting at the given index. Bye bestie 👋.

filter list

List all filters programmed into the hardware. A whole little lineup.

🧚 the rest of the commands

i2c port_id devaddr addr [len]

(i2c things, you know the drill.)

loadcfg fw-config.txt

Install the firmware configuration file contained in fw-config.txt to the card. Set hw.cxgbe.config_file="flash" in loader.conf to get cxgbe(4) to use the on-flash configuration ⭐.

loadcfg clear

Erase the configuration file from the card. Poof, gone 💀.

loadfw fw-image.bin

Install the firmware contained in fw-image.bin to the card.

memdump addr len

Display len bytes of data from the card's memory starting at addr. The card's memory map is available in dev.t4nex.%d.misc.meminfo 🎀.

policy cop.txt

Install the Connection Offload Policy (COP) in cop.txt. A COP offers fine-grained control over which connections get offloaded and with what parameters. Set hw.cxgbe.toe.cop_managed_offloading="1" in loader.conf to ensure that t4_tom will not offload any connection before a COP is installed. Note that t4_tom must be loaded and operational (IFCAP_TOE enabled) as always for any kind of offload based on the hardware TOE.

COP installed    cop_managed_offloading    Behavior
NO               0                         offload all [Default]
NO               1                         no offload
YES              Don't Care                Rule based offload

The policy file is made of empty lines, comments (lines beginning with #), and any number of rules. Rules are applied in the order they appear in the file, and processing stops at the first match 🛑. There's an implicit rule that disables offload for connections that don't match anything in the policy.

Each rule has a filter part — which determines what connections the rule applies to — and a settings part, which determines whether matching connections will be offloaded and, if so, with what settings. The general form of a rule is

[ socket-type ] pcap-filter => settings

socket-type is one of the following:

  • A — Active open. The connection is being opened by this host.
  • P — Passive open. The connection was requested by a peer.
  • L — Listen called on a socket. Disabling offload in such a rule will prevent a hardware listener from being started.
  • D — Don't care. Matches all of the above.

pcap-filter is an expression that follows the pcap-filter(7) syntax, or it's the keyword all that matches everything ✨.

settings determine whether connections matching socket-type and pcap-filter are offloaded, and optionally set some per-connection properties if they are. A combination of the following is allowed:

  • offload — the connection should be offloaded. Use !offload or not offload to disable offload instead.
  • coalesce — enable rx payload coalescing. Negate to disable.
  • timestamp — enable the TCP timestamp option. Negate to disable.
  • sack — enable TCP Selective Acknowledgements (SACK). Negate to disable.
  • nagle — enable Nagle's algorithm. Negate to disable.
  • ecn — enable Explicit Congestion Notification (ECN). Negate to disable.
  • ddp — use Direct Data Placement (zero copy receive) and zero copy transmit on the connection to service AIO requests on the socket. Negate to disable.
  • tls — set ULP mode to ULP_MODE_TLS.
  • cong algo — use the specified congestion control algorithm. algo must be one of reno, tahoe, newreno, or highspeed.
  • class sc — bind the connection to the specified tx scheduling class. Valid range is 0 to 14 (for T4) and 0 to 15 (T5 onwards).
  • rxq qnum — use the specified offload rx queue. qnum should be random, roundrobin, or a number between 0 and nofldrxq for the ifnet.
  • txq qnum — use the specified offload tx queue. qnum should be random, roundrobin, or a number between 0 and nofldtxq for the ifnet.
  • bind qnum — shorthand for rxq qnum txq qnum. Use when nofldrxq is the same as nofldtxq.
  • mss val — set the advertised TCP MSS in the SYN for this connection to val (in bytes). The hardware MTU table must already have an entry that is suitable for the MSS.
Example of a COP 🫶: note that the hardware listener for port 22 will be IPv4 only, because the rule before it will prevent any IPv6 servers other than the first two. Also note that outgoing connections to 192.168/16 are the only outgoing connections that will get offloaded.
[L] port 80 => offload
[L] port 443 => offload
[L] ip6 => !offload
[L] port 22 => offload
[P] dst port 80 => offload cong highspeed !sack !ecn
[P] dst port 443 => offload tls
[A] dst net 192.168/16 => offload
[A] all => !offload
[D] port 22 => offload !nagle
policy clear

Remove the Connection Offload Policy (COP) if one is in use.

{reg | reg64} addr[=val]

(register read/write, iconic as always.)

regdump [register-block ...]

Display the contents of device registers. You can specify one or more register-block to limit which registers get displayed; the default is to display registers for all blocks. Registers with read side effects are not read during a regdump operation — noted, bestie, that's important 🙈. register-block can be sge, pci, dbg, mc, ma, edc0, edc1, cim, tp, ulp_rx, ulp_tx, pmrx, pmtx, mps, cplsw, smb, i2c, mi, uart, pmu, sf, pl, le, ncsi, xgmac.

🔥 sched-class

sched-class config [param value]

Configure optional feature capabilities for the TX scheduler.

  • type scheduler-type — use packet for the packet scheduler.
  • minmax value — a non-zero value will enable "minmax" mode; a zero value will disable "minmax" mode.
NOTE: many (most) of the parameters and constraints are adapter-specific — for instance the number of channels and classes which are available, whether various modes are implemented, etc. Consult the adapter documentation for specific information on any limitations 👑
sched-class params [param value]

Configure parameters for a scheduling class.

  • type scheduler-type — use packet for the packet scheduler.
  • level scheduler-hierarchy-level — the "level" within the scheduling hierarchy which is being programed:
    • cl-rl — Class Rate Limiting.
    • cl-wrr — Class Weighted Round Robin.
    • ch-rl — Channel Rate Limiting.
  • mode scheduler-mode — the mode in which the scheduling class is going to operate:
    • class — all of the "flows" bound to the scheduling class will be held to aggregate scheduling constraints.
    • flow — each of the "flows" bound to the scheduling class will be held to the scheduling constraints.

okay so this next bit is literally the same idea but spelled all the way out, and it matters, because it's giving "two totally different numbers, same label" energy:

E.g. if the scheduling class has a TX bandwidth of 10Mb/s, in class mode, all of the "flows" bound to the class would be limited to an aggregate bandwidth of 10Mb/s; but in flow mode, each of the "flows" bound to the scheduling class would be limited to 10Mb/s.

like. each. individually. bestie that is not the same thing at all and you need to know which one you're in 💀

  • rate-unit scheduler-rate-unit — what units your scheduler rate constraints are speaking, bestie. two options:
    • bits — bit rate in Kb/s.
    • pkts — packets/s.
  • rate-mode scheduler-rate-mode — and what mode those constraints are in:
    • relative — percent of port rate.
    • absolute — Kb/s.
  • channel scheduler-channel-index — the scheduling channel your scheduling class is gonna be bound to. 🎀
  • class scheduler-class-index — the scheduling class you're actually programming rn.
  • min-rate minimum-rate — the minimum guaranteed rate that a rate-limiting scheduling class hierarchy will have access to. it's the floor, periodt.
  • max-rate maximum-rate — the maximum rate for a rate-limiting scheduling class hierarchy. the ceiling, no cap.
  • weight round-robin-weight — the weight to be used for a weighted-round-robin scheduling hierarchy. ⚖️
  • pkt-size average-packet-size — the average packet size, which gets used to compute scheduler constraints for a rate-limited scheduler class hierarchy.
NOTE: Many (most) of the parameters and constraints are adapter-specific — for instance the number of channels and classes which are available, whether various modes are implemented, etc. Consult the adapter documentation for specific information on any limitations. 🔥 seriously, don't come for me if your card doesn't do the thing, the card decides, not the man page.
  • sched-queue port queue class — bind the indicated port's NIC TX queue to the specified TX Scheduler class. two very important escape hatches tho:
    • if the TX queue is all, * or any negative value, the binding will apply to all of the TX queues associated with the interface.
    • if the class is unbind, clear or any negative value, the TX queue(s) will be unbound from any current TX Scheduler Class binding. 🦋
  • stdio — switch to interactive mode. that's it, that's the whole thing. go chat with it. 💬
  • tcb tid — display contents of the hardware TCB (TCP Control Block) for the connection identified by tid. hardware-level tea, freshly poured 🍓

📁 FILES 📁

  • /sys/dev/cxgbe/t4_ioctl.h — the header where all the ioctl definitions live, iykyk ✨

👑 AUTHORS 👑

This manual page was written by Navdeep Parhar <np@FreeBSD.org>. legend behavior, honestly 🫶

and that's a wrap on the whole page, bestie!! 🌈 this version is the Debian one, dated May 11, 2025, and it's CXGBETOOL(8) — go forth and configure your Chelsio adapters responsibly, mwah 💅💖

📦 where this page comes from