infiniband-diags(8)

💖 infiniband-diags 💖

okay bestie, welcome to infiniband-diags(8) from Open IB Diagnostics — this is part 1 of 2 so buckle up, we're going in 🌸

---

✨ NAME ✨

infiniband-diags — Diagnostics for InfiniBand Fabrics 🦋

---

🌈 DESCRIPTION 🌈

so infiniband-diags is a whole set of utilities designed to help you configure, debug, and maintain infiniband fabrics. like, a lot of tools and utilities are provided here. some of them literally do basically the same thing as each other (we contain multitudes 💅).

the base utilities use directed route MAD's to do their thing. which means — and this is the slay part — they may work even in unconfigured subnets ✨. the other, higher level utilities, they need LID routed MAD's and to some extent SA/SM access. so yeah, tier list exists, no cap.

---

👑 THE USE OF SMPs (QP0) 👑

many of the tools in this package rely on the use of SMPs via QP0 to acquire data directly from the SMA. and like, while this mode of operation is not technically in compliance with the InfiniBand specification — we know, we know 🙈 — practical experience has found that this level of diagnostics is genuinely valuable when you're working with a fabric which is broken or only partially configured. for this reason many of these tools may require the use of an MKey, or operation from Virtual Machines may be restricted for security reasons. just so you know. 🫶

---

🎀 COMMON OPTIONS 🎀

most OpenIB diagnostics take some of the following common flags. the exact list of supported flags per utility can be found in the documentation for those commands — so don't come for me if your fav tool doesn't take all of them, iykyk 💅

💫 Addressing Flags 💫

the -D and -G option have two forms:

• -D, --Direct — the address specified is a directed route 🛣️
• -D, --Direct <dr_path> — the address specified is a directed route 🛣️ (yes, again, she's versatile)

Examples for the no-argument form:

[options] -D [options] "0"          # self port
[options] -D [options] "0,1,2,1,4"  # out via port 1, then 2, ...

(Note the second number in the path specified must match the port being
used.  This can be specified using the port selection flag '-P' or the
port found through the automatic selection process.)

Examples for the <dr_path> form:

-D "0"          # self port
-D "0,1,2,1,4"  # out via port 1, then 2, ...

(Note the second number in the path specified must match the port being
used.  This can be specified using the port selection flag '-P' or the
port found through the automatic selection process.)

• -G, --Guid — the address specified is a Port GUID 🪪
• --port-guid, -G <port_guid> — specify a port_guid 🪪
• -L, --Lid — the address specified is a LID 🔢
• -s, --sm_port <smlid> — use smlid as the target lid for SA queries. 📮

🌸 Port Selection flags 🌸

• -C, --Ca <ca_name> — use the specified ca_name. 🎯
• -P, --Port <ca_port> — use the specified ca_port. 🎯
• -Y, --pkey-idx <idx> — P_Key table index (0-65535, default 0) used for the outgoing SA/GMP MADs. SMP (SMI directed-route/LID-routed) MADs always use P_Key index 0. 🔑

🌼 Local port Selection 🌼

Multiple port/Multiple CA support: when no IB device or port is specified (see the "local umad parameters" below), the libibumad library selects the port to use by the following criteria:

1. the first port that is ACTIVE.
2. if not found, the first port that is UP (physical link up).

if a port and/or CA name is specified, the libibumad library attempts to fulfill the user request, and will fail if it is not possible. 😭

for example:

ibaddr                 # use the first port (criteria #1 above)
ibaddr -C mthca1       # pick the best port from "mthca1" only.
ibaddr -P 2            # use the second (active/up) port from the first available IB device.
ibaddr -C mthca0 -P 2  # use the specified port only.

🦋 Debugging flags 🦋

• -d — raise the IB debugging level. may be used several times (-ddd or -d -d -d). 🔍
• -e — show send and receive errors (timeouts and others) 💀
• -h, --help — show the usage message 🙋
• -v, --verbose — increase the application verbosity level. may be used several times (-vv or -v -v -v) 🗣️
• -V, --version — show the version info. 📌

⭐ Configuration flags ⭐

• -t, --timeout <timeout_ms> — override the default timeout for the solicited mads. ⏳
• --outstanding_smps, -o <val> — specify the number of outstanding SMP's which should be issued during the scan. default: 2 🔥
• --node-name-map <node-name-map> — specify a node name map. this file maps GUIDs to more user friendly names. see FILES section. 🗺️
• --config, -z <config_file> — specify alternate config file. default: /etc/infiniband-diags/ibdiag.conf ⚙️

---

🧚 COMMON FILES 🧚

the following config files are common amongst many of the utilities. 🍓

💜 CONFIG FILE 💜

/etc/infiniband-diags/ibdiag.conf

a global config file is provided to set some of the common options for all tools. see the supplied config file for details. 🌈

🎀 NODE NAME MAP FILE FORMAT 🎀

the node name map is used to specify user friendly names for nodes in the output. GUIDs are used to perform the lookup. 🔎

this functionality is provided by the opensm-libs package. see opensm(8) for the file location for your installation. 👑

generically:

# comment
<guid> "<name>"

example:

# IB1
# Line cards
0x0008f104003f125c "IB1 (Rack 11 slot 1   ) ISR9288/ISR9096 Voltaire sLB-24D"
0x0008f104003f125d "IB1 (Rack 11 slot 1   ) ISR9288/ISR9096 Voltaire sLB-24D"
0x0008f104003f10d2 "IB1 (Rack 11 slot 2   ) ISR9288/ISR9096 Voltaire sLB-24D"
0x0008f104003f10d3 "IB1 (Rack 11 slot 2   ) ISR9288/ISR9096 Voltaire sLB-24D"
0x0008f104003f10bf "IB1 (Rack 11 slot 12  ) ISR9288/ISR9096 Voltaire sLB-24D"

# Spines
0x0008f10400400e2d "IB1 (Rack 11 spine 1   ) ISR9288 Voltaire sFB-12D"
0x0008f10400400e2e "IB1 (Rack 11 spine 1   ) ISR9288 Voltaire sFB-12D"
0x0008f10400400e2f "IB1 (Rack 11 spine 1   ) ISR9288 Voltaire sFB-12D"
0x0008f10400400e31 "IB1 (Rack 11 spine 2   ) ISR9288 Voltaire sFB-12D"
0x0008f10400400e32 "IB1 (Rack 11 spine 2   ) ISR9288 Voltaire sFB-12D"

# GUID   Node Name
0x0008f10400411a08 "SW1  (Rack  3) ISR9024 Voltaire 9024D"
0x0008f10400411a28 "SW2  (Rack  3) ISR9024 Voltaire 9024D"
0x0008f10400411a34 "SW3  (Rack  3) ISR9024 Voltaire 9024D"
0x0008f104004119d0 "SW4  (Rack  3) ISR9024 Voltaire 9024D"

🌈 TOPOLOGY FILE FORMAT 🌈

the topology file format is human readable and largely intuitive. most identifiers are given textual names like vendor ID (vendid), device ID (device ID), GUIDs of various types (sysimgguid, caguid, switchguid, etc.). PortGUIDs are shown in parentheses (). for switches, this is shown on the switchguid line. for CA and router ports, it is shown on the connectivity lines. the IB node is identified followed by the number of ports and a quoted the node GUID. on the right of this line is a comment (#) followed by the NodeDescription in quotes. if the node is a switch, this line also contains whether switch port 0 is base or enhanced, and the LID and LMC of port 0. subsequent lines pertaining to this node show the connectivity. on the left is the port number of the current node. on the right is the peer node (node at other end of link). it is identified in quotes with nodetype followed by - followed by NodeGUID with the port number in square brackets. further on the right is a comment (#). what follows the comment is dependent on the node type. if it it a switch node, it is followed by the NodeDescription in quotes and the LID of the peer node. if it is a CA or router node, it is followed by the local LID and LMC and then followed by the NodeDescription in quotes and the LID of the peer node. the active link width and speed are then appended to the end of this output line. 💅

an example of this is:

#
# Topology file: generated on Tue Jun  5 14:15:10 2007
#
# Max of 3 hops discovered
# Initiated from node 0008f10403960558 port 0008f10403960559

Non-Chassis Nodes

vendid=0x8f1
devid=0x5a06
sysimgguid=0x5442ba00003000
switchguid=0x5442ba00003080(5442ba00003080)
Switch  24 "S-005442ba00003080"         # "ISR9024 Voltaire" base port 0 lid 6 lmc 0
[22]    "H-0008f10403961354"[1](8f10403961355)         # "MT23108 InfiniHost Mellanox Technologies" lid 4 4xSDR
[10]    "S-0008f10400410015"[1]         # "SW-6IB4 Voltaire" lid 3 4xSDR
[8]     "H-0008f10403960558"[2](8f1040396055a)         # "MT23108 InfiniHost Mellanox Technologies" lid 14 4xSDR
[6]     "S-0008f10400410015"[3]         # "SW-6IB4 Voltaire" lid 3 4xSDR
[12]    "H-0008f10403960558"[1](8f10403960559)         # "MT23108 InfiniHost Mellanox Technologies" lid 10 4xSDR

vendid=0x8f1
devid=0x5a05
switchguid=0x8f10400410015(8f10400410015)
Switch  8 "S-0008f10400410015"          # "SW-6IB4 Voltaire" base port 0 lid 3 lmc 0
[6]     "H-0008f10403960984"[1](8f10403960985)         # "MT23108 InfiniHost Mellanox Technologies" lid 16 4xSDR
[4]     "H-005442b100004900"[1](5442b100004901)        # "MT23108 InfiniHost Mellanox Technologies" lid 12 4xSDR
[1]     "S-005442ba00003080"[10]                # "ISR9024 Voltaire" lid 6 1xSDR
[3]     "S-005442ba00003080"[6]         # "ISR9024 Voltaire" lid 6 4xSDR

vendid=0x2c9
devid=0x5a44
caguid=0x8f10403960984
Ca      2 "H-0008f10403960984"          # "MT23108 InfiniHost Mellanox Technologies"
[1](8f10403960985)     "S-0008f10400410015"[6]         # lid 16 lmc 1 "SW-6IB4 Voltaire" lid 3 4xSDR

vendid=0x2c9
devid=0x5a44
caguid=0x5442b100004900
Ca      2 "H-005442b100004900"          # "MT23108 InfiniHost Mellanox Technologies"
[1](5442b100004901)     "S-0008f10400410015"[4]         # lid 12 lmc 1 "SW-6IB4 Voltaire" lid 3 4xSDR

vendid=0x2c9
devid=0x5a44
caguid=0x8f10403961354
Ca      2 "H-0008f10403961354"          # "MT23108 InfiniHost Mellanox Technologies"
[1](8f10403961355)     "S-005442ba00003080"[22]                # lid 4 lmc 1 "ISR9024 Voltaire" lid 6 4xSDR

vendid=0x2c9
devid=0x5a44
caguid=0x8f10403960558
Ca      2 "H-0008f10403960558"          # "MT23108 InfiniHost Mellanox Technologies"
[2](8f1040396055a)     "S-005442ba00003080"[8]         # lid 14 lmc 1 "ISR9024 Voltaire" lid 6 4xSDR
[1](8f10403960559)     "S-005442ba00003080"[12]                # lid 10 lmc 1 "ISR9024 Voltaire" lid 6 1xSDR

when grouping is used, IB nodes are organized into chassis which are numbered. nodes which cannot be determined to be in a chassis are displayed as "Non-Chassis Nodes". external ports are also shown on the connectivity lines. 🫶

---

🌸 bestie, that's the end of part 1 — but we are SO not done, more parts are coming, don't go anywhere 💖

🌈 Utilities list 🌈

okay bestie, here's the whole girlies lineup — the tools are grouped by what kind of detective work you're doing 💖

🦋 Basic fabric connectivity

like, is everybody talking to everybody? find out here.

• See: ibnetdiscover, iblinkinfo

🎀 Node information

who even is on your fabric 💅

• See: ibnodes, ibswitches, ibhosts, ibrouters

🌸 Port information

the deets on one specific port, no cap.

• See: ibportstate, ibaddr

🍓 Switch Forwarding Table info

the switch's personal map of where stuff goes 🗺️✨

• See: ibtracert, ibroute, dump_lfts, dump_mfts, check_lft_balance, ibfindnodesusing

⭐ Performance counters

the numbers, the stats, the receipts 📊

• See: ibqueryerrors, perfquery

💜 Local HCA info

your own local card's whole vibe check.

• See: ibstat, ibstatus

🧚 Connectivity check

poke a host, see if it pokes back 🫶

• See: ibping, ibsysstat

👑 Low level query tools

down in the weeds, raw query energy, iykyk.

• See: smpquery, smpdump, saquery, sminfo

🔥 Fabric verification tools

making sure your fabric isn't lying to you 🙈

• See: ibidsverify

---

💀 Backwards compatibility scripts 💀

ngl bestie, these scripts have been identified as redundant and/or lower performing compared to the scripts above. they're provided as legacy scripts — and literally only when --enable-compat-utils is specified at build time. no flag, no scripts, periodt.

ibcheckerrors, ibclearcounters, ibclearerrors, ibdatacounters ibchecknet, ibchecknode, ibcheckport, ibcheckportstate, ibcheckportwidth, ibcheckstate, ibcheckwidth, ibswportwatch, ibprintca, ibprintrt, ibprintswitch, set_nodedesc.sh

---

🫶 AUTHORS 🫶

• Ira Weiny — < ira.weiny@intel.com >

---

and that's the whole page, babe ✨ this one's from infiniband-diags(8), dated 2017-08-21 — go forth and interrogate your fabric 💅🦋🎀