IBDIAGNET(1)

💖 IBDIAGNET(1) 💖

okay bestie, welcome to the IB DIAGNOSTIC PACKAGE — this is the manual page for ibdiagnet, and yes the tool is literally called "IB diagnostic net" and it's giving main character energy for your whole InfiniBand fabric. sit down, grab a strawberry 🍓, and let's get into it.

---

✨ NAME ✨

ibdiagnet — IB diagnostic net

---

🦋 SYNOPSYS 🦋

ibdiagnet [-c <count>] [-v] [-r] [-vlr] [-u] [-o <out-dir>]
  [-t <topo-file>] [-s <sys-name>] [-i <dev-index>] [-p <port-num>]
[-wt]
  [-pm] [-pc] [-P <<PM>=<Value>>]
  [-lw <1x|4x|12x>] [-ls <2.5|5|10>]
  [-skip <ibdiag_check/s>] [-load_db <db_file>]

---

🌸 DESCRIPTION 🌸

so ibdiagnet scans the entire fabric using directed route packets and pulls out literally every scrap of info it can about the connectivity and the devices. then it drops a whole collection of files into the output directory (and that directory is the one you set with the -o option — more on that below). 💅

---

🎀 FILES 🎀

here's the lil squad of files it makes for you:

• ibdiagnet.log — a dump of all the application reports generate according to the provided flags
• ibdiagnet.lst — list of all the nodes, ports and links in the fabric
• ibdiagnet.fdbs — a dump of the unicast forwarding tables of the fabric switches
• ibdiagnet.mcfdbs — a dump of the multicast forwarding tables of the fabric switches
• ibdiagnet.masks — in case of duplicate port/node Guids, these file include the map between masked Guid and real Guids
• ibdiagnet.sm — list of all the SM (state and priority) in the fabric
• ibdiagnet.pm — a dump of the pm Counters values, of the fabric links
• ibdiagnet.pkey — a dump of the the existing partitions and their member host ports
• ibdiagnet.mcgs — a dump of the multicast groups, their properties and member host ports
• ibdiagnet.psl — a dump of the path record SL of each possible source node guid and DLID pairs. Created only if -vlr option is provided
• ibdiagnet.slvl — a dump of the SL2VL Tables from all the switches. Created only if -vlr option is provided
• ibdiagnet.db — a dump of the internal subnet database. This file can be loaded in later runs using the -load_db option

and that's not even the whole vibe — on top of generating all those files, the discovery phase also checks for duplicate node/port GUIDs in the IB fabric. if it catches an error like that, it shows up on standard output. once the discovery phase is done, directed route packets get sent multiple times (however many times you told it with the -c option) to sniff out any problematic paths where packets might get lost. those paths get explored, and a report of the suspected bad links gets displayed on standard output. 🔥

after it's done scanning the fabric, if you passed -r, you get a full report of the fabric qualities. and that report includes:

• SM report
• Number of nodes and systems
• Hop-count information
  • maximal hop-count, an example path, and a hop-count histogram
• All CA-to-CA paths traced
• Credit loop report
• mgid-mlid-HCAs multicast group and report
• Partitions report
• IPoIB report

Note: In case the IB fabric includes only one CA, then CA-to-CA paths are not reported. Furthermore, if a topology file is provided, ibdiagnet uses the names defined in it for the output reports.

---

💅 OPTIONS 💅

• -c <count> — min number of packets to be sent across each link (default = 10)
• -v — enable verbose mode
• -r — provides a report of the fabric qualities
• -u — credit loop check based on UpDown rules
• -t <topo-file> — specifies the topology file name
• -s <sys-name> — specifies the local system name. Meaningful only if a topology file is specified
• -i <dev-index> — specifies the index of the device of the port used to connect to the IB fabric (in case of multiple devices on the local system)
• -p <port-num> — specifies the local device's port num used to connect to the IB fabric
• -o <out-dir> — specifies the directory where the output files will be placed (default = /var/cache/ibutils)
• -lw <1x|4x|12x> — specifies the expected link width
• -ls <2.5|5|10> — specifies the expected link speed
• -pm — dump all the fabric links, pm Counters into ibdiagnet.pm
• -pc — reset all the fabric links pmCounters
• -P <PM=<Trash>> — if any of the provided pm is greater then its provided value, print it to screen
• -skip <skip-option(s)> — skip the executions of the selected checks. Skip options (one or more can be specified): dup_guids zero_guids pm logical_state part ipoib all
• -wt <file-name>> — write out the discovered topology into the given file. This flag is useful if you later want to check for changes from the current state of the fabric. A directory named ibdiag_ibnl is also created by this option, and holds the IBNL files required to load this topology. To use these files you will need to set the environment variable named IBDM_IBNL_PATH to that directory. The directory is located in /var/cache/ibutils or in the output directory provided by the -o flag.
• -load_db <file-name>> — load subnet data from the given .db file, and skip subnet discovery stage. Note: Some of the checks require actual subnet discovery, and therefore would not run when load_db is specified. These checks are: Duplicated/zero guids, link state, SMs status.
• -vlr — VL based Routing - provide this option to create the paths SL file (psl), SL2VL file (slvl) and use them in credit loops check (if -r option is provided).

NOTE: Since number of paths is N^2 extracting the PSL file may take some time.

• -h|--help — prints the help page information
• -V|--version — prints the version of the tool
• --vars — prints the tool's environment variables and their values

---

👑 ERROR CODES 👑

when it's not slaying, it's returning one of these:

1. Failed to fully discover the fabric
2. Failed to parse command line options
3. Failed to intract with IB fabric
4. Failed to use local device or local port
5. Failed to use Topology File
6. Failed to load requierd Package

---

and that's the whole page bestie, no cap 🫶 go diagnose something iconic.

this has been IBDIAG 1.0, dated 2024-04-22 — periodt. 💖✨