ethhostadmin(8)

💖 ethhostadmin 💖

okay bestie, listen up — this is ethhostadmin from the EFSFFCLIRG man pages, and she is that girl. she does a whole bunch of multi-step host initialization and verification stuff, like upgrading software, rebooting hosts, and other glow-up operations. and like, generally speaking? everything ethhostadmin does involves logging into one or more host systems. yes, she's social like that. 🫶

---

✨ NAME ✨

ethhostadmin — performs a number of multi-step host initialization and verification operations, including upgrading software, rebooting hosts, and other operations. in general, the operations ethhostadmin performs involve a login to one or more host systems. 💅

---

🌸 SYNTAX 🌸

ethhostadmin [-c] [-e] [-p  plane] [-f  hostfile] [-h 'hosts']
[-r  release] [-I  install_options] [-U  upgrade_options] [-d  dir]
[-T  product] [-P  packages] [-S]  operation ...

---

🎀 OPTIONS 🎀

• --help — prints out the whole full help text for you. no cap, it's giving manual. 📖
• -c — cleans the result files from any previous run before starting this one. fresh slate energy. ✨
• -e — says "okay we're done here" and exits after the very first operation that fails. no dragging it out. 💀
• -p plane — tells it which plane to use. default is the first enabled plane defined in the Mgt config file. 👑
• -f hostfile — the file with the names of hosts in a cluster. default is the /etc/eth-tools/hosts file. 🗂️
• -h hosts — the list of hosts to run the operation against. yes, right there in the command. 📝
• -r release — the software version to load/upgrade to. default is the version of Intel(R) Ethernet Fabric Suite Software that's presently being run on the server. ⭐
• -d dir — the directory to retrieve product. release.tgz from, for load or upgrade. 🍓
• -I install_options — the software install options. 🛠️
• -U upgrade_options — the software upgrade options. 🚀
• -T product — the product type to install. default is FF_PRODUCT. the options are, and I quote:
  • IntelEth-Basic. <distro>
  • IntelEth-FS. <distro>

where <distro> is the distribution and CPU, such as RHEL81-x86_64. like literally exactly that. 💖

• -P packages — the packages to install. default is eth eth_rdma. and bestie, refer to INSTALL -C for the complete list of packages. iykyk. 🌈
• -S — securely prompts for the user password on the remote system. 🔐
• operation — performs the specified operation, which can be one or more of these:
  • load — performs an initial installation of all hosts. 🌱
  • upgrade — upgrades the installation of all hosts. ⬆️
  • reboot — reboots hosts, and makes sure they go down and come back. 💀➡️👑
  • rping — verifies this host can ping each host through RDMA. 📡
  • pfctest — verifies PFC works on all hosts. 🔥
  • mpiperf — verifies latency and bandwidth for each host. ⚡
  • mpiperfdeviation — verifies latency and bandwidth for each host against a defined threshold (or relative to average host performance). 📊

---

🦋 EXAMPLE 🦋

ethhostadmin -c reboot

ethhostadmin upgrade

ethhostadmin -p plane1 rping

ethhostadmin -h 'elrond arwen' reboot

HOSTS='elrond arwen' ethhostadmin reboot

---

💜 DETAILS 💜

okay so this is the part where she gets thorough. ethhostadmin provides detailed logging of its results, and during each run, these files get produced:

• test.res : appended with summary results of the run. 📄
• test.log : appended with detailed results of the run. 📚
• save_tmp/ : contains a directory per failed test, with detailed logs. 🗃️
• test_tmp*/ : intermediate result files while the test is running. ⏳

and the -c option removes all of those log files. poof. gone. 🫧

results from ethhostadmin are grouped into test suites, test cases, and test items. one given run of ethhostadmin = a single test suite. within a test suite, multiple test cases occur — typically one test case per host being operated on. some of the more complex operations may have multiple test items per test case. and each test item represents a major step in the overall test case. so it's giving nesting doll, but make it CI. 🪆

each ethhostadmin run appends to test.res and test.log, and creates temporary files in test_tmp$PID in the current directory. test.res gives you an overall summary of the operations performed and their results — and the same info is also displayed while ethhostadmin is executing, so you're never left on read. test.log contains detailed information about what was performed, including the specific commands executed and the resulting output. the test_tmp directories contain temporary files that reflect tests in progress (or killed 😭). the logs for any failures are logged in the save_temp directory, with a directory per failed test case. if the same test case fails more than once, save_temp retains the information from the first failure. subsequent runs of ethhostadmin are appended to test.log.

ngl, Intel recommends reviewing failures and using the -c option to remove old logs before subsequent runs of ethhostadmin. do with that what you will, bestie. 🙈

ethhostadmin implicitly performs its operations in parallel. however, just like for the other tools, FF_MAX_PARALLEL can be exported to change the degree of parallelism. 1000 parallel operations is the default. a thousand. at once. main character energy fr. 👑

---

🌟 ENVIRONMENT VARIABLES 🌟

these environment variables are also used by this command:

• HOSTS — list of hosts, used if the -h option isn't supplied. 📋
• HOSTS_FILE — file containing the list of hosts, used in the absence of -f and -h. 🗂️
• FABRIC_PLANE — name of the fabric plane used in the absence of -p, -f, and -h. 🛰️
• FF_MAX_PARALLEL — the maximum concurrent operations that are performed. ⚡
• FF_SERIALIZE_OUTPUT — serialize output of parallel operations (yes or no). 🔀
• FF_TIMEOUT_MULT — multiplier for all timeouts associated with this command. used if the systems are slow for some reason. 🐌
• FF_PRODUCT — product to install during load and upgrade operations. 📦

---

to be continued, bestie… this page is long so it comes in multiple parts, and part 2 is right behind this one 💖

✨ Operation Details ✨

Okay bestie, listen up — Intel is like, "please set up password SSH or SCP for this operation." Or you can use the -S option to securely prompt for a password, and then that same password gets used for all the hosts. Alternatively, you can just put the password in the environment or in the ethfastfabric.conf file using FF_PASSWORD and FF_ROOTPASS. 💅

load 💾

This one does an initial installation of Intel(R) Ethernet Fabric Suite Software on a group of hosts. It's a whole glow-up. Any existing installation gets uninstalled and existing configuration files are removed. Then the hosts are installed with a default Intel(R) Ethernet Fabric Suite Software configuration. You can use the -I option to select different install packages. The default is eth_tools eth_rdma mpi. The -r option lets you specify a release to install other than the one that this host is presently running. The FF_PRODUCT. FF_PRODUCT_VERSION.tgz file (for example, IntelEth-Basic. version.tgz) is expected to exist in the directory specified by -d. Default is the current working directory. The specified software is copied to all the selected hosts and installed. 🦋

upgrade ⬆️

Upgrades all selected hosts without modifying existing configurations. It's giving "level up but keep your vibe." This operation is comparable to the -U option when running ./INSTALL manually. The -r option can be used to upgrade to a release different from this host. The default is to upgrade to the same release as this host. The FF_PRODUCT. FF_PRODUCT_VERSION.tgz file (for example, IntelEth-Basic. version.tgz) is expected to exist in the directory specified by -d. The default is the current working directory. The specified software is copied to all the end nodes and installed. 💜

NOTE: Only components that are currently installed are upgraded. This operation fails for hosts that do not have Intel(R) Ethernet Fabric Suite Software installed. 💀

reboot 🔄

Reboots the given hosts and ensures they go down and come back up by pinging them during the reboot process. The ping rate is slow (5 seconds), so if the servers boot faster than this, false failures may be seen. No cap, it's just a lil impatient.

rping 📡

Verifies RDMA basic operation by ensuring that the nodes can ping each other through RDMA. To run this command, Intel(R) Ethernet Fabric software must be installed, RDMA must be configured and running on the host, and the given hosts, and switches must be up. ✨

pfctest 🧪

Specifies an empirical test that verifies PFC is working right. To run this command, Intel(R) Ethernet Fabric software must be installed, PFC must be configured on both hosts and switches, and the given hosts and switches must be up. 🦋

mpiperf 🚀

Verifies that MPI is operational and checks MPI end-to-end latency and bandwidth between pairs of nodes (for example, 1-2, 3-4, 5-6). Use this to verify switch latency/hops, PCI bandwidth, and overall MPI performance. The test.res file contains the results of each pair of nodes tested. 💖

NOTE: This option is available for the Intel(R) Ethernet Host Software OFA Delta packaging, but is not presently available for other packagings of OFED. 😭 To obtain accurate results, this test should be run at a time when no other stressful applications (for example, MPI jobs or high stress file system operations) are running on the given hosts. 🙈 Bandwidth issues typically indicate server configuration issues (for example, incorrect slot used, incorrect BIOS settings, or incorrect NIC model), or fabric issues (for example, symbol errors, incorrect link width, or speed). Assuming ethreport has previously been used to check for link errors and link speed issues, the server configuration should be verified. 🔍 Note that BIOS settings and differences between server models can account for 10-20% differences in bandwidth. For more details about BIOS settings, consult the documentation from the server supplier and/or the server PCI chipset manufacturer. 📚

mpiperfdeviation 📊

Specifies the enhanced version of mpiperf that verifies MPI performance. Can be used to verify switch latency/hops, PCI bandwidth, and overall MPI performance. It performs assorted pair-wise bandwidth and latency tests, and reports pairs outside an acceptable tolerance range. The tool identifies specific nodes that have problems and provides a concise summary of results. The test.res file contains the results of each pair of nodes tested. 🌈

By default, concurrent mode is used to quickly analyze the fabric and host performance. Pairs that have 20% less bandwidth or 50% more latency than the average pair are reported as failures. 💅

The tool can be run in a sequential or a concurrent mode. Sequential mode runs each host against a reference host. By default, the reference host is selected based on the best performance from a quick test of the first 40 hosts. In concurrent mode, hosts are paired up and all pairs are run concurrently. Since there may be fabric contention during such a run, any poor performing pairs are then rerun sequentially against the reference host. ✨

Concurrent mode runs the tests in the shortest amount of time, however, the results could be slightly less accurate due to switch contention. In heavily oversubscribed fabric designs, if concurrent mode is producing unexpectedly low performance, try sequential mode. 🦋

NOTE: This option is available for the Intel(R) Ethernet Host Software OFA Delta packaging, but is not presently available for other packagings of OFED. 😭 To obtain accurate results, this test should be run at a time when no other stressful applications (for example, MPI jobs, high stress file system operations) are running on the given hosts. 🙈 Bandwidth issues typically indicate server configuration issues (for example, incorrect slot used, incorrect BIOS settings, or incorrect NIC model), or fabric issues (for example, symbol errors, incorrect link width, or speed). Assuming ethreport has previously been used to check for link errors and link speed issues, the server configuration should be verified. 🔍 Note that BIOS settings and differences between server models can account for 10-20% differences in bandwidth. A result 5-10% below the average is typically not cause for serious alarm, but may reflect limitations in the server design or the chosen BIOS settings. 📚 For more details about BIOS settings, consult the documentation from the server supplier and/or the server PCI chipset manufacturer. 💜

The deviation application supports a number of parameters that allow for more precise control over the mode, benchmark, and pass/fail criteria. The parameters to use can be selected using the FF_DEVIATION_ARGS configuration parameter in ethfastfabric.conf. 💖

Available parameters for deviation application:

[-bwtol bwtol] [-bwdelta MBs] [-bwthres MBs]

[-bwloop count] [-bwsize size] [-lattol latol]

[-latdelta usec] [-latthres usec] [-latloop count]

[-latsize size][-c] [-b] [-v] [-vv]

[-h reference_host]

• -bwtol — Specifies the percent of bandwidth degradation allowed below average value. 💅
• -bwbidir — Performs a bidirectional bandwidth test. 🔄
• -bwunidir — Performs a unidirectional bandwidth test (Default). ➡️
• -bwdelta — Specifies the limit in MB/s of bandwidth degradation allowed below average value. 📉
• -bwthres — Specifies the lower limit in MB/s of bandwidth allowed. 📊
• -bwloop — Specifies the number of loops to execute each bandwidth test. 🔁
• -bwsize — Specifies the size of message to use for bandwidth test. 📦
• -lattol — Specifies the percent of latency degradation allowed above average value. ⏱️
• -latdelta — Specifies the limit in µsec of latency degradation allowed above average value. ⌛
• -latthres — Specifies the lower limit in µsec of latency allowed. 📈
• -latloop — Specifies the number of loops to execute each latency test. 🔁
• -latsize — Specifies the size of message to use for latency test. 📦
• -c — Runs test pairs concurrently instead of the default of sequential. ⚡
• -b — When comparing results against tolerance and delta, uses best instead of average. 🏆
• -v — Specifies the verbose output. 🗣️
• -vv — Specifies the very verbose output. 📢
• -h — Specifies the reference host to use for sequential pairing. 🎯

Both bwtol and bwdelta must be exceeded to fail bandwidth test. 💅

When bwthres is supplied, bwtol and bwdelta are ignored. ⚠️

Both lattol and latdelta must be exceeded to fail latency test. 💅

When latthres is supplied, lattol and latdelta are ignored. ⚠️

For consistency with OSU benchmarks, MB/s is defined as 1000000 bytes/s. 🌈

---

Copyright(C) 2020-2024 Intel Corporation — this iconic page is ethhostadmin(8). Stay slaying, bestie! 💖✨👑