ibacm(8)

💖 ibacm — bestie edition 💖

okay so. welcome to ibacm(8), the address and route resolution services for InfiniBand, and honestly? she's a whole mood. let's get into it 🫶

✨ NAME ✨

ibacm — address and route resolution services for InfiniBand. that's it, that's the name, and she serves. 💅

🌸 SYNOPSIS 🌸

ibacm [-D] [-P] [-A addr_file] [-O option_file]

like. look at that. so clean. so minimal. main character energy fr fr ✨

🦋 DESCRIPTION 🦋

okay so the IB ACM? she implements and provides a framework for name, address, and route (path) resolution services over InfiniBand. why do we care? because she's literally here to fix connection setup scalability issues when you're running MPI applications on large clusters. no cap. the IB ACM gives you the information you need to establish a connection, but — and this is important bestie — she does not implement the CM protocol. she's the wingwoman, not the whole relationship. 💀

a primary user of the ibacm service is the librdmacm library. and the way this slays is that it lets applications make use of the ibacm service without code changes or even needing to be aware that the service is in use. like, she's just there, doing her thing, unnoticed, iconic. librdmacm versions 1.0.12 - 1.0.15 can invoke IB ACM services when built using the --with-ib_acm option. version 1.0.16 and newer of librdmacm will automatically use the IB ACM if it is installed. she doesn't even need to be asked anymore. 😌

the IB ACM services tie in under the rdma_resolve_addr, rdma_resolve_route, and rdma_getaddrinfo routines. for maximum benefit, the rdma_getaddrinfo routine should be used, however existing applications should still see significant connection scaling benefits using the calls available in librdmacm 1.0.11 and previous releases. so like, you don't have to upgrade, but you should, obviously. 🎀

the IB ACM is focused on being scalable, efficient, and extensible. she implements a plugin architecture that allows a vendor to supply its proprietary provider in addition to the default provider. the current default provider implementation ibacmp limits network traffic, SA interactions, and centralized services. ibacmp supports multiple resolution protocols in order to handle different fabric topologies. flexible queen. 👑

the IB ACM package is comprised of three components: the ibacm core service, the default provider ibacmp shared library, and a test/configuration utility — ib_acme. all three are userspace components and are available for Linux. additional details are given below. keep reading bestie, there's lore. 📖

✨ OPTIONS ✨

• -D — run in daemon mode (default). she's always in daemon mode unless you say otherwise. the default. the moment. 💖
• -P — run as standard process. for when you want her to just. be a normal process. no daemon drama. 🫶
• -A addr_file — address configuration file. tell her where the address file lives.
• -O option_file — option configuration file. tell her where the options file lives.
• --systemd — Enable systemd integration. This includes optional socket activation of the daemon's listening socket. systemd girlies rise up 🌈

🌟 QUICK START GUIDE 🌟

1. Prerequisites: libibverbs and libibumad must be installed. The IB stack should be running with IPoIB configured. These steps assume that the user has administrative privileges. yes bestie you need admin. put the sudo down and pick it back up correctly. 💅
2. Install the IB ACM package. This installs ibacm, ibacmp, ib_acme, and init.d scripts. a whole little family moving in. 🏠
3. Run ibacm as administrator to start the ibacm daemon. go go go ✨
4. Optionally, run ib_acme -d <dest_ip> -v to verify that the ibacm service is running. vibe check complete. ✅
5. Install librdmacm, using the build option --with-ib_acm if needed. This build option is not needed with librdmacm 1.0.17 or newer. The librdmacm will automatically use the ibacm service. On failures, the librdmacm will fall back to normal resolution. she has a backup plan, she's prepared. 🎀
6. You can use ib_acme -P to gather performance statistics from the local ibacm daemon to see if the service is working correctly. Similarly, the command ib_acme -e could be used to enumerate all endpoints created by the local ibacm service. stats? enumerated. we love a diagnostic moment. 📊

💜 NOTES 💜

ib_acme:

the ib_acme program serves a dual role. It acts as a utility to test ibacm operation and help verify if the ibacm service and selected protocol is usable for a given cluster configuration. Additionally, it automatically generates ibacm configuration files to assist with or eliminate manual setup. dual role. double threat. she contains multitudes. 💖

ibacm configuration files:

the ibacm service relies on two configuration files.

the ibacm_addr.cfg file contains name and address mappings for each IB <device, port, pkey> endpoint. Although the names in the ibacm_addr.cfg file can be anything, ib_acme maps the host name to the IB endpoints. IP addresses, on the other hand, are assigned dynamically. If the address file cannot be found, the ibacm service will attempt to create one using default values. she'll figure it out herself, honestly. resourceful. 🌸

the ibacm_opts.cfg file provides a set of configurable options for the ibacm core service and default provider, such as timeout, number of retries, logging level, etc. ib_acme generates the ibacm_opts.cfg file using static information. If an option file cannot be found, ibacm will use default values. defaults are always there for you bestie. 🫶

ibacm:

the ibacm service is responsible for resolving names and addresses to InfiniBand path information and caching such data. It should execute with administrative privileges. yes. admin. again. it's giving required. 🔐

the ibacm implements a client interface over TCP sockets, which is abstracted by the librdmacm library. One or more providers can be loaded by the core service, depending on the configuration. In the default provider ibacmp, one or more back-end protocols are used to satisfy user requests. Although ibacmp supports standard SA path record queries on the back-end, it also supports a resolution protocol based on multicast traffic. The latter is not usable on all fabric topologies, specifically ones that may not have reversible paths or fabrics using torus routing. Users should use the ib_acme utility to verify that multicast protocol is usable before running other applications. ⚠️ okay so listen — verify first. don't just assume. run the utility. protect your cluster. 💀

Conceptually, the default provider ibacmp implements an ARP like protocol and either uses IB multicast records to construct path record data or queries the SA directly, depending on the selected route protocol. By default, the ibacmp provider uses and caches SA path record queries.

Specifically, all IB endpoints join a number of multicast groups. Multicast groups differ based on rates, mtu, sl, etc., and are prioritized. All participating endpoints must be able to communicate on the lowest priority multicast group. The ibacmp assigns one or more names/addresses to each IB endpoint using the ibacm_addr.cfg file. Clients provide source and destination names or addresses as input to the service, and receive as output path record data. input in, path record data out. the pipeline is pipelining. 🔥

The service maps a client's source name/address to a local IB endpoint. If the destination name/address is not cached locally in the default provider, it sends a multicast request out on the lowest priority multicast group on the local endpoint. The request carries a list of multicast groups that the sender can use. The recipient of the request selects the highest priority multicast group that it can use as well and returns that information directly to the sender. The request data is cached by all endpoints that receive the multicast request message. The source endpoint also caches the response and uses the multicast group that was selected to construct or obtain path record data, which is returned to the client. everybody's caching, everybody's communicating, it's a whole group chat and everyone's invited. 📣

The current implementation of the provider ibacmp has several additional restrictions:

• The ibacmp is limited in its handling of dynamic changes. ibacm must be stopped and restarted if a cluster is reconfigured. no hot-swapping bestie. restart her. 🔄
• Support for IPv6 has not been verified. 🙈 not verified. don't come for me.
• The number of multicast groups that an endpoint can support is limited to 2. just 2. two. that's the limit. 😭

The ibacmp contains several internal caches. These include caches for GID and LID destination addresses. These caches can be optionally preloaded. ibacm supports the OpenSM dump_pr plugin "full" PathRecord format which is used to preload these caches. The file format is specified in the ibacm_opts.cfg file via the route_preload setting which should be set to full_opensm_v1 for this file format. Default format is none which does not preload these caches. See dump_pr.notes.txt in dump_pr for more information on the full_opensm_v1 file format and how to configure OpenSM to generate this file. 📄 (yes, go read the notes file, she's not gonna explain it twice)

Additionally, the name, IPv4, and IPv6 caches can be be preloaded by using the addr_preload option. The default is none which does not preload these caches. To preload these caches, set this option to acm_hosts and configure the addr_data_file appropriately. preload girlies stay winning. ⭐

🌈 SEE ALSO 🌈

• ibacm(7)
• ib_acme(1)
• rdma_cm(7)

okay that's the whole page bestie, you ate it, you're literally an InfiniBand expert now 💅

this was ibacm, from the ibacm source, dated 2014-06-16, section ibacm(8). mwah. 🎀✨