RFC 4392

IP over InfiniBand (IPoIB) Architecture

📅 published April 2006, from ietf

RFC 4392: IP over InfiniBand (IPoIB) Architecture 💖

okay bestie, this one is from April 2006 and it's an Informational RFC, so it's sharing knowledge rather than setting a standard 🎀. It was written by V. Kashyap from IBM, and the title block doesn't list any RFCs it obsoletes or updates, so it's standing on its own 🦋. The whole point is IP over InfiniBand (IPoIB) architecture, which is lowkey the foundation for everything after it ✨.

Status of This Memo 🌸

this memo is just here to give information to the Internet community. it's not a standard of any kind, and you're totally allowed to distribute it as much as you want 💅

Copyright (C) The Internet Society (2006).

Abstract 🔥

so, InfiniBand is a high-speed, channel-based interconnect between systems and devices. it's giving fast, direct connections 🧚

this doc gives an overview of the InfiniBand architecture. it also lays out the requirements and guidelines for sending IP over InfiniBand. the stuff in here works for both IPv4 and IPv6, unless it says otherwise, which is so convenient 💜 and the way IP over InfiniBand gets encapsulated, plus how IP addresses get resolved on IB fabrics, is covered in other documents 📚

Table of Contents 🍓

   1. Introduction to InfiniBand ......................................2
      1.1. InfiniBand Architecture Specification ......................2
      1.2. Overview of InfiniBand Architecture ........................2
           1.2.1. InfiniBand Addresses ................................6
                  1.2.1.1. Unicast GIDs ...............................7
                  1.2.1.2. Multicast GIDs .............................7
      1.3. InfiniBand Multicast Group Management ......................9
           1.3.1. Multicast Member Record ............................10
                  1.3.1.1. JoinState .................................10
           1.3.2. Join and Leave Operations ..........................11
                  1.3.2.1. Creating a Multicast Group ................11
                  1.3.2.2. Deleting a Multicast Group ................11
                  1.3.2.3. Multicast Group Create/Delete Traps .......12
   2. Management of InfiniBand Subnet ................................12
   3. IP over IB .....................................................12
      3.1. InfiniBand as Datalink ....................................13
      3.2. Multicast Support .........................................13
           3.2.1. Mapping IP Multicast to IB Multicast ...............14
           3.2.2. Transient Flag in IB MGIDs .........................14
      3.3. IP Subnets Across IB Subnets ..............................14
   4. IP Subnets in InfiniBand Fabrics ...............................14
      4.1. IPoIB VLANs ...............................................16
      4.2. Multicast in IPoIB subnets ................................16
           4.2.1. Sending IP Multicast Datagrams .....................17
           4.2.2. Receiving Multicast Packets ........................18
           4.2.3. Router Considerations for IPoIB ....................18
           4.2.4. Impact of InfiniBand Architecture Limits ...........19
           4.2.5. Leaving/Deleting a Multicast Group .................19
      4.3. Transmission of IPoIB Packets .............................20
      4.4. Reverse Address Resolution Protocol (RARP) and
           Static ARP Entries ........................................20
      4.5. DHCPv4 and IPoIB ..........................................21
   5. QoS and Related Issues .........................................21
   6. Security Considerations ........................................21
   7. Acknowledgements ...............................................21
   8. References .....................................................21
      8.1. Normative References ......................................21
      8.2. Informative References ....................................22

1. Introduction to InfiniBand ✨

okay so the InfiniBand Trade Association (IBTA) was formed to build an I/O specification for a channel-based, switched fabric technology. the whole goal is meeting the needs of servers in data centers, and that means scalability, reliability, availability, and performance, all at once 💅 it's literally the main character of data center networking 👑

1.1. InfiniBand Architecture Specification 🦋

the InfiniBand Trade Association specification is up for download at http://www.infinibandta.org. go grab it bestie 🌈

1.2. Overview of InfiniBand Architecture ✨

Okay bestie, if you want the full tea, go read chapter 3 of the InfiniBand specification 📚💅

InfiniBand Architecture (IBA) defines a System Area Network (SAN) for connecting multiple independent processor platforms, I/O platforms, and I/O devices 🦋 It's a communications and management infrastructure that handles both I/O and inter-processor communications for one or more computer systems. Iconic. 👑

An IBA SAN is made of processor nodes and I/O units, all linked through an IBA fabric built from cascaded switches and IB routers (which connect IB subnets) 🌸 I/O units can be super simple, like one single ASIC IBA-attached device (think a LAN adapter), or a massive, memory-rich Redundant Array of Independent Disks (RAID) subsystem 💾🔥

An IBA network can be split into subnets that get connected by routers. These are IB routers and IB subnets, NOT IP routers or IP subnets, so don't mix them up 🙈 In this doc, InfiniBand routers and subnets are called 'IB routers' and 'IB subnets', and IP routers and IP subnets are just 'routers' and 'subnets' 🧚

Each IB node or switch can attach to a single switch or multiple switches, or it can connect directly to other nodes or switches 🍓 Each IB unit hooks into the link through channel adapters (CAs). The architecture allows multiple CAs per unit, and each CA gives you one or more ports that plug into the fabric. From the fabric's point of view, each CA shows up as a node 💖

The ports are the endpoints where data gets sent. Each port can hold multiple QPs (Queue Pairs) that a remote peer can address directly, and from a data transfer standpoint, the QP number (QPN) is part of the address 🎀

IBA supports both connection-oriented and datagram service between ports 💜 Peers are identified by a QPN plus the port identifier. There are two exceptions, though 👀 QPNs aren't used when packets are multicast, and they aren't used in Raw Datagram mode either.

A port, inside a data packet, gets identified by a Local Identifier (LID) and, optionally, a Global Identifier (GID) 🧚 The GID only has to show up in the packet when you're talking across an IB subnet, but it can always be included if you want.

The GID is 128 bits long, built by sticking together a 64-bit IB subnet prefix and a 64-bit EUI-64-compliant portion 🌈 The EUI-64 part of a GID is called the Global Unique Identifier (GUID), where EUI stands for Extended Unique Identifier ✨ The LID is a 16-bit value that gets assigned when the port becomes active. The GUID is the only persistent identifier for a port, but it can't be used as an address in a packet 😭 If the prefix changes, the GID may change too. The subnet manager MAY try to keep the LID values constant across reboots, but that is NOT a requirement 🙈

Now, who hands out the GIDs and LIDs? The subnet manager! 👑 Every IB subnet has at least one subnet manager component that controls the fabric. It assigns the LIDs and GIDs, and it programs the switches so they route packets to their destinations 🦋 The subnet manager (SM) and its buddy, the subnet administrator (SA), are the central hub for all the information needed to set up and bring up the fabric. Main character energy 💅

IB routers route packets between IB subnets based on GIDs 🌸 So inside an IB subnet, a packet MAY or MAY NOT include a GID, but when it's going across an IB subnet, the GID MUST be included. A LID is always needed in a packet, because the destination within a subnet is determined by it 💖

A CA and a switch can each have multiple ports. Every CA port gets its own LID or a range of LIDs 🎀 The ports of a switch are not addressable by LIDs or GIDs, which in other words means they're transparent to other end nodes 🫶 Each port has its own set of buffers, and the buffering runs through virtual lanes (VL), where each VL has its own flow control. There MAY be up to 16 VLs 🌈

VLs let you make multiple virtual links inside a single physical link ✨ All ports MUST support VL15, which is reserved exclusively for subnet management datagrams, so it doesn't concern the IP over Infiniband (IPoIB) discussions 🙌 The actual VL a packet uses is configured by the SM in the switch and channel adapter tables, and it's determined by the Service Level (SL) in every packet. There are 16 possible SLs 💜

Besides QPs, SLs, and addressing (GID/LID), IBA also defines the following 💅

Partitioning:

Every packet, except for raw datagrams, carries the partition key (P_Key) 🔑 These values are used for isolation in the fabric. A switch (this is an OPTIONAL feature) MAY be programmed by the SM to drop packets that don't have a certain key 🙅‍♀️ The CA ports always check the P_Keys. A CA port MAY belong to multiple partitions, and P_Key checking is OPTIONAL at IB routers 🌸

A P_Key can be described as having 'limited membership' or 'full membership' 🦋 For a packet to be accepted, at least one of the P_Keys (meaning the P_Key in the packet or the P_Key in the port) MUST be a 'full membership' P_Key 🫶

Q_Keys:

Q_Keys enforce access rights for reliable and unreliable IB datagram services 🔐 Raw datagram services don't use Q_Keys. When communication gets set up, the endpoints swap Q_Keys and MUST always use the relevant Q_Keys when talking to each other. Multicast packets use the Q_Key tied to the multicast group 💖

Q_Keys with the most significant bit set are called controlled Q_Keys (like the General Service Interface (GSI) Q_Key [IB_ARCH]) 👑 A Host Channel Adapter (HCA) does NOT let a consumer arbitrarily specify a controlled Q_Key. If someone tries to send a controlled Q_Key, the Q_Key in the QP context gets used instead. That means the Operating System stays in control, since it can configure the QP context for the controlled Q_Key for privileged consumers 🛡️ It's worth noting that even though the IB specification suggests the idea of a 'controlled Q_Key', it doesn't require its use or implementation 🤷‍♀️

Multicast support:

A switch MAY support multicasting, which means replicating packets across multiple output ports 📡 This is an OPTIONAL feature. Similarly, support for sending and receiving multicast packets is OPTIONAL in CAs 🦋 A multicast group is identified by a GID. The GID format is as defined in RFC 2373 on IPv6 addressing [IB_ARCH] ✨ So from an IPv6-over-InfiniBand point of view, the data link multicast address looks like the network address 🌈 An IB port MUST explicitly join a multicast group by sending a request to the SM to receive multicast packets. A port MAY send packets to any multicast group. In both cases, the multicast LID used in the packets comes from the SM 💜

There are six methods for data transfer in the IB architecture 🎀

  1. Unreliable Datagram (unacknowledged - connectionless)

The Unreliable Datagram (UD) service is connectionless and unacknowledged, and it lets the QP talk to any unreliable datagram QP on any node 🕊️

The switches, and therefore each link, can only support a certain MTU. The MTU ranges are 256 octets, 512 octets, 1024 octets, 2048 octets, and 4096 octets. A UD packet cannot be larger than the link MTU between the two peers 📏

  1. Reliable Datagram (acknowledged - multiplexed)

The Reliable Datagram (RD) service is multiplexed over connections between nodes called End-to-End Contexts (EEC). This lets each RD QP talk to any RD QP on any node that has an established EEC 🤝 Multiple QPs can share the same EEC, and a single QP can use multiple EECs (one for each remote node per reliable datagram domain) 💫

  1. Reliable Connected (acknowledged - connection oriented)

The Reliable Connected (RC) service ties a local QP to one and only one remote QP 💞 Message sizes can be as large as 2^31 octets. The CA implementation handles segmentation and assembly for you, so you don't have to 🧁

  1. Unreliable Connected (unacknowledged - connection oriented)

The Unreliable Connected (UC) service ties one local QP to one and only one remote QP 🔗 There's no acknowledgement, so there's no resending lost or corrupted packets. Those packets just get dropped 😬 Otherwise, it's similar to RC.

  1. Raw Ethertype (unacknowledged - connectionless)

The Ethertype raw datagram packet has a generic transport header that the CA doesn't interpret, but it does specify the protocol type 🧩 The Ethertype values are the same as the ones defined by the Internet Assigned Numbers Authority (IANA) [IANA] 🌸

  1. Raw IPv6 (unacknowledged - connectionless)

With the IPv6 raw datagram service, the IBA CA can support standard protocol layers on top of IPv6 (like TCP/UDP) 🌐 So native IPv6 packets can get bridged into the IBA SAN and delivered straight to a port and its IPv6 raw datagram QP 🚀

The first four types are called IB transports, and the last two are classified as raw datagrams 💅 Raw datagram packets don't show the QP number, and they're limited by the link MTU in size 📦

The two connected modes and the Reliable Datagram mode MAY also support Automatic Path Migration (APM) 🛤️ This is an OPTIONAL facility that gives you hardware-based path failover. An alternate path gets associated with the QP when the connection or EE context is first created. If unrecoverable errors show up, the connection switches over to the alternate path 🦸‍♀️✨

1.2.1. InfiniBand Addresses

The InfiniBand architecture borrows HEAVILY from the IPv6 architecture when it comes to InfiniBand subnet structure and GIDs 💜

The InfiniBand architecture defines the GID associated with a port as a 128-bit unicast or multicast identifier 🎀 IBA takes the GID address format from RFC 2373 [IB_ARCH] and adds some extra properties and restrictions to make discovery, communication, and routing more efficient 🧠

Note: The IBA explicitly refers to RFC 2373, which is obsolete [RFC3513]. It MUST be noted that IBA is therefore unaffected by any further changes introduced in the IPv6 addressing architecture. 🙌

IBA defines two types of GIDs: unicast and multicast 🦋

1.2.1.1. Unicast GIDs ✨

okay so unicast GIDs are giving IPv6 vibes 🧚 they come in three scopes, and the IB specification spells them out like this:

  • a. link local: FE80/10 🔒
    • The IB routers will not forward packets with a link-local address in source or destination beyond the IB subnet.
  • b. site local: FEC0/10 🏢
    • A unicast GID used within a collection of subnets that is unique within that collection (e.g., a data center or campus) but is not necessarily globally unique. IB routers must not forward any packets with either a site-local Source GID or a site-local Destination GID outside of the site.
  • c. global: 🌍
    • A unicast GID with a global prefix. An IB router may use this GID to route packets throughout an enterprise or internet.

1.2.1.2. Multicast GIDs 📣

the multicast GIDs also copy the IPv6 multicast addresses (main character energy 👑). The IB specification defines them like this:

FFxy:<112 bits>

Flag bits: the nibble, denoted by x above, holds the 4 flag bits, which look like 000T. The first 3 bits are reserved and are set to zero. The last bit means this:

  • T=0: denotes a permanently assigned, that is, well-known GID 📌
  • T=1: denotes a transient group ⏳

Scope bits: the 4 bits, denoted by y in the GID above, are the scope bits. Table 1 shows what each scope value means:

         scope value        Address value

         0                        Reserved
         1                        Unassigned
         2                        Link-local
         3                        Unassigned
         4                        Unassigned
         5                        Site-local
         6                        Unassigned
         7                        Unassigned
         8                        Organization-local
         9                        Unassigned
         0xA                      Unassigned
         0xB                      Unassigned
         0xC                      Unassigned
         0xD                      Unassigned
         0xE                      Global
         0xF                      Reserved

                 Table 1

The IB specification also refers to RFC 2373 and RFC 2375 while defining the well-known multicast addresses. But then it says the well-known addresses apply to IB raw IPv6 datagrams only. 🤔 It must be noted though that a multicast group can be associated with only a single Multicast Global Identifier (MGID). So the same MGID cannot be associated with the UD mode and the Raw Datagram mode. not it being a one-MGID-one-mode situation 💀

1.3. InfiniBand Multicast Group Management 🎀

IB multicast groups, identified by MGIDs, are managed by the SM. The SM explicitly programs the IB switches in the fabric so that packets are received by all the members of the multicast group that request the reception of packets. The SM also needs to program the switches so that packets sent to the group by any group member reach all receivers in the multicast group. 🫶

IBA distinguishes between multicast senders and receivers. Even though all members of a multicast group can transmit to the group (and expect their packets to be correctly forwarded), not all members of the group are receivers. A port needs to explicitly request that multicast packets addressed to the group be forwarded to it.

A multicast group is created by sending a join request to the SM. As will be explained later, IBA defines multiple modes for joining a multicast group. The subnet manager records the group's multicast GID and the associated characteristics. The group characteristics are the group path MTU, whether the group will be used for raw datagrams or unreliable datagrams, the service level, the partition key associated with the group, the Local Identifier (LID) associated with the group, and so on. These characteristics are set at the time of group creation. The interested reader may look up the 'MCMemberRecord' attribute in the IB architecture specification [IB_ARCH] for the complete list of characteristics that define a group. 📖

A LID is associated with the multicast group by the SM at the time of the multicast group creation. The SM determines the multicast tree based on all the group members and programs the relevant switches. The Multicast LID (MLID) is what the switches use to route the packets. 🦋

Any member IB port that wants to take part in the multicast group must join the group. As part of the join operation, the node receives the group characteristics from the SM. At the same time, the subnet manager makes sure the requester can actually participate in the group by checking that it can support the group MTU and that it can reach the rest of the group members. Other group characteristics may need checking too.

The SM, for groups that span IB subnet boundaries, must interact with IB routers to find out whether this group exists in other IB subnets. If it does, the MTU must match across the IB subnets. 🌐

P_Key is another characteristic that must match across IB subnets, since the P_Key inserted into a packet is not modified by the IB switches or IB routers. So if the P_Keys did not match, the IB router(s) itself might drop the packets, or destinations on other subnets might drop the packets. 💅

A join operation may cause the SM to reprogram the fabric so that the new member can take part in the multicast group. By the same token, a leave may cause the SM to reprogram the fabric to stop forwarding packets to the requester. 🔄

1.3.1. Multicast Member Record 🗂️

The multicast group is maintained by the SM, with each group member represented by an MCMemberRecord [IB_ARCH]. Some of its components are these:

MGID      - Multicast GID for this multicast group
PortGID   - Valid GID of the port joining this multicast group
Q_Key     - Q_Key to be used by this multicast group
MLID      - Multicast LID for this multicast group
MTU       - MTU for this multicast group
P_Key     - Partition key for this multicast group
SL        - Service level for this multicast group
Scope     - Same as MGID address scope
JoinState - Join/Leave status requested by the port:
            bit 0: FullMember
            bit 1: NonMember
            bit 2: SendOnlyNonMember

1.3.1.1. JoinState 🔑

The JoinState shows the membership qualities a port wants to add while joining or creating a group, or to delete when leaving a group. Here's what each JoinState bit means:

  • FullMember: messages destined for the group are routed to and from the port. The SM may delete a group if there are no FullMembers in the group. 🗑️
  • NonMember: messages destined for the group are routed to and from the port. The port is not counted as a member for purposes of group creation or deletion. 👀
  • SendOnlyNonMember: group messages are only routed from the port, not to the port. The port is not counted as a member for purposes of group creation or deletion. 📤

A port may have multiple bits set in its record. When that happens, the membership qualities are the union of the JoinStates. A port may leave the multicast group for each of the JoinStates individually, or in any combination of JoinState bits [IB_ARCH]. 🧩

1.3.2. Join and Leave Operations 🚪

An IB port joins a multicast group by sending a join request (SubnAdmSet() method) to the SM, and leaves a multicast group by sending a leave message (SubnAdmDelete() method). The IBA specification [IB_ARCH] describes the methods and attributes to use when sending these messages. 📬

1.3.2.1. Creating a Multicast Group 🌱

There is no 'create' command to form a new multicast group. The FullMember bit in the JoinState must be set to create a multicast group. In other words, the first FullMember join request causes the group to be created as a side effect of that join request. Later join or leave requests may contain any combination of the JoinState bits. 🫧

The creator of the group specifies the Q_Key, MTU, P_Key, SL, FlowLabel, TClass, and the Scope value. A creator may ask for a suitable MGID to be created for it. Or the request can specify the MGID it wants. In both cases, the SM assigns the MLID. 🏷️

So a group is created with the specified values when the requester sets the FullMember bit and no such group already exists in the subnet. ✨

1.3.2.2. Deleting a Multicast Group 🧹

When the last FullMember leaves the multicast group, the SM may delete the multicast group, releasing all resources associated with the group, including those that might exist in the fabric itself. 🌪️

Note that a special 'delete' message does not exist. It is a side effect of the last FullMember 'leave' operation. 🙈

1.3.2.3. Multicast Group Create/Delete Traps 🚨

Ports can ask the SA to report whenever a multicast group is created or deleted. A port can choose the multicast group(s) it cares about by using their MGID, or by submitting a wild card request. The SA reports these events using traps 66 (for creates) and 67 (for deletes)[IB_ARCH].

So a port that wants to join a group without creating it itself can request a create notification. Or a port might even ask for notifications about every group that gets created (a wild card request). The SA will diligently tell them about each creation using those traps, and the requester can then join the multicast group indicated. Similarly, a SendOnlyNonMember or a NonMember might ask the SA to tell it about group deletions. The endnode, on receiving a delete report, can safely release the resources associated with the group. The associated MLID is no longer valid for the group and may be reassigned to a new multicast group by the SM. 🔁

2. Management of InfiniBand Subnet 🛠️

To help monitor and configure InfiniBand subnet components, a set of MIB modules needs to be defined. MIB modules are needed for the channel adapters, InfiniBand interfaces, InfiniBand subnet manager, and InfiniBand subnet management agents, and to allow management of specific device properties. It must be noted that the management objects addressed in the IPoIB documents cover all of the IB subnet components and are not limited to IP (over IB). The relevant MIB modules are described in separate documents and are not covered here. 📚

3. IP over IB 💖

As described in section 1.0, the InfiniBand architecture provides a broad set of capabilities to choose from when implementing IP over InfiniBand networks. 🧚‍♀️

The IPoIB specification must not, and does not, require changes in IP and higher-layer protocols. Nor does it mandate requirements on IP stacks to implement special user-level programs. It is an aim of the IPoIB specification that IPoIB changes be amenable to modularization and incorporation into existing implementations at the same level as other media types. 🎀

okay so InfiniBand gives two endpoints a few different ways to swap data, as we said earlier. here they are, bestie 🦋

           Reliable Connected (RC)
           Reliable Datagram  (RD)
           Unreliable Connected (UC)
           Unreliable Datagram (UD)
           Raw Datagram : Raw IPv6 (R6)
                        : Raw Ethertype (RE)

IPoIB can be built on any, several, or all of these services 💖 there's a case to be made for supporting any of the transport methods, depending on which features you're after.

The IB specification requires Unreliable Datagram mode to be supported by all the IB nodes 🫶 Host channel adapters (HCAs) are specifically required to support Reliable Connected (RC) and Unreliable Connected (UC) modes, but target channel adapters (TCAs) don't have the same requirement. Support for the two Raw Datagram modes is entirely optional 🌸 Raw Datagram mode uses a 16-bit Cyclic Redundancy Check (CRC), while the other modes get the stronger protection of a 32-bit CRC. so the raw modes are a little less protected, not gonna lie 😭

For simplicity, easy implementation, and playing nicely with existing stacks, it's desirable for the fabric to support multicasting. That's only possible in Unreliable Datagram (UD) mode and IB's Raw Datagram modes ✨

So it's only UD mode that is universal, supports multicast, AND supports a robust CRC 👑 with all three conditions in its corner, it's the obvious choice for IP over InfiniBand [RFC4391]. no notes, it's iconic 💅

Future documents might look at the connected modes 🦋 Compared to the limited link MTU that UD mode offers, the connected modes can give a big performance boost by using a larger MTU. Reliability also gets better if the connected modes' automatic path migration feature is used 🌈

3.2. Multicast Support 🎀

InfiniBand specification makes support of multicasting in the switches optional 💀 but multicast is a basic requirement in IP networks. so IPoIB requires that multicast-capable InfiniBand fabrics be used to implement IPoIB subnets 🔥

3.2.1. Mapping IP Multicast to IB Multicast 🧚

Well-known IP multicast groups are defined for both IPv4 and IPv6 [IANA, RFC3513]. Multicast groups can also be created dynamically at any time ✨ To avoid making unnecessary duplicates of multicast packets in the fabric, and to avoid unnecessary handling of those packets at the hosts, each IP multicast group should be linked to a different IB multicast group wherever possible 💜 There's a process defined in [RFC4391] for mapping IP multicast addresses to unique IB multicast addresses 🫶

3.2.2. Transient Flag in IB MGIDs 🌸

The IB specification describes the flag bits, as discussed in section 1.2 😭 It also defines some well-known IB MGIDs. Those MGIDs are reserved for IB's Raw Datagram mode, which is incompatible with IB's other transports. So any mapping defined from IP multicast addresses must not fall into IB's definition of a well-known address 🙈

That's why all IPoIB-related multicast GIDs always set the transient bit 💅

3.3. IP Subnets Across IB Subnets 🌈

some implementations might want to support multiple clusters of machines in their own IB subnets, while still being part of a common IP subnet 👀 For that solution, the IB specification needs several upgrades. The required enhancements are:

  1. A method for creating IB multicast GIDs that span multiple IB subnets. The partition keys and other parameters need to be consistent across IB subnets.
  2. Developing an IB routing protocol to determine the IB topology across IB subnets.
  3. Defining the process and protocols needed between IB nodes and IB routers.

until all of that is sorted, it's not possible to implement IPoIB subnets that span IB subnets 😤 the IPoIB standards have, however, been defined with this possibility in mind 🫶

4. IP Subnets in InfiniBand Fabrics 🏰

the IPoIB subnet gets overlaid on top of the IB subnet 💖 It comes up in these steps:

Note: the join/leave operation at the IP level will be referred to as IP_join/IP_leave, and the join/leave operations at the IB level will be referred to as IB_join in this document. 🎀

1. The all-IPoIB nodes IB multicast group is created

The fabric administrator creates an IB multicast group (henceforth called 'broadcast group') when the IP subnet is set up 🔥 The 'broadcast group' is defined in [RFC4391]. IPoIB doesn't define how the broadcast group gets set up. The group may be set up at the SM by the administrator, or by the first IB_join 🧚

As noted earlier, when you create an IB multicast group, you have to specify multiple values like the P_Key, Q_Key, Service Level, Hop Limit, Flow ID, TClass, MTU, etc. 📝 These values should be chosen so that all potential members of the IB multicast group can communicate with one another when using them. In the future, as the IB specification attaches more meaning to the various parameters and defines IB Quality of Service (QoS), different values for IP multicast traffic may be possible ✨ All unicast packets also need to use the P_Key and Q_Key specified in the broadcast group [RFC4391]. It's obvious that a thought-out configuration is needed for a successful IPoIB subnet setup 💅

2. All IPoIB interfaces IB_join the broadcast group

The broadcast group defines the span and the members of the IPoIB link 🦋 This link gets built up as IPoIB nodes IB_join the broadcast group.

Joining the broadcast group has the extra perk of distributing those multicast group parameters to all the members of the subnet 🌈

Note that this IB_join to the broadcast group is a FullMember join 🫶 If any of the ports, or the switches linking the port to the rest of the IPoIB subnet, can't support the parameters (e.g., path MTU or P_Key) associated with the broadcast group, then the IB_join request will fail and the requesting port will not become part of the IPoIB subnet 😭

3. Configuration Parameters

As noted above, parameters like Q_Key and Path MTU, which every IPoIB communication needs, are returned to the IPoIB node when it IB_joins the 'broadcast group' 🎀 [RFC4391] also notes that the parameters used in the broadcast group are used when creating other multicast groups.

However, the P_Key must still be known to the IPoIB endnode before it can join the broadcast group 👀 The P_Key is included in the mapping of the broadcast group [RFC4391]. Another parameter, the scope of the broadcast group, also needs to be known to the endnode before it can join 🧚 How the P_Key and the scope bits related to the IPoIB subnet get determined is an implementation choice. They could be configuration parameters that the administrator initializes by some means ✨

The methods an implementation uses to determine the P_Key and scope bits are not specified by IPoIB 💅

4.1. IPoIB VLANs 💜

The endpoints in an IB subnet must have compatible P_Keys to communicate with one another 🫶 So the administrator, when setting up an IP subnet over an IB subnet, must make sure all the members have compatible P_Keys. An IP subnet can have only one P_Key associated with it, so that all IP nodes in it can talk to one another. An endpoint may, however, have multiple P_Keys 🌸

The IB architecture specifies that there can be only one MGID associated with a multicast group in the IB subnet 🔥 The P_Key is included in the MGID mappings from IP multicast addresses [RFC4391]. Since the P_Key is unique in the IB subnet, including it in the IB MGIDs makes sure unique MGID mappings get created 🙌 Every unique broadcast group MGID formed this way creates a separate abstract IPoIB link, and therefore an IPoIB VLAN 👑

4.2. Multicast in IPoIB subnets 🦋

IP multicast on InfiniBand subnets follows the same concepts and rules as on any other media 💖 But unlike most other media, multicast over InfiniBand requires interaction with another entity: the IB subnet manager 😤 This section describes the outline of the process and suggests some guidelines 📝

IB architecture specifies the following format for IB multicast packets when used over Unreliable Datagram (UD) mode 👀

   +--------+-------+---------+---------+-------+---------+---------+
   |Local   |Global |Base     |Datagram |Packet |Invariant| Variant |
   |Routing |Routing|Transport|Extended |Payload| CRC     |  CRC    |
   |Header  |Header |Header   |Transport| (IP)  |         |         |
   |        |       |         |Header   |       |         |         |
   +--------+-------+---------+---------+-------+---------+---------+

For details about the various headers, please refer to the InfiniBand Architecture Specification [IB_ARCH] 📚

The Global Routing Header (GRH) includes the IB multicast group GID 🧚 The Local Routing Header (LRH) includes the Local Identifier (LID). The IB switches in the fabric route the packet based on the LID ✨

The GID is made available to the receiving IB user (the IPoIB interface driver, for example) 🫶 So the driver can work out which IB group the packet belongs to 💅

IPv4 defines three levels of multicast conformance [RFC1112] 🌈

  • Level 0: No support for IP multicasting
  • Level 1: Support for sending but not receiving multicasts
  • Level 2: Full support for IP multicasting

In IPv6 there's no such distinction 💜 Full multicast support is mandatory. Also, all IPv4 subnets support broadcast (255.255.255.255) 🔥 IPv4 broadcast can always be sent and received by all IPv4 interfaces 🙌

Every IPoIB subnet requires the broadcast GID to be defined, so a packet can always be broadcast 🎀

4.2.1. Sending IP Multicast Datagrams 📮

An IP host may send a multicast packet at any time, to any multicast address 🦋

The IP layer hands the multicast packet to the IPoIB interface driver/module ✨ That module tries to IB_join the relevant IB multicast group 🫶 This is needed, because otherwise the InfiniBand architecture does not guarantee that the packet will reach its destinations 😭

A pure sender may choose to join the multicast group as a FullMember 💖 In that case, the sender will receive all the multicast packets transmitted to the IB group. Also, the IB group won't be deleted until the sender leaves the group 👑

Alternatively, a sender might IB_join as a SendOnlyNonMember 🌸 In that case, packets are not routed to the sender, though packets it transmits can still reach the other group members. Also, the group can be deleted once all FullMembers have left it. The sender can further request delete updates from the SM 🧚

If the sender doesn't find the group in existence, [RFC4391] recommends that the packets be sent to the MGID corresponding to the all-IP routers address 🌈 A sender could also send the packets to the broadcast group 💅 The sender might also choose to request 'creation' reports from the SM ✨

4.2.2. Receiving Multicast Packets

okay so receivers have a job too, bestie 🎀 The IP host MUST join the IB multicast group that matches its IP address 🫶 This is because of the IBA requirement that the receiver MUST join the relevant IB multicast group. If the group doesn't exist yet, it gets automatically created [IB_ARCH] ✨

The IP receivers MUST IB_leave the IB group when the IP layer stops listening to that IP address 💅 After that, the SM can choose to delete the group 🗑️

4.2.3. Router Considerations for IPoIB

okay this section is giving router drama 🦋 IP routers learn about new IP groups in the subnet through protocols like Internet Group Management Protocol (IGMPv3) / Multicast Listener Discovery (MLD) [RFC3376, RFC2710]. But that's NOT enough for IPoIB, ngl 😭 The router also needs to IB_join the relevant IB groups so it can receive and transmit the packets. And there is no promiscuous mode for listening to all packets, so no sneaking around 🙈

So the IPoIB routers need to ask the SM to report every creation of IB groups in the fabric 📣 Then the router can IB_join the reported group 💖 Here's the tea: it's not desirable for the router's IB_join to count the same as a receiver's IB_join. The router's IB_join SHOULD NOT disallow the group's deletion when all the receivers leave 💀 To handle exactly this situation, IBA gives us the NonMember IB_join mode 🧚

The NonMember IB_join mode can be used by IP routers when they join in response to create reports ✨ A router SHOULD, ideally, request the delete reports too, so it can free up all the resources tied to the group 🧹 Fair warning: the MLID tied to a deleted MGID can be reassigned by the SM. So if a router caches the MLID, it could end up transmitting to the wrong place 😬 A router that does NOT request delete reports still works fine, though, because it gets the correct MLID and purges any old cached value when it IB_joins in response to a create report 🌸

It's reasonable for a router to IB_join as a FullMember if it's joining in response to an application or routing daemon request 👑 In that case, the router might end up controlling whether the IB group exists, since it's a FullMember of the group.

4.2.4. Impact of InfiniBand Architecture Limits

Real talk: an HCA or TCA may have a limit on the number of MGIDs it can support 🥲 So even if the groups aren't limited at the subnet manager or across the subnet, they can still be limited at a particular interface. It is advisable to pick an adequately provisioned HCA/TCA when you set up an IPoIB subnet 🍓

4.2.5. Leaving/Deleting a Multicast Group

Okay this one's about tidying up 🧹 An IPv4 sender (level 1 compliance) IB_joins the IB multicast group only because that's the only way to guarantee every group recipient gets the packets 🦋 The sender MUST, however, IB_leave the group at some point. A sender that isn't a receiver on the group can start a timer for each multicast group it sends to. When the timer goes off, the sender leaves the IB group. If another message gets sent, it restarts the timer ⏰

This doesn't apply to the IB broadcast group, and it also doesn't apply to the IB group for the all-hosts multicast group 🙅‍♀️ An IPv4 host MUST always remain a member of the broadcast group. Non-negotiable, bestie 👑

An IP multicast receiver IB_leaves the matching IB multicast group when it IP_leaves the IP multicast group 🫶 For IPv4 implementations, the receiver MAY choose to keep being a sender (level 1 compliance). In that case, it MAY choose NOT to IB_leave the IB group and instead start a timer like we talked about above ✨

As noted elsewhere, the SM can choose to free up resources (for example, routing entries in the switches) tied to the IB group once the last FullMember IB_leaves the group 🧹 At that point the MLID becomes invalid for the group, and it can be reassigned when a new group gets created. So SendOnlyNonMember/NonMember ports caching the MLID need to avoid this. The way out is for them to request group delete reports 🙌 An IP router that requests reports for all groups doesn't need to request the delete report, because an IB_join in response to a create report will hand it the new MLID association 💜

Last thing: a router might prefer to IB_leave the IB multicast group when there are no members of the IP multicast address in the subnet, and it has no explicit knowledge of any need to forward those packets 🌈

4.3. Transmission of IPoIB Packets

The encapsulation of IP packets in InfiniBand is described in [RFC4391] 📦

IPoIB uses an 'Ethertype' value [IANA] in all IPoIB communication packets 🏷️ The link-layer address is made up of the GID and the Queue Pair Number (QPN) [RFC4391] 🦋

Now the clever part ✨ To let IPoIB subnets span multiple IB-subnets, the spec uses the GID as part of the link-layer address. Here's the catch: every packet in IB has to use the Local Identifier (LID), so address resolution gets an extra step. The destination GID (returned in response to an Address Resolution Protocol (ARP) / Neighbor Discover (ND) request) has to be resolved to the LID [RFC4391] 🧩 This phase might also figure out other essential parameters (e.g., the SL, path rate, etc.) for successful IB communication between two peers 🚀

As noted earlier, all communication in the IPoIB subnet derives the Q_Key to use from the Q_Key specified in the broadcast group 🔑

4.4. Reverse Address Resolution Protocol (RARP) and Static ARP Entries

RARP entries and static ARP entries are based on invariant link addresses 🧊 With IPoIB, though, the link address includes the QPN, and that might not stay constant across reboots or even across network interface resets 😬 So static ARP entries or RARP server entries will only work if the implementation(s) using these options can make sure the QPN for an interface stays invariant across reboots/network resets [RFC4391] 🔒

4.5. DHCPv4 and IPoIB

DHCPv4 [RFC2131] uses a 'client identifier' field (expected to hold the link-layer address) that's 16 octets long 📏 The IPoIB address is 20 octets, so that's a problem 🙃 To get around it, IPoIB specifies [RFC4390] that the client uses the 'broadcast flag' when requesting an IP address 📢

The IB specification suggests using service levels for load balancing, QoS, and deadlock avoidance within an IB subnet 💅 But it leaves the usage and the way the SL gets determined up to the application 🤷‍♀️ The SL and the list of SLs are available in the SA, but it's on the endnode's application to choose the 'right' value 🎯

Every IPoIB implementation determines the relevant SL value based on its own policy 🌸 No method or process for choosing the SL has been defined by the IPoIB standards, so it's really a vibe check per implementation ✨

6. Security Considerations

Okay bestie, read this part carefully 🔐

This document describes the IB architecture as it's relevant to IPoIB 📘 It also restates issues specified in other documents. It does not itself specify any requirements. There are no security issues introduced by this document. IPoIB-related security issues are described in [RFC4391] and [RFC4390] 🛡️

7. Acknowledgements

This document has benefited from the comments and suggestions of the members of the IPoIB working group and the members of the InfiniBand(SM) Trade Association 🫶💜

8. References

8.1. Normative References

  • [IB_ARCH] InfiniBand Architecture Specification, Volume 1, Release 1.2, October, 2004.
  • [RFC4391] Chu, J. and V. Kashyap, "Transmission of IP over InfiniBand (IPoIB)", RFC 4391, April 2006.
  • [RFC4390] Kashyap, V., "Dynamic Host Configuration Protocol (DHCP) over InfiniBand", RFC 4390, April 2006.
  • [RFC2131] Droms, R., "Dynamic Host Configuration Protocol", RFC 2131, March 1997.

8.2. Informative References

  • [RFC3513] Hinden, R. and S. Deering, "Internet Protocol Version 6 (IPv6) Addressing Architecture", RFC 3513, April 2003.
  • [RFC2375] Hinden, R. and S. Deering, "IPv6 Multicast Address Assignments", RFC 2375, July 1998.
  • [IANA] Internet Assigned Numbers Authority, URL http://www.iana.org
  • [RFC1112] Deering, S., "Host extensions for IP multicasting", STD 5, RFC 1112, August 1989.
  • [RFC3376] Cain, B., Deering, S., Kouvelas, I., Fenner, B., and A. Thyagarajan, "Internet Group Management Protocol, Version 3", RFC 3376, October 2002.
  • [RFC2710] Deering, S., Fenner, W., and B. Haberman, "Multicast Listener Discovery (MLD) for IPv6", RFC 2710, October 1999.

Author's Address

Vivek Kashyap IBM 15450, SW Koll Parkway Beaverton, OR 97006

Phone: +1 503 578 3422 EMail: vivk@us.ibm.com

Copyright (C) The Internet Society (2006).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document, or the extent to which any license under such rights might or might not be available. It also doesn't represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.

Acknowledgement

Funding for the RFC Editor function is provided by the IETF Administrative Support Activity (IASA) 💖🎀

📦 where this page comes from