RFC 4391
Transmission of IP over InfiniBand (IPoIB)
RFC 4391: Transmission of IP over InfiniBand (IPoIB) 🦋
Okay bestie, here's the title block tea ☕:
- Series: Network Working Group
- Authors: J. Chu (Sun Microsystems) and V. Kashyap (IBM)
- Category: Standards Track
- Date: April 2006
- Obsoletes / Updates: the title block doesn't list any RFCs it obsoletes or updates 🙅♀️
Status of This Memo 📜
So this doc is an Internet standards track protocol for the Internet community, and it's basically inviting everyone to chat about it and send in improvements 💬. For the official standardization state and status of this protocol, check the current edition of "Internet Official Protocol Standards" (STD 1). And yes, distribution of this memo is totally unlimited 🌈.
Copyright Notice ©️
Copyright (C) The Internet Society (2006). Legal stuff, but it's cute that it's still here 🎀.
Abstract ✨
This doc is all about how to wrap up IPv4 and IPv6 packets, plus Address Resolution Protocol (ARP) packets, and send them over InfiniBand (IB) 🦋. It explains which link-layer address you use when you're resolving IP addresses in IP over InfiniBand (IPoIB) subnets. It also covers how IP multicast addresses get turned into InfiniBand multicast addresses, and it defines how to set up and configure IPoIB links 🔧.
Table of Contents 🗂️
1. Introduction ....................................................2
2. IP over UD Mode .................................................2
3. InfiniBand Datalink .............................................3
4. Multicast Mapping ...............................................3
4.1. Broadcast-GID Parameters ...................................5
5. Setting Up an IPoIB Link ........................................6
6. Frame Format ....................................................6
7. Maximum Transmission Unit .......................................8
8. IPv6 Stateless Autoconfiguration ................................8
8.1. IPv6 Link-Local Address ....................................9
9. Address Mapping - Unicast .......................................9
9.1. Link Information ...........................................9
9.1.1. Link-Layer Address/Hardware Address ................11
9.1.2. Auxiliary Link Information .........................12
9.2. Address Resolution in IPv4 Subnets ........................13
9.3. Address Resolution in IPv6 Subnets ........................14
9.4. Cautionary Note on QPN Caching ............................14
10. Sending and Receiving IP Multicast Packets ....................14
11. IP Multicast Routing ..........................................16
12. New Types of Vulnerability in IB Multicast ....................17
13. Security Considerations .......................................17
14. IANA Considerations ...........................................18
15. Acknowledgements ..............................................18
16. References ....................................................18
16.1. Normative References .....................................18
16.2. Informative References ...................................191. Introduction ✨
Okay so the InfiniBand spec [IBTA] lives at http://www.infinibandta.org 🌐. The doc [RFC4392] is the short-and-sweet overview of InfiniBand architecture (IBA), plus what to think about when you're specifying IP over InfiniBand networks 🧠.
IBA has multiple transport modes that IP could be run over 🚇. The Unreliable Datagram (UD) mode is the best match for IP's needs, and for the universality described in [RFC4392] 💖.
This doc only specifies IPoIB over IB's UD mode. Running IP subnets over IB's other transport mechanisms is out of scope here 🙅♀️.
It walks through the steps you need to lay an IP network on top of an IB network 🏗️. It breaks down all the elements of an IPoIB link, how to configure its associated attributes, and how to set up basic broadcast and multicast services for it 📡.
It also covers IP address resolution, plus how IP and Address Resolution Protocol (ARP) packets get encapsulated in an InfiniBand frame 🎁.
The BCP 14 key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be read as described in RFC 2119 [RFC2119]. These are the official words that carry the weight in this spec, so keep an eye on them 👀.
2. IP over UD Mode 🌸
UD mode is supported by every IB element, whether it's an IB router, a Host Channel Adapter (HCA), or a Target Channel Adapter (TCA) 🙌. It's not just the only universal transmission method: it also supports multicasting, partitioning, and a 32-bit Cyclic Redundancy Check (CRC) [IBTA] 🔐. Multicasting support is optional in IB fabrics, but the IPoIB architecture requires the participating components to support it 💅.
All IPoIB implementations MUST support IP over the UD transport mode of IBA. This one is non-negotiable, no cap 🚨.
3. InfiniBand Datalink 🔗
An IB subnet is a network of IB nodes, connected directly or through IB switches 🕸️. You can connect IB subnets with IB routers to make a fabric made of multiple IB subnets 🏙️. Nodes in different IB subnets can talk to each other directly through IB routers at the IB network layer 💬. And you can overlay multiple IP subnets on top of this IB network 🍰.
An IP subnet gets configured over a communication facility or medium where nodes can talk at the "link" layer [IPV6] 🛣️. For example, an ethernet segment is a link formed by interconnected switches, hubs, or bridges 🔌. That segment is defined by the physical topology of the network. IPoIB is different, though 🙃. IPoIB subnets are built over an abstract "link", which is defined by its members and common characteristics, like the P_Key, the link MTU, and the Q_Key 🧩.
Any two ports using UD communication mode in an IB fabric can only communicate if they're in the same partition (meaning they have the same P_Key and the same Q_Key) [RFC4392] 🤝. The link MTU limits the size of the payload that can be used 📦. Packet transmission and routing within the IB fabric are also shaped by other parameters, like the traffic class (TClass), hop limit (HopLimit), service level (SL), and the flow label (FlowLabel) [RFC4392] 🗺️. How these values get determined and used for IPoIB communication is covered in the sections that follow ⭐.
4. Multicast Mapping 🦋
IB identifies multicast groups by their Multicast Global Identifiers (MGIDs) 🆔. These follow the same rules as IPv6 multicast addresses, so they follow the same rules for transient addresses and scope bits, just in the context of the IB fabric 🌈. That means the resulting address looks a lot like an IPv6 multicast address 💜. For a detailed description of IB multicast, check out [IBTA, RFC4392] 📚.
The IPoIB multicast mapping is shown in figure 1 ✨. The same mapping function is used for both IPv4 and IPv6, except for the IPoIB signature field 🎀.
Unless it's explicitly stated otherwise, all addresses and fields in the protocol headers in this document are stored in network byte order 🔢.
| 8 | 4 | 4 | 16 bits | 16 bits | 80 bits |
+------ -+----+----+-----------------+---------+-------------------+
|11111111|0001|scop|<IPoIB signature>|< P_Key >| group ID |
+--------+----+----+-----------------+---------+-------------------+Figure 1
An MGID allocated for carrying IP multicast datagrams is only treated as a transient link-layer multicast address [RFC4392], so all IB MGIDs allocated for IPoIB use MUST set T-flag to 1 [IBTA] 🚩.
There's a special signature embedded in the MGID to mark it as IPoIB-only 🔖. For IPv4 over IB, the signature MUST be "0x401B". For IPv6 over IB, the signature MUST be "0x601B" 💫.
The IP multicast address gets combined with a given IPoIB link P_Key to form the MGID of the IB multicast group 🧁. For IPv6, the lower 80 bits of the group ID go straight into the lower 80 bits of the MGID. For IPv4, the group ID is only 28 bits long, and it goes straight into the lower 28 bits of the MGID. Any remaining group ID bits in the MGID get filled with 0 🫧.
E.g., on an IPoIB link that's fully contained within a single IB subnet with a P_Key of 0x8000, the MGIDs for the all-router multicast group with group ID 2 [AARCH, IGMP3] are:
FF12:401B:8000::2, for IPv4 in compressed format, and
FF12:601B:8000::2, for IPv6 in compressed format.There's one special case: the IPv4 limited broadcast address "255.255.255.255" [HOSTS] 📣. That address SHALL be mapped to the "broadcast-GID", which is laid out like this:
| 8 | 4 | 4 | 16 bits | 16 bits | 48 bits | 32 bits |
+--------+----+----+----------------+---------+----------+---------+
|11111111|0001|scop|0100000000011011|< P_Key >|00.......0|<all 1's>|
+--------+----+----+----------------+---------+----------+---------+Figure 2
Every MGID used in the IPoIB subnet MUST use the same scop bits as the corresponding broadcast-GID 🌟.
4.1. Broadcast-GID Parameters 💅
The broadcast-GID gets set up with these attributes 🎀:
- P_Key
A "Full Membership" P_Key (where the high-order bit is set to 1) MUST be used, so that all members can communicate with one another 🫶.
- Q_Key
It is RECOMMENDED that a controlled Q_Key be used with the high-order bit set 🔒. The reason is to stop non-privileged software from making up and sending out bogus IP datagrams 🙅♀️.
- IB MTU
The value assigned to the broadcast-GID must not be greater than any physical link MTU spanned by the IPoIB subnet 📏.
The following attributes are required for multicast transmissions, and also for unicast transmissions if an IPoIB link covers more than a single IB subnet 🌐.
- Other parameters
The choice of TClass, FlowLabel, and HopLimit values is implementation dependent 🛠️. But they must take into account the topology of the IB subnets that make up the IPoIB link, so that any two nodes on the same IPoIB link can communicate successfully 🗺️.
An SL also needs to be assigned to the broadcast-GID 🏷️. This SL gets used for all multicast communication in the subnet 📡.
The broadcast-GID's scope bits need to be set based on whether the IPoIB link is confined within one IB subnet or spans multiple IB subnets 🌍. A default of local-subnet scope (i.e., 0x2) is RECOMMENDED 💖. A node might work out which scope bits to use by interactively searching for a broadcast-GID of ever-greater scope, starting with local scope 🔍. Or an implementation might let the scope bits be set as a configuration parameter ⚙️.
5. Setting Up an IPoIB Link ✨
okay bestie, the broadcast-GID from the last part MUST be set up before an IPoIB subnet can even exist 🫶 Every IPoIB interface MUST "FullMember" join the IB multicast group that the broadcast-GID defines. From here on we call that group the broadcast group 📣 When a node joins, it gets back the MTU, the Q_Key, and a bunch of other params tied to the broadcast group, and then it hooks those up to its IPoIB interface 🔥 The broadcast group is also basically the link-layer broadcast service for stuff like ARP, net-directed, subnet-directed, and all-subnets-directed broadcasts in IPv4 over IB networks 📡
The join only works if the Subnet Manager (SM) decides the joining node can support the MTU registered with the broadcast group [RFC4392], so everyone shares one common link MTU 💅 The SM also makes sure every node in the broadcast-GID has paths to each other, so they can send and receive unicast packets. And it checks that they really form a multicast tree, where a packet from any member gets copied out to every other member 🌳 So the IPoIB link is basically formed by IPoIB nodes joining the broadcast group. There's no physical boundary for the link, the group membership is the boundary 🦋
The P_Key is a config parameter that must be known before the broadcast-GID can be formed. For a node to join a partition, one of its ports must be assigned the relevant P_Key by the SM [RFC4392] 🎀
How the broadcast group gets created and which parameters it gets is up to the implementation and/or whoever administers the IPoIB subnet 🍓 It can be made by the first IPoIB node that boots up, or it can be set up administratively before the subnet goes live. It is RECOMMENDED that the creation and deletion of the broadcast group be under administrative control 🙌
InfiniBand multicast management, which covers creating, joining, and leaving IB multicast groups, is described in [RFC4392] 📚
6. Frame Format 🎀
Every IP and ARP datagram sent over InfiniBand gets a 4-octet encapsulation header slapped on the front 🎁 Here's the layout:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
| Type | Reserved |
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 3The Reserved field MUST be set to zero when you send, and ignored when you receive, unless a future document says otherwise 🙅♀️
The Type field SHALL tell you which protocol is inside, based on this table 💖
+----------+-------------+
| Type | Protocol |
|------------------------|
| 0x800 | IPv4 |
|------------------------|
| 0x806 | ARP |
|------------------------|
| 0x8035 | RARP |
|------------------------|
| 0x86DD | IPv6 |
+------------------------+
Table 1These values come from the ETHER TYPE numbers that the Internet Assigned Numbers Authority (IANA) [IANA] assigns. Other network protocols, identified by different ETHER TYPE values, may use this encapsulation format too, but that's outside the scope of this document 🧚
|<------ IB Frame headers -------->|<- Payload ->|<- IB trailers ->|
+-------+------+---------+---------+-------------+---------+-------+
|Local | |Base |Datagram | 4-octet | | |
|Routing| GRH* |Transport|Extended | header |Invariant|Variant|
|Header |Header|Header |Transport| + | CRC | CRC |
| | | |Header | IP/ARP | | |
+-------+------+---------+---------+-------------+---------+-------+
Figure 4Figure 4 shows the IB frame wrapping an IP/ARP datagram 📦 The InfiniBand specification requires the use of Global Routing Header (GRH) [RFC4392] when multicasting, or when an InfiniBand packet goes from one IB subnet to another through an IB router. Its use is optional for unicast between nodes within the same IB subnet 🌸 The IPoIB implementation MUST be able to handle packets received with or without GRH 💜
7. Maximum Transmission Unit 🌈
Let's get the MTU terms straight first 📏
- IB MTU: The IB components, meaning IB links, switches, Channel Adapters (CAs), and IB routers, may support maximum payloads of 256, 512, 1024, 2048, or 4096 octets. The maximum IB payload supported by the IB components in any IB path is the IB MTU for that path.
- IPoIB-Link MTU: This is the MTU value associated with the broadcast group. It can be set to any value up to the smallest IB MTU supported by the IB components that make up the IPoIB link.
To cut down on problems with fragmentation and path-MTU discovery, this document requires that all IPoIB implementations support an MTU of 2044 octets. That's a 2048-octet IPoIB-link MTU minus the 4-octet encapsulation overhead 🫧 Larger and smaller MTUs MAY be supported, subject to other existing MTU requirements [IPV6], but the default configuration must support an MTU of 2044 octets 🍰
8. IPv6 Stateless Autoconfiguration 🧚
IB architecture gives each port an EUI-64 identifier called the Globally Unique Identifier (GUID) [RFC4392, IBTA] 🆔 The Local Identifier (LID) is unique only within an IB subnet.
The interface identifier can come from any of these:
- The EUI-64-compliant GUID assigned by the manufacturer.
- If the IPoIB subnet is fully contained within an IB subnet, any of the unique 16-bit LIDs of the port associated with the IPoIB interface.
Heads up about LIDs! The LID value of a port may change after a reboot or power-cycle of the IB node. So if you want a persistent value, it's prudent not to use the LID to form the interface identifier 🙈
On the flip side, the LID isn't globally unique and it can change over time, so it can help create a more anonymous IPv6 address 🕶️
It is RECOMMENDED that the link-local address be constructed from the port's EUI-64 identifier, as given below 💅
[AARCH] requires that the interface identifier be created in the "Modified EUI-64" format when it's derived from an EUI-64 identifier. [IBTA] is unclear about whether the GUID should use IEEE EUI-64 format or the "Modified EUI-64" format. So when creating an interface identifier from the GUID, an implementation MUST do the following:
- Determine if the GUID is a modified EUI-64 identifier (the "u" bit is toggled) as defined by [AARCH].
- If the GUID is a modified EUI-64 identifier, then the "u" bit MUST NOT be toggled when creating the interface identifier.
- If the GUID is an unmodified EUI-64 identifier, then the "u" bit MUST be toggled in compliance with [AARCH].
8.1. IPv6 Link-Local Address 🔗
The IPv6 link-local address for an IPoIB interface is formed as described in [AARCH], using the interface identifier from the previous section ✨
9. Address Mapping - Unicast 🗺️
Address resolution in IPv4 subnets is done through the Address Resolution Protocol (ARP) [ARP]. In IPv6 subnets, it's done using the Neighbor Discovery protocol [DISC] 🫶
9.1. Link Information 💡
An InfiniBand packet over UD mode has several headers, including the LRH (local route header), GRH (global route header), BTH (base transport header), and DETH (datagram extended transport header), as shown in figure 4 and specified in the InfiniBand architecture [IBTA]. All of these headers together make up the link layer of an IPoIB link 🧱
The parameters in these IBA headers are the link-layer information that must be determined before an IP packet can be sent across the IPoIB link. Here's what needs to be sorted out 👀
a) LID
The LID is always needed. A packet always includes an LRH aimed at the remote node's LID, or at an IB router's LID to get to a remote node in another IB subnet 🎯
b) Global Identifier (GID)
The GID isn't needed when exchanging information within an IB subnet, though it may be included in any packet. It's an absolute necessity when transmitting across IB subnets, since IB routers use the GID to forward the packets correctly. The source and destination GIDs are fields included in the GRH. A GID formed from the GUID can be used to unambiguously identify an endpoint 🔍
c) Queue Pair Number (QPN)
Every unicast UD communication is always directed to a particular queue pair (QP) at the peer 💌
d) Q_Key
A Q_Key is associated with each Unreliable Datagram QPN. Received packets must contain a Q_Key that matches the QP's Q_Key to be accepted 🔐
e) P_Key
Two IB nodes can communicate successfully using UD mode only if they have compatible P_Keys. This is referred to as being in the same partition [IBTA] 🎉
f) SL
Every IBA packet contains an SL value. A path in IBA is defined by the three-tuple (source LID, destination LID, SL). The SL is in turn mapped to a virtual lane (VL) at every CA and switch that sends or forwards the packet [RFC4392]. Multiple SLs may be used between two endpoints to provide for load balancing. SLs may be used to provide a Quality of Service (QoS) infrastructure, or may be used to avoid deadlocks in the IBA fabric 🚦
There's one more auxiliary piece of information, which isn't included in the IBA headers 👇
g) Path rate
IBA defines multiple link speeds. A higher-speed transmitter can swamp switches and CAs. To avoid that congestion, every source transmitting at greater than 1x speeds is required to determine the "path rate" before the data may be transmitted [IBTA] 🏎️
9.1.1. Link-Layer Address/Hardware Address ✨
okay so the list of info you need for a successful IPoIB packet is literally huge 😭 but not all of it has to be figured out during the IP address resolution process 🧚
the 20-octet IPoIB link-layer address is the same format whether it's in the source/target link-layer address option in IPv6 or the "hardware address" in IPv4/ARP 💅 here's how it's laid out:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved | Queue Pair Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ +
| |
+ GID +
| |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 5a) Reserved Flags 🌸 these 8 bits are reserved for future use. They MUST be set to zero on send and ignored on receive, unless a future document says otherwise.
b) QPN 🦋 every unicast communication in IB architecture gets directed to a specific QP [IBTA]. This QP number is included in the link description. ALL IP communication to the relevant IPoIB interface MUST be directed to this QPN. For IPv4 subnets, the Address Resolution Protocol (ARP) reply packets get sent to the same QPN too. Picking the QPN for IP/ARP communication is up to the implementation, so no drama there.
c) GID 🍓 this is one of the GIDs of the port tied to the IPoIB interface [IBTA]. IB can link multiple GIDs to one port. It is RECOMMENDED that the GID made from the IB subnet prefix plus the port's "Port GUID" [IBTA] gets included in the link-layer/hardware address.
9.1.2. Auxiliary Link Information
the rest of the parameters get sorted out like this 💜
a) LID the way you figure out the peer's LID isn't defined in this document. The implementation can use any IBA-approved method to find the destination LID. One option is to take the GID you got during address resolution and use it to look up the matching LID from the IB routing infrastructure or the Subnet Administrator (SA).
it's the administrator's job to make sure the IB subnet(s) have unicast connectivity between the IPoIB nodes 🙈 a GID swapped in a multicast message (ARP/ND) doesn't guarantee a unicast path between the two, so don't assume it does.
there may be multiple LIDs, and therefore multiple paths, between the endpoints. How you pick among them is beyond the scope of this document.
b) Q_Key the Q_Key you receive on joining the broadcast group MUST be used for all IPoIB communication over that particular IPoIB link. non-negotiable bestie 🔥
c) P_Key the P_Key for the IP subnet isn't discovered. It's a configuration parameter instead.
d) SL how you determine the SL isn't defined in this document. Any IBA-approved method will do ✨
e) Path rate the implementation must leverage IB methods to determine the path rate as required 💅
9.2. Address Resolution in IPv4 Subnets
the ARP packet header is defined in [ARP] 🎀 the hardware type gets set to 32 (decimal), as specified by IANA [IANA], and the rest of the fields follow [ARP]. the layout looks like this:
16 bits: hardware type
16 bits: protocol
8 bits: length of hardware address
8 bits: length of protocol address
16 bits: ARP operationthe remaining fields in the packet hold the sender/target hardware and protocol addresses:
[ sender hardware address ]
[ sender protocol address ]
[ target hardware address ]
[ target protocol address ]the hardware address in the ARP packet is whatever section 9.1.1 and figure 5 describe 🦋 which means the length of the hardware address in the ARP packet header is 20. obsessed with how neat that is 🥹
9.3. Address Resolution in IPv6 Subnets
the Source/Target Link-layer address option shows up in Router Solicit, Router advertisements, Redirect, Neighbor Solicitation, and Neighbor Advertisement messages, but only when those messages are sent over InfiniBand networks 🧚 the option is set up like this:
Type:
Source Link-layer address 1
Target Link-layer address 2
Length: 3Link-layer address: this is the link-layer address from section 9.1.1, as shown in figure 5 🌈
[DISC] says the option length is counted in 8-octet units, which is why the length is '3' above. Since the IPoIB link-layer address is only 20 octets long, two octets of zero MUST be prepended to fill the full option length of 24 octets. that's the padding moment 🙈
9.4. Cautionary Note on QPN Caching
⚠️ the link-layer address for IPoIB includes the QPN, and that QPN might not stay constant across reboots or even across network interface resets. Cached QPN entries (like the ones in static ARP entries or in Reverse Address Resolution Protocol (RARP) servers) will only work if the implementations using them make sure the QPN for an interface stays invariant across reboots/network resets.
It is RECOMMENDED that implementations revalidate ARP caches periodically, because of that QPN-induced volatility in IPoIB link-layer addresses. stale caches are not it, bestie 💀
10. Sending and Receiving IP Multicast Packets 🌈
multicast in InfiniBand is different from multicast on ethernet in a bunch of ways, and that adds some complexity when an IPoIB implementation supports IP multicast over IB 😭
A) IB multicast groups have to be made first. an IB multicast group must be explicitly created through the SA before it can be used.
so to send a packet to an IP multicast address, the IPoIB implementation has to check with the SA on the outbound link first for an "MCMemberRecord" that matches the MGID 👑 if one exists, the Multicast Local Identifier (MLID) for that group becomes the Destination Local Identifier (DLID) for the packet. if not, that means no member exists on the local link. If the IP multicast group's scope goes beyond link-local, the packet must be sent to the on-link routers using the all-router multicast group or the broadcast group. This lets local routers forward the packet to multicast listeners on remote networks. The all-router multicast group is preferred over the broadcast group for better efficiency. If the all-router multicast group doesn't exist, the sender can assume there are no routers on the local link, so the packet can be safely dropped.
B) senders have to join first. a multicast sender must join the target multicast group before outgoing multicast messages from it can be successfully routed. 💅 the "SendOnlyNonMember" join is different from the regular "FullMember" join in two ways. First, both types of joins let multicast packets be routed FROM the local port, but only the "FullMember" join lets multicast packets be routed TO the port. Second, the sender port of a "SendOnlyNonMember" join doesn't count as a member of the multicast group for purposes of group creation and deletion.
here's the typical egress flow when a multicast packet is being sent 🦋
if the egress port is already a "SendOnlyNonMember", or a
"FullMember"
=> send the packet
else if the target multicast group exists
=> do "SendOnlyNonMember" join
=> send the packet
else if scope > link-local AND the all-router multicast group exists
=> send the packet to all routers
else
=> drop the packetimplementations should cache info about whether an IB multicast group exists, its MLID, and its other attributes 🧠 that way they don't have to make expensive SA calls on every outgoing multicast packet. Senders MUST subscribe to the multicast group create and delete traps so they can keep an eye on the status of specific IB multicast groups. For example, multicast packets sent to the all-router multicast group because nobody was listening on the local subnet must get forwarded to the right multicast group if that group gets created later. This happens when a listener shows up on the local subnet ✨
a node joining an IP multicast group must first construct an MGID according to the rule described in section 4 above. Once the correct MGID is calculated, the node must call the SA of the outbound link to attempt a "FullMember" join of the IB multicast group that corresponds to the MGID. If the IB multicast group doesn't already exist, one must be created first with the IPoIB link MTU. The MGID MUST use the same P_Key, Q_Key, SL, MTU, and HopLimit as the ones used in the broadcast-GID. The rest of the attributes SHOULD follow the values used in the broadcast-GID as well 🎀
the join request adds the local port to the multicast group. It also lets the SM program IB switches and routers with the new multicast info, so multicast packets for the group get forwarded correctly 🦋
when a node leaves an IP multicast group, it SHOULD make a "FullMember" leave request to the SA 🫶 that gives the SM a chance to update the relevant forwarding info, delete an IB multicast group if the local port was the last FullMember to leave, and free up the MLID that was allocated for it. The specific algorithm is implementation-dependent and out of scope for this document.
one more thing 👀 for an IPoIB link that spans more than one IB subnet connected by IB routers, adequate multicast forwarding support at the IB level is required for multicast packets to reach listeners on a remote IB subnet. The specific mechanism for this is beyond the scope of IPoIB.
11. IP Multicast Routing ✨
okay so IP multicast routing needs every interface the router is running on to listen to ALL the link-layer multicast addresses that IP generates [IPMULT, IP6MLD] 🎀. On an Ethernet interface, people usually get this by turning on promiscuous multicast mode 🦋
IBA doesn't give you any hardware support for promiscuous multicast mode 💀 but literally no worries, bestie, because you can fake it in software on the router with these steps:
A) Grab a list of all the active IB multicast groups from the local SA ✨
B) Send a "NonMember" join request to the SA for every group whose MGID has a signature matching the one for IPv4 or IPv6 🌸
C) Subscribe to IB multicast group creation events using a wildcarded MGID, so the router can "NonMember" join every new IB multicast group created for IPv4 or IPv6 later on 🧚
The "NonMember" join does the same thing as a "FullMember" join, with ONE difference: it doesn't count as a member of the group when it comes to creating or deleting the group 👑 So when the last "FullMember" leaves a multicast group, the SA can safely delete the group without worrying about any "NonMember" routers 💅
12. New Types of Vulnerability in IB Multicast 🥀
Lots of IB multicast functions can fail because of different resource limits 😭 That includes creating IB multicast groups, the join calls ("SendOnlyNonMember", "FullMember", and "NonMember"), and attaching a QP to a multicast group 🫶
Honestly, how these failures show up is very implementation-dependent, and they're believed to be rare ✨ Usually a failed multicast operation at the IB level can get passed back up to the IP level, so the original operation fails and whoever started it gets notified 🔥 BUT some IB multicast functions aren't tied to any foreground operation, so their failures are hard to spot 🙈 For example, if an IP multicast router tries to "NonMember" join a brand new multicast group in the local subnet and that join call fails, packet forwarding for that group will likely fail silently, meaning local multicast senders won't even notice 💀 This kind of problem adds even more vulnerability to IP multicast operations that were already unreliable 😩
Implementations SHOULD log error messages upon any failure from an IB multicast operation. 🚨
Network administrators should be aware of this vulnerability and keep enough multicast resources around wherever IP multicast is going to get used heavily 🌈 For example, HCAs with plenty of multicast resources should be used at any IP multicast router 👑
13. Security Considerations 🔐
Okay, this part is SERIOUS, so pay attention 💜 This document specifies IP transmission over a multicast network. Any network like this is vulnerable to a sender pretending to be someone else, forging traffic, or eavesdropping 😱 It's the responsibility of the higher layers or applications to build suitable countermeasures if that's a problem for them 🛡️
Successful IP packet transmission depends on getting a bunch of things right: setting up the IPoIB link, creating the broadcast-GID, creating the QP and attaching it to the broadcast-GID, and figuring out link parameters like the LID, service level, and path rate correctly ✨ Lots of these operations involve talking to the SM/SA, and they MUST be protected by the underlying operating system 🚨 That stops malicious, non-privileged software from hijacking important resources and configurations, and nobody wants that drama 💀
Controlled Q_Keys SHOULD be used in all transmissions. This is to stop non-privileged software from fabricating IP datagrams. 🧿
14. IANA Considerations 🍓
So, for ARP over InfiniBand to work, the Address Resolution Parameter "Number Hardware Type (hrd)" needs a value 🦋 IANA already assigned the number "32" to mean InfiniBand [IANA_ARP] ✨
Future uses of the reserved bits in the frame format (Figure 3) and the link-layer address (Figure 5) MUST be published as RFCs 📜 Also, this document requires that reserved bits get set to zero on send and ignored on receive 🌸
15. Acknowledgements 🫶
The authors want to give a huge shoutout to Bruce Beukema, David Brean, Dan Cassiday, Aditya Dube, Yaron Haviv, Michael Krause, Thomas Narten, Erik Nordmark, Greg Pfister, Jim Pinkerton, Renato Recio, Kevin Reilly, Kanoj Sarcar, Satya Sharma, Madhu Talluri, and David L. Stevens 👑 Thank you for the suggestions and all the clarifications on the IBA specification 💖
16. References 📚
16.1. Normative References 💅
[AARCH] Hinden, R. and S. Deering, "Internet Protocol Version 6
(IPv6) Addressing Architecture", RFC 3513, April 2003.
[ARP] Plummer, David C., "Ethernet Address Resolution
Protocol: Or converting network protocol addresses to
48.bit Ethernet address for transmission on Ethernet
hardware ", STD 37, RFC 826, November 1982.
[DISC] Narten, T., Nordmark, E., and W. Simpson, "Neighbor
Discovery for IP Version 6 (IPv6)", RFC 2461, December
1998.
[IANA] Internet Assigned Numbers Authority, URL
http://www.iana.org
[IANA_ARP] URL http://www.iana.org/assignments/arp-parameters
[IBTA] InfiniBand Architecture Specification, URL
http://www.infinibandta.org/specs
[RFC4392] Kashyap, V., "IP over InfiniBand (IPoIB) Architecture",
RFC 4392, April 2006.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.16.2. Informative References 🦋
[HOSTS] Braden, R., "Requirements for Internet Hosts -
Communication Layers", STD 3, RFC 1122, October 1989.
[IGMP3] Cain, B., Deering, S., Kouvelas, I., Fenner, B., and A.
Thyagarajan, "Internet Group Management Protocol,
Version 3", RFC 3376, October 2002.
[IP6MLD] Deering, S., Fenner, W., and B. Haberman, "Multicast
Listener Discovery (MLD) for IPv6", RFC 2710, October
1999.
[IPMULT] Deering, S., "Host extensions for IP multicasting", STD
5, RFC 1112, August 1989.
[IPV6] Deering, S. and R. Hinden, "Internet Protocol, Version 6
(IPv6) Specification", RFC 2460, December 1998.Authors' Addresses 🎀
H.K. Jerry Chu
17 Network Circle, UMPK17-201
Menlo Park, CA 94025
USA
Phone: +1 650 786 5146
EMail: jerry.chu@sun.com
Vivek Kashyap
15350, SW Koll Parkway
Beaverton, OR 97006
USA
Phone: +1 503 578 3422
EMail: vivk@us.ibm.comFull Copyright Statement 🌈
Copyright (C) The Internet Society (2006) ✨
This document is subject to the rights, licenses and restrictions in BCP 78, and except for what's set out there, the authors keep all their rights 💜
This document and the info in it are provided on an "AS IS" basis, and the disclaimer below applies in full:
THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 💅
Intellectual Property 🔥
The IETF takes no position on whether any Intellectual Property Rights or other rights claimed to cover the tech in this document are valid or in scope, or whether a license for them might be available 🙈 It doesn't claim to have done any independent work to find those rights either 🫶 For the procedures around rights in RFC documents, check BCP 78 and BCP 79 📜
Copies of IPR disclosures sent to the IETF Secretariat, any assurances of licenses, and the results of any attempt to get a general license or permission for implementers or users are all available from the IETF's online IPR repository at http://www.ietf.org/ipr ✨
The IETF invites any interested party to tell it about any copyrights, patents, patent applications, or other proprietary rights that might cover technology needed to implement this standard 🧚 Send that info to the IETF at ietf-ipr@ietf.org 💖
Acknowledgement 💜
Funding for the RFC Editor function comes from the IETF Administrative Support Activity (IASA) 🌸
📦 where this page comes from
- ietf: rfc4391 2006-04
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms