RFC 4755

RFC 4755 is IP over InfiniBand: Connected Mode 🦋 by V. Kashyap of IBM, a Standards Track doc from December 2006. It doesn't list anything it obsoletes or updates, so there's no lineage drama here. ✨

Status of This Memo

This doc is an Internet standards track protocol for the Internet community 💖 and it's openly asking for discussion and suggestions for improvements. For the current standardization state and status of this protocol, check the current edition of the "Internet Official Protocol Standards" (STD 1). Distribution of this memo is unlimited 🌈.

Copyright Notice

Copyright (C) The IETF Trust (2006). 🎀

Abstract

This doc specifies how IPv4/IPv6 packets get transmitted and how address resolution works over the connected modes of InfiniBand 🧚.

Table of Contents

1. Introduction ....................................................2
2. IPoIB-connected Mode ............................................3
   2.1. Multicasting ...............................................3
   2.2. Outline of Address Resolution ..............................4
   2.3. Outline of Connection Setup ................................4
3. Address Resolution ..............................................4
   3.1. Link-layer Address .........................................4
   3.2. IB Connection Setup ........................................6
   3.3. Simultaneous IB Connections ................................6
   3.4. IPoIB-CM IB Connection Teardown ............................7
   3.5. Service-ID .................................................7
4. Frame Format ....................................................8
5. Maximum Transmission Unit .......................................8
   5.1. Per-Connection MTU .........................................9
6. Private-Data Format .............................................9
7. IPoIB-CM Considerations ........................................10
   7.1. A Cautionary Note on IPoIB-RC .............................10
   7.2. IPoIB-CM Per-Destination MTU ..............................10
8. Security Considerations ........................................11
9. IANA Considerations ............................................11
10. Acknowledgements ..............................................11
11. Normative References ..........................................11
12. Informative References ........................................11

1. Introduction ✨

okay so the InfiniBand spec [IB_ARCH] lives at www.infinibandta.org 💅. The doc [RFC4392] gives a short overview of InfiniBand architecture and also looks at what it'd take to specify IP over InfiniBand networks 🫶.

The InfiniBand Architecture (IBA) has multiple modes of transports, and the unreliable datagram (UD) one is the best match for what IP needs 😭. IP over InfiniBand (IPoIB) over UD is described in [RFC4391]. This doc is all about IP transmission over the connected modes of IBA instead 🔥.

IBA has two connected modes:

1. Reliable Connected (RC)
2. Unreliable Connected (UC)

As the names suggest, the two modes mostly differ in how reliable data delivery across the connection is 💀. This doc applies equally to both connected modes. IPoIB over these two modes is called IPoIB-CM (connected mode) in this doc, and for clarity, IPoIB over the unreliable datagram mode as described in [RFC4391] is called IPoIB-UD 🌸.

IBA requires that all Host Channel Adapters (HCAs) support the reliable and unreliable connected modes [IB_ARCH]. It is optional for Target Channel Adapters (TCAs) to support the connected modes 🍓.

The connected modes offer link MTUs of up to 2^31 octets in length 👑. So using the connected modes can offer significant benefits by supporting reasonably large MTUs. The datagram modes of InfiniBand Architecture (IBA) are limited to 4096 octets 🙈.

Reliability also gets a boost if you use the underlying feature of "automatic path migration", which the connected modes support ⭐.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119] 🎀.

2. IPoIB-connected Mode

IPoIB over connected mode is an OPTIONAL extension to IPoIB-UD. Every IPoIB implementation MUST support [RFC4391] and MAY support the extensions described in this doc 💖.

So IP encapsulation, default MTU, link-layer address format, and the IPv6 stateless autoconfiguration mechanism apply to IPoIB-CM exactly as described in [RFC4391]. No changes, no drama 💅.

2.1. Multicasting

The connected modes of IBA define a non-broadcast, multiple-access network. They don't support multicasting, though every node can still talk to every other node if it wants to 🦋.

That means multicasting has to be emulated in some form by the network. But with InfiniBand, you don't have to emulate anything: an unreliable datagram (UD) queue pair (QP) can handle multicasting while the connected mode QP handles unicast traffic ✨. Since every IPoIB implementation is required to support the UD mode, every implementation supporting IPoIB-CM can use the pre-existing IPoIB-UD QP for all broadcast/multicast communications 🧚. Multicast mapping, transmission, and reception of multicast packets, and multicast routing MUST use the UD QP associated with the IPoIB interface 🔥.

2.2. Outline of Address Resolution

Every IPoIB-CM interface MUST have two sets of QPs associated with it:

1. One unreliable datagram QP
2. One or more connected mode QPs

[RFC4391] describes the address resolution method for figuring out the link address of the peer. The response comes back on the UD QP associated with the IPoIB interface 🫶.

2.3. Outline of Connection Setup

Once you know the link address of the remote node, an IB connection has to be set up between the nodes before any IP communication can happen 🌈.

To make a connection, the sender must know the service-ID to use in the request to make a connection [IB_ARCH]. It also has to supply the "connection mode" queue pair to the remote node. The peer replies with its own queue pair. Each IB connection is peer to peer and uses one connected mode QP at each end 💜.

Address resolution happens at the individual IP address level, but the connection between the nodes is at the IB layer. So every individual address resolution doesn't mean a brand new connection between the peers 🌸.

3. Address Resolution

Address resolution queries go out on the "broadcast-GID" (Broadcast-Group Identifier) over the UD QP associated with the IPoIB interface [RFC4391]. A unicast reply comes back on the UD QP 💅.

3.1. Link-layer Address

IPoIB encapsulation [RFC4391] describes the link-layer address like this:

<1 octet reserved>:QP: GID

This doc extends the link-layer address like this:

<Flags>:QPN:GID

Flags:

This is a single-octet field 🍓. The bits show which connected modes the interface supports.

Bit 0 specifies support for the "reliable connected" (RC) mode. Bit 1 indicates support for the "unreliable connected" (UC) mode. All other bits in the octet are reserved and MUST be set to 0 on transmits and ignored on receives. The format of the flags looks like this:

          +--+--+--+--+--+--+--+--+
          |RC|UC| 0| 0| 0| 0| 0| 0|
          +--+--+--+--+--+--+--+--+

Both the RC and UC MAY be set at the same time if the interface supports both modes 🙈. Since the IPoIB-UD mode is always supported, there are no flags to indicate IPoIB-UD support.

If IPoIB-CM is not supported, i.e., if the implementation only supports IPoIB-UD, then the implementation MUST ignore the <Flags> on reception. It MUST set the <Flags> octet to all zeros on transmission as specified in [RFC4391] 👑.

QPN:

This is the queue-pair number (QPN) on which the unicast address resolution replies will be received [RFC4391]. An IPoIB interface has only one UD QP associated with it, whether or not it supports this extension ✨.

The QPN also has a second job. It's used to form the Service-ID that sets up the IB connection 🧚.

On receiving the multicast/broadcast address resolution request, the receiver replies with its own link address, including the associated UD QPN and the appropriate flags 🌈.

The receiver's reply is unicast back to the sender after the receiver has, just like in IPoIB-UD, resolved the GID to the Local Identifier (LID) and determined other required parameters [RFC4391]. Once address resolution is done, the underlying IB connection on the supported connection modes can be set up. An implementation is NOT REQUIRED to set up a connection merely because the peer indicates the capability. The decision to make such a connection is left to the implementation 💖.

3.2. IB Connection Setup

Once address resolution is complete, either peer can set up the IB connection 🦋. To do that, IB Management Datagrams (MADs) are directed to the peer's communication manager (CM). The connection request always contains a Service-ID so the peer can associate the request with the right service. If the request is accepted, the peer returns the relevant connected mode QPN in the response MAD. The format of the CM connection messages and the IB connection setup process is described in [IB_ARCH]. The overall handshake looks like this:

          REQ ---->
               <---- REP [or REJ(reject)]
          RTA ---->
          [or REJ(reject)]

The CM messages include, among other parameters, the Service-ID, Local connection-mode QPN, and the payload size to use over the connection 🫶.

Note: The IB connection is set up using the Service-ID as defined in Section 3.5 below. The node MUST keep a record of IB connections it is participating in. The node MAY attempt another connection to the remote peer using the same Service-ID as used for an existing IB connection. Similarly, the receiver of such a connection MAY drop the request with a suitable error indication in the CM response. The decision to accept or initiate multiple connections from or to an IPoIB interface is left to the implementation 🔥.

The node that started the connection already knows the target node's IP address, as described above. But the node receiving the IB connection request can't determine the initiating node's link address. To make that possible, every CM message exchanged while setting up the IB connection MUST include the sender's IPoIB-UD QPN in the "private data" [IB_ARCH] field 😭. The IPoIB-UD QPN MUST also be included in all "REJ" [IB_ARCH] messages 💜.

Part 1 of 3 is done, bestie 🌈✨ Part 2 picks up right after this, so stay tuned 💅🎀

3.3. Simultaneous IB Connections ✨

okay so, bestie, this is the nightmare where two IB connections get set up between the same two peers because their REQ messages cross paths 💀 To stop that, these rules MUST be followed:

• The receiver builds the remote node's link-layer address from the UD QPN it got in the "private data" field of the "REQ" message, plus the sender's GID from that same "REQ" message. It then checks whether the local interface already has an outstanding connection request "REQ" sent to that link-layer address. If it does, the two link-layer addresses (local and remote) get compared numerically. If the local one is numerically smaller, the connection is accepted, and otherwise it's rejected. The error code in the "REJ" MAD is set to "Consumer Reject" [IB_ARCH].
  • Note: The link-layer addresses formed for comparison zero out the connection mode flags specified in Section 3.1. The comparison runs from the most significant octet down to the least significant octet of the link-layer address.
• The above still holds even if the receiver supports multiple IB connections from the same peer. This makes sure that only one more connection gets set up when the "REQ" messages cross. Crossing REQs, handled 💅

3.4. IPoIB-CM IB Connection Teardown ✨

IB connections created through IPoIB-CM are considered part of an IPoIB interface, so they SHOULD be torn down when the IPoIB interfaces they're attached to get torn down. It's giving tidy house rules 🧹

Also, either peer MAY tear down the IB connection between two peers whenever the address resolution entry expires. An implementation is free to come up with its own alternative policies for tearing down IB connections between peers. Totally up to you, bestie 🦋

3.5. Service-ID ✨

The InfiniBand spec sets aside a block of Service-IDs for IETF use, and it has left the definition and management of that block to the IETF [IB_ARCH]. The 64-bit block looks like this:

  +--------+--------+--------+--------+-------+--------+--------+------+
  |00000001|<-------------------IETF use------------------------------>|
  +--------+--------+--------+--------+-------+--------+--------+------+

Service-IDs used by IPoIB follow this format:

  +--------+--------+--------+--------+-------+-------+--------+-------+
  |00000001|  Type  |         Reserved        |        QPN             |
  +--------+--------+--------+--------+-------+-------+--------+-------+

• The "Type" field MUST be set to 0.
• The "Reserved" field MUST be set to zeros.
• The QPN MUST be the UD QP exchanged during address resolution. the QPN is the main character here 🧚

4. Frame Format ✨

All IP datagrams transported over InfiniBand get prefixed by a 4-octet encapsulation header, as described in [RFC4391]. 🎀

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                               |                               |
  |         Type                  |       Reserved                |
  |                               |                               |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The type field SHALL indicate the encapsulated protocol, following this table:

                 +----------+-------------+
                 | Type     |    Protocol |
                 |------------------------|
                 | 0x800    |    IPv4     |
                 |------------------------|
                 | 0x86DD   |    IPv6     |
                 +------------------------+

These values come from the "ETHER TYPE" numbers assigned by the Internet Assigned Numbers Authority (IANA). Other network protocols, identified by different values of "ETHER TYPE", may use the encapsulation format defined here, but using it that way is outside the scope of this document. 🌸

5. Maximum Transmission Unit ✨

The IB connection setup might be used for both IPv4 and IPv6, or it could be used for only one of them while a different connection handles the other. The link MTU MUST be able to support the minimum MTU required by the protocols. no notes, just vibes 💅

The default MTU of the IPoIB-CM interface is 2044 octets, i.e., the 2048-octet IPoIB-link MTU minus the 4-octet encapsulation header. 🍓

However, connected modes of InfiniBand allow message sizes up to 2^31 octets. So IPoIB-CM can use a much larger MTU for unicast communication between any two endpoints. The maximum and/or optimal payload that can be sent or received over an InfiniBand connection depends on the implementation, the IB Channel Adapter, and the configured resources. 🚀

An implementation MAY utilize the following mechanism to exchange the optimal message size across the IB connection.

5.1. Per-Connection MTU ✨

Every IB connection setup message includes a "private data" field [IB_ARCH]. The "private data" field in the connection setup message (CM REQ) MUST include the "Receive MTU". This tells the other side the maximum packet size the requester can accept. The requester MUST be able to accept smaller MTU sizes as well. 🫶

It's up to the implementation to use this mechanism for setting the per-IB connection MTU. To work out the resulting IPoIB MTU over the connection, the smaller of the two IB "Receive MTU" values is used by both peers. The IPoIB interface must also account for the 4-octet encapsulation header, so the IPoIB MTU over the connection gets reduced by that amount too. math is my love language 🧮

6. Private-Data Format ✨

The "private data" field in every CM message for connection establishment must include the following values:

1. UD QPN of the sender
2. Receive MTU supported by the sender

The format of the "private data" field MUST be as follows:

          0        7       15       23       31
          +--------+--------+--------+--------+
          |Reserved|         UD QPN           |
          +--------+--------+--------+--------+
          |            Receive MTU            |
          +--------+--------+--------+--------+

The Reserved value MUST be set to zero on transmit and ignored on receive. 🎀

7. IPoIB-CM Considerations ✨

Every IPoIB interface supports IPoIB-UD. It may additionally support one or both of the IPoIB-CM modes, so there can be multiple ways of communicating between any two peers. That means an interface MAY transmit/receive a packet over any of the RC, UC, or UD modes, depending on which modes are supported between it and the peer. It also follows that every IPoIB implementation compliant with this document MUST accept all IP unicast transmissions over any of the IPoIB modes it supports. Multicast and broadcast packets, by their nature, will always be transmitted and received over the IPoIB-UD QP. Additionally, all address resolution responses (ARP or Neighbor Discovery) MUST always be encapsulated in a UD mode packet. 🦋

7.1. A Cautionary Note on IPoIB-RC ✨

💀 Read this one carefully, bestie. The RC mode of InfiniBand guarantees in-order delivery of packets.

The RC connection breaks every message into physical MTU-sized packets. If any packet gets lost, it's retransmitted until the complete message has been exchanged. So there's a possibility that an upper transport layer hits a timeout while the RC layer is still busy transferring the complete message. TCP will read that timeout as a sign of congestion and enter slow-start, which can tank throughput drastically [RFC2581]. Other upper-layer protocols might insert retransmissions into the fabric, piling onto congestion that's already there. 😭

The reliability of InfiniBand only really applies to a fabric with short latencies, not a wide area one. So the RC timer values should be short compared with the starting minimum time values used by the upper end-to-end transports. Also, because the RC mode doesn't have measurement-based reliable transmission, using it over fabrics with long or very dynamic latency may be a concern for congestion-aware traffic traversing those fabrics. not the vibe for long-haul links 🙈

7.2. IPoIB-CM Per-Destination MTU ✨

As described above, interfaces on the same subnet may support different link MTUs, based on the negotiated value or on the link type (UD or connected mode). So an implementation might choose to define a large IP MTU, which then gets reduced based on the MTU to the destination. The relevant MTU may be stored in a suitable per-destination object, such as a route cache or a neighbor cache. The IPoIB-CM interface knows the per-destination MTU as described in Section 5. 🗺️

Implementations might choose not to support differing MTU values and always support an MTU equal to the IPoIB-UD MTU determined from the broadcast GID. Simple and iconic 👑

8. Security Considerations ✨

🚨 Serious business, bestie. An impostor may return a false set of flags to an IPOIB interface. This may cause unnecessary attempts and some delay or disruption in IPoIB communication. The same is the case if wrong or spurious QPN values are provided during address resolution broadcast or multicast. 💀

9. IANA Considerations ✨

Future uses of the reserved bits and octets in the link-layer address (Section 3.1), Service-ID (Section 3.5), and "Private-Data Format" (Section 6) MUST be published as RFCs. This document requires that the reserved bits be set to zero on sends. 📚

10. Acknowledgements ✨

The author thanks the IPoIB Working Group for the various comments and suggestions. A special thanks to Bernie King-Smith and Dror Goldenberg for the detailed review and suggestions. 🫶 the real MVPs 🌈

11. Normative References ✨

[IB_ARCH]    InfiniBand Architecture Specification, version 1.2
             www.infinibandta.org

[RFC4392]    Kashyap, V., "IP over InfiniBand (IPoIB) Architecture",
             RFC 4392, April 2006.

[RFC4391]    Chu, J. and V. Kashyap, "Transmission of IP over
             InfiniBand (IPoIB)", RFC 4391, April 2006.

[RFC2119]    Bradner, S., "Key words for use in RFCs to Indicate
             Requirement Levels", BCP 14, RFC 2119, March 1997.

12. Informative References ✨

[RFC2581]    Allman, M., Paxson, V., and W. Stevens, "TCP Congestion
             Control ", RFC 2581, April 1999.

Author's Address

Vivek Kashyap
15350, SW Koll Parkway
Beaverton
OR 97006

Phone: +1 503 578 3422
EMail: vivk@us.ibm.com

Full Copyright Statement ✨

Here's the legal bit, kept exactly as it is 📜

Copyright (C) The IETF Trust (2006).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST, AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property ✨

okay so, the IETF is not picking a side on whether any Intellectual Property Rights or other rights that someone claims over the tech in this doc are valid, or how far they reach 💅. It also doesn't promise that any license for those rights is actually available, and it hasn't independently gone hunting for any such rights either 🫣. If you want the deets on how rights work for RFC documents, check out BCP 78 and BCP 79 📚.

Copies of IPR disclosures that were sent to the IETF Secretariat 🗂️, any assurances of licenses that are supposed to be made available, and the results of anyone trying to get a general license or permission to use those proprietary rights (for implementers or users of this spec) can all be found in the IETF on-line IPR repository at http://www.ietf.org/ipr 🔍.

The IETF invites any interested party to speak up 📣. If you know about copyrights, patents, patent applications, or other proprietary rights that might cover technology needed to implement this standard, bring it to the IETF's attention 💌. Send that info to ietf-ipr@ietf.org 🦋.

Acknowledgement

Funding for the RFC Editor function is currently provided by the Internet Society 🌈💜. Shoutout to the Internet Society for keeping that whole operation funded, bestie 🫶✨.