RFC 6581

Enhanced Remote Direct Memory Access (RDMA) Connection Establishment

📅 published April 2012, from ietf

Enhanced Remote Direct Memory Access (RDMA) Connection Establishment 💖

Okay bestie, this one is RFC 6581 🎀 It's a Standards Track document from the IETF, published in April 2012, with ISSN 2070-1721 ✨ A. Kanevsky (Ed.) from Dell Inc. and C. Bestler (Ed.) from Nexenta Systems lead the writing, with R. Sharp from Intel and S. Wise from Open Grid Computing helping out 🫶 It updates RFC 5043 and RFC 5044, and those are the only two RFCs the title block lists. No Obsoletes line shows up, so nothing gets replaced 🌸

Abstract

This doc updates RFC 5043 and RFC 5044 by upgrading Marker PDU Aligned Framing (MPA) negotiation so RDMA connections can be set up 🦋 The first upgrade extends RFC 5044 so peer-to-peer connection establishment works over MPA / Transmission Control Protocol (TCP) 👑 The second extends both RFC 5043 and RFC 5044 by giving you an option for a standardized exchange of RDMA-layer connection configuration 🔥

Status of This Memo

This is an Internet Standards Track document 💜

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community 🫶 It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more info on Internet Standards, check out Section 2 of RFC 5741 ✨

Info about the current status of this document, any errata, and how to send feedback is at http://www.rfc-editor.org/info/rfc6581 🧚

Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved 💅

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they're in effect on the date this document was published. Please actually read these, because they spell out your rights and restrictions for this document 🙈 Code Components pulled from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License 📜

Table of Contents

   1. Introduction ....................................................3
      1.1. Summary of Changes Affecting RFC 5044 ......................4
      1.2. Summary of Changes Affecting RFC 5043 ......................4
   2. Requirements Language ...........................................4
   3. Definitions .....................................................4
   4. Motivations .....................................................7
      4.1. Standardization of RDMA Read Parameter Configuration .......7
      4.2. Enabling MPA Mode ..........................................9
      4.3. Lack of Explicit RTR in MPA Request/Reply Exchange ........10
      4.4. Limitations on ULP Workaround .............................11
           4.4.1. Transport Neutral APIs .............................11
           4.4.2. Work/Completion Queue Accounting ...................11
           4.4.3. Host-based Implementation of MPA Fencing ...........12
   5. Enhanced MPA Connection Establishment ..........................13
   6. Enhanced MPA Request/Reply Frames ..............................14
   7. Enhanced SCTP Session Control Chunks ...........................15
   8. MPA Error Reporting ............................................16
   9. Enhanced RDMA Connection Establishment Data ....................17
      9.1. IRD and ORD Negotiation ...................................18
      9.2. Peer-to-Peer Connection Negotiation .......................20
      9.3. Enhanced Connection Negotiation Flow ......................21
   10. Interoperability ..............................................21
   11. IANA Considerations ...........................................22
   12. Security Considerations .......................................23
   13. Acknowledgements ..............................................23
   14. References ....................................................23
      14.1. Normative References .....................................23
      14.2. Informative References ...................................24

1. Introduction ✨

When you're running over the Transmission Control Protocol (TCP), the current Remote Direct Data Placement (RDDP) [RFC5041] suite relies on the MPA [RFC5044] protocol for both connection establishment and for markers for TCP layering 🌈

Here's the usual flow for setting up an RDMA connection 💖

  • The passive side (responder) Upper Layer Protocol (ULP) listens for connection requests.
  • The active side (initiator) ULP submits a connection request using an RDMA endpoint, the desired destination, and the parameters to be used for the connection. Those parameters include both RDMA-layer characteristics, such as the number of simultaneous RDMA Read Requests to be allowed, and application-specific data.
  • The passive side ULP receives a connection request that includes the identity of the active side and the requested connection characteristics. The passive side ULP uses this information to decide whether to accept the connection, and if it is to be accepted, how to create and/or configure the local RDMA endpoint.
  • If accepting, the responder submits its acceptance of the connection request, which in turn generates the accept message to the initiator. This responder accept operation includes the RDMA endpoint to be used and the connection characteristics (both the RDMA configuration and any application-specific Private Data to be transferred to the initiator).
  • The active side receives confirmation that the connection has been accepted, what the configured connection characteristics are, and any application-supplied Private Data.

Right now, MPA only supports a client-server model for connection establishment, which forces peer-to-peer apps to act like they're in a client-server relationship 🙄 On top of that, negotiation of some parameters specific to the Remote Direct Memory Access Protocol (RDMAP) [RFC5040] is left to ULP negotiation. An optional ULP-independent format for exchanging these parameters would help transport neutral RDMA applications 🦋

1.1. Summary of Changes Affecting RFC 5044

This document enhances the MPA connection setup protocol [RFC5044] 🔥 First, it adds exchange and negotiation of the parameters needed to support RDMA Read Requests. Second, it adds a message that works as a Ready to Receive (RTR) indication from the initiator to the responder, sent as the last message of connection establishment. It also adds negotiation of which type of message to use for carrying the RTR indication into MPA Request/Reply Frames 💅

RTR indications are optional and are carried by existing RDMA message types: a zero-length FULPDU Send message, a zero-length RDMA Read message, or a zero-length RDMA write message. Whether the RTR indication is present or absent, and which RDMA message type gets used, are negotiated by control flags in Enhanced RDMA connection establishment data specified by this document (see Section 9) 🫶 RDMA implementations are often tightly integrated with application libraries and hardware, so being able to use more than one type of RDMA message lets implementations pick the one that's least disruptive to their structure ✨ When an RTR indication is used, and MPA connection setup negotiation shows that both the initiator and responder support multiple RDMA message types as RTR indications, the initiator picks one of the supported types at the initiator's sole discretion 👑

1.2. Summary of Changes Affecting RFC 5043

This document enhances [RFC5043] by adding new Enhanced Session Control Chunks. They extend the currently defined Chunks with Inbound RDMA Read Queue Depth (IRD) and Outbound RDMA Read Queue Depth (ORD) negotiation 🌸

2. Requirements Language 🎀

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 💜

3. Definitions ✨

Active Side: See Initiator. 🦋

Consumer: The ULPs or applications that sit above MPA and Direct Data Placement (DDP). The Consumer is responsible for making TCP or Stream Control Transmission Protocol (SCTP) connections, starting MPA and DDP connections, and generally controlling operations. See [RFC5044] and [RFC5043]. 👑

CRC: Cyclic Redundancy Check

Completion Queue (CQ): A Consumer-accessible queue where the RDMA device reports completions of Work Requests. A Consumer can reap completions from a CQ without needing per-transaction support from the kernel or another privileged entity. See [RDMAC]. 💅

Completion Queue Entry (CQE): A transport- and device-specific representation of a Work Completion. A CQ holds CQEs. See [RDMAC]. 🎀

FULPDU: Framed Upper Layer Protocol PDU. See FPDU of [RFC5044]. 🌸

Inbound RDMA Read Request Queue (IRRQ): A queue associated with an RDMA connection that tracks active incoming simultaneous RDMA Read Request Messages. See [RDMAC]. 🧚

Inbound RDMA Read Queue Depth (IRD): The maximum number of incoming simultaneous RDMA Read Request Messages an RDMA connection can handle. See [RDMAC]. 🔥

Initiator: The endpoint of a connection that sends the MPA Request Frame. The initiator is the active side of the connection establishment. See [RFC5044]. 🍓

IRD: See Inbound RDMA Read Queue Depth.

MPA Fencing: MPA responder connection establishment logic that makes sure no ULP messages get transferred until the initiator's first message has been received. 🙈

MPA Request Frame: Data sent from the MPA initiator to the MPA responder during the Startup Phase. See [RFC5044]. ✨

MPA Reply Frame: Data sent from the MPA responder to the MPA initiator during the Startup Phase. See [RFC5044]. ✨

ORD: See Outbound RDMA Read Queue Depth.

Outbound RDMA Read Queue Depth (ORD): The maximum number of simultaneous RDMA Read Requests that can be issued for the RDMA connection. This should be less than or equal to the peer's IRD. See [RDMAC]. 🌈

Passive Side: See Responder.

Private Data: A block of data exchanged between MPA endpoints during initial connection setup. See [RFC5044]. 🫶

Queue Pair (QP): A Queue Pair is the set of Work Queues associated exclusively with a single Endpoint (first defined in [VIA]). The Send Queue (SQ), Receive Queue (RQ), and Inbound RDMA Read Queue (IRQ) are considered part of the Queue Pair. The potentially shared Completion Queue (CQ) and Shared Receive Queue (SRQ) are not. See [RDMAC]. 👯

Remote Peer: The MPA protocol implementation on the opposite end of the connection. Used to refer to the remote entity when describing protocol exchanges or other interactions between two nodes. See [RFC5044]. 🦋

Responder: The connection endpoint that responds to an incoming MPA connection request (the MPA Request Frame). The responder is the passive side of the connection establishment. See [RFC5044]. 🌙

Ready to Receive (RTR): RTR is an indication provided by the last connection establishment message sent from the initiator to the responder. An RTR indicates that the initiator is ready to receive messages and that connection establishment is completed. 💖

Startup Phase: The initial exchanges of an MPA connection that help the MPA endpoints more fully identify each other and pass connection-specific setup information to each other. See [RFC5044]. 🚀

Shared Receive Queue (SRQ): A shared pool of Receive Work Requests posted by the Consumer that can be allocated by multiple RDMA endpoints (QP). See [RDMAC]. 🛍️

Tagged (DDP) Message: A DDP Message that targets a Tagged Buffer that is explicitly advertised to the Remote Peer through an exchange of an STag (memory handle), an offset in the memory region identified by the STag, and a length [RFC5040]. 🎯

Untagged (DDP) Message: A DDP Message that targets an Untagged Buffer associated with a queue specified by the Queue Number (QN). [RFC5040]. 📬

Work Queue: An element of a QP that lets user-space applications submit Work Requests directly to network hardware (first defined in [VIA]). Specific Work Queues include the Send Queue (SQ) for transmit requests, the Receive Queue (RQ) for receive requests specific to a single endpoint, and Shared Receive Queues (SRQs) for receive requests that can be allocated by one or more endpoints. See [RDMAC]. 🧁

Work Queue Element (WQE): A transport- and device-specific representation of a Work Request. See [RDMAC]. 🍭

Work Request: An elementary object Consumers use to enqueue a requested operation (WQEs) onto a Work Queue. See [RDMAC]. 🌷

4. Motivations 💜

The goal of this document is two-fold. 🎀 The first part is to extend support from the current client-server model for RDMA connection setup to a peer-to-peer model. The second part is to add negotiation of the RDMA Read Queue size for both sides of an RDMA connection. It's giving planning ahead energy. 🔮

4.1. Standardization of RDMA Read Parameter Configuration ⭐

Most RDMA applications are built on a transport-neutral Application Programming Interface (API) that accesses RDMA services using a "Queue Pair" paradigm. That paradigm was originally defined by the Virtual Interface Architecture [VIA], refined by the Direct Access Programming Library [DAPL], and is most commonly deployed with the OpenFabrics API [OFA]. 🌈

These transport-neutral APIs aim to provide a common set of RDMA services, whether the underlying transport is, for example, RDDP over MPA, RDDP over SCTP, or InfiniBand. The way they do this is so clean. 🫶

The common model for establishing an RDMA connection has the following steps: ✨

  • The passive side ULP listens for connection requests.
  • The active side ULP submits a connection request using an RDMA endpoint ("Queue Pair"), the desired destination, and the parameters to be used for the connection. Those parameters include both RDMA-layer characteristics, such as the number of simultaneous RDMA Read Requests to be allowed, and application-specific data (typically referred to as "Private Data"). 💅
  • The passive side ULP receives a connection request, which includes the identity of the active side and the requested connection characteristics. The passive side ULP uses this information to decide whether to accept the connection and, if it is to be accepted, how to create and/or configure the RDMA endpoint. 🧐
  • If accepting, the passive side ULP submits its acceptance of the connection request. This local accept operation includes the RDMA endpoint to be used and the connection characteristics (both the RDMA configuration and any application-specific Private Data to be returned). 🎁
  • The active side receives confirmation that the connection has been accepted, what the configured connection characteristics are, and any application-supplied Private Data. 🥳

Here's the tea: as currently defined, DDP connection establishment requires the ULP to encode the RDMA configuration in the application-specific Private Data. This leads to undesirable duplication of logic, since each ULP has to cover the RDMA characteristics of both InfiniBand and RDDP, and specify for both InfiniBand and RDDP how to extract those characteristics. 😭

Both RDDP and InfiniBand support an initial Private Data exchange. Because of that, a standard definition of the RDMA characteristics within the Private Data section would let common connection establishment APIs format the RDMA characteristics from the same API information used when establishing either protocol to form the connection. The application would then only have to indicate that it was using this standard format for common connection establishment procedures to apply common code that properly parses these fields and configures the RDMA endpoints accordingly. Exchanging the parameters needed to perform RDMA Read operations is a common use of the initial Private Data exchange. 🌸

One of the RDMA operations defined in [RDMAC] is an RDMA Read. RDMA Read operations are performed using an untagged message sent from a Queue Pair (QP) on the local endpoint to a QP on the remote endpoint, targeting the Inbound RDMA Read Request Queue (QN=1 or Inbound RDMA Read Request Queue (IRRQ)) associated with the connection. RDMA Read responses transfer data associated with each RDMA Read Request from the remote endpoint to the local endpoint using tagged messages. An inbound RDMA Read Request remains on the IRRQ from the time it is received until the time the last tagged message associated with the RDMA request is acknowledged. The IRRQ is associated with a QP but is not a Work Queue. Instead, the IRRQ is a stand-alone queue used to manage RDMA Read Requests associated with a QP. See [RDMAC], Section 6 for more information regarding QPs and IRRQ. One of the characteristics that must be configured for a QP is the size of the IRRQ. This parameter is called the Inbound RDMA Read Queue Depth (IRD). Another characteristic of a QP that must be configured is a local limit on the number of simultaneous outbound RDMA Read Requests, based on the size of the remote endpoint QP's IRRQ. This parameter is called the Outbound RDMA Read Queue Depth (ORD). ORD is used to limit the number of simultaneous RDMA Read Requests so that the local endpoint does not overrun the remote endpoint's IRRQ depth or IRD. Note that outbound RDMA Reads are submitted to a QP's Send Queue at the local peer, not to a separate outbound RDMA Read Request queue on the local peer. The local endpoint uses ORD to strictly limit simultaneous Read Requests so that IRRQ overruns don't occur at the remote endpoint. 🧠

Determination of the values of ORD and IRD is left to the ULP by the current RDDP suite of protocols and also by [RDMAC]. Since this negotiation of ORD and IRD is so typical, it's desirable to provide a common mechanism, as described in this document. Periodt. 💖

4.2. Enabling MPA Mode ✨

okay so MPA defines how DDP Segments get encoded inside Framed Upper Layer Protocol PDUs (FULPDUs) 🎀. To generate FULPDUs you need to periodically insert MPA Markers and compute the MPA CRC-32c for each frame. On the receiving end you may need to parse and strip those markers, use them to find MPA Frame boundaries, and validate the MPA-CRC32c 🔍.

The big design goal for MPA was that the resulting TCP stream be fully compliant for any and all TCP-aware middleboxes 🛡️. The tricky part is that only some TCP payload streams are valid streams of MPA FULPDUs, but any sequence of bytes is a valid TCP payload stream 🤯. So nobody at the MPA or TCP layer can figure out whether a given stream is in a specific MPA mode. That's why enabling MPA mode is handled by the ULP 👑.

The MPA protocol is basically two parts:

  • a spec for generating and receiving MPA FULPDUs. This part is unchanged by enhanced RDMA connection establishment 💅.
  • a pre-MPA exchange of messages that turns on a specific MPA mode for the TCP connection. Enhanced RDMA connection establishment adds two new features to this part 🦋.

In typical implementations, the hardware handles generating and receiving MPA FULPDUs, and host software handles the exchange of MPA Request and Reply Frames 💻. As we'll see, this split gets in the way of applications that don't fit the client-server assumptions baked into the current MPA Request/Reply exchange 😭.

4.3. Lack of Explicit RTR in MPA Request/Reply Exchange 🌸

The MPA Request and Reply exchange that puts a TCP connection into MPA mode is specified in [RFC5044]. It's honestly super good for MPA FULPDU hardware design, because:

  • The ULP is in charge of specifying the exact MPA Mode (Markers enabled or disabled, CRC-32c enabled or suppressed) and the point in the TCP streams (inbound and outbound) where MPA Frames will begin 🎯.
  • Before the first MPA Frame is transmitted, every pre-MPA mode TCP payload has already been acknowledged by the peer. So it's never necessary to generate a retransmission that mixes pre-MPA and MPA payload 🙌.
  • Before MPA reception is enabled, every incoming pre-MPA mode TCP payload has already been acknowledged. So the host never receives a TCP segment that mixes pre-MPA and MPA payload 🧼.

The catch is that the current MPA Request/Reply exchange doesn't define a Ready to Receive (RTR) indication that the active side would send. That means the passive side has no way to know the last non-MPA payload (the MPA Reply) was received 🥲.

Instead, the RTR indication gets piggybacked on the first MPA FULPDU sent by the active side. Honestly this is a valuable optimization for every app that fits the classic client-server model 💖. The client only starts the connection when it has a request to send to the server, and the server has nothing to send until it has received and processed that request 🧁.

Even apps where the server sends some config data right away can just send that same info as application Private Data in the MPA Reply 📦. So the current exchange works for basically every app 👏.

Now, lots of peer-to-peer apps, especially cluster calculation stuff (often using Message Passing Interface (MPI) [UsingMPI] or [RDS]), have no natural client or server roles ([PPMPI] [OpenMP]) 🧩. Usually one member of the cluster is arbitrarily picked to start the connection when the distributed task launches, while the other one accepts it 🎲. But at startup there's no way to predict which node will have the first message to actually send 🤷‍♀️. Connecting right away is super valuable because it cuts latency once results are ready to transmit, and it checks connectivity across the whole cluster ⚡.

Basically, the missing explicit RTR indication in the MPA Request/Reply exchange forces every app to have the connection initiator send the first message, whether or not that matches how the app actually communicates 🙃.

4.4. Limitations on ULP Workaround 🛠️

Honestly, the rule that the RDMA connection initiator sends the first message doesn't seem too bad at first glance 👀. The obvious question is why the application layer doesn't just generate a dummy message when there's nothing else to submit 🤔.

There are three factors that make this workaround a no-go for many peer-to-peer apps 🚫:

  • Transport-Neutral APIs.
  • Work/Completion Queue Accounting.
  • Host-based implementation of MPA Fencing.

4.4.1. Transport-Neutral APIs 🌍

A lot of these apps access RDMA services through a transport-neutral API like [DAPL] or [OFA]. Only RDDP over TCP [RFC5044] has a first message requirement 🙋‍♀️. Other RDMA transports, including RDDP over SCTP (see [RFC5043]) and InfiniBand (see [IBTA]), don't have one 🙅‍♀️.

App or middleware communications can be expressed as transport-neutral RDMA operations, which lets the lower software layers translate to transport and device specifics 🧚. Having a distinct extra message that's only required for one transport totally undermines the app's goal of being transport neutral 💔.

4.4.2. Work/Completion Queue Accounting 📚

RDMA local APIs usually use Work Queues to submit requests (Work Queue elements or WQEs) and to asynchronously receive completions (in Completion Queues or CQs) 📬.

Each Work Request can generate a Completion Queue Entry (CQE). Completions for successful transmit Work Requests are often suppressed, but the CQ capacity has to account for the possibility that each one completes in error 😬. A CQ can receive completions from multiple Work Queues 🫧.

CQs are designed so hardware RDMA implementations can write CQEs straight into a user-space-mapped buffer 🪄. That lets a user-space RDMA Consumer reap completions without needing kernel intervention 🏃‍♀️.

A hardware RDMA implementation can't reasonably wait for an open slot in the CQ 🙅. So the queue has to be sized so an overflow never happens. When an overflow does happen, it's treated as a catastrophic error and will typically mean tearing down all RDMA connections using that CQ 💀.

This style of interface is super efficient, but it puts the burden on the app to properly size each CQ to match the Work Queues feeding it 🧮.

Even though the format of both WQEs and CQEs is transport and device dependent, a transport-neutral API can treat WQEs and CQEs as abstract, transport- and device-neutral objects 🎨. So the number of WQEs and CQEs an app needs can also be transport and device neutral 🫶.

The capacity of the Work Queues and CQs can be calculated in an abstract, transport- and device-neutral way 🧠. If a dummy operation approach were used, lower layers would need to know the usage model, and it would mess up those calculations by inserting a dummy "operation" Work Request and then filtering out the matching completion 😵‍💫. The lower layer doesn't know the usage model the queue sizes are built on, and it doesn't know how often an insertion would be needed 🙈.

4.4.3. Host-based Implementation of MPA Fencing 🖥️

Lots of hardware implementations of RDDP using MPA/TCP don't handle the MPA Request/Reply exchange in hardware. Instead, the host processor handles it in software 🫠. With designs like that, it's common for MPA Fencing to live in the user-space, device-specific library (usually called a 'User Verbs' library or module) 📖.

When generating and receiving MPA FULPDUs are already dedicated to hardware, a Work Completion can only be generated by an untagged message 🎯. That's because a message arriving for a tagged buffer doesn't necessarily generate a completion, and it happens without any interaction with the ULP [RFC5040] 👻.

5. Enhanced MPA Connection Establishment 🦄

Here's the overview of Enhanced Connection Setup 💫. The goal is to allow standard negotiation of the ORD/IRD setting on both sides of the RDMA connection, and/or to negotiate the initial data transfer operation by the initiator when the existing 'client sends first' rule doesn't match the app's requirements 🎀.

The RDMA connection initiator sends an MPA Request, as specified in [RFC5044]. The new format defined here allows for:

  • Standardized negotiation of ORD and IRD 📏.
  • Negotiation of RTR functionality and the RDMA message type to use as the RTR indication 🚦.

The RDMA connection responder processes the MPA Request and generates an MPA Reply, as specified in [RFC5044]. The new format completes the negotiation 🏁.

The local interface needs a way for a ULP to request the use of an explicit RTR indication on a per-application or per-connection basis, for when an explicit RTR indication will be required 🙏. Piggybacking the RTR on a Client's first message is still a valuable optimization for most connections 💜.

⚠️ The RDMA connection initiator MUST NOT allow any later FULPDUs to be transmitted before the RTR indication. One method to achieve this is to delay notifying the ULP that the RDMA connection has been established until after any required RTR indication has been transmitted 🚨.

All MPA exchanges happen over TCP before RDMA establishment, so they're signaled via TCP and not via RDMA completion 🧚‍♀️💅.

6. Enhanced MPA Request/Reply Frames 🎀

Okay so enhanced RDMA connection establishment has its own alternate outfit for MPA Requests and Replies, and honestly it's giving main character 💅 Here's the layout:

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 0  |                                                               |
    +         Key (16 bytes containing "MPA ID Req Frame")          +
 4  |      (4D 50 41 20 49 44 20 52 65 71 20 46 72 61 6D 65)        |
    +         Or  (16 bytes containing "MPA ID Rep Frame")          +
 8  |      (4D 50 41 20 49 44 20 52 65 70 20 46 72 61 6D 65)        |
    +                                                               +
 12 |                                                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 16 |M|C|R|S| Res   |     Rev       |          PD_Length            |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                                                               |
    ~                                                               ~
    ~                   Private Data                                ~
    |                                                               |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Key: Unchanged from [RFC5044]. 🫶

M: Unchanged from [RFC5044].

C: Unchanged from [RFC5044].

R: Unchanged from [RFC5044].

S: This one is set to One if the Private Data begins with the enhanced RDMA connection establishment data, and 0 otherwise. The vibe is very "is this the enhanced one or nah" 🔥

Res: This field is one bit smaller than in [RFC5044] but otherwise unchanged. In [RFC5044], the 'Res' field, where the newly defined 'S' bit lives, is reserved for future use. [RFC5044] says that 'Res' MUST be set to zero when sending and MUST NOT be checked on reception, which is what makes the 'S' bit backwards compatible with the original MPA Frame format 🦋 When the 'S' bit is set to zero, no additional Private Data is used for enhanced RDMA connection establishment, so the resulting MPA Request and Reply Frames are identical to the unenhanced protocol. Same frame, no drama 🌸

Rev: This field holds the revision of MPA. To use any enhanced connection establishment feature, this MUST be set to two or higher. If no enhanced connection establishment features are desired, it MAY be set to one. A host accepting MPA connections MUST continue to accept MPA Requests with version one, even if it supports version two. Old friends still get invited to the party 🎀

PD_Length: Unchanged from [RFC5044]. This is the total length of the Private Data field, including the enhanced RDMA connection establishment data, if present. 🧚

Private Data: Unchanged from [RFC5044]. However, if the 'S' flag is set, Private Data MUST begin with enhanced RDMA connection establishment data (see Section 9). ✨

7. Enhanced SCTP Session Control Chunks 🦋

Okay bestie, enhanced RDMA connection establishment uses the first 32 bits of the Private Data field for IRD and ORD negotiation in the "DDP Stream Session Initiate" and "DDP Stream Session Accept" SCTP Session Control Chunks. Iconic 👑

The type of the SCTP Session Control Chunk is defined by a Function Code (see [RFC4960]). [RFC5043] already defines codes for 'DDP Stream Session Initiate' and 'DDP Stream Session Accept', which are equivalent to an MPA Request Frame and an accepting MPA Reply Frame.

Enhanced RDMA connection establishment needs three additional function codes, listed below 💖

  • Enhanced DDP Stream Session Initiate: 0x005
  • Enhanced DDP Stream Session Accept: 0x006
  • Enhanced DDP Stream Session Reject: 0x007

The Enhanced Reject function code MUST be used to indicate rejection of enhanced DDP stream session for a configuration that would have been accepted for unenhanced DDP stream session negotiation. Rejecting the enhanced version is a whole different energy 😭

The enhanced DDP stream session establishment follows the same rules as the standard DDP stream session establishment as defined in [RFC5043]. ULP-supplied Private Data MUST be included for Enhanced DDP Stream Session Initiate, Enhanced DDP Stream Session Accept, and Enhanced DDP Stream Session Reject messages, and MUST follow the enhanced RDMA connection establishment data in the DDP Stream Session Initiate and the Enhanced DDP Stream Session Accept messages. The ordering is strict, so keep it neat 🎀

Private Data length MUST NOT exceed 512 bytes in any message, including enhanced RDMA connection establishment data. No oversized fits allowed 🙈

Private Data MUST NOT be included in the DDP Stream Session TERM message. TERM stays clean 🧼

Received Extended DDP Stream Session Control messages SHOULD be reported to the ULP. If they are reported, any supplied Private Data MUST be available for the ULP to examine. For example, a received Extended DDP Stream Session Control message is not reported to the ULP if none of the requested RTR indication types are supported by the receiver. In this case, the Provider MAY generate a reject reply message indicating which RTR indication types it supports. Basically, if nobody can do what you asked, they're allowed to tell you what they can do 💅

The enhanced DDP stream management MUST use the DDP stream session termination function code to terminate a stream established using enhanced DDP stream session function codes. Same exit, different entrance ✨

[RFC5043] already supports either side sending the first DDP Message, since the Payload Protocol Identifier (PPID) already tells apart Session Establishment from DDP Segments. The enhanced RDMA connection establishment gives the ULP a transport-independent way to support the peer-to-peer model. Both sides get to make the first move, and it's so healthy 🫶

The following additional Legal Sequences of DDP Stream Session messages are defined:

  • Enhanced Active/Passive Session Accepted: as with Section 6.2 of [RFC5043], but with the extended opcodes as defined in this document.
  • Enhanced Active/Passive Session Rejected: as with Section 6.3 of [RFC5043], but with the extended opcodes as defined in this document.
  • Enhanced Active/Passive Session Non-ULP Rejected: as with Section 6.4 of [RFC5043], but with the extended opcodes as defined in this document.

8. MPA Error Reporting 🚨

Ngl this part is lowkey dramatic. The RDMA connection establishment protocol is layered upon the protocols defined in [RFC5040] and [RFC5041]. Any enhanced RDMA connection establishment error generates an MPA termination message to a peer. [RFC5040] defines a triplet of protocol layers, error types, and error codes for error specification. MPA negotiation for RDMA connection establishment uses the following layer and error type for MPA error reporting:

Layer: 0x2 - LLP Error Type: 0x0 - MPA

While [RFC5044] defines four error codes, [RFC5043] does not define any. Enhanced RDMA connection establishment extends the error codes defined in [RFC5044] by adding three new error codes. Thus, enhanced RDMA connection establishment is backward compatible with both [RFC5043] and [RFC5044]. Everybody still gets a seat at the table 🪑💜

The following error codes are defined for enhanced RDMA connection establishment negotiation:

   Error Code         Description
   --------------------------------------------------------
   0x05               Local catastrophic
   0x06               Insufficient IRD resources
   0x07               No matching RTR option

9. Enhanced RDMA Connection Establishment Data 🍓

Okay this is the heart of it. Enhanced RDMA connection establishment places the following 32 bits at the beginning of the Private Data field of the MPA Request and Reply Frames, or the "DDP Stream Session Initiate" and "DDP Stream Session Accept" SCTP Session Control Chunks. ULP-specified Private Data follows this field, so the maximum amount of ULP-specified Private Data is reduced by 4 bytes. Note that this field MUST be sent in network byte order, with the IRD and ORD encoded as 14-bit unsigned integers. Byte order is not the place to freestyle 🙈

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 0  |A|B|        IRD                |C|D|        ORD                |
 4  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

IRD: Inbound RDMA Read Queue Depth. 📥

ORD: Outbound RDMA Read Queue Depth. 📤

A: Control Flag for connection model. 🌈

B: Control Flag for use of a zero-length FULPDU (Send) RTR indication. 🧚

C: Control Flag for use of a zero-length RDMA Write RTR indication. 💫

D: Control Flag for use of a zero-length RDMA Read RTR indication. 🦋

9.1. IRD and ORD Negotiation 🎀

okay so IRD and ORD are the two numbers both endpoints use to agree on how many RDMA Read Requests can be in flight at once 💅. IRD sets how deep the Inbound RDMA Read Request Queue (IRRQ) is on each endpoint. ORD limits how many outbound RDMA Read operations you can have going at any given moment, so you never overrun the IRRQ on the other side 😭. To describe the negotiated local and remote values, there are four terms:

  • Initiator IRD: the IRD value sent in the MPA Request or "DDP Stream Session Initiate" SCTP Session Control Chunk 🦋. It's the initiator's IRD at the moment the MPA Request is generated. The responder sets its local ORD to this value or less. It's the most simultaneous inbound RDMA Read Requests the initiator can handle for that connection.
  • Initiator ORD: the ORD value in the MPA Request or "DDP Stream Session Initiate" SCTP Session Control Chunk ✨. It's the initiator's starting ORD when the request is generated, and it's also a request asking the responder to support a responder IRD of at least this value. It's the most simultaneous outbound RDMA Read operations the initiator wants the responder to support for that connection.
  • Responder IRD: the IRD value returned in the MPA Reply or "DDP Stream Session Accept" SCTP Session Control Chunk 🌸. This is the real value the responder sets for its local IRD. For a successful negotiation it's greater than or equal to the initiator ORD. It's the most simultaneous inbound RDMA Read Requests the responder can actually support for that connection.
  • Responder ORD: the ORD value returned in the MPA Reply or "DDP Stream Session Accept" SCTP Session Control Chunk 💜. This is the real value the responder used for ORD. For a successful negotiation it's less than or equal to the initiator IRD. It's the most simultaneous outbound RDMA Read operations the responder will allow for that connection.

once a negotiation succeeds, these relationships hold (iykyk 🙈):

initiator ORD <= responder IRD

responder ORD <= initiator IRD

The responder and initiator MUST pass the peer's provided IRD and ORD values to the ULP 🫶, on top of using the values calculated by the rules above.

The responder ORD SHOULD be set to a value less than or equal to the initiator IRD 💅. If the initiator ORD isn't enough for the connection model you picked, the responder IRD MAY be increased. For example, if the initiator ORD is 0 (so the ULP won't use RDMA Reads) and the responder supports a zero-length RDMA Read RTR indication, the responder IRD can be set to 1 🧚. The responder MUST set its ORD at most to the initiator IRD. The responder MAY reject the connection request if the initiator IRD isn't enough for the ULP-required ORD, and it can put the required ORD in the MPA Reject Frame responder ORD field. So the TERM message MUST contain Layer 2, Error Type 0, Error Code 6 ⭐.

When the initiator gets the MPA Accept Frame from the responder, it MUST set its IRD at least to the responder ORD, and its ORD at most to the responder IRD 🔥. If the initiator doesn't have enough resources for the required IRD, it MUST send a TERM message to the responder saying there are insufficient resources, then terminate the connection for that reason. So again, the TERM message MUST contain Layer 2, Error Type 0, Error Code 6 💀.

The initiator MUST pass the responder's provided IRD and ORD to the ULP for both MPA Accept and Reject messages 🌈. The initiator ULP then decides what to do. For example, it might terminate the established connection and renegotiate the responder ORD. main character decisions only 👑.

An all ones value (0x3FFF) means automatic negotiation of the IRD or ORD is NOT wanted, and the ULP will handle it instead 🍓. The responder MUST respond to an initiator ORD of 0x3FFF by leaving its local endpoint IRD unchanged and setting IRD to 0x3FFF in its reply. The initiator MUST leave its local endpoint ORD unchanged when it receives a responder IRD of 0x3FFF. The responder MUST respond to an initiator IRD of 0x3FFF by leaving its local endpoint ORD unchanged and setting ORD to 0x3FFF in its reply. The initiator MUST leave its local endpoint IRD unchanged when it receives a responder ORD of 0x3FFF 🦋.

9.2. Peer-to-Peer Connection Negotiation 💖

okay this is where we pick the connection model and the RTR indication options 🎀. Control Flag A set to 1 means a peer-to-peer connection model is happening, and 0 means client-server 💅. Control Flag B set to 1 means a zero-length FULPDU (Send) RTR indication is requested by the initiator and supported by the responder, and 0 otherwise 🌸. Control Flag C set to 1 means a zero-length RDMA Write RTR indication is requested by the initiator and supported by the responder, and 0 otherwise ✨. Control Flag D set to 1 means a zero-length RDMA Read RTR indication is requested by the initiator and supported by the responder, and 0 otherwise 🫶.

The initiator MUST set Control Flag A to 1 for the peer-to-peer model. If Control Flag A is set to 1, the initiator MUST set each of Control Flags B, C, and D to 1 for each option it supports 🔥.

The responder MUST support at least one RTR indication option if it supports Enhanced RDMA connection establishment 👑. If Control Flag A is 1 in the MPA Request message, the responder MUST set Control Flag A to 1 in the MPA reply message 🧚. For each initiator-supported RTR indication option, the responder SHOULD set the matching Control Flag if it can support that option in an MPA reply 🍓. The responder isn't required to specify every RTR indication option it supports 🙈. The responder MUST set at least one RTR indication option if it supports more than one initiator-specified RTR indication option. The responder MAY include extra RTR indication options it supports, even if no initiator-specified option asked for them 🌈. If the responder doesn't support any of the initiator-specified RTR indication options, it MUST set at least one RTR indication type option it does support 💀.

When the initiator gets the MPA Accept Frame with Control Flag A set to 1, it MUST generate one of the negotiated RTR indications 💅. If the initiator can't generate any of the responder-supported RTR indications, it MUST send a TERM message to the responder saying it failed to negotiate a mutually compatible connection model or RTR option, and then terminate the connection. So the TERM message MUST contain Layer 2, Error Type 0, Error Code 7 😭. The ULP can negotiate a ULP-level RTR indication when a Provider-level RTR indication can't be negotiated 🦋.

The initiator MUST set Control Flag A to 0 for the client-server model 🎀. The responder MUST set Control Flag A to 0 if Control Flag A is 0 in the request. If Control Flag A is 0, then Control Flags B, C, and D MUST also be 0 ✨. On reception, if Control Flag A is 0, Control Flags B, C, and D MUST be ignored 💜.

9.3. Enhanced Connection Negotiation Flow 🧚

the RTR indication type and the ORD/IRD negotiation go in this order 🦋:

initiator (MPA Request) -->  The initiator sets Control Flag A to 1
   to indicate the peer-to-peer connection model and sets its initial
   IRD/ORD on the local endpoint of the connection.  The initiator
   also sets Control Flags B, C, and D to 1 for each initiator-
   supported option of RTR indication.

responder (MPA Reply) <--  The responder matches the initiator's
   Control Flag A value and sets ORD/IRD to its local endpoint values
   based upon the initiator's initial ORD/IRD values and the number
   of simultaneous RDMA Read Requests required by the ULP.  The
   responder sets Control Flags B, C, and D to 1 for each responder-
   supported option of RTR indication options for the peer-to-peer
   connection model.  The responder also sets its IRD/ORD to actual
   values.

initiator (First RDMA Message) -->  After the initiator modifies its
   ORD/IRD to match the responder's values as stated above, the
   initiator sends the first message of the negotiated RTR indication
   option.  If no matching RTR indication option exists, then the
   initiator sends a TERM message.

Now the tricky bit 😤. The initiator or responder MUST generate the TERM message containing Layer 2, Error Type 0, Error Code 5 whenever it hits a local error that doesn't have its own special Error Code defined in Section 8, and it MUST do this before resetting the connection 🙈.

10. Interoperability ✨

okay so here's how enhanced and unenhanced endpoints get along 🫶. The initiator asks for enhanced RDMA connection establishment by sending an enhanced RDMA establishment request. An enhanced responder is REQUIRED to answer with an enhanced RDMA connection establishment response 💖. An unenhanced responder, though, treats the enhanced request as improperly formatted and closes the TCP connection 😭. All responders are REQUIRED to send unenhanced RDMA connection establishment responses to unenhanced RDMA connection establishment requests 🌸.

The initiator MUST NOT use the enhanced RDMA connection establishment formats or function codes when no enhanced functionality is wanted 🙅‍♀️. The responder MUST continue to accept unenhanced connection requests 🎀.

There are three initiator/responder cases involving enhanced MPA: both the initiator and responder are enhanced, only the responder is enhanced, or only the initiator is enhanced. The enhanced MPA Frame is marked by field 'S' set to 1 🦋.

Enhanced MPA initiator and responder: if the responder receives an enhanced MPA message, it MUST respond with an enhanced MPA message. Both sides are on the same page, bestie 🥰.
Enhanced MPA responder only: if the responder receives an unenhanced MPA message ('S' is set to 0), it MUST respond with an unenhanced MPA message 🍓.
Enhanced MPA initiator only: if the responder receives an enhanced MPA message and doesn't support enhanced RDMA connection establishment, it MUST close the TCP connection and exit MPA 💀. From a standard RDMA connection establishment point of view, the enhanced MPA Frame is improperly formatted as stated in [RFC5044]. So both the initiator and responder report the TCP connection termination to an application locally 😭. In this case, the initiator MAY try to set up an RDMA connection using the unenhanced MPA protocol as defined in [RFC5044], if that protocol works with the application, and let the ULP deal with ORD and IRD and peer-to-peer negotiations 🧚.

a note for future enhancements to connection establishment negotiation 👀: it's possible to extend the formatting of Private Data in the MPA Request and Reply Frames even more, and to use other bits from the "Res" field to signal additional Private Data formatting 🌈.

11. IANA Considerations 🎀

okay bestie, IANA (the folks who keep all the official registries 📋) gets some fresh entries! first up, three new codes go into the "SCTP Function Codes for DDP Session Control" registry that Section 3.5 of [RFC6580] created 🦋

0x0005,  Enhanced DDP Stream Session Initiate, [RFC6581]

0x0006,  Enhanced DDP Stream Session Accept, [RFC6581]

0x0007,  Enhanced DDP Stream Session Reject, [RFC6581]

then IANA also adds three entries to the "MPA Errors" registry from Section 3.3 of [RFC6580] 💀

0x2/0x0/0x05,  - MPA Error / Local catastrophic error, [RFC6581]

0x2/0x0/0x06  - MPA Error / Insufficient IRD resources, [RFC6581]

0x2/0x0/0x07  - MPA Error / No matching RTR option, [RFC6581]

12. Security Considerations 🔐

okay this part is serious, so read it carefully 🫶 the security considerations from RFC 5044 and RFC 5043 still apply here, and the changes in this document do not bring in any new security considerations ✨

however, it is recommended that implementations do sanity checking on the input parameters. that includes ORD, IRD, and the control flags used for RTR indication option negotiation. so check your inputs, babe 💅

13. Acknowledgements 🌸

the authors want to thank Sean Hefty, Dave Minturn, Tom Talpey, David Black, and David Harrington 💜 for their valuable contributions and reviews of this document. the real MVPs 👑

14. References 📚

14.1. Normative References 🧱

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC4960]  Stewart, R., "Stream Control Transmission Protocol", RFC
           4960, September 2007.

[RFC5040]  Recio, R., Metzler, B., Culley, P., Hilland, J., and D.
           Garcia, "A Remote Direct Memory Access Protocol
           Specification", RFC 5040, October 2007.

[RFC5041]  Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
           Data Placement over Reliable Transports", RFC 5041,
           October 2007.

[RFC5043]  Bestler, C. and R. Stewart, "Stream Control Transmission
           Protocol (SCTP) Direct Data Placement (DDP) Adaptation",
           RFC 5043, October 2007.

[RFC5044]  Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
           Carrier, "Marker PDU Aligned Framing for TCP
           Specification", RFC 5044, October 2007.

[RFC6580]  Ko, M. and D. Black, "IANA Registries for the Remote
           Direct Data Placement (RDDP) Protocols", RFC 6580, April
           2012.

14.2. Informative References 🦋

[DAPL]     "Direct Access Programming Library",
           <http://www.datcollaborative.org/uDAPL_doc_062102.pdf>.

[IBTA]     "InfiniBand Architecture Specification Release 1.2.1",
           <http://www.infinibandta.org>.

[OFA]      "OFA verbs & APIs", <http://www.openfabrics.org/>.

[OpenMP]   McGraw-Hill, "Parallel Programming in C with MPI and
           OpenMP", 2003.

[PPMPI]    Morgan Kaufmann Publishers Inc., "Parallel Programming
           with MPI", 2008.

[RDMAC]    "RDMA Protocol Verbs Specification (Version 1.0)",
           <http://www.rdmaconsortium.org/home/
           draft-hilland-iwarp-verbs-v1.0-RDMAC.pdf>.

[RDS]      Open Fabrics Association, "Reliable Datagram Socket",
           2008,
           <http://www.openfabrics.org/archives/spring2008sonoma>.

[UsingMPI] MIT Press, "Using MPI-2: Advanced Features of the Message
           Passing Interface", 1999.

[VIA]      Cameron, Don and Greg Regnier, "Virtual Interface
           Architecture", Intel, April 2002.

Authors' Addresses 🧚

Arkady Kanevsky (editor)
Dell Inc.
One Dell Way, MS PS2-47
Round Rock, TX 78682
USA

Phone: +1-512-728-0000
EMail: arkady.kanevsky@gmail.com


Caitlin Bestler (editor)
Nexenta Systems
555 E El Camino Real #104
Sunnyvale, CA 94087
USA

Phone: +1-949-528-3085
EMail: Caitlin.Bestler@nexenta.com


Robert Sharp
Intel
LAD High Performance Message Passing, Mailstop: AN1-WTR1
1501 South Mopac, Suite 400
Austin, TX 78746
USA

Phone: +1-512-493-3242
EMail: robert.o.sharp@intel.com


Steve Wise
Open Grid Computing
4030 Braker Lane STE 130
Austin, TX 78759
USA

Phone: +1-512-343-9196 x101
EMail: swise@opengridcomputing.com
📦 where this page comes from