RFC 7306
Remote Direct Memory Access (RDMA) Protocol Extensions
Remote Direct Memory Access (RDMA) Protocol Extensions 💖
Okay bestie, let's start with the basics 🎀. This is RFC 7306, a Standards Track doc from the IETF, dated June 2014 (ISSN 2070-1721). The authors are H. Shah from Broadcom Corporation, A. Eiriksson, F. Marti and W. Noureddine from Chelsio Communications, Inc., and R. Sharp from Intel Corporation 👑. The title block doesn't list anything this RFC obsoletes or updates.
Abstract
okay so this doc extends the IETF Remote Direct Memory Access Protocol (RDMAP) from RFC 5040 🧚. RDMAP gives applications read and write services straight to memory, so data can land directly in Upper-Layer Protocol (ULP) Buffers with no intermediate copies 💅. The extensions here add two capabilities and improvements: Atomic Operations and Immediate Data ✨.
Status of This Memo
this is an Internet Standards Track document 🌈.
It comes from the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community. It went through public review and got approved for publication by the Internet Engineering Steering Group (IESG) 🫶. For more on Internet Standards, check Section 2 of RFC 5741.
You can find the current status, any errata, and a way to send feedback at http://www.rfc-editor.org/info/rfc7306.
Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. 🔒
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the date of publication. Please read those carefully, because they spell out your rights and restrictions for this doc 📜. Code Components pulled out of this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License.
Table of Contents
1. Introduction ....................................................4
1.1. Discovery of RDMAP Extensions ..............................5
2. Requirements Language ...........................................5
3. Glossary ........................................................6
4. Header Format Extensions ........................................7
4.1. RDMAP Control and Invalidate STag Fields ...................7
4.2. RDMA Message Definitions ...................................9
5. Atomic Operations ...............................................9
5.1. Atomic Operation Details ..................................10
5.1.1. FetchAdd ...........................................10
5.1.2. CmpSwap ............................................12
5.2. Atomic Operations .........................................13
5.2.1. Atomic Operation Request Message ...................14
5.2.2. Atomic Operation Response Message ..................17
5.3. Atomicity Guarantees ......................................18
5.4. Atomic Operations Ordering and Completion Rules ...........18
6. Immediate Data .................................................20
6.1. RDMAP Interactions with ULP for Immediate Data ............20
6.2. Immediate Data Header Format ..............................21
6.3. Immediate Data or Immediate Data with SE Message ..........21
6.4. Ordering and Completions ..................................22
7. Ordering and Completions Table .................................22
8. Error Processing ...............................................25
8.1. Errors Detected at the Local Peer .........................25
8.2. Errors Detected at the Remote Peer ........................26
9. Security Considerations ........................................26
10. IANA Considerations ...........................................27
10.1. RDMAP Message Atomic Operation Subcodes ..................27
10.2. RDMAP Queue Numbers ......................................28
11. References ....................................................29
11.1. Normative References .....................................29
11.2. Informative References ...................................29
12. Acknowledgments ...............................................30
Appendix A. DDP Segment Formats for RDMA Messages .................31
A.1. DDP Segment for Atomic Operation Request ..................32
A.2. DDP Segment for Atomic Response ...........................33
A.3. DDP Segment for Immediate Data and Immediate Data with SE .331. Introduction
okay so the RDMA Protocol [RFC5040] gives you zero-copy data communications that keep memory protection semantics intact, which makes network protocol implementations way more efficient 🦋. It's part of the iWARP family, which also includes RFC 5041 [RFC5041], RFC 5044 [RFC5044], and RFC 6581 [RFC6581]. This document adds the following extensions to the RDMA Protocol (RDMAP):
- Atomic Operations can be performed on remote memory locations. Atomic Operations make RDMAP way more useful in distributed shared-memory environments 💜.
- Immediate Data messages let the ULP at the sender include a small amount of data. When an Immediate Data message follows an RDMA Write Message, the pair works like the RDMA Write with Immediate message you'll find in other RDMA transport protocols 🍓.
Other RDMA transport protocols already have the functionality these extensions add, and that creates differences in RDMA applications and/or Upper-Layer Protocols. Getting rid of those differences makes applications and ULPs simpler, and that's the main motivation for this doc ✨.
RSockets [RSOCKETS] is an example of RDMA-enabled middleware. It gives you a socket interface on the upper edge and uses RDMA underneath for more efficient networking for socket-based applications 🌸. RSockets knows about Immediate Data support in InfiniBand [IB], but it can't use the RDMA Write with Immediate Data operation from InfiniBand 🙈. Adding the Immediate Data operation from this document closes that gap for RSockets on InfiniBand and iWARP alike.
Structured high-performance computing applications built on the Message-Passing Interface [MPI] may use the Atomic Operations defined here 🔥. DAT Atomics [DAT_ATOMICS] is an example of RDMA-enabled middleware that gives you a portable RDMA programming interface across various RDMA transport protocols. DAT Atomics includes a primitive for InfiniBand that iWARP RDMA-enabled Network Interface Controllers (RNICs) don't support. Adding Atomic Operations from this document lets Atomic Operations in DAT Atomics work interchangeably on InfiniBand and RNICs 👑.
For more background on where RDMA Protocol applies, check "Applicability of Remote Direct Memory Access Protocol (RDMA) and Direct Data Placement Protocol (DDP)" [RFC5045] 🧚.
1.1. Discovery of RDMAP Extensions
the situation right now is that some RDMA applications and/or ULPs already know about Atomic and Immediate Data operations on RDMA transports like InfiniBand, and on application programming interfaces such as Open Fabrics Verbs [OFAVERBS] 🙈. Those apps have to know that RDMAP doesn't support some of these operations. Usually, software adapter query interfaces expose these capabilities to applications, and then apps decide whether to use Immediate Data or Atomic Operations based on what the query returns. The catch is that those query interfaces usually report the scope of atomicity guarantees, not each individual Atomic Operation that's supported. So this specification requires that if an RNIC supports any Atomic Operations, it supports all of the Atomic Operations defined within 💅.
When heterogeneous hardware gets deployed for RDMA applications and/or ULPs, and that hardware supports Atomic Operations and Immediate Data Operations to different degrees, the apps are either statically configured to use or skip optional features, or they use application-specific negotiation mechanisms 😭. For the extensions in this document, RDMA applications and/or ULPs are RECOMMENDED to negotiate usage of these extensions at the application or ULP level 🫶. The definition of those application-specific mechanisms is outside the scope of this specification. For backward compatibility, existing applications and/or ULPs should not assume that these extensions are supported.
Without application-specific negotiation of the features defined here, you can just attempt the new operations and use the errors that get reported to figure out what a remote peer can do ✨. For Atomics, a FetchAdd operation with "Add Data" set to 0 can safely determine whether Atomic Operations exist, without changing the content of the remote peer's memory 🎀. If there's an Immediate Data or Atomic Operation the remote peer doesn't support, the remote peer reports a Remote Operation Error or Unexpected OpCode error.
2. Requirements Language
okay so the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119] 💜. These are the BCP 14 key words, so read them with care!
3. Glossary ✨
okay so this doc is a extension of RFC 5040, and the key words (the MUST/SHOULD stuff 🫶) are defined in that doc's glossary, so go check there too 🧚
- Atomic Operation: an operation that runs a memory op at a specific ULP Buffer address on a remote node, using the Tagged Buffer data transfer model. The consumer can read, modify, and write memory at the destination ULP Buffer address 💅 and it's guaranteed that no other Atomic Operation read or write hits that same address across any other RDMAP Streams on an RNIC at the Responder. iconic 👑
- Atomic Operation Request: an RDMA Message the Data Source uses to do an Atomic Operation at the Responder.
- Atomic Operation Response: an RDMA Message the Responder uses to say an Atomic Operation is done.
- CmpSwap: an Atomic Operation that compares and swaps a value at a specific address on a remote node 🔄
- FetchAdd: an Atomic Operation that atomically increments a value at a specific ULP Buffer address on a remote node ✨
- Immediate Data: a small, fixed-size bit of data sent from the Data Source to a Data Sink 🍓
- Immediate Data Message: an RDMA Message the Data Source uses to send Immediate Data to the Data Sink.
- Immediate Data with Solicited Event (SE) Message: an RDMA Message the Data Source uses to send Immediate Data with Solicited Event to the Data Sink 🔔
- iWARP: a suite of wire protocols made up of RFC 5040, RFC 5041, RFC 5044, and RFC 6581 🌈
- Requester: the sender of an RDMA Atomic Operation request.
- Responder: the receiver of an RDMA Atomic Operation request.
- RNIC: RDMA-enabled Network Interface Controller. In this context, that's a network I/O adapter or embedded controller with iWARP functionality 🦋
- ULP: Upper-Layer Protocol, aka the protocol layer above the one being referenced. The ULP for RFC 5040 / RFC 5041 is expected to be an OS, Application, adaptation layer, or proprietary device. The RFC 5040 / RFC 5041 docs don't specify a ULP, they just give you semantics so a ULP can be designed to use them 🧠
4. Header Format Extensions
okay this part is about the headers 🎀 The control info of RDMA Messages lives in header fields defined in RFC 5041, the Direct Data Placement (DDP) protocol. RFC 5040 defines the RDMAP header formats that sit on top of the DDP header definition. This spec extends RFC 5040 with these new formats:
- Four new RDMA Messages carry extra RDMAP headers. The Immediate Data operation and the Immediate Data with Solicited Event operation each include 8 bytes of data after the RDMAP header. Atomic Operations include Atomic Request or Atomic Response headers after the RDMAP header. The RDMAP header for Atomic Request messages is 52 bytes long, as specified in Figure 4. The RDMAP header for Atomic Response messages is 32 bytes long, as specified in Figure 5 💜
- A new queue for Untagged Buffers (QN=3) is introduced, used for Atomic Response tracking 🌸
4.1. RDMAP Control and Invalidate STag Fields
for reference, here's the DDP Control and RDMAP Control Fields layout, in the style and convention of RFC 5040 📐
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|T|L| Resrv | DV| RV|Rsv| Opcode|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Invalidate STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 1: DDP Control and RDMAP Control Fieldsthe DDP Control Field is made up of the T (Tagged), L (Last), Resrv, and DV (DDP protocol Version) fields [RFC5041]. The RDMAP Control Field is made up of the RV (RDMA Version), Rsv, and Opcode fields [RFC5040] 🫧
this spec adds new values for the RDMA Opcode field on top of the ones in RFC 5040. Figure 2 defines the new Opcode values used by the RDMA Messages in this spec 🔥
as Figure 2 shows, STag (Steering Tag) and Tagged Offset don't apply to the RDMA Messages defined here. Figure 2 also shows the right Queue Number for each Opcode 🎯
All RDMA Messages defined in this specification MUST have:
- The RDMA Version (RV) field:
01b. - Opcode field: set to one of the values in Figure 2.
- Invalidate STag: set to zero by the sender, ignored by the receiver.
-------+-----------+-------+------+-------+---------+-------------
RDMA | Message | Tagged| STag | Queue | In- | Message
Opcode | Type | Flag | and | Number| validate| Length
| | | TO | | STag | Communicated
| | | | | | between DDP
| | | | | | and RDMAP
-------+-----------+-------+------+-------+---------+-------------
1000b | Immediate | 0 | N/A | 0 | N/A | Yes
| Data | | | | |
-------+-----------+----------------------------------------------
1001b | Immediate | 0 | N/A | 0 | N/A | Yes
| Data with | | | | |
| SE | | | | |
-------+-----------+----------------------------------------------
1010b | Atomic | 0 | N/A | 1 | N/A | Yes
| Request | | | | |
-------+-----------+----------------------------------------------
1011b | Atomic | 0 | N/A | 3 | N/A | Yes
| Response | | | | |
-------+-----------+----------------------------------------------
Figure 2: Additional RDMA Usage of DDP FieldsNote: N/A means Not Applicable 🙈
this extension defines RDMAP use of Queue Number 3 for Untagged Buffers for Atomic Responses. That queue is used for tracking outstanding Atomic Requests 🕵️♀️
all the other DDP and RDMAP Control Fields are set the way RFC 5040 describes 💅
4.2. RDMA Message Definitions
this figure shows which RDMA Headers each new RDMA Message uses, and which new RDMA Messages are allowed to carry ULP payload 🧁
-------+-----------+-------------------+-------------------------
RDMA | Message | RDMA Header Used | ULP Message allowed in
Message| Type | | the RDMA Message
OpCode | | |
| | |
-------+-----------+-------------------+-------------------------
1000b | Immediate | Immediate Data | No
| Data | Header |
-------+-----------+-------------------+-------------------------
1001b | Immediate | Immediate Data | No
| Data with | Header |
| SE | |
-------+-----------+-------------------+-------------------------
1010b | Atomic | Atomic Request | No
| Request | Header |
-------+-----------+-------------------+-------------------------
1011b | Atomic | Atomic Response | No
| Response | Header |
-------+-----------+-------------------+-------------------------
Figure 3: RDMA Message Definitions5. Atomic Operations
RFC 5040 doesn't include support for Atomic Operations, and those are a seriously important building block for distributed shared memory 🧩
this doc extends the RDMA Protocol with a set of basic Atomic Operations and specifies their resource and ordering rules. The Atomic Operations here give you the same functionality as the InfiniBand RDMA transport and the extended Atomic Operations in Open Fabrics Verbs, so apps using these primitives can work interchangeably over iWARP 🌈 Other operations are left for future consideration 🔮
Atomic Operations as specified here execute a 64-bit memory operation at a specified destination ULP Buffer address on a Responder node, using the Tagged Buffer data transfer model. They atomically read, modify, and write back the contents of that destination ULP Buffer address, and guarantee that Atomic Operations on that address from other RDMAP Streams on the same RNIC don't happen between the read and the write caused by the Atomic Operation 🔒 So the Responder RNIC MUST implement mechanisms that stop Atomic Operations to memory registered for Atomic Operations while an Atomic Operation targeting that memory is in progress. The Requester of an Atomic Operation can't rely on Atomic Operation behavior at the Responder across multiple RNICs, or with respect to other applications/ULPs running at the Responder that can access the ULP Buffer. It is OPTIONAL for an RNIC to provide that behavior when implementing the Atomic Operations in this doc 🤷♀️ An RNIC that supports Atomic Operations as specified here MUST implement both the FetchAdd operation as specified in Section 5.1.1 and the CmpSwap operation as specified in Section 5.1.2. The advertisement of Tagged Buffer information for Atomic Operations is outside the scope of this specification and is handled by the ULPs 📣
Implementation note: It is RECOMMENDED that applications don't use the ULP Buffer addresses used for Atomic Operations for other RDMA operations, because there are no atomicity guarantees between operations other than Atomic Operations 🚫
Implementation note: Errors related to alignment in the following sections cover Atomic Operations targeted at a ULP Buffer address that isn't aligned to a 64-bit boundary 📏
Atomic Operation Request Messages use the same remote addressing mechanism as RDMA Reads and Writes. The ULP Buffer address in the request is in the address space of the Remote Peer the Atomic Operation is targeted at 🎯
Atomic Operation Response Messages MUST use the Untagged Buffer model with QN=3. Queue number 3 is used to track outstanding Atomic Operation Request messages at the Requester. When an Atomic Operation Response message arrives, the Message Sequence Number (MSN) is used to find the matching Atomic Operation request so the Atomic Operation request can be completed 🎀
5.1. Atomic Operation Details
the following subsections go into more detail on the Atomic Operations 🔍
5.1.1. FetchAdd ✨
okay so FetchAdd is the Atomic op where the Requester tells the Responder to grab a 64-bit Original Remote Data Value sitting at a 64-bit aligned ULP Buffer address 💖. The Responder reads it, does a FetchAdd across several fields of whatever length you pick (set by a 64-bit "Add Mask"), and writes the result back to that same address 🎀. The addition happens separately on each field. A bit set in the Add Mask marks a field boundary, so each field gets a bit at its most significant position, and any carry that comes out of that spot just gets thrown away when the add happens 🦋.
FetchAdd Atomic Operations MUST target ULP Buffer addresses that are 64-bit aligned. FetchAdd Atomic Operations that target ULP Buffer addresses that are not 64-bit aligned MUST be surfaced as errors, and the Responder's memory MUST NOT be modified in those cases. On top of that, an error MUST be surfaced and a terminate message MUST be generated 🚨. Setting the Add Mask field to 0x0000000000000000 gives you a plain Atomic Add of the 64-bit Original Remote Data Value and the 64-bit "Add Data" 🌸.
here's the pseudocode for a masked FetchAdd, bestie 🧚:
bit_location = 1
carry = 0
Remote Data Value = 0
for bit = 0 to 63
{
if (bit != 0 ) bit_location = bit_location << 1
val1 = (Original Remote Data Value & bit_location) >> bit
val2 = (Add Data & bit_location) >> bit
sum = carry + val1 + val2
carry = (sum & 2) >> 1
sum = sum & 1
if (sum)
Remote Data Value |= bit_location
carry = ((carry) && (!(Add Mask & bit_location)))
}The FetchAdd operation runs in the endian format of the target memory 🌈. The "Original Remote Data Value" gets converted from the target memory's endian format before it's returned to the Requester, but the fields are in big-endian format on the wire 💜.
The Requester specifies:
Remote STagRemote Tagged OffsetAdd DataAdd Mask
The Responder returns:
Original Remote Data
5.1.2. CmpSwap 🔥
the CmpSwap Atomic Operation is the "only swap if it matches" move 👑. The Responder reads a 64-bit value at a 64-bit aligned ULP Buffer address in its memory, ANDs it with the 64-bit Compare Mask from the Atomic Operation Request header, and then compares that to the result of ANDing the Compare Mask with the Compare Data from the same header. If the two values match, the Responder is required to swap the masked bits at that address with the masked Swap Data 😭. If the masked compare values don't match, the Responder's memory stays exactly as it was 🙈. Either way, the original value it read gets converted from the target memory's endian format and sent back to the Requester. The fields are in big-endian format on the wire 🍓.
The Requester specifies:
Remote STagRemote Tagged OffsetSwap DataSwap MaskCompare DataCompare Mask
The Responder returns:
Original Remote Data Value
here's the masked CmpSwap result in pseudocode ✨:
if (!((Compare Data ^ Original Remote Data Value) &
Compare Mask))
then
Remote Data Value =
(Original Remote Data Value & ~(Swap Mask))
| (Swap Data & Swap Mask)
else
Remote Data Value = Original Remote Data ValueAfter the operation, the remote data Buffer MUST contain the "Original Remote Data Value" if the comparison didn't match, or the masked "Swap Data" if it did. CmpSwap Atomic Operations MUST target ULP Buffer addresses that are 64-bit aligned 💅.
If a CmpSwap Atomic Operation is attempted on a target ULP Buffer address that is not 64-bit aligned:
- The operation MUST NOT be performed,
- The Responder's memory MUST NOT be modified,
- The result MUST be surfaced as an error, and
- A terminate message MUST be generated. (See Section 8.2 for the contents of the terminate message.)
⚠️ bestie, that alignment rule is not optional. Both FetchAdd and CmpSwap need it, and getting it wrong means errors and terminate messages 😱.
5.2. Atomic Operations 💜
The Atomic Operation Request and Response are RDMA Messages 🦋. An Atomic Operation uses the DDP Untagged Buffer Model. Atomic Operation Request messages MUST use the same Queue Number as RDMA Read Requests (QN=1). Reusing that Queue Number means Atomic Requests get to ride on the same infrastructure as RDMA Read Requests, like the Outbound and Inbound RDMA Read Queue Depth (ORD/IRD) flow control 🫶. Atomic Operation Response messages MUST set Queue Number (QN) to 3 in the DDP header 🎀.
The RDMA Message OpCode for an Atomic Request Message is 1010b. The RDMA Message OpCode for an Atomic Response Message is 1011b ✨.
5.2.1. Atomic Operation Request Message ✨
okay so the Atomic Operation Request Message is the one that carries an Atomic Operation Header describing the ULP Buffer address in the Responder's memory 💖 The header sits right after the DDP header, and the RDMAP layer hands the DDP layer an RDMAP Control Field to go with it. Here's the layout for every Atomic Operation Request Message 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved (Not Used) |AOpCode|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Request Identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Remote STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Remote Tagged Offset |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Add or Swap Data |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Add or Swap Mask |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Compare Data |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Compare Mask |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 4: Atomic Operation Request Header
Reserved (Not Used): 28 bits
This one gets set to zero when sending and simply ignored when receiving. Nothing to see here, bestie 🫶
Atomic Operation Code (AOpCode): 4 bits.
Check Figure 5 for the full list. Every Atomic Operation Code in Figure 5 MUST be implemented by an RNIC that supports Atomic Operations, so no skipping any 🔥
Request Identifier: 32 bits.
This is the number that identifies the Atomic Operation Request Message. The sending RNIC picks the value, and it comes back to the Local Peer in the Atomic Operation Response message, so it's like a little receipt 🧾✨
Remote STag: 32 bits.
The Remote STag points to the Remote Peer's Tagged Buffer that the Atomic Operation is aiming at. How the STag gets tied to the RDMAP Stream is a mechanism outside the RDMAP specification, so that's a different department 💅
Remote Tagged Offset: 64 bits.
This is the starting offset, in octets, from the base of the Remote Peer's Tagged Buffer that the Atomic Operation targets. The offset MAY start at an arbitrary position, but it MUST represent a ULP Buffer address that is 64-bit aligned. So it can start anywhere, but the address itself has to be 64-bit aligned, no exceptions 🌸
Add or Swap Data: 64 bits.
This field holds the 64-bit "Add Data" value for an Atomic FetchAdd Operation, or the 64-bit "Swap Data" value for an Atomic Swap or CmpSwap Operation. Same slot, different job depending on the op 🧚
Add or Swap Mask: 64 bits
This one is used in the masked Atomic Operations (FetchAdd and CmpSwap) to do a bitwise logical AND, exactly as those operations define it. For the non-masked Atomic Operation (Swap), it's set to ffffffffffffffffh on transmit and the receiver ignores it 🎀
Compare Data: 64 bits.
The Compare Data field carries the 64-bit "Compare Data" value for an Atomic CmpSwap Operation. For Atomic Operations FetchAdd and Atomic Swap, it's set to zero on transmit and ignored by the receiver 💜
Compare Mask: 64 bits
This field is used in the masked Atomic Operation CmpSwap, again doing a bitwise logical AND as that operation defines. For Atomic Operations FetchAdd and Swap, it's set to ffffffffffffffffh on transmit and ignored by the receiver 🦋
---------+-----------+----------+----------+---------+---------
Atomic | Atomic | Add or | Add or | Compare | Compare
Operation| Operation | Swap | Swap | Data | Mask
Code | | Data | Mask | |
---------+-----------+----------+----------+---------+---------
0000b | FetchAdd | Add Data | Add Mask | N/A | N/A
---------+-----------+----------+----------+---------+---------
0010b | CmpSwap | Swap Data| Swap Mask| Valid | Valid
---------+-----------+-----------------------------------------Figure 5: Atomic Operation Message Definitions
Okay, so here's what the Atomic Operation Request Message is supposed to do, step by step 👑
- An Atomic Operation Request Message MUST reference an Untagged Buffer. That is, the Local Peer's RDMAP layer MUST request that the DDP mark the Message as Untagged.
- One Atomic Operation Request Message MUST consume one Untagged Buffer.
- The Responder's RDMAP layer MUST process an Atomic Operation Request Message. A valid Atomic Operation Request Message MUST NOT be delivered to the Responder's ULP (i.e., it is processed by the RDMAP layer). Main character energy, the RDMAP layer handles it all 💅
- At the Responder, an error MUST be surfaced in response to delivery to the Remote Peer's RDMAP layer of an Atomic Operation Request Message with an Atomic Operation Code that the RNIC does not support. 😭
- An Atomic Operation Request Message MUST reference the RDMA Read Request Queue. That is, the Requester's RDMAP layer MUST request that the DDP layer set the Queue Number field to one.
- The Requester MUST pass to the DDP layer Atomic Operation Request Messages in the order they were submitted by the ULP.
- The Responder MUST process the Atomic Operation Request Messages in the order they were sent. Order matters, periodt 🌈
- If the Responder receives a valid Atomic Operation Request Message, it MUST respond with a valid Atomic Operation Response Message. ✨
5.2.2. Atomic Operation Response Message 🦋
The Atomic Operation Response Message carries an Atomic Operation Response Header holding the "Original Request Identifier" and the "Original Remote Data Value". Like the request, this header comes right after the DDP header, and the RDMAP layer passes the DDP layer an RDMAP Control Field. The layout is below 🌸
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Original Request Identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Original Remote Data Value |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 6: Atomic Operation Response Header
Original Request Identifier: 32 bits.
This one is set to whatever was in the Request Identifier field of the Atomic Operation Request Message it's answering. It's the matching ticket 🎟️
Original Remote Data Value: 64 bits.
This is the original 64-bit value that was stored at the ULP Buffer address the Atomic Operation targeted. Basically, what the value was before the op touched it 👀
The Atomic Operation Response Message has these semantics:
- The Atomic Operation Response Message for the associated Atomic Operation Request Message travels in the opposite direction. Reverse uno card 🔄
- An Atomic Operation Response Message MUST consume an Untagged Buffer. That is, the Responder RDMAP layer MUST request that the DDP mark the Message as Untagged.
- An Atomic Operation Response Message MUST reference the Queue Number 3. That is, the Responder's RDMAP layer MUST request that the DDP layer set the Queue Number field to 3.
- The Responder MUST ensure that a sufficient number of Untagged Buffers are available on the RDMA Read Request Queue (Queue with DDP Queue Number 1) to support the maximum number of Atomic Operation Requests negotiated by the ULP in addition to the maximum number of RDMA Read Requests negotiated by the ULP. Buffer planning is everything 🧠
- The Requester MUST ensure that a sufficient number of Untagged Buffers are available on the RDMA Atomic Response Queue (Queue with DDP Queue Number 3) to support the maximum number of Atomic Operation Requests negotiated by the ULP.
- The RDMAP layer MUST Deliver the Atomic Operation Response Message to the ULP. 💌
- At the Requester, when an invalid Atomic Operation Response Message is delivered to the Remote Peer's RDMAP layer, an error is surfaced. 🚨
- When the Responder receives Atomic Operation Request messages, the Responder RDMAP layer MUST pass Atomic Operation Response Messages to the DDP layer, in the order that the Atomic Operation Request Messages were received by the RDMAP layer, at the Responder. Same order in, same order out, iconic ✨
5.3. Atomicity Guarantees 🔒
Okay, the big promise here: the Read-Modify-Write (RMW) that the Atomic Operation does on the Responder's node MUST be assured, even when other RDMAP Streams on the same RNIC are making concurrent atomic accesses. No sneaking in and messing up someone else's update, we don't do that here 💖
5.4. Atomic Operations Ordering and Completion Rules
okay bestie, this is the ordering and completion rulebook for Atomic Operations 💅 On top of the rules in RFC 5040, these apply to any implementation doing Atomic Operations 🔥
- Requester, for an Atomic Operation you MUST NOT count the Tagged Buffer at the Responder as modified by that specific Atomic Operation until the Atomic Operation Response Message has been Delivered to RDMAP at the Requester 🙈
- Atomicity guarantees MUST be provided within the scope of a single RNIC ✨
Implementation Note: This atomicity requirement among operations only covers a single RNIC 🦋 Atomicity guarantees are OPTIONAL when the Tagged Buffer is accessed by any method other than an Atomic Operation through that same RNIC. Accesses that may not be atomic with respect to an Atomic Operation include accesses through other RNICs and local processor memory access to the Tagged Buffer 💜
- Atomic Operation Request Messages MUST NOT start processing at the Responder until they have been Delivered to RDMAP by DDP 🧚
- Atomic Operation Response Messages MAY be generated at the Responder after subsequent RDMA Write Messages or Send Messages have been Placed or Delivered 🌸
- Atomic Operation Response Message processing at the Responder MUST be started only after the Atomic Operation Request Message has been Delivered by the DDP layer (so all previous RDMA Messages on that DDP Stream have been Delivered too) 👑
- Send Messages MAY be Completed at the Responder before prior incoming Atomic Operation Request Messages have completed their response processing 🫶
- An Atomic Operation MUST NOT be Completed at the Requester until the DDP layer Delivers the associated incoming Atomic Operation Response Message 😭
- If more than one outstanding Atomic Request Message is supported by both peers, the Atomic Operation Request Messages MUST be processed in the order they were delivered by the DDP layer on the Responder. Atomic Operation Response Messages MUST be submitted to the DDP layer on the Responder in the order the Atomic Operation Request Messages were Delivered by DDP ✨
6. Immediate Data 🎀
The Immediate Data operation is typically used together with an RDMA Write Operation to improve ULP processing efficiency 💖 The efficiency comes from causing an RDMA Completion to be generated right after the RDMA Write operation. That RDMA Completion delivers 8 bytes of Immediate Data at the Remote Peer 🍓 The combination of an RDMA Write Message followed by an Immediate Data Operation has the same behavior as the RDMA Write with Immediate Data operation found in InfiniBand. An Immediate Data operation that is not preceded by an RDMA Write operation causes an RDMA Completion 🌈
6.1. RDMAP Interactions with ULP for Immediate Data
For Immediate Data operations, here are the interactions between the RDMAP Layer and the ULP 🧁
- At the Data Source:
- The ULP passes to the RDMAP Layer the following:
- 8 bytes of ULP Immediate Data
- When the Immediate Data operation Completes, an indication of the Completion results is passed along 💅
- The ULP passes to the RDMAP Layer the following:
- At the Data Sink:
- If the Immediate Data operation is Completed successfully, the RDMAP Layer passes the following information to the ULP Layer:
- 8 bytes of Immediate Data
- An Event, if the Data Sink is configured to generate an Event
- If the Immediate Data operation is Completed in error, the Data Sink RDMAP Layer passes the corresponding error information up to the Data Sink ULP and sends a Terminate Message to the Data Source RDMAP Layer. The Data Source RDMAP Layer then passes the Terminate Message up to the ULP 😭
- If the Immediate Data operation is Completed successfully, the RDMAP Layer passes the following information to the ULP Layer:
6.2. Immediate Data Header Format
The Immediate Data and Immediate Data with SE Messages carry Immediate Data as shown in Figure 7 🦋 The RDMAP layer passes to the DDP layer an RDMAP Control Field and 8 bytes of Immediate Data. The first 8 bytes of the data following the DDP header contain the Immediate Data. See Appendix A.3 for the DDP segment format of an Immediate Data or Immediate Data with SE Message 📚
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Immediate Data |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 7: Immediate Data or Immediate Data with SE Message Header
Immediate Data: 64 bits. 8 bytes of data transferred from the Data Source to an untagged Buffer at the Data Sink 🌸
6.3. Immediate Data or Immediate Data with SE Message
The Immediate Data or Immediate Data with SE Message uses the DDP Untagged Buffer Model to transfer Immediate Data from the Data Source to the Data Sink 💜
- An Immediate Data or Immediate Data with SE Message MUST reference an Untagged Buffer. That is, the Local Peer's RDMAP Layer MUST request that the DDP layer mark the Message as Untagged 🧚
- One Immediate Data or Immediate Data with SE Message MUST consume one Untagged Buffer 🎀
- At the Remote Peer, the Immediate Data and Immediate Data with SE Messages MUST be Delivered to the Remote Peer's ULP in the order they were sent 🍓
- For an Immediate Data or Immediate Data with SE Message, the Local Peer's RDMAP Layer MUST request that the DDP layer set the Queue Number field to zero 🌈
- For an Immediate Data or Immediate Data with SE Message, the Local Peer's RDMAP Layer MUST request that the DDP layer transmit 8 bytes of data 💅
- The Local Peer MUST issue Immediate Data and Immediate Data with SE Messages in the order they were submitted by the ULP 🫶
- The Remote Peer MUST check that Immediate Data and Immediate Data with SE Messages include exactly 8 bytes of data from the DDP layer. The DDP header carries the length field that is reported by the DDP layer 🔥
6.4. Ordering and Completions
Ordering and completion rules for Immediate Data are the same as those for a Send operation, as described in Section 5.5 of RFC 5040 ✨
7. Ordering and Completions Table ✨
okay so this is the big cheat sheet 🫶 it lays out the ordering relationships for Atomic and Immediate Data operations, seen from the Local Peer that's issuing the Operations. quick heads up: in this table, Send is the whole Send family (Send, Send with Invalidate, Send with Solicited Event, and Send with Solicited Event and Invalidate) 💅 and Immediate Data covers both Immediate Data and Immediate Data with Solicited Event 🦋
---------+----------+-------------+-------------+------------------
First | Second | Placement | Placement | Ordering
Operation| Operation| Guarantee at| Guarantee at| Guarantee at
| | Remote Peer | Local Peer | Remote Peer
---------+----------+-------------+-------------+------------------
Immediate| Send | No Placement| Not | Completed in
Data | | Guarantee | Applicable | Order
| | between Send| |
| | Payload and | |
| | Immediate | |
| | Data | |
---------+----------+-------------+-------------+------------------
Immediate| RDMA | No Placement| Not | Not
Data | Write | Guarantee | Applicable | Applicable
| | between RDMA| |
| | Write | |
| | Payload and | |
| | Immediate | |
| | Data | |
---------+----------+-------------+-------------+------------------
Immediate| RDMA | No Placement| RDMA Read | RDMA Read
Data | Read | Guarantee | Response | Response
| | between | will not be | Message will
| | Immediate | Placed until| not be
| | Data and | Immediate | generated
| | RDMA Read | Data is | until
| | Request | Placed at | Immediate Data
| | | Remote Peer | has been
| | | | Completed
---------+----------+-------------+-------------+------------------
Immediate| Atomic | No Placement| Atomic | Atomic
Data | | Guarantee | Response | Response
| | between | will not be | Message will
| | Immediate | Placed until| not be
| | Data and | Immediate | generated
| | Atomic | Data is | until
| | Request | Placed at | Immediate Data
| | | Remote Peer | has been
| | | | Completed
---------+----------+-------------+-------------+------------------
Immediate| Immediate| No Placement| Not | Completed in
Data or | Data | Guarantee | Applicable | Order
Send | | | |
---------+----------+-------------+-------------+------------------
RDMA | Immediate| No Placement| Not | Immediate Data
Write | Data | Guarantee | Applicable | is Completed
| | | | after RDMA
| | | | Write is Placed
| | | | and Delivered
---------+----------+-------------+-------------+------------------
RDMA Read| Immediate| No Placement| Immediate | Not Applicable
| Data | Guarantee | Data MAY be |
| | between | Placed |
| | Immediate | before |
| | Data and | RDMA Read |
| | RDMA Read | Response is |
| | Request | generated |
---------+----------+-------------+-------------+------------------
Atomic | Immediate| No Placement| Immediate | Not Applicable
| Data | Guarantee | Data MAY be |
| | between | Placed |
| | Immediate | before |
| | Data and | Atomic |
| | Atomic | Response is |
| | Request | generated |
---------+----------+-------------+-------------+------------------
Atomic | Send | No Placement| Send Payload| Not Applicable
| | Guarantee | MAY be |
| | between Send| Placed |
| | Payload and | before |
| | Atomic | Atomic |
| | Request | Response is |
| | | generated |
---------+----------+-------------+-------------+------------------
Atomic | RDMA | No Placement| RDMA Write | Not
| Write | Guarantee | Payload MAY | Applicable
| | between RDMA| be Placed |
| | Write | before |
| | Payload and | Atomic |
| | Atomic | Response is |
| | Request | generated |
---------+----------+-------------+-------------+------------------
Atomic | RDMA | No Placement| No Placement| RDMA Read
| Read | Guarantee | Guarantee | Response
| | between | between | Message will
| | Atomic | Atomic | not be
| | Request and | Response | generated
| | RDMA Read | and RDMA | until Atomic
| | Request | Read | Response Message
| | | Response | has been
| | | | generated
---------+----------+-------------+-------------+------------------
Atomic | Atomic | Placed in | No Placement| Second Atomic
| | order | Guarantee | Request
| | | between two | Message will
| | | Atomic | not be
| | | Responses | processed
| | | | until first
| | | | Atomic Response
| | | | has been
| | | | generated
---------+----------+-------------+-------------+------------------
Send | Atomic | No Placement| Atomic | Atomic Response
| | Guarantee | Response | Message will not
| | between Send| will not be | be generated
| | Payload and | Placed at | until Send has
| | Atomic | the Local | been Completed
| | Request | Peer until |
| | | Send Payload|
| | | is Placed |
| | | at the |
| | | Remote Peer |
---------+----------+-------------+-------------+------------------
RDMA | Atomic | No Placement| Atomic | Not
Write | | Guarantee | Response | Applicable
| | between RDMA| will not be |
| | Write | Placed at |
| | Payload and | the Local |
| | Atomic | Peer until |
| | Request | RDMA Write |
| | | Payload |
| | | is Placed |
| | | at the |
| | | Remote Peer |
---------+----------+-------------+-------------+------------------
RDMA | Atomic | No Placement| No Placement| Atomic Response
Read | | Guarantee | Guarantee | Message will
| | between | between | not be generated
| | Atomic | Atomic | until RDMA
| | Request and | Response | Read Response
| | RDMA Read | and RDMA | has been
| | Request | Read | generated
| | | Response |
---------+----------+-------------+-------------+------------------8. Error Processing 🔥
so error handling 🙈 the stuff in Section 7 of RFC 5040 still applies, and then these extra rules kick in for the new RDMA Messages defined in this spec. let's gooo
8.1. Errors Detected at the Local Peer
the Local Peer MUST send a Terminate Message for each of the following cases:
- for errors found while creating an Atomic Request, Atomic Response, Immediate Data, or Immediate Data with SE Message, or for other reasons not directly tied to an incoming Message, the Terminate Message and Error code get sent instead of the Message. in this case the Error Type and Error Code fields are included in the Terminate Message, but the Terminated DDP Header and Terminated RDMA Header fields are set to zero 💀
- for errors found on an incoming Atomic Request, Atomic Response, Immediate Data, or Immediate Data with SE (after the Message has been Delivered by DDP), the Terminate Message is sent at the earliest possible opportunity, preferably in the next outgoing RDMA Message 🦋 in this case the Error Type, Error Code, and Terminated DDP Header fields are included in the Terminate Message, but the Terminated RDMA Header field is set to zero.
8.2. Errors Detected at the Remote Peer
on incoming Atomic Requests, Atomic Responses, Immediate Data, and Immediate Data with Solicited Event, the following MUST be validated:
- The DDP layer MUST validate all DDP Segment fields.
- The RDMA OpCode MUST be valid.
- The RDMA Version MUST be valid.
on incoming Atomic requests the following additional validation MUST be performed:
- The RDMAP layer MUST validate that the Remote Peer's Tagged ULP Buffer address references a ULP Buffer address that is 64-bit aligned. In the case of an error, the RDMAP layer MUST generate a Terminate Message indicating RDMA Layer Remote Operation Error with Error Code Name "Catastrophic error, localized to RDMAP Stream" as described in Section 4.8 of RFC 5040. Implementation Note: A ULP implementation can avoid this error by having the target ULP Buffer of an Atomic Operation 64-bit aligned. 🌸
9. Security Considerations 🔒✨
okay bestie, this part is SERIOUS 😭 but also kind of fascinating. This document extends the RDMA Protocol from RFC 5040, so all the Security Considerations in Section 8 of RFC 5040 apply here too 💅. The big one: Atomic Operations use ULP Buffer addresses for the Remote Peer Buffer addressing from RFC 5040, as required by the security model in RFC 5042 [RFC5042] 🔐.
RDMAP and related protocols can be used by apps with really distinctive traffic patterns, like message timing, source, destination, and size. Structured high-performance computing apps built on the MPI interface are a prime example 🧚. For those apps, just analyzing encrypted traffic could leak sensitive stuff: what the app is, how big the data set is, and how fast it's progressing 👀. The fix is ESP version 3 (ESPv3) Traffic Flow Confidentiality [RFC4303], which obfuscates the encrypted traffic's characteristics and hides that info from passive observers 🦋. The ESPv3 implementation requirements for RDMAP are in [RFC7146] 💜.
10. IANA Considerations 📋🌸
IANA added these entries to the "RDMAP Message Operation Codes" registry of the "Remote Direct Data Placement (RDDP)" registry:
0x8, Immediate Data, this specification
0x9, Immediate Data with Solicited Event, this specification
0xA, Atomic Request, this specification
0xB, Atomic Response, this specificationThey also added a brand new registry to the "Remote Direct Data Placement (RDDP)" registry. The next section explains the registry, its starting contents, and how it gets administered in more detail 🎀.
10.1. RDMAP Message Atomic Operation Subcodes 💖
Name of the registry: "RDMAP Message Atomic Operation Subcodes"
Namespace details: RDMAP Message Atomic Operation Subcodes are 4-bit values.
Information that must be provided to assign a new value: An IESG-approved Standards Track specification defining the semantics and interoperability requirements of the proposed new value and the fields to be recorded in the registry.
Fields to record in the registry: RDMAP Message Atomic Operation Subcode, Atomic Operation, RFC Reference.
Initial registry contents:
0x0, FetchAdd, this specification
0x1, Reserved, this specification
0x2, CmpSwap, this specificationNote: An experimental RDMAP Message Operation Code has already been allocated, so there's no need for an experimental RDMAP Message Atomic Operation Subcode 🙈.
All other values are Unassigned and available to IANA for assignment. New RDMAP Message Atomic Operation Subcodes should be assigned sequentially, so that implementations processing RDMAP Message Atomic Operations in hardware are better supported 🔥.
Allocation Policy: Standards Action [RFC5226]
10.2. RDMAP Queue Numbers 🍓
Name of the registry: "RDMAP DDP Untagged Queue Numbers"
Namespace details: RDMAP DDP Untagged Queue numbers are 32-bit values.
Information that must be provided to assign a new value: An IESG-approved Standards Track specification defining the semantics and interoperability requirements of the proposed new value and the fields to be recorded in the registry.
Fields to record in the registry: RDMAP DDP Untagged Queue Numbers, Queue Usage Description, RFC Reference.
Initial registry contents:
0x00000000, Queue 0 (Send operation Variants), [RFC5040]
0x00000001, Queue 1 (RDMA Read Request operations), [RFC5040]
0x00000002, Queue 2 (Terminate operations), [RFC5040]
0x00000003, Queue 3 (Atomic Response operations), this specificationNote: An experimental RDMAP Message Operation Code has already been allocated, so there's no need for an experimental RDMAP DDP Untagged Queue Number ⭐.
All other values are Unassigned and available to IANA for assignment. New RDMAP queue numbers should be assigned sequentially, so that implementations doing RDMAP queue selection in hardware are better supported 🧚.
Allocation Policy: Standards Action [RFC5226]
11. References 📚
11.1. Normative References 💜
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC4303] Kent, S., "IP Encapsulating Security Payload (ESP)", RFC
4303, December 2005.
[RFC5040] Recio, R., Metzler, B., Culley, P., Hilland, J., and D.
Garcia, "A Remote Direct Memory Access Protocol
Specification", RFC 5040, October 2007.
[RFC5041] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
Data Placement over Reliable Transports", RFC 5041,
October 2007.
[RFC5042] Pinkerton, J. and E. Deleganes, "Direct Data Placement
Protocol (DDP) / Remote Direct Memory Access Protocol
(RDMAP) Security", RFC 5042, October 2007.
[RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an
IANA Considerations Section in RFCs", BCP 26, RFC 5226,
May 2008.
[RFC7146] Black, D. and P. Koning, "Securing Block Storage Protocols
over IP: RFC 3723 Requirements Update for IPsec v3", RFC
7146, April 2014.11.2. Informative References 🦋
[DAT_ATOMICS]
DAT Collaborative, "IB Transport Specific Extensions for
DAT 2.0", User Direct Access Programming Library,
<http://www.datcollaborative.org/DAT_IB_Extensions.pdf>.
[IB] InfiniBand Trade Association, "InfiniBand Architecture
Specification Volumes 1 and 2", Release 1.1, November
2002, <http://www.infinibandta.org/specs>.
[MPI] Message Passing Interface Forum, "MPI: A Message-Passing
Interface Standard, Version 3.0", September 2012,
<http://www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf>.
[OFAVERBS] Rosenstock, H., "Subject: Re: [PATCH 0/2] Add support for
enhanced atomic operations", message to the linux-rdma
mailing list,
<http://www.spinics.net/lists/linux-rdma/msg02405.html>.
[RFC5044] Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
Carrier, "Marker PDU Aligned Framing for TCP
Specification", RFC 5044, October 2007.
[RFC5045] Bestler, C., Ed., and L. Coene, "Applicability of Remote
Direct Memory Access Protocol (RDMA) and Direct Data
Placement (DDP)", RFC 5045, October 2007.
[RFC6581] Kanevsky, A., Ed., Bestler, C., Ed., Sharp, R., and S.
Wise, "Enhanced Remote Direct Memory Access (RDMA)
Connection Establishment", RFC 6581, April 2012.
[RSOCKETS] Hefty, S., "RDMA CM - RDMA enabled Sockets library for
Open Fabrics", <http://git.openfabrics.org/?p=~shefty/
librdmacm.git;a=summary>.12. Acknowledgments 🫶
The authors want to thank these lovely people for their valuable comments and suggestions 🌈:
- David Black
- Arkady Kanevsky
- Bernard Metzler
- Jim Pinkerton
- Tom Talpey
- Steve Wise
- Don Wood
Appendix A. DDP Segment Formats for RDMA Messages 🎀
NOT part of the standard, bestie. This appendix is for information only, and it just depicts the DDP Segment format for the various RDMA Messages 📖.
A.1. DDP Segment for Atomic Operation Request 💅
The following figure depicts an Atomic Operation Request, DDP Segment:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved (Not Used) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Atomic Operation Request) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Atomic Operation Request) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Atomic Operation Request) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved (Not Used) |AOpCode|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Request Identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Remote STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Remote Tagged Offset |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Add or Swap Data |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Add or Swap Mask |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Compare Data |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Compare Mask |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+A.2. DDP Segment for Atomic Response 🦋
The following figure depicts an Atomic Operation Response, DDP Segment:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved (Not Used) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Atomic Operation Request) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Atomic Operation Request) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Atomic Operation Request) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Original Request Identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Original Remote Value |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+A.3. DDP Segment for Immediate Data and Immediate Data with SE ✨
okay bestie, this is the last little packet diagram 🦋 it's the segment for Immediate Data and Immediate Data with SE. same vibe as before, just with its own fields 💅
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved (Not Used) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Immediate Data |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Authors' Addresses 🎀
Hemal Shah Broadcom Corporation 5300 California Avenue Irvine, CA 92617 US Phone: 1-949-926-6941 EMail: hemal@broadcom.com
Felix Marti Chelsio Communications, Inc. 370 San Aleso Ave. Sunnyvale, CA 94085 US Phone: 1-408-962-3600 EMail: felix@chelsio.com
Asgeir Eiriksson Chelsio Communications, Inc. 370 San Aleso Ave. Sunnyvale, CA 94085 US Phone: 1-408-962-3600 EMail: asgeir@chelsio.com
Wael Noureddine Chelsio Communications, Inc. 370 San Aleso Ave. Sunnyvale, CA 94085 US Phone: 1-408-962-3600 EMail: wael@chelsio.com
Robert Sharp Intel Corporation 1300 South Mopac Expy, Mailstop: AN4-4B Austin, TX 78746 US Phone: 1-512-362-1407 EMail: robert.o.sharp@intel.com
that's the whole spec, bestie!! 🌈👑 the squad of authors who made this happen is iconic 🫶 thanks for hanging with me through all 8 parts 💖
📦 where this page comes from
- ietf: rfc7306 2014-06
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms