RFC 5044
Marker PDU Aligned Framing for TCP Specification
Marker PDU Aligned Framing for TCP Specification 🎀
okay bestie, this is RFC 5044, and it's giving standards track 💅 It's from the Network Working Group and was published in October 2007. The authors are P. Culley (Hewlett-Packard Company), U. Elzur (Broadcom Corporation), R. Recio (IBM Corporation), S. Bailey (Sandburst Corporation), and J. Carrier (Cray Inc.). The header doesn't list anything it obsoletes or updates, so it's just doing its own thing 🦋
Status of This Memo 📜
so this doc is an Internet standards track protocol for the Internet community, and it wants discussion and suggestions for improvements 🫶 If you want to know where this protocol stands in the official standardization state, check the current edition of the "Internet Official Protocol Standards" (STD 1). And the memo can be distributed with no limits, which is so generous of them 🌈
Abstract ✨
Marker PDU Aligned Framing (MPA) is designed to work as an "adaptation layer" between TCP and the Direct Data Placement protocol (DDP) as described in RFC 5041. It keeps TCP's reliable, in-order delivery, and it also adds the preservation of higher-level protocol record boundaries that DDP needs 🔥 MPA is fully compliant with applicable TCP RFCs and works with existing TCP implementations, so no drama there 💖 It can also be used in integrated implementations that combine TCP, MPA, and DDP. The goal is less buffering and better performance at the system level 🧚
Table of Contents 🗂️
Table of Contents
1. Introduction ....................................................4
1.1. Motivation .................................................4
1.2. Protocol Overview ..........................................5
2. Glossary ........................................................8
3. MPA's Interactions with DDP ....................................11
4. MPA Full Operation Phase .......................................13
4.1. FPDU Format ...............................................13
4.2. Marker Format .............................................14
4.3. MPA Markers ...............................................14
4.4. CRC Calculation ...........................................16
4.5. FPDU Size Considerations ..................................21
5. MPA's interactions with TCP ....................................22
5.1. MPA transmitters with a standard layered TCP ..............22
5.2. MPA receivers with a standard layered TCP .................23
6. MPA Receiver FPDU Identification ...............................24
7. Connection Semantics ...........................................24
7.1. Connection Setup ..........................................24
7.1.1. MPA Request and Reply Frame Format .................26
7.1.2. Connection Startup Rules ...........................28
7.1.3. Example Delayed Startup Sequence ...................30
7.1.4. Use of Private Data ................................33
7.1.4.1. Motivation ................................33
7.1.4.2. Example Immediate Startup Using
Private Data ..............................35
7.1.5. "Dual Stack" Implementations .......................37
7.2. Normal Connection Teardown ................................38
8. Error Semantics ................................................39
9. Security Considerations ........................................40
9.1. Protocol-Specific Security Considerations .................40
9.1.1. Spoofing ...........................................40
9.1.1.1. Impersonation .............................41
9.1.1.2. Stream Hijacking ..........................41
9.1.1.3. Man-in-the-Middle Attack ..................41
9.1.2. Eavesdropping ......................................42
9.2. Introduction to Security Options ..........................42
9.3. Using IPsec with MPA ......................................43
9.4. Requirements for IPsec Encapsulation of MPA/DDP ...........43
10. IANA Considerations ...........................................44
Appendix A. Optimized MPA-Aware TCP Implementations ...............45
A.1. Optimized MPA/TCP Transmitters ............................46
A.2. Effects of Optimized MPA/TCP Segmentation .................46
A.3. Optimized MPA/TCP Receivers ...............................48
A.4. Re-segmenting Middleboxes and Non-Optimized MPA/TCP
Senders ...................................................49
A.5. Receiver Implementation ...................................50
A.5.1. Network Layer Reassembly Buffers ...................51
A.5.2. TCP Reassembly Buffers .............................52
Appendix B. Analysis of MPA over TCP Operations ...................52
B.1. Assumptions ...............................................53
B.1.1. MPA Is Layered beneath DDP .........................53
B.1.2. MPA Preserves DDP Message Framing ..................53
B.1.3. The Size of the ULPDU Passed to MPA Is Less Than
EMSS Under Normal Conditions .......................53
B.1.4. Out-of-Order Placement but NO Out-of-Order Delivery.54
B.2. The Value of FPDU Alignment ...............................54
B.2.1. Impact of Lack of FPDU Alignment on the Receiver
Computational Load and Complexity ..................56
B.2.2. FPDU Alignment Effects on TCP Wire Protocol ........60
Appendix C. IETF Implementation Interoperability with RDMA
Consortium Protocols ..................................62
C.1. Negotiated Parameters ......................................63
C.2. RDMAC RNIC and Non-Permissive IETF RNIC ....................64
C.2.1. RDMAC RNIC Initiator ................................65
C.2.2. Non-Permissive IETF RNIC Initiator ..................65
C.2.3. RDMAC RNIC and Permissive IETF RNIC .................65
C.2.4. RDMAC RNIC Initiator ................................66
C.2.5. Permissive IETF RNIC Initiator ......................67
C.3. Non-Permissive IETF RNIC and Permissive IETF RNIC ..........67
Normative References ..............................................68
Informative References ............................................68
Contributors ......................................................70Table of Figures 🖼️
Table of Figures
Figure 1: ULP MPA TCP Layering .....................................5
Figure 2: FPDU Format .............................................13
Figure 3: Marker Format ...........................................14
Figure 4: Example FPDU Format with Marker .........................16
Figure 5: Annotated Hex Dump of an FPDU ...........................19
Figure 6: Annotated Hex Dump of an FPDU with Marker ...............20
Figure 7: Fully Layered Implementation ............................22
Figure 8: MPA Request/Reply Frame .................................26
Figure 9: Example Delayed Startup Negotiation .....................31
Figure 10: Example Immediate Startup Negotiation ..................35
Figure 11: Optimized MPA/TCP Implementation .......................45
Figure 12: Non-Aligned FPDU Freely Placed in TCP Octet Stream .....56
Figure 13: Aligned FPDU Placed Immediately after TCP Header .......58
Figure 14: Connection Parameters for the RNIC Types ...............63
Figure 15: MPA Negotiation between an RDMAC RNIC and a
Non-Permissive IETF RNIC ...............................65
Figure 16: MPA Negotiation between an RDMAC RNIC and a Permissive
IETF RNIC ..............................................66
Figure 17: MPA Negotiation between a Non-Permissive IETF RNIC and
a Permissive IETF RNIC .................................671. Introduction 🦋
okay so this section explains why MPA exists on top of TCP, plus a quick overview of the protocol 🧚
1.1. Motivation 💭
the Direct Data Placement protocol [DDP], when used with TCP [RFC793], needs a way to detect record boundaries 🫣 DDP records are called Upper Layer Protocol Data Units in this document. Finding the Upper Layer Protocol Data Unit (ULPDU) boundary is super useful for a hardware network adapter. It uses DDP to place the data straight into the application buffer, based on the control information in the ULPDU header, and it can do this without the packets needing to arrive in order 😭 The perks are avoiding the memory copy overhead and needing less memory to handle out-of-order or dropped packets. Iconic behavior 👑
there are lots of approaches to a generalized framing mechanism, and they're either probabilistic or deterministic 💅 An example of the probabilistic kind uses a detectable value embedded in the octet stream, but nothing stops that same value from showing up somewhere else in the user data. That's why it's probabilistic: under some conditions the receiver might misread application data as that detectable value, and then the protocol could fail more often than anyone would like 💀 The deterministic approach puts embedded controls at known locations in the octet stream. Since the receiver only ever looks at the spots that are known to hold an embedded control, it can never mistake application data for control data. For handling an out-of-order packet unambiguously, the deterministic approach is the one preferred 🌸
and that's the vibe: MPA gives DDP over TCP a framing mechanism that uses the deterministic approach 🎀 It lets you find the ULPDU in the TCP stream even when the TCP segments arrive out of order ✨
1.2. Protocol Overview ✨
okay so here's the layer cake 🎀 MPA sits right in the middle, and Figure 1 shows exactly how everything stacks:
+------------------+
| ULP client |
+------------------+ <- Consumer messages
| DDP |
+------------------+ <- ULPDUs
| MPA* |
+------------------+ <- FPDUs (containing ULPDUs)
| TCP* |
+------------------+ <- TCP Segments (containing FPDUs)
| IP etc. |
+------------------+
* These may be fully layered or optimized together.
Figure 1: ULP MPA TCP Layeringso MPA is an extra layer that lives above TCP and below DDP 💅 here's the operation sequence, step by step:
- A TCP connection gets set up because the ULP says so 🦋 the way it happens is totally out of scope for this spec. The ULP can send some data in streaming mode before it starts MPA, but it doesn't have to.
- The Consumer negotiates DDP and MPA at both ends of the connection ✨ this spec doesn't cover how that negotiation works. It might happen in streaming mode, or some other way (like a pre-arranged port number). ngl that's giving flexibility.
- The ULP turns on MPA at each end during the Startup Phase, and each end can be an Initiator or a Responder, whichever the ULP decides 👑 this is where MPA checks that it's being used, sets up CRC and Markers, and lets the ULP send some extra info through a Private Data exchange. For the full startup tea, see Section 7.1, Connection Setup.
- When the Startup Phase wraps up, the ULP puts MPA (and DDP) into Full Operation and starts sending DDP data 🌸 the DDP data chunks are called ULPDUs, and you can read about the DDP data over at [DDP].
Once MPA is in Full Operation, data transfer goes like this 💖
- DDP figures out the Maximum ULPDU (MULPDU) size by asking MPA for it 🧚 MPA gets that from TCP or IP when that info is available, or it picks a reasonable value.
- DDP makes ULPDUs of MULPDU size or smaller, then hands them to MPA at the sender 🍓
- MPA builds a Framed Protocol Data Unit (FPDU) by adding a header, optionally inserting Markers, and tacking a CRC field on after the ULPDU and PAD (if there is any). Then MPA delivers the FPDU to TCP ✨
- The TCP sender puts the FPDUs into the TCP stream 🫶 if the sender is optimized MPA/TCP, it segments the stream so that a TCP Segment boundary is also an FPDU boundary. TCP then passes each segment down to the IP layer for transmission.
- The receiver might be optimized or might not be 🌈 if it's optimized MPA/TCP, it can split up passing the TCP payload to MPA and passing the TCP payload ordering information to MPA. Either way, RFC-compliant TCP wire behavior shows up at both the sender and the receiver, so nobody's breaking the rules.
- The MPA receiver finds and assembles complete FPDUs in the stream, checks their integrity, and strips off MPA Markers (when they're there), ULPDU_Length, PAD, and the CRC field 🔥
- MPA then gives the complete ULPDUs to DDP. MPA can also split passing the MPA payload to DDP from passing the MPA payload ordering information 💜
A fully layered MPA on TCP is implemented as a data stream ULP for TCP, so it's RFC compliant. literally no notes 💅
An optimized DDP/MPA/TCP uses a TCP layer that might have some extra behaviors, which this document suggests 🦋 when DDP/MPA/TCP are cross-layer optimized, TCP's behavior (especially its sender segmentation) can shift from the un-optimized version. The changes still stay inside what the TCP RFC specifications allow, and they'll interoperate with an un-optimized TCP 🫶 the extra behaviors are in Appendix A and they are not normative 🙈 they're described at a TCP interface layer as a convenience. Implementations can get the described functionality any way they like, including cross-layer optimizations between TCP, MPA, and DDP ✨
An optimized DDP/MPA/TCP sender can segment the data stream so that TCP segments begin with FPDUs. This is called FPDU Alignment, and it's a big win for receivers 👑 when segments arrive with aligned FPDUs, the receiver usually doesn't need to buffer any part of the segment. That lets DDP put it straight into its destination memory, which skips copies from intermediate buffers (aka DDP's whole reason for existing 💀).
An optimized DDP/MPA/TCP receiver lets a DDP on MPA implementation find the start of ULPDUs that might show up out of order 🧚 it also lets the implementation figure out whether the entire ULPDU has arrived. Because of that, MPA can pass out-of-order ULPDUs to DDP for immediate use 🍓 this lets a DDP on MPA implementation save a ton of intermediate storage by dropping ULPDUs into the right spots in the application buffers as they arrive, instead of waiting until full ordering can be restored. main character energy 🌸
Out-of-order recovery is optional, and the receiver declares during startup whether it supports it 🎀 when the receiver says it doesn't support out-of-order recovery, the transmitter doesn't add the control information to the data stream that's needed for that recovery.
If the receiver is fully layered, MPA gets a strictly ordered stream of data and never has to deal with out-of-order ULPDUs 💖 in that case, MPA passes each ULPDU to DDP when the last bytes arrive from TCP, along with an indication that they're in order.
MPA implementations that support recovery of out-of-order ULPDUs MUST support a mechanism to indicate the ordering of ULPDUs as the sender transmitted them, and to indicate when missing intermediate segments arrive. These mechanisms let DDP put records back in order and report Delivery of complete messages (groups of records) 🫶
MPA also handles enhanced data integrity 🔥 some TCP users have noticed that the TCP checksum isn't as strong as anyone would like (see [CRCTCP]). Studies such as [CRCTCP] have shown that the TCP checksum flags segments in error at a much higher rate than the underlying link characteristics would suggest. With those higher error rates, the chance of an error slipping past detection, when only the TCP checksum is used for data integrity, becomes a real concern 😭 a stronger integrity check can reduce the chance of data errors getting missed.
MPA includes a CRC check to bring ULPDU data integrity up to the level other modern protocols provide, like SCTP [RFC4960] 💜 it's possible to disable this CRC check, but CRCs MUST be enabled unless it's clear that the end-to-end connection through the network has data integrity at least as good as MPA with CRC enabled (for example, when IPsec is implemented end to end) ✨ DDP's ULP expects this level of data integrity, so the ULP doesn't have to provide its own duplicate data integrity and error recovery for lost data. no notes, that's iconic 👑
2. Glossary ✨
okay bestie, before we get into the good stuff we need the vocab 🎀 the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this doc mean exactly what [RFC2119] says they mean 💅 no vibes-based interpretation allowed 😭
Consumer - the ULPs or applications that sit above MPA and DDP. The Consumer is the one in charge of making TCP connections, starting MPA and DDP connections, and generally running the show 👑
CRC - Cyclic Redundancy Check 🔁
Delivery - (Delivered, Delivers) - For MPA, Delivery means telling DDP that a particular PDU is ordered for use. A PDU is Delivered in the exact order the original sender sent it, and MPA uses TCP's byte stream ordering to figure out when Delivery is possible. this is totally different from "passing the PDU to DDP", which can happen in any order. the order of Delivery is strict, the order of passing is not 🦋
EMSS - Effective Maximum Segment Size. EMSS is the smaller of the TCP maximum segment size (MSS) from RFC 793 [RFC793] and the current path Maximum Transmission Unit (MTU) [RFC1191] 📏
FPDU - Framed Protocol Data Unit. It's the unit of data an MPA sender creates 📦
FPDU Alignment - the property that an FPDU is Header Aligned with the TCP segment, and the TCP segment includes a whole number of FPDUs. a TCP segment with FPDU Alignment can be processed right away, no waiting on other TCP segments to arrive and no combining with earlier segments. chef's kiss 🤌
FPDU Pointer (FPDUPTR) - this field of the Marker is used to indicate where an FPDU begins 📍
Full Operation (Full Operation Phase) - once the Startup Phase is done, MPA starts exchanging FPDUs 🚀
Header Alignment - the property that a TCP segment begins with an FPDU. The FPDU is Header Aligned when its FPDU header sits exactly at the start of the TCP segment (right behind the TCP headers on the wire) 🎯
Initiator - the endpoint of a connection that sends the MPA Request Frame, meaning the first one to actually send data. this might not be the same endpoint that sent the TCP SYN 🙈
Marker - a four-octet field placed in the MPA data stream at fixed octet intervals (every 512 octets) 🧚
MPA-aware TCP - a TCP implementation that knows about the receiver efficiencies of MPA FPDU Alignment and can send TCP segments that begin with an FPDU ✨
MPA-enabled - MPA is enabled if the MPA protocol is visible on the wire. When the sender is MPA-enabled, it's inserting framing and Markers. When the receiver is MPA-enabled, it's interpreting framing and Markers 👀
MPA Request Frame - data sent from the MPA Initiator to the MPA Responder during the Startup Phase 📨
MPA Reply Frame - data sent from the MPA Responder to the MPA Initiator during the Startup Phase 📬
MPA - Marker-based ULP PDU Aligned Framing for TCP protocol. This document defines the MPA protocol 💜
MULPDU - Maximum ULPDU. It's the current maximum size of the record that DDP is allowed to pass to MPA for transmission 📐
Node - a computing device attached to one or more links of a network. In this context, a Node does not mean a specific application or protocol instance running on the computer. A Node may consist of one or more MPA on TCP devices installed in a host computer 🖥️
PAD - a 1-3 octet group of zeros used to fill an FPDU up to an exact modulo 4 size 🫧
PDU - Protocol data unit 🍓
Private Data - a block of data exchanged between MPA endpoints during initial connection setup 🔐
Protection Domain - an RDMA concept (see [VERBS-RDMA] and [RDMASEC]) that ties use of various endpoint resources (memory access, etc.) to one specific RDMA/DDP/MPA connection 🛡️
RDDP - a suite of protocols including MPA, [DDP], [RDMAP], an overall security document [RDMASEC], a problem statement [RFC4297], an architecture document [RFC4296], and an applicability document [APPL]. the whole family 👩👧👦
RDMA - Remote Direct Memory Access, a protocol that uses DDP and MPA to let applications move data directly between memory buffers. See [RDMAP] 🌸
Remote Peer - the MPA protocol implementation on the opposite end of the connection. It's used to refer to the remote entity when describing protocol exchanges or other interactions between two Nodes 🤝
Responder - the connection endpoint that responds to an incoming MPA connection request (the MAP Request Frame). This may not be the endpoint that waited for the TCP SYN 🫡
Startup Phase - the initial exchanges of an MPA connection, which help the MPA endpoints get to know each other better and pass connection-specific setup info between them 🌅
ULP - Upper Layer Protocol. It's the protocol layer above the one currently being referenced. The ULP for MPA is DDP [DDP] 🍰
ULPDU - Upper Layer Protocol Data Unit. It's the data record defined by the layer above MPA (DDP). ULPDU corresponds to DDP's DDP segment 🧁
ULPDU_Length - a field in the FPDU describing the length of the included ULPDU 🔢
3. MPA's Interactions with DDP
okay so DDP needs MPA to keep DDP record boundaries intact from sender to receiver 🫶 when you're using MPA on TCP to send data, DDP hands MPA its records (ULPDUs). MPA then uses TCP's reliable transmission abilities to get the data across, and it slips extra info into the TCP stream so the MPA receiver can find where each record boundary is 🔍
so basically, MPA takes complete records (ULPDUs) from DDP at the sender and hands them back to DDP at the receiver. the whole thing is giving organized 📚
MPA MUST encapsulate the ULPDU such that there is exactly one ULPDU contained in one FPDU. no more, no less, one ULPDU per FPDU 🎀
MPA over a standard TCP stack can usually provide FPDU Alignment with the TCP Header if the FPDU is equal to TCP's EMSS. An optimized MPA/TCP stack can also keep alignment as long as the FPDU is less than or equal to TCP's EMSS. Since the receiver generally wants FPDU Alignment, DDP works with MPA to make sure FPDU lengths don't go past the EMSS under normal conditions. This is done with the MULPDU mechanism 🧠
MPA MUST provide information to DDP on the current maximum size of the record that is acceptable to send (MULPDU). DDP SHOULD limit each record size to MULPDU. The range of MULPDU values MUST be between 128 octets and 64768 octets, inclusive. bestie, that's the range, no going outside it 📏
The sending DDP MUST NOT post a ULPDU larger than 64768 octets to MPA. DDP MAY post a ULPDU of any size between one and 64768 octets; however, MPA is not REQUIRED to support a ULPDU Length that is greater than the current MULPDU. ngl the "not REQUIRED" part is so specific it's kind of iconic 💅
While the maximum theoretical length supported by the MPA header ULPDU_Length field is 65535, TCP over IP requires the IP datagram maximum length to be 65535 octets. To let MPA support FPDU Alignment, the maximum size of the FPDU has to fit within an IP datagram. So the ULPDU limit of 64768 octets was worked out by taking the maximum IP datagram length, subtracting the maximum total length of the IPv4 header, TCP header, IPv4 options, TCP options, and the worst-case MPA overhead, and then rounding the result down to a 128-octet boundary. the math is literally so precise 🤓🫶
Note that MULPDU will be a lot smaller than the theoretical maximum in most implementations for most circumstances, due to link MTUs, use of extra headers such as those required for IPsec, etc. 🙃
On receive, MPA MUST pass each ULPDU with its length to DDP when it has been validated. ✅
If an MPA implementation supports passing out-of-order ULPDUs to DDP, the MPA implementation SHOULD:
- Pass each ULPDU with its length to DDP as soon as it has been fully received and validated. 🏃♀️
- Provide a mechanism to indicate the ordering of ULPDUs as the sender transmitted them. One possible mechanism might be providing the TCP sequence number for each ULPDU. 🔢
- Provide a mechanism to indicate when a given ULPDU (and prior ULPDUs) are complete (Delivered to DDP). One possible mechanism might be to allow DDP to see the current outgoing TCP ACK sequence number. 🎯
- Provide an indication to DDP that the TCP has closed or has begun to close the connection (e.g., received a FIN). 👋
MPA MUST provide the protocol version negotiated with its peer to DDP. DDP will use this version to set the version in its header and to report the version to [RDMAP]. 🏷️
4. MPA Full Operation Phase
the following sections describe the main semantics of the Full Operation Phase of MPA 🌈
4.1. FPDU Format
MPA senders create FPDUs out of ULPDUs. The format of an FPDU shown below MUST be used for all MPA FPDUs. For purposes of clarity, Markers are not shown in Figure 2. this is the main layout, so no freestyling 💖
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ULPDU_Length | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
| |
~ ~
~ ULPDU ~
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | PAD (0-3 octets) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2: FPDU FormatULPDU_Length: 16 bits (unsigned integer). This is the number of octets of the contained ULPDU. It does not include the length of the FPDU header itself, the pad, the CRC, or any Markers that fall within the ULPDU. The 16-bit ULPDU Length field is big enough to support the largest IP datagrams for IPv4 or IPv6. 📏✨
PAD: The PAD field trails the ULPDU and contains between 0 and 3 octets of data. The pad data MUST be set to zero by the sender and ignored by the receiver (except for CRC checking). The length of the pad is set so the size of the FPDU is an integral multiple of four. 🫧
CRC: 32 bits. When CRCs are enabled, this field contains a CRC32c check value, which is used to verify the entire contents of the FPDU, using CRC32c. See Section 4.4, CRC Calculation. When CRCs are not enabled, this field is still present, may contain any value, and MUST NOT be checked. 🔐 okay this one is a little dramatic but it's giving protection
The FPDU adds a minimum of 6 octets to the length of the ULPDU. In addition, the total length of the FPDU will include the length of any Markers and from 0 to 3 pad octets added to round up the ULPDU size. 🧮
4.2. Marker Format ✨
okay so the format of a Marker MUST be exactly what's drawn in Figure 3, no freestyling 💅
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RESERVED | FPDUPTR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 3: Marker FormatRESERVED 🎀: the Reserved field MUST be set to zero on transmit and ignored on receive (except for CRC calculation). it's giving "don't touch it, just zero it" energy 🙈
FPDUPTR 🦋: this is the FPDU Pointer, a relative pointer that's 16 bits long and read as an unsigned integer. it counts how many octets are in the TCP stream from the start of the ULPDU Length field to the first octet of the whole Marker. the least significant two bits MUST always be set to zero at the transmitter, and the receivers MUST always treat these as zero for calculations. main character energy for those two bits, tbh 👑
4.3. MPA Markers 💖
okay bestie, Markers are how you find where FPDUs start when packets show up out of order 🧚 they get placed at fixed intervals in the data stream (which is tied to the TCP sequence number), and the Marker's value helps you find the start of the FPDU just before it.
all MPA Markers are included in the containing FPDU CRC calculation (when both CRCs and Markers are in use) 🔥
whether an MPA receiver can find out-of-order FPDUs and pass the ULPDUs to DDP is up to the implementation 🤷♀️ MPA/DDP lets receivers that can handle out-of-order FPDUs ask for Markers to be inserted in the data stream. if a receiver can't deal with out-of-order FPDUs like that, it may turn off Marker insertion at the sender. all MPA senders MUST be able to generate Markers when the opposing receiver declares they're being used (see Section 7.1, Connection Setup) ✨
when Markers are enabled, MPA senders MUST insert a Marker into the data stream at a 512-octet periodic interval in the TCP Sequence Number Space. each Marker holds a 16-bit unsigned integer called the FPDUPTR (FPDU Pointer) 🌸
if the FPDUPTR's value isn't zero, the FPDU Pointer is a 16-bit relative back-pointer. FPDUPTR MUST contain the number of octets in the TCP stream from the start of the ULPDU Length field to the first octet of the Marker, unless the Marker falls between FPDUs. so to find the first octet of the previous FPDU header, you subtract the Marker's value from the current octet-stream sequence number (aka the TCP sequence number) of the Marker's first octet. and this math MUST take into account that the TCP sequence number could have wrapped between the Marker and the header. it's giving math-bestie era 🧮💀
an FPDUPTR value of 0x0000 is a special case 😭 it's used when the Marker lands exactly between FPDUs (between the preceding FPDU's CRC field and the next FPDU's ULPDU Length field). in that situation the Marker counts as part of the following FPDU, and the Marker MUST be included in the CRC calculation of that FPDU (if CRCs are being generated or checked). so 0x0000 just means an FPDU header (the ULPDU Length field) comes right after the Marker 🍓
since every FPDU is a whole multiple of 4 octets, the bottom two bits of the FPDUPTR the sender calculates are always zero. MPA reserves those bits, so the receiver MUST treat them as zero when it does its math 🫶
when Markers are enabled (see Section 7.1, Connection Setup), the MPA Markers MUST be inserted immediately before the first FPDU of Full Operation Phase, and then at every 512th octet of the TCP octet stream after that. so the first Marker has an FPDUPTR value of 0x0000. if the first Marker starts at octet sequence number SeqStart, Markers go in such that the first octet of a Marker sits at octet sequence number SeqNum whenever the remainder of (SeqNum - SeqStart) mod 512 is zero. and yes, SeqNum can wrap 🌈
for example, if you used the TCP sequence number to figure out where Markers go, the starting TCP sequence number probably won't be zero, and the 512-octet multiples probably won't land on a modulo 512 of zero. so if the MPA connection starts at TCP sequence number 11, the 1st Marker begins at 11, and the next ones begin at 523, 1035, etc. 🧚
if an FPDU is big enough to hold multiple Markers, they MUST all point to the same spot in the TCP stream: the first octet of that FPDU's ULPDU Length field 💜
if a Marker interval has multiple FPDUs (because they're small), the Marker MUST point to the start of the ULPDU Length field of the FPDU that contains the Marker, unless the Marker falls between FPDUs. in that case the Marker MUST be zero 🙈
here's an example of an FPDU with a Marker in it 🎀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ULPDU Length (0x0010) | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
| |
+ +
| ULPDU (octets 0-9) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (0x0000) | FPDU ptr (0x000C) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ULPDU (octets 10-15) |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | PAD (2 octets:0,0) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 4: Example FPDU Format with MarkerMPA Receivers MUST preserve ULPDU boundaries when passing data to DDP ✨ and MPA Receivers MUST pass the ULPDU data and the ULPDU Length to DDP, but not the Markers, headers, or CRC. only the good stuff gets through, periodt 💅🦋
4.4. CRC Calculation 💖
Okay bestie, CRCs are the little checksums that catch errors, and every MPA implementation MUST implement CRC support 💅 and it MUST pick one of these two routes:
(1) always use CRCs. The MPA provider is not REQUIRED to go along with an admin who wants CRCs turned off 🙅♀️
or
(2a) only show a preference to skip CRCs when the system administrator explicitly asks, through an interface this spec doesn't define. The default for a connection MUST be to use CRCs ✨
(2b) turn off CRC checking (and maybe generation too) if both the local and remote endpoints say they'd rather skip CRCs 🦋
Now the tea ☕ An admin deciding to have a host request CRC suppression SHOULD NOT happen unless they're sure the TCP connection gives protection from undetected errors that's at least as strong as an end-to-end CRC32c. Using an IPsec cryptographic integrity check end-to-end is one way to get that protection, and the ULP using channel bindings [NFSv4CHANNEL] can give a high level of assurance that the IPsec protection scope is end-to-end with respect to the ULP. Lowkey a big deal 🔥
The process MUST be invisible to the ULP 🫥 The ULP should never even notice it's happening.
Here's the rule for when peers disagree 👀 After getting an MPA startup declaration saying its peer requires CRCs, an MPA instance MUST keep generating and checking CRCs until the connection ends. If an MPA instance has declared that it does not require CRCs, it MUST turn off CRC checking right after receiving an MPA mode declaration saying its peer also doesn't require CRCs. It MAY keep generating CRCs, though 🌸 For the details on MPA startup, check Section 7.1, Connection Setup.
When you send an FPDU, the sender MUST include a CRC field 🎀 When CRCs are enabled, the CRC field in the MPA FPDU MUST be computed with the CRC32c polynomial, done the way the iSCSI Protocol [iSCSI] document describes for Header and Data Digests. It's giving precision 💜
The fields that MUST be included in the CRC calculation when sending an FPDU go like this:
- If a Marker does not immediately precede the ULPDU Length field, the CRC-32c is calculated from the first octet of the ULPDU Length field, through all the ULPDU and Markers (if present), to the last octet of the PAD (if present), inclusive. If there is a Marker immediately following the PAD, the Marker is included in the CRC calculation for this FPDU.
- If a Marker immediately precedes the first octet of the ULPDU Length field of the FPDU (aka the Marker fell between FPDUs, so it has to be included in the second FPDU), the CRC-32c is calculated from the first octet of the Marker, through the ULPDU Length header, through all the ULPDU and Markers (if present), to the last octet of the PAD (if present), inclusive.
- After calculating the CRC-32c, the resulting value goes into the CRC field at the end of the FPDU 🧚
When an FPDU is received and CRC checking is enabled, the receiver MUST first do the following:
- Calculate the CRC of the incoming FPDU the same way as defined above.
- Check that the calculated CRC-32c value matches the received CRC-32c value in the FPDU CRC field. If it doesn't, the receiver MUST treat the FPDU as an invalid FPDU 💀
What happens next with invalid FPDUs is covered in Section 8, Error Semantics.
Now for the hex dumps, which are honestly so satisfying 🤓 This is an annotated hex dump of an example FPDU sent as the first FPDU on the stream. So it starts with a Marker. The FPDU has a 42 octet ULPDU (an example DDP segment), which holds 24 octets of the contained ULPDU, a data load that's all zeros. The CRC32c has been calculated correctly, so you can use it as a reference. For what the DDP Control field, Queue, MSN, MO, and Send Data mean, check the [DDP] and [RDMAP] specifications.
Octet Contents Annotation
Count
0000 00 Marker: Reserved
0001 00
0002 00 Marker: FPDUPTR
0003 00
0004 00 ULPDU Length
0005 2a
0006 41 DDP Control Field, Send with Last flag set
0007 43
0008 00 Reserved (DDP STag position with no STag)
0009 00
000a 00
000b 00
000c 00 DDP Queue = 0
000d 00
000e 00
000f 00
0010 00 DDP MSN = 1
0011 00
0012 00
0013 01
0014 00 DDP MO = 0
0015 00
0016 00
0017 00
0018 00 DDP Send Data (24 octets of zeros)
...
002f 00
0030 52 CRC32c
0031 23
0032 99
0033 83Figure 5: Annotated Hex Dump of an FPDU
Next up is an example sent as the second FPDU of the stream. The first FPDU (not shown here) had a length of 492 octets and was also a Send to Queue 0 with the Last Flag set. This one contains a Marker 🍓
Octet Contents Annotation
Count
01ec 00 Length
01ed 2a
01ee 41 DDP Control Field: Send with Last Flag set
01ef 43
01f0 00 Reserved (DDP STag position with no STag)
01f1 00
01f2 00
01f3 00
01f4 00 DDP Queue = 0
01f5 00
01f6 00
01f7 00
01f8 00 DDP MSN = 2
01f9 00
01fa 00
01fb 02
01fc 00 DDP MO = 0
01fd 00
01fe 00
01ff 00
0200 00 Marker: Reserved
0201 00
0202 00 Marker: FPDUPTR
0203 14
0204 00 DDP Send Data (24 octets of zeros)
...
021b 00
021c 84 CRC32c
021d 92
021e 58
021f 98Figure 6: Annotated Hex Dump of an FPDU with Marker
4.5. FPDU Size Considerations 📏
MPA defines the Maximum Upper Layer Protocol Data Unit (MULPDU) as the size of the largest ULPDU that fits in an FPDU. For an empty TCP Segment, MULPDU is EMSS minus the FPDU overhead (6 octets) minus space for Markers and pad octets.
When Markers are present, the maximum ULPDU Length for a single ULPDU MUST be computed as:
MULPDU = EMSS - (6 + 4 * Ceiling(EMSS / 512) + EMSS mod 4)That formula already accounts for the worst-case number of Markers 🙌
When Markers are not present, the maximum ULPDU Length for a single ULPDU MUST be computed as:
MULPDU = EMSS - (6 + EMSS mod 4)For extra wire efficiency, an MPA implementation MAY dynamically adjust the MULPDU (see Section 5 for the latency and wire efficiency trade-offs) ✨ When one or more FPDUs are already packed into a TCP Segment, MULPDU MAY be reduced to match.
DDP SHOULD provide ULPDUs that are as large as possible, while staying less than or equal to MULPDU 🍰
If the TCP implementation has to adjust EMSS for MTU changes or changing TCP options, the MULPDU value changes to match 🔄
In certain rare situations, EMSS may shrink below 128 octets. If that happens, the MPA on the TCP sender MUST NOT shrink the MULPDU below 128 octets, and it is not required to follow the segmentation rules in Section 5.1 and Appendix A. Okay that's a real plot twist 😭
If one or more FPDUs are already packed into a TCP segment, so the remaining room is less than 128 octets, MPA MUST NOT provide a MULPDU smaller than 128. In this case, MPA would typically provide a MULPDU for the next full sized segment, but it may still pack the next FPDU into the small remaining room, as long as that FPDU is small enough to fit 🧩
The value 128 was picked so DDP designers have room for the DDP Header and some user data. Smart, right? 💜
5. MPA's interactions with TCP ✨
The following sections describe MPA's interactions with TCP. This section covers using a standard layered TCP stack, with MPA sitting on top of a TCP socket. Using an optimized MPA-aware TCP, with an MPA implementation that takes advantage of the extra optimizations, is covered in Appendix A 🌈
+-----------------------------------+
| +-----+ +-----------------+ |
| | MPA | | Other Protocols | |
| +-----+ +-----------------+ |
| || || |
| ----- socket API -------------- |
| || |
| +-----+ |
| | TCP | |
| +-----+ |
| || |
| +-----+ |
| | IP | |
| +-----+ |
+-----------------------------------+Figure 7: Fully Layered Implementation
The fully layered implementation is described for completeness, but the user is cautioned 🚨 that with this setup, FPDU alignment is less likely when transmitting. That tends to add higher overhead at optimized receivers. On top of that, the lack of out-of-order receive processing will seriously reduce the value of DDP/MPA, because the local receiver ends up with higher buffering and copying overhead 😩
5.1. MPA transmitters with a standard layered TCP 🚀
MPA transmitters SHOULD calculate a MULPDU as described in Section 4.5. If the TCP implementation lets MPA determine EMSS, that value should be used. If the transmit side TCP implementation can't report EMSS, MPA SHOULD use the current MTU value to estimate a likely FPDU size, while taking the various expected header sizes into account.
MPA transmitters SHOULD also use whatever facilities the TCP stack offers to make the TCP transmitter start TCP segments at FPDU boundaries. Multiple FPDUs MAY be packed into a single TCP segment, as determined by the EMSS calculation, as long as they're entirely contained in that segment 📦
For example, passing FPDU buffers sized to the current EMSS to the TCP socket, and using the TCP_NODELAY socket option to disable the Nagle [RFC896] algorithm, will usually result in many segments starting with an FPDU 🎯
It's recognized that various effects can cause FPDU Alignment to be lost. A few of those effects are:
- ULPDUs that are smaller than the MULPDU. If these are sent in a continuous stream, FPDU Alignment will be lost. Careful use of a dynamic MULPDU can help here: the MULPDU for future FPDUs can be adjusted to re-establish alignment with the segments based on the current EMSS 🧠
- Sending enough data that the TCP receive window limit is reached. TCP may send a smaller segment to exactly fill the receive window 🪟
- Sending data while TCP is operating up against the congestion window. If TCP isn't tracking the congestion window in segments, it may transmit a smaller segment to exactly fill the receive window 🌊
- Changes in EMSS due to varying TCP options, or changes in MTU 🔧
If FPDU Alignment with TCP segments is lost for any reason, it's regained after a break in transmission where the TCP send buffers are emptied. Lots of usage models for DDP/MPA include breaks like that 🫶
MPA receivers are REQUIRED to be able to operate correctly even if alignment is lost (see Section 6). No matter what, the receiver has to keep it together 💪
5.2. MPA receivers with a standard layered TCP
okay so MPA receivers get their TCP data in the usual ordered stream 🌊 and they MUST figure out where each FPDU starts and ends using the ULPDU_LENGTH field, as described in Section 6. 💖 They MAY use markers to double-check that the FPDU boundaries line up, but they are NOT required to look at the markers to find those boundaries. the length field does the heavy lifting here, bestie 💅
6. MPA Receiver FPDU Identification ✨
an MPA receiver MUST verify the FPDU before it passes the ULPDU to DDP 🫶 to do that, the receiver MUST:
- locate the start of the FPDU unambiguously,
- verify its CRC (if CRC checking is enabled).
if both of those are true, the MPA receiver passes the ULPDU to DDP. easy peasy 🎀
to detect the start of the FPDU unambiguously, one of the following MUST be used:
- In an ordered TCP stream, the ULPDU Length field in the current FPDU can be used to identify the beginning of the next FPDU, but only when that FPDU has a valid CRC.
- For optimized MPA/TCP receivers that support out-of-order reception of FPDUs (see Section 4.3, MPA Markers), a Marker can always be used to locate the beginning of an FPDU (in FPDUs with valid CRCs). the location of the Marker is known in the octet stream (sequence number space), so the Marker can always be found 🦋
- Having found an FPDU by means of a Marker, an optimized MPA/TCP receiver can find the following contiguous FPDUs by using the ULPDU Length fields (from FPDUs with valid CRCs) to establish the next FPDU boundary 🧚
the ULPDU Length field (see Section 4) MUST be used to determine whether the entire FPDU is present before the ULPDU gets forwarded to DDP. no half-baked FPDUs allowed 💀
CRC calculation is discussed in Section 4.4 above. 🌸
7. Connection Semantics 👑
7.1. Connection Setup
MPA requires that the Consumer MUST activate MPA, and any TCP enhancements for MPA, on a TCP half connection at the same location in the octet stream at both the sender and the receiver. this is non-negotiable, because it's how the Marker scheme finds the Markers (if they're enabled) and how everyone finds the first FPDU 🔥
MPA, and any TCP enhancements for MPA, get enabled by the ULP in both directions at once at an endpoint. ✨
there are a few ways to make this happen, and it's up to DDP's ULP to pick:
- DDP's ULP MAY require DDP on MPA startup immediately after TCP connection setup. this is lowkey the simplest option, because no streaming mode negotiation is needed 🌈 an example of this kind of protocol is shown in Figure 10: Example Immediate Startup negotiation.
this may be done by using a well-known port, or a service locator protocol to find the right port where DDP on MPA is expected to run.
- DDP's ULP MAY negotiate the start of DDP on MPA sometime after a normal TCP startup, using TCP streaming data exchanges on the same connection. the exchange establishes that DDP on MPA (as well as other ULPs) will be used, and pins down exactly where in the octet stream MPA is to begin operating 🦋 note that a negotiation protocol like this is outside the scope of this specification. a simplified example is shown in Figure 9: Example Delayed Startup negotiation on page 33.
An MPA endpoint operates in two distinct phases 🎀
the Startup Phase is used to verify correct MPA setup, exchange CRC and Marker configuration, and optionally pass Private Data between endpoints before a DDP connection is completed. during this phase, specifically formatted frames get exchanged as TCP byte streams without using CRCs or Markers. during this phase, a DDP endpoint doesn't need to be "bound" to the MPA connection. honestly the choice of DDP endpoint and its operating parameters might not even be known until the Consumer has looked at the supplied Private Data (if there is any) 🙈
the second phase is Full Operation, where FPDUs are sent using all the rules that apply (CRCs, Markers, MULPDU restrictions, etc.). a DDP endpoint MUST be "bound" to the MPA connection when it enters this phase 💜
when Private Data is passed between ULPs in the Startup Phase, the ULP is responsible for interpreting that data, and then placing MPA into Full Operation. ✨
Note: The following text differentiates the two endpoints by calling them Initiator and Responder. this is pretty arbitrary and has NOTHING to do with the TCP startup (the SYN, SYN/ACK sequence). the Initiator is the side that sends first in the MPA startup sequence (the MPA Request Frame).
Note: The possibility that both endpoints would be allowed to make a connection at the same time, sometimes called an active/active connection, was considered by the work group and rejected 🙅♀️ there were a few reasons for that. one was that the applications needing this were few (none other than theoretical at the time of this document). another was that the feature created some implementation difficulties, especially with the "dual stack" designs described later on. the last issue was that dealing with rejected connections at startup would have needed at least an additional frame type and more recovery actions, which makes the protocol messier. none of these was overwhelming on its own, but the group and implementers weren't motivated to do the work to fix them. the protocol does include a way to detect these active/active startup attempts, so they can be rejected and an error reported ✨
the ULP is responsible for deciding which side is Initiator or Responder. for client/server type ULPs this is easy peasy. for peer-peer ULPs (which might use a TCP style active/active startup), some mechanism (not defined by this specification) must be established, or some streaming mode data must be exchanged before MPA startup to figure out which side starts in Initiator mode and which starts in Responder mode 🌸
7.1.1. MPA Request and Reply Frame Format
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 | |
+ Key (16 bytes containing "MPA ID Req Frame") +
4 | (4D 50 41 20 49 44 20 52 65 71 20 46 72 61 6D 65) |
+ Or (16 bytes containing "MPA ID Rep Frame") +
8 | (4D 50 41 20 49 44 20 52 65 70 20 46 72 61 6D 65) |
+ +
12 | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
16 |M|C|R| Res | Rev | PD_Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ ~
~ Private Data ~
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 8: MPA Request/Reply Frame 🦋
Key: this field holds the "key" that proves the sender is an MPA sender 🔑 Initiator mode senders MUST set this field to the fixed value "MPA ID Req Frame", or (in byte order) 4D 50 41 20 49 44 20 52 65 71 20 46 72 61 6D 65 (in hexadecimal). Responder mode receivers MUST check this field for that same value, and close the connection and report an error locally if any other value shows up. Responder mode senders MUST set this field to the fixed value "MPA ID Rep Frame", or (in byte order) 4D 50 41 20 49 44 20 52 65 70 20 46 72 61 6D 65 (in hexadecimal). Initiator mode receivers MUST check this field for that same value, and close the connection and report an error locally if any other value is detected.
M: this bit declares an endpoint's REQUIRED Marker usage 📍 when this bit is '1' in an MPA Request Frame, the Initiator declares that Markers are REQUIRED in FPDUs sent from the Responder. when it's set to '1' in an MPA Reply Frame, the bit declares that Markers are REQUIRED in FPDUs sent from the Initiator. when a received MPA Request Frame or MPA Reply Frame has the value '0', Markers MUST NOT be added to the data stream by that endpoint. when it's '1', Markers MUST be added as described in Section 4.3, MPA Markers.
C: this bit declares an endpoint's preferred CRC usage 🧮 when this field is '0' in the MPA Request Frame and the MPA Reply Frame, CRCs MUST not be checked and need not be generated by either endpoint. when this bit is '1' in either the MPA Request Frame or the MPA Reply Frame, CRCs MUST be generated and checked by both endpoints. even when they're not in use, the CRC field stays present in the FPDU. when CRCs are not in use, the CRC field MUST be considered valid for FPDU checking regardless of its contents. 💀
R: this bit is set to zero, and it's not checked on reception in the MPA Request Frame. in the MPA Reply Frame, this bit is the Rejected Connection bit, which the Responder's ULP sets to indicate acceptance ('0') or rejection ('1') of the connection parameters provided in the Private Data. 🙅♀️
Res: this field is reserved for future use 🌙 it MUST be set to zero when sending, and not checked on reception.
Rev: this field contains the revision of MPA. for this version of the specification, senders MUST set this field to one. MPA receivers compliant with this version of the specification MUST check this field. if the MPA receiver can't interoperate with the received version, then it MUST close the connection and report an error locally. otherwise, the MPA receiver should report the received version to the ULP. ✨
PD_Length: this field MUST contain the length in octets of the Private Data field. a value of zero means there is no Private Data field present at all. if the receiver detects that the PD_Length field doesn't match the length of the Private Data field, or if the Private Data field is longer than 512 octets, the receiver MUST close the connection and report an error locally. otherwise, the MPA receiver should pass the PD_Length value and the Private Data to the ULP 🎀
Private Data: this field may contain any value defined by ULPs, or it may not be present at all 🫶 the Private Data field MUST be between 0 and 512 octets in length. ULPs define how to size, set, and validate this field within these limits. Private Data usage is further discussed in Section 7.1.4. 🌈
7.1.2. Connection Startup Rules ✨
Okay bestie, this is the startup playbook for MPA connections 🎀 There are ten rules for the Startup Phase, and every number matters, so no skipping. 💅
- When MPA is started in the Initiator mode, the MPA implementation MUST send a valid MPA Request Frame. The MPA Request Frame MAY include ULP-supplied Private Data. 💌
- When MPA is started in the Responder mode, the MPA implementation MUST wait until an MPA Request Frame is received and validated before entering Full MPA/DDP Operation.
If the MPA Request Frame is improperly formatted, the implementation MUST close the TCP connection and exit MPA.
If the MPA Request Frame is properly formatted but the Private Data is not acceptable, the implementation SHOULD return an MPA Reply Frame with the Rejected Connection bit set to '1'. The MPA Reply Frame MAY include ULP-supplied Private Data. The implementation MUST exit MPA, leaving the TCP connection open. The ULP may close TCP or use the connection for other purposes. 🙅♀️
If the MPA Request Frame is properly formatted and the Private Data is acceptable, the implementation SHOULD return an MPA Reply Frame with the Rejected Connection bit set to '0'. The MPA Reply Frame MAY include ULP-supplied Private Data. The Responder SHOULD prepare to interpret any data received as FPDUs and pass any received ULPDUs to DDP. 🦋
Note: Since the receiver's ability to deal with Markers is unknown until the Request and Reply Frames have been received, sending FPDUs before this occurs is not possible. 🙈
Note: The requirement to wait on a Request Frame before sending a Reply Frame is a design choice. It makes for a well-ordered sequence of events at each end, and avoids having to specify how to deal with situations where both ends start at the same time. It's giving organized. 🧚
- MPA Initiator mode implementations MUST receive and validate an MPA Reply Frame.
If the MPA Reply Frame is improperly formatted, the implementation MUST close the TCP connection and exit MPA.
If the MPA Reply Frame is properly formatted but is the Private Data is not acceptable, or if the Rejected Connection bit is set to '1', the implementation MUST exit MPA, leaving the TCP connection open. The ULP may close TCP or use the connection for other purposes. 😤
If the MPA Reply Frame is properly formatted and the Private Data is acceptable, and the Reject Connection bit is set to '0', the implementation SHOULD enter Full MPA/DDP Operation Phase, interpreting any received data as FPDUs and sending DDP ULPDUs as FPDUs. 🥳
- MPA Responder mode implementations MUST receive and validate at least one FPDU before sending any FPDUs or Markers.
Note: This requirement is present to allow the Initiator time to get its receiver into Full Operation before an FPDU arrives, avoiding potential race conditions at the Initiator. This was also subject to some debate in the work group before rough consensus was reached. Eliminating this requirement would allow faster startup in some types of applications. However, that would also make certain implementations (particularly "dual stack") much harder. 🧠
- If a received "Key" does not match the expected value (see Section 7.1.1, MPA Request and Reply Frame Format) the TCP/DDP connection MUST be closed, and an error returned to the ULP. 🚫
- The received Private Data fields may be used by Consumers at either end to further validate the connection and set up DDP or other ULP parameters. The Initiator ULP MAY close the TCP/MPA/DDP connection as a result of validating the Private Data fields. The Responder SHOULD return an MPA Reply Frame with the "Reject Connection" bit set to '1' if the validation of the Private Data is not acceptable to the ULP. 🔍
- When the first FPDU is to be sent, then if Markers are enabled, the first octets sent are the special Marker 0x00000000, followed by the start of the FPDU (the FPDU's ULPDU Length field). If Markers are not enabled, the first octets sent are the start of the FPDU (the FPDU's ULPDU Length field). 🌸
- MPA implementations MUST use the difference between the MPA Request Frame and the MPA Reply Frame to check for incorrect "Initiator/Initiator" startups. Implementations SHOULD put a timeout on waiting for the MPA Request Frame when started in Responder mode, to detect incorrect "Responder/Responder" startups. 👀
- MPA implementations MUST validate the PD_Length field. The buffer that receives the Private Data field MUST be large enough to receive that data; the amount of Private Data MUST not exceed the PD_Length or the application buffer. If any of the above fails, the startup frame MUST be considered improperly formatted. 📏
- MPA implementations SHOULD implement a reasonable timeout while waiting for the entire set of startup frames; this prevents certain denial-of-service attacks. ULPs SHOULD implement a reasonable timeout while waiting for FPDUs, ULPDUs, and application level messages to guard against application failures and certain denial-of-service attacks. 🛡️
7.1.3. Example Delayed Startup Sequence 🕰️
A variety of startup sequences are possible when using MPA on TCP. Following is an example of an MPA/DDP startup that occurs after TCP has been running for a while and has exchanged some amount of streaming data. This example does not use any Private Data (an example that does is shown later in Section 7.1.4.2, Example Immediate Startup Using Private Data), although it is perfectly legal to include the Private Data. Note that since the example does not use any Private Data, there are no ULP interactions shown between receiving "startup frames" and putting MPA into Full Operation. 🎞️
Initiator Responder
+---------------------------+
|ULP streaming mode |
| <Hello> request to |
| transition to DDP/MPA | +---------------------------+
| mode (optional). | --------> |ULP gets request; |
+---------------------------+ | enables MPA Responder |
| mode with last (optional)|
| streaming mode |
| <Hello Ack> for MPA to |
| send. |
+---------------------------+ |MPA waits for incoming |
|ULP receives streaming | <-------- | <MPA Request Frame>. |
| <Hello Ack>; | +---------------------------+
|Enters MPA Initiator mode; |
|MPA sends |
| <MPA Request Frame>; |
|MPA waits for incoming | +---------------------------+
| <MPA Reply Frame>. | - - - - > |MPA receives |
+---------------------------+ | <MPA Request Frame>. |
|Consumer binds DDP to MPA; |
|MPA sends the |
| <MPA Reply Frame>. |
|DDP/MPA enables FPDU |
+---------------------------+ | decoding, but does not |
|MPA receives the | < - - - - | send any FPDUs. |
| <MPA Reply Frame> | +---------------------------+
|Consumer binds DDP to MPA; |
|DDP/MPA begins Full |
| Operation. |
|MPA sends first FPDU (as | +---------------------------+
| DDP ULPDUs become | ========> |MPA receives first FPDU. |
| available). | |MPA sends first FPDU (as |
+---------------------------+ | DDP ULPDUs become |
<====== | available). |
+---------------------------+Figure 9: Example Delayed Startup Negotiation 🖼️
An example Delayed Startup sequence is described below: 📝
- Active and passive sides start up a TCP connection in the usual fashion, probably using sockets APIs. They exchange some amount of streaming mode data. At some point, one side (the MPA Initiator) sends streaming mode data that effectively says "Hello, let's go into MPA/DDP mode".
- When the remote side (the MPA Responder) gets this streaming mode message, the Consumer would send a last streaming mode message that effectively says "I acknowledge your Hello, and am now in MPA Responder mode". The exchange of these messages establishes the exact point in the TCP stream where MPA is enabled. The Responding Consumer enables MPA in the Responder mode and waits for the initial MPA startup message.
- The Initiating Consumer would enable MPA startup in the Initiator mode which then sends the MPA Request Frame. It is assumed that no Private Data messages are needed for this example, although it is possible to do so. The Initiating MPA (and Consumer) would also wait for the MPA connection to be accepted.
- The Responding MPA would receive the initial MPA Request Frame and would inform the Consumer that this message arrived. The Consumer can then accept the MPA/DDP connection or close the TCP connection.
- To accept the connection request, the Responding Consumer would use an appropriate API to bind the TCP/MPA connections to a DDP endpoint, thus enabling MPA/DDP into Full Operation. In the process of going to Full Operation, MPA sends the MPA Reply Frame. MPA/DDP waits for the first incoming FPDU before sending any FPDUs.
- If the initial TCP data was not a properly formatted MPA Request Frame, MPA will close or reset the TCP connection immediately.
- The Initiating MPA would receive the MPA Reply Frame and would report this message to the Consumer. The Consumer can then accept the MPA/DDP connection, or close or reset the TCP connection to abort the process.
- On determining that the connection is acceptable, the Initiating Consumer would use an appropriate API to bind the TCP/MPA connections to a DDP endpoint thus enabling MPA/DDP into Full Operation. MPA/DDP would begin sending DDP messages as MPA FPDUs. 🚀
7.1.4. Use of Private Data 🎀
This section is advisory in nature, in that it suggests a method by which a ULP can deal with pre-DDP connection information exchange. 💜
7.1.4.1. Motivation ✨
Okay so, older RDMA protocols already had Private Data floating around, just passed through out-of-band mechanisms 🫶. Because of that, a bunch of apps now expect some kind of Private Data to be ready before the DDP/RDMA connection even exists 💅. Here are some examples, bestie 🎀.
An RDMA endpoint (called a Queue Pair, or QP, in InfiniBand and the [VERBS-RDMA]) must be associated with a Protection Domain. No receive operations may be posted to the endpoint before it is associated with a Protection Domain. Honestly, under both InfiniBand and the proposed RDMA/DDP verbs [VERBS-RDMA], an endpoint/QP is created inside a Protection Domain, so it's not optional 😭.
Some apps pick their Protection Domain based on who the remote ULP client is 👑. For example, if a user session needs multiple connections, it is highly desirable for all of those connections to use a single Protection Domain 💜. Note: Use of Protection Domains is further discussed in [RDMASEC].
InfiniBand, the DAT APIs [DAT-API], and the IT-API [IT-API] all let the active-side ULP send Private Data when it requests a connection 🦋. That data gets handed to the ULP so it can decide whether to accept the connection and, if so, which endpoint to use (and, implicitly, which Protection Domain) 🧚.
Private Data can also make sure both ends of the connection configured their RDMA endpoints compatibly, like the RDMA Read capacity (see [RDMAP]) 🌸. Other ULP-specific uses are also presumed, such as figuring out the identity of the client 🙈.
Private Data is also allowed when accepting the connection, so the two sides can finish any negotiation on RDMA resources and handle other ULP stuff too 🍓.
There are several possible ways to swap this Private Data. For example, the InfiniBand specification includes a connection management protocol that lets a small amount of Private Data go over datagrams before the RDMA connection actually starts 🌈.
This document lets small amounts of Private Data be exchanged as part of the MPA startup sequence ✨. The actual Private Data fields are carried in the MPA Request Frame and the MPA Reply Frame 💖.
If you need larger amounts of Private Data or more negotiation, TCP streaming mode messages may be exchanged before MPA is enabled 🫶.
7.1.4.2. Example Immediate Startup Using Private Data
Initiator Responder
+---------------------------+
|TCP SYN sent. | +--------------------------+
+---------------------------+ --------> |TCP gets SYN packet; |
+---------------------------+ | sends SYN-Ack. |
|TCP gets SYN-Ack | <-------- +--------------------------+
| sends Ack. |
+---------------------------+ --------> +--------------------------+
+---------------------------+ |Consumer enables MPA |
|Consumer enables MPA | |Responder mode, waits for |
|Initiator mode with | | <MPA Request frame>. |
|Private Data; MPA sends | +--------------------------+
| <MPA Request Frame>; |
|MPA waits for incoming | +--------------------------+
| <MPA Reply Frame>. | - - - - > |MPA receives |
+---------------------------+ | <MPA Request Frame>. |
|Consumer examines Private |
|Data, provides MPA with |
|return Private Data, |
|binds DDP to MPA, and |
|enables MPA to send an |
| <MPA Reply Frame>. |
|DDP/MPA enables FPDU |
+---------------------------+ |decoding, but does not |
|MPA receives the | < - - - - |send any FPDUs. |
| <MPA Reply Frame>. | +--------------------------+
|Consumer examines Private |
|Data, binds DDP to MPA, |
|and enables DDP/MPA to |
|begin Full Operation. |
|MPA sends first FPDU (as | +--------------------------+
|DDP ULPDUs become | ========> |MPA receives first FPDU. |
|available). | |MPA sends first FPDU (as |
+---------------------------+ |DDP ULPDUs become |
<====== |available). |
+--------------------------+
Figure 10: Example Immediate Startup NegotiationNote: The exact order of when MPA is started in the TCP connection sequence is implementation dependent; the above diagram shows one possible sequence. Also, the Initiator "Ack" to the Responder's "SYN-Ack" may be combined into the same TCP segment containing the MPA Request Frame (as is allowed by TCP RFCs). 🎀
Here's how the example immediate startup goes, step by step 💅:
- The passive side (Responding Consumer) listens on the TCP destination port, which basically says "I'm ready to accept a connection" 👂.
- The active side (Initiating Consumer) asks for a connection from a TCP endpoint (one that expects to upgrade to MPA/DDP/RDMA and expects the Private Data) to a destination address and port 🦋.
- The Initiating Consumer starts a TCP connection to the destination port. Accepting or rejecting the connection happens just like normal TCP connection establishment 🌸.
- The passive side (Responding Consumer) gets the TCP connection request as usual, so normal TCP gatekeepers like INETD and TCPserver can do their usual safeguard and logging stuff 🛡️. Once the TCP connection is accepted, the Responding Consumer enables MPA in Responder mode and waits for the initial MPA startup message ⏳.
- The Initiating Consumer enables MPA startup in Initiator mode to send an initial MPA Request Frame with the Private Data message it wants to send 📨. The Initiating MPA (and Consumer) also waits for the MPA connection to be accepted, plus any returned Private Data 💌.
- The Responding MPA receives the initial MPA Request Frame with the Private Data message and passes the Private Data through to the Consumer 🧚. The Consumer can then accept the MPA/DDP connection, close the TCP connection, or reject the MPA connection with a return message 💅.
- To accept the connection request, the Responding Consumer uses an appropriate API to bind the TCP/MPA connections to a DDP endpoint, which turns on MPA/DDP Full Operation 👑. While heading into Full Operation, MPA sends the MPA Reply Frame, which includes the Consumer-supplied Private Data with whatever response the Consumer wants to give 🌈. MPA/DDP waits for the first incoming FPDU before sending any FPDUs 🫶.
- If the initial TCP data isn't a properly formatted MPA Request Frame, MPA closes or resets the TCP connection right away 💀.
- To reject the MPA connection request, the Responding Consumer sends an MPA Reply Frame with any ULP-supplied Private Data (including the reason for rejection), with the "Rejected Connection" bit set to '1', and may close the TCP connection 🙅♀️.
- The Initiating MPA receives the MPA Reply Frame with the Private Data message and reports it to the Consumer, including the supplied Private Data 📬.
If the "Rejected Connection" bit is set to a '1', MPA closes the TCP connection and exits 🚪.
If the "Rejected Connection" bit is set to a '0', and the Initiating Consumer decides from the Private Data in the MPA Reply Frame that the connection is acceptable, it uses an appropriate API to bind the TCP/MPA connections to a DDP endpoint, which turns on MPA/DDP Full Operation 🥳. MPA/DDP then starts sending DDP messages as MPA FPDUs 🚀.
7.1.5. "Dual Stack" Implementations
Okay, real talk: MPA/DDP implementations are commonly expected to be built as part of a "dual stack" architecture 🧁. One stack is the traditional TCP stack, usually with a sockets interface API (Application Programming Interface) 🔌. The second stack is the MPA/DDP stack, with its own API and potentially separate code or hardware to handle the MPA/DDP data 🛠️. Implementations can vary a lot, so the following comments are advisory only 🫶.
Using two stacks has some real perks ✨:
- TCP connection setup is usually done with the TCP stack. That means you get the usual naming and addressing mechanisms, and any mechanisms used to "harden" the connection setup against security threats are also used when starting MPA/DDP 🔒.
- Some apps were originally designed for TCP but get "enhanced" to use MPA/DDP after a negotiation shows the other side can do it 🦋. That negotiation happens in TCP's streaming mode, using the usual TCP APIs 🌸.
- Some new apps, designed for RDMA or DDP, still need to swap some data before starting MPA/DDP 💌. That exchange can be any length or complexity, but it often boils down to a small amount of Private Data, maybe just a single message 🍓. Using TCP streaming mode for this lets it happen with well-understood methods 💜.
The main downside of using two stacks is converting an active TCP connection from one stack to the other 😬. You have to do this carefully so you don't lose any data 🥹.
To dodge some of the problems with a "dual stack" architecture, the implementation may require these extra restrictions 👑:
- Enabling the DDP/MPA stack SHOULD be done only when no incoming stream data is expected. The ULP protocol typically manages this 🎀. When following the recommended startup sequence, the Responder side enters DDP/MPA mode, sends the last streaming mode data, and then waits for the MPA Request Frame. No additional streaming mode data is expected 🙅♀️. The Initiator side ULP receives the last streaming mode data, and then enters DDP/MPA mode. Again, no additional streaming mode data is expected 💅.
- The DDP/MPA MAY provide the ability to send a "last streaming message" as part of its Responder DDP/MPA enable function. This lets the DDP/MPA stack handle the switch to DDP/MPA mode more easily, and avoids problems with a very fast return of the MPA Request Frame from the Initiator side 🧚.
Note: Regardless of the "stack" architecture used, TCP's rules MUST be followed. For example, if network data is lost, re-segmented, or re-ordered, TCP MUST recover appropriately even when this occurs while switching stacks. 😭🔥
7.2. Normal Connection Teardown
Each half connection of MPA terminates when DDP closes the corresponding TCP half connection 🌙.
A mechanism SHOULD be provided by MPA to DDP so DDP can find out that a graceful close of the TCP connection has been received by the TCP (for example, when a FIN is received) 🫶.
8. Error Semantics ✨
okay so this is the part where MPA has to be the responsible friend and tell everyone when things go sideways 🙈 The following errors MUST be detected by MPA, and the codes SHOULD be handed to DDP or another Consumer 💅
Code Error
1 TCP connection closed, terminated, or lost. This includes lost
by timeout, too many retries, RST received, or FIN received.
2 Received MPA CRC does not match the calculated value for the
FPDU.
3 In the event that the CRC is valid, received MPA Marker (if
enabled) and ULPDU Length fields do not agree on the start of an
FPDU. If the FPDU start determined from previous ULPDU Length
fields does not match with the MPA Marker position, MPA SHOULD
deliver an error to DDP. It may not be possible to make this
check as a segment arrives, but the check SHOULD be made when a
gap creating an out-of-order sequence is closed and any time a
Marker points to an already identified FPDU. It is OPTIONAL for
a receiver to check each Marker, if multiple Markers are present
in an FPDU, or if the segment is received in order.
4 Invalid MPA Request Frame or MPA Response Frame received. In
this case, the TCP connection MUST be immediately closed. DDP
and other ULPs should treat this similar to code 1, above.So here's the breakdown, bestie 🫶
Code 1 is the "the connection is gone" one 💀 It covers a TCP connection that's closed, terminated, or lost, which includes lost by timeout, too many retries, RST received, or FIN received.
Code 2 is the CRC mismatch. The received MPA CRC doesn't match the value MPA calculated for the FPDU, so something got scrambled 😭
Code 3 is the tricky one, ngl 🧚 If the CRC is valid, but the received MPA Marker (when it's enabled) and the ULPDU Length fields don't agree on where an FPDU starts, MPA SHOULD deliver an error to DDP. It may not be possible to check this as each segment arrives, so the check SHOULD be made when a gap that created an out-of-order sequence gets closed, and any time a Marker points to an FPDU that was already identified. It is OPTIONAL for a receiver to check each Marker when multiple Markers show up in one FPDU, or when the segment arrives in order.
Code 4 is for when an Invalid MPA Request Frame or MPA Response Frame shows up. In this case the TCP connection MUST be immediately closed, and DDP and other ULPs should treat it similar to code 1, above 🦋
Optimized MPA/TCP implementations get a little extra wiggle room here. When conditions 2 or 3 are detected, an optimized MPA/TCP implementation MAY choose to silently drop the TCP segment instead of reporting the error to DDP. In that case the sending TCP will retry the segment, which usually fixes the error, unless the problem was at the source. If it was at the source, the source will usually exceed the number of retries and terminate the connection.
One more rule that's a whole vibe: once MPA delivers an error of any type, it MUST NOT pass or deliver any additional FPDUs on that half connection 🙅♀️
For Error codes 2 and 3, MPA MUST NOT close the TCP connection after a reported error. Closing the connection is the responsibility of DDP's ULP, so it's not MPA's job to slam the door 🚪
Note that since MPA will not Deliver any FPDUs on a half connection following an error detected on the receive side of that connection, DDP's ULP is expected to tear down the connection. This may not occur until after one or more last messages are transmitted on the opposite half connection. This allows a diagnostic error message to be sent.
9. Security Considerations 🔐
okay this section is serious, so I'm putting on my detective hat 🕵️♀️ This section discusses the security considerations for MPA.
9.1. Protocol-Specific Security Considerations
The vulnerabilities of MPA to third-party attacks are no greater than any other protocol running over TCP. A third party, by sending packets into the network that are delivered to an MPA receiver, could launch a variety of attacks that take advantage of how MPA operates. For example, a third party could send random packets that are valid for TCP, but contain no FPDU headers. An MPA receiver reports an error to DDP when any packet arrives that cannot be validated as an FPDU when properly located on an FPDU boundary. A third party could also send packets that are valid for TCP, MPA, and DDP, but do not target valid buffers. These types of attacks ultimately result in loss of connection and thus become a type of DOS (Denial Of Service) attack. Communication security mechanisms such as IPsec [RFC2401, RFC4301] may be used to prevent such attacks.
Independent of how MPA operates, a third party could use ICMP messages to reduce the path MTU to such a small size that performance would likewise be severely impacted. Range checking on path MTU sizes in ICMP packets may be used to prevent such attacks.
[RDMAP] and [DDP] are used to control, read, and write data buffers over IP networks. Therefore, the control and the data packets of these protocols are vulnerable to the spoofing, tampering, and information disclosure attacks listed below. In addition, connection to/from an unauthorized or unauthenticated endpoint is a potential problem with most applications using RDMA, DDP, and MPA.
9.1.1. Spoofing
Spoofing attacks can be launched by the Remote Peer or by a network based attacker. A network-based spoofing attack applies to all Remote Peers. Because the MPA Stream requires a TCP Stream in the ESTABLISHED state, certain types of traditional forms of wire attacks do not apply -- an end-to-end handshake must have occurred to establish the MPA Stream. So, the only form of spoofing that applies is one when a remote node can both send and receive packets. Yet even with this limitation the Stream is still exposed to the following spoofing attacks.
9.1.1.1. Impersonation
A network-based attacker can impersonate a legal MPA/DDP/RDMAP peer (by spoofing a legal IP address) and establish an MPA/DDP/RDMAP Stream with the victim. End-to-end authentication (i.e., IPsec or ULP authentication) provides protection against this attack.
9.1.1.2. Stream Hijacking
Stream hijacking happens when a network-based attacker follows the Stream establishment phase, and waits until the authentication phase (if such a phase exists) is completed successfully. He can then spoof the IP address and redirect the Stream from the victim to its own machine. For example, an attacker can wait until an iSCSI authentication is completed successfully, and hijack the iSCSI Stream.
The best protection against this form of attack is end-to-end integrity protection and authentication, such as IPsec, to prevent spoofing. Another option is to provide physical security. Discussion of physical security is out of scope for this document.
9.1.1.3. Man-in-the-Middle Attack
If a network-based attacker has the ability to delete, inject, replay, or modify packets that will still be accepted by MPA (e.g., TCP sequence number is correct, FPDU is valid, etc.), then the Stream can be exposed to a man-in-the-middle attack. The attacker could potentially use the services of [DDP] and [RDMAP] to read the contents of the associated Data Buffer, to modify the contents of the associated Data Buffer, or to disable further access to the buffer. Other attacks on the connection setup sequence and even on TCP can be used to cause denial of service. The only countermeasure for this form of attack is to either secure the MPA/DDP/RDMAP Stream (i.e., integrity protect) or attempt to provide physical security to prevent man-in-the-middle type attacks.
The best protection against this form of attack is end-to-end integrity protection and authentication, such as IPsec, to prevent spoofing or tampering. If Stream or session level authentication and integrity protection are not used, then a man-in-the-middle attack can occur, enabling spoofing and tampering.
Another approach is to restrict access to only the local subnet/link and provide some mechanism to limit access, such as physical security or 802.1.x. This model is an extremely limited deployment scenario and will not be further examined here.
9.1.2. Eavesdropping
Generally speaking, Stream confidentiality protects against eavesdropping. Stream and/or session authentication and integrity protection are a counter measurement against various spoofing and tampering attacks. The effectiveness of authentication and integrity against a specific attack depend on whether the authentication is machine-level authentication (as the one provided by IPsec) or ULP authentication.
9.2. Introduction to Security Options
The following security services can be applied to an MPA/DDP/RDMAP Stream:
- Session confidentiality - protects against eavesdropping.
- Per-packet data source authentication - protects against the following spoofing attacks: network-based impersonation, Stream hijacking, and man in the middle.
- Per-packet integrity - protects against tampering done by network-based modification of FPDUs (indirectly affecting buffer content through DDP services).
- Packet sequencing - protects against replay attacks, which is a special case of the above tampering attack.
If an MPA/DDP/RDMAP Stream may be subject to impersonation attacks, or Stream hijacking attacks, it is recommended that the Stream be authenticated, integrity protected, and protected from replay attacks. It may use confidentiality protection to protect from eavesdropping (in case the MPA/DDP/RDMAP Stream traverses a public network).
IPsec is capable of providing the above security services for IP and TCP traffic.
ULP protocols may be able to provide part of the above security services. See [NFSv4CHAN] for additional information on a promising approach called "channel binding". From [NFSv4CHAN]:
"The concept of channel bindings allows applications to prove that the end-points of two secure channels at different network layers are the same by binding authentication at one channel to the session protection at the other channel. The use of channel bindings allows applications to delegate session protection to lower layers, which may significantly improve performance for some applications."
9.3. Using IPsec with MPA
IPsec can be used to protect against the packet injection attacks outlined above. Because IPsec is designed to secure individual IP packets, MPA can run above IPsec without change. IPsec packets are processed (e.g., integrity checked and decrypted) in the order they are received, and an MPA receiver will process the decrypted FPDUs contained in these packets in the same manner as FPDUs contained in unsecured IP packets.
MPA implementations MUST implement IPsec as described in Section 9.4 below. The use of IPsec is up to ULPs and administrators.
9.4. Requirements for IPsec Encapsulation of MPA/DDP ✨
okay so the IP Storage working group already put a ton of time into writing the normative IPsec requirements for IP storage [RFC3723] 🔐. Parts of that spec fit a bunch of other protocols too, including the RDDP protocol suite, so nobody has to reinvent the wheel 💅. An MPA on TCP implementation MUST follow the requirements defined in RFC 3723, Sections 2.3 and 5, including the associated normative references for those sections 🫶.
Now for the slightly sneaky part 👀. IPsec acceleration hardware might only be able to juggle a limited number of active Internet Key Exchange Protocol (IKE) Phase 2 security associations (SAs) at once 🧚. So Phase 2 delete messages MAY be sent for idle SAs, just to keep the number of active Phase 2 SAs as low as possible 🍓. BUT the receipt of an IKE Phase 2 delete message MUST NOT be interpreted as a reason for tearing down a DDP/RDMA Stream 🙅♀️. It's preferable to leave the Stream up. If more traffic gets sent on it, you bring up another IKE Phase 2 SA to protect it. This avoids the potential for continually bringing Streams up and down, which is giving rollercoaster and nobody wants that 🎢.
The IPsec requirements for RDDP are based on the version of IPsec specified in RFC 2401 [RFC2401] and related RFCs, as profiled by RFC 3723 [RFC3723], even though a newer version of IPsec exists in RFC 4301 [RFC4301] and related RFCs 🤷♀️. One of the important early applications of the RDDP protocols is their use with iSCSI [iSER], so RDDP's IPsec requirements follow those of IPsec. That way, a common profile of IPsec can be used with iSCSI and the RDDP protocols, which is so iconic of them 👑. In the future, RFC 3723 may be updated to the newer version of IPsec. If that happens, the IPsec security requirements of any such update should apply uniformly to iSCSI and the RDDP protocols 🌈.
Note that there are serious security issues if IPsec is not implemented end-to-end. 😱 For example, if IPsec is implemented as a tunnel in the middle of the network, any hosts between the peer and the IPsec tunneling device can freely attack the unprotected Stream. Not me getting chills 🥶.
10. IANA Considerations 📋
No IANA actions are required by this document. Easy breezy 🌸.
If a well-known port is chosen as the mechanism to identify a DDP on MPA on TCP, the well-known port must be registered with IANA. Because the use of the port is DDP specific, registration of the port with IANA is left to DDP 🎀.
Appendix A. Optimized MPA-Aware TCP Implementations 🦋
This appendix is for information only and is NOT part of the standard. Just a friendly heads-up, bestie 💜.
This appendix covers some optimized MPA-aware TCP implementation guidance for implementers. It's aimed at implementations that want to send and receive as much traffic as possible in an aligned and zero-copy fashion 🚀.
+-----------------------------------+
| +-----------+ +-----------------+ |
| | Optimized | | Other Protocols | |
| | MPA/TCP | +-----------------+ |
| +-----------+ || |
| \\ --- socket API --- |
| \\ || |
| \\ +-----+ |
| \\ | TCP | |
| \\ +-----+ |
| \\ // |
| +-------+ |
| | IP | |
| +-------+ |
+-----------------------------------+Figure 11: Optimized MPA/TCP Implementation 🖼️
The diagram above shows a block diagram of a potential implementation. The network sub-system can support traditional sockets-based connections through the normal API, shown on the right side of the diagram. DDP/MPA/TCP connections run on the facilities shown on the left side 🧩.
DDP/MPA/TCP connections can be started using the facilities on the left side through some suitable API. Or they can be initiated using the facilities on the right side and then transitioned to the left side at the point in the connection setup where MPA goes to "Full MPA/DDP Operation Phase," as described in Section 7.1.2 ✨.
The optimized MPA/TCP implementations (the left side of the diagram, described below) only apply to MPA. All other TCP applications keep using the standard TCP stacks and interfaces shown on the right side of the diagram 💖.
A.1. Optimized MPA/TCP Transmitters
The various TCP RFCs give you a lot of choice when it comes to segmenting a TCP stream 🧁. To optimize FPDU recovery at the MPA receiver, an optimized MPA/TCP implementation uses some additional segmentation rules 🔍.
To get optimum performance, an optimized MPA/TCP transmit side implementation should be enabled to:
- With an EMSS large enough to contain the FPDU(s), segment the outgoing TCP stream so that the first octet of every TCP segment begins with an FPDU. Multiple FPDUs may be packed into a single TCP segment as long as they are entirely contained in that TCP segment 🌷.
- Report the current EMSS from the TCP to the MPA transmit layer 📢.
There are exceptions to the rule above, bestie 🙈. Once a ULPDU is provided to MPA, the MPA/TCP sender either transmits it or fails the connection. It cannot be repudiated, no take-backsies 😭. So during changes in MTU and EMSS, or when TCP's Receive Window size (RWIN) becomes too small, it may be necessary to send FPDUs that don't follow the segmentation rule above.
A possible, but less desirable, alternative is to use IP fragmentation on accepted FPDUs to deal with MTU reductions or an extremely small EMSS 🫠.
Even when alignment with TCP segments is lost, the sender still formats the FPDU according to the FPDU format shown in Figure 2 💅.
On a retransmission, TCP does not necessarily preserve the original TCP segmentation boundaries. This can lead to the loss of FPDU alignment and containment within a TCP segment during TCP retransmissions 💀. So an optimized MPA/TCP sender should try to preserve the original TCP segmentation boundaries on a retransmission 🫶.
A.2. Effects of Optimized MPA/TCP Segmentation
Optimized MPA/TCP senders will fill TCP segments up to the EMSS with a single FPDU when a DDP message is large enough 🍰. The DDP message may not fit into TCP segments perfectly, so a "message tail" often shows up and results in an FPDU smaller than a single TCP segment. Some DDP messages may also be considerably shorter than the EMSS. If a small FPDU is sent in a single TCP segment, you get a "short" TCP segment 🩳.
Applications expected to see strong advantages from Direct Data Placement include transaction-based applications and throughput applications 🚄. Request/response protocols typically send one FPDU per TCP segment and then wait for a response. Under these conditions, those "short" TCP segments are an appropriate and expected effect of the segmentation, so no panic 🌈.
Another possibility is that the application is sending multiple messages (FPDUs) to the same endpoint before waiting for a response. In that case, the segmentation policy would tend to reduce the available connection bandwidth by under-filling the TCP segments 📉.
Standard TCP implementations often use the Nagle [RFC896] algorithm to make sure segments are filled to the EMSS whenever the round-trip latency is large enough that the source stream can fully fill segments before ACKs arrive 🧠. It does this by delaying the transmission of TCP segments until a ULP can fill a segment, or until an ACK arrives from the far side. So the algorithm allows smaller segments when latencies are shorter, which keeps the ULP's end-to-end latency at reasonable levels ⏱️.
The Nagle algorithm is not mandatory to use [RFC1122] 🤓.
When used with optimized MPA/TCP stacks, Nagle and similar algorithms can result in the "packing" of multiple FPDUs into TCP segments 📦.
If a "message tail," small DDP messages, or the start of a larger DDP message are available, MPA may pack multiple FPDUs into TCP segments. When that happens, the TCP segments can be more fully utilized. But because of the size constraints of FPDUs, segments may not be filled all the way to the EMSS. A dynamic MULPDU, which tells DDP the size of the remaining TCP segment space, makes filling the TCP segment more effective 🧩.
Note that MPA receivers do more processing of a TCP segment that contains multiple FPDUs. This may affect the performance of some receiver implementations 👀.
It's up to the ULP to decide if Nagle is useful with DDP/MPA 🤔. Many of the applications expected to take advantage of MPA/DDP prefer to avoid the extra delays caused by Nagle, so in those scenarios there's expected to be minimal opportunity for packing at the transmitter, and receivers may choose to optimize their performance for that expected behaviour 💜.
So the application is expected to set TCP parameters so it can trade off latency and wire efficiency. Implementations should provide a connection option that disables Nagle for MPA/TCP, similar to the way the TCP_NODELAY socket option is provided for a traditional sockets interface 🎛️.
When latency isn't critical, the application is expected to leave Nagle enabled. In that case, the TCP implementation may pack any available FPDUs into TCP segments so the segments are filled to the EMSS. If there isn't enough data to fill the TCP segment when it's prepared for transmission, TCP can send the segment partly filled, or use the Nagle algorithm to wait for the ULP to post more data 🫧.
A.3. Optimized MPA/TCP Receivers ✨
okay so, when an MPA receive implementation AND the MPA-aware TCP receive side both support handling out-of-order ULPDUs, the TCP receive side does these things 💅:
- the implementation passes incoming TCP segments up to MPA as soon as they've been received and validated, even if they didn't show up in order 🫶. TCP has to commit to keeping each segment before it can be passed to MPA, and the segment has to pass TCP, IP, and lower layer data integrity validation (i.e., checksum), be in the receive window, be part of the same epoch (if timestamps are used to verify this), and pass any other checks required by TCP RFCs. 🔥
this does NOT mean the data has to be completely ordered before anyone uses it 👀. An implementation can accept out-of-order segments, SACK them [RFC2018], and pass them to MPA right away, before the segments that fill in the gaps show up. MPA expects to use these segments once they're complete FPDUs or can be combined into complete FPDUs, so ULPDUs can go to DDP when they arrive, no matter the order. DDP uses the passed ULPDU to "place" the DDP segments (see [DDP] for more details) 🧚.
since MPA does a CRC calculation and other checks on received FPDUs, the MPA/TCP implementation makes sure that any TCP segments that duplicate data already received and processed (which can happen during TCP retries) do NOT overwrite FPDUs that were already received and processed 🙅♀️. This way, duplicate data can't corrupt FPDUs that were already validated. bestie that's so important 😭.
- the implementation gives a way to indicate the ordering of TCP segments as the sender transmitted them. One possible mechanism might be attaching the TCP sequence number to each segment 🎀.
- the implementation also gives a way to indicate when a given TCP segment (and the prior TCP stream) is complete. One possible mechanism might be to use the leading (left) edge of the TCP Receive Window 🌸.
MPA uses the ordering and completion indications to tell DDP when a ULPDU is complete; MPA Delivers the FPDU to DDP 🦋. DDP uses those indications to "deliver" its messages to the DDP consumer (see [DDP] for more details).
DDP on MPA uses the two mechanisms above to set up the Delivery semantics that DDP's consumers agree to. These semantics are fully described in [DDP] 💜. They include requirements on DDP's consumer to respect ownership of buffers before DDP delivers them to the Consumer. it's giving responsibility, honestly 💅.
the use of SACK [RFC2018] significantly improves network utilization and performance, so it is recommended 🍓. When you combine it with out-of-order passing of segments to MPA and DDP, a lot of buffering and copying of received data can be avoided ✨.
A.4. Re-Segmenting Middleboxes and Non-Optimized MPA/TCP Senders 🌈
okay so MPA senders often start FPDUs on TCP segment boundaries, which means a receiving optimized MPA/TCP implementation may be able to optimize how it receives data in various ways 🤓.
BUT, MPA receivers MUST NOT depend on FPDU Alignment on TCP segment boundaries. no exceptions bestie 🙅♀️.
some MPA senders might not be able to meet the sender requirements, because their TCP implementation wasn't designed with MPA in mind 🙈. Even for optimized MPA/TCP senders, the network may have "middleboxes" that modify the TCP stream by changing the segmentation. This is generally interoperable with TCP and its users, and MPA must be no exception 💖.
the presence of Markers in MPA (when enabled) lets an optimized MPA/TCP receiver recover the FPDUs despite these obstacles, though it may be necessary to use additional buffering at the receiver to do so 🦋. main character recovery arc, iykyk 👑.
here are some of the cases a receiver might have to deal with, as a little reminder for the implementer 📝:
- A single aligned and complete FPDU, either in order or out of order: This can be passed to DDP as soon as it's validated, and Delivered when ordering is established ✨.
- Multiple FPDUs in a TCP segment, aligned and fully contained, either in order or out of order: These can be passed to DDP as soon as they're validated, and Delivered when ordering is established 🫧.
- Incomplete FPDU: The receiver should buffer until the remainder of the FPDU arrives. If the remainder of the FPDU is already available, this can be passed to DDP as soon as it's validated, and Delivered when ordering is established 🌷.
- Unaligned FPDU start: The partial FPDU must be combined with its preceding portion(s). If the preceding parts are already available and the whole FPDU is present, this can be passed to DDP as soon as it's validated, and Delivered when ordering is established. If the whole FPDU is not available, the receiver should buffer until the remainder of the FPDU arrives 🧩.
- Combinations of unaligned or incomplete FPDUs (and potentially other complete FPDUs) in the same TCP segment: If any FPDU is present in its entirety, or can be completed with portions already available, it can be passed to DDP as soon as it's validated, and Delivered when ordering is established 💅.
A.5. Receiver Implementation 🎀
Transport & Network Layer Reassembly Buffers:
okay so the use of reassembly buffers (either TCP reassembly buffers or IP fragmentation reassembly buffers) is up to the implementation 🤷♀️. When MPA is enabled, reassembly buffers are needed if out-of-order packets arrive and Markers are not enabled. Buffers are also needed if FPDU alignment is lost or if IP fragmentation occurs. That's because the incoming out-of-order segment might not have enough info for MPA to process all of the FPDU 😵💫. For cases where a re-segmenting middlebox is present, or where the TCP sender isn't optimized, the presence of Markers seriously cuts down on how much buffering is needed 🙌.
Recovery from IP fragmentation is transparent to the MPA Consumers 🦋.
A.5.1 Network Layer Reassembly Buffers 🌐
the MPA/TCP implementation should set the IP Don't Fragment bit at the IP layer 🚫. So, when the path MTU changes, intermediate devices drop the IP datagram if it's too large and reply with an ICMP message that tells the source TCP that the path MTU has changed. This makes TCP emit segments that fit the new path MTU size ✨. So IP fragments should basically never happen at the receiver under most conditions. But it's possible 👀.
there are a few options for implementing network layer reassembly buffers:
- drop any IP fragments, and reply with an ICMP message according to [RFC792] (fragmentation needed and DF set) to tell the Remote Peer to resize its TCP segment 🙅♀️.
- support an IP reassembly buffer, but make it of limited size (possibly the same size as the local link's MTU). The end node would normally never Advertise a path MTU larger than the local link MTU. It is recommended that a dropped IP fragment cause an ICMP message to be generated according to RFC 792 💌.
- multiple IP reassembly buffers, of effectively unlimited size 🧺.
- support an IP reassembly buffer for the largest IP datagram (64 KB) 📦.
- support for a large IP reassembly buffer that could span multiple IP datagrams 🪄.
an implementation should support at least 2 or 3 above, to avoid dropping packets that have traversed the entire fabric 💖.
there's no end-to-end ACK for IP reassembly buffers, so there's no flow control on the buffer 😬. The only end-to-end ACK is a TCP ACK, which can only happen when a complete IP datagram is delivered to TCP. Because of this, in the worst case, pathological scenarios, the largest IP reassembly buffer is the TCP receive window (to buffer multiple IP datagrams that have all been fragmented) 🫠.
note that if the Remote Peer doesn't implement re-segmentation of the data stream when it gets the ICMP reply updating the path MTU, it's possible to halt forward progress, because the opposite peer would keep retransmitting with a transport segment size that's too large 🛑. This deadlock scenario is no different than if the fabric MTU (not last-hop MTU) was reduced after connection setup, and the remote node's behavior is not compliant with [RFC1122] 💀.
A.5.2 TCP Reassembly Buffers 🧷
A TCP reassembly buffer is also needed 🫶. TCP reassembly buffers are needed if FPDU Alignment is lost when using TCP with MPA, or when the MPA FPDU spans multiple TCP segments. Buffers are also needed if Markers are disabled and out-of-order packets arrive.
since losing FPDU Alignment often means FPDUs are incomplete, an MPA on TCP implementation must have a reassembly buffer big enough to recover an FPDU that's less than or equal to the MTU of the locally attached link (this should be the largest possible Advertised TCP path MTU) 📏. If the MTU is smaller than 140 octets, a buffer of at least 140 octets long is needed to support the minimum FPDU size. The 140 octets allow for the minimum MULPDU of 128, 2 octets of pad, 2 of ULPDU_Length, 4 of CRC, and space for a possible Marker 🧮. As usual, extra buffering will probably give better performance ✨.
note that if the TCP segments weren't stored, it would be possible to deadlock the MPA algorithm 😱. If the path MTU is reduced, FPDU Alignment requires the source TCP to re-segment the data stream to the new path MTU. The source MPA will detect this condition and reduce the MPA segment size, but any FPDUs already posted to the source TCP will be re-segmented and lose FPDU Alignment. If the destination doesn't support a TCP reassembly buffer, those segments can never be successfully transmitted, and the protocol deadlocks 🥲.
When a complete FPDU is received, processing continues normally 🌸.
Appendix B. Analysis of MPA over TCP Operations 🦋
okay bestie, this appendix is for information only and is NOT part of the standard 💅 so think deep-dive vibes, not rulebook vibes.
It's an analysis of why it's actually useful to integrate MPA with TCP (with some tweaks to typical TCP implementations) to cut down on overall system buffering and overhead. 🧚
One of MPA's big goals is to give enough info, when combined with the Direct Data Placement Protocol [DDP], to let DDP payload get placed out of order straight into the final Upper Layer Protocol (ULP) Buffer. ✨ Note that DDP splits "putting data into a ULP Buffer" apart from "telling the ULP the buffer is ready to use." In DDP terms, the first is "Placement" and the second is "Delivery." MPA supports in-order Delivery of the data to the ULP, including Direct Data Placement in the final ULP Buffer location when TCP segments arrive out of order. Basically, the goal is to use the pre-posted ULP Buffers as the TCP receive buffer, where TCP reassembles the ULP Protocol Data Unit (PDU) (with MPA and DDP) right in place, in the ULP Buffer, with no data copies. 🫶 iconic, honestly.
This appendix walks through the pros and cons of the TCP sender changes MPA proposes:
- MPA prefers that the TCP sender do Header Alignment, where a TCP segment should begin with an MPA Framing Protocol Data Unit (FPDU) (if there is payload present).
- There be an integral number of FPDUs in a TCP segment (under conditions where the path MTU is not changing).
The conclusion? The scaling advantages of FPDU Alignment are strong 🔥, mostly because it drastically cuts TCP receive buffer requirements and makes receive handling way simpler. The analysis also shows there's little effect on TCP wire behavior. 💖
B.1. Assumptions
B.1.1 MPA Is Layered beneath DDP
MPA is an adaptation layer sitting between DDP and TCP. DDP needs DDP segment boundaries preserved and a CRC32c digest covering the DDP header and data. MPA adds these features to the TCP stream so DDP over TCP has the same basic properties as DDP over SCTP. 🌸
B.1.2. MPA Preserves DDP Message Framing
MPA was designed as a framing layer specifically for DDP. It was not intended as a general-purpose framing layer for any other ULP using TCP. 🙈
A framing layer lets the ULPs using it get indications from the transport layer only when complete ULPDUs are present. As a framing layer, MPA isn't aware of the content of the DDP PDU. It only knows it has received and, if necessary, reassembled a complete PDU for Delivery to the DDP. ✨
B.1.3. The Size of the ULPDU Passed to MPA Is Less Than EMSS under Normal Conditions
To make reception of a complete DDP PDU on every received segment possible, DDP passes MPA a PDU that is no larger than the EMSS of the underlying fabric. Each FPDU that MPA creates has enough info for the receiver to directly place the ULP payload in the correct location in the correct receive buffer. 🎀
Edge cases where this condition doesn't occur are dealt with, but they don't need to be on the fast path. 🧚
B.1.4. Out-of-Order Placement but NO Out-of-Order Delivery
DDP receives complete DDP PDUs from MPA. Each DDP PDU contains the info needed to place its ULP payload directly in the correct location in host memory. 💅
Because each DDP segment is self-describing, DDP segments received out of order can have their ULP payload placed immediately in the ULP receive buffer. 🦋
Data delivery to the ULP is guaranteed to be in the order the data was sent. DDP only indicates data delivery to the ULP after TCP has acknowledged the complete byte stream. 👑
B.2. The Value of FPDU Alignment
Big picture: when Header Alignment and complete FPDUs are the common case, receivers get way more optimized. 💜
Significant receiver optimizations are possible when Header Alignment and complete FPDUs are the common case. These optimizations let the receiver use significantly fewer buffers and do less computation per FPDU. The net effect is the ability to build a "flow-through" receiver, which lets TCP-based solutions scale to 10G and beyond in an economical way. 🚀 These optimizations are especially relevant to hardware implementations of receivers that process multiple protocol layers: Data Link Layer (e.g., Ethernet), Network and Transport Layer (e.g., TCP/IP), and even some ULP on top of TCP (e.g., MPA/DDP). As network speed goes up, there's an increasing desire to use a hardware-based receiver to get an efficient, high-performance solution. 💖
A TCP receiver, under worst-case conditions, has to allocate buffers (BufferSizeTCP) whose capacities are a function of the bandwidth-delay product. So:
BufferSizeTCP = K * bandwidth [octets/second] * Delay [seconds].Where bandwidth is the end-to-end bandwidth of the connection, delay is the round-trip delay of the connection, and K is an implementation-dependent constant.
So BufferSizeTCP scales with end-to-end bandwidth (10x more buffers for a 10x increase in end-to-end bandwidth). Since this buffering approach may scale poorly for hardware or software implementations alike, several approaches reduce the amount of buffering needed for high-speed TCP communication. 😩
The MPA/DDP approach is to let the ULP's Buffer serve as the TCP receive buffer. If the application pre-posts enough buffering, and each TCP segment has enough info to place its payload in the right application buffer, then when an out-of-order TCP segment arrives it could potentially be placed directly in the ULP Buffer. ✨ However, placement can only happen when a complete FPDU with the placement info is available to the receiver, and the FPDU contents contain enough info to place the data in the correct ULP Buffer (e.g., a DDP header is available).
When the FPDU isn't aligned with the TCP segment, it may take, on average, 2 TCP segments to assemble one FPDU. So the receiver has to allocate BufferSizeNAF (Buffer Size, Non-Aligned FPDU) octets:
BufferSizeNAF = K1* EMSS * number_of_connections + K2 * EMSSWhere K1 and K2 are implementation-dependent constants and EMSS is the effective maximum segment size.
For example, a 1 GB/sec link with 10,000 connections and an EMSS of 1500 B would need 15 MB of memory. 😱 Often the number of connections scales with network speed, which makes things even worse at higher speeds.
FPDU Alignment would let the receiver allocate BufferSizeAF (Buffer Size, Aligned FPDU) octets:
BufferSizeAF = K2 * EMSSfor the same conditions. An FPDU Aligned receiver may only need memory in the range of ~100s of KB, which is feasible for on-chip memory and enables a "flow-through" design, where data flows through the network interface card (NIC) and is placed directly in the destination buffer. 🍓 Assuming most connections support FPDU Alignment, receiver buffers no longer scale with the number of connections. 🫧
Additional optimizations are possible in a balanced I/O sub-system, where the system interface of the network controller provides ample bandwidth compared with the network bandwidth. For almost twenty years this has been the case, and the trend is expected to continue. Ethernet speeds have scaled by 1000 (from 10 megabit/sec to 10 gigabit/sec), while I/O bus bandwidth of volume CPU architectures has scaled from ~2 MB/sec to ~2 GB/sec (PC-XT bus to PCI-X DDR). 🌈 Under these conditions, the FPDU Alignment approach lets BufferSizeAF be indifferent to network speed. It's primarily a function of the local processing time for a given frame. 🦋
So when the FPDU Alignment approach is used, receive buffering is expected to scale gracefully (i.e., less than linear scaling) as network speed increases. That's giving scalable, main character energy. 👑💅
B.2.1. Impact of Lack of FPDU Alignment on the Receiver Computational Load and Complexity ✨
okay bestie, this is where the receiver's life gets a lil chaotic 😭 The receiver must do all the IP and TCP processing, and then run the FPDU CRC checks, before it can trust the FPDU header placement info 🫠 To keep things simple, we're assuming an FPDU rides in no more than 2 TCP segments. ngl, in real life with no FPDU Alignment, an FPDU can be spread across more than 2 TCP segments (like if the path MTU got reduced) 💀
----++-----------------------------++-----------------------++-----
+---||---------------+ +--------||--------+ +----------||----+
| TCP Seg X-1 | | TCP Seg X | | TCP Seg X+1 |
+---||---------------+ +--------||--------+ +----------||----+
----++-----------------------------++-----------------------++-----
FPDU #N-1 FPDU #NFigure 12: Non-Aligned FPDU Freely Placed in TCP Octet Stream 🌈
okay so here's the receiver algorithm for processing TCP segments (like TCP segment #X in Figure 12) that carry non-aligned FPDUs. It works whether they arrive in order or out of order 🦋
Data Link Layer processing (whole frame), which typically includes a CRC calculation.
- Network Layer processing. Assuming it's not an IP fragment, the whole Data Link Layer frame contains one IP datagram. IP fragments should be reassembled in a local buffer. This is not a performance optimization goal.
- Transport Layer processing: TCP protocol processing, plus header and checksum checks.
- a. Classify the incoming TCP segment using the 5 tuple (IP SRC, IP DST, TCP SRC Port, TCP DST Port, protocol) 🎀
- Find FPDU message boundaries.
- a. Get the MPA state information for the connection.
If the TCP segment is in order, use the receiver-managed MPA state info to calculate where the previous FPDU message (#N-1) ends in the current TCP segment X. (Back when the MPA receiver processed the first part of FPDU #N-1, it worked out how many bytes were left to finish FPDU #N-1 by using the MPA Length field.) Then:
- Get the stored partial CRC for FPDU #N-1.
- Complete the CRC calculation for FPDU #N-1 data (the first portion of TCP segment #X).
- Check the CRC calculation for FPDU #N-1.
- If there are no FPDU CRC errors, placement is allowed.
- Locate the local buffer for the first portion of FPDU#N-1, CopyData(local buffer of first portion of FPDU #N-1, host buffer address, length).
- Compute the host buffer address for the second portion of FPDU #N-1.
- CopyData (local buffer of second portion of FPDU #N-1, host buffer address for second portion, length).
- Calculate the octet offset into the TCP segment for the next FPDU #N.
- Start the CRC calculation for the available data for FPDU #N.
- Store the partial CRC results for FPDU #N.
- Store the local buffer address of the first portion of FPDU #N.
- No further action is possible on FPDU #N before it is completely received 🫶
If the TCP segment is out of order, the receiver must buffer the data until at least one complete FPDU has been received. Typically, buffering for more than one TCP segment per connection is required. Use the MPA-based Markers to calculate where the FPDU boundaries are 🧚
- When a complete FPDU is available, a similar procedure to the in-order algorithm above is used. There's extra complexity, though, because when the missing segment finally arrives, this TCP segment has to go through the CRC engine after the CRC is calculated for the missing segment 😩
okay now, if we do assume FPDU Alignment, this diagram and the algorithm below apply. Heads up: when using MPA, the receiver is assumed to actively detect the presence or loss of FPDU Alignment for every TCP segment it receives 👑
+--------------------------+ +--------------------------+
+--|--------------------------+ +--|--------------------------+
| | TCP Seg X | | | TCP Seg X+1 |
+--|--------------------------+ +--|--------------------------+
+--------------------------+ +--------------------------+
FPDU #N FPDU #N+1Figure 13: Aligned FPDU Placed Immediately after TCP Header 💖
The receiver algorithm for FPDU Aligned frames (in order or out of order) goes like this:
- Data Link Layer processing (whole frame), typically including a CRC calculation.
- Network Layer processing. Assuming it's not an IP fragment, the whole Data Link Layer frame contains one IP datagram. IP fragments should be reassembled in a local buffer. This is not a performance optimization goal.
- Transport Layer processing: TCP protocol processing, plus header and checksum checks.
- a. Classify the incoming TCP segment using the 5 tuple (IP SRC, IP DST, TCP SRC Port, TCP DST Port, protocol).
- Check for Header Alignment (described in detail in Section 6). Assuming Header Alignment for the rest of the algorithm below.
- a. If the header is not aligned, see the algorithm defined in the prior section.
- If the TCP segment is in order or out of order, the MPA header is at the beginning of the current TCP payload. Get the FPDU length from the FPDU header.
- Calculate the CRC over the FPDU.
- Check the CRC calculation for FPDU #N.
- If there are no FPDU CRC errors, placement is allowed.
- CopyData(TCP segment #X, host buffer address, length).
- Loop back to #5 until all the FPDUs in the TCP segment are consumed in order, to handle FPDU packing. 🌸
Implementation note: in both cases, the receiver has to classify the incoming TCP segment and associate it with one of the flows it maintains. With no FPDU Alignment, the receiver is forced to classify incoming traffic before it can calculate the FPDU CRC. With FPDU Alignment, the order of operations is left up to the implementer 🍓
The FPDU Aligned receiver algorithm is significantly simpler, and I'm obsessed 😭 There's no need to buffer portions of FPDUs locally. Accessing state info is also way simpler. In the normal case, you don't have to look up where an FPDU starts and ends, or fetch a partial CRC before the CRC calculation can start. This avoids adding internal latencies, multiple data passes through the CRC machine, or scheduling multiple commands to move data into the host buffer. Iconic behavior from the Aligned approach 👑
The aligned FPDU approach works for both in-order and out-of-order reception. The receiver can use the same mechanisms for data storage in both cases, and it only needs to track when all the TCP segments have arrived so it can enable Delivery. The Header Alignment, along with the high probability that at least one complete FPDU is found in every TCP segment, lets the receiver place data for out-of-order TCP segments with no intermediate buffering needed. Basically, the TCP receive buffer is eliminated, and TCP reassembly happens in place within the ULP Buffer 🦋✨
In case FPDU Alignment is not found, the receiver should follow the algorithm for non-aligned FPDU reception, which may be slower and less efficient 🥲
B.2.2. FPDU Alignment Effects on TCP Wire Protocol 🎀
Okay so bestie, in an optimized MPA/TCP setup, TCP tells MPA its EMSS 🦋. MPA uses that EMSS to work out its MULPDU, then hands the MULPDU up to DDP, its ULP. DDP slices its payload so that every FPDU MPA sends fits completely inside one TCP segment. This has no impact on the wire protocol, and the info is already exposed on loads of TCP implementations, including all modern flavors of BSD networking, through the TCP_MAXSEG socket option 💅✨.
In the common case, the ULP messages (DDP over MPA) given to TCP get segmented to MULPDU size. It's assumed the ULP message size is bounded by MULPDU, so a single ULP message can ride inside a single TCP segment. That means, usually, there's no increase in the number of TCP segments emitted 🌸. For smaller ULP messages, the sender can also do packing, which means it stuffs as many complete FPDUs as possible into one TCP segment. The rule to always have a complete FPDU might increase the number of segments, though 🙈. Typically a ULP message is anywhere from a few bytes up to multiple EMSSs (for example, 64 Kbytes). Sometimes the ULP posts more than one message at a time, which gives the sender a chance to pack 🍓.
Now the corner case. If more than one FPDU is ready to go, they get encapsulated into a TCP segment, and there's no room left for the next complete FPDU, then another TCP segment gets sent. So some segments come out not full size 😭. In the worst-case scenario, the ULP picks an FPDU size of EMSS/2 +1 and has multiple messages ready to send. For that unlucky FPDU size, the average TCP segment size lands around 1/2 of the EMSS, and the number of TCP segments emitted gets close to 2x what you'd get without the requirement to fit an integer number of complete FPDUs into every TCP segment. This is a dynamic situation that only lasts as long as the sender ULP has multiple non-optimal messages waiting to go, so the impact on wire utilization is minor 💜.
But honestly, it is not expected that requiring FPDU Alignment will have a measurable impact on the wire behavior of most applications 🫶. Throughput applications with large I/Os are expected to take full advantage of the EMSS 🔥. Another class of applications, the ones with many small outstanding buffers (compared to EMSS), is expected to use packing when applicable ⭐. Transaction-oriented applications are also optimal 👑.
TCP retransmission is another area that can affect sender behavior 🦋. TCP supports retransmission of the exact, originally transmitted segment (see [RFC793], Sections 2.6 and 3.7 (under "Managing the Window") and [RFC1122], Section 4.2.2.15). In the unlikely event that part of the original segment has already been received and acknowledged by the Remote Peer (for example, a re-segmenting middlebox, as documented in Appendix A.4, Re-Segmenting Middleboxes and Non-Optimized MPA/TCP Senders), better available bandwidth utilization may be possible by retransmitting only the missing octets 🧚. If an optimized MPA/TCP retransmits complete FPDUs, there may be some marginal bandwidth loss 💀.
Another area where a change in the TCP segment number may have an impact is slow start and congestion avoidance 🌈. Slow-start exponential increase is measured in segments per second, since the algorithm focuses on the per-segment overhead at the source for congestion that eventually results in dropped segments. Slow-start exponential bandwidth growth for optimized MPA/TCP is similar to any TCP implementation 💖. Congestion avoidance allows linear growth in available bandwidth when recovering after a packet drop. Just like with slow start, optimized MPA/TCP doesn't change how the algorithm behaves 🎀. So the average size of the segment versus EMSS isn't a major factor when assessing bandwidth growth for a sender. Both slow start and congestion avoidance for an optimized MPA/TCP behave similarly to any TCP sender and let it enjoy the theoretical performance limits of the algorithms ✨.
So the main takeaway: the ULP messages generated at the sender (for example, how many messages get grouped for each transmission request) and the message size distribution have the biggest impact on how many TCP segments get emitted 💅. The worst-case effect for certain ULPs (average message size of EMSS/2+1 to EMSS) is bounded by an increase of up to 2x in the number of TCP segments and acknowledges. In reality, the effect is expected to be marginal 🦋🫶.
Appendix C. IETF Implementation Interoperability with RDMA Consortium Protocols ✨
This appendix is for information only and is NOT part of the standard 📚.
This appendix covers methods of making MPA implementations interoperate with both the IETF and RDMA Consortium versions of the protocols 🌸.
The RDMA Consortium created early specifications of the MPA/DDP/RDMA protocols, and some manufacturers built implementations of those protocols before the IETF versions were finalized 🧚. These protocols are very similar to the IETF versions, which makes it possible to create or modify implementations so they support either set of specifications 💜.
For those who are interested, the RDMA Consortium protocol documents (draft-culley-iwarp-mpa-v1.0.pdf [RDMA-MPA], draft-shah-iwarp-ddp-v1.0.pdf [RDMA-DDP], and draft-recio-iwarp-rdmac-v1.0.pdf [RDMA-RDMAC]) can be obtained at http://www.rdmaconsortium.org/home 📖.
In this section, implementations of MPA/DDP/RDMA that conform to the RDMAC specifications are called RDMAC RNICs. Implementations of MPA/DDP/RDMA that conform to the IETF RFCs are called IETF RNICs 🎀.
Without the exchange of MPA Request/Reply Frames, there is no standard mechanism for enabling RDMAC RNICs to interoperate with IETF RNICs 😭. Even if a ULP uses a well-known port to start an IETF RNIC immediately in RDMA mode (that is, without exchanging the MPA Request/Reply messages), there is no reason to believe an IETF RNIC will interoperate with an RDMAC RNIC, because of the differences in the version number in the DDP and RDMAP headers on the wire 💀.
Therefore, the ULP or other supporting entity at the RDMAC RNIC must implement MPA Request/Reply Frames on behalf of the RNIC in order to negotiate the connection parameters 🔥. The following section describes the results after the exchange of the MPA Request/Reply Frames, before the conversion from streaming to RDMA mode 🦋.
C.1. Negotiated Parameters 💅
Three types of RNICs are considered 👑:
- Upgraded RDMAC RNIC - an RNIC implementing the RDMAC protocols that has a ULP or other supporting entity that exchanges the MPA Request/Reply Frames in streaming mode before the conversion to RDMA mode 🌈.
- Non-permissive IETF RNIC - an RNIC implementing the IETF protocols that is not capable of implementing the RDMAC protocols. Such an RNIC can only interoperate with other IETF RNICs 🙈.
- Permissive IETF RNIC - an RNIC implementing the IETF protocols that is capable of implementing the RDMAC protocols on a per-connection basis ✨.
The Permissive IETF RNIC is recommended for those implementers who want maximum interoperability with other RNIC implementations 💖.
The values used by these three RNIC types for the MPA, DDP, and RDMAP versions, as well as MPA Markers and CRC, are summarized in Figure 14 🍓.
+----------------++-----------+-----------+-----------+-----------+
| RNIC TYPE || DDP/RDMAP | MPA | MPA | MPA |
| || Version | Revision | Markers | CRC |
+----------------++-----------+-----------+-----------+-----------+
+----------------++-----------+-----------+-----------+-----------+
| RDMAC || 0 | 0 | 1 | 1 |
| || | | | |
+----------------++-----------+-----------+-----------+-----------+
| IETF || 1 | 1 | 0 or 1 | 0 or 1 |
| Non-permissive || | | | |
+----------------++-----------+-----------+-----------+-----------+
| IETF || 1 or 0 | 1 or 0 | 0 or 1 | 0 or 1 |
| permissive || | | | |
+----------------++-----------+-----------+-----------+-----------+
Figure 14: Connection Parameters for the RNIC Types
for MPA Markers and MPA CRC, enabled=1, disabled=0.It is assumed there is no mixing of versions allowed between MPA, DDP, and RDMAP. The RNIC either generates the RDMAC protocols on the wire (version is zero) or uses the IETF protocols (version is one) 🫶.
During the exchange of the MPA Request/Reply Frames, each peer provides its MPA Revision, Marker preference (M: 0=disabled, 1=enabled), and CRC preference. The MPA Revision provided in the MPA Request Frame and the MPA Reply Frame may differ 🦋.
From the information in the MPA Request/Reply Frames, each side sets the Version field (V: 0=RDMAC, 1=IETF) of the DDP/RDMAP protocols, as well as the state of the Markers for each half connection. Between DDP and RDMAP, no mixing of versions is allowed. Moreover, the DDP and RDMAP version MUST be identical in the two directions 🔥. The RNIC either generates the RDMAC protocols on the wire (version is zero) or uses the IETF protocols (version is one) 💅.
In the following sections, the figures do not discuss CRC negotiation, because there is no interoperability issue for CRCs 🌸. Since the RDMAC RNIC will always request CRC use, then, according to the IETF MPA specification, both peers MUST generate and check CRCs 🎀✨.
C.2. RDMAC RNIC and Non-Permissive IETF RNIC ✨
okay so 🎀 Figure 15 is giving us the tea: a Non-permissive IETF RNIC cannot interoperate with an RDMAC RNIC, even though both peers swap MPA Request/Reply Frames. For a Non-permissive IETF RNIC, the MPA negotiation has zero effect on the DDP/RDMAP version, so it just can't vibe with the RDMAC RNIC 💀
Quick decoder ring for the figure 💅: the rows show the state of the Marker field in the MPA Request Frame sent by the MPA Initiator, and the columns show the Marker field in the MPA Reply Frame sent by the MPA Responder. Each type of RNIC shows up as both an Initiator and a Responder. The results live in the lower right corner, where the RNIC types intersect. V=0 is the RDMAC DDP/RDMAP version, V=1 is the IETF DDP/RDMAC version, M=0 means MPA Markers are disabled, and M=1 means MPA Markers are enabled. The negotiated Marker state is written as X/Y, for the receive direction of the Initiator/Responder 🦋
+---------------------------++-----------------------+
| MPA || MPA |
| CONNECT || Responder |
| MODE +-----------------++-------+---------------+
| | RNIC || RDMAC | IETF |
| | TYPE || | Non-permissive|
| | +------++-------+-------+-------+
| | |MARKER|| M=1 | M=0 | M=1 |
+---------+----------+------++-------+-------+-------+
+---------+----------+------++-------+-------+-------+
| | RDMAC | M=1 || V=0 | close | close |
| | | || M=1/1 | | |
| +----------+------++-------+-------+-------+
| MPA | | M=0 || close | V=1 | V=1 |
|Initiator| IETF | || | M=0/0 | M=0/1 |
| |Non-perms.+------++-------+-------+-------+
| | | M=1 || close | V=1 | V=1 |
| | | || | M=1/0 | M=1/1 |
+---------+----------+------++-------+-------+-------+Figure 15: MPA Negotiation between an RDMAC RNIC and a Non-Permissive IETF RNIC
C.2.1. RDMAC RNIC Initiator 🌸
so if the RDMAC RNIC is the MPA Initiator, its ULP sends an MPA Request Frame with the Rev field set to zero and the M and C bits set to one. The Non-permissive IETF RNIC can't dynamically downgrade the version number it uses for DDP and RDMAP, so it sends back an MPA Reply Frame with the Rev field equal to one and then gracefully closes the connection. bye bye 👋
C.2.2. Non-Permissive IETF RNIC Initiator 🧚
now flip it. If the Non-permissive IETF RNIC is the MPA Initiator, it sends an MPA Request Frame with the Rev field equal to one. The ULP or supporting entity for the RDMAC RNIC responds with an MPA Reply Frame that has the Rev field equal to zero and the M bit set to one. The Non-permissive IETF RNIC will gracefully close the connection after it reads that incompatible Rev field in the MPA Reply Frame 🥲
C.2.3. RDMAC RNIC and Permissive IETF RNIC 💜
Figure 16 is where it gets better bestie 🙌 A Permissive IETF RNIC can interoperate with an RDMAC RNIC regardless of its Marker preference. The figure uses the same format as the Non-permissive IETF RNIC one, so you already know how to read it ✨
+---------------------------++-----------------------+
| MPA || MPA |
| CONNECT || Responder |
| MODE +-----------------++-------+---------------+
| | RNIC || RDMAC | IETF |
| | TYPE || | Permissive |
| | +------++-------+-------+-------+
| | |MARKER|| M=1 | M=0 | M=1 |
+---------+----------+------++-------+-------+-------+
+---------+----------+------++-------+-------+-------+
| | RDMAC | M=1 || V=0 | N/A | V=0 |
| | | || M=1/1 | | M=1/1 |
| +----------+------++-------+-------+-------+
| MPA | | M=0 || V=0 | V=1 | V=1 |
|Initiator| IETF | || M=1/1 | M=0/0 | M=0/1 |
| |Permissive+------++-------+-------+-------+
| | | M=1 || V=0 | V=1 | V=1 |
| | | || M=1/1 | M=1/0 | M=1/1 |
+---------+----------+------++-------+-------+-------+Figure 16: MPA Negotiation between an RDMAC RNIC and a Permissive IETF RNIC
A truly Permissive IETF RNIC is the real MVP 👑 It recognizes an RDMAC RNIC from the Rev field of the MPA Req/Rep Frames, then adjusts its receive Marker state and DDP/RDMAP version to accommodate the RDMAC RNIC. Because of that, as an MPA Responder, the Permissive IETF RNIC will never return an MPA Reply Frame with the M bit set to zero. That case shows up as not applicable (N/A) in Figure 16 🙈
C.2.4. RDMAC RNIC Initiator 🍓
same vibe as C.2.1, just with a Permissive IETF Responder this time 🫶 When the RDMAC RNIC is the MPA Initiator, its ULP or other supporting entity prepares an MPA Request message and sets the revision to zero and the M bit and C bit to one.
The Permissive IETF Responder receives that MPA Request message and checks the revision field. Since it's capable of generating RDMAC DDP/RDMAP headers, it sends an MPA Reply message with the revision set to zero and the M and C bits set to one. The Responder must inform its ULP that it is generating version zero DDP/RDMAP messages. 📣
C.2.5 Permissive IETF RNIC Initiator 🦋
if the Permissive IETF RNIC is the MPA Initiator, it prepares the MPA Request Frame with the Rev field set to one. No matter what the M bit is in that MPA Request Frame, the ULP or other supporting entity for the RDMAC RNIC will create an MPA Reply Frame with Rev equal to zero and the M bit set to one.
When the Initiator reads the Rev field of the MPA Reply Frame and finds that its peer is an RDMAC RNIC, it must inform its ULP that it should generate version zero DDP/RDMAP messages and enable MPA Markers and CRC. this is the glow up 🌈
C.3. Non-Permissive IETF RNIC and Permissive IETF RNIC ⭐
for completeness, Figure 17 shows the results of MPA negotiation between a Non-permissive IETF RNIC and a Permissive IETF RNIC. The important takeaway 🔥 is that an IETF RNIC cannot detect whether its peer is a Permissive or Non-permissive RNIC.
+---------------------------++-------------------------------+
| MPA || MPA |
| CONNECT || Responder |
| MODE +-----------------++---------------+---------------+
| | RNIC || IETF | IETF |
| | TYPE || Non-permissive| Permissive |
| | +------++-------+-------+-------+-------+
| | |MARKER|| M=0 | M=1 | M=0 | M=1 |
+---------+----------+------++-------+-------+-------+-------+
+---------+----------+------++-------+-------+-------+-------+
| | | M=0 || V=1 | V=1 | V=1 | V=1 |
| | IETF | || M=0/0 | M=0/1 | M=0/0 | M=0/1 |
| |Non-perms.+------++-------+-------+-------+-------+
| | | M=1 || V=1 | V=1 | V=1 | V=1 |
| | | || M=1/0 | M=1/1 | M=1/0 | M=1/1 |
| MPA +----------+------++-------+-------+-------+-------+
|Initiator| | M=0 || V=1 | V=1 | V=1 | V=1 |
| | IETF | || M=0/0 | M=0/1 | M=0/0 | M=0/1 |
| |Permissive+------++-------+-------+-------+-------+
| | | M=1 || V=1 | V=1 | V=1 | V=1 |
| | | || M=1/0 | M=1/1 | M=1/0 | M=1/1 |
+---------+----------+------++-------+-------+-------+-------+Figure 17: MPA negotiation between a Non-permissive IETF RNIC and a Permissive IETF RNIC.
Normative References 📚
[iSCSI] Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and E. Zeidner, "Internet Small Computer Systems Interface (iSCSI)", RFC 3720, April 2004.
[RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191, November 1990.
[RFC2018] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP Selective Acknowledgment Options", RFC 2018, October 1996.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2401] Kent, S. and R. Atkinson, "Security Architecture for the Internet Protocol", RFC 2401, November 1998.
[RFC3723] Aboba, B., Tseng, J., Walker, J., Rangan, V., and F. Travostino, "Securing Block Storage Protocols over IP", RFC 3723, April 2004.
[RFC793] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981.
[RDMASEC] Pinkerton, J. and E. Deleganes, "Direct Data Placement Protocol (DDP) / Remote Direct Memory Access Protocol (RDMAP) Security", RFC 5042, October 2007.
Informative References ✨
Okay, these are the bonus reads that back up the spec. Bestie, bookmark them 📚🦋
[APPL] Bestler, C. and L. Coene, "Applicability of Remote
Direct Memory Access Protocol (RDMA) and Direct Data
Placement (DDP)", RFC 5045, October 2007.
[CRCTCP] Stone J., Partridge, C., "When the CRC and TCP checksum
disagree", ACM Sigcomm, Sept. 2000.
[DAT-API] DAT Collaborative, "kDAPL (Kernel Direct Access
Programming Library) and uDAPL (User Direct Access
Programming Library)", Http://www.datcollaborative.org.
[DDP] Shah, H., Pinkerton, J., Recio, R., and P. Culley,
"Direct Data Placement over Reliable Transports", RFC
5041, October 2007.
[iSER] Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah,
H., and P. Thaler, "Internet Small Computer System
Interface (iSCSI) Extensions for Remote Direct Memory
Access (RDMA)" RFC 5046, October 2007.
[IT-API] The Open Group, "Interconnect Transport API (IT-API)"
Version 2.1, http://www.opengroup.org.
[NFSv4CHAN] Williams, N., "On the Use of Channel Bindings to Secure
Channels", Work in Progress, June 2006.
[RDMA-DDP] "Direct Data Placement over Reliable Transports (Version
1.0)", RDMA Consortium, October 2002,
<http://www.rdmaconsortium.org/home/draft-shah-iwarp-
ddp-v1.0.pdf>.
[RDMA-MPA] "Marker PDU Aligned Framing for TCP Specification
(Version 1.0)", RDMA Consortium, October 2002,
<http://www.rdmaconsortium.org/home/draft-culley-iwarp-
mpa-v1.0.pdf>.
[RDMA-RDMAC] "An RDMA Protocol Specification (Version 1.0)", RDMA
Consortium, October 2002,
<http://www.rdmaconsortium.org/home/draft-recio-iwarp-
rdmac-v1.0.pdf>.
[RDMAP] Recio, R., Culley, P., Garcia, D., Hilland, J., and B.
Metzler, "A Remote Direct Memory Access Protocol
Specification", RFC 5040, October 2007.
[RFC792] Postel, J., "Internet Control Message Protocol", STD 5,
RFC 792, September 1981.
[RFC896] Nagle, J., "Congestion control in IP/TCP internetworks",
RFC 896, January 1984.
[RFC1122] Braden, R., "Requirements for Internet Hosts -
Communication Layers", STD 3, RFC 1122, October 1989.
[RFC4960] Stewart, R., Ed., "Stream Control Transmission
Protocol", RFC 4960, September 2007.
[RFC4296] Bailey, S. and T. Talpey, "The Architecture of Direct
Data Placement (DDP) and Remote Direct Memory Access
(RDMA) on Internet Protocols", RFC 4296, December 2005.
[RFC4297] Romanow, A., Mogul, J., Talpey, T., and S. Bailey,
"Remote Direct Memory Access (RDMA) over IP Problem
Statement", RFC 4297, December 2005.
[RFC4301] Kent, S. and K. Seo, "Security Architecture for the
Internet Protocol", RFC 4301, December 2005.
[VERBS-RMDA] "RDMA Protocol Verbs Specification", RDMA Consortium
standard, April 2003, <http://www.rdmaconsortium.org/
home/draft-hilland-iwarp-verbs-v1.0-RDMAC.pdf>.Contributors 🫶
Huge shoutout to these legends who helped shape this spec. Their contact details are below, exactly as they were listed 💜
Dwight Barron
Hewlett-Packard Company
20555 SH 249
Houston, TX 77070-2698 USA
Phone: 281-514-2769
EMail: dwight.barron@hp.com
Jeff Chase
Department of Computer Science
Duke University
Durham, NC 27708-0129 USA
Phone: +1 919 660 6559
EMail: chase@cs.duke.edu
Ted Compton
EMC Corporation
Research Triangle Park, NC 27709 USA
Phone: 919-248-6075
EMail: compton_ted@emc.com
Dave Garcia
24100 Hutchinson Rd.
Los Gatos, CA 95033
Phone: 831 247 4464
EMail: Dave.Garcia@StanfordAlumni.org
Hari Ghadia
Gen10 Technology, Inc.
1501 W Shady Grove Road
Grand Prairie, TX 75050
Phone: (972) 301 3630
EMail: hghadia@gen10technology.com
Howard C. Herbert
Intel Corporation
MS CH7-404
5000 West Chandler Blvd.
Chandler, AZ 85226
Phone: 480-554-3116
EMail: howard.c.herbert@intel.com
Jeff Hilland
Hewlett-Packard Company
20555 SH 249
Houston, TX 77070-2698 USA
Phone: 281-514-9489
EMail: jeff.hilland@hp.com
Mike Ko
IBM
650 Harry Rd.
San Jose, CA 95120
Phone: (408) 927-2085
EMail: mako@us.ibm.com
Mike Krause
Hewlett-Packard Corporation, 43LN
19410 Homestead Road
Cupertino, CA 95014 USA
Phone: +1 (408) 447-3191
EMail: krause@cup.hp.com
Dave Minturn
Intel Corporation
MS JF1-210
5200 North East Elam Young Parkway
Hillsboro, Oregon 97124
Phone: 503-712-4106
EMail: dave.b.minturn@intel.com
Jim Pinkerton
Microsoft, Inc.
One Microsoft Way
Redmond, WA 98052 USA
EMail: jpink@microsoft.com
Hemal Shah
Broadcom Corporation
5300 California Avenue
Irvine, CA 92617 USA
Phone: +1 (949) 926-6941
EMail: hemal@broadcom.com
Allyn Romanow
Cisco Systems
170 W Tasman Drive
San Jose, CA 95134 USA
Phone: +1 408 525 8836
EMail: allyn@cisco.com
Tom Talpey
Network Appliance
1601 Trapelo Road #16
Waltham, MA 02451 USA
Phone: +1 (781) 768-5329
EMail: thomas.talpey@netapp.com
Patricia Thaler
Broadcom
16215 Alton Parkway
Irvine, CA 92618
Phone: 916 570 2707
EMail: pthaler@broadcom.com
Jim Wendt
Hewlett Packard Corporation
8000 Foothills Boulevard MS 5668
Roseville, CA 95747-5668 USA
Phone: +1 916 785 5198
EMail: jim_wendt@hp.com
Jim Williams
Emulex Corporation
580 Main Street
Bolton, MA 01740 USA
Phone: +1 978 779 7224
EMail: jim.williams@emulex.comAuthors' Addresses 🎀
And last but definitely not least, here's how to reach the authors who wrote the spec. Slay, every one of them 👑
Paul R. Culley
Hewlett-Packard Company
20555 SH 249
Houston, TX 77070-2698 USA
Phone: 281-514-5543
EMail: paul.culley@hp.com
Uri Elzur
5300 California Avenue
Irvine, CA 92617, USA
Phone: 949.926.6432
EMail: uri@broadcom.com
Renato J Recio
IBM
Internal Zip 9043
11400 Burnett Road
Austin, Texas 78759
Phone: 512-838-3685
EMail: recio@us.ibm.com
Stephen Bailey
Sandburst Corporation
600 Federal Street
Andover, MA 01810 USA
Phone: +1 978 689 1614
EMail: steph@sandburst.com
John Carrier
Cray Inc.
411 First Avenue S, Suite 600
Seattle, WA 98104-2860
Phone: 206-701-2090
EMail: carrier@cray.comFull Copyright Statement 📜
Copyright (C) The IETF Trust (2007). 💅
This document is subject to the rights, licenses and restrictions contained in BCP 78. Except as set out there, the authors keep all their rights 🔐
This document and the information in it are provided on an "AS IS" basis. THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 😭
Intellectual Property 🔥
The IETF doesn't take a side on whether any Intellectual Property Rights or other rights claimed to cover this technology are valid or how far they reach, or on whether a license under them would be available. It also doesn't claim to have checked for any such rights itself. For the rules on rights in RFC documents, see BCP 78 and BCP 79 🧐
Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses to be made available, and the results of any attempt to get a general license or permission to use these proprietary rights for implementers or users of this specification can be found in the IETF's online IPR repository at http://www.ietf.org/ipr. 🌈
The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology needed to implement this standard. Please send that information to the IETF at ietf-ipr@ietf.org. 🦋
📦 where this page comes from
- ietf: rfc5044 2007-10
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms