RFC 5045
Applicability of Remote Direct Memory Access Protocol (RDMA) and Direct Data Placement (DDP)
RFC 5045 📜🎀
Okay bestie, this is RFC 5045 💖, titled Applicability of Remote Direct Memory Access Protocol (RDMA) and Direct Data Placement Protocol (DDP). It's an Informational doc from October 2007, written by C. Bestler, Ed. (Neterion) and L. Coene (Nokia Siemens Networks). Informational means it's here to inform the Internet community and doesn't set a standard. The header doesn't list any RFCs it obsoletes or updates. 🦋
Status of This Memo
This memo is for information only, so the Internet community can read it. It is not an Internet standard of any kind 🙈. Anyone is free to distribute it, with no limits. 💅
Abstract
This doc explains where RDMAP (Remote Direct Memory Access Protocol) and DDP (Direct Data Placement Protocol) are a good fit ✨. It compares the different IP transports DDP can ride on, and gives ULP developers tips for picking one, or for not caring which transport is underneath at all 🫶. It also compares using DDP with using those transports directly, and DDP over IP transports with non-IP transports that have RDMA built in. Main character energy 👑.
Table of Contents 🌸
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Direct Placement . . . . . . . . . . . . . . . . . . . . . . . 5
3.1. Direct Placement Using Only the LLP . . . . . . . . . . . 5
3.2. Fewer Required ULP Interactions . . . . . . . . . . . . . 6
4. Tagged Messages . . . . . . . . . . . . . . . . . . . . . . . 6
4.1. Order-Independent Reception . . . . . . . . . . . . . . . 7
4.2. Reduced ULP Notifications . . . . . . . . . . . . . . . . 7
4.3. Simplified ULP Exchanges . . . . . . . . . . . . . . . . . 8
4.4. Order-Independent Sending . . . . . . . . . . . . . . . . 9
4.5. Untagged Messages and Tagged Buffers as ULP Credits . . . 10
5. RDMA Read . . . . . . . . . . . . . . . . . . . . . . . . . . 12
6. LLP Comparisons . . . . . . . . . . . . . . . . . . . . . . . 13
6.1. Multistreaming Implications . . . . . . . . . . . . . . . 13
6.2. Out-of-Order Reception Implications . . . . . . . . . . . 13
6.3. Header and Marker Overhead . . . . . . . . . . . . . . . . 13
6.4. Middlebox Support . . . . . . . . . . . . . . . . . . . . 14
6.5. Processing Overhead . . . . . . . . . . . . . . . . . . . 14
6.6. Data Integrity Implications . . . . . . . . . . . . . . . 14
6.6.1. MPA/TCP Specifics . . . . . . . . . . . . . . . . . . 15
6.6.2. SCTP Specifics . . . . . . . . . . . . . . . . . . . . 15
6.7. Non-IP Transports . . . . . . . . . . . . . . . . . . . . 15
6.7.1. No RDMA-Layer Ack . . . . . . . . . . . . . . . . . . 16
6.8. Other IP Transports . . . . . . . . . . . . . . . . . . . 16
6.9. LLP-Independent Session Establishment . . . . . . . . . . 17
6.9.1. RDMA-Only Session Establishment . . . . . . . . . . . 17
6.9.2. RDMA-Conditional Session Establishment . . . . . . . . 18
7. Local Interface Implications . . . . . . . . . . . . . . . . . 18
8. Security Considerations . . . . . . . . . . . . . . . . . . . 19
8.1. Connection/Association Setup . . . . . . . . . . . . . . . 19
8.2. Tagged Buffer Exposure . . . . . . . . . . . . . . . . . . 19
8.3. Impact of Encrypted Transports . . . . . . . . . . . . . . 19
9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 19
9.1. Normative References . . . . . . . . . . . . . . . . . . . 19
9.2. Informative References . . . . . . . . . . . . . . . . . . 191. Introduction ✨
RDMAP [RFC5040] and DDP [RFC5041] work as a team 👯♀️. Together they place application payload straight into the buffers the ULP (Upper Layer Protocol) asks for, in a way that doesn't depend on the application and runs efficiently.
DDP handles the direct placement of incoming payload into the buffers the ULP specifies. RDMAP gives the ULP completion notifications, and supports RDMA Reads, where the Data Sink fetches an Advertised Buffer 🦋.
Both DDP and RDMAP are application-independent, and both let the ULP do remote direct data placement. DDP can run over several standard IP transports, including SCTP and TCP 🌈.
By spelling out when these protocols are useful, this doc helps implementers, application designers, and protocol designers decide which ones to use 🧠.
The applicability of RDMAP/DDP comes from their unique features:
- This doc looks at when common data placement procedures help applications more than app-specific solutions built straight on the transport 💅.
- DDP supports both Untagged and Tagged Buffers. Tagged Buffers let the Data Sink ULP stop caring what order the Data Source sent things in, or which messages they came in, or what order packets arrive in 🤯. Usually tagged data is for moving payload, and untagged is best for control messages. But each ULP can decide the best use of Tagged and Untagged Messages for itself. This doc also covers when Data Source flexibility is useful for apps.
- RDMAP combines ULP notifications, which cuts down the number of required ULP interactions ⭐.
- RDMAP defines RDMA Reads, which give remote access to Advertised Buffers. This doc compares RDMA Reads with other options 🔥.
A more complete intro to RDMAP and DDP, along with their security considerations, is in [RFC5042] 📖.
Some non-IP transports, like InfiniBand, have RDMA features built right in. This doc looks at offering RDMA services over everyday IP transports, instead of custom transport protocols 🌸. Because DDP is cleanly defined as a layer on top of existing IP transports, its ordering rules are simpler than some earlier RDMA protocols. That can affect app designers, so keep it in mind 🙈.
RDMA and DDP only shine fully when apps are designed to use them. The doc also covers how RDMAP/DDP-aware local interfaces can live alongside traditional socket interfaces 🧚.
Last but not least, DDP is defined for at least two IP transports: SCTP [RFC5043] and TCP [RFC5044]. The doc explains why both are supported, and when each one is the right pick 💜.
2. Definitions 📚
- Advertisement - Telling a Remote Peer that a local RDMA Buffer is ready for it. A Node makes an RDMA Buffer available for incoming RDMA Read or RDMA Write access by telling its RDMA/DDP peer the Tagged Buffer identifiers (STag, base address, and buffer length). DDP/RDMA doesn't define how this Advertisement of Tagged Buffer information happens. That's up to the ULP. A common way is for the Local Peer to embed the Tagged Buffer's Steering Tag, base address, and length in a Send Message going to the Remote Peer 💌.
- Data Sink - The peer that receives a data payload. The Data Sink might have to both send and receive RDMA/DDP Messages to move a payload 🫧.
- Data Source - The peer that sends a data payload. The Data Source might also have to both send and receive RDMA/DDP Messages to move a payload 🫧.
- Lower Layer Protocol (LLP) - The transport protocol that provides services to DDP. This is an IP transport with whatever adaptation layer it needs. Adaptation layers are defined for SCTP and TCP 🔧.
- Steering Tag (STag) - An identifier for a Tagged Buffer on a Node. It's valid as defined inside a protocol spec 🏷️.
- Tagged Message - A DDP message that goes into a ULP-specified buffer based on embedded addressing info. The message sender is the one who picks the destination buffer, right away. The receiver gets no separate signal that a Tagged Message arrived 🎯.
- Untagged Message - A DDP message that goes into a ULP-specified buffer by matching a Message Sequence Number with a buffer the receiver supplied. The receiver picks the destination buffer. Some other mechanism tells the receiver that an Untagged Message arrived 📬.
- Upper Layer Protocol (ULP) - The direct user of RDMAP/DDP services. Besides protocols like iSER [RFC5046] and NFSv4 over RDMA [NFSDIRECT], the ULP can live inside an app or a middleware layer, which is often how the Sockets Direct Protocol (SDP) and Remote Procedure Call (RPC) protocols do it 🍓.
3. Direct Placement 🎀
Direct Data Placement makes it so ULP Payload lands in the correct destination buffers. Usually that means no intermediate copying, which is huge ⚡. Placement works no matter what order things arrive or get sent in, and it doesn't need an interaction with the ULP for every single placement 🙌.
RDMAP cuts down on the required ULP interactions. This helps the most with apps that need several transport layer packets for each required ULP interaction 🚀.
3.1. Direct Placement Using Only the LLP 💅
Direct data placement can happen without RDMA. Pre-posting receive buffers could let a non-RDMA network stack place data directly into user buffers 🌷.
How much DDP helps depends on which transport it's being compared with, and on what the local interface looks like. Without RDMAP/DDP, pre-posting buffers means the receiving side has to predict the exact buffers and their sizes ahead of time. That's not doable for every ULP 😬. DDP only asks the ULP to predict the sequence and size of incoming Untagged Messages, which is way easier 💖.
An app that can predict incoming messages and only needs direct placement into buffers might manage it with a well-designed local interface to native SCTP or TCP (no RDMA). That's easier with native SCTP, because the app only has to predict the sequence of messages and the maximum size of each one, not the exact size 🦋.
The main perk DDP gives such an app is that pre-posting receive buffers is a required local interface capability. Predictions can also always be made per message, not per byte 🎁.
The Lower Layer Protocol, LLP, can also be used directly, if ULP-specific knowledge is built into the protocol stack so it can do "parse and place" handling of received packets. That kind of solution either needs interaction with the ULP, or needs the protocol stack to know the ULP's specific syntax rules 🤓.
DDP gets the benefits of directly placing incoming payload without tightly coupling the ULP and the protocol stack. Still, "parse and place" can give equivalent services to a limited number of ULPs 🌈.
To be continued! Part 2 is coming up next, bestie 💕🫶
3.2. Fewer Required ULP Interactions
Cutting down on required ULP interactions is already a whole vibe 💅, but it's absolutely critical for high-speed connections 🔥. The burst packet rate on a fast interface could easily outrun the host system's ability to switch ULP contexts 😭. Packets arriving faster than the host can keep up with context switching? Not it, bestie 💀.
Content access apps are the big examples of apps that need high bandwidth and can move a significant amount of content between required ULP interactions 🎀. That includes file access protocols (NAS), storage access (SAN), database access, and other app-specific forms of content access like HTTP, XML, and email 📬.
4. Tagged Messages ✨
This section covers the major benefits from using Tagged Messages 🦋.
Okay, the biggest perk of DDP is that the Data Source can use Tagged Buffers 👑. Tagging messages lets the Data Source choose the ordering and packetization of its payload deliveries 🧚. With direct data placement based only on pre-posted receives, the packetization and delivery of payload has to be agreed on by the ULP peers in advance 🙈.
The Upper Layer Protocol can split content between Untagged and/or Tagged Messages to get the most out of the optimizations 🌈. Putting content in an Untagged Message can deliver it in the same packet that signals completion to the receiver, which can improve latency 💖. It can even get rid of round trips 🔥. But it means larger anonymous buffers have to be available 🥺.
Some examples of data that typically belongs in an Untagged Message:
- short fixed-size control data that is inherently part of the control message. This is especially true when the data is a required part of the control message.
- relatively short payload that is almost always needed, especially when including it would eliminate a round-trip to fetch the data. Examples would include the initial data on a write request and Advertisements of Tagged Buffers.
Tagged Messages standardize direct placement of data without per-packet interaction with the upper layers 🎯. Even if there's an upper-layer protocol encoding of what's being transferred, like with middleware solutions, that info isn't understood at the application-independent layers 🤷. The directions for where to place incoming data can't be accessed without switching to the ULP first 🙃. DDP gives you a standardized 'packing list' that can be interpreted without ULP interaction 📝. And honestly? It's designed to be implementable in hardware 🤖.
4.1. Order-Independent Reception
Tagged Messages get directed to a buffer based on an included Steering Tag 🎯. Also, no notice is given to the ULP for each individual Tagged Message's arrival 🤫. Together, these let Tagged Messages that arrive out of order get processed without intermediate buffering or additional notifications to the ULP 💅.
4.2. Reduced ULP Notifications
RDMAP offers both Tagged and Untagged Messages 🦋. No receiving-side ULP interactions are required for Tagged Messages ✨. By dividing traffic optimally between Tagged and Untagged Messages, the ULP can limit the number of events it has to deal with at the ULP layer 🎀. This typically reduces the number of context switches required and improves performance 📈.
RDMAP goes even further to reduce required ULP interactions, consolidating completion notifications of Tagged Messages with the completion notification of a trailing Untagged Message 🫶. For most ULPs, this radically reduces the number of required ULP interactions even further 🔥.
While RDMAP's consolidation of notices is beneficial to most applications, it may be detrimental to some apps that benefit from streamed delivery, where the ULP processes received data as promptly as possible 😬. A ULP that uses RDMAP can't begin processing any portion of an exchange until it gets notification that the entire exchange has been placed 🙅. An "exchange" here is a set of zero or more Tagged Messages and a single terminating Untagged Message 🧩. An app that would prefer to start on the received payload as soon as possible, no matter what order it arrived in, might prefer to work directly with the LLP 🏃. RDMAP is optimized for apps that care more about when the entire exchange is complete 💜.
An app that benefits from being able to begin processing each received packet as quickly as possible may find RDMAP interferes with that goal 🚧.
Such an app might be able to keep most of the benefits of RDMAP by using the DDP layer directly 💡. However, on top of taking on the responsibilities of the RDMAP layer, the app would likely have more difficulty finding support for a DDP-only API 😵. Many hardware implementations may choose to tightly couple RDMAP and DDP, and might not provide an API directly to DDP services 🔌.
These features minimize the required interactions with the ULP, which is extremely beneficial for apps that use multiple transport layer packets to accomplish what is a single ULP interaction 🙌.
4.3. Simplified ULP Exchanges
The notification rules for Tagged Messages let ULPs create multi-message "exchanges" made of zero or more Tagged Messages that represent a single step in the ULP interaction 🧸. The receiving ULP is notified that the Untagged Message has arrived, and is implicitly notified of any associated Tagged Messages too 🔔.
If a ULP can't effectively use Tagged Messages, it would get little benefit from RDMAP/DDP compared to using SCTP directly 🥱. But while Tagged Buffers are the justification for RDMAP/DDP, Untagged Buffers are still necessary 💗. Without Untagged Buffers, the only way to exchange buffer Advertisements would be out-of-band communications 📡. Most RDMA-aware ULPs use Untagged Buffers for requests and responses, and buffer Advertisements are typically done within these Untagged Messages 📨.
More importantly, there would be no reliable method for the upper-layer peers to synchronize 😱. The absence of any guarantees about ordering within or between Tagged Messages is fundamental to allowing the DDP layer to optimize transfer of tagged payload 🎢.
Therefore, no ULP can be defined entirely in terms of Tagged Messages 🚫. Eventually, a notification that confirms delivery must be generated from the RDMAP/DDP layer ✅.
Limiting Untagged Buffers to requests and responses, and moving all bulk data using tagged transfers, can greatly simplify the amount of prediction the Data Sink has to do when pre-posting receive buffers 🧠. For example, a typical RDMA-enabled interaction would consist of the following:
- Client sends transaction request to server as an Untagged Message.
- This message includes buffer Advertisements for the buffers where the results are to be placed.
- The server sends multiple Tagged Messages to the Advertised buffers.
- The server sends transaction reply as an Untagged Message to the client.
- Client receives single notification, indicating completion of the interaction.
With this type of exchange, the pacing and required size of Untagged Buffers are highly predictable 📊. The variability of response sizes is absorbed by tagged transfers 🫧.
4.4. Order-Independent Sending
Tagged Messages are especially applicable when the Data Sink doesn't know the actual size, structure, or location of the content it's requesting (or updating) 🔍.
For example, suppose the Data Sink ULP needs to fetch four related pieces of data into four separate buffers 🧩. With SCTP, the Data Sink ULP could receive four messages into four separate buffers, only having to predict the maximum size of each 📦. However, it would have to dictate the order in which the Data Source supplied the separate pieces 😤. If the Data Source found it advantageous to fetch them in a different order, it would have to use intermediate buffering to re-order the pieces into the expected order, even though the application only required that all four be delivered and did not truly have an ordering requirement 🙄.
Techniques such as RAID striping and mirroring represent this same problem, but one step further 🪞. What appears to be a single resource to the Data Sink is actually stored in separate locations by the Data Source 🗂️. Non RDMA protocols would either require the Data Source to fetch the material in the desired order, or force the Data Source to use its own holding buffers to assemble an image of the destination buffer 😩.
While sometimes referred to as a "buffer-to-buffer" solution, RDMA more fundamentally enables remote buffer access 💫. The ULP is free to work with larger remote buffers than it has locally 🌟. This reduces buffering requirements and the number of times the data must be copied in an end-to-end transfer 💅.
There are numerous reasons why the Data Sink wouldn't know the true order or location of the requested data 🤷♀️. It could be different for each client, different records selected and/or different sort orders, as well as RAID striping, file fragmentation, volume fragmentation, volume mirroring, and server-side dynamic compositing of content (such as server-side includes for HTTP) 🌐.
In all of these cases, the Data Source is free to assemble the desired data in the Data Sink's buffers in whatever order the component data becomes available to it 🎉. It is not constrained on ordering 🙌. It does not have to assemble an image in its own memory before creating it in the Data Sink's buffers 🧠.
Note that while DDP enables use of Tagged Messages for bulk transfer, there are some application scenarios where Untagged Messages would still be used for bulk transfer 📌. For example, a file server may not expose its own memory to its clients 🗄️. A client wishing to write may Advertise a buffer upon which the server will issue RDMA Reads 📣. However, when performing a small write, it may be preferable to include the data in the Untagged Message rather than incurring an additional round trip with the RDMA Read and its response 🙏.
Generally, the best use of an Untagged Message is to synchronize and to deliver data that is naturally tied to the same message as the synchronization 💕. For initial data transfers, this has the additional benefit of avoiding the need to Advertise specific Tagged Buffers for indefinite time periods 🫠. Instead, anonymous buffers can be used for initial data reception 🫶. Because anonymous buffers do not need to be tied to specific messages in advance, this can be a major benefit 🏆.
4.5. Untagged Messages and Tagged Buffers as ULP Credits 🎀
okay so the end-to-end buffer credit situation is very different with DDP than when the ULP just uses TCP or SCTP directly 💅
with both TCP and SCTP, the credits are based on the receiver giving the sender permission to transmit according to the total number of bytes. these credits are about system buffering resources and/or simple flow control. they are NOT ULP resources 🙈
DDP doesn't define any standard flow control 😭 but it does assume a ULP mechanism exists. the mechanism it assumes is that the Data Sink ULP has issued credits to the Data Source, so the Data Source can send a specific number of Untagged Messages. bestie it's giving "you get exactly this many, no more" 💜
the ULP peers MUST ensure that the sender knows the maximum size that can be sent to any specific target buffer. one way to do this is to use one standard size for all Untagged Buffers within a given connection 🍓 for example, a ULP might specify an initial Untagged Buffer size to be used right after session establishment, and then optionally specify mechanisms for negotiating changes ✨
Tagged Buffers are ULP resources that get Advertised directly from ULP to ULP 🦋 a DDP put to a known Tagged Buffer is only constrained by transport level flow control, NOT by how much system buffering is available. iconic 👑
either Tagged or Untagged Buffers let you bypass system buffer resources. the difference is that using Tagged Buffers also lets the Data Source choose what order to use its credits in 🧚
to the extent the ULP allows it, Tagged Buffers are also divisible resources 🌸 the Data Sink can Advertise a single 100 KB buffer, and then get notifications from its peer saying it wrote 50 KB, 20 KB, and 30 KB to that buffer in three successive transactions. it's literally a buffer that gets eaten in bites 🍰
managing Tagged Buffer resources at the ULP level, separately from transport and DDP layer credits, is an extra perk of RDMA protocols 🔥 large bulk transfers can't get blocked by limited general-purpose buffering capacity. applications can do flow control based on higher level stuff, like the number of outstanding requests, no matter how much data has to move. main character energy ⭐
that said, using system buffering, the way you do when you use the underlying transports directly, can be the better choice in some situations 🙈
one example: when the number of target ULP Buffers is big enough, and writes arrive slowly enough, that pinning all the target ULP Buffers in memory would be a bad idea 💀 the maximum transfer rate, and therefore the maximum amount of system buffering needed, may be more stable and predictable than the total ULP Buffer exposure 🫶
another example: when the Data Sink wants to receive a stream of data at a predictable rate, but doesn't know ahead of time how big each packet will be. this is common with streaming media encoded at a variable bit rate 🎶 with DDP, the Data Sink would either have to use Untagged Buffers big enough for the largest packet, or Advertise a circular buffer. if, for security or other reasons, the Data Sink doesn't want its buffer size to be publicly known, using the underlying SCTP transport directly may be the better move, because of its byte-oriented credits 🔐
5. RDMA Read ✨
RDMA Reads are another service that RDMAP provides 🌈 they let the Data Sink fetch exactly the part of the peer ULP Buffer it needs on a "just in time" basis. and the Data Source ULP doesn't need to support every single fetch, which is so convenient 💖
storage servers might want to limit the maximum write buffer allocated to any one session 🗄️ the storage server might be a super minimal layer between the client and the disk storage media, or it might just want to cap the total resources that would be needed if every client could push its entire payload whenever it felt like it 😤
either way, there's not much benefit in pulling data from the Data Source way before it'll actually be written to persistent storage media 🙅♀️ RDMA Reads let the Storage Server fetch the payload "just in time." so a relatively small number of block-sized buffers can run one transaction that writes a large file 🧠 or a Storage Server with lots of clients can grab buffers from each client in whatever order is most convenient for the server 🫶
the same thing works when the part of the Advertised Buffer you want isn't known in advance 🔍 for example, the Advertised Buffer could hold performance statistics. the Data Sink could ask for only the portions it needs, without having to talk to the Data Source ULP at all 🦋
this is a great fit for lots of apps that publish semi-volatile data that doesn't need transactional validity checking (aka authorized users can read the whole data set) 📚 it's a worse fit when there are ULP consistency checks that have to run on the data 😬 those apps are better off having the client send a request, and having the server use RDMA Writes to publish the requested data 💌 neither RDMAP nor DDP gives you a way to bundle multiple disjoint updates into one atomic operation. so using an Advertised Buffer as a data resource comes with the same caveats as any randomly updated data resource, like flat files that don't enforce their own consistency 🥲
6. LLP Comparisons 🌸
normally, the choice of underlying IP transport doesn't matter to the ULP 💅 RDMAP and DDP provide the same services over either one. there can still be performance impacts from the choice though 📈 it's the ULP's job to figure out which IP transport is best for its needs 🧐
SCTP preserves message boundaries 📦 each DDP Segment gets delivered within a single SCTP packet. TCP can only offer equivalent services by using the MPA (Marker PDU Alignment) adaptation layer 🩵
6.1. Multistreaming Implications 🎀
SCTP also has multi-streaming 🧵 when the same pair of hosts needs multiple DDP streams, this can be a huge advantage 👑 a single SCTP association carries multiple DDP streams, which consolidates connection setup, congestion control, and acknowledgements. so much less overhead, it's so efficient 🥰
completions are controlled by the DDP Source Sequence Number (DDP-SSN) on a per-stream basis 🔢 so combining multiple DDP Streams into a single SCTP association can't mean that a dropped packet carrying data for one stream holds up completions on the others 🙌
6.2. Out-of-Order Reception Implications 🔀
the use of unordered Data Chunks with SCTP guarantees that the DDP layer will be able to do placements when IP datagrams arrive out of order 💖
placement of out-of-order DDP Segments carried over MPA/TCP isn't guaranteed, but it's certainly allowed 🤷♀️ the MPA receiver's ability to process out-of-order DDP Segments may get impaired when the alignment of TCP segments and MPA FPDUs is lost 😵💫 with SCTP, each DDP Segment is encoded in a single Data Chunk and never spread over multiple IP datagrams 🫶
6.3. Header and Marker Overhead 📏
MPA and TCP headers together are smaller than the headers used by SCTP and its adaptation layer 🤏 however, this advantage can be cut down by inserting MPA markers 📍 the difference in ULP Payload per IP Datagram isn't likely to be a significant factor 🙅♀️
6.4. Middlebox Support 🛡️
even with the MPA adaptation layer, DDP traffic carried over MPA/TCP will look like a normal TCP connection to every network middlebox 🫣 in lots of environments there may be a requirement to use only TCP connections, to satisfy existing network elements and/or make monitoring and control of connections easier 📊 SCTP is just as monitorable and controllable as TCP, sure, but there's no guarantee the network management infrastructure has the support it needs for both 😤
6.5. Processing Overhead 🧠
a DDP stream delivered via MPA/TCP will take more processing effort than one delivered over SCTP 🏋️♀️ but that extra work may be worth it in plenty of deployments, where full SCTP support isn't available in the endpoints of the network, or where middleboxes make SCTP hard to use ✨
6.6. Data Integrity Implications 🔐
both the SCTP [RFC4960] and MPA/TCP [RFC5044] adaptations provide end-to-end CRC32c protection against accidental data corruption, or its equivalent 🛡️
a ULP that needs more protection can add its own 💪 however, DDP and RDMAP headers are only guaranteed to have the equivalent of end-to-end CRC32c protection 😬 a ULP that needs data integrity checking more thorough than end-to-end CRC32c should first invalidate all STags that reference a buffer before applying its own integrity check 🚫
⚠️ CRC32c only protects against random corruption. to protect against unauthorized alteration or forging of data packets, security methods must be applied. the RDMA security document [RFC5042] specifies the use of RFC 2406 [RFC2406] for both adaptation layers. as [RFC5042] points out, the IPsec requirements for RDDP are based on the version of IPsec specified in RFC 2401 [RFC2401] and related RFCs, as profiled by RFC 3723 [RFC3723], even though a newer version of IPsec is specified in RFC 4301 [RFC4301] and related RFCs. ⚠️
6.6.1. MPA/TCP Specifics 🧩
it is mandatory for MPA/TCP implementations to implement CRC32c, but it is not mandatory to use the CRC32c during an RDMA connection 🤓 turning the CRC on or off is an administrative configuration operation at the local and remote end. the administration of the CRC (ON/OFF) is invisible to the ULP 🫥
applications should assume that disabling CRC32c will only be used when the end-to-end protection is at least as effective as a transport layer CRC32c 🥹 applications should not use additional integrity checks based only on the possibility that CRC32c could be disabled without equivalent integrity checks at a lower level 🙃
CRC32c must not be disabled unless equivalent or better end-to-end integrity protection is provided 🔒
if the CRC is active/used for one direction/end, then using the CRC is mandatory in both directions/ends 🔁
if both ends have been configured not to use the CRC, that's allowed as long as an equivalent protection (comparable to or better than CRC) against undetected errors on the connection is provided 🫶
6.6.2. SCTP Specifics 💜
SCTP provides CRC32c protection automatically ✅ the adaptation to SCTP has no option to turn off SCTP CRC32c protection. no escape hatch, bestie 🚪🔒
6.7. Non-IP Transports 🌐
DDP is defined to operate over ubiquitous IP transports like SCTP and TCP 🌍 this means a new DDP-enabled node can be added anywhere in an IP network 🎉 no DDP-specific support from middleboxes is required 🙌
there are non-IP transport fabrics that offer RDMA capabilities 🛸 because these capabilities are integrated with the transport protocol, they have some technical advantages over RDMA over IP 🤩 for example, fencing of RDMA Operations can be based on transport level acks. but because DDP is cleanly layered over an IP transport, any explicit RDMA layer ack has to be separate from the transport layer ack 🧱
there may be deployments where the benefits of RDMA/transport integration outweigh the benefits of being on an IP network 💫
6.7.1. No RDMA-Layer Ack 🫶
Okay so DDP doesn't have its own acks, bestie 💅 The only ack at the RDMAP layer is an RDMA Read Response. DDP and RDMAP lean almost entirely on other layers for flow control and pacing. The LLP is trusted to guarantee delivery and avoid network congestion, and ULP-level acking is relied on for ULP pacing and to avoid ULP Buffer overruns ✨
Previous RDMA protocols, like InfiniBand, could use their integration with the transport layer to give stronger ordering guarantees 🔥 So it's important that app designers who need those guarantees provide them through ULP interaction. Not optional, babe 🙈
Specifically:
- There is no way for a local interface to "fence" outbound messages to guarantee that prior Tagged Messages have been placed before sending a Tagged Message 😭 The only guarantees you can get from the other side are an RDMA Read Response (from the RDMAP layer) or a response from the ULP layer. Remember that the normal ordering rules only guarantee when the Data Sink ULP will be notified of Untagged Messages. They do not control when data gets placed into receive buffers.
- Re-use of Tagged Buffers must be done with extreme care 🧚 The fact that an Untagged Message indicates all prior Tagged Messages have been placed does not guarantee that no later Tagged Message has. The best strategy is to change only the state of any given Advertised Buffers with Untagged Messages.
- As covered elsewhere in this document, flow control of Untagged Messages is the responsibility of the ULP 👑
6.8. Other IP Transports
Both TCP and SCTP give DDP reliable transport with TCP-friendly rate control 🌸 Right now DDP is defined to work over reliable transports, and it implicitly relies on some form of rate control.
DDP is fully compatible with a non-reliable protocol 🦋 Out-of-order placement is obviously not dependent on whether the other DDP Segments ever actually arrive.
However, RDMAP requires the LLP to provide reliable service. An alternate completion handling protocol would be required if DDP were to be deployed over an unreliable IP transport.
As noted in the prior section on Tagged Buffers as ULP credits, neither RDMAP nor DDP provides any flow control for Tagged Messages. If no transport layer flow control is provided, an RDMAP/DDP application would be limited only by the link layer rate, almost inevitably resulting in severe network congestion 💀
RDMAP encourages applications to not care about the underlying transport path MTU ✨ The ULP is only notified when all messages ending in a single Untagged Message have completed. The ULP is not aware of the granularity or ordering of the underlying message. This approach assumes that the ULP is only interested in the complete set of messages, and has no use for a subset of them.
6.9. LLP-Independent Session Establishment
For an RDMAP/DDP application, the transport services provided by a pair of SCTP streams and by a TCP connection both provide the same service: reliable delivery of DDP Segments between two connected RDMAP/DDP endpoints 🎀
6.9.1. RDMA-Only Session Establishment
It is also possible to allow for transport-neutral establishment of RDMAP/DDP sessions between endpoints 🦋 Combined, these two features would let most applications be unconcerned as to which LLP is actually in use.
Specifically, the procedures for DDP Stream Session establishment discussed in section 3 of the SCTP mapping, and section 13.3 of the MPA/TCP mapping, both allow for the exchange of ULP-specific data ("Private Data") before enabling the exchange of DDP Segments. This delay can allow for proper selection and/or configuration of the endpoints based upon the exchanged data. For example, each DDP Stream Session associated with a single client session might be assigned to the same DDP Protection Domain 💜
To be transport neutral, applications should exchange Private Data as part of session establishment messages to determine how the RDMA endpoints are to be configured. One side must be the Initiator, and the other, the Responder.
With SCTP, a pair of SCTP streams can be used for successive sessions while the SCTP association remains open. With MPA/TCP, each connection can be used for, at most, one session. However, the same source/destination pair of ports can be re-used for a subsequent TCP connection, as allowed by TCP.
Both SCTP and MPA limit the private data size to a maximum of 512 bytes 🍓
MPA/TCP requires the end of the TCP connection that initiated the conversion to MPA mode to send the first DDP Segment. SCTP does not have this requirement. ULPs that wish to be transport neutral should require the initiating end to send the first message. A zero-length RDMA Write can be used for this purpose if the ULP logic itself does naturally support this restriction 🫶
6.9.2. RDMA-Conditional Session Establishment
It is sometimes desirable for the active side of a session to connect with the passive side before knowing whether the passive side supports RDMA 🙈
This style of session establishment can be supported with either TCP or SCTP, but not as transparently as for RDMA-only sessions. Pre-existing non-RDMA servers are also far more likely to be using TCP than SCTP.
With TCP, a normal TCP connection is established. It is then used by the ULP to determine whether or not to convert to MPA mode and use RDMA. This will typically be integral with other session-establishment negotiations.
With SCTP, the establishment of an association tests whether RDMA is supported. If it is not supported, the application simply requests the association without the RDMA adaptation indication.
One key difference is that with SCTP, the determination as to whether the peer can support RDMA is made before the transport layer association/connection is established. With TCP, the established connection itself is used to determine whether RDMA is supported 🌈
7. Local Interface Implications 🎀
Full utilization of DDP and RDMAP capabilities requires a local interface that explicitly requests these services. Protocols such as Sockets Direct Protocol (SDP) can allow applications to keep their traditional byte-stream or message-stream interface and still enjoy many of the benefits of the optimized wire level protocols ✨
8. Security Considerations 🔐
RDMA security considerations are discussed in the RDMA security document [RFC5042]. This document will only deal with the more usage-oriented aspects, and where there are implications in the choice of underlying transport.
8.1. Connection/Association Setup
Both the SCTP and TCP adaptations allow for existing procedures to be followed for the establishment of the SCTP association or TCP connection. Use of DDP does not impair the use of any security measures to filter, validate, and/or log the remote end of an association/connection 🛡️
8.2. Tagged Buffer Exposure
DDP only exposes ULP memory to the extent explicitly allowed by ULP actions. These include posting of receive operations and enabling of Steering Tags 🔒
Neither RDMAP nor DDP places requirements on how ULPs Advertise Buffers. A ULP may use a single Steering Tag for multiple buffer Advertisements. However, the ULP should be aware that enforcement on STag usage is likely limited to the overall range that is enabled. If the Remote Peer writes into the 'wrong' Advertised Buffer, neither the DDP nor the RDMAP layer will be aware of this. Nor is there any report to the ULP on how the Remote Peer specifically used Tagged Buffers.
Wrong-buffer writes are invisible to both DDP and RDMAP, and nobody reports to the ULP how the Remote Peer used the Tagged Buffers. Scary stuff, bestie 😱
Unless the ULP peers have an adequate basis for mutual trust, the receiving ULP might be well advised to use a distinct STag for each interaction, and to invalidate it after each use, or to require its peer to use the RDMAP option to invalidate the STag with its responding Untagged Message 🫶
8.3. Impact of Encrypted Transports
While DDP is cleanly layered over the LLP, its maximum benefit may be limited when the LLP Stream is secured with a streaming cypher, such as Transport Layer Security (TLS) [RFC4346]. If the LLP must decrypt in order, it cannot provide out-of-order DDP Segments to the DDP layer for placement purposes. IPsec [RFC2401] tunnel mode encrypts entire IP Datagrams. IPsec transport mode encrypts TCP Segments or SCTP packets, as does use of Datagram TLS (DTLS) [RFC4347] over UDP beneath TCP or SCTP. Neither IPsec nor this use of DTLS precludes providing out-of-order DDP Segments to the DDP layer for placement 🦋
Note that end-to-end use of cryptographic integrity protection may allow suppression of MPA CRC generation and checking under certain circumstances. This is one example where the LLP may be judged to have "or equivalent" protection to an end-to-end CRC32c 💖
9. References 📚
9.1. Normative References
[RFC2401] Kent, S. and R. Atkinson, "Security Architecture for the
Internet Protocol", RFC 2401, November 1998.
[RFC2406] Kent, S. and R. Atkinson, "IP Encapsulating Security
Payload (ESP)", RFC 2406, November 1998.
[RFC4960] Stewart, R., "Stream Control Transmission Protocol",
RFC 4960, September 2007.
[RFC5040] Recio, R., Metzler, B., Culley, P., Hilland, J., and D.
Garcia, "A Remote Direct Memory Access Protocol
Specification", RFC 5040, October 2007.
[RFC5041] Shah, H., Pinkerton, J., Recio, R., and P. Culley,
"Direct Data Placement over Reliable Transports",
RFC 5041, October 2007.
[RFC5042] Pinkerton, J. and E. Deleganes, "DDP/RDMAP Security",
RFC 5042, October 2007.
[RFC5043] Bestler, C. and R. Stewart, "Stream Control Transmission
Protocol (SCTP) Direct Data Placement (DDP) Adaptation",
RFC 5043, October 2007.
[RFC5044] Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
Carrier, "Marker PDU Aligned Framing for TCP
Specification", RFC 5044, October 2007.9.2. Informative References
[NFSDIRECT] Talpey, T., Callaghan, B., and I. Property, "NFS Direct
Data Placement", Work in Progress, June 2007.
[RFC3723] Aboba, B., Tseng, J., Walker, J., Rangan, V., and F.
Travostino, "Securing Block Storage Protocols over IP",
RFC 3723, April 2004.
[RFC4301] Kent, S. and K. Seo, "Security Architecture for the
Internet Protocol", RFC 4301, December 2005.
[RFC4346] Dierks, T. and E. Rescorla, "The Transport Layer
Security (TLS) Protocol Version 1.1", RFC 4346,
April 2006.
[RFC4347] Rescorla, E. and N. Modadugu, "Datagram Transport Layer
Security", RFC 4347, April 2006.
[RFC5046] Ko, M., Chadalapaka, M., Elzur, U., Shah, H., and P.
Thaler, "Internet Small Computer System Interface
(iSCSI) Extensions for Remote Direct Memory Access
(RDMA)", RFC 5046, October 2007.Authors' Addresses 🖋️
Caitlin Bestler (editor)
Neterion
20230 Stevens Creek Blvd.
Suite C
Cupertino, CA 95014
USA
Phone: 408-366-4639
EMail: caitlin.bestler@neterion.com
Lode Coene
Nokia Siemens Networks
Atealaan 26
Herentals 2200
Belgium
Phone: +32-14-252081
EMail: lode.coene@nsn.comFull Copyright Statement ✨
okay so the legal bit is up next, and it's giving very "read the fine print, bestie" 📜💅 The IETF Trust owns the copyright on this one, dated 2007. This document is covered by the rights, licenses and restrictions in BCP 78. Outside of what BCP 78 says, the authors keep all their rights 💖
THE DISCLAIMER, VERBATIM BECAUSE IT'S SERIOUS 🙈: This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property 🦋
the IETF is staying neutral here, lowkey iconic of them 🫶 It takes no position on whether any IPR or other rights that someone might claim over the tech in this document are valid or how far they reach. It also doesn't say whether a license is available. And it doesn't claim to have checked for any of those rights itself 🙈
If you want the details on how rights work for RFC documents, go read BCP 78 and BCP 79. They're the source of truth 📚✨
Copies of IPR disclosures sent to the IETF Secretariat, plus any assurances that licenses will be made available, live in the IETF online IPR repository at http://www.ietf.org/ipr. You can also find the result of any attempt to get a general license or permission to use those proprietary rights by implementers or users of this specification there 💜
The IETF also invites anyone to flag any copyrights, patents, patent applications, or other proprietary rights that might cover tech needed to implement this standard. Send that to the IETF at ietf-ipr@ietf.org 💌🌸
📦 where this page comes from
- ietf: rfc5045 2007-10
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms