RFC 5666
Remote Direct Memory Access Transport for Remote Procedure Call
RFC 5666: Remote Direct Memory Access Transport for Remote Procedure Call 💖
okay bestie, this is RFC 5666 💅 It's an Internet Standards Track document, so it's the real official deal ✨ It came out in January 2010 from the IETF (Internet Engineering Task Force). It has two authors: T. Talpey (Unaffiliated) and B. Callaghan (Apple). It's got ISSN 2070-1721 🎀 The header doesn't have any Obsoletes or Updates lines, so it isn't replacing or patching anything else.
Abstract ✨
okay so this doc describes a protocol that uses Remote Direct Memory Access (RDMA) as a brand new transport for Remote Procedure Call (RPC) 🦋 The RDMA transport binding brings the perks of efficient, bulk-data transport over high-speed networks, and it does that with minimal change to RPC applications. There's no required revision of the application RPC protocol, or of the RPC protocol itself 🔥
Status of This Memo 📜
This is an Internet Standards Track document 💜
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). If you want more info on Internet Standards, check out Section 2 of RFC 5741 🌸
Info about the current status of this document, any errata, and how to give feedback on it can be found at http://www.rfc-editor.org/info/rfc5666 ✨
Copyright Notice 🎀
Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved 💖
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication of this document. Please review these documents carefully, because they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they are provided without warranty as described in the Simplified BSD License 🫶
This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from the person(s) controlling the copyright in such materials, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format it for publication as an RFC or to translate it into languages other than English 🌈
Table of Contents 🗂️
1. Introduction ....................................................3
1.1. Requirements Language ......................................4
2. Abstract RDMA Requirements ......................................4
3. Protocol Outline ................................................5
3.1. Short Messages .............................................6
3.2. Data Chunks ................................................6
3.3. Flow Control ...............................................7
3.4. XDR Encoding with Chunks ...................................8
3.5. XDR Decoding with Read Chunks .............................11
3.6. XDR Decoding with Write Chunks ............................12
3.7. XDR Roundup and Chunks ....................................13
3.8. RPC Call and Reply ........................................14
3.9. Padding ...................................................17
4. RPC RDMA Message Layout ........................................18
4.1. RPC-over-RDMA Header ......................................18
4.2. RPC-over-RDMA Header Errors ...............................20
4.3. XDR Language Description ..................................20
5. Long Messages ..................................................22
5.1. Message as an RDMA Read Chunk .............................23
5.2. RDMA Write of Long Replies (Reply Chunks) .................24
6. Connection Configuration Protocol ..............................25
6.1. Initial Connection State ..................................26
6.2. Protocol Description ......................................26
7. Memory Registration Overhead ...................................28
8. Errors and Error Recovery ......................................28
9. Node Addressing ................................................28
10. RPC Binding ...................................................29
11. Security Considerations .......................................30
12. IANA Considerations ...........................................31
13. Acknowledgments ...............................................32
14. References ....................................................33
14.1. Normative References .....................................33
14.2. Informative References ...................................331. Introduction ✨
okay so Remote Direct Memory Access (RDMA) [RFC5040, RFC5041], [IB] is a technique for efficient movement of data between end nodes, and it's getting more and more iconic over high-speed transports 🚀 It works by directing data into destination buffers as it's sent on a network, then placing it via direct memory access by hardware. That gives you the double benefit of faster transfers and reduced host overhead 💅
Open Network Computing Remote Procedure Call (ONC RPC, or simply, RPC) [RFC5531] is a remote procedure call protocol that has been run over a variety of transports. Most RPC implementations today use UDP or TCP 🦋 RPC messages are defined in terms of an eXternal Data Representation (XDR) [RFC4506], which gives you a canonical data representation across a variety of host architectures. An XDR data stream gets conveyed differently on each type of transport. On UDP, RPC messages are encapsulated inside datagrams, while on a TCP byte stream, RPC messages are delineated by a record marking protocol. An RDMA transport also conveys RPC messages in a unique fashion that must be fully described if client and server implementations are going to interoperate 🫶
RDMA transports present new semantics that are unlike the behaviors of either UDP or TCP alone 🌸 They keep message delineations like UDP does, while also providing a reliable, sequenced data transfer like TCP. On top of that, they provide the new efficient, bulk-transfer service of RDMA. So RDMA transports are naturally viewed as a new transport type by RPC 💖
RDMA as a transport is going to boost the performance of RPC protocols that move large "chunks" of data, since RDMA hardware is extremely good at moving data efficiently between host memory and a high-speed network with little or no host CPU involvement 🔥 In this context, the Network File System (NFS) protocol, in all its versions [RFC1094] [RFC1813] [RFC3530] [RFC5661], is an obvious beneficiary of RDMA 👑 A complete problem statement is discussed in [RFC5532], and related NFSv4 issues are discussed in [RFC5661]. Many other RPC-based protocols will also benefit 🧚
Although the RDMA transport described here provides relatively transparent support for any RPC application, the proposal goes further by describing mechanisms that can optimize the use of RDMA with more active participation by the RPC application ✨
1.1. Requirements Language 🎀
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 💜
2. Abstract RDMA Requirements 🌈
okay this part is the foundation, so pay attention 🤓 An RPC transport is responsible for conveying an RPC message from a sender to a receiver. An RPC message is either an RPC call from a client to a server, or an RPC reply from the server back to the client. An RPC message contains an RPC call header followed by arguments if the message is an RPC call, or an RPC reply header followed by results if the message is an RPC reply. The call header contains a transaction ID (XID) followed by the program and procedure number as well as a security credential. An RPC reply header begins with an XID that matches that of the RPC call message, followed by a security verifier and results. All data in an RPC message is XDR encoded. For a complete description of the RPC protocol and XDR encoding, see [RFC5531] and [RFC4506] 🫶
This protocol assumes the following abstract model for RDMA transports. These terms are common in the RDMA lexicon, and they're used in this document. A more complete glossary of RDMA terms can be found in [RFC5040] 💅
- Registered Memory: All data moved via tagged RDMA operations is resident in registered memory at its destination. This protocol assumes that each segment of registered memory MUST be identified with a steering tag of no more than 32 bits and memory addresses of up to 64 bits in length 🔥
- RDMA Send: The RDMA provider supports an RDMA Send operation with completion signaled at the receiver when data is placed in a pre-posted buffer. The amount of transferred data is limited only by the size of the receiver's buffer. Sends complete at the receiver in the order they were issued at the sender 🦋
- RDMA Write: The RDMA provider supports an RDMA Write operation to directly place data in the receiver's buffer. An RDMA Write is initiated by the sender and completion is signaled at the sender. No completion is signaled at the receiver. The sender uses a steering tag, memory address, and length of the remote destination buffer. RDMA Writes are not necessarily ordered with respect to one another, but they are ordered with respect to RDMA Sends. A subsequent RDMA Send completion obtained at the receiver guarantees that prior RDMA Write data has been successfully placed in the receiver's memory 🌸
- RDMA Read: The RDMA provider supports an RDMA Read operation to directly place peer source data in the requester's buffer. An RDMA Read is initiated by the receiver and completion is signaled at the receiver. The receiver provides steering tags, memory addresses, and a length for the remote source and local destination buffers. Since the peer at the data source receives no notification of RDMA Read completion, there is an assumption that on receiving the data, the receiver will signal completion with an RDMA Send message, so that the peer can free the source buffers and the associated steering tags 💖
This protocol is designed to be carried over all RDMA transports meeting the stated requirements. It conveys to the RPC peer enough information for that peer to direct an RDMA layer to perform transfers containing RPC data and to communicate their result(s). For example, it is readily carried over RDMA transports such as Internet Wide Area RDMA Protocol (iWARP) [RFC5040, RFC5041], or InfiniBand [IB] 👑
3. Protocol Outline ✨
okay so an RPC message gets sent the exact same way whether it's a call or a reply 💖 Before the actual message shows up, there's a transport-specific header for RPC-over-RDMA transports to read first. It's kind of like the record marking used for RPC over TCP, except way more extensive, because RDMA transports support several modes of data transfer 🦋 The upper-layer protocol gets to pick the most efficient mode for each segment of a message, which is honestly so smart. One message can have its segments moved in different ways to different remote memory destinations 🧚
Every transfer of a call or reply starts with an RDMA Send that carries at least the RPC-over-RDMA header 🎀 Usually the call or reply is attached right behind it, or at least some part of it is. The catch is that an RDMA Send can only carry as much as the receiver's pre-posted buffer can hold, so the transport gives you a bunch of ways to shrink what goes through the RDMA Send when it's needed. Parts of the message get moved with RDMA Read and RDMA Write instead 😭🫶
RPC-over-RDMA framing replaces all the other RPC framing (like TCP record marking) once you're on an RPC/RDMA association 💅 Even if the underlying RDMA protocol is itself layered on top of something with its own defined RPC framing (like TCP), that's still how it works. It's also possible to turn on RPC/RDMA dynamically while you're negotiating RDMA through an upper-layer exchange 🌸 Since RPC framing marks the start and end of a whole RPC request or reply, the framing shift has to happen between distinct RPC messages, and it has to be coordinated with the transport. Main character timing, no cap 👑
3.1. Short Messages
Lots of RPC messages are super short 🍓 For example, an NFS version 3 GETATTR request is only 56 bytes: 20 bytes of RPC header, plus a 32-byte file handle argument and 4 bytes of length. The reply to this very common request is about 100 bytes 🙈
Moving a tiny message with an RDMA Read or Write is literally not worth it 💀 Setting up steering tags and memory addresses only pays off for big transfers. The critical message size where RDMA transfer starts making sense depends on the RDMA implementation and the network, but it's typically in the range of a few kilobytes ✨ For short messages, an RDMA Send into a pre-posted buffer is the right call 🦋 The RPC-over-RDMA header plus the short message (call or reply) right after it all goes out in one single RDMA Send operation 🎀
Short RPC messages over an RDMA transport:
RPC Client RPC Server
| RPC Call |
Send | ------------------------------> |
| |
| RPC Reply |
| <------------------------------ | Send3.2. Data Chunks
Some protocols, like NFS, have RPC procedures that can move really big chunks of data inside the RPC call or reply 💜 If you tried to send those as part of the RDMA Send, you'd blow past the maximum send size, which is a whole mess 😭 These large chunks usually range from a kilobyte to a megabyte or more. An RDMA transport can move them way more efficiently by using direct placement with an RDMA Read or RDMA Write operation 🔥 Direct placement beats inline transfer because it skips expensive data copies, and it also gives you correct data alignment at the destination. Iconic efficiency behavior 👑
3.3. Flow Control
Flow control for RDMA Send is critical for an RDMA connection, bestie 🚨 RDMA receive operations fail if there isn't a pre-posted receive buffer ready to take an incoming RDMA Send, and if that keeps happening it can be fatal to the connection 💀 This is a big departure from conventional TCP/IP networking, where buffers get allocated dynamically as needed and nobody has to pre-post anything 🌈
Fixed credit limits at the RPC server aren't practical, honestly 🙅♀️ Fixed limits scale poorly, because posted buffers stay locked to the connection until receive operations consume them. Also, for protocol correctness, the RPC server must always be able to reply to client requests, whether or not new buffers have been posted for future receives 🫶 (Side note: the RPC server might actually be a client at some other layer. For example, NFSv4 callbacks are processed by the NFSv4 client, acting as an RPC server. The credit discussions apply equally in either case 🧚)
Flow control for RDMA Send is a simple request/grant protocol in the RPC-over-RDMA header attached to each RPC message 💖 The header on RPC call messages carries a requested credit value for the RPC server, and the caller MAY dynamically adjust it to match what it expects to need 🎀 The header on RPC reply messages carries the granted result. That value MAY have any value, except it MUST NOT be zero when there are no in-progress operations at the server, because zero would cause deadlock 💅 The value MAY be adjusted up or down at each opportunity to match the server's needs or policies ✨
The RPC client MUST NOT send unacknowledged requests beyond this granted RPC server credit limit 🚫 If the limit gets exceeded, the RDMA layer may signal an error, and it might terminate the connection. Even if no error happens, it is OPTIONAL for the server to handle the excess request(s), and it MAY return an RPC error to the client 🙈 Also, the never-zero rule means an RPC server MUST always provide at least one credit to each connected RPC client that has no outstanding requests. Otherwise the client would deadlock, unable to send another request, and that's a total vibe killer 😭
RPC calls can complete in any order, but the RPC client always knows the current flow control limit at the RPC server from the Send ordering properties 🦋 It's always the most recent server-granted credit value minus the number of requests in flight. Math that slaps 💅
Some RDMA implementations may impose extra flow control restrictions, like limits on RDMA Read operations in progress at the responder 🌸 Those operations are outside the scope of this protocol, so they aren't addressed here, and other layers SHOULD be the ones to provide for them 💜 For example, a simple upper-layer RPC consumer might do single-issue RDMA Read requests, while a more sophisticated, multithreaded RPC consumer might build its own First In, First Out (FIFO) queue of those operations 🧚 If you want more on possible protocol implementations that can negotiate these values, check out Section 6 "Connection Configuration Protocol" of this document, or [RFC5661] 🎀🌈
3.4. XDR Encoding with Chunks ✨
Okay so an RPC call or reply message gets marshaled (aka serialized) into one contiguous stream by an XDR routine 📦. XDR data types like integers, strings, arrays, and linked lists are usually built on two super simple functions: one that encodes an XDR data unit (32 bits), and one that encodes an array of bytes 💅
Normally, the separate data items in an RPC call or reply get encoded as a contiguous sequence of bytes for network transmission over UDP or TCP. BUT, when the transport is RDMA, local routines like XDR encode can figure out that (for instance) an opaque byte array is big enough to be moved more efficiently with an RDMA data transfer operation like RDMA Read or RDMA Write 🚀 main character energy for that byte array, honestly 🎀
Semantically speaking, the protocol has no restriction on which data types can or can't be represented by a read or write chunk 🌈 In practice though, efficiency means certain data types are not generally "chunkable" 🙈 Typically, only the opaque and aggregate data types that can get substantial size are considered eligible. With today's hardware, that size may be a kilobyte or more. However, any object MAY be chosen for chunking in any given message 💖
The eligibility of XDR data items to be candidates for being moved as data chunks (as opposed to being marshaled inline) is not specified by the RPC-over-RDMA protocol 🦋 Chunk eligibility criteria MUST be determined by each upper layer so that the spec stays interoperable 🫶 One example, with rationale, for the NFS protocol family is provided in [RFC5667] 🌸
The interface that lets an upper-layer implementation tell RPC locally that a data item is eligible for chunking is out of scope for this specification 🧚 In many implementations it's possible to build a transparent RPC chunking facility. However, those implementations may lead to inefficiencies. They might make the RPC layer do expensive registration and de-registration of memory "on the fly", or they might require RDMA chunks in reply messages, along with the resulting extra handshaking with the RPC-over-RDMA peer 😭 But these issues are internal and generally confined to the local interface between RPC and its upper layers, which is a space where implementations are free to innovate ✨ The only requirement is that the resulting RPC RDMA protocol sent to the peer is valid for the upper layer. See, for example, [RFC5667] 💜
When sending any message (request or reply) that contains an eligible large data chunk, the XDR encoding routine avoids moving the data into the XDR stream 🙅♀️ Instead, it doesn't encode the data portion. It records the address and size of each chunk in a separate "read chunk list", which gets encoded within the RPC RDMA transport-specific headers 📝 Those chunks will be transferred via RDMA Read operations initiated by the receiver 🔥
When the read chunks are going to be moved via RDMA, the memory for each chunk gets registered. This registration may take place within XDR itself, which gives full transparency to upper layers, or it may be performed by any other specific local implementation 🎀
Also, when making an RPC call that can result in bulk data transferred in the reply, write chunks MAY be provided to accept the data directly via RDMA Write 💅 These write chunks will be pre-filled by the RPC server before it responds, so the client doesn't need XDR decode of the data 🙌 They go through a similar registration and advertisement via "write chunk lists" that get built as part of XDR encoding 🌈
Some RPC client implementations can't determine where an RPC call's results reside during the "encode" phase 😵💫 That makes it difficult or impossible for the RPC client layer to encode the write chunk list at the time it builds the request. In this case, it's hard for the RPC implementation to stay transparent to the RPC consumer, who might need to recode to provide result information at this earlier stage 🥺
So, if the RPC client doesn't make a write chunk list available to receive the result, then the RPC server MAY return data inline in the reply. Or, if the upper-layer specification permits, it MAY be returned via a read chunk list 🧚 It is NOT RECOMMENDED that upper-layer RPC client protocol specifications omit write chunk lists for eligible replies 🚫 That's because of the lower performance from the additional handshaking needed to transfer the data, plus the server has to expose (and preserve) the reply data for a period of time. Without a server-provided read chunk list in the reply, if the encoded reply overflows the posted receive buffer, the RPC will fail with an RDMA transport error 💀
When any data within a message is provided via read or write chunks, the chunk itself refers only to the data portion of the XDR stream element 🍓 For counted fields (e.g., a "<>" encoding), the byte count encoded as part of the field stays in the XDR stream, and it's also encoded in the chunk list. The data portion is elided from the encoded XDR stream and gets transferred as part of chunk list processing 🎀 It's important to keep upper-layer implementation compatibility: both the count and the data must be transferred as part of the logical XDR stream 🫶 Once chunk list processing has made the data available to the upper-layer peer for XDR decoding, the length in the chunk list entries doesn't get passed along 🙈 Any byte count in the XDR stream MUST match the sum of the byte counts present in the corresponding read or write chunk list. If they don't agree, an RPC protocol encoding error results 🚨
The following items are contained in a chunk list entry 📋
- Handle Steering tag or handle obtained when the chunk memory is registered for RDMA.
- Length The length of the chunk in bytes.
- Offset The offset or beginning memory address of the chunk. To support the widest array of RDMA implementations, as well as the most general steering tag scheme, this field is unconditionally included in each chunk list entry 💖
Zero-based offset schemes are available in many RDMA implementations, but using them with RPC requires individual registration of each read or write chunk 😩 On many such implementations, that can be a significant overhead. By providing an offset in each chunk, many pre-registration or region-based registrations can be readily supported. Using a single, universal chunk representation also keeps the RPC RDMA protocol implementation as simple as it can get, in its most general form ✨
- Position For data that is to be encoded, the position in the XDR stream where the chunk would normally reside. The chunk inserts its data into the XDR stream at this position, but its transfer is no longer "inline". So all chunks belonging to a single RPC argument or result will have the same position 🧩 For data that is to be decoded, no position is used.
When XDR marshaling is complete, the chunk list is XDR encoded, then sent to the receiver prepended to the RPC message 📬 Any source data for a read chunk, or the destination of a write chunk, stays behind in the sender's registered memory. Their actual payload isn't marshaled into the request or reply 🦋
+----------------+----------------+-------------
| RPC-over-RDMA | |
| header w/ | RPC Header | Non-chunk args/results
| chunks | |
+----------------+----------------+-------------Read chunk lists and write chunk lists are structured somewhat differently 👀 That's because they're used differently. Read chunks are decoded and indexed by their argument's or result's position in the XDR data stream, and their size is always known. Write chunks, on the other hand, are only used for results. They have neither a preassigned offset in the XDR stream nor a size until the results are produced, since the buffers may be only partially filled, or may not be used for results at all. So their presence in the XDR stream isn't known until the reply is processed 🤯 The mapping of write chunks onto designated NFS procedures and their results is described in [RFC5667] 💜
So, read chunks are encoded into a read chunk list as a single array, with each entry tagged by its (known) size and its argument's or result's position in the XDR stream 📚 Write chunks are encoded as a list of arrays of RDMA buffers, with each list element (an array) providing buffers for a separate result. Individual write chunk list elements MAY thereby result in being partially or fully filled, or in fact not being filled at all 🎀 Unused write chunks, or unused bytes in write chunk buffer lists, aren't returned as results, and their memory is returned to the upper layer as part of RPC completion. However, the RPC layer MUST NOT assume that the buffers have not been modified 🙅♀️
3.5. XDR Decoding with Read Chunks 🎀
The XDR decode process moves data from an XDR stream into a data structure provided by the RPC client or server application 🧁 Where elements of the destination data structure are buffers or strings, the RPC application can either pre-allocate storage to receive the data, or leave the string or buffer fields null and let the XDR decode stage of RPC processing automatically allocate storage of sufficient size 🙌
When decoding a message from an RDMA transport, the receiver first XDR decodes the chunk lists from the RPC-over-RDMA header, then proceeds to decode the body of the RPC message (arguments or results) 📖 Whenever the XDR offset in the decode stream matches that of a chunk in the read chunk list, the XDR routine initiates an RDMA Read to bring the chunk data over into locally registered memory for the destination buffer 🚀
When processing an RPC request, the RPC receiver (RPC server) acknowledges that it's done using the source buffers simply by replying to the RPC sender (client). Then the peer may free all source buffers advertised by the request ✨
When processing an RPC reply, after completing such a transfer, the RPC receiver (client) MUST issue an RDMA_DONE message (described in Section 3.8) to notify the peer (server) that the source buffers can be freed. 🚨
The read chunk list is constructed and used entirely within the RPC/XDR layer 🧚 Other than specifying the minimum chunk size, the management of the read chunk list is automatic and transparent to an RPC application 💖
3.6. XDR Decoding with Write Chunks ✨
okay so when the client sends a write chunk list with its RPC call, the RPC server MUST provide the matching data by doing an RDMA Write into the memory that the chunk list entries point at 🦋 The RPC reply then sends the write chunk list back to the client with the lengths updated to match what was actually transferred. So the XDR decode of the reply doesn't move any data locally. It just reads the length from the reply 💅
Each decoded result uses up one entry in the write chunk list, and each entry is an array of RDMA segments 🌸 That means the length is the total of all the returned lengths across all the segments in that entry. Once you've decoded an entry, the whole entry is used up 💜
The write chunk list is built and used by the RPC application itself 🎀 The RPC/XDR layer just passes the list between client and server and kicks off the RDMA Writes back to the client. How write chunk list entries map to procedure arguments MUST be decided for each protocol 🧚 There's an example of one mapping in [RFC5667] if you want to see it in action.
3.7. XDR Roundup and Chunks 🍓
okay this one is a bit of a deep dive but it's giving very important 🤓 XDR wants every new encoded element to start on a 4-byte boundary in the XDR stream. The reason is efficiency and easier decoding on the receiving end. If the stream buffer starts on a native machine boundary, the XDR elements land at predictable offsets in memory 🌈
So when you encode something that isn't a multiple of 4 bytes, like an odd-length string or bulk data, the length gets encoded as-is, but the data itself gets padded so the next element starts on a 4-byte boundary 💖 For TCP or RDMA inline encoding, that small overhead is needed, because the transport framing depends on the fact that an element's offset from the start of the message tells you its XDR position during decode 🫶
RPC/RDMA Read chunks are a different story though 👑 Each chunk carries the XDR position of the chunked element and the length of the Chunk segment. The receiver can place the data exactly where it belongs in its memory, no matter how the position lines up in the XDR stream. Any rounded-up padding isn't actually part of the upper layer's message, so the receiver never references it, and there's no reason to set it to any particular value in its memory 🦋
When roundup shows up at the end of a sequence of chunks, the length of that sequence ends it at a position that isn't 4-byte aligned in XDR. When the receiver goes on to decode the rest of the XDR stream, it checks the XDR position of the next chunk. That position won't match, because if it did, roundup wouldn't have happened 🙈 So the receiver falls back to looking at the remaining inline portion. If there's no data left to decode in the inline portion, the receiver MUST conclude that roundup is present, and it moves the XDR decode position forward to the one the next chunk indicates, if there is a next chunk. That's how roundup gets passed without ever transferring any extra XDR bytes 🎀
Some protocol operations over RPC/RDMA, like NFS writes of data at the end of a file or in direct I/O situations, often end up with roundups inside RDMA Read Chunks 🌸 Since those roundup bytes aren't really in the data buffers being written, their memory would have to come from noncontiguous buffers, either as an extra memory registration segment or as an extra Chunk. That overhead can be significant for the sender doing the marshaling, and even more for the receiver that has to transfer them 😭 So Senders SHOULD therefore avoid encoding individual RDMA Read Chunks for roundup whenever possible. It's acceptable, but not necessary, to include roundup data in an existing RDMA Read Chunk, but only if it's already in the XDR stream carrying upper-layer data 💅
Also, leaving roundup out of the XDR stream doesn't expose any extra data on the sender, since any extra sender buffers never get shown to the peer. The data is literally not there to be transferred ✨
For RDMA Write Chunks, there's a simpler method 🧚 Again, roundup bytes aren't transferred. Instead, the chunk length sent back to the receiver in the reply just gets increased to include any roundup. Because RDMA Write Chunks are filled sequentially with no gaps, this can only happen on the final chunk that receives data. So roundup data has no chance to put misalignment or positional gaps into the XDR stream 🫶
3.8. RPC Call and Reply 💜
okay so the RDMA transport for RPC gives you three ways to move data between the RPC client and server 🦋
- Inline: Data is moved between RPC client and server within an RDMA Send.
- RDMA Read: Data is moved between RPC client and server via an RDMA Read operation via steering tag, address and offset obtained from a read chunk list.
- RDMA Write: Result data is moved from RPC server to client via an RDMA Write operation via steering tag, address and offset obtained from a write chunk list or reply chunk in the client's RPC call message.
These methods can be mixed inside a single RPC 🌈 For example, an RPC call might carry some inline data along with some big chunks that the server pulls over with RDMA Read. The reply to that call might have some result chunks that the server RDMA Writes back to the client. Here's how these interactions look 👀
An RPC with write chunks in the call message:
RPC Client RPC Server
| RPC Call + Write Chunk list |
Send | ------------------------------> |
| |
| Chunk 1 |
| <------------------------------ | Write
| : |
| Chunk n |
| <------------------------------ | Write
| |
| RPC Reply |
| <------------------------------ | SendWith write chunks in play, RDMA ordering guarantees that all the data in the RDMA Write operations has been placed in memory before the client processes the RPC reply 🔥
An RPC with read chunks in the call message:
RPC Client RPC Server
| RPC Call + Read Chunk list |
Send | ------------------------------> |
| |
| Chunk 1 |
| +------------------------------ | Read
| v-----------------------------> |
| : |
| Chunk n |
| +------------------------------ | Read
| v-----------------------------> |
| |
| RPC Reply |
| <------------------------------ | SendAn RPC with read chunks in the reply message:
RPC Client RPC Server
| RPC Call |
Send | ------------------------------> |
| |
| RPC Reply + Read Chunk list |
| <------------------------------ | Send
| |
| Chunk 1 |
Read | ------------------------------+ |
| <-----------------------------v |
| : |
| Chunk n |
Read | ------------------------------+ |
| <-----------------------------v |
| |
| Done |
Send | ------------------------------> |The final Done message lets the RPC client tell the server it has received the chunks, so the server can de-register and free the memory holding them 🧹 A Done completion isn't needed for an RPC call, since the RPC reply Send is already a receive completion notification. If the client doesn't return the Done message within some timeout period, the server MAY conclude that a protocol violation has occurred and close the RPC connection, or it MAY proceed with a de-register and free its chunk buffers. That can lead to a fatal RDMA error if the client later tries an RDMA Read operation, which amounts to the same thing 💀
Read chunks in RPC reply messages are way less efficient than write chunks in the originating RPC call 😩 That's because of the extra message exchanges, the RPC server needing to advertise buffers to the peer, the server having to keep a timer to recover from misbehaving clients, and the need for extra memory registration. Their use is NOT RECOMMENDED by upper layers where efficiency is a primary concern [RFC5667]. However, they MAY be employed by upper-layer protocol bindings that are mainly concerned with transparency, since they can often be implemented completely within the RPC lower layers 🌸
It's important to know that the Done message uses up a credit at the RPC server 🎟️ The RPC server SHOULD provide enough credits to the client so the Done message can be sent without deadlock (driving the outstanding credit count to zero). The RPC client MUST account for its required Done messages to the server in its available credit accounting, and the server SHOULD replenish any credit used by such exchanges at its earliest opportunity 💖
Last thing: it's possible to imagine RPC exchanges that combine any or all of write chunks in the RPC call, read chunks in the RPC call, and read chunks in the RPC reply 🦄 Protocol-wise that's straightforward, but in practice these would be pretty rare, mostly limited to upper-layer protocol exchanges that move bulk data in both the call and the matching reply 👑
3.9. Padding 🎀
Lining up certain opaque data is a sneaky trick that unlocks some scatter/gather optimizations ✨. The best part is that RDMA transfers keep alignment intact, even when Sends land in pre-posted receive buffers 🫶.
Lots of servers can really glow up with this. Padding lets you chain RDMA receive buffers so any data moved by RDMA on behalf of RPC requests lands in properly aligned buffers on the system receiving it. That means servers don't have to do RDMA Read for everything except the biggest client writes 💅 no more chaos for every little write.
Here's what padding looks like. The "XXX" is the stuff that came before on the XDR stream, then comes an opaque field with four length bytes ("LLLL") and then the data bytes ("DDD"). The receiver of the RDMA Send has posted two chained receive buffers. Without padding, the opaque data gets split across both buffers 💀. With padding bytes ("ppp") slipped in before the first data byte, the data lines up perfectly in the second buffer 😭✨.
Buffer 1 Buffer 2
Unpadded -------------- --------------
XXXXXXXLLLLDDDDDDDDDDDDDD ---> XXXXXXXLLLLDDD DDDDDDDDDDD
Padded
XXXXXXXLLLLpppDDDDDDDDDDDDDD ---> XXXXXXXLLLLppp DDDDDDDDDDDDDDPadding is handled completely inside the RDMA transport encoding, and it gets flagged with a specific message type 🦋. When padding is applied, two values go to the peer: rdma_align, which is the padding value used, and rdma_thresh, which is the opaque data size that padding kicks in at. For instance, if the server is using chained 4 KB receive buffers, then up to (4 KB - 1) padding bytes could be used to get the data aligned 🌸. The XDR routine at the peer MUST consult these values when decoding opaque values. Where the decoded length exceeds the rdma_thresh, the XDR decode MUST skip over the appropriate padding, as indicated by rdma_align and the current XDR stream position 🧚.
4. RPC RDMA Message Layout 🦋
RPC call and reply messages travel across an RDMA transport with a prepended RPC-over-RDMA header 💖. That header carries data for RDMA flow control credits, padding parameters, and lists of addresses that give you direct data placement via RDMA Read and Write operations 🔥. The layout of the RPC message itself is unchanged from what [RFC5531] describes, except that large data chunks might be left out because they'll be moved by RDMA Read or Write operations. If the RPC message (along with the RPC-over-RDMA header) is too long for the posted receive buffer, even after the large chunks are removed, then the entire RPC message MAY be moved separately as a chunk, leaving just the RPC-over-RDMA header in the RDMA Send 🫠.
4.1. RPC-over-RDMA Header ✨
The RPC-over-RDMA header starts with four 32-bit fields that are always there and that control the RDMA interaction, including RDMA-specific flow control 👑. After those come a bunch of items like chunk lists and padding, which MAY or MUST NOT be present depending on the type of transmission. The four always-present fields are:
- Transaction ID (XID). This is the XID generated for the RPC call and reply. Having it at the very front makes it easy to figure out the message context 🍓. This XID MUST be the same as the XID in the RPC header. The receiver MAY do its processing based solely on the XID in the RPC-over-RDMA header, and thereby ignore the XID in the RPC header, if it so chooses 💅.
- Version number. This version of the RPC RDMA message protocol is 1. The version number MUST be increased by 1 whenever the format of the RPC RDMA messages is changed 🌈.
- Flow control credit value. In an RPC call message, the requested value is provided. In an RPC reply message, the granted value is returned. RPC calls SHOULD NOT be sent in excess of the currently granted limit 🙈 don't be greedy bestie.
- Message type.
RDMA_MSG= 0 indicates that chunk lists and RPC message follow.RDMA_NOMSG= 1 indicates that after the chunk lists there is no RPC message. In this case, the chunk lists give the info needed to move the message proper using RDMA Read or Write, so it isn't appended to the RPC-over-RDMA header 🧚.RDMA_MSGP= 2 indicates that a chunk list and RPC message with some padding follow.RDMA_DONE= 3 indicates that the message signals the completion of a chunk transfer via RDMA Read 💜.RDMA_ERROR= 4 is used to signal any detected error(s) in the RPC RDMA chunk encoding 💀.
Because the version number is encoded as part of this header, and the RDMA_ERROR message type is used to indicate errors, these first four fields and the start of the following message body MUST always remain aligned at these fixed offsets for all versions of the RPC-over-RDMA header 🔒 this part is non-negotiable, iykyk.
For a message of type RDMA_MSG or RDMA_NOMSG, the Read and Write chunk lists follow. If the Read chunk list is null (a 32-bit word of zeros), then there are no chunks to transfer separately and the RPC message follows in its entirety 🎀. If it's non-null, then it's the beginning of an XDR encoded sequence of Read chunk list entries. If the Write chunk list is non-null, then an XDR encoded sequence of Write chunk entries follows 🫶.
If the message type is RDMA_MSGP, then two extra fields that specify the padding alignment and threshold get inserted before the Read and Write chunk lists ✨.
A header of message type RDMA_MSG or RDMA_MSGP MUST be followed by the RPC call or RPC reply message body, beginning with the XID. The XID in the RDMA_MSG or RDMA_MSGP header MUST match this 💖.
+--------+---------+---------+-----------+-------------+----------
| | | | Message | NULLs | RPC Call
| XID | Version | Credits | Type | or | or
| | | | | Chunk Lists | Reply Msg
+--------+---------+---------+-----------+-------------+----------Note that for RDMA_DONE and RDMA_ERROR, no chunk list or RPC message follows. As an implementation hint: a gather operation on the Send of the RDMA RPC message can be used to marshal the initial header, the chunk list, and the RPC message itself 🍓.
4.2. RPC-over-RDMA Header Errors 🚨
When a peer gets an RPC RDMA message, it MUST perform the following basic validity checks on the header and chunk contents. If it detects any such errors in the request, an RDMA_ERROR reply MUST be generated 🙅♀️.
There are two types of errors defined: version mismatch and invalid chunk format. When the peer spots an RPC-over-RDMA header version it doesn't support (right now this document only defines version 1), it replies with an error code of ERR_VERS, and it provides the low and high inclusive version numbers it actually does support 🌸. The version number in this reply MUST be any value otherwise valid at the receiver. When other decoding errors show up in the header or chunks, either an RPC decode error MAY be returned, or the RPC/RDMA error code ERR_CHUNK MUST be returned 😭.
4.3. XDR Language Description 📜
Here's the message layout written in XDR language 🧁.
struct xdr_rdma_segment {
uint32 handle; /* Registered memory handle */
uint32 length; /* Length of the chunk in bytes */
uint64 offset; /* Chunk virtual address or offset */
};
struct xdr_read_chunk {
uint32 position; /* Position in XDR stream */
struct xdr_rdma_segment target;
};
struct xdr_read_list {
struct xdr_read_chunk entry;
struct xdr_read_list *next;
};
struct xdr_write_chunk {
struct xdr_rdma_segment target<>;
};
struct xdr_write_list {
struct xdr_write_chunk entry;
struct xdr_write_list *next;
};
struct rdma_msg {
uint32 rdma_xid; /* Mirrors the RPC header xid */
uint32 rdma_vers; /* Version of this protocol */
uint32 rdma_credit; /* Buffers requested/granted */
rdma_body rdma_body;
};
enum rdma_proc {
RDMA_MSG=0, /* An RPC call or reply msg */
RDMA_NOMSG=1, /* An RPC call or reply msg - separate body */
RDMA_MSGP=2, /* An RPC call or reply msg with padding */
RDMA_DONE=3, /* Client signals reply completion */
RDMA_ERROR=4 /* An RPC RDMA encoding error */
};
union rdma_body switch (rdma_proc proc) {
case RDMA_MSG:
rpc_rdma_header rdma_msg;
case RDMA_NOMSG:
rpc_rdma_header_nomsg rdma_nomsg;
case RDMA_MSGP:
rpc_rdma_header_padded rdma_msgp;
case RDMA_DONE:
void;
case RDMA_ERROR:
rpc_rdma_error rdma_error;
};
struct rpc_rdma_header {
struct xdr_read_list *rdma_reads;
struct xdr_write_list *rdma_writes;
struct xdr_write_chunk *rdma_reply;
/* rpc body follows */
};
struct rpc_rdma_header_nomsg {
struct xdr_read_list *rdma_reads;
struct xdr_write_list *rdma_writes;
struct xdr_write_chunk *rdma_reply;
};
struct rpc_rdma_header_padded {
uint32 rdma_align; /* Padding alignment */
uint32 rdma_thresh; /* Padding threshold */
struct xdr_read_list *rdma_reads;
struct xdr_write_list *rdma_writes;
struct xdr_write_chunk *rdma_reply;
/* rpc body follows */
};
enum rpc_rdma_errcode {
ERR_VERS = 1,
ERR_CHUNK = 2
};
union rpc_rdma_error switch (rpc_rdma_errcode err) {
case ERR_VERS:
uint32 rdma_vers_low;
uint32 rdma_vers_high;
case ERR_CHUNK:
void;
default:
uint32 rdma_extra[8];
};5. Long Messages 🌊
The receiver of RDMA Send messages is required by RDMA to have previously posted one or more adequately sized buffers. The RPC client can tell the server the maximum size of its RDMA Send messages through the Connection Configuration Protocol, which is described later in this document 💅.
Since RPC messages are usually small, you can save memory by posting small buffers ✨. Even big messages like NFS READ or WRITE end up being quite small once the chunks are removed from the message. But there can be large messages that would demand a really big buffer to be posted, where the buffer's contents may not be a chunkable XDR element 🙈. A good example is an NFS READDIR reply, which can hold a large number of small filename strings. Also, the NFS version 4 protocol [RFC3530] has COMPOUND request and reply messages of unbounded length 👑 unbounded. no thoughts. just vibes..
Ideally, each upper layer negotiates these limits. However, it's frequently necessary to provide a transparent solution 🫶.
5.1. Message as an RDMA Read Chunk
okay so the simplest move 💅 is for the client to spot any RPC message that's bigger than the server's posted buffer and just move it separately as a chunk. basically you reference it as the first entry in the read chunk list with an XDR position of zero 🎀
Normal Message
+--------+---------+---------+------------+-------------+----------
| | | | | | RPC Call
| XID | Version | Credits | RDMA_MSG | Chunk Lists | or
| | | | | | Reply Msg
+--------+---------+---------+------------+-------------+----------Long Message
+--------+---------+---------+------------+-------------+
| | | | | |
| XID | Version | Credits | RDMA_NOMSG | Chunk Lists |
| | | | | |
+--------+---------+---------+------------+-------------+
|
| +----------
| | Long RPC Call
+->| or
| Reply Message
+----------ok so if the receiver gets an RPC-over-RDMA header with a message type of RDMA_NOMSG and finds an initial read chunk list entry with a zero XDR position, it allocates a registered buffer and issues an RDMA Read of the long RPC message into it 🔥 then it XDR decodes the RPC message like it came in inline with the Send data. if decoding needs more chunks, it can just issue more RDMA Reads to bring them over ✨
ngl the long message path costs one extra network turnaround 🙈 but in practice these will be rare if the posted receive buffers are sized correctly, and they'll be literally non-existent for RDMA-aware upper layers 💖
A long call RPC with request supplied via RDMA Read
RPC Client RPC Server
| RDMA-over-RPC Header |
Send | ------------------------------> |
| |
| Long RPC Call Msg |
| +------------------------------ | Read
| v-----------------------------> |
| |
| RDMA-over-RPC Reply |
| <------------------------------ | SendAn RPC with long reply returned via RDMA Read
RPC Client RPC Server
| RPC Call |
Send | ------------------------------> |
| |
| RDMA-over-RPC Header |
| <------------------------------ | Send
| |
| Long RPC Reply Msg |
Read | ------------------------------+ |
| <-----------------------------v |
| |
| Done |
Send | ------------------------------> |it's totally possible for one RPC procedure to use both a long call for its arguments AND a long reply for its results 🧚 but that's atypical, since not many upper layers define exchanges like that 🙃
5.2. RDMA Write of Long Replies (Reply Chunks)
okay this one is the superior method 👑 instead of the server reading stuff out, the RPC client posts a large buffer and the server just writes a big RPC reply straight into it. the perks: an RDMA Write can be slightly faster in network latency than an RDMA Read 🦋 and the server doesn't have to wait for a completion the way it does for an RDMA Read. plus, for a reply, it kills the need for an RDMA_DONE message if the large reply is returned as a Read chunk 😭
this protocol supports direct return of a large reply through an OPTIONAL rdma_reply write chunk that sits after the read chunk list and the write chunk list 🎀 the client allocates a buffer sized to receive a large reply and drops its steering tag, address and length into the rdma_reply write chunk. if the reply is too long to go back inline with an RDMA Send (bigger than the client's posted receive buffer), even with read chunks removed, then the RPC server performs an RDMA Write of the RPC reply message into the buffer the rdma_reply chunk points to 🔥 and if the client doesn't provide an rdma_reply chunk, or it's too small, then the message MAY be returned as a Read chunk, as long as the upper-layer specification permits it 💅
An RPC with long reply returned via RDMA Write
RPC Client RPC Server
| RPC Call with rdma_reply |
Send | ------------------------------> |
| |
| Long RPC Reply Msg |
| <------------------------------ | Write
| |
| RDMA-over-RPC Header |
| <------------------------------ | Sendok but real talk, using RDMA Write for long replies means the client apps have to see a long reply coming and know roughly how big it'll be, so they can allocate a big enough buffer 🙈 this is totally true for NFS READDIR replies, where the client already gives an upper bound on the size of the encoded directory fragment the server will return 🌸
these "reply chunks" are highly efficient and super convenient for both the RPC client and server 🫶 which is why they're encouraged for eligible RPC operations like NFS READDIR. otherwise you'd be stuck doing extensive chunk management inside the results, or using RDMA Read plus a Done message [RFC5667] 💀
6. Connection Configuration Protocol ✨
okay so RDMA Send operations need the receiver to post one or more buffers at the RDMA connection endpoint, and each one has to be big enough for the largest Send message 🎀 buffers get used up as Send messages come in. if a buffer is too small, or there aren't any posted, the RDMA transport MAY return an error and break the RDMA connection 😭 so the receiver MUST post sufficient, adequately sized buffers to avoid buffer overrun or capacity errors 🔥
the protocol we just went through only has a way to manage how many of those receive buffers there are. there's no explicit feature for the RPC client and server to provision or control buffer sizing, or any other session parameters 🙈
honestly, this kind of connection management hasn't been needed for RPC before. RPC over UDP or TCP doesn't have a protocol to negotiate the link 🦋 the server can get a rough idea of the maximum message size from its server protocol code, but a protocol to negotiate transport features on a more dynamic basis is desirable 💜
so the Connection Configuration Protocol lets the client pass its connection requirements to the server, and lets the server tell the client its connection limits ✨
using this protocol from an upper layer is OPTIONAL 🌈
6.1. Initial Connection State
this protocol MAY be used for connection setup before another RPC protocol that uses the RDMA transport gets going 💖 it works in-band, meaning it uses the connection itself to negotiate the connection parameters. to give the negotiation a basis, the connection is assumed to provide a basic level of interoperability: the ability to exchange at least one RPC message at a time that's at least 1 KB in size 🧚 the server MAY exceed this basic level, but the client MUST NOT assume more than one, and MUST receive a valid reply from the server carrying the actual number of available receive messages, before it sends its next request 👑
6.2. Protocol Description
okay so version 1 of the Connection Configuration Protocol is just one procedure 💅 it lets the client tell the server its connection requirements, and lets the server send connection info back to the client 🎀
the maxcall_sendsize argument is the maximum size of an RPC call message that the client MAY send inline in an RDMA Send message to the server 🔥 the server MAY return a maxcall_sendsize value that's smaller or larger than what the client asked for. the client MUST NOT send an inline call message bigger than what the server will accept 🙈 and just so we're clear, maxcall_sendsize only limits inline RPC calls. it does NOT limit long RPC messages that get moved as an initial chunk in the Read chunk list 🫶
the maxreply_sendsize is the maximum size of an inline RPC message that the client will accept from the server 🌸
the maxrdmaread is the maximum number of RDMA Reads that can be active at the peer 🦋 this number lines up with the RDMA incoming RDMA Read count ("IRD") configured into each originating endpoint by the client or server. if more RDMA Read operations than this get issued by the connected peer at the same time, you can get connection loss or suboptimal flow control 😭 so the value SHOULD be observed at all times 💜 the peers' values don't have to match. and if it's zero, the peer MUST NOT issue requests that require RDMA Read to satisfy, because no transfer would be possible 💀
the align value is what the server recommends for opaque data values like strings and counted byte arrays ✨ the client MAY use this value to compute how many prepended pad bytes to add when XDR encoding opaque values in the RPC call message 🌈
typedef unsigned int uint32;
struct config_rdma_req {
uint32 maxcall_sendsize;
/* max size of inline RPC call */
uint32 maxreply_sendsize;
/* max size of inline RPC reply */
uint32 maxrdmaread;
/* max active RDMA Reads at client */
};
struct config_rdma_reply {
uint32 maxcall_sendsize;
/* max call size accepted by server */
uint32 align;
/* server's receive buffer alignment */
uint32 maxrdmaread;
/* max active RDMA Reads at server */
};
program CONFIG_RDMA_PROG {
version VERS1 {
/*
* Config call/reply
*/
config_rdma_reply CONF_RDMA(config_rdma_req) = 1;
} = 1;
} = 100417;7. Memory Registration Overhead 💅
okay so RDMA needs ALL the data to move between registered memory regions at both the source and destination 🧠 every protocol header AND every separately sent data chunk has to live in registered memory. registering and de-registering memory can eat a huge chunk of the cost of an RDMA transaction 😭 so minimizing registration activity is a whole priority. the easy move is to allocate chunk list data and RPC headers in a reusable way out of pre-registered pools, all inside RPC-controlled memory 🫶
the data chunks sent via RDMA MAY occupy memory that sticks around past the end of the RPC transaction 👀 so the default behavior of an RPC-over-RDMA transport is to register and de-register those chunks on every single transaction. but this isn't a limit of the protocol, it's just how the existing local RPC API works 🙈 the API can be extended with stuff like rpc_control(3) to change that default, so the application can take over memory registration through an RPC-provided registered memory allocator. main character energy 👑
8. Errors and Error Recovery ✨
RPC RDMA protocol errors are described in Section 4. RPC errors and RPC error recovery aren't affected by the protocol at all, they just work like they would for any other RPC error condition 💖 and RDMA transport error reporting and recovery are outside the scope of this protocol 🙅♀️
it's assumed the link itself will provide some level of error detection and retransmission 🔥 the iWARP Marker PDU Aligned (MPA) layer (when it's used over TCP), Stream Control Transmission Protocol (SCTP), and the InfiniBand link layer all give CRC (Cyclic Redundancy Check) protection to the RDMA payload, and CRC-class protection is a general attribute of these transports 🌸 on top of that, the RPC layer can accept errors from the link level and recover through retransmission. RPC recovery can even handle a complete loss and re-establishment of the link 🦋
see Section 11 for more on using RPC-level integrity schemes to detect errors, plus the efficiency issues that come with them 🧚
9. Node Addressing 🎀
when setting up a brand new RDMA connection, the very first thing an RPC client does is get a transport address for the server 💜 how it gets that address, and how it opens the RDMA connection, depends on the type of RDMA transport. that's the job of each RPC protocol binding and its local implementation, not this doc's problem 😌
10. RPC Binding 🌈
okay this one is lowkey important. RPC services normally register with a portmap or rpcbind [RFC1833] service, which links an RPC program number to a service address 🗺️ (for UDP or TCP, the NFS service address is normally port 2049.) this policy doesn't change with RDMA interconnects, though it might mean allocating port numbers for each upper-layer binding that uses the RPC framing defined here 🍓
when it's mapped on top of iWARP [RFC5040, RFC5041], the transport uses IP port addressing because it's layered on TCP and/or SCTP. so port mapping is super easy 🥰 it just means sending the port during the connection process. IANA has assigned port 20049 to the NFS/RDMA protocol service, for both iWARP/TCP and iWARP/SCTP ✨
when it's mapped on top of InfiniBand [IB], which uses a Group Identifier (GID)-based service endpoint naming scheme, a translation MUST be employed 🚨 one translation that's defined is in the InfiniBand Port Addressing Annex [IBPORT], which is right for translating IP port addressing to the InfiniBand network. so in this case, IP port addressing may be readily employed by the upper layer 💅
when a mapping standard or convention exists for IP ports on an RDMA interconnect, each upper layer has a few possibilities to consider 🧠
- one option is to have an upper-layer server register its mapped IP port with the rpcbind service, under the netid (or netids) defined here. an RPC/RDMA-aware client can then resolve its desired service to a mappable port and go ahead and connect. this is the most flexible and compatible approach, for upper layers that are defined to use the rpcbind service 🫶
- a second option is to have the server's portmapper register itself on the RDMA interconnect at a "well known" service address. (on UDP or TCP, that's port
111.) a client could connect to this service address and use the portmap protocol to get a service address back in response to a program number, like an iWARP port number or an InfiniBand GID 🦋 - or, the client could just connect straight to the mapped well-known port for the service itself, if it's been defined that way. by convention, when NFS/RDMA runs on top of an InfiniBand fabric, it uses the same
20049assignment as iWARP 🌸
different RPC protocols have historically taken different approaches to port assignment, so the specific method is left to each RPC/RDMA-enabled upper-layer binding and isn't addressed here 🙈
in Section 12, "IANA Considerations", this spec defines two new "netid" values, to be used for registering upper layers on top of iWARP [RFC5040, RFC5041] and (when a suitable port translation service is available) InfiniBand [IB] 🎀 additional RDMA-capable networks MAY define their own netids, or, if they provide a port translation, they MAY share the one defined here 👑
11. Security Considerations 🔐
RPC provides its own security through the RPCSEC_GSS framework [RFC2203]. RPCSEC_GSS can provide message authentication, integrity checking, and privacy 💖 and this security mechanism is unaffected by the RDMA transport. the data integrity and privacy features alter the body of the message, presenting it as a single chunk. for large messages, that chunk might be big enough to qualify for an RDMA Read transfer 👀 however, there's a ton of data movement tied up in computing and verifying integrity, or in encryption and decryption, so certain performance advantages may be lost 😭
for efficiency, a better fit for RDMA links may be link-level protection, like certain configurations of IPsec, which may be co-located in the RDMA hardware 🔥 the use of link-level protection MAY be negotiated through the new RPCSEC_GSS mechanism defined in [RFC5403], together with the Channel Binding mechanism [RFC5056] and IPsec Channel Connection Latching [RFC5660]. using these mechanisms is REQUIRED where integrity and/or privacy is desired, and where efficiency is required 🫶
there's also the protection of the integrity and privacy of local memory by the RDMA transport itself 🧚 the use of RDMA by RPC MUST NOT introduce any vulnerabilities to system memory contents, or to memory owned by user processes. these protections come from the RDMA layer specs, and specifically their security models. it is REQUIRED that any RDMA provider used for RPC transport conforms to the requirements of [RFC5042] to satisfy these protections 🛡️
once the RDMA provider has delivered things securely, any RDMA-exposed addresses will only contain RPC payloads in the chunk lists, transferred under the protection of RPCSEC_GSS integrity and privacy. this way the data stays protected end-to-end, as the RPC layer security model requires 💜
⚠️ the denial-of-service situation: where upper-layer protocols choose to supply results to the requester via read chunks, a server resource deficit can happen if the client doesn't promptly acknowledge their status via the RDMA_DONE message. this can potentially lead to a denial-of-service situation, with a single client unfairly (and unnecessarily) hogging server RDMA resources. servers for such upper-layer protocols MUST protect against this situation, whether it comes from one client or many. for example, a time-based window of buffer availability may be offered. if the client doesn't get the data within the window, it just retries using ordinary RPC retry semantics. or, a more severe method is for the server to simply close the client's RDMA connection, freeing the RDMA resources so the server can reclaim them 😱
a fairer and more useful method is provided by the protocol itself 🥹 the server MAY use the rdma_credit value to limit the number of outstanding requests for each client. by including the number of outstanding RDMA_DONE completions in the computation of available client credits, the server can limit its exposure to each client, and so provide uninterrupted service as its resources permit 🌈
but the server must make sure it doesn't decrease the credit count to zero with this method, since the RDMA_DONE message is not acknowledged 😬 if the credit count dropped to zero solely because of outstanding RDMA_DONE messages, the client would deadlock, since it would never get a new credit to keep going. so, if the server adjusts credits to zero for outstanding RDMA_DONE, it MUST withhold its reply to at least one message in order to provide the next credit 🙏 the time-based window (or any other appropriate method) SHOULD be used by the server to recover resources in case the client never comes back 💀
finally, the Connection Configuration Protocol, when used, MUST be protected by an appropriate RPC security flavor, so it doesn't get attacked while an RPC/RDMA connection is being set up 🔒✨
12. IANA Considerations 📋
three new assignments are specified by this document 🎀
- a new set of RPC "netids" for resolving RPC/RDMA services
- optional service port assignments for upper-layer bindings
- an RPC program number assignment for the configuration protocol
these assignments have been established, as shown below 💖
the new RPC transport has been assigned an RPC "netid". that's an rpcbind [RFC1833] string used to describe the underlying protocol, so RPC can pick the right transport framing and the right format for service addresses and ports 🧠
the following "Netid" registry strings are defined for this purpose:
NC_RDMA "rdma"
NC_RDMA6 "rdma6"these netids MAY be used for any RDMA network that satisfies the requirements of Section 2, and that can identify service endpoints using IP port addressing, possibly through a translation service like the one described above in Section 10, "RPC Binding" 🦋 the rdma netid is used when IPv4 addressing is employed by the underlying transport, and rdma6 is for IPv6 addressing 🌸
the netid assignment policy and registry are defined in [RFC5665] 📜
as a new RPC transport, this protocol has no effect on RPC program numbers or existing registered port numbers 🙌 however, new port numbers MAY be registered for use by RPC/RDMA-enabled services, as appropriate to the new networks those services will run over 🌈
for example, the NFS/RDMA service defined in [RFC5667] has been assigned port 20049 in the IANA registry:
nfsrdma 20049/tcp Network File System (NFS) over RDMA
nfsrdma 20049/udp Network File System (NFS) over RDMA
nfsrdma 20049/sctp Network File System (NFS) over RDMAthe OPTIONAL Connection Configuration Protocol described here requires an RPC program number assignment 🧚 the value 100417 has been assigned:
rdmaconfig 100417 rpc.rdmaconfigthe RPC program number assignment policy and registry are defined in [RFC5531] 💜
13. Acknowledgments ✨
The authors wanted to shout out Rob Thurlow, John Howard, Chet Juszczak, Alex Chiu, Peter Staubach, Dave Noveck, Brian Pawlowski, Steve Kleiman, Mike Eisler, Mark Wittle, Shantanu Mehendale, David Robinson, and Mallikarjun Chadalapaka 🫶💖 for their contributions to this document. Literally a whole squad of legends, no cap 👑.
14. References 📚
14.1. Normative References
- [RFC1833] Srinivasan, R., "Binding Protocols for ONC RPC Version 2", RFC 1833, August 1995.
- [RFC2203] Eisler, M., Chiu, A., and L. Ling, "RPCSEC_GSS Protocol Specification", RFC 2203, September 1997.
- [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
- [RFC4506] Eisler, M., Ed., "XDR: External Data Representation Standard", STD 67, RFC 4506, May 2006.
- [RFC5042] Pinkerton, J. and E. Deleganes, "Direct Data Placement Protocol (DDP) / Remote Direct Memory Access Protocol (RDMAP) Security", RFC 5042, October 2007.
- [RFC5056] Williams, N., "On the Use of Channel Bindings to Secure Channels", RFC 5056, November 2007.
- [RFC5403] Eisler, M., "RPCSEC_GSS Version 2", RFC 5403, February 2009.
- [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol Specification Version 2", RFC 5531, May 2009.
- [RFC5660] Williams, N., "IPsec Channels: Connection Latching", RFC 5660, October 2009.
- [RFC5665] Eisler, M., "IANA Considerations for Remote Procedure Call (RPC) Network Identifiers and Universal Address Formats", RFC 5665, January 2010.
14.2. Informative References
- [RFC1094] Sun Microsystems, "NFS: Network File System Protocol specification", RFC 1094, March 1989.
- [RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS Version 3 Protocol Specification", RFC 1813, June 1995.
- [RFC3530] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame, C., Eisler, M., and D. Noveck, "Network File System (NFS) version 4 Protocol", RFC 3530, April 2003.
- [RFC5040] Recio, R., Metzler, B., Culley, P., Hilland, J., and D. Garcia, "A Remote Direct Memory Access Protocol Specification", RFC 5040, October 2007.
- [RFC5041] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct Data Placement over Reliable Transports", RFC 5041, October 2007.
- [RFC5532] Talpey, T. and C. Juszczak, "Network File System (NFS) Remote Direct Memory Access (RDMA) Problem Statement", RFC 5532, May 2009.
- [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "Network File System Version 4 Minor Version 1 Protocol", RFC 5661, January 2010.
- [RFC5667] Talpey, T. and B. Callaghan, "Network File System (NFS) Direct Data Placement", RFC 5667, January 2010.
- [IB] InfiniBand Trade Association, InfiniBand Architecture Specifications, available from http://www.infinibandta.org.
- [IBPORT] InfiniBand Trade Association, "IP Addressing Annex", available from http://www.infinibandta.org.
Authors' Addresses 💌
Tom Talpey 170 Whitman St. Stow, MA 01775 USA
EMail: tmtalpey@gmail.com
Brent Callaghan Apple Computer, Inc. MS: 302-4K 2 Infinite Loop Cupertino, CA 95014 USA
EMail: brentc@apple.com
📦 where this page comes from
- ietf: rfc5666 2010-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms