RFC 8797

Remote Direct Memory Access - Connection Manager (RDMA-CM) Private Data for RPC-over-RDMA Version 1

📅 published June 2020, from ietf

RFC 8797: Remote Direct Memory Access - Connection Manager (RDMA-CM) Private Data for RPC-over-RDMA Version 1 🦋

Okay bestie, this one's from the IETF (Internet Engineering Task Force) and it's officially RFC 8797 💅. It's a Standards Track doc, dated June 2020, written by C. Lever at Oracle 👑. It updates RFC 8166 and doesn't obsolete anything, and its ISSN is 2070-1721 🌸.

Abstract ✨

So this doc defines the format of the RDMA-CM Private Data that RPC-over-RDMA version 1 peers swap when they're setting up a connection 🔌. The Private Data payload is an optional extension, and it does not change the RPC-over-RDMA version 1 protocol itself. This document updates RFC 8166 💖.

Status of This Memo 📜

This is an Internet Standards Track document 🌈.

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841.

Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc8797 🧚.

Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents 🗂️

   1.  Introduction
   2.  Requirements Language
   3.  Advertised Transport Properties
     3.1.  Inline Threshold Size
     3.2.  Remote Invalidation
   4.  Private Data Message Format
     4.1.  Using the R Field
     4.2.  Send and Receive Size Values
   5.  Interoperability Considerations
     5.1.  Interoperability with RPC-over-RDMA Version 1
           Implementations
     5.2.  Interoperability amongst RDMA Transports
   6.  Updating the Message Format
   7.  Security Considerations
   8.  IANA Considerations
     8.1.  Guidance for Designated Experts
   9.  References
     9.1.  Normative References
     9.2.  Informative References
   Acknowledgments
   Author's Address

1. Introduction 💫

The RPC-over-RDMA version 1 transport protocol [RFC8166] lets upper-layer protocols built on Remote Procedure Calls (RPCs) [RFC5531] move payload data using Remote Direct Memory Access (RDMA) 🚀. The terms "Remote Direct Memory Access" (RDMA) and "Direct Data Placement" (DDP) are introduced in [RFC5040].

Okay, so there are two big shortcomings of RPC-over-RDMA version 1 that are the most immediate 😭:

  1. Setting up an RDMA data transfer (via RDMA Read or Write) can be costly 💸. The small default size of messages sent using RDMA Send means you're forced into RDMA Read or Write operations even for relatively small messages and data payloads.

The original RPC-over-RDMA version 1 spec had an out-of-band protocol for passing inline threshold values between connected peers [RFC5666]. But [RFC8166] got rid of support for that protocol, so it's not available for this purpose 🙃.

  1. Unlike most other contemporary RDMA-enabled storage protocols, RPC-over-RDMA version 1 has no facility for using remote invalidation [RFC5042] 🥲.

Each RPC-over-RDMA version 1 Transport Header follows the External Data Representation (XDR) definition [RFC4506] specified in [RFC8166]. But RPC-over-RDMA version 1 has no way to extend that definition while keeping interoperability with existing implementations intact 😤. So an out-of-band mechanism is needed to help relieve these constraints for existing RPC-over-RDMA version 1 implementations.

This document specifies a simple, non-XDR-based message format that's meant to be passed between RPC-over-RDMA version 1 peers the moment each RDMA transport connection is first established 🤝. The mechanism assumes the underlying RDMA transport has a Private Data field that gets passed between peers at connection time. Examples include the Marker PDU Aligned Framing (MPA) protocol (described in Section 7.1 of [RFC5044] and extended in [RFC6581]) and the InfiniBand Connection Manager [IBA] ✨.

So current RPC-over-RDMA version 1 implementations can still interoperate with implementations that support the message format in this document, implementing the Private Data exchange is OPTIONAL 🫶. When Private Data has been successfully exchanged, peers may choose to perform extended RDMA semantics. Even so, this exchange does not alter the XDR definition specified in [RFC8166] 💜.

The message format is designed to be further extensible within the normal scope of such IETF work (see Section 6 for further details). Section 8 of this document defines an IANA registry for this purpose 📋. Also, interoperation between implementations of RPC-over-RDMA version 1 that present this message format to peers and those that don't recognize it is guaranteed 🌸.

2. Requirements Language 🔑

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 🎀.

3. Advertised Transport Properties 🦋

3.1. Inline Threshold Size 📏

Section 3.3.2 of [RFC8166] defines the term "inline threshold". An inline threshold is the maximum number of bytes that can be transmitted using one RDMA Send and one RDMA Receive. A connection has a pair of inline thresholds: one client-to-server threshold and one server-to-client threshold 🔁.

If an incoming RDMA message is bigger than a receiver's inline threshold, the Receive operation fails, and the RDMA provider typically terminates the connection 💀. To convey an RPC message that's larger than the receiver's inline threshold without risking a receive failure, a sender must use explicit RDMA data transfer operations. Those are more expensive than an RDMA Send 💸. See Sections 3.3 and 3.5 of [RFC8166] for a complete discussion.

The default inline threshold for RPC-over-RDMA version 1 connections is 1024 bytes (as defined in Section 3.3.3 of [RFC8166]). That's honestly enough for nearly all NFS version 3 procedures 🍓.

NFS version 4 COMPOUND operations [RFC7530] are larger on average than NFS version 3 procedures [RFC1813]. That forces clients to use explicit RDMA operations for frequently issued requests like LOOKUP and GETATTR 😩. Using RPCSEC_GSS security also makes the average RPC message bigger, because the RPCSEC_GSS credential material included in RPC headers is larger [RFC7861].

If a sender and receiver could somehow agree on larger inline thresholds, frequently used RPC transactions could skip the cost of explicit RDMA operations 🧚‍♀️✨.

3.2. Remote Invalidation 🫧

After an RDMA data transfer operation completes, an RDMA consumer can ask its peer's RDMA Network Interface Card (RNIC) to invalidate the Steering Tag (STag) associated with that data transfer [RFC5042] 🪄.

An RDMA consumer requests remote invalidation by posting an RDMA Send with Invalidate operation in place of an RDMA Send operation. Each RDMA Send with Invalidate carries one STag to invalidate. The receiver of an RDMA Send with Invalidate performs the requested invalidation, then reports that invalidation as part of the completion of a waiting Receive operation 📬.

If both peers support remote invalidation, an RPC-over-RDMA responder might use remote invalidation when replying to an RPC request that provided chunks. Because one of the chunks has already been invalidated, finalizing the results of the RPC gets simpler and faster 🏎️💨.

However, there are some important caveats that rule out using remote invalidation everywhere 🚨:

  • Remote invalidation is not supported by all RNICs.
  • Not all RPC-over-RDMA responder implementations can generate RDMA Send with Invalidate operations.
  • Not all RPC-over-RDMA requester implementations can recognize when remote invalidation has occurred.
  • On one connection, across different RPC-over-RDMA transactions or within a single RPC-over-RDMA transaction, an RPC-over-RDMA requester can expose a mix of STags that may be invalidated remotely and some that must not be. No indication is provided at the RDMA layer as to which is which.
A responder therefore must not employ remote invalidation unless it is aware of support for it in its own RDMA stack, and on the requester. And, without altering the XDR structure of RPC-over-RDMA version 1 messages, it is not possible to support remote invalidation with requesters that include an STag that must not be invalidated remotely in an RPC with STags that may be invalidated. Likewise, it is not possible to support remote invalidation with requesters that mix RPCs with STags that may be invalidated with RPCs with STags that must not be invalidated on the same connection 🙅‍♀️.

There are some NFS/RDMA client implementations whose STags are always safe to invalidate remotely. For those clients, indicating to the responder that remote invalidation is always safe can enable it without the need for any additional protocol elements to be defined 💖.

  1. Private Data Message Format ✨

Okay so, bestie, picture this 🎀 With an InfiniBand lower layer, RDMA connection setup uses a Connection Manager (CM) when it's time to establish a Reliable Connection [IBA]. 🦋 When an RPC-over-RDMA version 1 transport connection gets set up, the client (the one actively reaching out to connect) and the server (the one passively accepting connections) both fill in the CM Private Data field that gets swapped as part of CM connection establishment. 💖

The transport properties exchanged through this mechanism are locked in for the life of the connection. 🔒 Every new connection is a fresh chance for a new exchange 🌸 So an implementation of the extension described in this document MUST be prepared for the settings to change upon a reconnection. ✨ Be ready for a glow-up every time!

For RPC-over-RDMA version 1, the CM Private Data field is formatted the way described below, and RPC clients and servers use the same format. 💅 If the Private Data field is too small to hold this message format, or if the underlying RDMA transport isn't managed by a CM, then the CM Private Data field cannot be used on behalf of RPC-over-RDMA version 1. 💀

The first eight octets of the CM Private Data field are to be formatted as follows:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                       Format Identifier                       |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |    Version    |  Reserved   |R|   Send Size   | Receive Size  |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Format Identifier: This field is a fixed 32-bit value that marks the content of the Private Data field as an RPC-over-RDMA version 1 CM Private Data message. 🧚 In RPC-over-RDMA version 1 Private Data, the value of this field is always 0xf6ab0e18, in network byte order. How this field gets used is expanded on in Section 5.2. 🫶

Version: This 8-bit field holds a message format version number. The value "1" here means that exactly eight octets are present, that they show up in the order described in this section, and that each one has the meaning defined in this section. 🌈 More thoughts on using this field are in Section 6.

Reserved: This 7-bit field is unused. Senders MUST set these bits to zero, and receivers MUST ignore their value. 🙈

R: This 1-bit field indicates that the sender supports remote invalidation. ⭐ It gets set and interpreted as described in Section 4.1.

Send Size: This 8-bit field contains an encoded value for the maximum number of bytes this peer is prepared to transmit in a single RDMA Send on this connection. The value is encoded as described in Section 4.2. 🍓

Receive Size: This 8-bit field contains an encoded value for the maximum number of bytes this peer is prepared to receive with a single RDMA Receive on this connection. The value is encoded as described in Section 4.2. 👑

4.1. Using the R Field 🦋

The R field indicates limited support for remote invalidation, as described in Section 3.2. ✨ When both connection peers have set this bit flag in their CM Private Data, the responder MAY use RDMA Send with Invalidate operations when transmitting RPC Replies. Each RDMA Send with Invalidate MUST invalidate an STag associated only with the Transaction ID (XID) in the rdma_xid field of the RPC-over-RDMA Transport Header it carries. 💜

When either peer on a connection clears this flag, the responder MUST use only RDMA Send when transmitting RPC Replies. 🌸

4.2. Send and Receive Size Values 🎀

Inline threshold sizes from 1024 to 262144 octets can be represented in the Send Size and Receive Size fields. 🔥 The inline threshold values give a pair of 1024-octet-aligned maximum message lengths that guarantee Send and Receive operations do not fail due to length errors. 💅

The minimum inline threshold for RPC-over-RDMA version 1 is 1024 octets (see Section 3.3.3 of [RFC8166]). The values in the Send Size and Receive Size fields represent the unsigned number of additional kilo-octets of length beyond the first 1024 octets. 🧮 So a sender computes the encoded value by dividing its actual buffer size, in octets, by 1024 and subtracting one from the result. A receiver decodes an incoming Size value by doing the inverse: it adds one to the encoded value and then multiplies that result by 1024. 🦋

The client uses the smaller of its own send size and the server's reported receive size as the client-to-server inline threshold. ✨ The server uses the smaller of its own send size and the client's reported receive size as the server-to-client inline threshold. 💖

5. Interoperability Considerations 🌈

The extension described in this document is designed to let RPC-over-RDMA version implementations that use CM Private Data interoperate fully with RPC-over-RDMA version 1 implementations that don't exchange this information. 🫶 Implementations that use this extension must also interoperate fully with RDMA implementations that use CM Private Data for other purposes. 🧚 Making these goals happen requires that implementations of this extension follow the practices described in the rest of this section. 💅

5.1. Interoperability with RPC-over-RDMA Version 1 Implementations 🎀

When a peer doesn't receive a CM Private Data message that conforms to Section 4, it needs to act as if the remote peer supports only the default RPC-over-RDMA version 1 settings, as defined in [RFC8166]. 🙈 In other words, the peer MUST behave as if a Private Data message was received in which (1) bit 15 of the Flags field is zero and (2) both Size fields contain the value zero. 💀

5.2. Interoperability amongst RDMA Transports 🦋

The Format Identifier field defined in Section 4 exists so implementations can tell the Private Data defined in this document apart from Private Data inserted at other layers, such as the additional Private Data defined by the MPAv2 protocol described in [RFC6581], and others. 👑

As part of connection establishment, the buffer holding the received Private Data is searched for the Format Identifier word. The offset of the Format Identifier is not restricted to any alignment. 🍓 If the RPC-over-RDMA version 1 CM Private Data Format Identifier is not present, an RPC-over-RDMA version 1 receiver MUST behave as if no RPC-over-RDMA version 1 CM Private Data has been provided. 🔥

Once the RPC-over-RDMA version 1 CM Private Data Format Identifier is found, the receiver parses the subsequent octets as RPC-over-RDMA version 1 CM Private Data. ✨ As extra assurance that the content is valid, the receiver should check that the format version number field contains a valid and recognized version number, and that the size of the content does not overrun the length of the buffer. 🌸

6. Updating the Message Format 🌸

Although the message format described in this document lets the client and server exchange particular information about the local RPC-over-RDMA implementation, there may be a future need to exchange additional properties. ✨ That would make it necessary to extend or otherwise modify the format described in this document. 🔮

Any modification has to deal with interoperating properly with implementations of RPC-over-RDMA version 1 that are unaware of the new format. 😭 These include implementations that don't recognize the exchange of CM Private Data, as well as those that recognize only the format described in this document. 💀

Given the message format described in this document, these interoperability constraints could be met by the following sorts of new message formats:

  • A format that uses a different value for the first four bytes of the format, as provided for in the registry described in Section 8.
  • A format that uses the same value for the Format Identifier field and a value other than one (1) in the Version field.

Although it is possible to reorganize the last three of the eight bytes in the existing format, extended formats are unlikely to do so. 🙄 New formats would take the form of extensions of the format described in this document, with added fields starting at byte eight of the format, or changes to the definition of bits in the Reserved field. 🧸

7. Security Considerations 🚨

Okay this section is SERIOUS bestie, read it all 😭🚨

The reader is directed to the Security Considerations section of [RFC8166] for background and further discussion. 📚

The RPC-over-RDMA version 1 protocol framework depends on the semantics of the Reliable Connected (RC) queue pair (QP) type, as defined in Section 9.7.7 of [IBA]. 🔐 The integrity of CM Private Data and the authenticity of its source are ensured by the exclusive use of RC QPs. 🛡️ Any attempt to interfere with or hijack data in transit on an RC connection results in the RDMA provider terminating the connection. 💥

The Security Considerations section of [RFC5042] refers the reader to further relevant discussion of generic RDMA transport security. 📖 That document recommends IPsec as the default transport-layer security solution. 🔒 When deployed with the Remote Direct Memory Access Protocol (RDMAP) [RFC5040], DDP [RFC5041], and MPA [RFC5044], IPsec establishes a protected channel before any operations are exchanged, so it protects the exchange of Private Data. 🫡 However, IPsec is not available for InfiniBand or RDMA over Converged Ethernet (RoCE) deployments. 😬 Those fabrics rely on physical security and cyclic redundancy checks to protect network traffic. 🧱

Exchanging the information contained in the message format defined in this document does not expose upper-layer payloads to an attacker. 🙅‍♀️ Furthermore, the behavior changes that occur as a result of exchanging the Private Data described in the current document do not introduce any new risk of exposure of upper-layer payload data. 🛑

Improperly setting one of the fields in version 1 Private Data can result in an increased risk of disconnection (i.e., self-imposed Denial of Service). 🚫 A similar risk can arise if non-RPC-over-RDMA CM Private Data inadvertently contains the Format Identifier that identifies this protocol's data structure. 😱 Additional checking of incoming Private Data, as described in Section 5.2, can help reduce this risk. 🧠

In addition to describing the structure of a new format version, any document that extends the Private Data format described in the current document must discuss security considerations of new data items exchanged between connection peers. 📝 Such documents should also explore the risks of erroneously identifying non-RPC-over-RDMA CM Private Data as the new format. 🕵️‍♀️

8. IANA Considerations 🎀

Okay bestie, IANA has a whole registry situation going on 💅 They made the "RDMA-CM Private Data Identifiers" subregistry, which lives inside the "Remote Direct Data Placement" protocol category group 🌸 It's a subregistry of 32-bit numbers that identify the upper-layer protocol behind the data in the application-specific RDMA-CM Private Data area ✨ Every entry has a Format Identifier, a Length (format length, in octets), a Description, and a Reference.

The registry starts with just one entry, and it's giving main character energy 👑

+===================+========+=======================+===========+
| Format Identifier | Length | Description           | Reference |
+===================+========+=======================+===========+
| 0xf6ab0e18        | 8      | RPC-over-RDMA version | RFC 8797  |
|                   |        | 1 CM Private Data     |           |
+-------------------+--------+-----------------------+-----------+

Table 1: New "RDMA-CM Private Data Identifiers" Registry

New entries get added later through the Specification Required policy, as defined in Section 4.6 of [RFC8126] 🦋

8.1. Guidance for Designated Experts

Okay so the Designated Expert (DE), who gets appointed by the IESG, SHOULD check that there's actual documentation defining the semantics and format of the Private Data 🧐 That doc has to be permanently and publicly available, too, no cap. Documentation written outside the IETF MUST NOT conflict with work that's active or already published within the IETF, so no drama between the two 🙈 The new Reference field SHOULD point to that documentation.

The Description field SHOULD hold the name of the upper-layer protocol that generates and uses the Private Data 🍓

The DE SHOULD give each new Format Identifier a number that doesn't clash with anything already in the registry, and that won't be easy to mistake for part of the payload of some other registered format 💜

Here's the process bit: the DE SHALL post the request to the NFSV4 Working Group mailing list (or a successor list, if one exists) for comment and review 📬 Then the DE SHALL approve or deny the request and publish notice of the decision within 30 days ⭐

9. References

9.1. Normative References

[IBA]      InfiniBand Trade Association, "InfiniBand Architecture
           Specification Volume 1", Release 1.3, March 2015,
           <https://www.infinibandta.org/>.

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119,
           DOI 10.17487/RFC2119, March 1997,
           <https://www.rfc-editor.org/info/rfc2119>.

[RFC4506]  Eisler, M., Ed., "XDR: External Data Representation
           Standard", STD 67, RFC 4506, DOI 10.17487/RFC4506, May
           2006, <https://www.rfc-editor.org/info/rfc4506>.

[RFC5040]  Recio, R., Metzler, B., Culley, P., Hilland, J., and D.
           Garcia, "A Remote Direct Memory Access Protocol
           Specification", RFC 5040, DOI 10.17487/RFC5040, October
           2007, <https://www.rfc-editor.org/info/rfc5040>.

[RFC5042]  Pinkerton, J. and E. Deleganes, "Direct Data Placement
           Protocol (DDP) / Remote Direct Memory Access Protocol
           (RDMAP) Security", RFC 5042, DOI 10.17487/RFC5042, October
           2007, <https://www.rfc-editor.org/info/rfc5042>.

[RFC8126]  Cotton, M., Leiba, B., and T. Narten, "Guidelines for
           Writing an IANA Considerations Section in RFCs", BCP 26,
           RFC 8126, DOI 10.17487/RFC8126, June 2017,
           <https://www.rfc-editor.org/info/rfc8126>.

[RFC8166]  Lever, C., Ed., Simpson, W., and T. Talpey, "Remote Direct
           Memory Access Transport for Remote Procedure Call Version
           1", RFC 8166, DOI 10.17487/RFC8166, June 2017,
           <https://www.rfc-editor.org/info/rfc8166>.

[RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
           2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
           May 2017, <https://www.rfc-editor.org/info/rfc8174>.

9.2. Informative References

[RFC1813]  Callaghan, B., Pawlowski, B., and P. Staubach, "NFS
           Version 3 Protocol Specification", RFC 1813,
           DOI 10.17487/RFC1813, June 1995,
           <https://www.rfc-editor.org/info/rfc1813>.

[RFC5041]  Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
           Data Placement over Reliable Transports", RFC 5041,
           DOI 10.17487/RFC5041, October 2007,
           <https://www.rfc-editor.org/info/rfc5041>.

[RFC5044]  Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
           Carrier, "Marker PDU Aligned Framing for TCP
           Specification", RFC 5044, DOI 10.17487/RFC5044, October
           2007, <https://www.rfc-editor.org/info/rfc5044>.

[RFC5531]  Thurlow, R., "RPC: Remote Procedure Call Protocol
           Specification Version 2", RFC 5531, DOI 10.17487/RFC5531,
           May 2009, <https://www.rfc-editor.org/info/rfc5531>.

[RFC5666]  Talpey, T. and B. Callaghan, "Remote Direct Memory Access
           Transport for Remote Procedure Call", RFC 5666,
           DOI 10.17487/RFC5666, January 2010,
           <https://www.rfc-editor.org/info/rfc5666>.

[RFC6581]  Kanevsky, A., Ed., Bestler, C., Ed., Sharp, R., and S.
           Wise, "Enhanced Remote Direct Memory Access (RDMA)
           Connection Establishment", RFC 6581, DOI 10.17487/RFC6581,
           April 2012, <https://www.rfc-editor.org/info/rfc6581>.

[RFC7530]  Haynes, T., Ed. and D. Noveck, Ed., "Network File System
           (NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530,
           March 2015, <https://www.rfc-editor.org/info/rfc7530>.

[RFC7861]  Adamson, A. and N. Williams, "Remote Procedure Call (RPC)
           Security Version 3", RFC 7861, DOI 10.17487/RFC7861,
           November 2016, <https://www.rfc-editor.org/info/rfc7861>.

Acknowledgments 🫶

Thanks to Christoph Hellwig and Devesh Sharma for suggesting this approach, and to Tom Talpey and David Noveck for their expert comments and review 💖 The author also wants to thank Bill Baker and Greg Marsden for their support of this work 🌈 Also, a big shoutout to expert reviewers Sean Hefty and Dave Minturn ✨

Special thanks go to document shepherd Brian Pawlowski, Transport Area Director Magnus Westerlund, NFSV4 Working Group Chairs David Noveck and Spencer Shepler, and NFSV4 Working Group Secretary Thomas Haynes 👑

Author's Address 🦋

Charles Lever
Oracle Corporation
United States of America

Email: chuck.lever@oracle.com
📦 where this page comes from