RFC 8167

Bidirectional Remote Procedure Call on RPC-over-RDMA Transports

📅 published June 2017, from ietf

Bidirectional RPC on RPC-over-RDMA, a.k.a. RFC 8167 💖

Okay bestie, so this is RFC 8167, titled Bidirectional Remote Procedure Call on RPC-over-RDMA Transports 🎀. It was written by C. Lever over at Oracle 💅. It's a Standards Track doc, dated June 2017, with ISSN 2070-1721 ✨. The header doesn't list anything it obsoletes or updates, but the body says outright that it does not update the protocol in [RFC8166] 🦋. Let's get into it!!

Abstract ✨

Okay so minor versions of NFS version 4 that are newer than minor version 0 work best when RPC transports can send RPC transactions in both directions on the same connection 🌈. This doc explains how RPC transport endpoints that can do RDMA (Remote Direct Memory Access) send RPCs both ways on a single connection 💜.

Status of This Memo 🌸

This is an Internet Standards Track document 💅.

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community 🫶. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG) 👑. If you want more info on Internet Standards, check Section 2 of RFC 7841 🍓.

Info about the current status of this doc, any errata, and how to send feedback can be found at http://www.rfc-editor.org/info/rfc8167 ✨.

Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved 🔥.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date this document was published 📜. Please read these carefully, bestie, because they spell out your rights and restrictions for this doc 🙈. Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License 💀.

Table of Contents 🧚

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
2.  Understanding RPC Direction . . . . . . . . . . . . . . . . .   3
3.  Immediate Uses of Bidirectional RPC-over-RDMA . . . . . . . .   5
4.  Flow Control  . . . . . . . . . . . . . . . . . . . . . . . .   6
5.  Sending and Receiving Operations in the Reverse Direction . .   8
6.  In the Absence of Support for Reverse-Direction Operation . .  11
7.  Considerations for ULBs . . . . . . . . . . . . . . . . . . .  11
8.  Security Considerations . . . . . . . . . . . . . . . . . . .  12
9.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  12
10. Normative References  . . . . . . . . . . . . . . . . . . . .  12
Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  13
Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  13

1. Introduction 🦋

Okay so RPC-over-RDMA transports, which were introduced in [RFC8166], send Remote Procedure Call (RPC) transactions efficiently over transport layers that can do Remote Direct Memory Access (RDMA) 🚀. The whole point of this doc is to let things run concurrently in both directions on a single transport connection, using RPC-over-RDMA protocol versions that don't have any special features for reverse-direction operation 💖.

Reverse-direction RPC transactions are necessary for version 4.1 of the Network File System (NFS), and in particular for Parallel NFS (pNFS) [RFC5661] ✨. Any Upper-Layer Protocol (ULP) implementation is allowed to use them too, though 🌈. An Upper-Layer Binding (ULB) for NFS version 4.x callback operation is also required (see Section 7), but this document does not provide one 🙈.

For example, using the approach in here, RPC transactions can go both ways on the same RPC-over-RDMA version 1 connection, and you don't need to change the RPC-over-RDMA version 1 protocol at all 🎀. This document does not update the protocol specified in [RFC8166] 💅.

The rest of this document assumes you're already familiar with the terminology and concepts in [RFC8166], especially Sections 2 and 3 🧚.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 🫶.

2. Understanding RPC Direction 🌸

The Open Network Computing Remote Procedure Call (ONC RPC) protocol, as described in [RFC5531], is built as a message-passing protocol between one server and one or more clients 🍓. ONC RPC transactions are made up of two types of messages 💜.

A CALL message, or "Call", requests work 💅. A Call is identified by the value CALL in the message's msg_type field ✨. An arbitrary unique value goes in the message's Transaction ID (XID) field 🔥. A host that starts a Call is called a "Requester" in this document 🎀.

A REPLY message, or "Reply", reports the results of work that a Call asked for 👑. A Reply is identified by the value REPLY in the message's msg_type field 🌈. The value in the message's XID field is copied from the Call whose results are being sent back 🦋. A host that sends a Reply is called a "Responder" 💖.

Typically, a Call gets a matching Reply 🧚. A Reply is never sent without a matching Call 🙈.

RPC-over-RDMA is a connection-oriented RPC transport 🚀. In every case where a connection-oriented transport is used, ONC RPC client endpoints are the ones responsible for starting transport connections, while ONC RPC service endpoints just sit there passively waiting for incoming connection requests 💅.

RPC direction on connectionless RPC transports is not covered in this document 💀.

2.1. Forward Direction 🦋

Traditionally, an ONC RPC client acts as a Requester, while an ONC RPC service acts as a Responder ✨. This way of passing messages is called "forward-direction" operation 🌸.

2.2. Reverse Direction 🌈

The ONC RPC specification [RFC5531] doesn't forbid passing messages the other way 🫶. An ONC RPC service endpoint can act as a Requester, and in that case an ONC RPC client endpoint acts as a Responder 🔥. This way of passing messages is called "reverse-direction" operation 💜.

During reverse-direction operation, the ONC RPC client is still the one responsible for setting up transport connections, even though the RPC Call messages come from the ONC RPC server 🎀.

ONC RPC clients and servers are optimized to perform and scale well when handling forward-direction traffic, and they might not be ready to handle reverse-direction operation 🙈. It wasn't until NFS version 4.1 [RFC5661] that there was a strong need to handle reverse-direction operation 👑.

2.3. Bidirectional Operation 💅

A pair of connected RPC endpoints can choose to use only forward-direction operations or only reverse-direction operations on a particular transport connection 🧚. Or, these endpoints can send Calls in both directions at the same time on the same transport connection ✨.

"Bidirectional operation" happens when both transport endpoints act as a Requester and a Responder at the same time 🌷.

Bidirectionality is an extension of RPC transport connection sharing 🍓. Two RPC endpoints want to swap independent RPC messages over a shared connection, but going in opposite directions 💖. These messages may or may not be related to the same workloads or RPC Programs 🌈.

2.4. XID Values 🎀

Section 9 of [RFC5531] introduces the ONC RPC transaction identifier, or "XID" for short 💜. The value of an XID is interpreted in the context of the message's msg_type field 🫶.

  • The XID of a Call is arbitrary, but it's unique among the outstanding Calls from that Requester 💅.
  • The XID of a Reply always matches the XID of the Call that started it 🔥.

When a Requester receives a Reply, it matches the XID value in the Reply with a Call it sent earlier 🦋.

2.4.1. XID Generation 🧚

During bidirectional operation, forward- and reverse-direction XIDs are usually generated on different hosts, sometimes using different algorithms 🌸. There's no coordination between forward- and reverse-direction XID generation 💀.

So, a forward-direction Requester MAY use the same XID value at the same time as a reverse-direction Requester on the same transport connection 🌈. Even though these concurrent requests use the same XID value, they represent distinct ONC RPC transactions 👑.

3. Immediate Uses of Bidirectional RPC-over-RDMA 🌷

3.1. NFS Version 4.0 Callback Operation 🍓

An NFS version 4.0 client uses a traditional ONC RPC client to send NFS requests to an NFS version 4.0 server's traditional ONC RPC service [RFC7530] 💖. NFS version 4.0 requests flow in the forward direction on a connection the client sets up 💜. This connection is called a "forechannel" connection 🫶.

An NFS version 4.x "delegation" is basically a promise from a server that it will notify a client before another client or program running on the server is allowed to access a file 🔥. With that guarantee, the client can act as the sole accessor of the file 🎀. In particular, it can manage the file's data and metadata caches aggressively 💅.

To manage file delegations, NFS version 4.0 brings in callback operations, or "callbacks," described in Section 10.2 of [RFC7530] ✨. An NFS version 4.0 server sets up a forward-direction ONC RPC client, and an NFS version 4.0 client sets up a forward-direction ONC RPC service 🌈. Callbacks flow in the forward direction on a connection set up between the server's callback client and the client's callback service 🦋. This connection is separate from the connections used as forechannels, and it's called a "backchannel connection" 🧚.

When an RDMA transport is used as a forechannel, an NFS version 4.0 client usually provides a TCP-based callback service 🌸. The client's SETCLIENTID operation advertises the callback service endpoint with a tcp or tcp6 netid 💜. Then the server connects to this service using a TCP socket 👑.

NFS version 4.0 implementations can work without a backchannel in place 🙈. When that happens, the NFS server doesn't grant file delegations 💀. This might have a negative performance effect, but correctness isn't affected 🍓.

3.2. NFS Version 4.1 Callback Operation 🦋

okay so NFS version 4.1 is giving the same file delegation energy as 4.0 🎀 but it also stretches the callback mechanism to manage pNFS layouts. You can read all about that in Section 12 of [RFC5661] 🧚

NFS version 4.1 transport connections are started by NFS version 4.1 clients, so NFS version 4.1 servers send callbacks back to clients in the reverse direction on connections those clients made 🌸

NFS version 4.1 clients and servers let each other know a backchannel is available on a given transport connection. They do this in the arguments and results of the NFS CREATE_SESSION or BIND_CONN_TO_SESSION operations 💌

NFS version 4.1 clients can use distinct transport connections for forechannel and backchannel operation, or they can squish both onto one transport connection using bidirectional operation 👯‍♀️

Without a reverse-direction RPC-over-RDMA capability, an NFS version 4.1 client also connects using a transport that does have reverse-direction capability, and uses that as its backchannel. In this case, opening an independent TCP socket is the only choice for an NFS version 4.1 backchannel connection 🔌

Implementations often find a single combined transport (one that can do bidirectional operation) way more convenient 💖 It simplifies connection establishment and recovery during network partitions or when one endpoint restarts. It can also enable better scaling, since fewer transport connections get the same work done ✨

As with NFS version 4.0, if a backchannel is not in use, an NFS version 4.1 server does not grant delegations. And because NFS version 4.1 relies on callbacks to manage pNFS layout state, pNFS operation is not possible without a backchannel 💀

4. Flow Control 💅

For an RDMA Send operation to work properly, the receiving peer has to have already posted a Receive buffer to accept the incoming message. If a receiver hasn't posted enough buffers for each incoming Send operation, the receiving RDMA provider is allowed to terminate the RDMA connection 😭

RPC-over-RDMA transport protocols have built-in send flow control so nobody overruns the pre-posted Receive buffers on a connection's receive endpoint. They do this with a "credit grant" mechanism. The use of credits in RPC-over-RDMA version 1 is described in Section 3.3.1 of [RFC8166] 🎀

4.1. Reverse-Direction Credits

RPC-over-RDMA credits work the same way in the reverse direction as they do in the forward direction. The one twist is that forward-direction credits and reverse-direction credits on the same connection are accounted for separately. This direction-independent accounting means head-of-line blocking in one direction can't mess with operation in the other direction 🦋

The forward-direction credit value keeps the same meaning whether or not there are reverse-direction resources associated with an RPC-over-RDMA transport connection. It's the number of RPC requests the forward-direction Responder (the ONC RPC server) is prepared to receive concurrently 🌈

The reverse-direction credit value is the number of RPC requests the reverse-direction Responder (the ONC RPC client) is prepared to receive concurrently. It MAY be different than the forward-direction credit value 🧚

During bidirectional operation, each receiver has to figure out whether an incoming message contains a credit request (the receiver is acting as a Responder) or a credit grant (the receiver is acting as a requester). Then it applies the credit value accordingly 👑

Now the tricky bit. When message direction is not fully determined by context (e.g., suggested by the definition of the RPC-over-RDMA version that is in use) or by an accompanying RPC message payload with a call direction field, the receiver can't tell with certainty whether the header credit value is a request or a grant. In such cases, the receiver MUST ignore the header's credit value 🙈

4.2. Inline Thresholds

Forward- and reverse-direction operation on the same connection share the same Receive buffers, so the inline threshold values for both directions are the same. The call inline threshold for the reverse direction is the same as the reply inline threshold for the forward direction, and vice versa. For more info, see Section 3.3.2 of [RFC8166] 🍓

4.3. Managing Receive Buffers

An RPC-over-RDMA transport endpoint posts Receive buffers before it can receive and process incoming RPC-over-RDMA messages. If a sender transmits a message for a receiver with no posted Receive buffer, the RDMA provider is allowed to drop the RDMA connection 😬

4.3.1. Client Receive Buffers

Typically, an RPC-over-RDMA Requester only posts as many Receive buffers as there are outstanding RPC Calls. So a client endpoint without reverse-direction support might, at times, have no available Receive buffers left 🥺

To receive incoming reverse-direction Calls, an RPC-over-RDMA client endpoint posts enough additional Receive buffers to match its advertised reverse-direction credit value. Each outstanding forward-direction RPC needs one more Receive buffer on top of that minimum 🫶

When an RDMA transport connection is lost, all active Receive buffers get flushed and are no longer available to receive incoming messages. When a fresh transport connection is established, a client endpoint posts a Receive buffer to handle the Reply for each retransmitted forward-direction Call. It also posts enough Receive buffers to handle reverse-direction Calls 🔄

4.3.2. Server Receive Buffers

A forward-direction RPC-over-RDMA service endpoint posts as many Receive buffers as it expects incoming forward-direction Calls. In other words, it posts no fewer buffers than the number of credits granted in the rdma_credit field of forward-direction RPC replies 🎀

To receive incoming reverse-direction replies, an RPC-over-RDMA server endpoint posts enough additional Receive buffers to handle replies for each reverse-direction Call it sends 💜

When the existing transport connection is lost, all active Receive buffers are flushed and are no longer available to receive incoming messages. When a fresh transport connection is established, a server endpoint posts a Receive buffer to handle the Reply for each retransmitted reverse-direction Call. It also posts enough Receive buffers to handle incoming forward-direction Calls 🌸

5. Sending and Receiving Operations in the Reverse Direction 🌈

The operation of RPC-over-RDMA transports in the forward direction is defined in [RFC5531] and [RFC8166]. In this section, we define a mechanism for reverse-direction operation on RPC-over-RDMA. Reverse-direction operation used together with forward-direction operation enables bidirectional communication on a common RPC-over-RDMA transport connection 🦋

Certain fields in the RPC-over-RDMA header have a fixed position in all versions of RPC-over-RDMA. The normative specification of these fields is in Section 4 of [RFC8166] 📜

5.1. Sending a Call in the Reverse Direction

To form a reverse-direction RPC-over-RDMA Call message, an ONC RPC service endpoint builds an RPC-over-RDMA header containing a fresh RPC XID in the rdma_xid field (see Section 2.4 for full requirements) 💌

The rdma_vers field MUST contain the same value in reverse- and forward-direction Call messages on the same connection. Think of it as matching outfits, bestie 👗

The number of requested reverse-direction credits goes in the rdma_credit field (see Section 4) 🎟️

Whether it's presented inline or as a separate chunk, the ONC RPC Call header MUST start with the same XID value that is present in the RPC-over-RDMA header. Also, the RPC header's msg_type field MUST contain the value CALL 🫶

5.2. Sending a Reply in the Reverse Direction

To form a reverse-direction RPC-over-RDMA Reply message, an ONC RPC client endpoint builds an RPC-over-RDMA header containing a copy of the matching ONC RPC Call's RPC XID in the rdma_xid field (see Section 2.4 for full requirements) 🧁

The rdma_vers field MUST contain the same value in a reverse-direction Reply message as in the matching Call message. Same energy as before, no cap 🤝

The number of granted reverse-direction credits goes in the rdma_credit field (see Section 4) 💖

Whether it's presented inline or as a separate chunk, the ONC RPC Reply header MUST start with the same XID value that is present in the RPC-over-RDMA header. Also, the RPC header's msg_type field MUST contain the value REPLY ✨

5.3. Using Chunks in Reverse-Direction Operations

A "chunk" refers to a portion of a message's Payload stream that is DDP-eligible and gets placed directly in the receiver's memory by the transport. Chunk data may be moved by an explicit RDMA operation, for example. Chunks are defined in Section 3.4.4, and DDP-eligibility is covered in Section 6.1 of [RFC8166] 🧩

Chunks MAY be used in the reverse direction. They work the same way as in the forward direction 🌷

An implementation might support only ULPs that have no DDP-eligible data items. Those ULPs may use only small messages, or they may have a native mechanism for restricting the size of reverse-direction RPC messages. That way, there's no need to handle Long Messages in the reverse direction 🍃

When there's no ULP requirement for chunks in the reverse direction, implementers can choose not to support chunks there. This avoids the complexity of adding support for performing RDMA Reads and Writes in the reverse direction 🙌

When chunks aren't implemented, RPC messages in the reverse direction are always sent using a Short Message. So they can be no larger than what can be sent inline (that is, without chunks). Sending an inline message larger than the inline threshold can result in loss of connection 🚨

If a reverse-direction requester provides a non-empty chunk list to a Responder that does not support chunks, the Responder MUST reply with an RDMA_ERROR message with the rdma_err field set to ERR_CHUNK. Big "nope" energy, and it's a must 🙅‍♀️

5.4. Reverse-Direction Retransmission 🔁

okay so every once in a while an ONC RPC service just can't finish an RPC transaction and send back a reply 😭. That can happen because the transport connection got lost, because the Call or Reply message got dropped, or because the ULP delayed or dropped the ONC RPC request. Usually the Requester sends the whole RPC transaction again, reusing the same RPC XID. This is known as an "RPC retransmission" ✨

In the forward direction, the Requester is the ONC RPC client. The client is always the one responsible for setting up a transport connection before it sends again 💅

With reverse-direction operation, the Requester is the ONC RPC server 🙈. Here's the tea: an ONC RPC server doesn't set up transport connections with clients, so it can't retransmit if there's no transport connection. It's stuck waiting for the ONC RPC client to re-establish a transport connection before it can retransmit ONC RPC transactions in the reverse direction. Not fair, but that's the vibe 💀

If the ONC RPC client peer has no work to do, it can be a while before it re-establishes a transport connection 😴. A waiting reverse-direction ONC RPC Call may time out, so it doesn't wait forever for a connection to show up ⏳

So, forward-direction Requesters SHOULD maintain a transport connection as long as there's a possibility that the connection peer can send reverse-direction requests. For example, while an NFS version 4.1 client has open delegated files or active pNFS layouts, it keeps one or more transport connections open so the NFS server can perform callback operations 🦋

6. In the Absence of Support for Reverse-Direction Operation 🌸

An RPC-over-RDMA transport endpoint might not support reverse-direction operation, which means it doesn't support bidirectional operation either 🥺. There might be no mechanism in the transport implementation to do that. Or, in an implementation that can support reverse-direction operation, the ULP might not have configured or enabled the transport to handle reverse-direction traffic yet.

If an endpoint isn't ready to receive an incoming reverse-direction message, the RDMA connection might get lost. So denial of service could happen if a sender keeps sending reverse-direction messages after every transport reconnect to an endpoint that isn't prepared to receive them 🚫

When dealing with the possibility that the remote peer has no transport-level support for reverse-direction operation, the ULP becomes responsible for telling peers when reverse-direction operation is supported. Otherwise, even a simple reverse-direction RPC NULL procedure from a peer could lose the connection 😱

So, a ULP MUST NOT perform reverse-direction ONC RPC operations until the peer has indicated it's ready to handle them 🛑. How ULPs do that indication is outside the scope of this document, so that's a whole other story 📖

For example, an NFS version 4.1 server doesn't send backchannel messages to an NFS version 4.1 client before that client has sent a CREATE_SESSION or a BIND_CONN_TO_SESSION operation. As long as the NFS version 4.1 client has prepared appropriate resources to receive reverse-direction operations before sending one of those NFS operations, denial of service is avoided 🙌

7. Considerations for ULBs 🎀

A ULP that operates on RPC-over-RDMA transports may have procedures that include DDP-eligible data items. DDP-eligibility is specified in an Upper-Layer Binding (ULB). Direction of operation doesn't cancel out the need for DDP-eligibility statements, bestie 💖

Reverse-direction-only operation requires the client endpoint to establish a fresh connection. The ULB can specify appropriate RPC binding parameters for those connections 🧚

Bidirectional operation happens on an already-established connection. Specifying RPC binding parameters is usually not necessary in this case ✨

For bidirectional operation, other considerations may apply when distinct RPC Programs share an RPC-over-RDMA transport connection at the same time. Check out Section 6 of [RFC8166] for details about what else may be contained in a ULB 📚

8. Security Considerations 🔐

RPC security is handled in the RPC layer, which sits above the transport layer where RPC-over-RDMA operates. Lowkey the most important thing to know here 👑

Reverse-direction operations use an authentication mechanism and credentials that are independent of forward-direction operation. Otherwise they work the same way outlined in Section 8.2 of [RFC8166] 🔒

9. IANA Considerations 🌈

This document does not require any IANA actions. Nothing to register, nothing to update, we're chillin' 😌

10. Normative References 📜

Acknowledgements 🫶

Tom Talpey was an indispensable resource, and he created the foundation this whole work is built on. The author's warmest regards go to him for his help and support 💜

Dave Noveck provided excellent review, constructive suggestions, and navigational guidance throughout the process of drafting this document 🧭

Dai Ngo was a solid partner and collaborator. Together they constructed and tested independent prototypes of the changes described in this document 🤝

The author wishes to thank Bill Baker and Greg Marsden for their unwavering support of this work. The author also gratefully acknowledges the expert contributions of Karen Deitke, Chunli Zhang, Mahesh Siddheshwar, Steve Wise, and Tom Tucker 🌟

Special thanks go to Transport Area Director Spencer Dawkins, NFSV4 Working Group Chair and Document Shepherd Spencer Shepler, and NFSV4 Working Group Secretary Tom Haynes for their support 🎉

Author's Address 📮

Charles Lever
Oracle Corporation
1015 Granger Avenue
Ann Arbor, MI  48104
United States of America

Phone: +1 248 816 6463
Email: chuck.lever@oracle.com
📦 where this page comes from