RFC 9754

Extensions for Opening and Delegating Files in NFSv4.2

📅 published March 2025, from ietf

RFC 9754: Extensions for Opening and Delegating Files in NFSv4.2 ✨

Okay bestie, here's the title block 🎀

  • Who wrote it: T. Haynes and T. Myklebust, both from Hammerspace 🧑‍💻
  • Stream and category: Internet Engineering Task Force (IETF), Standards Track 🌈
  • Date: March 2025 🗓️
  • ISSN: 2070-1721
  • Obsoletes/Updates: the header doesn't list any obsoleted or updated documents. What it does say is that it extends NFSv4.2, which the Abstract covers below. 🦋

Abstract

Okay so NFSv4 lets a client open a file and get a delegation for that same file 🔥 A delegation basically means the client gets to be the authority for that file's metadata and cache it locally, which is so iconic 💅 This doc adds several extensions for both opening the file and delegating it to the client ✨ It extends NFSv4.2 (see RFC 7863) 🎀

Status of This Memo

This is an Internet Standards Track document 🌸

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community, it got public review, and the Internet Engineering Steering Group (IESG) approved it for publication 🫶 For more on Internet Standards, check Section 2 of RFC 7841.

Want the current status, any errata, or a way to send feedback? Go to https://www.rfc-editor.org/info/rfc9754 🦋

Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved. 💖

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the publication date. Please actually read these, because they spell out your rights and restrictions for this document 🙈 Any Code Components pulled out of this document must include Revised BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as described in the Revised BSD License.

Table of Contents

   1.  Introduction
     1.1.  Definitions
     1.2.  Requirements Language
   2.  Offline Files
     2.1.  XDR for the Offline Attribute
   3.  Determining OPEN Feature Support
     3.1.  XDR for Open Arguments
   4.  OPEN Grants Either an Open or a Delegation Stateid
     4.1.  Implementation Experience
   5.  Proxying of Times
     5.1.  Use Case for NFSv3 Client Proxy
     5.2.  XDR for Proxying of Times
   6.  Extraction of XDR
   7.  Security Considerations
   8.  IANA Considerations
   9.  Normative References
   Acknowledgments
   Authors' Addresses

1. Introduction ✨

In NFSv4, a client can be granted a delegation for a file (see Section 1.8.4 of [RFC8881]). That makes the client the authority for the file's data and metadata, which is giving main character energy 👑 This document adds a bunch of extensions that level up opens and delegations. They let the client:

  • detect an offline file, which may require significant effort to obtain;
  • determine which extensions of OPEN flags are supported by the server;
  • retrieve either the open or delegation stateid, but not both simultaneously, during the OPEN procedure; and
  • cache both the access and modify timestamps, thereby reducing the frequency with which the client must query the server for this information.

Using the process in [RFC8178], the revisions in this document become an extension of NFSv4.2 [RFC7862]. They're built on top of the External Data Representation (XDR) [RFC4506] generated from [RFC7863] 🧚

1.1. Definitions

Here's the terminology this doc uses 📖

  • offline file: A file that lives on a device that isn't connected to the server. Bringing it online usually costs something. Historically that meant a file on tape media, and the cost was finding and loading the tape 📼 These days it's more likely a file in the cloud, where the cost is money for downloading it 💸
  • proxy: Attributes get proxied when a client has the authority, granted by the right delegation, to represent the attributes the server normally maintains. For read attributes, that happens when the client has either a read or write delegation for the file. For write attributes, it happens when the client has a write delegation for the file. The client with this authority is the "proxy" for those attributes 🫶

The following terms are referenced as well, with their definitions found where shown:

  • CB_GETATTR (Section 20.1 of [RFC8881])
  • change (Section 5.8.1.4 of [RFC8881])
  • CLOSE (Section 18.2 of [RFC8881])
  • compound (Section 2.3 of [RFC8881])
  • DELEGRETURN (Section 18.6 of [RFC8881])
  • GETATTR (Section 18.7 of [RFC8881])
  • LAYOUTGET (Section 18.43 of [RFC8881])
  • LOCK (Section 18.10 of [RFC8881])
  • NFS4ERR_DELAY (Section 15.1.1.3 of [RFC8881])
  • OPEN (Section 18.16 of [RFC8881])
  • open_delegation_type4 (Section 18.16.1 of [RFC8881])
  • READ (Section 18.22 of [RFC8881])
  • READDIR (Section 18.23 of [RFC8881])
  • SETATTR (Section 18.30 of [RFC8881])
  • stateid (Section 8.2 of [RFC8881])
  • time_access (Section 5.8.2.37 of [RFC8881])
  • time_metadata (Section 5.8.2.42 of [RFC8881])
  • time_modify (Section 5.8.2.43 of [RFC8881])
  • WRITE (Section 18.32 of [RFC8881])

1.2. Requirements Language

Okay, keyword rules time 🔑 The words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document carry the meaning from BCP 14 [RFC2119] [RFC8174]. That only applies when, and only when, they're written in all capitals, the way they are right here 💅

2. Offline Files 🧊

If a file is offline, the server has immediate, high-performance access to its attributes, but not to its content. Pulling the data content is expensive, so the content should only be retrieved if it's actually going to be used 😭 For example, a graphical file manager (such as Finder in Mac OS X) may want to peek at the beginning of a file so it can preview it while a user hovers over the file name, without actually opening it. If the file does get retrieved, it'll most likely be either immediately thrown away or returned 🙈

A compound with a GETATTR or READDIR can report the file's attributes without bringing the file online. But either an OPEN or a LAYOUTGET might make the file server retrieve the archived data contents, which brings the file online ✨ For non-parallel NFS systems (see Section 12 of [RFC8881]), the OPEN operation requires a filehandle to retrieve the data content. For parallel NFS (pNFS) systems, the filehandle retrieved from an OPEN doesn't need to cause the data content to be retrieved. However, when the LAYOUTGET operation is processed, a layout-type-specific mapping will cause the data content to be retrieved from offline storage 🧚

If the client doesn't know the file is offline, it might accidentally open the file just to figure out what type of file it is 😬 By checking the new attribute fattr4_offline, a client can predetermine whether the file is available, so it doesn't have to open it at all 🦋 Being offline can also mean the file is archived in the cloud, i.e., there can be a cost both for retrieving the file to bring it online and for sending it back to offline status 💸

2.1. XDR for the Offline Attribute

<CODE BEGINS>
///
/// typedef bool            fattr4_offline;
///
///
/// const FATTR4_OFFLINE            = 83;
///
<CODE ENDS>

3. Determining OPEN Feature Support 🔍

Section 4.4.2 of [RFC8178] lets you extend a particular minor version of the NFSv4 protocol without defining a brand-new minor version, which is so smart 🧠 The client can probe the server's capabilities and, based on the result, figure out whether both it and the server support optional features that weren't previously specified as part of the minor version 🌈

The fattr4_open_arguments attribute is a new XDR extension that's super helpful when OPEN gets extended this way 💖 It models all of the parameters with bitmap4 data structures, which means you can add a new flag to any of the OPEN arguments. The scope of this attribute applies to all objects with a matching fsid 🎀

Two new flags are provided:

  • OPEN4_SHARE_ACCESS_WANT_OPEN_XOR_DELEGATION (see Section 4)
  • OPEN4_SHARE_ACCESS_WANT_DELEG_TIMESTAMPS (see Section 5)

Future extensions can use this framework when they add new OPTIONAL functionality to OPEN, by creating a new flag for each OPTIONAL parameter 🫶

Since fattr4_open_arguments is a RECOMMENDED attribute, if the server tells the client via NFS4ERR_ATTRNOTSUPP that it doesn't support this new attribute, the client MUST take that to mean the additional new OPTIONAL functionality to OPEN isn't supported either. Not me assuming otherwise 🙅‍♀️

There's also the question of handling flags that are currently REQUIRED alongside OPTIONAL flags that become REQUIRED in the future 🤔 The server MUST mark REQUIRED flags as being supported. And these flags MUST only change from OPTIONAL to REQUIRED when the NFSv4 minor version is incremented. No sneaky changes mid-version, periodt 💅✨

3.1. XDR for Open Arguments 🎀

okay so this part is the XDR (the type language this protocol uses to describe its data) for the OPEN arguments 💅. Here's the whole thing:

///
/// struct open_arguments4 {
///   bitmap4  oa_share_access;
///   bitmap4  oa_share_deny;
///   bitmap4  oa_share_access_want;
///   bitmap4  oa_open_claim;
///   bitmap4  oa_create_mode;
/// };
///
///
/// enum open_args_share_access4 {
///    OPEN_ARGS_SHARE_ACCESS_READ  = 1,
///    OPEN_ARGS_SHARE_ACCESS_WRITE = 2,
///    OPEN_ARGS_SHARE_ACCESS_BOTH  = 3
/// };
///
///
/// enum open_args_share_deny4 {
///    OPEN_ARGS_SHARE_DENY_NONE  = 0,
///    OPEN_ARGS_SHARE_DENY_READ  = 1,
///    OPEN_ARGS_SHARE_DENY_WRITE = 2,
///    OPEN_ARGS_SHARE_DENY_BOTH  = 3
/// };
///
///
/// enum open_args_share_access_want4 {
///    OPEN_ARGS_SHARE_ACCESS_WANT_ANY_DELEG           = 3,
///    OPEN_ARGS_SHARE_ACCESS_WANT_NO_DELEG            = 4,
///    OPEN_ARGS_SHARE_ACCESS_WANT_CANCEL              = 5,
///    OPEN_ARGS_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_RESRC_AVAIL
///                                                    = 17,
///    OPEN_ARGS_SHARE_ACCESS_WANT_PUSH_DELEG_WHEN_UNCONTENDED
///                                                    = 18,
///    OPEN_ARGS_SHARE_ACCESS_WANT_DELEG_TIMESTAMPS    = 20,
///    OPEN_ARGS_SHARE_ACCESS_WANT_OPEN_XOR_DELEGATION = 21
/// };
///
///
/// enum open_args_open_claim4 {
///    OPEN_ARGS_OPEN_CLAIM_NULL          = 0,
///    OPEN_ARGS_OPEN_CLAIM_PREVIOUS      = 1,
///    OPEN_ARGS_OPEN_CLAIM_DELEGATE_CUR  = 2,
///    OPEN_ARGS_OPEN_CLAIM_DELEGATE_PREV = 3,
///    OPEN_ARGS_OPEN_CLAIM_FH            = 4,
///    OPEN_ARGS_OPEN_CLAIM_DELEG_CUR_FH  = 5,
///    OPEN_ARGS_OPEN_CLAIM_DELEG_PREV_FH = 6
/// };
///
///
/// enum open_args_createmode4 {
///    OPEN_ARGS_CREATEMODE_UNCHECKED4     = 0,
///    OPEN_ARGS_CREATE_MODE_GUARDED       = 1,
///    OPEN_ARGS_CREATEMODE_EXCLUSIVE4     = 2,
///    OPEN_ARGS_CREATE_MODE_EXCLUSIVE4_1  = 3
/// };
///
///
/// typedef open_arguments4 fattr4_open_arguments;
///
///
/// %/*
/// % * Determine what OPEN supports.
/// % */
/// const FATTR4_OPEN_ARGUMENTS     = 86;
///
///
/// const OPEN4_SHARE_ACCESS_WANT_OPEN_XOR_DELEGATION = 0x200000;
///
///
/// const OPEN4_RESULT_NO_OPEN_STATEID = 0x00000010;

4. OPEN Grants Either an Open or a Delegation Stateid ✨

okay bestie, so the OPEN procedure hands the client an open stateid 🦋 so it can reference the state of the file. The client can also ask for a delegation stateid in the OPEN arguments. The file counts as open for the client as long as the count of open and delegated stateids is greater than 0 💖. Either type is enough for the server to treat the file as open, which lets READ, WRITE, LOCK, and LAYOUTGET operations go through. If the client gets both an open and a delegation stateid as part of the OPEN, it has to return them both to the server 🫶. And one more thing: during each operation, the client can send a costly GETATTR 😭.

If the client knows that the server supports the OPEN4_SHARE_ACCESS_WANT_OPEN_XOR_DELEGATION flag (as determined by an earlier GETATTR operation that queried for the fattr4_open_arguments attribute), then the client can supply that flag during the OPEN and get either an open or a delegation stateid 🌸.

The client is already prepared to not get a delegation stateid, even if it asked for one 🧚. To not send an open stateid, the server MUST indicate that fact with the result flag OPEN4_RESULT_NO_OPEN_STATEID. The open stateid field, OPEN4resok.stateid, MUST be set to the special all-zero stateid in this case.

Note that the OPEN4_SHARE_ACCESS_WANT_OPEN_XOR_DELEGATION flag is a hint 👀. The server might return both stateids. Picture this: the client opens a file read-only (with OPEN4_SHARE_ACCESS_WANT_OPEN_XOR_DELEGATION set) and only gets an open stateid. Then it opens the same file read-write (again with OPEN4_SHARE_ACCESS_WANT_OPEN_XOR_DELEGATION set). The server can return one of these three options:

  1. Only an open stateid with the correct seqid.
  2. Only a delegation stateid, with the open stateid now having an incorrect seqid as it needs to be upgraded.
  3. Both an open stateid (which will be upgraded) and a delegation stateid.
In that scenario, returning just a delegation stateid would hide information from the client. So if the client already has an open stateid, then the server SHOULD ignore the OPEN4_SHARE_ACCESS_WANT_OPEN_XOR_DELEGATION flag and return both the open and delegation stateids 🙈.

4.1. Implementation Experience 🍓

The CLOSE operation neither explicitly nor implicitly releases any delegation stateids. This is not symmetrical with the OPEN operation, which can grant both an open and a delegation stateid. This specification could have tried to extend the CLOSE operation to release both stateids, but implementation experience shows that is more costly than the approach that has been proposed 💅.

Consider a small workload of creating a file with content. With existing implementations, this involves three synchronous operations and one asynchronous operation:

  • The first synchronous operation has to OPEN the file.
  • The second synchronous operation performs the WRITE to the file.
  • The third synchronous operation has to CLOSE the file.
  • The asynchronous operation uses DELEGRETURN to return the delegation stateid.
      SEQ PUTFH OPEN GETFH GETATTR
      SEQ PUTFH WRITE GETATTR
      SEQ PUTFH CLOSE
      ...
      SEQ PUTFH DELEGRETURN

With the proposed approach of setting the OPEN_ARGS_SHARE_ACCESS_WANT_OPEN_XOR_DELEGATION flag during the OPEN, the number of operations is always three 🌈. The first two compounds are still synchronous, but the last one is asynchronous. That's because the client no longer has to send a CLOSE operation, so it can delay the DELEGRETURN until either the server requests it back via delegation recall, or garbage collection causes the client to return the stateid 🦋.

      SEQ PUTFH OPEN(OPEN_ARGS_SHARE_ACCESS_WANT_OPEN_XOR_DELEGATION)
          GETFH GETATTR
      SEQ PUTFH WRITE GETATTR
      ...
      SEQ PUTFH DELEGRETURN

This approach reduces the cost of synchronous operations by 33% and the total number of operations by 25% 🔥. Contrast that with the alternative proposal of having CLOSE return both stateids, which would not reduce the number of synchronous operations 💀.

5. Proxying of Times ✨

Okay so, bestie, when a client gets a write delegation on a file, it becomes the boss of the file contents and the associated attributes 👑. If the server asks what's going on with the file via a CB_GETATTR, then under the unextended NFSv4 protocol, the server can only figure out the file size and the change attribute 🤷‍♀️. Meanwhile the delegation-holding client has the current access and modify times, and literally no other client can see those values 🙈. To tell the server about these proxied values, the client could send a compound of the form SEQ, PUTFH, SETATTR (time_modify | time_access), DELEGRETURN. The catch is that the SETATTR would make either or both of the change attribute or time_metadata attribute jump to the current time on the server 😩. There's also no current way to grab these values before the delegation comes back using CB_GETATTR. So it can't pass these times along to an application that expects POSIX compliance, which is often super necessary for things to work right 💀.

With the new OPEN4_SHARE_ACCESS_WANT_DELEG_TIMESTAMPS flag, the client and server can negotiate that the client is the authority for these values 🦋. Then, when the delegation stateid gets returned via a DELEGRETURN, the times get passed back to the server 💖. If another client queries the server for either the size or the times, the server will need to use a CB_GETATTR to ask the client holding the delegation 🫶.

If a server tells the client through the fattr4_open_arguments attribute that it supports OPEN_ARGS_SHARE_ACCESS_WANT_DELEG_TIMESTAMPS, and it returns a valid delegation stateid for an OPEN operation that sets the OPEN4_SHARE_ACCESS_WANT_DELEG_TIMESTAMPS flag, then it MUST query the client via a CB_GETATTR for the fattr4_time_deleg_access attribute (see Section 5.2) and the fattr4_time_deleg_modify attribute (see Section 5.2). (Note that the change time can be derived from the modify time 🧚.) Further, when a server gets a SETATTR with those attributes set, then it MUST accept those changes in the fattr4_time_deleg_access and fattr4_time_deleg_modify attributes and derive the change time, or it MUST reject the changes with NFS4ERR_DELAY 🎀.

When the server grants a delegation stateid, it MUST inform the client by setting the appropriate flag in the open_delegation_type4 response ✨. The server MUST set OPEN_DELEGATE_READ_ATTRS_DELEG when it grants a read attribute delegation, and MUST set OPEN_DELEGATE_WRITE_ATTRS_DELEG when it grants a write attribute delegation 🌈.

These new attributes are invalid to use with GETATTR, VERIFY, and NVERIFY, and they can only be used with CB_GETATTR and SETATTR by a client holding an appropriate delegation 🚫. The SETATTR SHOULD be either 1) in a separate compound before the one containing the DELEGRETURN, or 2) in the same compound as an operation before the DELEGRETURN. Failure to sequence the operations properly may lead to race conditions 😭.

A key prerequisite of this approach is that the server and client are in time synchronization with each other ⏰. Note that while base NFSv4.2 doesn't require this synchronization, using RPCSEC_GSS typically makes it a requirement 🔐. When the client presents either the fattr4_time_deleg_access or the fattr4_time_deleg_modify attribute to the server, the server MUST decide, for both of them, whether the time presented is:

  • before the corresponding time_access attribute or time_modify attribute on the file, or
  • past the current server time.

When the time presented is before the original time, the update is ignored 🙅‍♀️. When the time presented is in the future, the server can either clamp the new time to the current time or return NFS4ERR_DELAY to the client, letting it retry 🔁. Note that if the clock skew is large, the delay approach would result in access to the file being denied until the clock skew is exceeded 😬.

A change in the access time MUST NOT advance the change time, also known as the time_metadata attribute 🚫. However, a change in the modify time might advance the change time (and in turn, the change attribute) 💜. If the modify time is greater than the change time and before the current time, then the change time is adjusted to the modify time, and not the current time (as is most likely done on most SETATTR calls that change the metadata) 💅. If the modify time is in the future, it will be clamped to the current time 🧃.

Note that each of the possible times (access, modify, and change) is compared to the current time. They should all be compared against the same time value for the current time (i.e., they do not retrieve a different value of the current time for each calculation) ⭐.

If the client sets the OPEN4_SHARE_ACCESS_WANT_DELEG_TIMESTAMPS flag in an OPEN operation, then it MUST support the fattr4_time_deleg_access and fattr4_time_deleg_modify attributes in both the CB_GETATTR and SETATTR operations 🫶.

5.1. Use Case for NFSv3 Client Proxy

Let's say there's an NFSv3 client that wants to access data on a server that only supports NFSv4.2 🌸. An implementation can introduce an NFSv3 server that acts as an NFSv4.2 client, serving as a gateway between the two otherwise incompatible systems 🌉. NFSv3 is a stateless protocol, so the state isn't kept on the client, but rather on the NFSv3 server 🧠. Since the NFSv3 server is already managing the state, it can proxy file delegations to avoid spurious GETATTRs 🎯. That means that as the client queries the NFSv3 server for attributes, they can be served without the NFSv3 server sending a GETATTR to the NFSv4.2 server 🦋.

5.2. XDR for Proxying of Times

///
/// /*
///  * attributes for the delegation times being
///  * cached and served by the "client"
///  */
/// typedef nfstime4        fattr4_time_deleg_access;
/// typedef nfstime4        fattr4_time_deleg_modify;
///
///
/// %/*
/// % * New RECOMMENDED Attribute for
/// % * delegation caching of times
/// % */
/// const FATTR4_TIME_DELEG_ACCESS  = 84;
/// const FATTR4_TIME_DELEG_MODIFY  = 85;
///
///
/// const OPEN4_SHARE_ACCESS_WANT_DELEG_TIMESTAMPS = 0x100000;
///
/// enum open_delegation_type4 {
///        OPEN_DELEGATE_NONE                  = 0,
///        OPEN_DELEGATE_READ                  = 1,
///        OPEN_DELEGATE_WRITE                 = 2,
///        OPEN_DELEGATE_NONE_EXT              = 3, /* new to v4.1 */
///        OPEN_DELEGATE_READ_ATTRS_DELEG      = 4,
///        OPEN_DELEGATE_WRITE_ATTRS_DELEG     = 5
/// };

6. Extraction of XDR 🧩

This document contains the XDR [RFC4506] description of the new open flags for delegating the file to the client 💌. The XDR description is embedded in this document in a way that makes it simple for the reader to extract it into a ready-to-compile form 🛠️. The reader can feed this document into the following shell script to produce the machine-readable XDR description of the new flags:

#!/bin/sh
grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??'

So if the above script is stored in a file called "extract.sh" and this document is in a file called "spec.txt", then the reader can do the following:

sh extract.sh < spec.txt > delstid_prot.x

The effect of the script is to remove leading blank space from each line, plus a sentinel sequence of "///". XDR descriptions with the sentinel sequence are embedded throughout the document 🍓.

Note that the XDR code in this document depends on types from the NFSv4.2 nfs4_prot.x file (generated from [RFC7863]). This includes both nfs types that end with a 4 (such as offset4 and length4) as well as more generic types (such as uint32_t and uint64_t) 🌈.

While the XDR can be appended to that from [RFC7863], the various code snippets belong in their respective areas of that XDR 📎.

7. Security Considerations 🚨

Okay, this part is serious bestie 🥺. While this document extends some capabilities for client delegation, there are no new security concerns. The client cannot be queried by other clients as to the cached attributes. The client could report false data for the cached attributes, but it already has this ability via a SETATTR operation 🙈💀.

8. IANA Considerations 🗂️

This document has no IANA actions ✨.

9. Normative References 📚

Acknowledgments 💜

Trond Myklebust, Tom Haynes, and David Flynn all worked on the prototype at Hammerspace 🙌.

Dave Noveck, Chuck Lever, Rick Macklem, and Zaheduzzaman Sarker provided reviews of the document 👀.

Jeff Layton provided experience from an implementation he authored 🫶.

Authors' Addresses 🎀

Thomas Haynes Hammerspace Email: loghyr@hammerspace.com

Trond Myklebust Hammerspace Email: trondmy@hammerspace.com

📦 where this page comes from