RFC 9766
Extensions for Weak Cache Consistency in NFSv4.2's Flexible File Layout
RFC 9766: Extensions for Weak Cache Consistency in NFSv4.2's Flexible File Layout 🎀
okay bestie, the header facts first 💅 This is an IETF RFC (Request for Comments, the Internet's official protocol docs) numbered 9766. The authors are T. Haynes and T. Myklebust, both from Hammerspace. It's a Standards Track document, which means it's a real spec people implement, not just an opinion piece. The ISSN is 2070-1721 and it was published in April 2025. The header doesn't list anything this RFC obsoletes or updates, so there's nothing to chase there 🦋
Abstract ✨
okay so this doc adds extensions to NFSv4.2 to make Weak Cache Consistency (WCC) better 🔥 The big idea is adding mechanisms so partial writes made under a Parallel NFS (pNFS) layout stay coherent and get tracked properly. It's built for the drama that happens when several clients write to the same file through separate data servers 😭 By defining extra interactions between clients, metadata servers, and data servers, it makes NFSv4 more reliable in parallel-access setups and keeps things consistent across different deployment scenarios 💜
Status of This Memo 📜
This is an Internet Standards Track document. It's a product of the Internet Engineering Task Force (IETF), it represents the consensus of the IETF community, it got public review, and the Internet Engineering Steering Group (IESG) approved it for publication. More info on Internet Standards is in Section 2 of RFC 7841.
You can check the current status of this document, any errata, and how to send feedback at https://www.rfc-editor.org/info/rfc9766 🌈
Copyright Notice 🎀
Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as in effect on the date of publication. Please review those documents carefully, because they explain your rights and restrictions for this document. Code Components pulled out of this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Revised BSD License 📜
Table of Contents 🧚
1. Introduction
1.1. Definitions
1.2. Requirements Language
2. Weak Cache Consistency (WCC)
3. Operation 77: LAYOUT_WCC - Layout Weak Cache Consistency
3.1. ARGUMENT
3.2. RESULT
3.3. DESCRIPTION
3.4. Implementation
3.4.1. Examples of When to Use LAYOUT_WCC
3.4.2. Examples of What to Send in LAYOUT_WCC
3.5. Allowed Errors
3.6. Extension of Existing Implementations
3.7. Flexible File Layout Type
4. Extraction of XDR
5. Security Considerations
6. IANA Considerations
7. References
7.1. Normative References
7.2. Informative References
Acknowledgments
Authors' Addresses1. Introduction 🌸
so the Parallel NFS (pNFS) flexible file layout (see [RFC8435]) has a gap 👀 There's no mechanism for data servers to tell the metadata servers when the data part of the file gets modified. The metadata server needs that info so it can update the metadata part of the file to match. If the client talks to the data server over NFSv3, it can use Weak Cache Consistency (WCC) to tell the metadata server about attribute changes. In this doc we introduce a new NFSv4.2 operation called LAYOUT_WCC, which lets the client periodically report the attributes of the data files to the metadata server ✨
The revisions here become an extension of NFSv4.2 [RFC7862] by following the process in [RFC8178] 💅 They're built on top of the External Data Representation (XDR) [RFC4506] generated from [RFC7863].
1.1. Definitions 🦋
For a bigger set of definitions, check Section 1.1 of [RFC8435].
- (file) data: the part of the file system object that holds the data to be read or written. It's the contents of the object, not the attributes of the object.
- data server (DS): a pNFS server that provides the file's data when the file system object is accessed over a file-based protocol.
- (file) metadata: the part of the file system object that holds descriptive info about the file object, as opposed to the file data itself. This could include the time of last modification, access time, EOF position, etc.
- metadata server (MDS): the pNFS server that provides metadata info for a file system object.
- storage device: the target that clients can send I/O requests to when they hold an appropriate layout. Each data server is a storage device, but some storage device are not data servers. (See Section 2.1 of [RFC8434] for a discussion on the difference between a data server and a storage device.)
- weak cache consistency (WCC): the NFSv3 mechanism that lets the client check for file attribute changes before and after an operation (see Section 2.6 of [RFC1813]).
1.2. Requirements Language 🔑
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
2. Weak Cache Consistency (WCC) 💖
A pNFS layout type lets the metadata server tell the client two things: which storage protocol to use, and where the data lives that the client should use when talking to the storage devices 🧚 The flexible file layout type, from [RFC8435], explains how data servers using NFSv3 can be accessed. The client is restricted to these NFSv3 operations on the filehandles given in the layout: READ, WRITE, and COMMIT (see Sections 3.3.6, 3.3.7, and 3.3.21 of [RFC1813], respectively). In other words, the client may only use NFSv3 operations that act directly on the data portion of the file 🎀
Not every data server can support a control protocol (see [RFC8434]), so NFSv3 is used as the control protocol instead 💅 The metadata server commonly uses these NFSv3 operations: CREATE, GETATTR, and SETATTR (see Sections 3.3.8, 3.3.1, and 3.3.2 of [RFC1813], respectively). So the metadata server is only allowed to use NFSv3 operations that act directly on the metadata portion of the data file. GETATTR mostly lets the metadata server grab the mtime (modify time), ctime (change time), and atime (access time) ⏰ The metadata server can use that to figure out whether the client modified the file while it held an iomode of LAYOUTIOMODE4_RW (see Section 3.3.20 of [RFC8881]). Then it can work out the following for the metadata file: time_modify, time_metadata, and time_access (see Sections 5.8.2.43, 5.8.2.42, and 5.8.2.37 of [RFC8881], respectively). Basically, it can figure out what to return to clients in an NFSv4.2 GETATTR response 🙈
For example, the metadata server might send an NFSv3 GETATTR to the data server, usually because the client sent an NFSv4 GETATTR to the metadata server. On top of the cost of each GETATTR, the data server can get totally swamped by a huge pile of these requests 😭 NFSv3 dealt with a similar problem by putting a post-operation attribute in the READ and WRITE operations so they could report WCC data (see Section 2.6 of [RFC1813]).
Every NFSv3 operation takes one round trip between client and server. So a WRITE followed by a GETATTR would take two round trips 💀 In that case, the attribute info you get back counts as having strict server-client consistency. NFSv4 is different: it lets a WRITE and GETATTR be combined in one compound operation, which only needs one round trip ✨ That combined approach also counts as strict server-client consistency. Basically, NFSv4 READ and WRITE operations leave out post-operation attributes, so the client decides for itself whether it needs that info 💜
NFSv4 got rid of the need for the WRITE or READ operations to supply WCC info, but pNFS brings the same problem back 😩 The metadata server has to talk to the data server to get data that a WCC model could have handed over for free.
With the flexible file layout type, the client can use the NFSv3 WCC to handle the proxying of times (see Section 5 of [RFC9754]), but the granularity of that data is limited 🍓 With client-side mirroring (see Section 8 of [RFC8435]), the client has to combine the N mirrored files so it can send one piece of info instead of N pieces. On top of that, the client can only send that info when it returns the delegation 🥺
Here's the fix: this document adds a new NFSv4.2 operation, LAYOUT_WCC, which lets the client give the metadata server info it got from the data server 👑 The client collects the NFSv3 WCC data returned by the three permitted NFSv3 operations, then sends it back to the metadata server as part of the NFSv4.2 attributes. The metadata server MAY therefore skip the expensive NFSv3 GETATTR calls to the data servers. Because this approach relies on a weak model, the metadata server MAY still make those calls if it wants to strengthen the model 🦋
3. Operation 77: LAYOUT_WCC - Layout Weak Cache Consistency 🌈
3.1. ARGUMENT 🎀
<CODE BEGINS>
/// struct LAYOUT_WCC4args {
/// stateid4 lowa_stateid;
/// layouttype4 lowa_type;
/// opaque lowa_body<>;
/// };
<CODE ENDS>stateid4 is defined in Section 3.3.12 of [RFC8881]. layouttype4 is defined in Section 3.3.13 of [RFC8881].
3.2. RESULT 🍓
<CODE BEGINS>
/// struct LAYOUT_WCC4res {
/// nfsstat4 lowr_status;
/// };
<CODE ENDS>nfsstat4 is defined in Section 3.2 of [RFC8881].
3.3. DESCRIPTION 💜
The current filehandle and the lowa_stateid together identify the specific layout for the LAYOUT_WCC operation 🧚 The lowa_type tells you how to read the layout-type-specific payload in the lowa_body field. The lowa_type is the matching value from the "pNFS Layout Types" IANA registry for whichever layout type is in use.
The lowa_body carries the data file attributes. The client is responsible for mapping NFSv3 post-operation attributes into the fattr4 representation 💅 Just like post-operation attributes, the client can ignore these attributes, and the server can also choose to ignore any attributes included in LAYOUT_WCC. Still, the server can use them to avoid asking the data server for data file attributes 👀 These attributes are optional, and the client has no recourse if the server decides to ignore them. So there's no requirement to return a bitmap4 saying which attributes were accepted in the LAYOUT_WCC result ✨
3.4. Implementation 🌸
(continued in part 2 💫)
3.4.1. Examples of When to Use LAYOUT_WCC ✨
okay bestie, so the ONLY way the metadata server can tell the data file changed is by poking the data servers with a GETATTR 👀. If it compares the mtime results across multiple calls, it can spot an NFSv3 WRITE from the client. The atime results basically scream that the client did an NFSv3 READ 📖. So the client can totally use LAYOUT_WCC whenever it thinks the metadata server would need to refresh the data file attributes 💅. It can send LAYOUT_WCC at any time, but there are a few moments where it's especially iconic to do it, because it saves the metadata server from sending NFSv3 GETATTR requests to the data servers:
- Whenever it sends a GETATTR for any of the following attributes:
- Whenever it sends an
NFS4ERR_ACCESSerror viaLAYOUTRETURNorLAYOUTERROR. It could have already gotten the NFSv3 uid and gid values back in the WCC of the WRITE, READ, or COMMIT operation that got the error. So it can report that info back to the metadata server, which saves it from querying that info via an NFSv3 GETATTR 🙈. - Whenever it sends a SETATTR to refresh the proxied times (see Section 5 of [RFC9754]). The metadata server will correlate these times to detect later modification to the data file 🕵️♀️.
3.4.2. Examples of What to Send in LAYOUT_WCC 🎀
The NFSv3 attributes returned in the WCC of WRITE, READ, and COMMIT operations are a smaller subset of what can be transmitted as an NFSv4 attribute. The mapping of NFSv3 to NFSv4 attributes is in Table 1. The LAYOUT_WCC MUST provide all of these attributes to the metadata server, no exceptions 🫶. Both the uid and gid get stringified into their respective attributes, owner and owner_group. For NFS4ERR_ACCESS, the reason to provide these two attributes is that the metadata server can compare what it expects the uid and gid of the data file to be versus the actual values. Then it can repair the permissions as needed, or update the expected values it has cached.
+=================+===================+
| NFSv3 Attribute | NFSv4.2 Attribute |
+=================+===================+
| size | size |
+-----------------+-------------------+
| used | space_used |
+-----------------+-------------------+
| mode | mode |
+-----------------+-------------------+
| uid | owner |
+-----------------+-------------------+
| gid | owner_group |
+-----------------+-------------------+
| atime | time_access |
+-----------------+-------------------+
| mtime | time_modify |
+-----------------+-------------------+
| ctime | time_metadata |
+-----------------+-------------------+
Table 1: NFSv3 to NFSv4.2 Attribute
Mappings3.5. Allowed Errors 🚨
The LAYOUT_WCC operation can raise the errors listed in Table 2. When an error shows up, the metadata server can decide to ignore the entire operation. Or, depending on the layout-type-specific payload, it could decide to apply a portion of the payload. Note that there are no new errors introduced for LAYOUT_WCC, and each error in Table 2 is defined in Section 15.1 of [RFC8881]. Honestly, Table 2 can be thought of as an extension of Section 15.2 of [RFC8881] 💖.
+============+====================================================+
| Operation | Errors |
+============+====================================================+
| LAYOUT_WCC | NFS4ERR_ADMIN_REVOKED, NFS4ERR_BADXDR, |
| | NFS4ERR_BAD_STATEID, NFS4ERR_DEADSESSION, |
| | NFS4ERR_DELAY, NFS4ERR_DELEG_REVOKED, |
| | NFS4ERR_EXPIRED, NFS4ERR_FHEXPIRED, NFS4ERR_GRACE, |
| | NFS4ERR_INVAL, NFS4ERR_ISDIR, NFS4ERR_MOVED, |
| | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTSUPP, |
| | NFS4ERR_NO_GRACE, NFS4ERR_OLD_STATEID, |
| | NFS4ERR_OP_NOT_IN_SESSION, NFS4ERR_REP_TOO_BIG, |
| | NFS4ERR_REP_TOO_BIG_TO_CACHE, NFS4ERR_REQ_TOO_BIG, |
| | NFS4ERR_RETRY_UNCACHED_REP, NFS4ERR_SERVERFAULT, |
| | NFS4ERR_STALE, NFS4ERR_TOO_MANY_OPS, |
| | NFS4ERR_UNKNOWN_LAYOUTTYPE, NFS4ERR_WRONG_CRED, |
| | NFS4ERR_WRONG_TYPE |
+------------+----------------------------------------------------+
Table 2: Operations and Their Valid Errors3.6. Extension of Existing Implementations 🧩
The new LAYOUT_WCC operation is OPTIONAL for both NFSv4.2 [RFC7863] and the flexible file layout type [RFC8435]. So it's totally up to implementers, no pressure 🌸.
3.7. Flexible File Layout Type 🦋
/// struct ff_data_server_wcc4 {
/// deviceid4 ffdsw_deviceid;
/// stateid4 ffdsw_stateid;
/// nfs_fh4 ffdsw_fh_vers<>;
/// fattr4 ffdsw_attributes;
/// };
///
/// struct ff_mirror_wcc4 {
/// ff_data_server_wcc4 ffmw_data_servers<>;
/// };
///
/// struct ff_layout_wcc4 {
/// ff_mirror_wcc4 fflw_mirrors<>;
/// };The results specific to the flexible file layout type MUST correspond to the ff_layout4 data structure as defined in Section 5.1 of [RFC8435]. There MUST be a one-to-one correspondence between the following:
ff_data_server4 -> ff_data_server_wcc4ff_mirror4 -> ff_mirror_wcc4ff_layout4 -> ff_layout_wcc4
Each ff_layout4 has an array of ff_mirror4, and each of those has an array of ff_data_server4. Based on the current filehandle and the lowa_stateid, the server can match up the reported attributes 🔍.
But positional correspondence between the elements isn't enough to figure out which attributes to update. Picture this: a layout has three mirrors, and two of them have updated attributes but the third doesn't. A client could decide to present all three mirrors, with one mirror having an attribute mask with no attributes present. Or it could decide to present only the two mirrors that had been changed. Either way works 💅.
In either case, the combination of ffdsw_deviceid, ffdsw_stateid, and ffdsw_fh_vers will uniquely identify the attributes to be updated. All three arguments are required. A layout might have multiple data files on the same storage device, in which case ffdsw_deviceid and ffdsw_stateid would match, but ffdsw_fh_vers would not. Bestie, that's exactly why you need all three 🔥.
The ffdsw_attributes are processed similarly to the obj_attributes in the SETATTR arguments (see Section 18.30 of [RFC8881]).
4. Extraction of XDR 🌈
This document contains the XDR [RFC4506] description of the new NFSv4.2 operation LAYOUT_WCC. The XDR description is embedded in this document in a way that makes it simple for the reader to extract it into a ready-to-compile form. The reader can feed this document into the following shell script to produce the machine-readable XDR description of the new NFSv4.2 operation LAYOUT_WCC.
#!/bin/sh
grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??'So if the above script is stored in a file called 'extract.sh', and this document is in a file called 'spec.txt', then the reader can do:
sh extract.sh < spec.txt > layout_wcc.xThe effect of the script is to remove leading blank space from each line, plus a sentinel sequence of '///'. XDR descriptions with the sentinel sequence are embedded throughout the document ✨.
Note that the XDR code contained in this document depends on types from the NFSv4.2 nfs4_prot.x file (generated from [RFC7863]). This includes both nfs types that end with a 4 (such as offset4 and length4) as well as more generic types (such as uint32_t and uint64_t) 🧁.
While the XDR can be appended to the XDR from [RFC7863], the various code snippets belong in their respective areas of that XDR.
5. Security Considerations 🔐
There are no new security considerations beyond those in [RFC8435]. Okay, so this one isn't a drama queen moment, but it still matters, so don't skip it bestie 😭.
6. IANA Considerations 📋
This document has no IANA actions.
7. References 📚
7.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.
[RFC4506] Eisler, M., Ed., "XDR: External Data Representation Standard", STD 67, RFC 4506, DOI 10.17487/RFC4506, May 2006, <https://www.rfc-editor.org/info/rfc4506>.
[RFC7862] Haynes, T., "Network File System (NFS) Version 4 Minor Version 2 Protocol", RFC 7862, DOI 10.17487/RFC7862, November 2016, <https://www.rfc-editor.org/info/rfc7862>.
[RFC7863] Haynes, T., "Network File System (NFS) Version 4 Minor Version 2 External Data Representation Standard (XDR) Description", RFC 7863, DOI 10.17487/RFC7863, November 2016, <https://www.rfc-editor.org/info/rfc7863>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8178] Noveck, D., "Rules for NFSv4 Extensions and Minor Versions", RFC 8178, DOI 10.17487/RFC8178, July 2017, <https://www.rfc-editor.org/info/rfc8178>.
[RFC8434] Haynes, T., "Requirements for Parallel NFS (pNFS) Layout Types", RFC 8434, DOI 10.17487/RFC8434, August 2018, <https://www.rfc-editor.org/info/rfc8434>.
[RFC8435] Halevy, B. and T. Haynes, "Parallel NFS (pNFS) Flexible File Layout", RFC 8435, DOI 10.17487/RFC8435, August 2018, <https://www.rfc-editor.org/info/rfc8435>.
[RFC8881] Noveck, D., Ed. and C. Lever, "Network File System (NFS) Version 4 Minor Version 1 Protocol", RFC 8881, DOI 10.17487/RFC8881, August 2020, <https://www.rfc-editor.org/info/rfc8881>.
[RFC9754] Haynes, T. and T. Myklebust, "Extensions for Opening and Delegating Files in NFSv4.2", RFC 9754, DOI 10.17487/RFC9754, March 2025, <https://www.rfc-editor.org/info/rfc9754>.
7.2. Informative References
[RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS Version 3 Protocol Specification", RFC 1813, DOI 10.17487/RFC1813, June 1995, <https://www.rfc-editor.org/info/rfc1813>.
Acknowledgments 🫶
Dave Noveck, Tigran Mkrtchyan, and Rick Macklem provided reviews of the document. Thank you, legends 👑.
Authors' Addresses 💌
Thomas Haynes Hammerspace Email: loghyr@gmail.com
Trond Myklebust Hammerspace Email: trondmy@hammerspace.com
📦 where this page comes from
- ietf: rfc9766 2025-04
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms