RFC 8881

Network File System (NFS) Version 4 Minor Version 1 Protocol

📅 published August 2020, from ietf

RFC 8881: Network File System (NFS) Version 4 Minor Version 1 Protocol 🦋

okay bestie, so this is RFC 8881 💖 and it's the Network File System (NFS) version 4 minor version 1 Protocol. It came out in August 2020 as a Standards Track doc, which means it's the real deal, not a "just an idea" kind of thing ✨. The editor is D. Noveck from NetApp, and C. Lever from ORACLE helped write it 🫶. It's part of the IETF stream and its ISSN is 2070-1721 🌸. And the big tea is that it obsoletes RFC 5661 💅, so this one replaces that older version. No "updates" are listed on it, so nothing else gets changed by this one 🍓.

Abstract

okay so the abstract is giving us the whole plot 🎀. This doc describes NFS version 4 minor version 1, and it keeps the features from the base protocol (that's NFS version 4 minor version 0, which lives in RFC 7530) plus all the extensions that came after 🧚. The really iconic part is that the later minor version has no dependencies on minor version 0. So it's basically its own protocol, not a sequel that needs the original to work ✨.

It also says this doc obsoletes RFC 5661 and substantially revises how it handles the multi-server namespace stuff 💜. That means the new description of those features supersedes whatever RFC 5661 said about them. Main character energy, honestly 👑.

Status of This Memo

so this section is the paperwork part, but it's lowkey important 📜. This is an Internet Standards Track document 🔥. It's a product of the Internet Engineering Task Force (IETF), and it represents the consensus of the IETF community, so a lot of people agreed on it 🌈. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG) 💅.

If you want more info on Internet Standards, check Section 2 of RFC 7841 🦋. And if you want the current status, any errata, or a way to send feedback, go to https://www.rfc-editor.org/info/rfc8881 💖.

Copyright Notice

okay this part is the legal stuff, so read it carefully bestie, it's not a vibe to skip 🙈. The copyright is Copyright (c) 2020 IETF Trust and the persons identified as the document authors ✨, and all rights are reserved 🔒.

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, which you can find at https://trustee.ietf.org/license-info (as in effect on the date it was published) 📜. Please actually review those, because they explain your rights and restrictions with this document 🫶. Any Code Components pulled out of this doc must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in that same license 💀.

Heads up: some of this doc may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008 🚨. The people who control the copyright on some of that material may not have given the IETF Trust permission to allow modifications outside the IETF Standards Process. So unless you get an adequate license from those copyright holders, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process. The only exceptions are formatting it for publication as an RFC or translating it into languages other than English 🌸.

Okay that's the first part of the tea, bestie 🧚 more is coming in the next parts! 💜

okay bestie, here's the map of everything coming up 🗺️✨ it's a loooong list, but every single number and name is exactly as the spec has it, because implementers literally look these up by number 💅

Table of Contents

   1.  Introduction
     1.1.  Introduction to This Update
     1.2.  The NFS Version 4 Minor Version 1 Protocol
     1.3.  Requirements Language
     1.4.  Scope of This Document
     1.5.  NFSv4 Goals
     1.6.  NFSv4.1 Goals
     1.7.  General Definitions
     1.8.  Overview of NFSv4.1 Features
     1.9.  Differences from NFSv4.0
   2.  Core Infrastructure
     2.1.  Introduction
     2.2.  RPC and XDR
     2.3.  COMPOUND and CB_COMPOUND
     2.4.  Client Identifiers and Client Owners
     2.5.  Server Owners
     2.6.  Security Service Negotiation
     2.7.  Minor Versioning
     2.8.  Non-RPC-Based Security Services
     2.9.  Transport Layers
     2.10. Session
   3.  Protocol Constants and Data Types
     3.1.  Basic Constants
     3.2.  Basic Data Types
     3.3.  Structured Data Types
   4.  Filehandles
     4.1.  Obtaining the First Filehandle
     4.2.  Filehandle Types
     4.3.  One Method of Constructing a Volatile Filehandle
     4.4.  Client Recovery from Filehandle Expiration
   5.  File Attributes
     5.1.  REQUIRED Attributes
     5.2.  RECOMMENDED Attributes
     5.3.  Named Attributes
     5.4.  Classification of Attributes
     5.5.  Set-Only and Get-Only Attributes
     5.6.  REQUIRED Attributes - List and Definition References
     5.7.  RECOMMENDED Attributes - List and Definition References
     5.8.  Attribute Definitions
     5.9.  Interpreting owner and owner_group
     5.10. Character Case Attributes
     5.11. Directory Notification Attributes
     5.12. pNFS Attribute Definitions
     5.13. Retention Attributes
   6.  Access Control Attributes
     6.1.  Goals
     6.2.  File Attributes Discussion
     6.3.  Common Methods
     6.4.  Requirements
   7.  Single-Server Namespace
     7.1.  Server Exports
     7.2.  Browsing Exports
     7.3.  Server Pseudo File System
     7.4.  Multiple Roots
     7.5.  Filehandle Volatility
     7.6.  Exported Root
     7.7.  Mount Point Crossing
     7.8.  Security Policy and Namespace Presentation
   8.  State Management
     8.1.  Client and Session ID
     8.2.  Stateid Definition
     8.3.  Lease Renewal
     8.4.  Crash Recovery
     8.5.  Server Revocation of Locks
     8.6.  Short and Long Leases
     8.7.  Clocks, Propagation Delay, and Calculating Lease Expiration
     8.8.  Obsolete Locking Infrastructure from NFSv4.0
   9.  File Locking and Share Reservations
     9.1.  Opens and Byte-Range Locks
     9.2.  Lock Ranges
     9.3.  Upgrading and Downgrading Locks
     9.4.  Stateid Seqid Values and Byte-Range Locks
     9.5.  Issues with Multiple Open-Owners
     9.6.  Blocking Locks
     9.7.  Share Reservations
     9.8.  OPEN/CLOSE Operations
     9.9.  Open Upgrade and Downgrade
     9.10. Parallel OPENs
     9.11. Reclaim of Open and Byte-Range Locks
   10. Client-Side Caching
     10.1.  Performance Challenges for Client-Side Caching
     10.2.  Delegation and Callbacks
     10.3.  Data Caching
     10.4.  Open Delegation
     10.5.  Data Caching and Revocation
     10.6.  Attribute Caching
     10.7.  Data and Metadata Caching and Memory Mapped Files
     10.8.  Name and Directory Caching without Directory Delegations
     10.9.  Directory Delegations
   11. Multi-Server Namespace
     11.1.  Terminology
     11.2.  File System Location Attributes
     11.3.  File System Presence or Absence
     11.4.  Getting Attributes for an Absent File System
     11.5.  Uses of File System Location Information
     11.6.  Trunking without File System Location Information
     11.7.  Users and Groups in a Multi-Server Namespace
     11.8.  Additional Client-Side Considerations
     11.9.  Overview of File Access Transitions
     11.10. Effecting Network Endpoint Transitions
     11.11. Effecting File System Transitions
     11.12. Transferring State upon Migration
     11.13. Client Responsibilities When Access Is Transitioned
     11.14. Server Responsibilities Upon Migration
     11.15. Effecting File System Referrals
     11.16. The Attribute fs_locations
     11.17. The Attribute fs_locations_info
     11.18. The Attribute fs_status
   12. Parallel NFS (pNFS)
     12.1.  Introduction
     12.2.  pNFS Definitions
     12.3.  pNFS Operations
     12.4.  pNFS Attributes
     12.5.  Layout Semantics
     12.6.  pNFS Mechanics
     12.7.  Recovery
     12.8.  Metadata and Storage Device Roles
     12.9.  Security Considerations for pNFS
   13. NFSv4.1 as a Storage Protocol in pNFS: the File Layout Type
     13.1.  Client ID and Session Considerations
     13.2.  File Layout Definitions
     13.3.  File Layout Data Types
     13.4.  Interpreting the File Layout
     13.5.  Data Server Multipathing
     13.6.  Operations Sent to NFSv4.1 Data Servers
     13.7.  COMMIT through Metadata Server
     13.8.  The Layout Iomode
     13.9.  Metadata and Data Server State Coordination
     13.10. Data Server Component File Size
     13.11. Layout Revocation and Fencing
     13.12. Security Considerations for the File Layout Type
   14. Internationalization
     14.1.  Stringprep Profile for the utf8str_cs Type
     14.2.  Stringprep Profile for the utf8str_cis Type
     14.3.  Stringprep Profile for the utf8str_mixed Type
     14.4.  UTF-8 Capabilities
     14.5.  UTF-8 Related Errors
   15. Error Values
     15.1.  Error Definitions
     15.2.  Operations and Their Valid Errors
     15.3.  Callback Operations and Their Valid Errors
     15.4.  Errors and the Operations That Use Them
   16. NFSv4.1 Procedures
     16.1.  Procedure 0: NULL - No Operation
     16.2.  Procedure 1: COMPOUND - Compound Operations
   17. Operations: REQUIRED, RECOMMENDED, or OPTIONAL
   18. NFSv4.1 Operations
     18.1.  Operation 3: ACCESS - Check Access Rights
     18.2.  Operation 4: CLOSE - Close File
     18.3.  Operation 5: COMMIT - Commit Cached Data
     18.4.  Operation 6: CREATE - Create a Non-Regular File Object
     18.5.  Operation 7: DELEGPURGE - Purge Delegations Awaiting
             Recovery
     18.6.  Operation 8: DELEGRETURN - Return Delegation
     18.7.  Operation 9: GETATTR - Get Attributes
     18.8.  Operation 10: GETFH - Get Current Filehandle
     18.9.  Operation 11: LINK - Create Link to a File
     18.10. Operation 12: LOCK - Create Lock
     18.11. Operation 13: LOCKT - Test for Lock
     18.12. Operation 14: LOCKU - Unlock File
     18.13. Operation 15: LOOKUP - Lookup Filename
     18.14. Operation 16: LOOKUPP - Lookup Parent Directory
     18.15. Operation 17: NVERIFY - Verify Difference in Attributes
     18.16. Operation 18: OPEN - Open a Regular File
     18.17. Operation 19: OPENATTR - Open Named Attribute Directory
     18.18. Operation 21: OPEN_DOWNGRADE - Reduce Open File Access
     18.19. Operation 22: PUTFH - Set Current Filehandle
     18.20. Operation 23: PUTPUBFH - Set Public Filehandle
     18.21. Operation 24: PUTROOTFH - Set Root Filehandle
     18.22. Operation 25: READ - Read from File
     18.23. Operation 26: READDIR - Read Directory
     18.24. Operation 27: READLINK - Read Symbolic Link
     18.25. Operation 28: REMOVE - Remove File System Object
     18.26. Operation 29: RENAME - Rename Directory Entry
     18.27. Operation 31: RESTOREFH - Restore Saved Filehandle
     18.28. Operation 32: SAVEFH - Save Current Filehandle
     18.29. Operation 33: SECINFO - Obtain Available Security
     18.30. Operation 34: SETATTR - Set Attributes
     18.31. Operation 37: VERIFY - Verify Same Attributes
     18.32. Operation 38: WRITE - Write to File
     18.33. Operation 40: BACKCHANNEL_CTL - Backchannel Control
     18.34. Operation 41: BIND_CONN_TO_SESSION - Associate Connection
             with Session
     18.35. Operation 42: EXCHANGE_ID - Instantiate Client ID
     18.36. Operation 43: CREATE_SESSION - Create New Session and
             Confirm Client ID
     18.37. Operation 44: DESTROY_SESSION - Destroy a Session
     18.38. Operation 45: FREE_STATEID - Free Stateid with No Locks
     18.39. Operation 46: GET_DIR_DELEGATION - Get a Directory
             Delegation
     18.40. Operation 47: GETDEVICEINFO - Get Device Information
     18.41. Operation 48: GETDEVICELIST - Get All Device Mappings for
             a File System
     18.42. Operation 49: LAYOUTCOMMIT - Commit Writes Made Using a
             Layout
     18.43. Operation 50: LAYOUTGET - Get Layout Information
     18.44. Operation 51: LAYOUTRETURN - Release Layout Information
     18.45. Operation 52: SECINFO_NO_NAME - Get Security on Unnamed
             Object
     18.46. Operation 53: SEQUENCE - Supply Per-Procedure Sequencing
             and Control
     18.47. Operation 54: SET_SSV - Update SSV for a Client ID
     18.48. Operation 55: TEST_STATEID - Test Stateids for Validity
     18.49. Operation 56: WANT_DELEGATION - Request Delegation
     18.50. Operation 57: DESTROY_CLIENTID - Destroy a Client ID
     18.51. Operation 58: RECLAIM_COMPLETE - Indicates Reclaims
             Finished
     18.52. Operation 10044: ILLEGAL - Illegal Operation
   19. NFSv4.1 Callback Procedures
     19.1.  Procedure 0: CB_NULL - No Operation
     19.2.  Procedure 1: CB_COMPOUND - Compound Operations
   20. NFSv4.1 Callback Operations
     20.1.  Operation 3: CB_GETATTR - Get Attributes
     20.2.  Operation 4: CB_RECALL - Recall a Delegation
     20.3.  Operation 5: CB_LAYOUTRECALL - Recall Layout from Client
     20.4.  Operation 6: CB_NOTIFY - Notify Client of Directory
             Changes
     20.5.  Operation 7: CB_PUSH_DELEG - Offer Previously Requested
             Delegation to Client
     20.6.  Operation 8: CB_RECALL_ANY - Keep Any N Recallable Objects
     20.7.  Operation 9: CB_RECALLABLE_OBJ_AVAIL - Signal Resources
             for Recallable Objects
     20.8.  Operation 10: CB_RECALL_SLOT - Change Flow Control Limits
     20.9.  Operation 11: CB_SEQUENCE - Supply Backchannel Sequencing
             and Control
     20.10. Operation 12: CB_WANTS_CANCELLED - Cancel Pending
             Delegation Wants
     20.11. Operation 13: CB_NOTIFY_LOCK - Notify Client of Possible
             Lock Availability
     20.12. Operation 14: CB_NOTIFY_DEVICEID - Notify Client of Device
             ID Changes
     20.13. Operation 10044: CB_ILLEGAL - Illegal Callback Operation
   21. Security Considerations
   22. IANA Considerations
     22.1.  IANA Actions
     22.2.  Named Attribute Definitions
     22.3.  Device ID Notifications
     22.4.  Object Recall Types
     22.5.  Layout Types
     22.6.  Path Variable Definitions
   23. References
     23.1.  Normative References
     23.2.  Informative References
   Appendix A.  The Need for This Update
   Appendix B.  Changes in This Update
     B.1.  Revisions Made to Section 11 of RFC 5661
     B.2.  Revisions Made to Operations in RFC 5661
     B.3.  Revisions Made to Error Definitions in RFC 5661
     B.4.  Other Revisions Made to RFC 5661
   Appendix C.  Security Issues That Need to Be Addressed
   Acknowledgments
   Authors' Addresses

1. Introduction ✨

1.1. Introduction to This Update

Okay so bestie, there are two big features that were introduced in minor version 0 but never fully finished in minor version 1 🎀. The first is trunking, which means using multiple connections between a client and server at the same time, possibly to different network addresses 🔥. The second is Transparent State Migration, which lets a file system move between servers while the client keeps its existing locking state across the transfer 🦋.

The revised NFSv4.1 description in this update is needed to fully use these features together with other multi-server namespace features ✨. This document is an updated description of the NFSv4.1 protocol that was previously defined in RFC 5661 [66]. RFC 5661 is obsoleted by this document 💀. The update has a limited scope, though, and it's focused on enabling full use of trunking and Transparent State Migration 💅. Appendix A explains why these changes are needed, and Appendix B walks through the specific changes that got us to the current text 🌸.

This limited-scope update replaces the current NFSv4.1 RFC. The goal is an authoritative and complete specification, and the motivation for that is discussed in [36], but only for the issues that fall within this update's scope 🧚. It won't address issues that are known but outside that limited scope, which you'd normally expect from a full update of the protocol 🙈. Here are some areas that are known to need work in a future update:

Work needs to be done with regard to RFC 8178 [67], which establishes NFSv4-wide versioning rules. Because RFC 5661 is currently inconsistent with that document, changes are needed so that RFC 8178 would no longer need to update the NFSv4.1 specification 🌈.
Work needs to be done with regard to RFC 8434 [70], which establishes the requirements for parallel NFS (pNFS) layout types. Those requirements aren't clearly defined in RFC 5661 😭. Once that work is done and the resulting documents are approved, the new NFSv4.1 specification will provide a clear set of requirements for layout types, along with a description of the file layout type that conforms to them. Other layout types will get their own specification documents that conform to those requirements too 👑.
Work needs to be done to address many errata reports relevant to RFC 5661, other than errata report 2006 [64], which is addressed in this document. Addressing that report couldn't be put off, because the changes suggested there interact with the newly described handling of state and session migration 🔥.

The errata reports that have been deferred, and that will need to be addressed in a later document, include reports with a range of statuses in the errata reporting system. That includes reports marked Accepted, and reports marked Hold For Document Update because the change was too minor to address right away ✨.

There's also a set of other reports, including at least one in state Rejected, that will need to be addressed in a later document 🫶. This will mean changing consensus decisions reflected in RFC 5661, in situations where the working group decided that RFC 5661's treatment is incorrect and needs revising. The goal is to reflect the working group's new consensus and to keep compatibility with existing implementations that don't follow the handling described in RFC 5661 💜.

Note that all of these errata reports are expected to stay relevant to implementors and to the authors of an eventual rfc5661bis, even though this document obsoletes RFC 5661 [66] 🍓.

There is a need for a new approach to describing internationalization, since the current internationalization section (Section 14) has never been implemented and doesn't meet the needs of the NFSv4 protocol 🙈. Possible solutions are to create a new internationalization section modeled on the one in [68], or to create a new document describing internationalization for all NFSv4 minor versions. That new document would then be referenced from the RFCs defining both NFSv4.0 and NFSv4.1 💖.
There is a need for a revised treatment of security in NFSv4.1 🔐. The issues with the existing treatment are discussed in Appendix C 😬.

Until the above work is done, there won't be a consistent set of documents describing the NFSv4.1 protocol, and any full description would involve documents updating other documents within the specification 💀. The updates applied by RFC 8434 [70] and RFC 8178 [67] to RFC 5661 also apply to this specification, and they'll keep applying to any later v4.1 specification until that work is done 🦋.

1.2. The NFS Version 4 Minor Version 1 Protocol

The NFS version 4 minor version 1 (NFSv4.1) protocol is the second minor version of the NFS version 4 (NFSv4) protocol 🌸. The first minor version, NFSv4.0, is now described in RFC 7530 [68]. NFSv4.1 generally follows the minor versioning guidelines listed in Section 10 of RFC 3530 [37] 📜. But it diverges from guideline 11 ("a client and server that support minor version X must support minor versions 0 through X-1") and guideline 12 ("no new features may be introduced as mandatory in a minor version") 🙃.

These divergences happen because of two things: the sessions model for managing non-idempotent operations, and the RECLAIM_COMPLETE operation 🎀. Both of these new features are infrastructural in nature, and they simplify the implementation of existing and other new features ✨. Making them anything but REQUIRED would add undue complexity to the protocol definition and implementation 😤. So NFSv4.1 accordingly updates the minor versioning guidelines (Section 2.7) 💅.

As a minor version, NFSv4.1 stays consistent with the overall goals for NFSv4. It also extends the protocol to better meet those goals, based on experience with NFSv4.0 🧁. On top of that, NFSv4.1 has adopted some additional goals, and those goals motivate some of the major extensions in NFSv4.1 👑.

1.3. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [1] 📖💜.

1.4. Scope of This Document

This document describes the NFSv4.1 protocol 🦋. With respect to NFSv4.0, this document does not:

describe the NFSv4.0 protocol, except where needed to contrast with NFSv4.1.
modify the specification of the NFSv4.0 protocol.
clarify the NFSv4.0 protocol.

1.5. NFSv4 Goals

The NFSv4 protocol is a further revision of the NFS protocol already defined by NFSv3 [38] 🌈. It keeps the essential characteristics of previous versions: easy recovery, independence of transport protocols, operating systems and file systems, simplicity, and good performance ✨. NFSv4 has the following goals:

Improved access and good performance on the Internet 🌐

The protocol is designed to transit firewalls easily, perform well where latency is high and bandwidth is low, and scale to very large numbers of clients per server 🚀.

Strong security with negotiation built into the protocol 🔐

The protocol builds on the work of the ONCRPC working group in supporting the RPCSEC_GSS protocol. On top of that, the NFSv4.1 protocol provides a mechanism that lets clients and servers negotiate security, and it requires clients and servers to support a minimal set of security schemes 🫶.

Good cross-platform interoperability 🧚

The protocol features a file system model that provides a useful, common set of features. It doesn't unduly favor one file system or operating system over another 🍓.

Designed for protocol extensions 🧩

The protocol is designed to accept standard extensions within a framework that enables and encourages backward compatibility 💖.

1.6. NFSv4.1 Goals

NFSv4.1 has the following goals, within the framework set by the overall NFSv4 goals 👑.

To correct significant structural weaknesses and oversights discovered in the base protocol 🛠️.
To add clarity and specificity to areas that the base protocol left unaddressed or didn't address in enough detail. However, as stated in Section 1.4, it is not a goal to clarify the NFSv4.0 protocol in the NFSv4.1 specification 🙈.
To add specific features based on experience with the existing protocol and recent industry developments ✨.
To provide protocol support for taking advantage of clustered server deployments, including the ability to provide scalable parallel access to files distributed among multiple servers 🌸.

1.7. General Definitions 💅

okay so before we get into the good stuff, let's make sure we're all speaking the same language 🫶 These definitions give the reader the context they need.

Byte: in this doc, a byte is an octet, aka a datum that's exactly 8 bits long 🔢

Client: the client is the entity that accesses the NFS server's resources 💻 It can be an app with the logic to hit the NFS server directly, OR it can be the classic operating system client that provides remote file system services for a whole bunch of apps ✨

A client is uniquely identified by a client owner 🪪

For byte-range locking specifically, the client is also the entity keeping a set of locks on behalf of one or more apps 🔐 It's also responsible for crash or failure recovery for the locks it manages 🛟

Note that multiple clients can share the same transport and connection, and multiple clients can exist on the same network node 🌸

Client ID: this is a 64-bit quantity used as a unique, short-hand reference to a client-supplied verifier and client owner 🎀 The server is the one responsible for supplying it 👑

Client Owner: this is a unique string, opaque to the server, that identifies a client 🦋 Multiple network connections (and the source network addresses coming from those connections) can share a client owner. The server is expected to treat requests from connections with the same client owner as coming from the same client 💜

File System: the file system is the collection of objects on a server that share the same fsid attribute (see Section 5.8.1.9) 🗂️ The server is identified by the major identifier of a server owner, which gets defined later in this section 👀

Lease: a lease is an interval of time the server defines for which the client is irrevocably granted locks ⏳ When a lease period ends, locks can be revoked if the lease hasn't been extended 💀 And a lock must be revoked if a conflicting lock got granted after the lease interval 🙈

A server grants a client a single lease for all state 🫧

Lock: the term "lock" refers to byte-range locks (also known as record locks in UNIX environments), share reservations, delegations, or layouts, unless it's specifically stated otherwise 🔒

Secret State Verifier (SSV): the SSV is a unique secret key shared between a client and a server 🤫 It serves as the secret key for an internal GSS mechanism (internal to NFSv4.1), called the SSV GSS mechanism (see Section 2.10.9). That mechanism uses the SSV to compute message integrity codes (MIC) and Wrap tokens 🧁 For more on how NFSv4.1 uses the SSV and the SSV GSS mechanism, see Section 2.10.8.3 ✨

Server: the Server is the entity responsible for coordinating client access to a set of file systems, and it's identified by a server owner 🏰 A server can span multiple network addresses 🌈

Server Owner: the server owner identifies the server to the client 🪞 It's made up of a major identifier and a minor identifier. When the client has two connections, each going to a peer with the same major identifier, the client assumes both peers are the same server (the server namespace is the same via each connection) and that lock state is shareable across both connections 🤝 When each peer has both the same major AND minor identifiers, the client assumes each connection might be associable with the same session 🧚

Stable Storage: stable storage is storage that data stored by an NFSv4.1 server can be recovered from without data loss, even through multiple power failures 🔋 That includes cascading power failures (several power failures in quick succession), operating system failures, and/or hardware failure of components other than the storage medium itself (such as disk, nonvolatile RAM, flash memory, etc.) 😭

Some examples of stable storage that are allowable for an NFS server include:

Media commit of data: the modified data has been successfully written to the disk media, for example, the disk platter 💿
An immediate reply disk drive with battery-backed, on-drive intermediate storage or uninterruptible power system (UPS) 🔌
Server commit of data with battery-backed intermediate storage and recovery software 🪫
Cache commit with uninterruptible power system (UPS) and recovery software 🛡️

Stateid: a stateid is a 128-bit quantity returned by a server that uniquely defines the open and locking states the server provides for a specific open-owner or lock-owner/open-owner pair, for a specific file and type of lock 🎟️

Verifier: a verifier is a 64-bit quantity generated by the client that the server can use to figure out whether the client has restarted and lost all previous lock state 🔁

1.8. Overview of NFSv4.1 Features ✨

okay now for the tour 🗺️ The major features of the NFSv4.1 protocol get reviewed in brief, so that both the reader who already knows the earlier NFS versions AND the reader who's brand new to NFS have the context they need 💖 If you're new to NFS, there's still some fundamental knowledge you're expected to have: the reader should be familiar with the External Data Representation (XDR) and Remote Procedure Call (RPC) protocols described in [2] and [3] 📚 A basic understanding of file systems and distributed file systems is expected too 🌷

In general, this spec won't distinguish the features added in minor version 1 from the ones in the base protocol. It treats NFSv4.1 as a unified whole 🫶 For a summary of the differences between NFSv4.0 and NFSv4.1, see Section 1.9 🦋

1.8.1. RPC and Security 🔐

Same as previous NFS versions, the XDR and RPC mechanisms used for NFSv4.1 are the ones defined in [2] and [3] 🧩 To meet end-to-end security requirements, the RPCSEC_GSS framework [4] extends the basic RPC security 🛡️ With RPCSEC_GSS, various mechanisms can offer authentication, integrity, and privacy to the NFSv4 protocol 💜 Kerberos V5 is used as described in [5] to provide one security framework 👑 With RPCSEC_GSS, other mechanisms may also be specified and used for NFSv4.1 security ✨

To enable in-band security negotiation, NFSv4.1 has operations that let the client ask the server about its policies on which security mechanisms must be used to access the server's file system resources 🕵️‍♀️ That way, the client can securely match a security mechanism that satisfies the policies at both the client and the server 🎀

NFSv4.1 introduces parallel access (see Section 1.8.2.2), which is called pNFS 🚀 The security framework described in this section gets significantly modified by pNFS (see Section 12.9), because data access sometimes isn't over RPC 😳 How big that impact is depends on the storage protocol (see Section 12.2.5), and it can be as low as zero impact (see Section 13.12) 🫧

1.8.2. Protocol Structure 🧱

1.8.2.1. Core Protocol 🌸

Unlike NFSv3, which used a series of ancillary protocols (e.g., NLM, NSM (Network Status Monitor), MOUNT), within all minor versions of NFSv4 a single RPC protocol gets used to make requests to the server 🙌 Things that used to be separate protocols, like locking, are now integrated into one unified protocol 💅

1.8.2.2. Parallel Access ⚡

Minor version 1 supports high-performance data access to a clustered server implementation 🏎️ It does this by separating metadata access from data access, and the data access gets done to multiple servers in parallel 🧚

This parallel data access is controlled by recallable objects known as "layouts" 🍓 Layouts are integrated into the protocol locking model 🔗 Clients direct data access requests to a set of data servers specified by the layout, via a data storage protocol 📡 That data storage protocol may be NFSv4.1, or it may be another protocol entirely 🌈

Because the protocols used for parallel data access aren't necessarily RPC-based, the RPC-based security model (Section 1.8.1) is obviously impacted (see Section 12.9) 😭 How much it's impacted depends on the storage protocol used for data access (see Section 12.2.5), and it can be as low as zero (see Section 13.12) 🫶

1.8.3. File System Model 🗂️

The general file system model for NFSv4.1 is the same as in previous versions 💖 The server file system is hierarchical, and the regular files inside it are treated as opaque byte streams 🫙 In a slight departure, file and directory names are encoded with UTF-8, to deal with the basics of internationalization 🌍

NFSv4.1 doesn't need a separate protocol to handle the initial mapping between path name and filehandle 🙅‍♀️ All file systems exported by a server are presented as a tree, so every file system is reachable from a special per-server global root filehandle 🌳 That means LOOKUP operations can do what the MOUNT protocol used to do 🎉 The server provides whatever pseudo file systems are needed to bridge any gaps caused by unexported space between exported file systems 🌉

1.8.3.1. Filehandles 🎟️

Just like in previous NFS versions, opaque filehandles are used to identify individual files and directories 🔍 Lookup-type and create operations translate file and directory names into filehandles, and those filehandles then identify objects in subsequent operations 🪄

NFSv4.1 supports persistent filehandles, which are guaranteed to stay valid for the lifetime of the file system object they designate 💎 It also gives servers the option of providing filehandles with more limited validity guarantees, called volatile filehandles 🫠

1.8.3.2. File Attributes

The NFSv4.1 protocol has a rich and extensible file object attribute structure, which is divided into REQUIRED, RECOMMENDED, and named attributes (see Section 5).

Several (but not all) of the REQUIRED attributes are derived from the attributes of NFSv3 (see the definition of the fattr3 data type in [38]). An example of a REQUIRED attribute is the file object's type (Section 5.8.1.2) so that regular files can be distinguished from directories (also known as folders in some operating environments) and other types of objects. REQUIRED attributes are discussed in Section 5.1.

An example of three RECOMMENDED attributes are acl, sacl, and dacl. These attributes define an Access Control List (ACL) on a file object (Section 6). An ACL provides directory and file access control beyond the model used in NFSv3. The ACL definition allows for specification of specific sets of permissions for individual users and groups. In addition, ACL inheritance allows propagation of access permissions and restrictions down a directory tree as file system objects are created. RECOMMENDED attributes are discussed in Section 5.2.

A named attribute is an opaque byte stream that is associated with a directory or file and referred to by a string name. Named attributes are meant to be used by client applications as a method to associate application-specific data with a regular file or directory. NFSv4.1 modifies named attributes relative to NFSv4.0 by tightening the allowed operations in order to prevent the development of non- interoperable implementations. Named attributes are discussed in Section 5.3.

1.8.3.3. Multi-Server Namespace ✨

NFSv4.1 contains a number of features to allow implementation of namespaces that cross server boundaries and that allow and facilitate a nondisruptive transfer of support for individual file systems between servers. They are all based upon attributes that allow one file system to specify alternate, additional, and new location information that specifies how the client may access that file system.

These attributes can be used to provide for individual active file systems:

Alternate network addresses to access the current file system instance.
The locations of alternate file system instances or replicas to be used in the event that the current file system instance becomes unavailable.

These file system location attributes may be used together with the concept of absent file systems, in which a position in the server namespace is associated with locations on other servers without there being any corresponding file system instance on the current server. For example,

These attributes may be used with absent file systems to implement referrals whereby one server may direct the client to a file system provided by another server. This allows extensive multi- server namespaces to be constructed.
These attributes may be provided when a previously present file system becomes absent. This allows nondisruptive migration of file systems to alternate servers.

1.8.4. Locking Facilities 🔒

As mentioned previously, NFSv4.1 is a single protocol that includes locking facilities. These locking facilities include support for many types of locks including a number of sorts of recallable locks. Recallable locks such as delegations allow the client to be assured that certain events will not occur so long as that lock is held. When circumstances change, the lock is recalled via a callback request. The assurances provided by delegations allow more extensive caching to be done safely when circumstances allow it.

The types of locks are:

Share reservations as established by OPEN operations.
Byte-range locks.
File delegations, which are recallable locks that assure the holder that inconsistent opens and file changes cannot occur so long as the delegation is held.
Directory delegations, which are recallable locks that assure the holder that inconsistent directory modifications cannot occur so long as the delegation is held.
Layouts, which are recallable objects that assure the holder that direct access to the file data may be performed directly by the client and that no change to the data's location that is inconsistent with that access may be made so long as the layout is held.

All locks for a given client are tied together under a single client- wide lease. All requests made on sessions associated with the client renew that lease. When the client's lease is not promptly renewed, the client's locks are subject to revocation. In the event of server restart, clients have the opportunity to safely reclaim their locks within a special grace period.

1.9. Differences from NFSv4.0 🌸

The following summarizes the major differences between minor version 1 and the base protocol:

Implementation of the sessions model (Section 2.10).
Parallel access to data (Section 12).
Addition of the RECLAIM_COMPLETE operation to better structure the lock reclamation process (Section 18.51).
Enhanced delegation support as follows.
- Delegations on directories and other file types in addition to regular files (Section 18.39, Section 18.49).
- Operations to optimize acquisition of recalled or denied delegations (Section 18.49, Section 20.5, Section 20.7).
- Notifications of changes to files and directories (Section 18.39, Section 20.4).
- A method to allow a server to indicate that it is recalling one or more delegations for resource management reasons, and thus a method to allow the client to pick which delegations to return (Section 20.6).
Attributes can be set atomically during exclusive file create via the OPEN operation (see the new EXCLUSIVE4_1 creation method in Section 18.16).
Open files can be preserved if removed and the hard link count ("hard link" is defined in an Open Group [6] standard) goes to zero, thus obviating the need for clients to rename deleted files to partially hidden names -- colloquially called "silly rename" (see the new OPEN4_RESULT_PRESERVE_UNLINKED reply flag in Section 18.16).
Improved compatibility with Microsoft Windows for Access Control Lists (Section 6.2.3, Section 6.2.2, Section 6.4.3.2).
Data retention (Section 5.13).
Identification of the implementation of the NFS client and server (Section 18.35).
Support for notification of the availability of byte-range locks (see the new OPEN4_RESULT_MAY_NOTIFY_LOCK reply flag in Section 18.16 and see Section 20.11).
In NFSv4.1, LIPKEY and SPKM-3 are not required security mechanisms [39].

2. Core Infrastructure 💖

2.1. Introduction

NFSv4.1 relies on core infrastructure common to nearly every operation. This core infrastructure is described in the remainder of this section.

2.2. RPC and XDR

The NFSv4.1 protocol is a Remote Procedure Call (RPC) application that uses RPC version 2 and the corresponding eXternal Data Representation (XDR) as defined in [3] and [2].

2.2.1. RPC-Based Security 🔐

Previous NFS versions have been thought of as having a host-based authentication model, where the NFS server authenticates the NFS client, and trusts the client to authenticate all users. Actually, NFS has always depended on RPC for authentication. One of the first forms of RPC authentication, AUTH_SYS, had no strong authentication and required a host-based authentication approach. NFSv4.1 also depends on RPC for basic security services and mandates RPC support for a user-based authentication model. The user-based authentication model has user principals authenticated by a server, and in turn the server authenticated by user principals. RPC provides some basic security services that are used by NFSv4.1.

2.2.1.1. RPC Security Flavors

As described in "Authentication", Section 7 of [3], RPC security is encapsulated in the RPC header, via a security or authentication flavor, and information specific to the specified security flavor. Every RPC header conveys information used to identify and authenticate a client and server. As discussed in Section 2.2.1.1.1, some security flavors provide additional security services.

NFSv4.1 clients and servers MUST implement RPCSEC_GSS. (This requirement to implement is not a requirement to use.) Other flavors, such as AUTH_NONE and AUTH_SYS, MAY be implemented as well.

2.2.1.1.1. RPCSEC_GSS and Security Services

RPCSEC_GSS [4] uses the functionality of GSS-API [7]. This allows for the use of various security mechanisms by the RPC layer without the additional implementation overhead of adding RPC security flavors.

2.2.1.1.1.1. Identification, Authentication, Integrity, Privacy

Via the GSS-API, RPCSEC_GSS can be used to identify and authenticate users on clients to servers, and servers to users. It can also perform integrity checking on the entire RPC message, including the RPC header, and on the arguments or results. Finally, privacy, usually via encryption, is a service available with RPCSEC_GSS. Privacy is performed on the arguments and results. Note that if privacy is selected, integrity, authentication, and identification are enabled. If privacy is not selected, but integrity is selected, authentication and identification are enabled. If integrity and privacy are not selected, but authentication is enabled, identification is enabled. RPCSEC_GSS does not provide identification as a separate service.

Although GSS-API has an authentication service distinct from its privacy and integrity services, GSS-API's authentication service is not used for RPCSEC_GSS's authentication service. Instead, each RPC request and response header is integrity protected with the GSS-API integrity service, and this allows RPCSEC_GSS to offer per-RPC authentication and identity. See [4] for more information.

NFSv4.1 client and servers MUST support RPCSEC_GSS's integrity and authentication service. NFSv4.1 servers MUST support RPCSEC_GSS's privacy service. NFSv4.1 clients SHOULD support RPCSEC_GSS's privacy service.

2.2.1.1.1.2. Security Mechanisms for NFSv4.1 🛡️

RPCSEC_GSS, via GSS-API, normalizes access to mechanisms that provide security services. Therefore, NFSv4.1 clients and servers MUST support the Kerberos V5 security mechanism.

The use of RPCSEC_GSS requires selection of mechanism, quality of protection (QOP), and service (authentication, integrity, privacy). For the mandated security mechanisms, NFSv4.1 specifies that a QOP of zero is used, leaving it up to the mechanism or the mechanism's configuration to map QOP zero to an appropriate level of protection. Each mandated mechanism specifies a minimum set of cryptographic algorithms for implementing integrity and privacy. NFSv4.1 clients and servers MUST be implemented on operating environments that comply with the REQUIRED cryptographic algorithms of each REQUIRED mechanism.

2.2.1.1.1.2.1. Kerberos V5 🧚

okay so the Kerberos V5 GSS-API mechanism described in [5] MUST be implemented with the RPCSEC_GSS services in this table, no cap 💅

      column descriptions:
      1 == number of pseudo flavor
      2 == name of pseudo flavor
      3 == mechanism's OID
      4 == RPCSEC_GSS service
      5 == NFSv4.1 clients MUST support
      6 == NFSv4.1 servers MUST support

      1      2        3                    4                     5   6
      ------------------------------------------------------------------
      390003 krb5     1.2.840.113554.1.2.2 rpc_gss_svc_none      yes yes
      390004 krb5i    1.2.840.113554.1.2.2 rpc_gss_svc_integrity yes yes
      390005 krb5p    1.2.840.113554.1.2.2 rpc_gss_svc_privacy    no yes

ngl the pseudo flavor number and name are only here as a mapping aid for the implementor 🎀. NFSv4.1 already has a way to negotiate security and it understands the GSS-API mechanism, so the pseudo flavor is not needed there. It is needed for NFSv3, though, since security negotiation happens through the MOUNT protocol (see [40]) 🦋.

Back when NFSv4.1 was specified, the Advanced Encryption Standard (AES) with HMAC-SHA1 was a REQUIRED algorithm set for Kerberos V5 ✨. NFSv4.0 was a different story 😭: weaker algorithm sets were REQUIRED for Kerberos V5, and were REQUIRED in the NFSv4.0 specification, because the Kerberos V5 spec at that time didn't specify stronger algorithms. NFSv4.1 does not specify REQUIRED algorithms for Kerberos V5. Instead, the implementor is expected to keep up with the evolution of the Kerberos V5 standard if and when stronger algorithms get specified 🌸.

2.2.1.1.1.2.1.1. Security Considerations for Cryptographic Algorithms in Kerberos V5 🔐

bestie, when you deploy NFSv4.1, the strength of the security you get depends on your existing Kerberos V5 infrastructure 🏰. The algorithms of Kerberos V5 aren't directly exposed to or selectable by the client or server, so the user of NFSv4.1 has to do some due diligence to make sure security is acceptable wherever it's needed. Not me being dramatic, this one matters 💀.

2.2.1.1.1.3. GSS Server Principal 👑

Regardless of which security mechanism under RPCSEC_GSS is in use, the NFS server MUST identify itself in GSS-API via a GSS_C_NT_HOSTBASED_SERVICE name type 🎀. These names are of the form:

service@hostname

For NFS, the "service" element is:

nfs

Implementations of security mechanisms will convert nfs@hostname to various different forms. For Kerberos V5, the following form is RECOMMENDED:

nfs/hostname

2.3. COMPOUND and CB_COMPOUND 🍓

A big glow-up from the NFS versions before NFSv4 is the introduction of the COMPOUND procedure 💖. For the NFSv4 protocol, in all minor versions, there are exactly two RPC procedures: NULL and COMPOUND. COMPOUND is defined as a series of individual operations, and these operations do the same sorts of functions that traditional NFS procedures used to do ✨.

The operations combined within a COMPOUND request are evaluated in order by the server, with no atomicity guarantees 👀, so it's not all-or-nothing. There's a limited set of facilities for passing results from one operation to another. Once an operation returns a failing result, evaluation ends and the results of all evaluated operations are returned to the client 💅.

With COMPOUND, the client can build simple or complex requests 🧚. These requests allow for a reduction in the number of RPCs needed for logical file system operations. For example, multi-component look up requests can be built by combining multiple LOOKUP operations. Those can be further combined with operations like GETATTR, READDIR, or OPEN plus READ to do more complicated sets of operations without incurring additional latency 🌸.

NFSv4.1 also has a whole considerable set of callback operations, where the server makes an RPC directed at the client 📞. Callback RPCs have a similar structure to normal server requests. In all minor versions of the NFSv4 protocol, there are two callback RPC procedures: CB_NULL and CB_COMPOUND. CB_COMPOUND is defined in an analogous fashion to COMPOUND, with its own set of callback operations 💜.

Adding new server and callback operations within the COMPOUND and CB_COMPOUND request framework gives a means of extending the protocol in subsequent minor versions 🌈.

Except for a small number of operations needed for session creation, server requests and callback requests are performed within the context of a session. Sessions provide a client context for every request and support robust replay protection for non-idempotent requests 🔥.

2.4. Client Identifiers and Client Owners

okay bestie, for every operation that grabs or depends on locking state, the server has to be able to tell exactly which client is asking 🔍✨

Each distinct client instance gets represented by a client ID 🆔. A client ID is a 64-bit identifier for one specific client at a given time 💜. It changes whenever the client re-initializes, and it may change when the server re-initializes too 🔄. Client IDs are what power lock identification and crash recovery 🦋.

During steady state operation (aka when everything is chill 🧃), the client ID for each operation comes from the session (see Section 2.10) that the operation gets sent on. And a session gets tied to a client ID the moment it's created 💅.

Unlike NFSv4.0 🙄, in NFSv4.1 the only operations you can run before a client ID exists are the ones needed to set that client ID up 🎀.

A sequence of an EXCHANGE_ID operation followed by a CREATE_SESSION operation using that client ID (eir_clientid as returned from EXCHANGE_ID) is required to establish and confirm the client ID on the server 🧾. Establishing identification as a new incarnation of the client also has the effect of immediately releasing any locking state that a previous incarnation of that same client might have had on the server 🧹. That released state would include all byte-range lock, share reservation, layout state, and -- where the server supports neither the CLAIM_DELEGATE_PREV nor CLAIM_DELEG_CUR_FH claim types -- all delegation state associated with the same client with the same identity 💥. For the delegation state recovery deets, see Section 10.2.1. For layout state recovery, see Section 12.7.1 📚.

Releasing that state requires the server to be able to figure out that one client instance is the successor of another 🕵️‍♀️. When it can't do that, for any of a number of reasons, the locking state will hang around for a while, subject to lease expiration (see Section 8.3). The new client will then have to wait for that state to be removed if it makes conflicting lock requests 😩.

Client identification is wrapped up in the following client owner data type 🎁:

struct client_owner4 {
        verifier4       co_verifier;
        opaque          co_ownerid<NFS4_OPAQUE_LIMIT>;
};

The first field, co_verifier, is a client incarnation verifier 🪪. It lets the server tell apart successive incarnations (e.g., reboots) of the same client 🔁. The server will start the process of canceling the client's leased state if co_verifier is different than what the server has previously recorded for the identified client (as specified in the co_ownerid field) 🚨.

The second field, co_ownerid, is a variable length string that uniquely defines the client, so that later instances of the same client bear the same co_ownerid with a different verifier 🏷️.

There are several things to think about when the client generates the co_ownerid string 🧠:

The string should be unique, so that multiple clients don't present the same string. The consequences of two clients presenting the same string range from one client getting an error to one client having its leased state abruptly and unexpectedly cancelled 😱.
The string should be selected so that later incarnations (e.g., restarts) of the same client present the same string. The implementor is cautioned against an approach that requires the string to be recorded in a local file, because that rules out using the implementation in an environment with no local disk, where all file access goes through an NFSv4.1 server 💾🚫.
The string should be the same for each server network address that the client accesses 🌐. That way, if a server has multiple interfaces, the client can trunk traffic over multiple network paths as described in Section 2.10.5. (Note: the precise opposite was advised in the NFSv4.0 specification [37].) 🙃
The algorithm for generating the string should not assume that the client's network address won't change, unless the client implementation knows it's using statically assigned network addresses 📍. This includes changes between client incarnations and even changes while the client is still running in its current incarnation. So with dynamic address assignment, if the client includes just its network address in the co_ownerid string, there's a real risk that after the client gives up that address, another client using a similar algorithm would generate a conflicting co_ownerid string 😬.

Given all that, a well-generated co_ownerid string is one that includes ✅:

If applicable, the client's statically assigned network address 🏠.
Additional information that tends to be unique, such as one or more of:
- The client machine's serial number (for privacy reasons, it's best to perform some one-way function on the serial number) 🔒.
- A Media Access Control (MAC) address (again, a one-way function should be performed) 🔐.
- The timestamp of when the NFSv4.1 software was first installed on the client (though this is subject to the earlier caution about using information stored in a file, because the file might only be accessible over NFSv4.1) 🗓️.
- A true random number. However, since this number ought to be the same between client incarnations, it has the same problem as using the timestamp of the software installation 🎲.
For a user-level NFSv4.1 client, additional information to tell it apart from other user-level clients running on the same host, such as a process identifier or other unique sequence 🧑‍💻.

The client ID is assigned by the server (the eir_clientid result from EXCHANGE_ID) and should be chosen so that it won't conflict with a client ID the server has previously assigned. This applies across server restarts 🔢.

If the server restarts, a client may find out that its current client ID is no longer valid when it receives an NFS4ERR_STALE_CLIENTID error 💀. The exact circumstances depend on the characteristics of the sessions involved, specifically whether the session is persistent (see Section 2.10.6.5). But in each case, the client gets this error when it tries to establish a new session with the existing client ID. The error says a new client ID needs to be obtained via EXCHANGE_ID, and the new session established with that client ID 🔄.

When a session is not persistent, the client finds out it needs to create a new session because it gets an NFS4ERR_BADSESSION, since the session was lost as part of a server restart 🫠. When the existing client ID is presented to a server as part of creating a session, and that client ID isn't recognized (as would happen after a server restart), the server rejects the request with NFS4ERR_STALE_CLIENTID 🚫.

If the session is persistent, the client re-establishes communication using the existing session after the restart 🧲. This session is associated with the existing client ID, but it may only be used to retransmit operations the client previously sent and didn't see replies to 📨. Replies to operations the server previously performed come from the reply cache. Otherwise, NFS4ERR_DEADSESSION gets returned. That's why such a session is called "dead" ☠️. To perform new operations in this situation, the client needs to establish a new session. If it tries to establish that new session with the existing client ID, the server rejects the request with NFS4ERR_STALE_CLIENTID 🙅‍♀️.

When NFS4ERR_STALE_CLIENTID shows up in either of these situations, the client needs to get a new client ID using the EXCHANGE_ID operation 🆕. Then it uses that client ID as the basis of a new session, and proceeds to any other necessary recovery for the server restart case (see Section 8.4.2) 🧩.

See the descriptions of EXCHANGE_ID (Section 18.35) and CREATE_SESSION (Section 18.36) for the full spec on these operations 📖.

2.4.1. Upgrade from NFSv4.0 to NFSv4.1

To make the upgrade from NFSv4.0 to NFSv4.1 smoother 🆙, a server may compare a value of data type client_owner4 in an EXCHANGE_ID with a value of data type nfs_client_id4 that was established using the SETCLIENTID operation of NFSv4.0. A server that does this lets an upgraded client skip waiting for the lease (i.e., the lease established by the NFSv4.0 instance of the client) to expire ⏩. This requires the value of data type client_owner4 to be constructed the same way as the value of data type nfs_client_id4. If the latter's contents included the server's network address (per the recommendations of the NFSv4.0 specification [37]), and the NFSv4.1 client doesn't want a client ID that prevents trunking, it should send two EXCHANGE_ID operations 2️⃣. The first EXCHANGE_ID will have a client_owner4 equal to the nfs_client_id4, which clears the state created by the NFSv4.0 client 🧽. The second EXCHANGE_ID won't include the server's network address. The state created for the second EXCHANGE_ID won't have to wait for lease expiration, because there's no state left to expire 🎉.

2.4.2. Server Release of Client ID

NFSv4.1 introduces a new operation called DESTROY_CLIENTID (Section 18.50), which the client SHOULD use to destroy a client ID it no longer needs 👋. This allows a graceful, bilateral release of a client ID 🤝. The operation can't be used if there are sessions associated with the client ID, or state with an unexpired lease 🚧.

If the server determines that the client holds no associated state for its client ID (associated state includes unrevoked sessions, opens, locks, delegations, layouts, and wants), the server MAY choose to unilaterally release the client ID to conserve resources 🧹. If the client contacts the server after this release, the server MUST ensure the client gets the appropriate error, so it will use the EXCHANGE_ID/CREATE_SESSION sequence to establish a new client ID 🚨. The server ought to be very hesitant to release a client ID, since the work the client has to do to recover from that event is the same burden as if the server had failed and restarted 😵. Typically, a server wouldn't release a client ID unless there had been no activity from that client for many minutes ⏳. As long as there are sessions, opens, locks, delegations, layouts, or wants, the server MUST NOT release the client ID 🙅. See Section 2.10.13.1.4 for the discussion on releasing inactive sessions 🌸.

2.4.3. Resolving Client Owner Conflicts ✨

okay so this is the part where the server has to figure out who's who when a client owner shows up again 🕵️‍♀️💅 bestie, it's giving detective energy, no cap.

When the server gets an EXCHANGE_ID for a client owner that currently has no state, or that has state but the lease has expired, the server MUST allow the EXCHANGE_ID and confirm the new client ID if followed by the appropriate CREATE_SESSION. 🌸 easy yes, the door is open!

When the server gets an EXCHANGE_ID for a new incarnation of a client owner that currently has an old incarnation with state and an unexpired lease, the server is allowed to dispose of the state of the previous incarnation of the client owner if one of the following is true: 👑

* The principal that created the client ID for the client owner is the same as the principal that is sending the EXCHANGE_ID operation. Note that if the client ID was created with SP4_MACH_CRED state protection (Section 18.35), the principal MUST be based on RPCSEC_GSS authentication, the RPCSEC_GSS service used MUST be integrity or privacy, and the same GSS mechanism and principal MUST be used as that used when the client ID was created. 🔐 ngl this one is strict, the same principal, the same mechanism, the same everything.

The client ID was established with SP4_SSV protection (Section 18.35, Section 2.10.8.3) and the client sends the EXCHANGE_ID with the security flavor set to RPCSEC_GSS using the GSS SSV mechanism (Section 2.10.9). 🦋 the SSV path, lowkey iconic.
The client ID was established with SP4_SSV protection, and under the conditions described herein, the EXCHANGE_ID was sent with SP4_MACH_CRED state protection. Because the SSV might not persist across client and server restart, and because the first time a client sends EXCHANGE_ID to a server it does not have an SSV, the client MAY send the subsequent EXCHANGE_ID without an SSV RPCSEC_GSS handle. Instead, as with SP4_MACH_CRED protection, the principal MUST be based on RPCSEC_GSS authentication, the RPCSEC_GSS service used MUST be integrity or privacy, and the same GSS mechanism and principal MUST be used as that used when the client ID was created. 🧚 okay this one is a lot, but the vibe is: SSVs don't always survive a restart, so the client can fall back to machine creds, but the auth rules still apply.

If none of the above situations apply, the server MUST return NFS4ERR_CLID_INUSE. 💀 not it, sorry bestie.

If the server accepts the principal and co_ownerid as matching that which created the client ID, and the co_verifier in the EXCHANGE_ID differs from the co_verifier used when the client ID was created, then after the server receives a CREATE_SESSION that confirms the client ID, the server deletes state. If the co_verifier values are the same (e.g., the client either is updating properties of the client ID (Section 18.35) or is attempting trunking (Section 2.10.5), the server MUST NOT delete state. 🫶 so different verifier means cleanup time, same verifier means hands off the state!

2.5. Server Owners 🏠

okay so the server owner is basically the client owner's cousin 💜, except there's no shorthand server ID. The server owner is defined in the following data type:

struct server_owner4 {
 uint64_t       so_minor_id;
 opaque         so_major_id<NFS4_OPAQUE_LIMIT>;
};

The server owner is returned from EXCHANGE_ID. When the so_major_id fields are the same in two EXCHANGE_ID results, the connections that each EXCHANGE_ID were sent over can be assumed to address the same server (as defined in Section 1.7). If the so_minor_id fields are also the same, then not only do both connections connect to the same server, but the session can be shared across both connections. 🎀 matching major means same server, matching minor means you can share the session too!

The reader is cautioned that multiple servers may deliberately or accidentally claim to have the same so_major_id or so_major_id/so_minor_id; the reader should examine Sections 2.10.5 and 18.35 in order to avoid acting on falsely matching server owner values. ⚠️ bestie, be careful, not every server is honest about its ID, so go read those sections before you trust a match!

The considerations for generating an so_major_id are similar to that for generating a co_ownerid string (see Section 2.4). The consequences of two servers generating conflicting so_major_id values are less dire than they are for co_ownerid conflicts because the client can use RPCSEC_GSS to compare the authenticity of each server (see Section 2.10.5). 🔍 a clash here is less of a disaster, since the client can check which server is legit with RPCSEC_GSS.

2.6. Security Service Negotiation 🤝

With the NFSv4.1 server potentially offering multiple security mechanisms, the client needs a method to determine or negotiate which mechanism is to be used for its communication with the server. The NFS server may have multiple points within its file system namespace that are available for use by NFS clients. These points can be considered security policy boundaries, and, in some NFS implementations, are tied to NFS export points. In turn, the NFS server may be configured such that each of these security policy boundaries may have different or multiple security mechanisms in use. 🌈 so many doors, so many locks, it's giving a whole building of security!

The security negotiation between client and server SHOULD be done with a secure channel to eliminate the possibility of a third party intercepting the negotiation sequence and forcing the client and server to choose a lower level of security than required or desired. See Section 21 for further discussion. 🛡️ bestie, negotiate over a secure channel so nobody sneaks in and downgrades you!

2.6.1. NFSv4.1 Security Tuples 🎟️

An NFS server can assign one or more "security tuples" to each security policy boundary in its namespace. Each security tuple consists of a security flavor (see Section 2.2.1.1) and, if the flavor is RPCSEC_GSS, a GSS-API mechanism Object Identifier (OID), a GSS-API quality of protection, and an RPCSEC_GSS service. 💖 a tuple is basically a little bundle of security settings!

2.6.2. SECINFO and SECINFO_NO_NAME 🔎

The SECINFO and SECINFO_NO_NAME operations allow the client to determine, on a per-filehandle basis, what security tuple is to be used for server access. In general, the client will not have to use either operation except during initial communication with the server or when the client crosses security policy boundaries at the server. However, the server's policies may also change at any time and force the client to negotiate a new security tuple. 😭 policies can change whenever, so stay alert!

Where the use of different security tuples would affect the type of access that would be allowed if a request was sent over the same connection used for the SECINFO or SECINFO_NO_NAME operation (e.g., read-only vs. read-write) access, security tuples that allow greater access should be presented first. Where the general level of access is the same and different security flavors limit the range of principals whose privileges are recognized (e.g., allowing or disallowing root access), flavors supporting the greatest range of principals should be listed first. 👑 the strongest-access tuples go first, and if access is equal, the ones that recognize the most principals go first.

2.6.3. Security Error 🚨

Based on the assumption that each NFSv4.1 client and server MUST support a minimum set of security (i.e., Kerberos V5 under RPCSEC_GSS), the NFS client will initiate file access to the server with one of the minimal security tuples. During communication with the server, the client may receive an NFS error of NFS4ERR_WRONGSEC. This error allows the server to notify the client that the security tuple currently being used contravenes the server's security policy. The client is then responsible for determining (see Section 2.6.3.1) what security tuples are available at the server and choosing one that is appropriate for the client. 🙈 if you get NFS4ERR_WRONGSEC, the tuple you're using is not allowed, so go find one that works!

2.6.3.1. Using NFS4ERR_WRONGSEC, SECINFO, and SECINFO_NO_NAME 🧩

This section explains the mechanics of NFSv4.1 security negotiation. 🎀 time to see how the pieces actually fit together!

2.6.3.1.1. Put Filehandle Operations 📦

The term "put filehandle operation" refers to PUTROOTFH, PUTPUBFH, PUTFH, and RESTOREFH. Each of the subsections herein describes how the server handles a subseries of operations that starts with a put filehandle operation. 🌸 these four are the put filehandle crew!

2.6.3.1.1.1. Put Filehandle Operation + SAVEFH 💾

The client is saving a filehandle for a future RESTOREFH, LINK, or RENAME. SAVEFH MUST NOT return NFS4ERR_WRONGSEC. To determine whether or not the put filehandle operation returns NFS4ERR_WRONGSEC, the server implementation pretends SAVEFH is not in the series of operations and examines which of the situations described in the other subsections of Section 2.6.3.1.1 apply. 🙅‍♀️ SAVEFH never gets the WRONGSEC error, it's just vibing in the background.

2.6.3.1.1.2. Two or More Put Filehandle Operations 🔁

For a series of N put filehandle operations, the server MUST NOT return NFS4ERR_WRONGSEC to the first N-1 put filehandle operations. The Nth put filehandle operation is handled as if it is the first in a subseries of operations. For example, if the server received a COMPOUND request with this series of operations -- PUTFH, PUTROOTFH, LOOKUP -- then the PUTFH operation is ignored for NFS4ERR_WRONGSEC purposes, and the PUTROOTFH, LOOKUP subseries is processed as according to Section 2.6.3.1.1.3. 💅 only the last put filehandle in a row gets to be judged!

PUTFH, PUTROOTFH, LOOKUP

2.6.3.1.1.3. Put Filehandle Operation + LOOKUP (or OPEN of an Existing Name) 🔍

This situation also applies to a put filehandle operation followed by a LOOKUP or an OPEN operation that specifies an existing component name. 🕵️‍♀️ same energy, just with a LOOKUP or OPEN after it!

In this situation, the client is potentially crossing a security policy boundary, and the set of security tuples the parent directory supports may differ from those of the child. The server implementation may decide whether to impose any restrictions on security policy administration. There are at least three approaches (sec_policy_child is the tuple set of the child export, sec_policy_parent is that of the parent). 🧠 three possible approaches, let's look at them!

(a) sec_policy_child <= sec_policy_parent (<= for subset). This means that the set of security tuples specified on the security policy of a child directory is always a subset of its parent directory. 🌷 approach A: the child is always a subset of the parent.

(b) sec_policy_child ^ sec_policy_parent != {} (^ for intersection, {} for the empty set). This means that the set of security tuples specified on the security policy of a child directory always has a non-empty intersection with that of the parent. 🌿 approach B: they always share at least one tuple.

(c) sec_policy_child ^ sec_policy_parent == {}. This means that the set of security tuples specified on the security policy of a child directory may not intersect with that of the parent. In other words, there are no restrictions on how the system administrator may set up these tuples. 🦄 approach C: anything goes, the admin is in charge!

In order for a server to support approaches (b) (for the case when a client chooses a flavor that is not a member of sec_policy_parent) and (c), the put filehandle operation cannot return NFS4ERR_WRONGSEC when there is a security tuple mismatch. Instead, it should be returned from the LOOKUP (or OPEN by existing component name) that follows. 🎯 so for B and C, the error waits until the LOOKUP or OPEN shows up.

Since the above guideline does not contradict approach (a), it should be followed in general. Even if approach (a) is implemented, it is possible for the security tuple used to be acceptable for the target of LOOKUP but not for the filehandles used in the put filehandle operation. The put filehandle operation could be a PUTROOTFH or PUTPUBFH, where the client cannot know the security tuples for the root or public filehandle. Or the security policy for the filehandle used by the put filehandle operation could have changed since the time the filehandle was obtained. 🤔 even with approach A, things can get sneaky, like when the client can't know the root's tuples or the policy changed on it.

Therefore, an NFSv4.1 server MUST NOT return NFS4ERR_WRONGSEC in response to the put filehandle operation if the operation is immediately followed by a LOOKUP or an OPEN by component name. 🌈 so the put filehandle operation gets a free pass when a LOOKUP or OPEN comes right after it!

2.6.3.1.1.4. Put Filehandle Operation + LOOKUPP ✨

Okay so, SECINFO only works its way down, which means there's literally no way LOOKUPP can return NFS4ERR_WRONGSEC without SECINFO_NO_NAME 😭. SECINFO_NO_NAME fixes this with style SECINFO_STYLE4_PARENT, which goes the opposite direction from SECINFO 🔄. Same rule as Section 2.6.3.1.1.3: a put filehandle operation that's followed by a LOOKUPP MUST NOT return NFS4ERR_WRONGSEC 🙅‍♀️. If the server doesn't support SECINFO_NO_NAME, the client's only option is to send the put filehandle operation, LOOKUPP, GETFH sequence with every security tuple it supports 🫠.

Regardless of whether SECINFO_NO_NAME is supported, an NFSv4.1 server MUST NOT return NFS4ERR_WRONGSEC in response to a put filehandle operation if the operation is immediately followed by a LOOKUPP. 💅

2.6.3.1.1.5. Put Filehandle Operation + SECINFO/SECINFO_NO_NAME 🔍

A security-sensitive client is allowed to pick a strong security tuple when it asks a server which security tuples a file object permits 🔐. The tuple the client picks doesn't have to be in the security policy list of either the parent directory from the put filehandle operation or the child file object from SECINFO (or any parent directory from SECINFO_NO_NAME). Of course, the server has to be configured for whatever tuple the client selects, or the request will fail at the RPC layer with an appropriate authentication error 🙃.

In theory, there's no connection between the security flavor used by SECINFO or SECINFO_NO_NAME and the ones in the security policy. But in practice, the client may start looking for strong flavors from the ones the security policy supports, followed by the ones in the REQUIRED set 👀.

The NFSv4.1 server MUST NOT return NFS4ERR_WRONGSEC to a put filehandle operation that is immediately followed by SECINFO or SECINFO_NO_NAME. The NFSv4.1 server MUST NOT return NFS4ERR_WRONGSEC from SECINFO or SECINFO_NO_NAME. 🚫

2.6.3.1.1.6. Put Filehandle Operation + Nothing 🫥

The NFSv4.1 server MUST NOT return NFS4ERR_WRONGSEC. Simple, clean, no drama 💖.

2.6.3.1.1.7. Put Filehandle Operation + Anything Else 🧩

"Anything Else" includes OPEN by filehandle, so yes, it counts too 📂.

The security policy enforcement applies to the filehandle in the put filehandle operation. That's why the put filehandle operation MUST return NFS4ERR_WRONGSEC when there's a security tuple mismatch. This avoids the mess of adding NFS4ERR_WRONGSEC as an allowable error to every other operation 🧹.

A COMPOUND with a put filehandle operation + SECINFO_NO_NAME (style SECINFO_STYLE4_CURRENT_FH) is an efficient way for the client to recover from NFS4ERR_WRONGSEC ⚡.

The NFSv4.1 server MUST NOT return NFS4ERR_WRONGSEC to any operation other than a put filehandle operation, LOOKUP, LOOKUPP, and OPEN (by component name). Only those four get to be the drama queens here 👑.

2.6.3.1.1.8. Operations after SECINFO and SECINFO_NO_NAME 🌀

Okay, this one's a whole plot twist 😮‍💨. Say a client sends a COMPOUND with the series SEQUENCE, PUTFH, SECINFO_NONAME, READ, and the security tuple it used doesn't match what the target file needs. By rule (see Section 2.6.3.1.1.5), neither PUTFH nor SECINFO_NO_NAME can return NFS4ERR_WRONGSEC. By rule (see Section 2.6.3.1.1.7), READ can't return NFS4ERR_WRONGSEC either 😤.

So how does it get resolved? SECINFO and SECINFO_NO_NAME consume the current filehandle (and yes, that's a change from NFSv4.0 🔄). That leaves no current filehandle for READ to use, so READ returns NFS4ERR_NOFILEHANDLE 🫣.

2.6.3.1.2. LINK and RENAME 🔗

The LINK and RENAME operations use both the current and saved filehandles 🧑‍🤝‍🧑. Technically, the server MAY return NFS4ERR_WRONGSEC from LINK or RENAME if the security policy of the saved filehandle rejects the security flavor used in the COMPOUND request's credentials. If the server does that, and there's no intersection between the security policies of the saved and current filehandles, then the client can never actually complete the intended LINK or RENAME 😭.

For example, say the client sends this COMPOUND request: SEQUENCE, PUTFH bFH, SAVEFH, PUTFH aFH, RENAME "c" "d", where bFH and aFH refer to different directories. Suppose no common security tuple exists between the policies of aFH and bFH. If the client sends the request using credentials that bFH's policy accepts but aFH's doesn't, the PUTFH aFH operation fails with NFS4ERR_WRONGSEC. After a SECINFO_NO_NAME request, the client sends SEQUENCE, PUTFH bFH, SAVEFH, PUTFH aFH, RENAME "c" "d" using credentials that aFH's policy accepts but bFH's doesn't. Now the server returns NFS4ERR_WRONGSEC on the RENAME operation 🌪️.

To stop a client from getting stuck in an endless loop of a request with LINK or RENAME, followed by a request with SECINFO_NO_NAME or SECINFO, the server MUST detect when the security policies of the current and saved filehandles have no mutually acceptable security tuple. In that situation, the server MUST NOT return NFS4ERR_WRONGSEC from LINK or RENAME. Instead, the server MUST do one of two things:

The server can return NFS4ERR_XDEV. 🙅
The server can allow the security policy of the current filehandle to override that of the saved filehandle, and so return NFS4_OK. ✅

2.7. Minor Versioning 🌱

To handle the need for an NFS protocol that can evolve as things change, the NFSv4.1 protocol includes the rules and framework for future minor changes or versioning 🦋.

The base assumption is that any future accepted minor version will be documented in one or more Standards Track RFCs 📜. Minor version 0 of the NFSv4 protocol is represented by [37], and minor version 1 is represented by this RFC. The COMPOUND and CB_COMPOUND procedures support encoding the minor version the client is requesting 💜.

Here are the basic rules for developing minor versions. Heads up: a future minor version may modify or add to these rules as part of its own definition 👀.

Procedures are not added or deleted.

To keep the general RPC model intact, NFSv4 minor versions will not add to or delete procedures from the NFS program 🧱.

Minor versions may add operations to the COMPOUND and CB_COMPOUND procedures.

Adding operations to COMPOUND and CB_COMPOUND doesn't affect the RPC model, so it's all good 🌸.

Minor versions may append attributes to the bitmap4 that represents sets of attributes and to the fattr4 that represents sets of attribute values.

This lets the attribute model grow and adapt for the future 🌿.

Minor version X must append any new attributes after the last documented attribute.

Attribute results are an opaque array of per-attribute, XDR-encoded results, so adding new attributes in the middle of the current definitions would be way too burdensome 😩.

Minor versions must not modify the structure of an existing operation's arguments or results.

Handling multiple structure definitions for one operation is too much of a headache, so new operations should be added instead of changing existing structures 🙅‍♀️.

This rule still allows these adaptations in a minor version:

adding bits to flag fields, such as new attributes to GETATTR's bitmap4 data type, and providing corresponding variants of opaque arrays, such as a notify4 used together with such bitmaps
adding bits to existing attributes like ACLs that have flag words
extending enumerated types (including NFS4ERR_*) with new values
adding cases to a switched union

Minor versions must not modify the structure of existing attributes. 🔒
Minor versions must not delete operations.

This stops an operation "slot" from being reused in a future minor version 🎰.

Minor versions must not delete attributes. 🫶
Minor versions must not delete flag bits or enumeration values. 💅
Minor versions may declare an operation MUST NOT be implemented.

Declaring an operation MUST NOT be implemented is the same as obsoleting it. For the client, that means the operation MUST NOT be sent to the server. For the server, an NFS error can be returned instead of "dropping" the request as an XDR decode error. This lets an operation be obsolete while keeping its structure, so a future minor version can bring it back 🔄.

Minor versions may declare that an attribute MUST NOT be implemented.
Minor versions may declare that a flag bit or enumeration value MUST NOT be implemented.

Minor versions may downgrade features from REQUIRED to RECOMMENDED, or RECOMMENDED to OPTIONAL. 📉
Minor versions may upgrade features from OPTIONAL to RECOMMENDED, or RECOMMENDED to REQUIRED. 📈
A client and server that support minor version X SHOULD support minor versions zero through X-1 as well. 🤝
Except for infrastructural changes, a minor version must not introduce REQUIRED new features.

This lets new functionality arrive while forcing real implementation experience before a feature gets marked REQUIRED 🧪. Still, some features are infrastructural and affect a lot of things, and letting those be RECOMMENDED or OPTIONAL would make the minor version really hard to implement 😵‍💫.

A client MUST NOT attempt to use a stateid, filehandle, or similar returned object from the COMPOUND procedure with minor version X for another COMPOUND procedure with minor version Y, where X != Y. 🚷

2.8. Non-RPC-Based Security Services 🛡️

As described in Section 2.2.1.1.1.1, NFSv4.1 relies on RPC for identification, authentication, integrity, and privacy. NFSv4.1 also provides or enables extra security services, covered in the next few subsections 🔐.

2.8.1. Authorization 🪪

Authorization to access a file object through an NFSv4.1 operation is ultimately decided by the NFSv4.1 server 👑. A client can predetermine its access to a file object with the OPEN (Section 18.16) and ACCESS (Section 18.1) operations.

Principals with the right access can change the authorization on a file object with the SETATTR (Section 18.30) operation. Attributes that affect access rights include mode, owner, owner_group, acl, dacl, and sacl. See Section 5 🗝️.

2.8.2. Auditing 📝

NFSv4.1 provides auditing per file object, through the acl and sacl attributes described in Section 6. Specifying audit log formats or management policies is outside the scope of this specification 🙈.

2.8.3. Intrusion Detection 🚨

NFSv4.1 provides alarm control per file object, through the acl and sacl attributes described in Section 6. Alarms may serve as the basis for intrusion detection. Specifying heuristics for detecting intrusion via alarms is outside the scope of this specification 🕵️‍♀️.

2.9. Transport Layers 🚌

2.9.1. REQUIRED and RECOMMENDED Properties of Transports

okay so NFSv4.1 can run over RDMA 🚀 and over non-RDMA transports too, as long as they have these two vibes ✨

The transport supports reliable delivery of data 💖. NFSv4.1 needs this, but neither NFSv4.1 nor RPC has facilities for making sure it happens [41].
The transport delivers data in the order it was sent 🎀. Ordered delivery makes it easier to spot transmit errors, and it makes sending arbitrary-sized requests and responses through the record marking protocol [3] simpler.

If an NFSv4.1 implementation supports operation over the IP network protocol, then any transport used between NFS and IP MUST be one of the IETF-approved congestion control transport protocols 🛡️. When this document was written, the only two transports with both of those attributes were TCP and the Stream Control Transmission Protocol (SCTP) 🦋. To boost interoperability, an NFSv4.1 implementation MUST support operation over the TCP transport protocol 🫶.

Even if NFSv4.1 runs over a non-IP network protocol, it is RECOMMENDED that the transport supports congestion control 🌸.

A connectionless transport is allowed under NFSv4.1, but reliable and in-order delivery of data combined with congestion control from that connectionless transport is REQUIRED 💅. Which means UDP by itself MUST NOT be used as an NFSv4.1 transport 💀. NFSv4.1 treats a client transport address and a server transport address used to send data over a transport as a pair that together constitutes a connection, even when the underlying transport doesn't really do connections 🧚.

2.9.2. Client and Server Transport Behavior

okay bestie, if you're using a connection-oriented transport (like TCP 💜), the client and server SHOULD use long-lived connections, and there are three reasons:

This keeps the transport's congestion control mechanisms from getting weakened by short-lived connections 🌈.
This makes performance better over the WAN, because it gets rid of the need for connection setup handshakes ⭐.
The NFSv4.1 callback model is different from NFSv4.0, and it requires the client and server to maintain a client-created backchannel (see Section 2.10.3.1) for the server to use 👑.

To reduce congestion, if a connection-oriented transport is used and the request is not the NULL procedure, then:

A requester MUST NOT retry a request unless the connection the request was sent over was lost before the reply was received 🙈.
A replier MUST NOT silently drop a request, even if that request is a retry 😭. (The silent drop behavior of RPCSEC_GSS [4] doesn't apply here, because that behavior happens at the RPCSEC_GSS layer, which is a lower layer in request processing.) Instead, the replier SHOULD return an appropriate error (see Section 2.10.6.1), or it MAY disconnect the connection 💅.

When sending a reply, the replier MUST send it to the same full network address the requester sent the request from. For example, with an IP-based transport, the source port of the requester is part of the full network address 🍓. If you're using a connection-oriented transport, replies MUST be sent on the same connection the request came in on 🦋.

If a connection gets dropped after the replier receives the request but before it sends the reply, the replier might have a pending reply. If a new connection is established with the same source and destination full network address as the dropped one, then the replier MUST NOT send the reply until the requester retries the request. The reason is that the requester MAY retry a request over a different connection, as long as that connection is associated with the original request's session 🎀.

When using RDMA transports, there are some extra reasons not to tolerate retries over the same connection 🚀:

RDMA transports use "credits" to enforce flow control, where a credit is a right for a peer to transmit a message. If one peer retransmits a request (or reply), it consumes an additional credit. If the replier retransmits a reply, it will almost certainly cause an RDMA connection loss, since the requester would typically only post a single receive buffer for each request. If the requester retransmits a request, the extra credit consumed on the server might lead to RDMA connection failure, unless the client accounted for it and decreased its available credit, which wastes resources 😩.
RDMA credits create a new issue for the reply cache in NFSv4.1. The reply cache may be used when a connection within a session is lost, such as after the client reconnects. Credit information is a dynamic property of the RDMA connection, so stale values must not be replayed from the cache. This means the reply cache contents must not be blindly used when replies are sent from it, and credit information appropriate to the channel must be refreshed by the RPC layer ✨.

Also, as described in Section 2.10.6.2, while a session is active, the NFSv4.1 requester MUST NOT stop waiting for a reply 🫶.

2.9.3. Ports

Historically, NFSv3 servers have listened on TCP port 2049 🌸. The registered port 2049 [42] for the NFS protocol should be the default configuration, so yay for consistency 🎀. NFSv4.1 clients SHOULD NOT use the RPC binding protocols described in [43] 🙈.

2.10. Session

NFSv4.1 clients and servers MUST support the session feature described in this section, and they MUST use it 👑✨.

2.10.1. Motivation and Overview

okay so previous versions and minor versions of NFS had some real problems 😭. Here's the tea:

No support for Exactly Once Semantics (EOS). This includes no support for EOS through server failure and recovery 💀.
Limited callback support, including no support for sending callbacks through firewalls, and races between replies to normal requests and callbacks 🧱.
Limited trunking over multiple network paths 🛤️.
Requiring machine credentials for fully secure operation 🔐.

Through the introduction of a session, NFSv4.1 fixes all of the above with practical solutions 💅:

EOS is enabled by a reply cache with a bounded size. That makes it feasible to keep the cache in persistent storage and enable EOS through server failure and recovery 🦋. One reason earlier revisions of NFS didn't support EOS was that some EOS approaches often limited parallelism. As explained in Section 2.10.6, NFSv4.1 supports both EOS and unlimited parallelism 🌈.
The NFSv4.1 client (defined in Section 1.7) creates transport connections and provides them to the server to use for sending callback requests, which solves the firewall issue (Section 18.34) 🔥. Races between responses to client requests and the callbacks those requests cause are detected through the session's sequencing properties, which come from EOS (Section 2.10.6.3) ⭐.
The NFSv4.1 client can associate an arbitrary number of connections with the session, which gives it trunking (Section 2.10.5) 🛤️.
The NFSv4.1 client and server produce a session key that is independent of client and server machine credentials. It can be used to compute a digest for protecting critical session management operations (Section 2.10.8.3) 🔑.
The NFSv4.1 client can also create secure RPCSEC_GSS contexts for use by the session's backchannel, and those don't require the server to authenticate to a client machine principal (Section 2.10.8.2) 🧚.

A session is a dynamically created, long-lived server object that a client creates and uses over time from one or more transport connections 💜. Its job is to maintain the server's state relative to the connection(s) belonging to a client instance. This state is entirely independent of the connection itself, and the state exists whether or not the connection exists 🌸. A client may have one or more sessions associated with it, so client-associated state can be accessed through any of the sessions associated with that client's client ID, as long as connections are associated with those sessions 🎀. When no connections are associated with any of a client ID's sessions for an extended time, objects such as locks, opens, delegations, layouts, etc. are subject to expiration ⏳. The session works as an object representing a way for a client to access its associated client state on the server, and it's independent of the physical means of access to that state 🫶.

A single client may create multiple sessions. A single session MUST NOT serve multiple clients 🙅‍♀️.

2.10.2. NFSv4 Integration

Sessions are part of NFSv4.1, not NFSv4.0 🧚. Normally, a major infrastructure change like sessions would require a new major version number for an Open Network Computing (ONC) RPC program like NFS. But NFSv4 wraps its functionality in a single procedure, COMPOUND, and COMPOUND can support an arbitrary number of operations, so sessions got added to NFSv4.1 with little difficulty 💅✨. COMPOUND includes a minor version number field, and for NFSv4.1 that minor version is set to 1. When the NFSv4 server processes a COMPOUND with the minor version set to 1, it expects a different set of operations than it does for NFSv4.0 🌈. NFSv4.1 defines the SEQUENCE operation, which is required for every COMPOUND that operates over an established session, with the exception of some session administration operations, such as DESTROY_SESSION (Section 18.37) 👑.

2.10.2.1. SEQUENCE and CB_SEQUENCE

okay now the important part 🔥. In NFSv4.1, when the SEQUENCE operation is present, it MUST be the first operation in the COMPOUND procedure 💖. Its main purpose is to carry the session identifier, and the session identifier associates all the other operations in the COMPOUND procedure with a particular session 🦋. SEQUENCE also contains the information required for maintaining EOS (see Section 2.10.6) 🛡️. So session-enabled NFSv4.1 COMPOUND requests look like this:

       +-----+--------------+-----------+------------+-----------+----
       | tag | minorversion | numops    |SEQUENCE op | op + args | ...
       |     |   (== 1)     | (limited) |  + args    |           |
       +-----+--------------+-----------+------------+-----------+----

and the replies look like this:

       +------------+-----+--------+-------------------------------+--//
       |last status | tag | numres |status + SEQUENCE op + results |  //
       +------------+-----+--------+-------------------------------+--//
               //-----------------------+----
               // status + op + results | ...
               //-----------------------+----

A CB_COMPOUND procedure request and reply has a similar form to COMPOUND, but instead of a SEQUENCE operation, it has a CB_SEQUENCE operation 🌸. CB_COMPOUND also has an extra field called "callback_ident", which is superfluous in NFSv4.1 and MUST be ignored by the client 🙈. CB_SEQUENCE has the same information as SEQUENCE, and it also includes other info needed to resolve callback races (Section 2.10.6.3) 🫶.

2.10.2.2. Client ID and Session Association ✨

Okay so, bestie, each client ID (Section 2.4) can have zero or more active sessions 💖. A client ID and its associated session are REQUIRED to do file access in NFSv4.1 🎀. Every single time a session gets used (whether the client is sending a request to the server, or the client is replying to a callback request from the server), the state leased to its associated client ID gets automatically renewed 🫶.

Now, state (which can be share reservations, locks, delegations, and layouts (Section 1.8.4)) is tied to the client ID. It is NOT tied to any individual session 💅. So successive state changing operations from a given state owner MAY go over different sessions, as long as the session is associated with the same client ID 🦋. A callback MAY arrive over a different session than the one that originally acquired the state the callback is about ✨. For example, if session A is used to acquire a delegation, a request to recall that delegation MAY arrive over session B, as long as both sessions are associated with the same client ID 🌸. The security considerations around callbacks are discussed in Sections 2.10.8.1 and 2.10.8.2 🙈💀.

2.10.3. Channels 🎀

A channel is NOT a connection 💅. A channel represents the direction ONC RPC requests are sent in 🦋.

Each session has one or two channels: the fore channel and the backchannel 🧚. Because there are at most two channels per session, and because each channel has a distinct purpose, channels are not assigned identifiers 🌸.

The fore channel is used for ordinary requests from the client to the server, and it carries COMPOUND requests and responses 💖. A session ALWAYS has a fore channel, no cap 👑.

The backchannel is used for callback requests from server to client, and it carries CB_COMPOUND requests and responses 🔥. Whether or not there is a backchannel is decided by the client. However, lowkey many features of NFSv4.1 REQUIRE a backchannel 😭. NFSv4.1 servers MUST support backchannels 🫶.

Each session has resources for each channel, including separate reply caches (see Section 2.10.6.1) ✨. Note that even the backchannel REQUIRES a reply cache (or at least a slot table, so it can detect retries), because some callback operations are non-idempotent 💀. Not me getting a retry storm 🙈.

2.10.3.1. Association of Connections, Channels, and Sessions 🦋

Each channel is associated with zero or more transport connections, whether they're the same transport protocol or different ones 🌈. A connection can be associated with one channel or with both channels of a session 💜. The client and server negotiate whether a connection will carry traffic for one channel or both via the CREATE_SESSION (Section 18.36) and BIND_CONN_TO_SESSION (Section 18.34) operations 🍓. When a session is created via CREATE_SESSION, the connection that carried the CREATE_SESSION request is automatically associated with the fore channel, and optionally the backchannel ✨. If the client specifies no state protection (Section 18.35) when the session is created, then when SEQUENCE is sent on a different connection, that connection is automatically associated with the fore channel of the session named in the SEQUENCE operation 🫶.

A connection's association with a session is NOT exclusive 💅. A connection associated with the channel(s) of one session MAY at the same time be associated with the channel(s) of other sessions, including sessions tied to other client IDs 🦋.

It is permissible for connections of multiple transport types to be associated with the same channel 🌸. For example, both TCP and RDMA connections can be associated with the fore channel 💖. If an RDMA connection and a non-RDMA connection are associated with the same channel, the maximum number of slots SHOULD be at least one more than the total number of RDMA credits (Section 2.10.6.1) 🔥. This way, if all RDMA credits are used, the non-RDMA connection can still have at least one outstanding request 👑. If a server supports multiple transport types, it MUST allow a client to associate connections from each transport with a channel 🫶.

It is permissible for a connection of one transport type to be associated with the fore channel, and a connection of a different type to be associated with the backchannel 🧚.

2.10.4. Server Scope 🌈

Servers each specify a server scope value, an opaque string called eir_server_scope, returned as part of the results of an EXCHANGE_ID operation 💜. The whole point of server scope is to let a group of servers tell clients that the servers sharing the same server scope value have agreed to use distinct values of opaque identifiers, so that the two servers NEVER assign the same value to two distinct objects 🍓. So the identifiers generated by two servers in that set can be assumed compatible, which means that in certain important cases, an identifier generated by one server in the set may be presented to another server with the same scope ✨.

Compatible values do NOT mean a value generated by one server will always be accepted by another 🙈. In most cases, it won't be 😭. But a server will not inadvertently accept a value generated by another server 🫶. When it does accept one, it will be because the value is recognized as valid and carries the same meaning as it does on another server of the same scope 💖.

When servers have the same server scope, this compatibility of values applies to the following identifiers:

Filehandle values. A filehandle value accepted by two servers of the same server scope denotes the same object 🎀. A WRITE operation sent to one server is reflected immediately in a READ sent to the other 🔥.
Server owner values. When the server scope values are the same, server owner values MAY be validly compared 🦋. When the server scope values are different, server owner values are treated as different even if they contain identical strings of bytes 💅.

The coordination among servers required to provide this compatibility can be quite minimal, and limited to a simple partition of the ID space 🌸. Recognizing common values requires additional implementation, but that can be tailored to the specific situations where that recognition is desired ✨.

Clients will have occasion to compare the server scope values of multiple servers under a number of circumstances, each of which is discussed under the appropriate functional section 🧚:

When server owner values received in response to EXCHANGE_ID operations sent to multiple network addresses are compared, in order to determine the validity of various forms of trunking, as described in Section 11.5.2 💜.
When network or server reconfiguration causes the same network address to possibly be directed to different servers, and the client has to determine when lock reclaim should be attempted, as described in Section 8.4.2.1 🍓.

When two replies from EXCHANGE_ID, each from a different server network address, have the same server scope, there are a number of ways a client can validate that the common server scope is due to two servers cooperating in a group 🌈.

If both EXCHANGE_ID requests were sent with RPCSEC_GSS ([4], [9], [27]) authentication and the server principal is the same for both targets, the equality of server scope is validated. It is RECOMMENDED that two servers intending to share the same server scope and server_owner major_id also share the same principal name. In some cases, this simplifies the client's task of validating server scope 👑.
The client MAY accept the appearance of the second server in the fs_locations or fs_locations_info attribute for a relevant file system. For example, if there is a migration event for a particular file system, or there are locks to be reclaimed on a particular file system, the attributes for that file system may be used 🫶. The client sends the GETATTR request to the first server for the fs_locations or fs_locations_info attribute with RPCSEC_GSS authentication. It may need to do this in advance of the need to verify the common server scope 🔥. If the client successfully authenticates the reply to GETATTR, and the GETATTR request and reply containing the fs_locations or fs_locations_info attribute refer to the second server, then the equality of server scope is supported 💖. A client may choose to limit the use of this form of support to information relevant to the specific file system involved (e.g., a file system being migrated) 🦋✨.

2.10.5. Trunking ✨

Okay so trunking is when a client and server use multiple connections at once to make data transfer faster 🏎️💨 NFSv4.1 supports two types: session trunking and client ID trunking. Both are iconic, fr.

When all the connections are using a single server network address, it's safe to assume they're all reaching the same server, and NFSv4.1 servers MUST support both forms of trunking. When multiple connections use a set of network addresses to reach the same server, the server MUST support both forms of trunking too 🫶 NFSv4.1 servers in a clustered setup MAY let network addresses for different servers use client ID trunking 🌸

Clients can use either form of trunking, as long as they don't break the servers' rules about which kinds of trunking are allowed when they trunk between different server network addresses (more on that below) 💅 For callback channels, the client MUST let the server pick among all the callback channels that are valid for a given client ID, and it MUST support trunking when the connections carrying the backchannel allow session or client ID trunking to be used for callbacks 🔥

Session Trunking. This is basically tying multiple connections, each with possibly different target and/or source network addresses, to the same session 🦋 When the target network addresses (the server addresses) of the two connections are the same, the server MUST support that session trunking. When the target network addresses are different, the server MAY say it supports this by using the data returned from the EXCHANGE_ID operation (see below) ✨

Client ID Trunking. This is tying multiple sessions to the same client ID 👑 Servers MUST support client ID trunking for two target network addresses whenever they allow session trunking for those same two network addresses. On top of that, a server MAY, by presenting the same major server owner ID (Section 2.5) and server scope (Section 2.10.4), allow an extra case of client ID trunking. When two servers return the same major server owner and server scope, it means they're cooperating on locking state management, and that's a must-have prerequisite for client ID trunking 🧚

Figuring out when a client is allowed to use session trunking versus client ID trunking means understanding how the results of the EXCHANGE_ID (Section 18.35) operation identify a server 🔍 Say a client sends EXCHANGE_IDs over two different connections, each with a possibly different target network address, but both EXCHANGE_ID operations have the same value in the eia_clientowner field. If the same NFSv4.1 server is listening over each connection, then each EXCHANGE_ID result MUST return the same values of eir_clientid, eir_server_owner.so_major_id, and eir_server_scope. The client can then treat each connection as referring to the same server (subject to verification, see Section 2.10.5.1 below), and it can use each connection to trunk requests and replies 💖 The client's choice is whether session trunking or client ID trunking applies 🎀

Session Trunking. If the eia_clientowner argument is the same in two different EXCHANGE_ID requests, and the eir_clientid, eir_server_owner.so_major_id, eir_server_owner.so_minor_id, and eir_server_scope results match in both EXCHANGE_ID results, then the client is permitted to perform session trunking 🌈 If the client has no session mapping to the tuple of eir_clientid, eir_server_owner.so_major_id, eir_server_scope, and eir_server_owner.so_minor_id, then it creates the session via a CREATE_SESSION operation over one of the connections, which associates the connection with the session. If there is already a session for the tuple, the client can send BIND_CONN_TO_SESSION to associate the connection with the session 🫧

Of course, if the client doesn't want to use session trunking, it isn't required to do so 🙈 It can invoke CREATE_SESSION on the connection, which will result in client ID trunking as described below. It can also decide to drop the connection if it doesn't want to trunk at all 💀

Client ID Trunking. If the eia_clientowner argument is the same in two different EXCHANGE_ID requests, and the eir_clientid, eir_server_owner.so_major_id, and eir_server_scope results match in both EXCHANGE_ID results, then the client is permitted to perform client ID trunking. This is true regardless of whether the eir_server_owner.so_minor_id results match 🍓 The client can associate each connection with different sessions, where each session is associated with the same server ✨

The client completes client ID trunking by invoking CREATE_SESSION on each connection, using the same client ID that was returned in eir_clientid. These invocations create two sessions and also associate each connection with its respective session 🌸 The client is free to pass on client ID trunking by simply dropping the connection at this point 🦋

When doing client ID trunking, locking state is shared across the sessions associated with that same client ID. This means the server has to coordinate state across sessions, and the client has to be able to associate the same locking state with multiple sessions 🔐

It's always possible that, because of various kinds of reconfiguration events, eir_server_scope and eir_server_owner values may be different on later EXCHANGE_ID requests made to the same network address 😭

In most cases, reconfiguration events like these are disruptive, and they mean an IP address that used to be connected to one server is now connected to a totally different one 💅

Here are some guidelines for how clients should handle these situations 🫶

When eir_server_scope changes, the client has no assurance that any IDs it got earlier (e.g., filehandles) can be validly used on the new server. Even if the new server accepts them, there's no assurance that this isn't just an accident. So it's best to treat all of that state as lost or stale, although a client may assume the probability of inadvertent acceptance is low and treat this situation as falling under the next case 🙈
When eir_server_scope stays the same but eir_server_owner.so_major_id changes, the client can keep using the filehandles it has, consider its locking state lost, and try to reclaim or otherwise re-obtain its locks. It might find that a filehandle is now stale. However, if NFS4ERR_STALE is not returned, it can proceed to reclaim or otherwise re-obtain its open locking state 👑
When eir_server_scope and eir_server_owner.so_major_id both stay the same, the client has to use the now-current values of eir_server_owner.so_minor_id when deciding on the right forms of trunking. This may result in connections being dropped or new sessions being created 🌈

2.10.5.1. Verifying Claims of Matching Server Identity 🔍

When the server responds over two different connections that claim matching or partially matching eir_server_owner, eir_server_scope, and eir_clientid values, the client does NOT have to trust the servers' claims 🙅‍♀️ The client may verify these claims before trunking traffic in the following ways:

For session trunking, clients SHOULD reliably verify whether connections between different network paths are actually associated with the same NFSv4.1 server and usable on the same session, and servers MUST allow clients to perform reliable verification. When a client ID is created, the client SHOULD specify that BIND_CONN_TO_SESSION is to be verified according to the SP4_SSV or SP4_MACH_CRED (Section 18.35) state protection options. For SP4_SSV, reliable verification depends on a shared secret (the SSV) that is established via the SET_SSV (see Section 18.47) operation 🔐

When a new connection is associated with the session (via the BIND_CONN_TO_SESSION operation, see Section 18.34), if the client specified SP4_SSV state protection for the BIND_CONN_TO_SESSION operation, the client MUST send the BIND_CONN_TO_SESSION with RPCSEC_GSS protection, using integrity or privacy, and an RPCSEC_GSS handle created with the GSS SSV mechanism (see Section 2.10.9) 🛡️

If the client mistakenly tries to associate a connection with a session on the wrong server, the server will either reject the attempt because it doesn't recognize the session identifier in the BIND_CONN_TO_SESSION arguments, or it will reject the attempt because the RPCSEC_GSS authentication fails. Even if the server mistakenly or maliciously accepts the connection association attempt, the RPCSEC_GSS verifier it computes in the response will not be verified by the client, so the client will know it can't use the connection for trunking the specified session 😤

If the client specified SP4_MACH_CRED state protection, the BIND_CONN_TO_SESSION operation will use RPCSEC_GSS integrity or privacy, using the same credential that was used when the client ID was created. Mutual authentication via RPCSEC_GSS assures the client that the connection is associated with the correct session on the correct server 🥰

For client ID trunking, the client has at least two options for verifying that the same client ID obtained from two different EXCHANGE_ID operations came from the same server 🧐 The first option is to use RPCSEC_GSS authentication when sending each EXCHANGE_ID operation. Each time an EXCHANGE_ID is sent with RPCSEC_GSS authentication, the client notes the principal name of the GSS target. If the EXCHANGE_ID results indicate that client ID trunking is possible, and the GSS targets' principal names are the same, then the servers are the same and client ID trunking is allowed ✨

The second option for verification is to use SP4_SSV protection. When the client sends EXCHANGE_ID, it specifies SP4_SSV protection. The first EXCHANGE_ID the client sends always has to be confirmed by a CREATE_SESSION call. The client then sends SET_SSV. Later, the client sends EXCHANGE_ID to a second destination network address, different from the one the first EXCHANGE_ID was sent to. The client checks that each EXCHANGE_ID reply has the same eir_clientid, eir_server_owner.so_major_id, and eir_server_scope. If so, the client verifies the claim by sending a CREATE_SESSION operation to the second destination address, protected with RPCSEC_GSS integrity using an RPCSEC_GSS handle returned by the second EXCHANGE_ID. If the server accepts the CREATE_SESSION request, and if the client verifies the RPCSEC_GSS verifier and integrity codes, then the client has proof that the second server knows the SSV. That means the two servers are cooperating for the purposes of specifying server scope and client ID trunking 🎉

2.10.6. Exactly Once Semantics ✨

okay bestie, this is the part where NFSv4.1 goes "nobody runs the same request twice on my watch" 💅 Via the session, NFSv4.1 offers exactly once semantics (EOS) for requests sent over a channel. EOS is supported on both the fore channel and backchannel 🎀

Each COMPOUND or CB_COMPOUND request that is sent with a leading SEQUENCE or CB_SEQUENCE operation MUST be executed by the receiver exactly once. This requirement holds regardless of whether the request is sent with reply caching specified (see Section 2.10.6.1.3). The requirement holds even if the requester is sending the request over a session created between a pNFS data client and pNFS data server 🫶 To understand the rationale for this requirement, divide the requests into three classifications:

Non-idempotent requests.
Idempotent modifying requests.
Idempotent non-modifying requests.

An example of a non-idempotent request is RENAME. Obviously, if a replier executes the same RENAME request twice, and the first execution succeeds, the re-execution will fail 💀 If the replier returns the result from the re-execution, this result is incorrect. Therefore, EOS is required for non-idempotent requests. Not me being dramatic about a RENAME, but it's literally the reason we're here 😭

An example of an idempotent modifying request is a COMPOUND request containing a WRITE operation. Repeated execution of the same WRITE has the same effect as execution of that WRITE a single time. Nevertheless, enforcing EOS for WRITEs and other idempotent modifying requests is necessary to avoid data corruption 🚨

Suppose a client sends WRITE A to a noncompliant server that does not enforce EOS, and receives no response, perhaps due to a network partition. The client reconnects to the server and re-sends WRITE A. Now, the server has outstanding two instances of A. The server can be in a situation in which it executes and replies to the retry of A, while the first A is still waiting in the server's internal I/O system for some resource. Upon receiving the reply to the second attempt of WRITE A, the client believes its WRITE is done so it is free to send WRITE B, which overlaps the byte-range of A. When the original A is dispatched from the server's I/O system and executed (thus the second time A will have been written), then what has been written by B can be overwritten and thus corrupted. 😱 This is the storyline nobody wants, bestie, so the spec is strict about it.

An example of an idempotent non-modifying request is a COMPOUND containing SEQUENCE, PUTFH, READLINK, and nothing else. The re-execution of such a request will not cause data corruption or produce an incorrect result. Nonetheless, to keep the implementation simple, the replier MUST enforce EOS for all requests, whether or not idempotent and non-modifying. Simple is iconic, periodt 💖

Note that true and complete EOS is not possible unless the server persists the reply cache in stable storage, and unless the server is somehow implemented to never require a restart (indeed, if such a server exists, the distinction between a reply cache kept in stable storage versus one that is not is one without meaning). See Section 2.10.6.5 for a discussion of persistence in the reply cache. Regardless, even if the server does not persist the reply cache, EOS improves robustness and correctness over previous versions of NFS because the legacy duplicate request/reply caches were based on the ONC RPC transaction identifier (XID). Section 2.10.6.1 explains the shortcomings of the XID as a basis for a reply cache and describes how NFSv4.1 sessions improve upon the XID 🌸

2.10.6.1. Slot Identifiers and Reply Cache 🦋

The RPC layer provides a transaction ID (XID), which, while required to be unique, is not convenient for tracking requests for two reasons. First, the XID is only meaningful to the requester; it cannot be interpreted by the replier except to test for equality with previously sent requests. When consulting an RPC-based duplicate request cache, the opaqueness of the XID requires a computationally expensive look up (often via a hash that includes XID and source address). NFSv4.1 requests use a non-opaque slot ID, which is an index into a slot table, which is far more efficient 🧚 Second, because RPC requests can be executed by the replier in any order, there is no bound on the number of requests that may be outstanding at any time. To achieve perfect EOS, using ONC RPC would require storing all replies in the reply cache. XIDs are 32 bits; storing over four billion (2^(32)) replies in the reply cache is not practical. In practice, previous versions of NFS have chosen to store a fixed number of replies in the cache, and to use a least recently used (LRU) approach to replacing cache entries with new entries when the cache is full. In NFSv4.1, the number of outstanding requests is bounded by the size of the slot table, and a sequence ID per slot is used to tell the replier when it is safe to delete a cached reply ✨

In the NFSv4.1 reply cache, when the requester sends a new request, it selects a slot ID in the range 0..N, where N is the replier's current maximum slot ID granted to the requester on the session over which the request is to be sent. The value of N starts out as equal to ca_maxrequests - 1 (Section 18.36), but can be adjusted by the response to SEQUENCE or CB_SEQUENCE as described later in this section. The slot ID must be unused by any of the requests that the requester has already active on the session. "Unused" here means the requester has no outstanding request for that slot ID.

Hmm, wait, I need to keep the key words exact, so let me restate that last sentence precisely: the slot ID MUST be unused by any of the requests that the requester has already active on the session. "Unused" here means the requester has no outstanding request for that slot ID.

Okay, the original text does not use MUST there, so I'm not adding one. Moving on 💅

A slot contains a sequence ID and the cached reply corresponding to the request sent with that sequence ID. The sequence ID is a 32-bit unsigned value, and is therefore in the range 0..0xFFFFFFFF (2^(32) - 1). The first time a slot is used, the requester MUST specify a sequence ID of one (Section 18.36). Each time a slot is reused, the request MUST specify a sequence ID that is one greater than that of the previous request on the slot. If the previous sequence ID was 0xFFFFFFFF, then the next request for the slot MUST have the sequence ID set to zero (i.e., (2^(32) - 1) + 1 mod 2^(32)) 🔥

The sequence ID accompanies the slot ID in each request. It is for the critical check at the replier: it is used to efficiently determine whether a request using a certain slot ID is a retransmit or a new, never-before-seen request. It is not feasible for the requester to assert that it is retransmitting to implement this, because for any given request the requester cannot know whether the replier has seen it unless the replier actually replies. Of course, if the requester has seen the reply, the requester would not retransmit 🙈

The replier compares each received request's sequence ID with the last one previously received for that slot ID, to see if the new request is:

A new request, in which the sequence ID is one greater than that previously seen in the slot (accounting for sequence wraparound). The replier proceeds to execute the new request, and the replier MUST increase the slot's sequence ID by one.
A retransmitted request, in which the sequence ID is equal to that currently recorded in the slot. If the original request has executed to completion, the replier returns the cached reply. See Section 2.10.6.2 for direction on how the replier deals with retries of requests that are still in progress.
A misordered retry, in which the sequence ID is less than (accounting for sequence wraparound) that previously seen in the slot. The replier MUST return NFS4ERR_SEQ_MISORDERED (as the result from SEQUENCE or CB_SEQUENCE).
A misordered new request, in which the sequence ID is two or more than (accounting for sequence wraparound) that previously seen in the slot. Note that because the sequence ID MUST wrap around to zero once it reaches 0xFFFFFFFF, a misordered new request and a misordered retry cannot be distinguished. Thus, the replier MUST return NFS4ERR_SEQ_MISORDERED (as the result from SEQUENCE or CB_SEQUENCE). 💀

Unlike the XID, the slot ID is always within a specific range; this has two implications. The first implication is that for a given session, the replier need only cache the results of a limited number of COMPOUND requests. The second implication derives from the first, which is that unlike XID-indexed reply caches (also known as duplicate request caches - DRCs), the slot ID-based reply cache cannot be overflowed. Through use of the sequence ID to identify retransmitted requests, the replier does not need to actually cache the request itself, reducing the storage requirements of the reply cache further. These facilities make it practical to maintain all the required entries for an effective reply cache 🌈

The slot ID, sequence ID, and session ID therefore take over the traditional role of the XID and source network address in the replier's reply cache implementation. This approach is considerably more portable and completely robust -- it is not subject to the reassignment of ports as clients reconnect over IP networks. In addition, the RPC XID is not used in the reply cache, enhancing robustness of the cache in the face of any rapid reuse of XIDs by the requester. While the replier does not care about the XID for the purposes of reply cache management (but the replier MUST return the same XID that was in the request), nonetheless there are considerations for the XID in NFSv4.1 that are the same as all other previous versions of NFS. The RPC XID remains in each message and needs to be formulated in NFSv4.1 requests as in any other ONC RPC request. The reasons include:

The RPC layer retains its existing semantics and implementation.
The requester and replier must be able to interoperate at the RPC layer, prior to the NFSv4.1 decoding of the SEQUENCE or CB_SEQUENCE operation.
If an operation is being used that does not start with SEQUENCE or CB_SEQUENCE (e.g., BIND_CONN_TO_SESSION), then the RPC XID is needed for correct operation to match the reply to the request.
The SEQUENCE or CB_SEQUENCE operation may generate an error. If so, the embedded slot ID, sequence ID, and session ID (if present) in the request will not be in the reply, and the requester has only the XID to match the reply to the request.

Given that well-formulated XIDs continue to be required, this raises the question: why do SEQUENCE and CB_SEQUENCE replies have a session ID, slot ID, and sequence ID? Having the session ID in the reply means that the requester does not have to use the XID to look up the session ID, which would be necessary if the connection were associated with multiple sessions. Having the slot ID and sequence ID in the reply means that the requester does not have to use the XID to look up the slot ID and sequence ID. Furthermore, since the XID is only 32 bits, it is too small to guarantee the re-association of a reply with its request [44]; having session ID, slot ID, and sequence ID in the reply allows the client to validate that the reply in fact belongs to the matched request 🎀

Okay so the SEQUENCE (and CB_SEQUENCE) op also carries a highest_slotid value 💅 which gives the replier extra info about how many slots the requester is actually using 🦋. The requester MUST always put in the slot ID of its outstanding request with the highest-numbered slot value 🫶. It should be as conservative as possible in all cases, though it can go a bit above what's really in use if it wants to keep some minimum or optimal level 🌸. This is how the requester hands unused request slots back to the replier, which can then reallocate them ✨.

The replier answers with two things: a new target highest_slotid and an enforced highest_slotid 🎀. Here's the tea:

The target highest_slotid tells the requester the highest highest_slotid the replier would like it to be using 💖. This lets the replier take back (or hand out) resources from a requester that isn't using them, so demand from everyone else gets shared more fairly 👑. The requester must always go along with the replier's updates, because they're newly set hard limits on what it can touch in the session 🔥. But since requests get pipelined, the requester might still have active requests in flight that reflect older values, so the replier must not immediately demand compliance 🙈.
The enforced highest_slotid is the highest slot ID the requester is allowed to use on a later SEQUENCE or CB_SEQUENCE op 💜. The replier's enforced highest_slotid SHOULD be no less than the highest_slotid the requester put in the SEQUENCE or CB_SEQUENCE arguments ✨.

Some requesters are super stubborn about lowering their highest_slotid argument on a Sequence op 💀. That means they keep ignoring the target highest_slotid in the response and keep setting their highest_slotid argument higher than the target. This can be seen as especially rude when the replier knows there are no outstanding requests with slot IDs above its target highest_slotid 😤. When that happens, the replier is free to get more forceful and MAY reply with a new enforced highest_slotid that's lower than its previous enforced highest_slotid 🧚. After that, if the requester keeps sending requests with a highest_slotid greater than the replier's new enforced highest_slotid, the server MAY return NFS4ERR_BAD_HIGH_SLOT, unless the slot ID in the request is greater than the new enforced highest_slotid and the request is a retry 🥲.

The replier SHOULD keep the slots it wants to retire until the requester sends a request with a highest_slotid less than or equal to the replier's new enforced highest_slotid 🍓.

The requester can also be stubborn about sending non-retry requests that have a slot ID bigger than the replier's highest_slotid 🙄. Once the replier has forcibly lowered the enforced highest_slotid, the requester is only allowed to send retries on slots that go above the replier's highest_slotid 🔒. If a request comes in with a slot ID higher than the new enforced highest_slotid, and its sequence ID is one higher than what's in that slot's reply cache, then the server can both retire the slot AND return NFS4ERR_BADSLOT 😭. (But the server MUST NOT do one and not the other, so no half-measures!) Retiring the slot is safe because by using the next sequence ID, the requester is basically saying "yep, I got the previous reply for this slot" 💅.

The requester SHOULD use the lowest available slot when it sends a new request 🌈. That way, the replier may be able to retire slot entries faster ⭐. But when the replier is actively adjusting its granted highest_slotid, it can't rely only on the slot ID and highest_slotid in the request to figure things out 🤔. Neither the slot ID nor the highest_slotid used in a request may reflect the replier's current idea of the requester's session limit, because the request might have been sent before the requester got the update 📬. So when things are being adjusted downward, the replier may have to keep a number of reply cache entries at least as large as the old maximum requests outstanding value, until it can tell that the requester has seen a reply with the new granted highest_slotid 🦋. The replier can tell that the requester has seen that reply when it gets a new request with the same slot ID as the request it replied to and the next higher sequence ID 🎀.

2.10.6.1.1. Caching of SEQUENCE and CB_SEQUENCE Replies

Whenever a SEQUENCE or CB_SEQUENCE op runs successfully, its reply MUST always be cached 💾. Specifically, the session ID, sequence ID, and slot ID MUST be cached in the reply cache 🫶. The SEQUENCE reply also includes the highest slot ID, target highest slot ID, and status flags. Instead of caching those values, the server MAY re-compute them from the current state of the fore channel, session, and/or client ID, as appropriate ✨. The CB_SEQUENCE reply includes a highest slot ID and target highest slot ID too. The client MAY re-compute those values from the current state of the session, as appropriate 🌸.

No matter whether a replier re-computes the highest slot ID, target slot ID, and status on replies to retries or not, the requester MUST NOT assume that those values are being re-computed whenever it gets a reply after sending a retry 🙈. It has no way of knowing if the reply it got was sent by the replier in response to the retry, or if it's just a late reply to the original request 🕰️. So the highest slot ID, target slot ID, or status bits might reflect how things looked when the request was first run 💀. Acting on that delayed info is totally valid, but it may make the receiver do unneeded work 😅. Requesters MAY choose to send extra requests to get the current state of affairs, or use the state reported by later requests, instead of acting right away on data that could be out of date 💜.

2.10.6.1.2. Errors from SEQUENCE and CB_SEQUENCE

Any time SEQUENCE or CB_SEQUENCE returns an error, the sequence ID of the slot MUST NOT change 🚫. The replier MUST NOT modify the reply cache entry for the slot whenever an error comes back from SEQUENCE or CB_SEQUENCE 🔒.

2.10.6.1.3. Optional Reply Caching

Per request, the requester can choose to tell the replier to cache the reply to all the operations after the first one (SEQUENCE or CB_SEQUENCE) using the sa_cachethis or csa_cachethis fields in the SEQUENCE or CB_SEQUENCE arguments 🎀. The reason it wouldn't tell the replier to cache the whole reply is that the request is made up of all idempotent operations [41] 🔁. Caching the reply may not buy much. If the reply is too big (see Section 2.10.6.4), it may not be cacheable anyway 📦. And even if the reply to an idempotent request is small enough to cache, caching it when it isn't needed just slows the server down and bumps up RPC latency 🐌.

Whether or not the requester asks for the reply to be cached has no effect on slot processing ✨. If the result of SEQUENCE or CB_SEQUENCE is NFS4_OK, then the slot's sequence ID MUST be incremented by one 📈. If a requester does not tell the replier to cache the reply, the replier MUST do one of the following 👇:

The replier can cache the entire original reply 🗂️. Even though sa_cachethis or csa_cachethis is FALSE, the replier is always free to cache. It might pick this route just to keep its implementation simple 🧸.
The replier puts into its reply cache a reply made up of the original results of the SEQUENCE or CB_SEQUENCE op, with the next operation in the COMPOUND or CB_COMPOUND returning the error NFS4ERR_RETRY_UNCACHED_REP 🌷. So if the requester later retries the request, it'll get NFS4ERR_RETRY_UNCACHED_REP 🔁. If a replier gets a retried Sequence op where the reply to the COMPOUND or CB_COMPOUND wasn't cached, then the replier:
- MAY return NFS4ERR_RETRY_UNCACHED_REP in reply to a Sequence op if the Sequence op is not the first operation (fair warning, a requester that does this is violating the NFSv4.1 protocol 🙅‍♀️).
- MUST NOT return NFS4ERR_RETRY_UNCACHED_REP in reply to a Sequence op if the Sequence op is the first operation 🚨.
If the second operation is an illegal operation, or an operation that was legal in a previous minor version of NFSv4 and MUST NOT be supported in the current minor version (for example, SETCLIENTID), the replier MUST NOT ever return NFS4ERR_RETRY_UNCACHED_REP 🙅. Instead, the replier MUST return NFS4ERR_OP_ILLEGAL, NFS4ERR_BADXDR, or NFS4ERR_NOTSUPP, as appropriate 🥀.
If the second operation can result in another error status, the replier MAY return a status other than NFS4ERR_RETRY_UNCACHED_REP, as long as the operation isn't executed in a way that changes the replier's state 🌟. Examples of such an error status include: NFS4ERR_NOTSUPP returned for an operation that's legal but not REQUIRED in the current minor versions, and so not supported by the replier 🫠; NFS4ERR_SEQUENCE_POS; and NFS4ERR_REQ_TOO_BIG 🧾.

The discussion above assumes that the retried request matches the original one 👯. Section 2.10.6.1.3.1 talks about what the replier might do, and MUST do, when the original and retried requests don't match 🔍. Since the replier may only cache a small amount of the info it would need to figure out whether this is a false retry, the replier may send the client any of the following responses 🫧:

The cached reply to the original request (if the replier has cached it in its entirety, and the users of the original request and retry match) 💌.
A reply that consists only of the Sequence op with the error NFS4ERR_SEQ_FALSE_RETRY 🚩.
A reply consisting of the response to Sequence with the status NFS4_OK, together with the second operation as it appeared in the retried request, with an error of NFS4ERR_RETRY_UNCACHED_REP or another error as described above 🌀.
A reply that consists of the response to Sequence with the status NFS4_OK, together with the second operation as it appeared in the original request, with an error of NFS4ERR_RETRY_UNCACHED_REP or another error as described above 🎭.

2.10.6.1.3.1. False Retry ✨

okay so bestie, this is the spot where the replier catches someone pretending to be a retry 👀💅 like, the requester sent a SEQUENCE with a slot ID and sequence ID that are already sitting in the reply cache, right? but the replier looks closer and goes "wait, this isn't the same request at all" 🙈 that could mean different operations, different arguments, or a totally different principal in the RPC credential that maps to a different user. and that's what we call a false retry 💀✨ main character energy for sure, but not in a good way lol

when the replier spots a false retry, it's allowed to send back NFS4ERR_SEQ_FALSE_RETRY in response to the SEQUENCE operation 🎀 (it doesn't HAVE to, though, it's optional, so no pressure 🫶)

now the user stuff is kinda important so pay attention 👑 if a particularly privileged user value gets translated to some other user because the replier's credentials aren't secure enough (that's configured on the replier), that translation should happen before the replier decides whether the users are the same or different 🧚 and if the replier figures out the users are different between the original and the retry, then it MUST return NFS4ERR_SEQ_FALSE_RETRY. no wiggle room, periodt 💜

this part is so important bestie!! 😭

next up, what if the retry has an operation that's illegal, or one that was legal in an earlier minor version but MUST NOT be supported anymore in the current one (SETCLIENTID is the example they give 🍓)? then the replier MAY return NFS4ERR_SEQ_FALSE_RETRY, and it MUST do so if the original and retry users are different 🔥 otherwise it MAY return NFS4ERR_OP_ILLEGAL, NFS4ERR_BADXDR, or NFS4ERR_NOTSUPP, whichever fits best ✨

one more thing that's lowkey confusing: this handling is kinda different from what happens when there's no cached reply at all 🦋 NFS4ERR_SEQ_FALSE_RETRY only makes sense for SEQUENCE ops, while NFS4ERR_RETRY_UNCACHED_REP is valid for basically every operation except the illegal ones and the ones that MUST NOT be supported in the current minor version 🌸 so yeah, two different error codes for two different situations, iykyk 🫠

2.10.6.2. Retry and Replay of Reply 💖

okay so now we talk about retries and replays, and the big rule is that a requester MUST NOT retry a request unless the connection it used to send it disconnects 🛑 once that happens, it can reconnect and re-send, or re-send over a different connection that's associated with the same session 🦋

if the requester is actually a server trying to re-send a callback over the session's backchannel, it can't reconnect because only the client can associate connections with the backchannel 😤 so the server can re-send over another connection bound to that same backchannel. and if there's no such connection, the server MUST indicate that the session has no backchannel by setting the SEQ4_STATUS_CB_PATH_DOWN_SESSION flag bit in the response to the client's next SEQUENCE operation 🌈 then the client MUST associate a connection with the session, or destroy the session 💅

it's not fatal to retry without a disconnect, just know that retries aren't free 🙈 especially with RDMA, where every request, retry or not, eats a credit 🍓 retries for no reason, especially right after the last attempt, are a waste of network bandwidth and kinda defeat the whole point of a transport's congestion control 😭

a requester MUST wait for a reply before using the slot for another request 💜 if it doesn't wait, it won't know what sequence ID to use next time, and that's a mess ✨ like, say a requester sends sequence ID 1 and then doesn't wait for the response. next time it uses that slot, it sends sequence ID 2. if the replier never saw sequence ID 1, it's not expecting 2, so it rejects the new request with NFS4ERR_SEQ_MISORDERED (from SEQUENCE or CB_SEQUENCE) 🫠

now RDMA stuff 🧚 RDMA fabrics don't guarantee that the memory handles (Steering Tags) in each RPC/RDMA "chunk" [32] stay valid outside a single connection 😬 so handles used by direct operations become invalid after a connection is lost. the server must make sure any RDMA operations that need replaying from the reply cache use the newly provided handle(s) from the most recent request 🎀

a retry might show up while the original is still running on the replier 👀 the replier SHOULD handle that by returning NFS4ERR_DELAY as the reply to SEQUENCE or CB_SEQUENCE, but implementations MAY return NFS4ERR_MISORDERED instead 🌸 errors from SEQUENCE and CB_SEQUENCE never get recorded in the reply cache, so the original's results can still be properly cached (assuming the requester asked for the reply to be cached) 🫶

2.10.6.3. Resolving Server Callback Races 🔥

so here's the situation: server callbacks can show up at the client before the reply to the related fore channel operations 😱 example: a client gets a delegation for a file it opened, but the OPEN reply that tells it about the delegation is stuck somewhere in the network 🐢 meanwhile a conflicting operation hits the server, which recalls the delegation over the backchannel. that backchannel might be a different transport connection, maybe even a different network, or a different session tied to the same client ID 🌍

the presence of a session basically fixes this 💖 when a session is in place, every client request is uniquely identified by its { session ID, slot ID, sequence ID } triple 🦋 and because of the rules for retiring slot entries (reply cache entries), the server knows whether the client has "seen" each of its replies ✨ so the server can give the client enough info to tell an erroneous or conflicting callback race apart from the real thing 👑

for each client operation that might lead to some kind of server callback, the server SHOULD "remember" that client request's { session ID, slot ID, sequence ID } triple until the slot ID retirement rules let it figure out the client has actually seen the reply 🧚 until that triple can be retired, any recalls of the related object MUST carry an array of these referring identifiers (inside the CB_SEQUENCE operation's arguments) so the client can use them 🎀 after that point, the server doesn't need to include this info anymore, since a race can't happen at that stage 🌸

the CB_SEQUENCE operation that starts each server callback carries a list of "referring" { session ID, slot ID, sequence ID } triples 💅 if the client finds the request matching the referring session ID, slot ID, and sequence ID is still outstanding (meaning the server's reply hasn't been seen by the client), it knows the callback raced the reply and can act on that 😤 if the client doesn't find the matching request outstanding (this includes a session ID for a destroyed session), then there's no race for that triple 🙈 and the server SHOULD limit the referring triples to just the requests that apply to the objects mentioned in the CB_COMPOUND procedure 🍓

the client must not just sit there forever waiting for the expected server reply before answering the CB_COMPOUND that won the race 😭 that reply could get delayed indefinitely 💀 the client should assume the reply will show up within the average round-trip time for COMPOUND requests to the server, and wait that long 🦋 if that time runs out, it can respond to the CB_COMPOUND with NFS4ERR_DELAY ✨ there are other situations where callbacks can race replies too, like pNFS layout recalls described in Section 12.5.5.2 👑 more parts coming, bestie 💖

2.10.6.4. COMPOUND and CB_COMPOUND Construction Issues 💅

okay bestie, huge requests and replies can cause a whole mess 😭 Buffer management gets tricky (especially with RDMA), and so does the reply cache. When the session is created (see Section 18.36), the client and server negotiate limits for each channel, fore and back. There are three of them: the biggest request they'll send or process (ca_maxrequestsize), the biggest reply they'll return or process (ca_maxresponsesize), and the biggest reply they'll stash in the reply cache (ca_maxresponsesize_cached) ✨

If a request goes over ca_maxrequestsize, the reply comes back with the status NFS4ERR_REQ_TOO_BIG. The replier MAY return NFS4ERR_REQ_TOO_BIG as the status for the first operation (SEQUENCE or CB_SEQUENCE) in the request. That means no operations in the request ran, and the slot in the reply cache is unchanged 🦋 Or it MAY opt to return it on a later operation in the same COMPOUND or CB_COMPOUND request. That means at least one operation did run, and the slot's state in the reply cache does change. The replier SHOULD set NFS4ERR_REQ_TOO_BIG on the operation that goes over ca_maxrequestsize.

If a reply goes over ca_maxresponsesize, the reply comes back with the status NFS4ERR_REP_TOO_BIG 🌸 The replier MAY return NFS4ERR_REP_TOO_BIG as the status for the first operation (SEQUENCE or CB_SEQUENCE) in the request. Or it MAY opt to return it on a later operation (in the same COMPOUND or CB_COMPOUND reply). The replier MAY also return NFS4ERR_REP_TOO_BIG in the reply to SEQUENCE or CB_SEQUENCE, even if the response would still exceed ca_maxresponsesize. Yes, it's allowed, and yes, it's kind of a plot twist 💀

Now the caching part, which is where the stakes get real 🔥 If sa_cachethis or csa_cachethis is TRUE, then the replier MUST cache a reply, except if an error is returned by the SEQUENCE or CB_SEQUENCE operation (see Section 2.10.6.1.2). If the reply exceeds ca_maxresponsesize_cached (and sa_cachethis or csa_cachethis is TRUE), then the server MUST return NFS4ERR_REP_TOO_BIG_TO_CACHE. Even if NFS4ERR_REP_TOO_BIG_TO_CACHE (or any other error, for that matter) is returned on an operation other than the first operation (SEQUENCE or CB_SEQUENCE), the reply MUST be cached if sa_cachethis or csa_cachethis is TRUE.

Here's an example so it clicks 🧚 Say a COMPOUND has eleven operations, including SEQUENCE. The fifth operation is a RENAME, and the tenth operation is a READ for one million bytes. The server may return NFS4ERR_REP_TOO_BIG_TO_CACHE on the tenth operation. But the server already executed several operations, including the non-idempotent RENAME. So the client's request to cache the reply needs to be honored for exactly once semantics to keep working correctly 😭 If the client retries, the server will have cached a reply that contains results for ten of the eleven requested operations, with the tenth operation having a status of NFS4ERR_REP_TOO_BIG_TO_CACHE. Iconic, but also a lot 💅

A client needs to be careful here. When it sends operations that change the current filehandle (except for PUTFH, PUTPUBFH, PUTROOTFH, and RESTOREFH), it must not exceed the maximum reply buffer before the GETFH operation. If it does, the client will have to retry the operation that changed the current filehandle, just to get the filehandle it wanted. For the OPEN operation (see Section 18.16), retry isn't always an option, so this matters a lot. These guidelines for handling filehandle-changing operations are advised:

Within the same COMPOUND procedure, a client SHOULD send GETFH immediately after a current filehandle-changing operation. A client MUST send GETFH after a current filehandle-changing operation that is also non-idempotent (e.g., the OPEN operation), unless the operation is RESTOREFH. RESTOREFH is the exception, because even though it is non-idempotent, the filehandle that RESTOREFH produced came from an operation that is either idempotent (e.g., PUTFH, LOOKUP) or non-idempotent (e.g., OPEN, CREATE). If the origin is non-idempotent, then because the client MUST send GETFH after the origin operation, the client can recover if RESTOREFH returns an error 🫶
A server MAY return NFS4ERR_REP_TOO_BIG or NFS4ERR_REP_TOO_BIG_TO_CACHE (if sa_cachethis is TRUE) on a filehandle-changing operation if the reply would be too large on the next operation.
A server SHOULD return NFS4ERR_REP_TOO_BIG or NFS4ERR_REP_TOO_BIG_TO_CACHE (if sa_cachethis is TRUE) on a filehandle-changing, non-idempotent operation if the reply would be too large on the next operation, especially if the operation is OPEN 👑
A server MAY return NFS4ERR_UNSAFE_COMPOUND to a non-idempotent current filehandle-changing operation if it looks at the next operation (in the same COMPOUND procedure) and finds that it is not GETFH. The server SHOULD do this if it is unable to determine in advance whether the total response size would exceed ca_maxresponsesize_cached or ca_maxresponsesize.

2.10.6.5. Persistence 🌈

Since the reply cache is bounded, it's practical for the reply cache to persist across server restarts 🎀 The replier MUST persist the following information if it agreed to persist the session (when the session was created; see Section 18.36):

The session ID.
The slot table, including the sequence ID and cached reply for each slot.

Those two things are enough for a replier to provide EOS semantics for any requests that were sent and executed before the server restarted ✨ If the replier is a client, it doesn't need to persist any more information, unless the client persists all other state across a client restart. In that case, the server will never see any NFSv4.1-level protocol manifestation of a client restart 🦋 If the replier is a server and only the slot table and session ID persist, then any requests the client retries after the server restart will return the results cached in the reply cache. Any new requests (i.e., the sequence ID is one greater than the slot's sequence ID) MUST be rejected with NFS4ERR_DEADSESSION (returned by SEQUENCE). Such a session is considered dead 💀 A server MAY re-animate a session after a server restart, so that the session accepts new requests as well as retries. To re-animate a session, the server needs to persist additional information through the restart:

The client ID. This is a prerequisite to let the client create more sessions associated with the same client ID as the re-animated session.
The client ID's sequence ID that is used for creating sessions (see Sections 18.35 and 18.36). This is a prerequisite to let the client create more sessions.
The principal that created the client ID. This allows the server to authenticate the client when it sends EXCHANGE_ID.
The SSV, if SP4_SSV state protection was specified when the client ID was created (see Section 18.35). This lets the client create new sessions and associate connections with the new and existing sessions.
The properties of the client ID as defined in Section 18.35.

A persistent reply cache puts certain demands on the server 😭 The execution of the sequence of operations (starting with SEQUENCE) and the placement of its results in the persistent cache MUST be atomic. If a client retries a sequence of operations that was previously executed on the server, the only acceptable outcomes are either the original cached reply, or an indication that the client ID or session has been lost. That second one would mean a catastrophic loss of the reply cache, or a session that was deleted because the client failed to use it for an extended period of time.

A server could fail and restart in the middle of a COMPOUND procedure that contains one or more non-idempotent or idempotent-but-modifying operations. That's an even bigger challenge for atomic execution and placement of results in the reply cache 🙈 One way to picture it is as a single transaction: each operation in the COMPOUND, followed by storing the result in persistent storage, and finally a transaction commit. If there's a failure before the transaction is committed, the server rolls back the transaction. If the server itself fails, then when it restarts, its recovery logic could roll back the transaction before starting the NFSv4.1 server.

The implementation details for atomic execution of the request and caching of the reply are beyond the scope of this document. An example implementation for NFSv2 [45] is described in [46] 🧠

2.10.7. RDMA Considerations 🦋

A complete discussion of the operation of RPC-based protocols over RDMA transports is in [32]. A discussion of the operation of NFSv4, including NFSv4.1, over RDMA is in [33]. Where RDMA is considered, this specification assumes the use of such a layering. It addresses only the upper-layer issues relevant to making the best use of RPC/RDMA 💖

2.10.7.1. RDMA Connection Resources ✨

RDMA requires its consumers to register memory and post buffers of a specific size and number for receive operations 🎀

Registration of memory can be a relatively high-overhead operation, since it requires pinning of buffers, assignment of attributes (e.g., readable/writable), and initialization of hardware translation. Preregistration is desirable to reduce overhead. These registrations are specific to hardware interfaces and even to RDMA connection endpoints, so negotiating their limits is desirable to manage resources effectively 💅

Following basic registration, these buffers must be posted by the RPC layer to handle receives. These buffers remain in use by the RPC/NFSv4.1 implementation. The size and number of them must be known to the remote peer, in order to avoid RDMA errors that would cause a fatal error on the RDMA connection 😭

NFSv4.1 manages slots as resources on a per-session basis (see Section 2.10), while RDMA connections manage credits on a per-connection basis. This means that in order for a peer to send data over RDMA to a remote buffer, it has to have both an NFSv4.1 slot and an RDMA credit. If multiple RDMA connections are associated with a session, and the total number of credits across all of them is X, while the number of slots in the session is Y, then the maximum number of outstanding requests is the lesser of X and Y 🔥

2.10.7.2. Flow Control 🌸

okay so, previous versions of NFS had NO flow control of their own 😭 they just leaned on the windowing from transports like TCP to throttle requests, and that's giving "trust the vibes" energy. 💀 RDMA is not about that life. It has no operation flow control at all, and if you blow past its limits it just terminates the connection in error. Harsh! 🙈 So NFSv4.1 negotiates limits when a session is created (check the ca_maxrequests field in Section 18.36 🎀). These limits set the maxima that each connection on the session's channel(s) has to stay within. RDMA connections get managed inside those limits like described in Section 3.3 of [32] ✨, and if there are multiple RDMA connections, the channel's max number of requests gets split between them. Put simply, the replier is the one who has to make sure the total RDMA credits across all of the replier's channel's connections does NOT exceed the channel's max number of outstanding requests. That's a whole responsibility, bestie 👑

The limits can also change dynamically whenever the replier wants, by tweaking certain parameters inside each NFSv4.1 reply. 🦋 And the CB_RECALL_SLOT callback operation (see Section 20.8) can be sent by a server to a client to hand RDMA credits back to the server. That lowers how many requests the client can have outstanding to the server. Iconic move, ngl 💅

2.10.7.3. Padding 🧚

okay the padding section is lowkey my favorite 🫶 Each peer asks for header padding at session initiation (see the ca_headerpadsize argument to CREATE_SESSION in Section 18.36 💖), and then the RPC RDMA layer uses it, as described in [32]. Zero padding is permitted. Zero padding is permitted, periodt 🌈

Here's the big idea: RDMA preserves alignment of data, even when the data gets placed into anonymous (untagged) buffers. ✨ So if padding is requested, client inline writes slide appropriate pad bytes into the request header so the data payload lands on the specified boundary. The client is encouraged to add enough padding (up to the negotiated size) so the "data" field of the WRITE operation is aligned. Most servers can absolutely use this 🙌 because it lets them chain receive buffers so any data in client requests drops straight into the right buffers at the server, ready for file system processing. The receiver's RPC layer skips over the pad bytes with zero overhead 💅, and the RDMA layer's high performance makes inserting and sending padding on the sender a big optimization. So servers don't need to do RDMA Read to grab all but the largest client writes anymore 🔥. Bonus: fewer message round trips on the network, which is a solid trade when latency is in the picture 🍓

Now the value you pick for padding has a bunch of criteria to juggle 🎀. The main source of variable-length data in the RPC header is the authentication info, and its form is picked by the client, maybe in response to what the server specifies. The contents of COMPOUNDs, the sizes of strings like the ones passed to RENAME, and so on all go into figuring out the max NFSv4.1 request size, and therefore the minimal buffer size. The client has to pick its offered value carefully so it doesn't overburden the server, and vice versa 🫶 A well-chosen padding value means higher performance. Period.

Sender gather:

RPC Request

Pad bytes

Length

User data...

------+----------------------/ \ \ \ \ Receiver scatter: -----------+- ... /-----+----------------\ \ \

RPC Request

Pad

Length

FS buffer

->...

In the diagram above, the server can recycle unused buffers into the next posted receive if the actual received request didn't use them, or it can pass the now-complete buffers by reference for normal write processing. For a server that can use it, this kills the need to copy incoming data 🙈, and you don't need complicated end-to-end buffer advertisement and management to get there. That includes most kernel-based and integrated server designs, among many others ✨. The client can do similar optimizations too, if it wants 💖

2.10.7.4. Dual RDMA and Non-RDMA Transports 🦋

Some RDMA transports (e.g., RFC 5040 [8]) allow a "streaming" (non-RDMA) phase, where ordinary traffic can flow before they "step up" to RDMA mode and start RDMA traffic. Others start connections in RDMA mode from the very beginning 🌸. NFSv4.1 allows a streaming phase before RDMA mode but does NOT assume one. When a connection gets associated with a session, the client and server negotiate whether the connection is used in RDMA or non-RDMA mode (see Sections 18.36 and 18.34) 👑

2.10.8. Session Security 🔥

2.10.8.1. Session Callback Security 🧚

Via session/connection association, NFSv4.1 improves security over what NFSv4.0 gave the backchannel 💅 The connection is client-initiated (see Section 18.34) and goes through the same firewall and routing checks as the fore channel. At the client's option (see Section 18.35), connection association is fully authenticated before it gets activated (see Section 18.34). Traffic from the server over the backchannel is authenticated exactly as the client specifies (see Section 2.10.8.2) ✨

2.10.8.2. Backchannel RPC Security 🎀

When the NFSv4.1 client sets up the backchannel, it tells the server which security flavors and principals to use when sending requests 🫶 If the flavor is RPCSEC_GSS, the client expresses the principal as an established RPCSEC_GSS context. The server is free to use any of the flavor/principal combinations the client offers, but it MUST NOT use combinations that weren't offered. Not me making the rules, but this is the rule 😭 This way the client doesn't need to provide a target GSS principal for the backchannel like it did with NFSv4.0, and the server doesn't have to implement an RPCSEC_GSS initiator like it did with NFSv4.0 [37] 🌈

The CREATE_SESSION (Section 18.36) and BACKCHANNEL_CTL (Section 18.33) operations let the client specify those flavor/principal combinations 💖

Also, the SP4_SSV state protection mode (see Sections 18.35 and 2.10.8.3) has a side benefit: it provides SSV-derived RPCSEC_GSS contexts (Section 2.10.9) 🦋

2.10.8.3. Protection from Unauthorized State Changes 🙈

Okay bestie, this is the scary part 💀 As described so far in the spec, NFSv4.1's state model is vulnerable to an attacker who sends a SEQUENCE operation with a forged session ID and a slot ID that the attacker expects the legit client to use next. If the legit client then uses that slot ID with the same sequence number, the server hands back the attacker's result from the reply cache. That disrupts the legit client and denies it service. Not okay 😤

It gets worse 😭 An attacker could also send a CREATE_SESSION with a forged client ID to make a new session tied to that client ID. Then the attacker could send requests on the new session that change locking state, like LOCKU operations that release locks the legit client acquired. One potential workaround is setting a security policy on the file that requires RPCSEC_GSS credentials whenever someone manipulates the file's state. But that has a downside: a legit client can't release state when RPCSEC_GSS is required and no GSS context can be obtained (maybe because the user logged off the client) 🥺

NFSv4.1 gives a client three options for state protection, which get specified when the client creates a client ID via EXCHANGE_ID (Section 18.35) ✨

The first one, SP4_NONE, just waives state protection entirely 🫠

The other two options, SP4_MACH_CRED and SP4_SSV, share a few traits:

An RPCSEC_GSS-based credential is used to authenticate client ID and session maintenance operations. That includes creating and destroying a session, associating a connection with the session, and destroying the client ID 🔥
Since RPCSEC_GSS authenticates client ID and session maintenance, the attacker can't associate a rogue connection with a legit session, or a rogue session with a legit client ID. That means they can't mess with the client ID's lock state via CLOSE, LOCKU, DELEGRETURN, LAYOUTRETURN, and so on 🙅‍♀️
When the server's security policies on part of its namespace require RPCSEC_GSS authentication, a client might have to use an RPCSEC_GSS credential to remove per-file state (e.g., LOCKU, CLOSE, etc.). The server may require that the principal removing the state match certain criteria (e.g., it might have to be the same principal that acquired the state). But the client might not have an RPCSEC_GSS context for that principal, and might not be able to create one (perhaps because the user logged off) 😩 When the client establishes SP4_MACH_CRED or SP4_SSV protection, it can specify a list of operations that the server MUST allow using the machine credential (if SP4_MACH_CRED is used) or the SSV credential (if SP4_SSV is used) 💖

The SP4_MACH_CRED option uses a machine credential, and the principal that creates the client ID MUST also be the principal that performs client ID and session maintenance operations. The security of this approach depends entirely on safeguarding the per-machine credential 🔐 As long as the per-machine credential is properly safeguarded for operations like CREATE_SESSION, BIND_CONN_TO_SESSION, DESTROY_SESSION, and DESTROY_CLIENTID, an attacker can't associate a rogue connection with a session, or a rogue session with a client ID 🛡️

There are at least three scenarios for SP4_MACH_CRED:

The system administrator configures a unique, permanent per-machine credential for one of the mandated GSS mechanisms (e.g., if Kerberos V5 is used, a "keytab" containing a principal derived from a client host name could be used) 🍓
The client is used by a single user, so the client ID and its sessions are used by just that user. If that user's credential expires, session and client ID maintenance can't happen, but since there's only one user, only that user is inconvenienced 🙃
The physical client has multiple users, but the client implementation gives each user a unique client ID. This is basically the same as the second scenario, but the downside is that each user needs at least one session, so the approach lacks economy 💅

The SP4_SSV protection option uses the SSV (Section 1.7), via RPCSEC_GSS and the SSV GSS mechanism (Section 2.10.9), to protect state from attack ✨ It's meant for a client with multiple active users, where a system administrator wants to avoid the hassle of installing a permanent machine credential on every client 👑 The SSV is established and updated on the server via SET_SSV (see Section 18.47). To prevent eavesdropping, a client SHOULD send SET_SSV via RPCSEC_GSS with the privacy service 🫶 Several aspects of the SSV make it intractable for an attacker to guess, so they can't associate rogue connections with a session, or rogue sessions with a client ID 🔒

okay so the arguments to and results of SET_SSV include digests of the old and new SSV, respectively ✨
bestie, the initial value of the SSV is zero, so it's known 🫶 The client that opts for SP4_SSV protection, and opts to apply SP4_SSV protection to BIND_CONN_TO_SESSION and CREATE_SESSION, MUST send at least one SET_SSV operation before the first BIND_CONN_TO_SESSION operation or before the second CREATE_SESSION operation on a client ID. If it doesn't, the SSV mechanism won't generate tokens (Section 2.10.9) 💀 A client SHOULD send SET_SSV as soon as a session is created 🌸
a SET_SSV request does not replace the SSV with the argument to SET_SSV 🙅‍♀️ Instead, the current SSV on the server is logically exclusive ORed (XORed) with the argument to SET_SSV 🧠 Each time a new principal uses a client ID for the first time, the client SHOULD send a SET_SSV with that principal's RPCSEC_GSS credentials, with RPCSEC_GSS service set to RPC_GSS_SVC_PRIVACY 🔒

Here are the types of attacks that can be attempted by an attacker named Eve on a victim named Bob, and how SP4_SSV protection foils each attack 😤🔥

Suppose Eve is the first user to log into a legitimate client 👀 Eve's use of an NFSv4.1 file system will cause the legitimate client to create a client ID with SP4_SSV protection, specifying that the BIND_CONN_TO_SESSION operation MUST use the SSV credential 🚨 Eve's use of the file system also causes an SSV to be created. The SET_SSV operation that creates the SSV will be protected by the RPCSEC_GSS context created by the legitimate client, which uses Eve's GSS principal and credentials. Eve can eavesdrop on the network while her RPCSEC_GSS context is created and the SET_SSV using her context is sent. Even if the legitimate client sends the SET_SSV with RPC_GSS_SVC_PRIVACY, because Eve knows her own credentials, she can decrypt the SSV 😭 Eve can compute an RPCSEC_GSS credential that BIND_CONN_TO_SESSION will accept, and so associate a new connection with the legitimate session. Eve can change the slot ID and sequence state of a legitimate session, and/or the SSV state, in such a way that when Bob accesses the server via the same legitimate client, the legitimate client will be unable to use the session 💔

The client's only recourse is to create a new client ID for Bob to use, and establish a new SSV for the client ID. The client will be unable to delete the old client ID, and will let the lease on the old client ID expire 🥀

Once the legitimate client establishes an SSV over the new session using Bob's RPCSEC_GSS context, Eve can use the new session via the legitimate client, but she cannot disrupt Bob 💅 Moreover, because the client SHOULD have modified the SSV due to Eve using the new session, Bob cannot get revenge on Eve by associating a rogue connection with the session 😈

The question is how did the legitimate client detect that Eve has hijacked the old session? When the client detects that a new principal, Bob, wants to use the session, it SHOULD have sent a SET_SSV, which leads to the following sub-scenarios 🧐

Let us suppose that from the rogue connection, Eve sent a SET_SSV with the same slot ID and sequence ID that the legitimate client later uses. The server will assume the SET_SSV sent with Bob's credentials is a retry, and return to the legitimate client the reply it sent Eve. However, unless Eve can correctly guess the SSV the legitimate client will use, the digest verification checks in the SET_SSV response will fail. That is an indication to the client that the session has apparently been hijacked 🚩
Alternatively, Eve sent a SET_SSV with a different slot ID than the legitimate client uses for its SET_SSV. Then the digest verification of the SET_SSV sent with Bob's credentials fails on the server, and the error returned to the client makes it apparent that the session has been hijacked 🫣
Alternatively, Eve sent an operation other than SET_SSV, but with the same slot ID and sequence that the legitimate client uses for its SET_SSV. The server returns to the legitimate client the response it sent Eve. The client sees that the response is not at all what it expects. The client assumes either session hijacking or a server bug, and either way destroys the old session 💥

Eve associates a rogue connection with the session as above, and then destroys the session. Again, Bob goes to use the server from the legitimate client, which sends a SET_SSV using Bob's credentials. The client receives an error that indicates that the session does not exist. When the client tries to create a new session, this will fail because the SSV it has does not match that which the server has, and now the client knows the session was hijacked 🕵️‍♀️ The legitimate client establishes a new client ID 🌈
If Eve creates a connection before the legitimate client establishes an SSV, because the initial value of the SSV is zero and therefore known, Eve can send a SET_SSV that will pass the digest verification check. However, because the new connection has not been associated with the session, the SET_SSV is rejected for that reason 🙃

In summary, an attacker's disruption of state when SP4_SSV protection is in use is limited to the formative period of a client ID, its first session, and the establishment of the SSV 🦋 Once a non-malicious user uses the client ID, the client quickly detects any hijack and rectifies the situation 🎀 Once a non-malicious user successfully modifies the SSV, the attacker cannot use NFSv4.1 operations to disrupt the non-malicious user 👑

Note that neither the SP4_MACH_CRED nor SP4_SSV protection approaches prevent hijacking of a transport connection that has previously been associated with a session 🙈 If the goal of a counter-threat strategy is to prevent connection hijacking, the use of IPsec is RECOMMENDED 🛡️

⚠️ If a connection hijack occurs, the hijacker could in theory change locking state and negatively impact the service to legitimate clients 😱 However, if the server is configured to require the use of RPCSEC_GSS with integrity or privacy on the affected file objects, and if EXCHGID4_FLAG_BIND_PRINC_STATEID capability (Section 18.35) is in force, this will thwart unauthorized attempts to change locking state 🫶✨

2.10.9. The Secret State Verifier (SSV) GSS Mechanism ✨

Okay bestie, the SSV is giving secret key energy 🔐✨. It's a GSS mechanism that lives inside NFSv4.1, and NFSv4.1 uses it for state protection 💅. These contexts are not set up through RPCSEC_GSS. They get created automatically when EXCHANGE_ID specifies SP4_SSV protection. The only tokens that exist are the PerMsgToken (emitted by GSS_GetMIC) and the SealedMessage token (emitted by GSS_Wrap) 🦋.

The mechanism OID for the SSV mechanism is iso.org.dod.internet.private.enterprise.Michael Eisler.nfs.ssv_mech (1.3.6.1.4.1.28882.1.1) 🌸. The SSV mechanism doesn't define any initial context tokens, but the OID still lets servers say the SSV mechanism is acceptable whenever the client sends a SECINFO or SECINFO_NO_NAME operation (see Section 2.6). That's the whole point of it being in the list 🫶.

The SSV mechanism defines four subkeys derived from the SSV value 🔑. Every time SET_SSV is invoked, the client and server recalculate the subkeys. Each of the four subkeys depends on its own respective ssv_subkey4 enumerated value. The calculation uses the HMAC [52] algorithm, with the current SSV as the key, the one-way hash algorithm negotiated by EXCHANGE_ID, and the input text being the XDR encoded enumeration value for that subkey, of data type ssv_subkey4 🧚. Now here's the important bit 👀: if the HMAC output is longer than the key length of the encryption algorithm (also negotiated by EXCHANGE_ID), then the subkey MUST be truncated from the HMAC output. So if the subkey is N bytes long, the first N bytes of the HMAC output MUST be used for it. EXCHANGE_ID also says the HMAC output length MUST NOT be less than the subkey length the encryption algorithm needs (see Section 18.35). No shortcuts, periodt 💅.

/* Input for computing subkeys */
enum ssv_subkey4 {
        SSV4_SUBKEY_MIC_I2T     = 1,
        SSV4_SUBKEY_MIC_T2I     = 2,
        SSV4_SUBKEY_SEAL_I2T    = 3,
        SSV4_SUBKEY_SEAL_T2I    = 4
};

So the subkey from SSV4_SUBKEY_MIC_I2T is used for MICs (message integrity codes) that come from the NFSv4.1 client, whether that's part of a request on the fore channel or a response on the backchannel 💜. The subkey from SSV4_SUBKEY_MIC_T2I is for MICs coming from the NFSv4.1 server. The subkey from SSV4_SUBKEY_SEAL_I2T is for encrypted text coming from the client, and SSV4_SUBKEY_SEAL_T2I is for encrypted text coming from the server. Each direction gets its own key, bestie, it's so organized 🎀.

The PerMsgToken description is based on this XDR definition:

/* Input for computing smt_hmac */
struct ssv_mic_plain_tkn4 {
  uint32_t        smpt_ssv_seq;
  opaque          smpt_orig_plain<>;
};

/* SSV GSS PerMsgToken token */
struct ssv_mic_tkn4 {
  uint32_t        smt_ssv_seq;
  opaque          smt_hmac<>;
};

The field smt_hmac is an HMAC calculated using the subkey from SSV4_SUBKEY_MIC_I2T or SSV4_SUBKEY_MIC_T2I as the key, the one-way hash algorithm negotiated by EXCHANGE_ID, and the input text being the data of type ssv_mic_plain_tkn4 🌈. The field smpt_ssv_seq is the same as smt_ssv_seq. The field smpt_orig_plain is the "message" input passed to GSS_GetMIC() (see Section 2.3.1 of [7]). The caller of GSS_GetMIC() hands over a pointer to a buffer with the plain text. The SSV mechanism's entry point for GSS_GetMIC() encodes that into an opaque array, and the encoding includes an initial four-byte length plus any necessary padding 🍓. Then the XDR encoded value of smpt_ssv_seq gets prepended to it, which makes up an XDR encoding of a ssv_mic_plain_tkn4 value. That whole thing is the input to the HMAC 🧠.

The token emitted by GSS_GetMIC() is XDR encoded and is of XDR data type ssv_mic_tkn4. The field smt_ssv_seq comes from the SSV sequence number. That number equals one after SET_SSV (Section 18.47) is called for the first time on a client ID, and then it goes up by one on each SET_SSV after that 🔢. So smt_ssv_seq tells you which version of the SSV was in use when GSS_GetMIC() was called. As noted in Section 18.35, the client and server can keep multiple concurrent versions of the SSV. That means the SSV can change without having to serialize every RPC call that uses the SSV mechanism around SET_SSV operations. Once the HMAC is calculated, it's XDR encoded into smt_hmac, which includes an initial four-byte length and any necessary padding. The XDR encoded value of smt_ssv_seq is prepended to that ✨.

The SealedMessage description is based on this XDR definition:

/* Input for computing ssct_encr_data and ssct_hmac */
struct ssv_seal_plain_tkn4 {
  opaque          sspt_confounder<>;
  uint32_t        sspt_ssv_seq;
  opaque          sspt_orig_plain<>;
  opaque          sspt_pad<>;
};

/* SSV GSS SealedMessage token */
struct ssv_seal_cipher_tkn4 {
  uint32_t      ssct_ssv_seq;
  opaque        ssct_iv<>;
  opaque        ssct_encr_data<>;
  opaque        ssct_hmac<>;
};

The token emitted by GSS_Wrap() is XDR encoded and is of XDR data type ssv_seal_cipher_tkn4 🔒. The ssct_ssv_seq field means the same thing as smt_ssv_seq. The ssct_encr_data field is the result of encrypting an XDR encoded value of data type ssv_seal_plain_tkn4. The encryption key is the subkey from SSV4_SUBKEY_SEAL_I2T or SSV4_SUBKEY_SEAL_T2I, and the encryption algorithm is whatever EXCHANGE_ID negotiated 💖.

The ssct_iv field is the initialization vector (IV) for the encryption algorithm, if one applies, and it's sent in clear text 👀. The content and size of the IV MUST comply with the specification of the encryption algorithm. For example, the id-aes256-CBC algorithm MUST use a 16-byte initialization vector (IV), and that IV MUST be unpredictable for each instance of a ssv_seal_plain_tkn4 value that gets encrypted with a particular SSV key. Unpredictable IVs are non-negotiable, bestie 🙈.

The ssct_hmac field is the result of computing an HMAC over the XDR encoded ssv_seal_plain_tkn4 value as the input text. The key is the subkey from SSV4_SUBKEY_MIC_I2T or SSV4_SUBKEY_MIC_T2I, and the one-way hash algorithm is the one negotiated by EXCHANGE_ID 🌸.

The sspt_confounder field is a random value 🎲. The sspt_ssv_seq field is the same as ssvt_ssv_seq. The field sspt_orig_plain is the original plaintext, and it's the "input_message" input passed to GSS_Wrap() (see Section 2.3.3 of [7]). Just like the GSS_GetMIC() entry point, the GSS_Wrap() entry point expects a pointer to the plaintext. It XDR encodes an opaque array into sspt_orig_plain representing the plain text, along with the other fields of a ssv_seal_plain_tkn4 instance 🧁.

The sspt_pad field is there to support encryption algorithms that need their inputs in fixed-size blocks. The content of sspt_pad is zero filled, except for the length. Beware, though: the XDR encoding of ssv_seal_plain_tkn4 has three variable-length arrays. Each array uses four bytes for its length, and each array that follows its length is always padded to a multiple of four bytes, per the XDR standard 🙃.

Here's an example, so the math is crystal clear 🧮. Say the encryption algorithm uses 16-byte blocks, sspt_confounder is three bytes long, and sspt_orig_plain is 15 bytes long. The XDR encoding of sspt_confounder uses eight bytes (4 + 3 + 1-byte pad). The XDR encoding of sspt_ssv_seq uses four bytes. The XDR encoding of sspt_orig_plain uses 20 bytes (4 + 15 + 1-byte pad). The smallest XDR encoding of the sspt_pad field is four bytes. That adds up to 36 bytes. The next multiple of 16 is 48, so the length field of sspt_pad needs to be set to 12 bytes, which gives a total encoding of 16 bytes. The total number of XDR encoded bytes is therefore 8 + 4 + 20 + 16 = 48 🎀.

GSS_Wrap() emits a token that is an XDR encoding of a ssv_seal_cipher_tkn4 value. Note that the token always has confidentiality, no matter whether the caller of GSS_Wrap() asked for it. That's because the SSV mechanism is for RPCSEC_GSS, and RPCSEC_GSS never produces GSS_Wrap() tokens without confidentiality 🔐💅.

There's one SSV per client ID. There's a single GSS context for each client ID / SSV pair, and every SSV mechanism RPCSEC_GSS handle for a client ID / SSV pair shares that same GSS context 🤝. SSV GSS contexts don't expire, except when the SSV is destroyed. Causes would include the client ID being destroyed or a server restart ⚠️. Here's the thing: one purpose of context expiration is to replace keys that have been in use for "too long", which makes them more vulnerable to compromise by brute force or accident. So the client can replace the SSV key by sending periodic SET_SSV operations, cycling through different users' RPCSEC_GSS credentials to do it. That way the SSV gets replaced without destroying its GSS contexts. Clever, right? 🧠✨

SSV RPCSEC_GSS handles can be expired or deleted by the server at any time, and the EXCHANGE_ID operation can be used to create more SSV RPCSEC_GSS handles 🦋. Expiring an SSV RPCSEC_GSS handle doesn't mean the SSV or its GSS context has expired. Different things, bestie.

The client MUST establish an SSV via SET_SSV before the SSV GSS context can be used to emit tokens from GSS_Wrap() and GSS_GetMIC(). If SET_SSV has not been successfully called, attempts to emit tokens MUST fail. No SET_SSV, no tokens. Period 💀.

The SSV mechanism doesn't support replay detection and sequencing in its tokens, because RPCSEC_GSS doesn't use those features (see "Context Creation Requests", Section 5.2.2 of [4]) 🙈. However, Section 2.10.10 covers the special considerations for the SSV mechanism when it's used with RPCSEC_GSS, so go read that next 🫶.

2.10.10. Security Considerations for RPCSEC_GSS When Using the SSV Mechanism 🔐

okay bestie, this one is SO important, so buckle up 💅 when a client ID gets created with SP4_SSV state protection (see Section 18.35), the client is allowed to hang multiple RPCSEC_GSS handles off of that one SSV GSS context (see Section 2.10.9). that's giving main character energy, but it also opens a door 🚪 because of how RPCSEC_GSS (both version 1 and version 2, see [4] and [9]) works out the verifier of the reply. so the NFSv4.1 client has to be extra careful to stop a man-in-the-middle 😈 from pulling stuff off.

here's the tea ☕ the verifier of an RPCSEC_GSS reply is the output of GSS_GetMIC() run on the seq_num field of the RPCSEC_GSS credential (data type rpc_gss_cred_ver_1_t) (see Section 5.3.3.2 of [4]). so if a bunch of RPCSEC_GSS handles share the same GSS context, and one handle sends a request with the same seq_num as another handle, an attacker could block the reply and swap in the verifier from the other handle. yikes 💀

there are a few ways to block this attack on the SSV RPCSEC_GSS verifier in the reply. the simplest one is believed to be this:

🌸 every time one or more new SSV RPCSEC_GSS handles get created via EXCHANGE_ID, the client SHOULD send a SET_SSV operation to change the SSV. once the SSV changes, the new handles can't end up reusing an SSV RPCSEC_GSS verifier in a reply. iconic 🦋
🧚 when a requester decides to use N SSV RPCSEC_GSS handles, it SHOULD give each handle its own unique, non-overlapping range of seq_nums. each range SHOULD be equal to MAXSEQ / N (see Section 5 of [4] for what MAXSEQ means). when an SSV RPCSEC_GSS handle hits its max, it SHOULD force the replier to destroy the handle by sending a NULL RPC request with seq_num set to MAXSEQ + 1 (see Section 5.3.3.3 of [4]). 🫶
🎀 when the requester wants to bump N up or down, it SHOULD force the replier to destroy all N handles by sending a NULL RPC request on each handle with seq_num set to MAXSEQ + 1. if the requester is the client, it SHOULD send a SET_SSV operation before using the new handles. if the requester is the server, then the client SHOULD send a SET_SSV operation when it notices the server has forced it to destroy a backchannel's SSV RPCSEC_GSS handle. by sending SET_SSV, the SSV changes, so the attacker can't successfully replay an old verifier in a reply to the requester. no cap, that's the whole trick 🔥

one more thing ✨ if the replier carefully creates the SSV RPCSEC_GSS handles, the splicing risk (where a man-in-the-middle swaps a forged SSV RPCSEC_GSS credential with a verifier meant for another handle) just doesn't exist. why? the verifier in an RPCSEC_GSS request is computed from input that includes both the RPCSEC_GSS handle and seq_num (see Section 5.3.1 of [4]). as long as the replier avoids reusing the value of an RPCSEC_GSS handle it creates, such as by putting a generation number in the handle, the man-in-the-middle can't successfully replay an old verifier in the request to a replier. periodt 💖

2.10.11. Session Mechanics - Steady State 🌈

2.10.11.1. Obligations of the Server 🖥️

the server has the main job of keeping an eye on the backchannel resources the client set up for it (RPCSEC_GSS contexts and backchannel connections). if those resources vanish, the server takes action as specified in Section 2.10.13.2. 👀

2.10.11.2. Obligations of the Client 🧁

the client SHOULD keep these obligations in order to use the session, bestie:

🌷 keep a session it needs from going idle on the server. a client that needs a session but isn't sending any operations risks the server destroying it. sessions eat resources, so the server may need to cull an inactive one. a server MAY consider a session inactive if the client hasn't used it before the session inactivity timer (Section 2.10.12) has expired. 😭
🗑️ destroy the session when it's not needed. if a client has several sessions and one has no requests waiting for replies and has been idle for a while, it SHOULD destroy that session. no clutter, no drama 💅
🔑 maintain GSS contexts and RPCSEC_GSS handles for the backchannel. if the client needs the server to use the RPCSEC_GSS security flavor for callbacks, it needs to make sure the RPCSEC_GSS handles and/or their GSS contexts handed to the server via BACKCHANNEL_CTL or CREATE_SESSION are unexpired. ⏰
🔌 preserve a connection for a backchannel. the server needs a backchannel to gracefully recall recallable state or tell the client about certain events. if the connection isn't being used for the fore channel, the client has no way to tell whether it's still alive (for example, the server restarted without sending a disconnect). the onus is on the server, not the client, to figure out if the backchannel's connection is alive, and to say in the response to a SEQUENCE operation when the last connection tied to a session's backchannel has disconnected. that's the server's job, not yours 🙈

2.10.11.3. Steps the Client Takes to Establish a Session 🚀

if the client doesn't have a client ID yet, it sends EXCHANGE_ID to get one. if it opts for SP4_MACH_CRED or SP4_SSV protection, it SHOULD at minimum list CREATE_SESSION, DESTROY_SESSION, BIND_CONN_TO_SESSION, BACKCHANNEL_CTL, and DESTROY_CLIENTID in the spo_must_enforce list of operations. if it opts for SP4_SSV protection, the client needs to ask for SSV-based RPCSEC_GSS handles. 🧚

the client uses the client ID to send a CREATE_SESSION on a connection to the server. the results of CREATE_SESSION show whether or not the server will persist the session reply cache through a server restart, and the client takes note of that for later 📝

if the client specified SP4_SSV state protection when the client ID was created, then it SHOULD send SET_SSV in the first COMPOUND after the session is created. every time a new principal goes to use the client ID, it SHOULD send SET_SSV again. 🎀

if the client wants delegations, layouts, directory notifications, or any other state that needs a backchannel, it needs to add a connection to the backchannel if CREATE_SESSION didn't already do that. the client creates a connection and calls BIND_CONN_TO_SESSION to link the connection to the session and the session's backchannel. if CREATE_SESSION didn't already do it, the client MUST tell the server what security is required for the client to accept callbacks. it does this with BACKCHANNEL_CTL. if the client picked SP4_MACH_CRED or SP4_SSV protection when it called EXCHANGE_ID, then the client SHOULD specify that the backchannel uses RPCSEC_GSS contexts for security. 🔐

if the client wants extra connections for the backchannel, it needs to call BIND_CONN_TO_SESSION on each connection it wants to use with the session. if it wants extra connections for the fore channel, it needs to call BIND_CONN_TO_SESSION if it specified SP4_SSV or SP4_MACH_CRED state protection when the client ID was created. 🦋

at this point, the session has reached steady state. ✨👑

2.10.12. Session Inactivity Timer ⏳

the server MAY keep a session inactivity timer for each session. if that timer expires, the server MAY destroy the session. to avoid losing a session to inactivity, the client MUST renew the session inactivity timer. the length of the session inactivity timer MUST NOT be less than the lease_time attribute (Section 5.8.1.11). just like lease renewal (Section 8.3), when the server gets a SEQUENCE operation, it resets the session inactivity timer, and MUST NOT let the timer expire while the rest of the operations in the COMPOUND procedure's request are still running. once the last operation finishes, the server MUST set the session inactivity timer to expire no sooner than the sum of the current time and the value of the lease_time attribute. ⏰💖

2.10.13. Session Mechanics - Recovery 🛠️

2.10.13.1. Events Requiring Client Action 🚨

the following events require client action to recover. 💅

2.10.13.1.1. RPCSEC_GSS Context Loss by Callback Path 🥀

if all RPCSEC_GSS handles the client granted to the server for callback use have expired, the client MUST establish a new handle via BACKCHANNEL_CTL. the sr_status_flags field of the SEQUENCE results indicates when callback handles are nearly expired, or fully expired (see Section 18.46.3). 🌸

2.10.13.1.2. Connection Loss 🔌

if the client loses the last connection of the session and wants to keep the session, it needs to create a new connection. and if BIND_CONN_TO_SESSION was listed in the spo_must_enforce list when the client ID was created, the client MUST use BIND_CONN_TO_SESSION to link the connection to the session. 🫶

if there was a request outstanding when the connection was lost, and the client wants to keep using the session, it MUST retry the request, as described in Section 2.10.6.2. it's not necessary to retry over a connection with the same source network address or the same destination network address as the lost one. as long as the session ID, slot ID, and sequence ID in the retry match the original request, the server will recognize the request as a retry if it ran the request before the disconnect. 🧠

if the connection that was lost was the last one tied to the backchannel, and the client wants to keep the backchannel and/or stop recallable state from being revoked, the client needs to reconnect. if it does, it MUST link the connection to the session and backchannel via BIND_CONN_TO_SESSION. the server SHOULD say when it has no callback connection through the sr_status_flags result from SEQUENCE. 📡

2.10.13.1.3. Backchannel GSS Context Loss 🧊

through the sr_status_flags result of the SEQUENCE operation or other means, the client will find out if some or all of the RPCSEC_GSS contexts it assigned to the backchannel have been lost. if the client wants to keep the backchannel and/or avoid putting recallable state at risk of revocation, it needs to use BACKCHANNEL_CTL to assign new contexts. 🦄

2.10.13.1.4. Loss of Session ✨

okay so the replier can lose its record of the session 😭 here's how that happens:

replier failure and restart 💀
a catastrophe corrupts or loses the reply cache on the media where it was stored. this still counts even if the replier said in the CREATE_SESSION results that it would persist the cache 🙈
the server purges the session of a client that's been inactive for a super long time 🕰️
configuration changes among a set of clustered servers move a network address that was connected to one server over to a different server that has zero clue about the session. this usually only happens when the original server stops working for a while 🌧️

so loss of the reply cache is basically the same thing as loss of session, no cap 💅 the replier tells the requester about it by returning NFS4ERR_BADSESSION on the next operation that uses the session ID for that lost session.

after something like a server restart, the client might have lost its connections 🔌 but it assumes for now that the session is still alive ✨ so it reconnects. if it asked for connection association enforcement when it created the session, it calls BIND_CONN_TO_SESSION using the session ID. otherwise it calls SEQUENCE. if BIND_CONN_TO_SESSION or SEQUENCE comes back with NFS4ERR_BADSESSION, the client knows the session isn't available to it through that network address. and if the connection survives the session loss, the next SEQUENCE the client sends over that connection will get NFS4ERR_BADSESSION too. so the client knows, again, that the session was lost 🥺

here's one suggested algorithm for the client when it gets NFS4ERR_BADSESSION 🧚 it's not obligatory, though. if a client doesn't want to use stuff like trunking, it can skip parts of it. it's still a useful example, though, because it shows off a bunch of possible recovery issues:

if the client has other connections to other server network addresses tied to the same session, try a COMPOUND with one operation, SEQUENCE, on each of those other connections.
if those attempts work, the session is still alive, which is a strong sign that the server's network address moved 👀 the client might send an EXCHANGE_ID on the connection that returned NFS4ERR_BADSESSION to check for client ID trunking opportunities (aka the same client ID and so_major_id value come back). the client might also use DNS to see if the moved network address got replaced with another one, so the performance and availability perks of session trunking can keep going 🦋
if the SEQUENCE requests fail with NFS4ERR_BADSESSION, the session doesn't exist on any of the server network addresses the client has connections for with that session ID. but it's possible the session is still alive on other network addresses 🤔 so the client sends an EXCHANGE_ID on all the connections to see if the server owner is still listening on those addresses. if the same server owner comes back with a new client ID, that's a strong sign of a server restart 🔄 if both the same server owner and the same client ID come back, that's a strong indication the server did delete the session. then the client needs to send a CREATE_SESSION if it doesn't have any other sessions for that client ID. if a different server owner comes back, the client can use DNS to find other network addresses. if it can't, or if DNS doesn't find any other addresses for the server, the client won't be able to provide NFSv4.1 service, and fatal errors should be returned to processes that were using the server 😱 if the client is using a "mount" paradigm, unmounting the server is advised.
if the client doesn't know of any other connections tied to the session ID and to server network addresses that are, or have been, associated with the session ID, it can use DNS to find other network addresses. if it can't, or if DNS doesn't find any other addresses for the server, the client won't be able to provide NFSv4.1 service, and fatal errors should be returned to processes that were using the server 💀 if the client is using a "mount" paradigm, unmounting the server is advised.

okay now the tea ☕ if a reconfiguration event assigns the same network address to servers where the eir_server_scope value is different, it can't be guaranteed that a session ID generated by the first server will be recognized as invalid by the first server. 😵‍💫 so when managing server reconfigurations among servers with different server scope values, it's necessary to make sure all clients have disconnected from the first server before the reconfiguration happens. still, clients shouldn't assume servers will always follow this rule. clients MUST be prepared to deal with unexpected effects of server reconfigurations 🫶 even when a session ID is wrongly recognized as valid, it's likely either that the connection won't be recognized as valid or that a sequence value for a slot won't be correct. so when a client gets results showing these unexpected errors, using EXCHANGE_ID to figure out the current server configuration is RECOMMENDED 🎀

one variation on the above: after a server's network address moves, there might be no NFSv4.1 server listening at all, like no listener on port 2049. in that case, one of these happens: the NFSv4 server returns NFS4ERR_MINOR_VERS_MISMATCH, the NFS server returns a PROG_MISMATCH error, the RPC listener on 2049 returns PROG_UNVAIL, or attempts to reconnect to the network address time out. these SHOULD be treated as equivalent to SEQUENCE returning NFS4ERR_BADSESSION for these purposes 🌈

when the client detects session loss, it needs to call CREATE_SESSION to recover 🔁 any non-idempotent operations that were in progress might have been performed on the server at the time of session loss. the client has no general way to recover from that one 😭

note that loss of session doesn't imply loss of byte-range lock, open, delegation, or layout state, because locks, opens, delegations, and layouts are tied to the client ID and depend on the client ID, not the session. and loss of byte-range lock, open, delegation, or layout state doesn't imply loss of session state either, because the session depends on the client ID. BUT loss of client ID does imply loss of session, byte-range lock, open, delegation, and layout state 💜 see Section 8.4.2. a session can survive a server restart, but lock recovery might still be needed 🔒

it's possible CREATE_SESSION fails with NFS4ERR_STALE_CLIENTID (for example, the server restarts and doesn't preserve client ID state). if that happens, the client needs to call EXCHANGE_ID, followed by CREATE_SESSION 🧸

2.10.13.2. Events Requiring Server Action

the following events require server action to recover 🛠️

2.10.13.2.1. Client Crash and Restart

as described in Section 18.35, a restarted client sends EXCHANGE_ID in a way that causes the server to delete any sessions it had 🧹

2.10.13.2.2. Client Crash with No Restart

if a client crashes and never comes back, it will never send EXCHANGE_ID with its old client owner 👻 so the server has session state that will never be used again. after an extended period of time, and if the server has resource constraints, it MAY destroy the old session as well as locking state 🗑️

2.10.13.2.3. Extended Network Partition

to the server, an extended network partition might look exactly like a client crash with no restart (see Section 2.10.13.2.2). unless the server can tell there's a network partition, it's free to treat the situation as if the client has crashed permanently 🫠

2.10.13.2.4. Backchannel Connection Loss

if there were callback requests outstanding when a connection was lost, then the server MUST retry the requests, as described in Section 2.10.6.2 🔁 it's not necessary to retry over a connection with the same source network address or the same destination network address as the lost connection. as long as the session ID, slot ID, and sequence ID in the retry match the original request, the callback target will recognize the request as a retry, even if it did see the request before the disconnect 🌸

if the connection that was lost is the last one associated with the backchannel, then the server MUST indicate that in the sr_status_flags field of every SEQUENCE reply until the backchannel is re-established 🚨 there are two situations, and each uses different status flags: no connectivity for the session's backchannel, and no connectivity for any session backchannel of the client. see Section 18.46 for a description of the appropriate flags in sr_status_flags.

2.10.13.2.5. GSS Context Loss

the server SHOULD monitor when the number of RPCSEC_GSS handles assigned to the backchannel reaches one, and when that one handle is near expiry (i.e., between one and two periods of lease time), and indicate so in the sr_status_flags field of all SEQUENCE replies 👀 the server MUST indicate when all of the backchannel's assigned RPCSEC_GSS handles have expired via the sr_status_flags field of all SEQUENCE replies 💔

2.10.14. Parallel NFS and Sessions

a client and server can potentially be a non-pNFS implementation, a metadata server implementation, a data server implementation, or two or three types of implementations 🧩 the EXCHGID4_FLAG_USE_NON_PNFS, EXCHGID4_FLAG_USE_PNFS_MDS, and EXCHGID4_FLAG_USE_PNFS_DS flags (not mutually exclusive) are passed in the EXCHANGE_ID arguments and results. they let the client say how it wants to use sessions created under the client ID, and let the server say how it'll allow the sessions to be used. see Section 13.1 for pNFS sessions considerations 💖

3. Protocol Constants and Data Types ✨

the syntax and semantics for describing the data types of the NFSv4.1 protocol are defined in the XDR (RFC 4506 [2]) and RPC (RFC 5531 [3]) documents 📜 the next sections build on the XDR data types to define constants, types, and structures specific to this protocol. the full list of XDR data types is in [10] 🦋

3.1. Basic Constants 🎀

okay bestie, this is the cheat sheet of sizes and limits that the rest of the spec leans on 💅 Here they are, all locked in:

   const NFS4_FHSIZE               = 128;
   const NFS4_VERIFIER_SIZE        = 8;
   const NFS4_OPAQUE_LIMIT         = 1024;
   const NFS4_SESSIONID_SIZE       = 16;

   const NFS4_INT64_MAX            = 0x7fffffffffffffff;
   const NFS4_UINT64_MAX           = 0xffffffffffffffff;
   const NFS4_INT32_MAX            = 0x7fffffff;
   const NFS4_UINT32_MAX           = 0xffffffff;

   const NFS4_MAXFILELEN           = 0xffffffffffffffff;
   const NFS4_MAXFILEOFF           = 0xfffffffffffffffe;

Except where noted, all of these are measured in bytes 📏 Here's what each one means:

NFS4_FHSIZE is the biggest a filehandle can be 🗂️
NFS4_VERIFIER_SIZE is the fixed size of a verifier 🔑
NFS4_OPAQUE_LIMIT is the cap on certain opaque info 🫙
NFS4_SESSIONID_SIZE is the fixed size of a session identifier 🆔
NFS4_INT64_MAX is the top value for a signed 64-bit integer 🔝
NFS4_UINT64_MAX is the top value for an unsigned 64-bit integer 🚀
NFS4_INT32_MAX is the top value for a signed 32-bit integer ✨
NFS4_UINT32_MAX is the top value for an unsigned 32-bit integer 💖
NFS4_MAXFILELEN is the longest a regular file can be 📁
NFS4_MAXFILEOFF is the furthest offset you can reach into a regular file 🦋

3.2. Basic Data Types 🧚

these are the base NFSv4.1 data types, the building blocks for everything else 🧱

     +===============+==============================================+
     | Data Type     | Definition                                   |
     +===============+==============================================+
     | int32_t       | typedef int int32_t;                         |
     +---------------+----------------------------------------------+
     | uint32_t      | typedef unsigned int uint32_t;               |
     +---------------+----------------------------------------------+
     | int64_t       | typedef hyper int64_t;                       |
     +---------------+----------------------------------------------+
     | uint64_t      | typedef unsigned hyper uint64_t;             |
     +---------------+----------------------------------------------+
     | attrlist4     | typedef opaque attrlist4<>;                  |
     |               |                                              |
     |               | Used for file/directory attributes.          |
     +---------------+----------------------------------------------+
     | bitmap4       | typedef uint32_t bitmap4<>;                  |
     |               |                                              |
     |               | Used in attribute array encoding.            |
     +---------------+----------------------------------------------+
     | changeid4     | typedef uint64_t changeid4;                  |
     |               |                                              |
     |               | Used in the definition of change_info4.      |
     +---------------+----------------------------------------------+
     | clientid4     | typedef uint64_t clientid4;                  |
     |               |                                              |
     |               | Shorthand reference to client                |
     |               | identification.                              |
     +---------------+----------------------------------------------+
     | count4        | typedef uint32_t count4;                     |
     |               |                                              |
     |               | Various count parameters (READ, WRITE,       |
     |               | COMMIT).                                     |
     +---------------+----------------------------------------------+
     | length4       | typedef uint64_t length4;                    |
     |               |                                              |
     |               | The length of a byte-range within a file.    |
     +---------------+----------------------------------------------+
     | mode4         | typedef uint32_t mode4;                      |
     |               |                                              |
     |               | Mode attribute data type.                    |
     +---------------+----------------------------------------------+
     | nfs_cookie4   | typedef uint64_t nfs_cookie4;                |
     |               |                                              |
     |               | Opaque cookie value for READDIR.             |
     +---------------+----------------------------------------------+
     | nfs_fh4       | typedef opaque nfs_fh4<NFS4_FHSIZE>;         |
     |               |                                              |
     |               | Filehandle definition.                       |
     +---------------+----------------------------------------------+
     | nfs_ftype4    | enum nfs_ftype4;                             |
     |               |                                              |
     |               | Various defined file types.                  |
     +---------------+----------------------------------------------+
     | nfsstat4      | enum nfsstat4;                               |
     |               |                                              |
     |               | Return value for operations.                 |
     +---------------+----------------------------------------------+
     | offset4       | typedef uint64_t offset4;                    |
     |               |                                              |
     |               | Various offset designations (READ, WRITE,    |
     |               | LOCK, COMMIT).                               |
     +---------------+----------------------------------------------+
     | qop4          | typedef uint32_t qop4;                       |
     |               |                                              |
     |               | Quality of protection designation in         |
     |               | SECINFO.                                     |
     +---------------+----------------------------------------------+
     | sec_oid4      | typedef opaque sec_oid4<>;                   |
     |               |                                              |
     |               | Security Object Identifier.  The sec_oid4    |
     |               | data type is not really opaque.  Instead, it |
     |               | contains an ASN.1 OBJECT IDENTIFIER as used  |
     |               | by GSS-API in the mech_type argument to      |
     |               | GSS_Init_sec_context.  See [7] for details.  |
     +---------------+----------------------------------------------+
     | sequenceid4   | typedef uint32_t sequenceid4;                |
     |               |                                              |
     |               | Sequence number used for various session     |
     |               | operations (EXCHANGE_ID, CREATE_SESSION,     |
     |               | SEQUENCE, CB_SEQUENCE).                      |
     +---------------+----------------------------------------------+
     | seqid4        | typedef uint32_t seqid4;                     |
     |               |                                              |
     |               | Sequence identifier used for locking.        |
     +---------------+----------------------------------------------+
     | sessionid4    | typedef opaque                               |
     |               | sessionid4[NFS4_SESSIONID_SIZE];             |
     |               |                                              |
     |               | Session identifier.                          |
     +---------------+----------------------------------------------+
     | slotid4       | typedef uint32_t slotid4;                    |
     |               |                                              |
     |               | Sequencing artifact for various session      |
     |               | operations (SEQUENCE, CB_SEQUENCE).          |
     +---------------+----------------------------------------------+
     | utf8string    | typedef opaque utf8string<>;                 |
     |               |                                              |
     |               | UTF-8 encoding for strings.                  |
     +---------------+----------------------------------------------+
     | utf8str_cis   | typedef utf8string utf8str_cis;              |
     |               |                                              |
     |               | Case-insensitive UTF-8 string.               |
     +---------------+----------------------------------------------+
     | utf8str_cs    | typedef utf8string utf8str_cs;               |
     |               |                                              |
     |               | Case-sensitive UTF-8 string.                 |
     +---------------+----------------------------------------------+
     | utf8str_mixed | typedef utf8string utf8str_mixed;            |
     |               |                                              |
     |               | UTF-8 strings with a case-sensitive prefix   |
     |               | and a case-insensitive suffix.               |
     +---------------+----------------------------------------------+
     | component4    | typedef utf8str_cs component4;               |
     |               |                                              |
     |               | Represents pathname components.              |
     +---------------+----------------------------------------------+
     | linktext4     | typedef utf8str_cs linktext4;                |
     |               |                                              |
     |               | Symbolic link contents ("symbolic link" is   |
     |               | defined in an Open Group [11] standard).     |
     +---------------+----------------------------------------------+
     | pathname4     | typedef component4 pathname4<>;              |
     |               |                                              |
     |               | Represents pathname for fs_locations.        |
     +---------------+----------------------------------------------+
     | verifier4     | typedef opaque                               |
     |               | verifier4[NFS4_VERIFIER_SIZE];               |
     |               |                                              |
     |               | Verifier used for various operations         |
     |               | (COMMIT, CREATE, EXCHANGE_ID, OPEN, READDIR, |
     |               | WRITE) NFS4_VERIFIER_SIZE is defined as 8.   |
     +---------------+----------------------------------------------+

Table 1 📋

End of Base Data Types 🎬

3.3. Structured Data Types 🍓

3.3.1. nfstime4 ⏰

this is how time gets packed up and passed around, so here's the struct:

   struct nfstime4 {
           int64_t         seconds;
           uint32_t        nseconds;
   };

okay so nfstime4 counts seconds and nanoseconds from midnight (zero hour) on January 1, 1970, Coordinated Universal Time (UTC) 🌍 A positive seconds value means a date after that moment, and a negative one means a date before it. Either way, nseconds gets added onto seconds to get the final time. For example, if you want to represent half a second before zero hour January 1, 1970, then seconds is negative one (-1) and nseconds is one-half second (500000000) 🧮 And nseconds above 999,999,999 is invalid, so don't even try 🙅

this type is used to pass time and date info around 📨 A server converts to and from its own local time format when it processes these values, and it keeps as much accuracy as it can 🎯 If the timestamps stored for a file system object are less precise than what's defined here, you can lose some precision, and that's just how it goes 🫠 An adjunct time maintenance protocol is RECOMMENDED to reduce client and server time skew ⏳ basically, keep your clocks friends 🫶

3.3.2. time_how4 ✨

okay so time_how4 is basically a two-option menu for timestamps 💅 💖

enum time_how4 {
        SET_TO_SERVER_TIME4 = 0,
        SET_TO_CLIENT_TIME4 = 1
};

3.3.3. settime4 🎀

settime4 is a union that switches on a time_how4. If you pick SET_TO_CLIENT_TIME4, you get to bring your own nfstime4 value. For anything else, the arm is just void, so there's nothing extra to send 🫶

union settime4 switch (time_how4 set_it) {
    case SET_TO_CLIENT_TIME4:
            nfstime4       time;
    default:
            void;
};

These two types are used to set timestamps in file object attributes. If set_it is SET_TO_SERVER_TIME4, the server grabs its own local idea of the time and uses that as the value 🕰️✨

3.3.4. specdata4 🧚

specdata4 is the device-number pair for the device file types NF4CHR and NF4BLK. Major and minor, bestie 🍓

struct specdata4 {
    uint32_t specdata1; /* major device number */
    uint32_t specdata2; /* minor device number */
};

3.3.5. fsid4 🌸

Here's the file system ID. It's two 64-bit numbers, major and minor, and that's it. Simple, iconic 👑

struct fsid4 {
        uint64_t        major;
        uint64_t        minor;
};

3.3.6. change_policy4 🦋

change_policy4 is the type behind the change_policy attribute, which is RECOMMENDED. It's kind of like the change attribute, but for sequencing changes. The clever part: it uses two 64-bit numbers so the server can keep a value that stays valid even after a restart, without needing any persistent storage. One number is a server instance ID, and the other gets bumped non-persistently within that same instance. Galaxy brain stuff 🌈🧠

struct change_policy4 {
        uint64_t        cp_major;
        uint64_t        cp_minor;
};

3.3.7. fattr4 💜

fattr4 is how we represent file and directory attributes. It has a bitmap saying which attributes are present, plus the values themselves 📦

struct fattr4 {
        bitmap4         attrmask;
        attrlist4       attr_vals;
};

The bitmap is a counted array of 32-bit integers used to hold bit values. To find the bit for bit n, you use the integer at position (n / 32), and then the bit inside that integer is (n mod 32). Picture it like this:

                 0            1
+-----------+-----------+-----------+--
|  count    | 31  ..  0 | 63  .. 32 |
+-----------+-----------+-----------+--

3.3.8. change_info4 🔥

change_info4 lets the client know the change attribute value for the directory where the target file system object lives. It shows up in the CREATE, LINK, OPEN, REMOVE, and RENAME operations, so you always know what changed 👀

struct change_info4 {
        bool            atomic;
        changeid4       before;
        changeid4       after;
};

3.3.9. netaddr4 🍓

netaddr4 is how we identify network transport endpoints. Its two fields hold a netid and a uaddr. Those concepts come from [12], and the formats for TCP over IPv4 and TCP over IPv6 are also in [12], specifically Tables 2 and 3 and Sections 5.2.3.3 and 5.2.3.4. Go read those if you want the full tea ☕

struct netaddr4 {
        /* see struct rpcb in RFC 1833 */
        string na_r_netid<>; /* network id */
        string na_r_addr<>;  /* universal address */
};

3.3.10. state_owner4 🫶

state_owner4 is the base type for both open_owner4 and lock_owner4. It's a clientid4 plus an opaque owner field with a size limit of NFS4_OPAQUE_LIMIT. The two owner types are just typedefs of it, so they're basically twins 👯✨

struct state_owner4 {
        clientid4       clientid;
        opaque          owner<NFS4_OPAQUE_LIMIT>;
};

typedef state_owner4 open_owner4;
typedef state_owner4 lock_owner4;

3.3.10.1. open_owner4 💖

This is the owner of OPEN state. Easy peasy 💅

3.3.10.2. lock_owner4 🔒

This one identifies the owner of byte-range locking state. Lock it in 🔐

3.3.11. open_to_lock_owner4 ✨

This is for the very first LOCK operation done for an open_owner4. It bundles up the open_stateid and the lock_owner, which makes the jump from a valid open stateid sequence over to a new lock stateid sequence smooth as butter 🧈. The best part is that it skips confirming the lock_owner/lock_seqid pair, because that pair is already tied to the established state through open_stateid/open_seqid. No extra confirmation step needed, no cap 🙌

struct open_to_lock_owner4 {
        seqid4          open_seqid;
        stateid4        open_stateid;
        seqid4          lock_seqid;
        lock_owner4     lock_owner;
};

3.3.12. stateid4 🦋

stateid4 is used for the different state sharing mechanisms between client and server. The client never modifies a stateid value, ever 🙅‍♀️. The starting value of seqid is undefined. The server is required to increment seqid by one at each transition of the stateid. This matters a LOT, because the client looks at the seqid in OPEN stateids to figure out what order the server processed OPENs in. Order is everything 👑

struct stateid4 {
        uint32_t        seqid;
        opaque          other[12];
};

3.3.13. layouttype4 🌈

layouttype4 tells you what kind of layout is in use. The file server advertises the layout types it supports through the fs_layout_type file system attribute (Section 5.12.1). Then the client asks for layouts of a particular type in LAYOUTGET, and handles them with its layout-type-specific logic 🧩

enum layouttype4 {
        LAYOUT4_NFSV4_1_FILES   = 0x1,
        LAYOUT4_OSD2_OBJECTS    = 0x2,
        LAYOUT4_BLOCK_VOLUME    = 0x3
};

The layouttype4 data type is 32 bits long, and its range gets split into three parts. Type 0x0 is reserved. Types in the range 0x00000001-0x7FFFFFFF are globally unique and get assigned according to Section 22.5, and they're maintained by IANA. Types in the range 0x80000000-0xFFFFFFFF are site specific and for private use only, so hands off, everyone else 🔒💅

The LAYOUT4_NFSV4_1_FILES enumeration says the NFSv4.1 file layout type from Section 13 is the one to use. The LAYOUT4_OSD2_OBJECTS enumeration says to use the object layout, as defined in [47]. And LAYOUT4_BLOCK_VOLUME says to use the block/volume layout, as defined in [48] ✨

3.3.14. deviceid4 🧚

Layout info includes device IDs, which point at a storage device through a compact handle. You get the addressing and type info with the GETDEVICEINFO operation. Device IDs aren't guaranteed to stay valid across metadata server restarts, so don't get too attached 💀. A device ID is unique per client ID and layout type. For more details, see Section 12.2.10 💕

const NFS4_DEVICEID4_SIZE = 16;

typedef opaque  deviceid4[NFS4_DEVICEID4_SIZE];

3.3.15. device_addr4 🌸

The device address is how you set up a communication channel with a storage device. Different layout types need different data to talk to storage devices, so the opaque da_addr_body field gets interpreted based on da_layout_type 🎀

struct device_addr4 {
        layouttype4             da_layout_type;
        opaque                  da_addr_body<>;
};

This document defines the device address for the NFSv4.1 file layout (see Section 13.3). That one identifies a storage device by network IP address and port number. It's enough for clients to talk to NFSv4.1 storage devices, and it might be enough for other layout types too. Device types for object-based storage devices and block storage devices (for example, Small Computer System Interface (SCSI) volume labels) are defined by their own layout specifications 🫧

3.3.16. layout_content4 🍑

layout_content4 is a layout type plus a body. The loc_body field gets interpreted based on the layout type (loc_type). This document defines loc_body for the NFSv4.1 file layout type, and you'll find its definition in Section 13.3 🧃

struct layout_content4 {
        layouttype4 loc_type;
        opaque      loc_body<>;
};

3.3.17. layout4 🌟

layout4 is the data type that defines a layout for a file. The layout type specific data stays opaque inside lo_content. Layouts can be subdivided, so the offset and length, together with the file's filehandle, the client ID, the iomode, and the layout type, are what identify a given layout 🔍

struct layout4 {
        offset4                 lo_offset;
        length4                 lo_length;
        layoutiomode4           lo_iomode;
        layout_content4         lo_content;
};

3.3.18. layoutupdate4 📮

The client uses layoutupdate4 to send updated layout info back to the metadata server through the LAYOUTCOMMIT (Section 18.42) operation. It's a channel for passing layout type specific info back in lou_body. For example, for the block/volume layout type, that could be the list of reserved blocks that got written. The contents of the opaque lou_body argument are decided by the layout type. The NFSv4.1 file-based layout doesn't use this type at all. If lou_type is LAYOUT4_NFSV4_1_FILES, then the lou_body field MUST have a zero length 🙅‍♀️📏

struct layoutupdate4 {
        layouttype4             lou_type;
        opaque                  lou_body<>;
};

3.3.19. layouthint4 💌

The client uses layouthint4 to pass in a hint about what type of layout it'd like created for a particular file. This is the data type from the layout_hint attribute in Section 5.12.4. The metadata server can ignore the hint, or it can selectively ignore fields inside it. This hint should be provided at create time, as part of the initial attributes within OPEN. The loh_body field depends on the layout type (loh_type). The NFSv4.1 file-based layout uses the nfsv4_1_file_layouthint4 data type from Section 13.3 🎯

struct layouthint4 {
        layouttype4             loh_type;
        opaque                  loh_body<>;
};

3.3.20. layoutiomode4 🔥

The iomode says whether the client just wants to read the data represented by the layout, or read and write it. LAYOUTIOMODE4_ANY MUST NOT be used in the arguments to the LAYOUTGET operation, but it MAY be used in the arguments to the LAYOUTRETURN and CB_LAYOUTRECALL operations. When LAYOUTIOMODE4_ANY shows up there, it means layouts for both LAYOUTIOMODE4_READ and LAYOUTIOMODE4_RW are being returned or recalled. How the metadata server uses the iomode can depend on the layout type. The storage devices MAY validate I/O accesses against the iomode and reject the invalid ones 🚫✨

enum layoutiomode4 {
        LAYOUTIOMODE4_READ      = 1,
        LAYOUTIOMODE4_RW        = 2,
        LAYOUTIOMODE4_ANY       = 3
};

3.3.21. nfs_impl_id4 🎀

This data type identifies the implementation details of the client and server 🤓. The nii_domain field is the DNS domain name the implementor is associated with. The nii_name field is the product name of the implementation, and it's totally free form. It is RECOMMENDED that nii_name be used to tell apart machine architecture, machine platforms, revisions, versions, and patch levels, so do that, babe 💅. The nii_date field is the timestamp for when the software instance was published or built ⏰

struct nfs_impl_id4 {
        utf8str_cis   nii_domain;
        utf8str_cs    nii_name;
        nfstime4      nii_date;
};

3.3.22. threshold_item4 🎀

Okay bestie, threshold_item4 is giving hint-list energy 💅 It holds a list of hints for one layout type, so the client can tell when it should send I/O straight through the metadata server (MDS) and when it should go to the storage devices 🧚 It has three parts: the layout type (thi_layout_type), a bitmap (thi_hintset) showing which hints the server supports (these can differ by layout type, fr), and a hint list (thi_hintlist) whose contents are decided by the hintset bitmap. Check the mdsthreshold attribute for the deets ✨

struct threshold_item4 {
        layouttype4     thi_layout_type;
        bitmap4         thi_hintset;
        opaque          thi_hintlist<>;
};

The thi_hintset field is a bitmap of the following values 🦋

+=========================+===+=========+===========================+
| name                    | # | Data    | Description               |
|                         |   | Type    |                           |
+=========================+===+=========+===========================+
| threshold4_read_size    | 0 | length4 | If a file's length is     |
|                         |   |         | less than the value of    |
|                         |   |         | threshold4_read_size,     |
|                         |   |         | then it is RECOMMENDED    |
|                         |   |         | that the client read      |
|                         |   |         | from the file via the     |
|                         |   |         | MDS and not a storage     |
|                         |   |         | device.                   |
+-------------------------+---+---------+---------------------------+
| threshold4_write_size   | 1 | length4 | If a file's length is     |
|                         |   |         | less than the value of    |
|                         |   |         | threshold4_write_size,    |
|                         |   |         | then it is RECOMMENDED    |
|                         |   |         | that the client write     |
|                         |   |         | to the file via the       |
|                         |   |         | MDS and not a storage     |
|                         |   |         | device.                   |
+-------------------------+---+---------+---------------------------+
| threshold4_read_iosize  | 2 | length4 | For read I/O sizes        |
|                         |   |         | below this threshold,     |
|                         |   |         | it is RECOMMENDED to      |
|                         |   |         | read data through the     |
|                         |   |         | MDS.                      |
+-------------------------+---+---------+---------------------------+
| threshold4_write_iosize | 3 | length4 | For write I/O sizes       |
|                         |   |         | below this threshold,     |
|                         |   |         | it is RECOMMENDED to      |
|                         |   |         | write data through the    |
|                         |   |         | MDS.                      |
+-------------------------+---+---------+---------------------------+

Table 2

3.3.23. mdsthreshold4 🌈

struct mdsthreshold4 {
        threshold_item4 mth_hints<>;
};

This one is the array holder 💜 mdsthreshold4 carries an array of threshold_item4 elements, and each element is valid for one particular layout type. The array exists because a server can support multiple layout types for a single file, so it needs more than one of these to cover them all 🫶

4. Filehandles ✨

The filehandle in the NFS protocol is a per-server unique identifier for a file system object 🌸 Its contents are opaque to the client, which means the server is the one responsible for translating the filehandle back into an internal representation of the file system object. The server handles the decoding, and the client just holds onto the bag 💅

4.1. Obtaining the First Filehandle

The operations of the NFS protocol are defined in terms of one or more filehandles, so the client needs a filehandle before it can even start talking to the server 🥺 With the NFSv3 protocol (RFC 1813 [38]), there was an ancillary protocol for getting that first filehandle. The MOUNT protocol, RPC program number 100005, handles translating a string-based file system pathname into a filehandle that the NFS protocols can then use.

The MOUNT protocol has deficiencies in the area of security and use via firewalls, which is one reason the public filehandle was introduced in RFC 2054 [49] and RFC 2055 [50]. With the public filehandle combined with the LOOKUP operation in the NFSv3 protocol, it was demonstrated that the MOUNT protocol is unnecessary for viable interaction between an NFS client and server. Iconic 🙌

So the NFSv4.1 protocol will not use an ancillary protocol to translate string-based pathnames into a filehandle. Instead, two special filehandles act as starting points for the NFS client 🦋

4.1.1. Root Filehandle

The first special filehandle is the ROOT filehandle, which is the "conceptual" root of the file system namespace at the NFS server 👑 The client uses or starts with the ROOT filehandle by employing the PUTROOTFH operation. That operation tells the server to set the "current" filehandle to the ROOT of the server's file tree. Once PUTROOTFH has been used, the client can traverse the entire server file tree with the LOOKUP operation. The full discussion of the server namespace is in Section 7 📚

4.1.2. Public Filehandle

The second special filehandle is the PUBLIC filehandle. Unlike the ROOT filehandle, the PUBLIC filehandle may be bound to or represent an arbitrary file system object at the server, and the server is the one responsible for that binding 🎀 The PUBLIC and ROOT filehandles may refer to the same file system object, but that's up to the administrative software at the server and the policies of the server administrator, who decide how PUBLIC gets bound. The client may not make any assumptions about this binding, so don't you dare guess 🙈 The client uses the PUBLIC filehandle via the PUTPUBFH operation.

4.2. Filehandle Types

In the NFSv3 protocol there was one type of filehandle with a single set of semantics. That type is called "persistent" in NFSv4.1, and its semantics stay the same as before 🍓 NFSv4.1 adds a new type called the "volatile" filehandle, which tries to accommodate certain server environments. Iykyk.

The volatile filehandle type was added to handle server functionality or implementation issues that make a correct persistent filehandle infeasible. Some server environments don't provide a file-system-level invariant that can be used to build a persistent filehandle. The underlying file system may not provide that invariant, or the server's file system programming interfaces may not give access to it. Volatile filehandles can make things like hierarchical storage management or file system reorganization or migration easier to implement, but they make life harder for the client 😭

Since the client has to handle persistent and volatile filehandles differently, a file attribute is defined that the client can use to figure out which filehandle types the server is returning 🧚

4.2.1. General Properties of a Filehandle

The filehandle contains all the information the server needs to tell one file apart from another. To the client it's opaque, so the client stores filehandles for use in later requests and can compare two filehandles from the same server for equality with a byte-by-byte comparison. The client MUST NOT otherwise interpret the contents of filehandles 🙅‍♀️ If two filehandles from the same server are equal, they MUST refer to the same file. Servers SHOULD try to keep a one-to-one correspondence between filehandles and files, but this is not required. Clients MUST use filehandle comparisons only to improve performance, not for correct behavior. Every client needs to be ready for situations where it can't tell whether two filehandles denote the same object, and in those cases it should avoid making invalid assumptions that could lead to incorrect behavior 💀 More on filehandle and attribute comparison in the context of data caching is in Section 10.3.4.

For example, if two different pathnames end up at the same file system object when traversed at the server, the server SHOULD return the same filehandle for each path. This can happen if a hard link (see [6]) is used to create two file names that refer to the same underlying file object and its associated data. So if paths /a/b/c and /a/d/c refer to the same file, the server SHOULD return the same filehandle for both pathnames' traversals. Main character energy for that file 🦋

4.2.2. Persistent Filehandle

A persistent filehandle has a fixed value for the lifetime of the file system object it refers to ✨ Once the server creates the filehandle for a file system object, the server MUST accept that same filehandle for the object for the object's entire lifetime. If the server restarts, the NFS server MUST honor the same filehandle value it used in its previous instantiation. Likewise, if the file system is migrated, the new NFS server MUST honor the same filehandle as the old NFS server. No drama, no switching filehandles mid-stream 💖

A persistent filehandle becomes stale or invalid when the file system object it refers to is removed. When the server gets a persistent filehandle that refers to a deleted object, it MUST return an error of NFS4ERR_STALE 😭 A filehandle can also go stale when the file system containing the object is no longer available. That can happen if the file system lives on removable media that's no longer available at the server, if the file system as a whole has been destroyed, or if it has simply been removed from the server's namespace (in a UNIX environment, that's unmounted) 🌈

4.2.3. Volatile Filehandle 🦋

okay bestie, a volatile filehandle does NOT have the same longevity as a persistent one 😭 The server can decide it's no longer valid at lots of different points in time. If the server can definitively tell that the object it refers to has been removed, it should send back NFS4ERR_STALE to the client (same as for persistent filehandles). In every other case where the server decides a volatile filehandle can't be used anymore, it should return NFS4ERR_FHEXPIRED 💅

The REQUIRED attribute "fh_expire_type" is how the client finds out what kind of filehandle the server is handing out for a particular file system ✨ It's a bitmask, and these are the values:

FH4_PERSISTENT: this one means a persistent filehandle, which stays valid until the object is removed from the file system 💖 The server will not return NFS4ERR_FHEXPIRED for it. FH4_PERSISTENT is defined as the value where none of the bits listed below are set.
FH4_VOLATILE_ANY: the filehandle can expire at any time, unless it's specifically excluded (i.e., FH4_NO_EXPIRE_WITH_OPEN) 🌸
FH4_NOEXPIRE_WITH_OPEN: May only be set when FH4_VOLATILE_ANY is set. When it's on, the FH4_VOLATILE_ANY meaning gets qualified so that the filehandle doesn't expire while it's open 🔥
FH4_VOL_MIGRATION: the filehandle will expire because of a file system transition (migration or replication), in the cases where the continuity of filehandle use isn't specified by handle class information in the fs_locations_info attribute. When this bit is set, clients without access to fs_locations_info information should assume filehandles will expire on file system transitions 🧚
FH4_VOL_RENAME: the filehandle will expire during a rename. That includes a rename by the requesting client or a rename by any other client 👑 If FH4_VOL_ANY is set, FH4_VOL_RENAME is redundant.

Servers that hand out volatile filehandles which can expire while open need extra care with RENAMEs and REMOVEs 🙈 This can happen if FH4_VOL_MIGRATION or FH4_VOL_RENAME is set, if FH4_VOLATILE_ANY is set and FH4_NOEXPIRE_WITH_OPEN is not set, or if a non-read-only file system has a transition target in a different handle class. In these cases, the server should deny a RENAME or REMOVE that would affect an OPEN file of any of the components leading to the OPEN file. The server should also deny all RENAME or REMOVE requests during the grace period, so that reclaims of files whose filehandles may have expired don't reclaim the wrong file 😭

Volatile filehandles are especially good for the pseudo file systems used to bridge exports 🌈 See Section 7.5 for a discussion of this.

4.3. One Method of Constructing a Volatile Filehandle 🍓

A volatile filehandle is opaque to the client, but it could contain:

[volatile bit = 1 | server boot time | slot | generation number]

slot is an index in the server volatile filehandle table
generation number is the generation number for the table entry/slot

When the client presents a volatile filehandle, the server runs these checks, which assume the check for the volatile bit has already passed. If the server boot time is less than the current server boot time, return NFS4ERR_FHEXPIRED. If the slot is out of range, return NFS4ERR_BADHANDLE. If the generation number does not match, return NFS4ERR_FHEXPIRED ✨

When the server restarts, the table is gone (it's volatile, bestie) 💀

If the volatile bit is 0, then it's a persistent filehandle with a different structure following it 💜

4.4. Client Recovery from Filehandle Expiration 🫶

If possible, the client SHOULD recover from receiving an NFS4ERR_FHEXPIRED error. The client must take on extra responsibility so it can get ready to recover from the expiration of a volatile filehandle. If the server returns persistent filehandles, the client doesn't need these extra steps 🎀

For volatile filehandles, the client will most commonly need to store the component names leading up to and including the file system object in question. With those names, the client should be able to recover by finding a filehandle in the namespace that's still available, or by starting over at the root of the server's file system namespace 🦋

If the expired filehandle refers to an object that has been removed from the file system, the client obviously won't be able to recover from it 😭

It's also possible that the expired filehandle refers to a file that has been renamed. If another client renamed the file, the original client may not be able to recover. But if the client itself is renaming the file and the file is open, the client may be able to recover 💅 The client can work out the new pathname from the processing of the rename request, and then regenerate the new filehandle from that pathname. The client could also use the COMPOUND procedure to build a series of operations like this:

RENAME A B
LOOKUP B
GETFH

Note that the COMPOUND procedure does not provide atomicity. This example just cuts down on the overhead of recovering from an expired filehandle ⭐

5. File Attributes 🌸

Attributes need to be handled in a flexible way so the protocol can keep up with new needs and play nicely with non-UNIX platforms 🌈 The NFSv3 fattr3 structure has a fixed list of attributes that not every client and server can support or cares about. It can't be extended as new needs come up, and it has no way to say "not supported." With the NFSv4.1 protocol, the client can ask what attributes the server supports and build requests with only those (or a subset of them) ✨

So attributes are split into three groups: REQUIRED, RECOMMENDED, and named 👑 Both REQUIRED and RECOMMENDED attributes get a specific, well-defined encoding in the NFSv4.1 protocol, and each one is identified by a number. The client asks for them by setting a bit in the bit vector sent in the GETATTR request, and the server's response includes a bit vector listing which attributes came back. New REQUIRED or RECOMMENDED attributes can be added to the NFSv4 protocol as part of a new minor version, by publishing a Standards Track RFC that allocates a new attribute number and defines the encoding for it. See Section 2.7 for more on this 🦋

Named attributes are accessed with the new OPENATTR operation, which reaches a hidden directory of attributes attached to a file system object. OPENATTR takes a filehandle for the object and returns the filehandle for the attribute hierarchy. That filehandle is a directory object you can reach with LOOKUP or READDIR, and it contains files whose names are the named attributes and whose data bytes are the attribute's value. Here's an example:

     +----------+-----------+---------------------------------+
     | LOOKUP   | "foo"     | ; look up file                  |
     +----------+-----------+---------------------------------+
     | GETATTR  | attrbits  |                                 |
     +----------+-----------+---------------------------------+
     | OPENATTR |           | ; access foo's named attributes |
     +----------+-----------+---------------------------------+
     | LOOKUP   | "x11icon" | ; look up specific attribute    |
     +----------+-----------+---------------------------------+
     | READ     | 0,4096    | ; read stream of bytes          |
     +----------+-----------+---------------------------------+

Table 3

Named attributes are meant for data that applications need, rather than data an NFS client implementation needs 🧚 NFS implementors are strongly encouraged to define their new attributes as RECOMMENDED attributes by bringing them through the IETF Standards Track process 💖

The REQUIRED set is deliberately small, because servers need to do whatever it takes to support those attributes 💪 A server should support as many RECOMMENDED attributes as it possibly can, but by definition the server isn't required to support all of them. Attributes count as REQUIRED only if the data is both needed by a large number of clients and not otherwise reasonably computable by the client when the server doesn't provide it 🔥

Note that the hidden directory returned by OPENATTR is a convenience for protocol processing 🙈 The client shouldn't make any assumptions about how the server implements named attributes, or whether the underlying file system on the server has a named attribute directory. So operations like SETATTR and GETATTR on the named attribute directory are undefined 💀

5.1. REQUIRED Attributes 👑

These MUST be supported by every NFSv4.1 client and server, so there's a minimum level of interoperability 💅 The server MUST store and return these attributes, and the client MUST be able to function with an attribute set limited to these attributes. With only the REQUIRED attributes, some client functionality may be impaired or limited in some ways. A client can ask for any of these attributes to be returned by setting a bit in the GETATTR request, and the server MUST return their value ✨

5.2. RECOMMENDED Attributes 🌷

These attributes are understood well enough to deserve support in the NFSv4.1 protocol. However, they may not be supported on all clients and servers 🫧 A client can ask for any of these attributes to be returned by setting a bit in the GETATTR request, but it must handle the case where the server doesn't return them. A client MAY ask for the set of attributes the server supports and SHOULD NOT request attributes the server doesn't support. A server should be tolerant of requests for unsupported attributes and simply not return them, rather than treating the request as an error. It's expected that servers will support every attribute they comfortably can, and only fail to support attributes that are hard to support in their operating environments 🌈 A server should provide attributes whenever it doesn't have to "tell lies" to the client. For example, a file modification time should be either an accurate time or not supported by the server at all. This is hard for clients sometimes, but a client is better placed to decide whether and how to fabricate or construct an attribute, or whether to do without it 🧚

5.3. Named Attributes ✨

okay so these attributes don't get a number like the others 🧚 they're stored as plain bytes and you find them by string name instead 💅 the server just keeps an uninterpreted stream of bytes hanging out with the file system object 🫶

to get at them you use the OPENATTR operation, which hands back a filehandle for a virtual "named attribute directory" 📂 then you can poke around in that namespace with operations you'd normally use on regular directories. READDIR lists the named attributes, LOOKUP and OPEN pick out one in particular, and creating a brand-new one can happen when an OPEN asks for file creation 🌸

once you've OPENed one, you read and write its bytes with normal READ and WRITE ops, using the filehandles and stateids that OPEN gave you 💖

named attributes and the named attribute directory can have their own non-named attributes too 🦋 every one of them MUST have all of the REQUIRED attributes, and they can also have extra RECOMMENDED ones 🎀 BUT the set can be (and usually will be) way smaller than what regular objects in that file system get 🙈

they could also be the target of delegations. for the named attribute directory those are directory delegations 👑 granting delegations is totally up to the server though, so a server doesn't have to support delegations on named attributes or the named attribute directory ngl 💀

it is RECOMMENDED that servers support arbitrary named attributes. clients should not depend on being able to store named attributes on the server's file system 😭

and if a server DOES support them, a client that can handle them should be able to copy a file's data and metadata from one place to another with complete transparency 🌈 which means names that work for regular directory entries should also work as named attribute names 🫶

now here's the lowkey important bit. in NFSv4.1 the named attribute directories are restricted in a few ways 🔥 the goal is to stop servers from ending up with different setups, where some have a full hierarchical directory structure for named attributes and others have a limited one. that's how you get non-interoperable implementations and clients start leaning on non-portable extensions 💅 the restrictions are:

CREATE is not allowed in a named attribute directory. so no symbolic links or special files as named attributes, and no creating directories in there either, which means no hierarchy of named attributes for a single object 🙅‍♀️
if OPENATTR is done on a named attribute directory or on a named attribute, the server MUST return NFS4ERR_WRONG_TYPE 😭
you can't RENAME a named attribute into a different named attribute directory or into an ordinary (i.e., non-named-attribute) directory 🚫
creating hard links between named attribute directories, or between a named attribute directory and an ordinary directory, is not allowed 🔗

names of attributes aren't controlled by this document or by other IETF Standards Track documents 🎀 see Section 22.2 for more on that 🦋

5.4. Classification of Attributes 🍓

every REQUIRED and RECOMMENDED attribute falls into one of three categories ✨ the first is per server, where the value is the same for every file object that shares the same server owner (see Section 2.5 for what a server owner is) 🏠 the second is per file system, where the value is the same for some or all file objects that share the same fsid attribute (Section 5.8.1.9) and server owner 🌈 and the last is per file system object 💜

heads up: some per file system attributes can vary inside the file system, depending on the value of the "homogeneous" (Section 5.8.2.16) attribute 👀

also, time_access_set and time_modify_set aren't listed in this section because they're write-only and just correspond to time_access and time_modify. they only show up in a special instance of SETATTR 🧚

here's the sorted-out list:

The per-server attribute is:

lease_time

The per-file system attributes are:

supported_attrs, suppattr_exclcreat, fh_expire_type,
link_support, symlink_support, unique_handles, aclsupport,
cansettime, case_insensitive, case_preserving,
chown_restricted, files_avail, files_free, files_total,
fs_locations, homogeneous, maxfilesize, maxname, maxread,
maxwrite, no_trunc, space_avail, space_free, space_total,
time_delta, change_policy, fs_status, fs_layout_type,
fs_locations_info, fs_charset_cap

The per-file system object attributes are:

type, change, size, named_attr, fsid, rdattr_error, filehandle,
acl, archive, fileid, hidden, maxlink, mimetype, mode,
numlinks, owner, owner_group, rawdev, space_used, system,
time_access, time_backup, time_create, time_metadata,
time_modify, mounted_on_fileid, dir_notif_delay,
dirent_notif_delay, dacl, sacl, layout_type, layout_hint,
layout_blksize, layout_alignment, mdsthreshold, retention_get,
retention_set, retentevt_get, retentevt_set, retention_hold,
mode_set_masked

for quota_avail_hard, quota_avail_soft, and quota_used, check out their definitions below for the right classification 💖

5.5. Set-Only and Get-Only Attributes 💅

some REQUIRED and RECOMMENDED attributes are set-only 🫶 meaning you can SETATTR them but you can't GETATTR them back. and some are get-only 🔥 you can GETATTR them but you can't SETATTR them. if a client tries to set a get-only attribute, or get a set-only attribute, the server MUST return NFS4ERR_INVAL 🙅‍♀️

5.6. REQUIRED Attributes - List and Definition References 📚

the REQUIRED attributes are listed in Table 4 ✨ here's what the columns mean:

Name: the name of the attribute 🏷️

Id: the number assigned to the attribute. if the assigned number ever clashes with [10], the latter is probably the one to trust, but it should get sorted out with Errata to this document and/or [10] 🛠️ check [51] for how the Errata process works 📝

Data Type: the XDR data type of the attribute 🧩

Acc: access allowed to the attribute. R is read-only (GETATTR can retrieve it, SETATTR can't set it). W is write-only (SETATTR can set it, GETATTR can't retrieve it). R W is read/write (GETATTR can retrieve it and SETATTR can set it) 🦋

Defined in: the section of this specification that describes the attribute 📖

     +====================+====+============+=====+==================+
     | Name               | Id | Data Type  | Acc | Defined in:      |
     +====================+====+============+=====+==================+
     | supported_attrs    | 0  | bitmap4    | R   | Section 5.8.1.1  |
     +--------------------+----+------------+-----+------------------+
     | type               | 1  | nfs_ftype4 | R   | Section 5.8.1.2  |
     +--------------------+----+------------+-----+------------------+
     | fh_expire_type     | 2  | uint32_t   | R   | Section 5.8.1.3  |
     +--------------------+----+------------+-----+------------------+
     | change             | 3  | uint64_t   | R   | Section 5.8.1.4  |
     +--------------------+----+------------+-----+------------------+
     | size               | 4  | uint64_t   | R W | Section 5.8.1.5  |
     +--------------------+----+------------+-----+------------------+
     | link_support       | 5  | bool       | R   | Section 5.8.1.6  |
     +--------------------+----+------------+-----+------------------+
     | symlink_support    | 6  | bool       | R   | Section 5.8.1.7  |
     +--------------------+----+------------+-----+------------------+
     | named_attr         | 7  | bool       | R   | Section 5.8.1.8  |
     +--------------------+----+------------+-----+------------------+
     | fsid               | 8  | fsid4      | R   | Section 5.8.1.9  |
     +--------------------+----+------------+-----+------------------+
     | unique_handles     | 9  | bool       | R   | Section 5.8.1.10 |
     +--------------------+----+------------+-----+------------------+
     | lease_time         | 10 | nfs_lease4 | R   | Section 5.8.1.11 |
     +--------------------+----+------------+-----+------------------+
     | rdattr_error       | 11 | enum       | R   | Section 5.8.1.12 |
     +--------------------+----+------------+-----+------------------+
     | filehandle         | 19 | nfs_fh4    | R   | Section 5.8.1.13 |
     +--------------------+----+------------+-----+------------------+
     | suppattr_exclcreat | 75 | bitmap4    | R   | Section 5.8.1.14 |
     +--------------------+----+------------+-----+------------------+

                                  Table 4

5.7. RECOMMENDED Attributes - List and Definition References 🎀

the RECOMMENDED attributes are defined in Table 5 🌸 the column headers mean the same thing as in Table 4, so check out Section 5.6 for what they all mean 💜

+====================+====+====================+=====+=============+

Name

Data Type

Acc

Defined in:

+====================+====+====================+=====+=============+

acl	12	nfsace4<>	R W	Section
				6.2.1

+--------------------+----+--------------------+-----+-------------+

aclsupport	13	uint32_t	R	Section
				6.2.1.2

+--------------------+----+--------------------+-----+-------------+

archive	14	bool	R W	Section
				5.8.2.1

+--------------------+----+--------------------+-----+-------------+

cansettime	15	bool	R	Section
				5.8.2.2

+--------------------+----+--------------------+-----+-------------+

case_insensitive	16	bool	R	Section
				5.8.2.3

+--------------------+----+--------------------+-----+-------------+

case_preserving	17	bool	R	Section
				5.8.2.4

+--------------------+----+--------------------+-----+-------------+

change_policy	60	chg_policy4	R	Section
				5.8.2.5

+--------------------+----+--------------------+-----+-------------+

chown_restricted	18	bool	R	Section
				5.8.2.6

+--------------------+----+--------------------+-----+-------------+

dacl	58	nfsacl41	R W	Section
				6.2.2

+--------------------+----+--------------------+-----+-------------+

dir_notif_delay	56	nfstime4	R	Section
				5.11.1

+--------------------+----+--------------------+-----+-------------+

dirent_notif_delay	57	nfstime4	R	Section
				5.11.2

+--------------------+----+--------------------+-----+-------------+

fileid	20	uint64_t	R	Section
				5.8.2.7

+--------------------+----+--------------------+-----+-------------+

files_avail	21	uint64_t	R	Section
				5.8.2.8

+--------------------+----+--------------------+-----+-------------+

files_free	22	uint64_t	R	Section
				5.8.2.9

+--------------------+----+--------------------+-----+-------------+

files_total	23	uint64_t	R	Section
				5.8.2.10

+--------------------+----+--------------------+-----+-------------+

fs_charset_cap	76	uint32_t	R	Section
				5.8.2.11

+--------------------+----+--------------------+-----+-------------+

fs_layout_type	62	layouttype4<>	R	Section
				5.12.1

+--------------------+----+--------------------+-----+-------------+

fs_locations	24	fs_locations	R	Section
				5.8.2.12

+--------------------+----+--------------------+-----+-------------+

fs_locations_info	67	fs_locations_info4	R	Section
				5.8.2.13

+--------------------+----+--------------------+-----+-------------+

fs_status	61	fs4_status	R	Section
				5.8.2.14

+--------------------+----+--------------------+-----+-------------+

hidden	25	bool	R W	Section
				5.8.2.15

+--------------------+----+--------------------+-----+-------------+

homogeneous	26	bool	R	Section
				5.8.2.16

+--------------------+----+--------------------+-----+-------------+

layout_alignment	66	uint32_t	R	Section
				5.12.2

+--------------------+----+--------------------+-----+-------------+

layout_blksize	65	uint32_t	R	Section
				5.12.3

+--------------------+----+--------------------+-----+-------------+

layout_hint	63	layouthint4	W	Section
				5.12.4

+--------------------+----+--------------------+-----+-------------+

layout_type	64	layouttype4<>	R	Section
				5.12.5

+--------------------+----+--------------------+-----+-------------+

maxfilesize	27	uint64_t	R	Section
				5.8.2.17

+--------------------+----+--------------------+-----+-------------+

maxlink	28	uint32_t	R	Section
				5.8.2.18

+--------------------+----+--------------------+-----+-------------+

maxname	29	uint32_t	R	Section
				5.8.2.19

+--------------------+----+--------------------+-----+-------------+

maxread	30	uint64_t	R	Section
				5.8.2.20

+--------------------+----+--------------------+-----+-------------+

maxwrite	31	uint64_t	R	Section
				5.8.2.21

+--------------------+----+--------------------+-----+-------------+

mdsthreshold	68	mdsthreshold4	R	Section
				5.12.6

+--------------------+----+--------------------+-----+-------------+

mimetype	32	utf8str_cs	R W	Section
				5.8.2.22

+--------------------+----+--------------------+-----+-------------+

mode	33	mode4	R W	Section
				6.2.4

+--------------------+----+--------------------+-----+-------------+

mode_set_masked	74	mode_masked4	W	Section
				6.2.5

+--------------------+----+--------------------+-----+-------------+

mounted_on_fileid	55	uint64_t	R	Section
				5.8.2.23

+--------------------+----+--------------------+-----+-------------+

no_trunc	34	bool	R	Section
				5.8.2.24

+--------------------+----+--------------------+-----+-------------+

numlinks	35	uint32_t	R	Section
				5.8.2.25

+--------------------+----+--------------------+-----+-------------+

owner	36	utf8str_mixed	R W	Section
				5.8.2.26

+--------------------+----+--------------------+-----+-------------+

owner_group	37	utf8str_mixed	R W	Section
				5.8.2.27

+--------------------+----+--------------------+-----+-------------+

quota_avail_hard	38	uint64_t	R	Section
				5.8.2.28

+--------------------+----+--------------------+-----+-------------+

quota_avail_soft	39	uint64_t	R	Section
				5.8.2.29

+--------------------+----+--------------------+-----+-------------+

quota_used	40	uint64_t	R	Section
				5.8.2.30

+--------------------+----+--------------------+-----+-------------+

rawdev	41	specdata4	R	Section
				5.8.2.31

+--------------------+----+--------------------+-----+-------------+

retentevt_get	71	retention_get4	R	Section
				5.13.3

+--------------------+----+--------------------+-----+-------------+

retentevt_set	72	retention_set4	W	Section
				5.13.4

+--------------------+----+--------------------+-----+-------------+

retention_get	69	retention_get4	R	Section
				5.13.1

+--------------------+----+--------------------+-----+-------------+

retention_hold	73	uint64_t	R W	Section
				5.13.5

+--------------------+----+--------------------+-----+-------------+

retention_set	70	retention_set4	W	Section
				5.13.2

+--------------------+----+--------------------+-----+-------------+

sacl	59	nfsacl41	R W	Section
				6.2.3

+--------------------+----+--------------------+-----+-------------+

space_avail	42	uint64_t	R	Section
				5.8.2.32

+--------------------+----+--------------------+-----+-------------+

space_free	43	uint64_t	R	Section
				5.8.2.33

+--------------------+----+--------------------+-----+-------------+

space_total	44	uint64_t	R	Section
				5.8.2.34

+--------------------+----+--------------------+-----+-------------+

space_used	45	uint64_t	R	Section
				5.8.2.35

+--------------------+----+--------------------+-----+-------------+

system

bool

R W

Section

   |                    |    |                    |     | 5.8.2.36    |
   +--------------------+----+--------------------+-----+-------------+
   | time_access        | 47 | nfstime4           | R   | Section     |
   |                    |    |                    |     | 5.8.2.37    |
   +--------------------+----+--------------------+-----+-------------+
   | time_access_set    | 48 | settime4           |   W | Section     |
   |                    |    |                    |     | 5.8.2.38    |
   +--------------------+----+--------------------+-----+-------------+
   | time_backup        | 49 | nfstime4           | R W | Section     |
   |                    |    |                    |     | 5.8.2.39    |
   +--------------------+----+--------------------+-----+-------------+
   | time_create        | 50 | nfstime4           | R W | Section     |
   |                    |    |                    |     | 5.8.2.40    |
   +--------------------+----+--------------------+-----+-------------+
   | time_delta         | 51 | nfstime4           | R   | Section     |
   |                    |    |                    |     | 5.8.2.41    |
   +--------------------+----+--------------------+-----+-------------+
   | time_metadata      | 52 | nfstime4           | R   | Section     |
   |                    |    |                    |     | 5.8.2.42    |
   +--------------------+----+--------------------+-----+-------------+
   | time_modify        | 53 | nfstime4           | R   | Section     |
   |                    |    |                    |     | 5.8.2.43    |
   +--------------------+----+--------------------+-----+-------------+
   | time_modify_set    | 54 | settime4           |   W | Section     |
   |                    |    |                    |     | 5.8.2.44    |
   +--------------------+----+--------------------+-----+-------------+

                                 Table 5

5.8. Attribute Definitions ✨

5.8.1. Definitions of REQUIRED Attributes 💖

5.8.1.1. Attribute 0: supported_attrs

Okay bestie, this one is a bit vector 🧚 that gets you every REQUIRED and RECOMMENDED attribute this object supports. Its scope covers every object with a matching fsid 💅

5.8.1.2. Attribute 1: type

This is how you figure out what kind of object you're looking at, using one of a bunch of special constants 🦋

NF4REG designates a regular file.
NF4DIR designates a directory.
NF4BLK designates a block device special file.
NF4CHR designates a character device special file.
NF4LNK designates a symbolic link.
NF4SOCK designates a named socket special file.
NF4FIFO designates a fifo special file.
NF4ATTRDIR designates a named attribute directory.
NF4NAMEDATTR designates a named attribute.

When the explanatory text and operation descriptions use these phrases, here's what they mean 🌸

The phrase "is a directory" means that the object's type attribute is NF4DIR or NF4ATTRDIR.
The phrase "is a special file" means that the object's type attribute is NF4BLK, NF4CHR, NF4SOCK, or NF4FIFO.
The phrases "is an ordinary file" and "is a regular file" mean that the object's type attribute is NF4REG or NF4NAMEDATTR.

5.8.1.3. Attribute 2: fh_expire_type

The server uses this to tell the client how filehandle expiration works. Check out Section 4 for the deets 🎀

5.8.1.4. Attribute 3: change

This is a value the server makes up 🫶 so the client can tell whether file data, directory contents, or attributes of the object got modified. The server may hand back the object's time_metadata attribute as this value, but only if the file system object can't be updated more often than the resolution of time_metadata. Lowkey clever ngl 🧠

5.8.1.5. Attribute 4: size

The size of the object in bytes 📏

5.8.1.6. Attribute 5: link_support

TRUE if the object's file system supports hard links 🔗

5.8.1.7. Attribute 6: symlink_support

TRUE if the object's file system supports symbolic links 🦋

5.8.1.8. Attribute 7: named_attr

TRUE if this object has named attributes. In other words, the object has a non-empty named attribute directory 📁

5.8.1.9. Attribute 8: fsid

This is the unique file system identifier for the file system holding this object 🆔 It has major and minor components, and each of them is of data type uint64_t 🔢

5.8.1.10. Attribute 9: unique_handles

TRUE if two distinct filehandles are guaranteed to refer to two different file system objects. No mix-ups, periodt 💜

5.8.1.11. Attribute 10: lease_time

The duration of the lease at the server, in seconds ⏳

5.8.1.12. Attribute 11: rdattr_error

This is the error returned when the server tries to retrieve attributes during a READDIR operation 🙈

5.8.1.13. Attribute 19: filehandle

The filehandle of this object. It's mainly for READDIR requests 🧚

5.8.1.14. Attribute 75: suppattr_exclcreat

This is the bit vector that would set all REQUIRED and RECOMMENDED attributes supported by the EXCLUSIVE4_1 method of file creation via the OPEN operation. Its scope applies to all objects with a matching fsid 🔥

5.8.2. Definitions of Uncategorized RECOMMENDED Attributes 🌈

Bestie, most of the RECOMMENDED attributes are defined right here 💅 The ones that share a common category get their own sections, so keep an eye out for those ✨

5.8.2.1. Attribute 14: archive

TRUE if this file has been archived since the last modification. (It's deprecated in favor of time_backup, so use that one instead 🍓)

5.8.2.2. Attribute 15: cansettime

TRUE if the server can change the times for a file system object as specified in a SETATTR operation ⏰

5.8.2.3. Attribute 16: case_insensitive

TRUE if file name comparisons on this file system are case insensitive 🔤

5.8.2.4. Attribute 17: case_preserving

TRUE if file name case on this file system is preserved 🎀

5.8.2.5. Attribute 60: change_policy

This is a value the server creates so the client can check whether some server policy tied to the current file system has changed 👀 If the value stays the same, the client can be sure the attributes related to fs location and the fss_type field of the fs_status attribute haven't changed. But if the value changes, that doesn't necessarily mean the policy changed. It's on the client to ask the server whether any policy it cares about actually changed 💀 See Section 3.3.6 for the full tea.

This attribute MUST change when the value returned by the fs_locations or fs_locations_info attribute changes, when a file system goes from read-only to writable or vice versa, or when the allowable set of security flavors for the file system or any part thereof is changed. This one is non-negotiable, no cap 🚨

5.8.2.6. Attribute 18: chown_restricted

If TRUE, the server will reject any request to change either the owner or the group of a file when the caller isn't a privileged user. Examples of privileged users are "root" in UNIX operating environments, or anyone holding the "Take Ownership" privilege in Windows 2000 👑

5.8.2.7. Attribute 20: fileid

A number that uniquely identifies the file within the file system 🪪

5.8.2.8. Attribute 21: files_avail

The file slots available to this user on the file system holding this object. This should be the smallest relevant limit 📉

5.8.2.9. Attribute 22: files_free

The free file slots on the file system holding this object. This should be the smallest relevant limit 🫧

5.8.2.10. Attribute 23: files_total

The total file slots on the file system holding this object 🗂️

5.8.2.11. Attribute 76: fs_charset_cap

The character set capabilities for this file system. See Section 14.4 🔤✨

5.8.2.12. Attribute 24: fs_locations

These are the locations where this file system may be found 🗺️ If the server returns NFS4ERR_MOVED as an error, this attribute MUST be supported. See Section 11.16 for more details.

5.8.2.13. Attribute 67: fs_locations_info

This is the full-function file system location. See Section 11.17.2 for more details 📍

5.8.2.14. Attribute 61: fs_status

This is the generic file system type information. See Section 11.18 for more details 🧾

5.8.2.15. Attribute 25: hidden

TRUE if the file is considered hidden with respect to the Windows API. Shh 🤫

5.8.2.16. Attribute 26: homogeneous

TRUE if this object's file system is homogeneous. That means all objects in the file system (all objects on the server with the same fsid) have common values for all per-file-system attributes 🧩

5.8.2.17. Attribute 27: maxfilesize

The maximum supported file size for the file system of this object 🐘

5.8.2.18. Attribute 28: maxlink

The maximum number of links for this object 🔗

5.8.2.19. Attribute 29: maxname

The maximum file name size supported for this object 🏷️

5.8.2.20. Attribute 30: maxread

The maximum amount of data the READ operation will return for this object 📖

5.8.2.21. Attribute 31: maxwrite

The maximum amount of data the WRITE operation will accept for this object ✍️ This attribute SHOULD be supported if the file is writable. Skipping it can lead to the client either wasting bandwidth or not getting the best performance, and nobody wants that 😭

5.8.2.22. Attribute 32: mimetype

The MIME body type/subtype of this object 🎀

5.8.2.23. Attribute 55: mounted_on_fileid 🦋

okay so mounted_on_fileid is like fileid's glow-up 💅 but if the target filehandle is the root of a file system, it gives you the fileid of the underlying directory instead 🌸

UNIX-based systems plug a file system into the namespace by mounting it onto an existing file object (the mount point, usually a directory) in another file system 🧚 When you read the mount point's parent directory with something like readdir(), you get directory entries, and each one has a component name and a fileid. Here's the tea: the fileid on the mount point's directory entry will be different from the one stat() returns 😭 stat() gives you the fileid of the root of the mounted file system, but readdir() gives you the fileid that stat() would have returned before any file systems were mounted on that mount point. Plot twist, right? 🙈

Unlike NFSv3, NFSv4.1 lets a client's LOOKUP request cross other file systems 🚗 The client can tell it crossed when the filehandle argument of LOOKUP has an fsid attribute that's different from the one on the filehandle LOOKUP returns. A UNIX-based client will call this a "mount point crossing" 💖 UNIX also has a legacy way for a process to figure out its current working directory. It relies on readdir() of a mount point's parent and stat() of the mount point, both returning fileids the way we just went over. The mounted_on_fileid attribute matches the fileid that readdir() would have returned 🫶

Now, the NFSv4.1 client could just make up a fileid that matches what mounted_on_fileid gives, and if the server doesn't support mounted_on_fileid, the client literally has no choice 🥲 But there's a risk that the made-up fileid collides with one already assigned to another object in the file system, and that's a mess 💀 So if the server can provide mounted_on_fileid, the chance of client operational problems in this area is totally eliminated ✨

If the server detects that there is no mount point at the target file object, then the value of mounted_on_fileid it returns is the same as the fileid attribute 🍓

The mounted_on_fileid attribute is RECOMMENDED, so the server SHOULD provide it if possible 💜 For a UNIX-based server, this is super straightforward 👑 Usually mounted_on_fileid gets requested during a READDIR operation, and then it's trivial (at least for UNIX-based servers) to return it, because it's equal to the fileid of a directory entry that readdir() returned. If mounted_on_fileid is requested in a GETATTR operation, the server should obey an invariant: the value it returns has to equal the file object's entry in the object's parent directory, i.e., what readdir() would have returned 🎀 Some operating environments let two or more file systems get mounted onto a single mount point. In that case, for the server to keep that invariant, it needs to find the base mount point, not the intermediate ones 🌈

5.8.2.24. Attribute 34: no_trunc 🌸

If this attribute is TRUE, then if the client uses a file name longer than name_max, an error gets returned instead of the name being truncated 🙅‍♀️ no silent chopping, bestie 💅

5.8.2.25. Attribute 35: numlinks 🔗

Number of hard links to this object. ✨

5.8.2.26. Attribute 36: owner 👤

The string name of the owner of this object. 🫶

5.8.2.27. Attribute 37: owner_group 👥

The string name of the group ownership of this object. 🧁

5.8.2.28. Attribute 38: quota_avail_hard 💾

The value in bytes that represents the amount of additional disk space beyond the current allocation that can be allocated to this file or directory before further allocations will be refused. It is understood that this space may be consumed by allocations to other files or directories. 🥹

5.8.2.29. Attribute 39: quota_avail_soft 💛

The value in bytes that represents the amount of additional disk space that can be allocated to this file or directory before the user may reasonably be warned. It is understood that this space may be consumed by allocations to other files or directories, though there is a rule as to which other files or directories. 🌤️

5.8.2.30. Attribute 40: quota_used 📊

The value in bytes that represents the amount of disk space used by this file or directory and possibly a number of other similar files or directories, where the set of "similar" meets at least the criterion that allocating space to any file or directory in the set will reduce the quota_avail_hard of every other file or directory in the set. 🧮

Note that there may be a number of distinct but overlapping sets of files or directories for which a quota_used value is maintained, e.g., "all files with a given owner", "all files with a given group owner", etc. The server is at liberty to choose any of those sets when providing the content of the quota_used attribute, but should do so in a repeatable way. The rule may be configured per file system or may be "choose the set with the smallest quota". 🎯 so many options, the server is in control, periodt 💅

5.8.2.31. Attribute 41: rawdev 🧸

Raw device number of file of type NF4BLK or NF4CHR. The device number is split into major and minor numbers. If the file's type attribute is not NF4BLK or NF4CHR, the value returned SHOULD NOT be considered useful. 🚫 literally don't trust it in that case 💀

5.8.2.32. Attribute 42: space_avail 🛏️

Disk space in bytes available to this user on the file system containing this object. This should be the smallest relevant limit. 📏

5.8.2.33. Attribute 43: space_free 🫧

Free disk space in bytes on the file system containing this object. This should be the smallest relevant limit. 🍬

5.8.2.34. Attribute 44: space_total 🌍

Total disk space in bytes on the file system containing this object. 🧺

5.8.2.35. Attribute 45: space_used 📦

Number of file system bytes allocated to this object. 🎈

5.8.2.36. Attribute 46: system 🪟

This attribute is TRUE if this file is a "system" file with respect to the Windows operating environment. 💻

5.8.2.37. Attribute 47: time_access ⏰

The time_access attribute represents the time of last access to the object by a READ operation sent to the server. What counts as an "access" depends on the server's operating environment and/or the server's file system semantics. For example, for servers obeying Portable Operating System Interface (POSIX) semantics, time_access would be updated only by the READ and READDIR operations and not by any of the operations that modify the content of the object [13], [14], [15]. Of course, setting the corresponding time_access_set attribute is another way to modify the time_access attribute 🕰️

Whenever the file object resides on a writable file system, the server should make its best efforts to record time_access into stable storage 💾 However, to mitigate the performance effects of doing that, and most especially whenever the server is satisfying the read of the object's content from its cache, the server MAY cache access time updates and lazily write them to stable storage 🦥 It's also acceptable to give administrators of the server the option to disable time_access updates 🙌

5.8.2.38. Attribute 48: time_access_set ✍️

Sets the time of last access to the object. SETATTR use only. 🚪

5.8.2.39. Attribute 49: time_backup 📼

The time of last backup of the object. 🗂️

5.8.2.40. Attribute 50: time_create 🎂

The time of creation of the object. This attribute does not have any relation to the traditional UNIX file attribute "ctime" or "change time". Not the same vibe, so don't mix them up 🙈

5.8.2.41. Attribute 51: time_delta ⏱️

Smallest useful server time granularity. 🔬

5.8.2.42. Attribute 52: time_metadata 🏷️

The time of last metadata modification of the object. 🧾

5.8.2.43. Attribute 53: time_modify ✏️

The time of last modification to the object. 🌷

5.8.2.44. Attribute 54: time_modify_set 🖊️

Sets the time of last modification to the object. SETATTR use only. 🌟

5.9. Interpreting owner and owner_group ✨

okay so the RECOMMENDED attributes owner and owner_group (plus the users and groups hiding inside the acl attribute) are all represented as a UTF-8 string 🫶. the whole point is NOT tying them to one client's or server's internal format, so UTF-8 is the neutral ground everyone can stand on 🌸. Section 6.1 of RFC 2624 [53] has more of the reasoning if you're curious 🧚.

each side keeps its own local way of storing owner and owner_group for its own stuff 📁. but when these travel between client and server, the local version gets translated into the form user@dns_domain 💅. that way a client and server that don't share the same local representation can still both read the same common syntax 🦋.

security principals can be represented in different ways depending on the security mechanism 🔐. servers usually translate those into a common format (generally the one used for local storage) so they can figure out which users each principal belongs to 👑. when those local identifiers get turned into the owner attribute for files created by those principals, they identify, in a common format, the users behind each set of principals ✨.

the translation used to interpret owner and group strings is NOT part of the protocol, so lots of solutions can work 🌈. for example, a local translation table might map a numeric ID to user@dns_domain, or a name service could handle it 📒. a server can also be more general and not limited by any particular translation, just storing owner and owner_group in local storage without translating anything 💜. or it can mix methods: store the entire string for attributes where no translation exists, and use the local representation wherever a translation IS available 🍓.

servers that don't support every possible value of owner and owner_group SHOULD return an error (NFS4ERR_BADOWNER) when a string with no translation shows up as the value to be set for a SETATTR of the owner, owner_group, or acl attributes 🙅‍♀️. when a server accepts an owner or owner_group value as valid on a SETATTR (same goes for the owner and group strings in an acl), it's promising to hand back that exact same string when a matching GETATTR happens 🤞. but configuration changes (including changes to how the string maps to the local representation) and ill-constructed name translations (the ones with aliasing) might make that promise impossible to keep 💀. servers should make appropriate efforts to avoid having these attributes change value when no real change to ownership has occurred 🫶.

the dns_domain portion of the owner string is meant to be a DNS domain name, like user@example.org 🌐. servers should accept as valid a set of users for at least one domain. a server may treat other domains as having no valid translations. a more general service happens when a server can accept users for multiple domains, or for all domains, subject to security constraints 🔒.

when there's no translation available to the client or server, the attribute value gets built WITHOUT the @ 🙈. so a missing @ means the sender is saying "no translation here," and the receiver should NOT use that string as a basis for translating into its own internal format 🚫. even though it can't be translated, it can still be useful: a client can use the string for local display of ownership 👀.

to be more compatible with NFSv3 (which identified users and groups by 32-bit unsigned user identifiers and group identifiers 🔢), owner and group strings made of decimal numeric values with no leading zeros can get a special interpretation from clients and servers that choose to support it 🧚. the receiver may treat such a string as representing the same user as an NFSv3 uid or gid with that numeric value. a server isn't obligated to accept such a string, and it may return an NFS4ERR_BADOWNER instead.

to stop this trick being used to dodge user and group translation (like a client sending every owner and group in numeric form 😤), a server SHOULD return an NFS4ERR_BADOWNER error when there IS a valid translation for the user or owner designated this way. in that case, the client must use the appropriate name@domain string, not the special numeric form 💅.

the owner string nobody may be used to designate an anonymous user, which gets associated with a file created by a security principal that can't be mapped through normal means to the owner attribute 👻. users and implementations of NFSv4.1 SHOULD NOT use nobody to designate a real user whose access isn't anonymous 🚫💀.

5.10. Character Case Attributes 💅

with respect to the case_insensitive and case_preserving attributes, each UCS-4 character (which UTF-8 encodes) can be mapped according to Appendix B.2 of RFC 3454 [16]. for general character handling and internationalization issues, see Section 14 🌍.

5.11. Directory Notification Attributes 🔔

as described in Section 18.39, the client can request a minimum delay for notifications of changes to attributes, but the server is free to ignore what the client requests 🙃. the client can figure out in advance which notification delays the server will accept by sending a GETATTR operation for either or both of two directory notification attributes. when the client calls the GET_DIR_DELEGATION operation and asks for attribute change notifications, it should request notification delays that are no less than the values in the server-provided attributes 📬.

5.11.1. Attribute 56: dir_notif_delay ⏰

the dir_notif_delay attribute is the minimum number of seconds the server will delay before notifying the client of a change to the directory's attributes.

5.11.2. Attribute 57: dirent_notif_delay ⏳

the dirent_notif_delay attribute is the minimum number of seconds the server will delay before notifying the client of a change to a file object that has an entry in the directory.

5.12. pNFS Attribute Definitions 🧩

5.12.1. Attribute 62: fs_layout_type 🗺️

the fs_layout_type attribute (see Section 3.3.13) applies to a file system and indicates what layout types are supported by the file system. when the client encounters a new fsid, the client SHOULD obtain the value for the fs_layout_type attribute associated with the new file system. this attribute is used by the client to determine if the layout types supported by the server match any of the client's supported layout types.

5.12.2. Attribute 66: layout_alignment 📐

when a client holds layouts on files of a file system, the layout_alignment attribute indicates the preferred alignment for I/O to files on that file system. where possible, the client should send READ and WRITE operations with offsets that are whole multiples of the layout_alignment attribute.

5.12.3. Attribute 65: layout_blksize 🧱

when a client holds layouts on files of a file system, the layout_blksize attribute indicates the preferred block size for I/O to files on that file system. where possible, the client should send READ operations with a count argument that is a whole multiple of layout_blksize, and WRITE operations with a data argument of size that is a whole multiple of layout_blksize.

5.12.4. Attribute 63: layout_hint 💡

the layout_hint attribute (see Section 3.3.19) may be set on newly created files to influence the metadata server's choice for the file's layout. if possible, this attribute is one of those set in the initial attributes within the OPEN operation. the metadata server may choose to ignore this attribute. the layout_hint attribute is a subset of the layout structure returned by LAYOUTGET. for example, instead of specifying particular devices, this would be used to suggest the stripe width of a file. the server implementation determines which fields within the layout will be used.

5.12.5. Attribute 64: layout_type 📋

this attribute lists the layout type(s) available for a file. the value returned by the server is for informational purposes only. the client will use the LAYOUTGET operation to obtain the information needed in order to perform I/O, for example, the specific device information for the file and its layout.

5.12.6. Attribute 68: mdsthreshold 🚦

this attribute is a server-provided hint used to communicate to the client when it is more efficient to send READ and WRITE operations to the metadata server or the data server. the two types of thresholds described are file size thresholds and I/O size thresholds. if a file's size is smaller than the file size threshold, data accesses SHOULD be sent to the metadata server. if an I/O request has a length that is below the I/O size threshold, the I/O SHOULD be sent to the metadata server. each threshold type is specified separately for read and write.

the server MAY provide both types of thresholds for a file. if both file size and I/O size are provided, the client SHOULD reach or exceed both thresholds before sending its read or write requests to the data server. alternatively, if only one of the specified thresholds is reached or exceeded, the I/O requests are sent to the metadata server.

for each threshold type, a value of zero indicates no READ or WRITE should be sent to the metadata server, while a value of all ones indicates that all READs or WRITEs should be sent to the metadata server.

the attribute is available on a per-filehandle basis. if the current filehandle refers to a non-pNFS file or directory, the metadata server should return an attribute that is representative of the filehandle's file system. it is suggested that this attribute is queried as part of the OPEN operation. due to dynamic system changes, the client should not assume that the attribute will remain constant for any specific time period; thus, it should be periodically refreshed.

5.13. Retention Attributes 🔒

retention is a concept whereby a file object can be placed in an immutable, undeletable, unrenamable state for a fixed or infinite duration of time. once in this "retained" state, the file cannot be moved out of the state until the duration of retention has been reached 🕰️.

when retention is enabled, retention MUST extend to the data of the file, and the name of file. the server MAY extend retention to any other property of the file, including any subset of REQUIRED, RECOMMENDED, and named attributes, with the exceptions noted in this section ✨.

servers MAY support or not support retention on any file object type 🎀.

the five retention attributes are explained in the next subsections 💖.

5.13.1. Attribute 69: retention_get 🎀

okay so if retention is turned on for the file, this attribute tells you the retention begin time 🕰️ bestie it's READ ONLY, you can only GETATTR it, and it MUST NOT be changed by SETATTR (Section 5.5) 🙅‍♀️ the value is built like this:

const RET4_DURATION_INFINITE    = 0xffffffffffffffff;
struct retention_get4 {
        uint64_t        rg_duration;
        nfstime4        rg_begin_time<1>;
};

rg_duration is how many seconds the file will be kept once retention is on ⏳ and rg_begin_time is an array that can hold up to ONE absolute time value 💅 if that array is zero length, it means no begin time was ever set and retention is NOT enabled. and if rg_duration is RET4_DURATION_INFINITE, the file gets kept forever 🌈✨

and here's the plot twist 😭 as soon as rg_duration hits zero, rg_begin_time becomes zero length too, and retention is not (no longer) enabled 💀 it's gone, babe.

5.13.2. Attribute 70: retention_set 💜

this one is how you SET the retention duration and, if you want, turn retention on for the file 💖 it's SETATTR only, you MUST NOT be able to GETATTR it (Section 5.5), and it's the write side of retention_get 🦋 the value looks like this:

struct retention_set4 {
        bool            rs_enable;
        uint64_t        rs_duration<1>;
};

if the client sets rs_enable to TRUE, retention turns on and the begin time is the server's current time and date ⏰ you can also pass a duration if rs_duration is length one. that duration counts in seconds from the begin time, and if it's RET4_DURATION_INFINITE, the file is kept forever 🫶 if retention gets enabled and nobody ever gave it a duration (not in this SETATTR, not in an earlier one), the duration just defaults to zero seconds 🙈

the server MAY restrict turning retention on, or how long it lasts, based on the ACE4_WRITE_RETENTION ACL permission 🔐 and enabling retention MUST NOT stop you from enabling event-based retention or changing the retention_hold attribute 💅

these rules hit both retention_set AND retentevt_set, so listen up 👑

as long as retention is NOT enabled, the client can decrease the duration 🌸
the duration can always be set to an equal or higher value, even if retention is enabled 🍓 but the real duration (what retention_get or retentevt_get shows; see Section 5.13.1 or Section 5.13.3) is literally counting down to zero, one unit per second 😭 unless it was set to RET4_DURATION_INFINITE. so no, you cannot precisely extend the duration on a file that's already retained. it is what it is 💀
while retention is enabled, trying to disable it or shorten its duration MUST fail with the error NFS4ERR_INVAL 🚫
if the principal trying to change retention_set or retentevt_set does NOT have ACE4_WRITE_RETENTION permissions, the attempt MUST fail with NFS4ERR_ACCESS 🧚

5.13.3. Attribute 71: retentevt_get 🦋

this one gets the event-based retention duration, and if it's enabled, the event-based retention begin time too 💖 it's basically retention_get but for event-based retention. the event that triggers event-based retention is NOT defined by the NFSv4.1 specification, so that's a whole other vibe 🙃

5.13.4. Attribute 72: retentevt_set 🌈

this sets the event-based retention duration and can turn event-based retention on for the file 💅 it's the set side of retentevt_get and works just like retention_set, but for event-based retention ✨ now the big rule: when event-based retention is set, the file MUST be retained even if non-event-based retention was set AND its duration already ran out 😭 and the flip side is just as strict: when non-event-based retention is set, the file MUST be retained even if event-based retention was set AND its duration already ran out 💜 both have to be satisfied, no shortcuts, bestie.

the server MAY restrict turning on event-based retention, or how long it lasts, based on the ACE4_WRITE_RETENTION ACL permission 🔐 and turning on event-based retention MUST NOT stop you from enabling non-event-based retention or from changing the retention_hold attribute 🫶

5.13.5. Attribute 73: retention_hold 🔒

this one gets OR sets the administrative retention holds, with one hold per bit position 🍓 you get up to 64 administrative holds, one for each bit on the attribute 🎀 if retention_hold is not zero, the file MUST NOT be deleted, renamed, or modified, EVEN IF the duration on enabled event or non-event-based retention has already been reached 😭 the server MAY restrict changing retention_hold based on the ACE4_WRITE_RETENTION_HOLD ACL permission 🔐 and turning on administrative holds does NOT stop you from enabling event-based or non-event-based retention 🦋

if the principal trying to change retention_hold doesn't have ACE4_WRITE_RETENTION_HOLD permissions, the attempt MUST fail with NFS4ERR_ACCESS 🚫

6. Access Control Attributes ✨

okay access control time, and this is where it gets really iconic 💅 ACLs (Access Control Lists) are file attributes that handle fine-grained access control 🔐 this section covers the "acl", "dacl", "sacl", "aclsupport", "mode", and "mode_set_masked" file attributes and how they interact with each other 🧚 and FYI, file attributes can apply to ANY file system object, not just regular files 🌸

6.1. Goals 🎯

ACLs and modes are two super established ways of specifying permissions 👑 this section lays out requirements that try to hit the following goals:

if a server supports the mode attribute, it should provide reasonable semantics to clients that only set and retrieve the mode attribute 🌸
if a server supports ACL attributes, it should provide reasonable semantics to clients that only set and retrieve those attributes 🍓
on servers that support the mode attribute, if ACL attributes have never been set on an object, either via inheritance or explicitly, the behavior should be traditional UNIX-like behavior 🦋
on servers that support the mode attribute, if the ACL attributes have been previously set on an object, either explicitly or via inheritance: 🧚
- setting only the mode attribute should effectively control the traditional UNIX-like permissions of read, write, and execute on owner, owner_group, and other 💅
- setting only the mode attribute should provide reasonable security. for example, setting a mode of 000 should be enough to make sure that future OPEN operations for OPEN4_SHARE_ACCESS_READ or OPEN4_SHARE_ACCESS_WRITE by any principal fail, no matter what ACL existed before or got inherited 😭🔥
NFSv4.1 may introduce different semantics for the mode and ACL attributes, but it does NOT render any previously existing implementations invalid ✨ this section also adds clarifications based on earlier implementations and the discussions around them 🫶
on servers that support both the mode and the acl or dacl attributes, the server must keep the two consistent with each other 💜 the value of the mode attribute (except for the three high-order bits described in Section 6.2.4) must be determined ENTIRELY by the ACL's value, so using the mode is never required for anything other than setting those three high-order bits. see Section 6.4.1 for the exact requirements 🌈
when a mode attribute is set on an object, the ACL attributes may need to be modified so they don't conflict with the new mode 👑 in those cases it's desirable for the ACL to keep as much information as possible. that includes inheritance info, AUDIT and ALARM ACEs, and any permissions granted or denied that don't conflict with the new mode 🎀

6.2. File Attributes Discussion 🦋

6.2.1. Attribute 12: acl 🎀

okay bestie, the acl attribute is a whole list of Access Control Entries (ACEs) attached to each file system object 💅 Clients can set and get it, but the server is the one actually using it to do the access control 👑 If you only want to check access without touching any data or metadata, you can use the OPEN or ACCESS operations 🦋

The NFS ACE structure looks like this:

typedef uint32_t        acetype4;

typedef uint32_t aceflag4;

typedef uint32_t        acemask4;

struct nfsace4 {
        acetype4        type;
        aceflag4        flag;
        acemask4        access_mask;
        utf8str_mixed   who;
};

Okay so here's how a request actually gets judged 🌸 The server goes through each nfsace4 entry in order. Only the ACEs whose "who" matches the requester even get a look 👀 Each ACE gets processed until every bit of the requester's access has been ALLOWED. Once a bit has been ALLOWED by an ACCESS_ALLOWED_ACE, it's done and no longer counts in the later ACEs. If an ACCESS_DENIED_ACE shows up and the requester's access still has unALLOWED bits in common with that ACE's "access_mask", the request is denied 🙅‍♀️ Once the whole ACL has been processed, if there are still bits in the requester's mask that were neither ALLOWED nor DENIED, access is denied. no cap, it's strict 💀

The ALARM and AUDIT ACE types are the side-quest ones ✨ They don't change anyone's access at all. Instead, they exist to trigger events when someone tries to get in. That's why AUDIT and ALARM ACEs only get processed after the ALLOW and DENY ACEs are done 🧚

The NFSv4.1 ACL model is super rich 💜 Some server platforms have access control that goes past the UNIX-style mode attribute, but it's still not as rich as the NFS ACL model. So that users can still use that more limited functionality, the server may support the acl attributes by mapping between its own ACL model and the NFSv4.1 ACL model. Servers must ensure that the ACL they actually store or enforce is at least as strict as the NFSv4 ACL that was set. It's tempting to just reject any ACL outside the small set that can be represented accurately, but that can leave ACLs totally unusable unless the client knows the server's mapping, and that defeats the whole point of a common NFSv4 ACL protocol 🙄 So servers should accept every ACL that they can without compromising security. To help with that, servers may make a special exception, for unsupported permission bits, to the rule that bits not ALLOWED or DENIED by an ACL must be denied. For example, a UNIX-style server might choose to silently allow read attribute permissions even though an ACL doesn't explicitly allow them. (An ACL that explicitly denies permission to read attributes should still be rejected 🚫)

The situation gets messy because a server might have multiple modules enforcing ACLs 🌈 For example, enforcement for NFSv4.1 access may be different from, but not weaker than, enforcement for local access, and both may be different from enforcement for access through other protocols like SMB (Server Message Block). So it may be useful for a server to accept an ACL even if not all of its modules can support it 🫶

The guiding principle for NFSv4 access is this: the server must not accept ACLs that appear to make access to the file more restrictive than it really is 🔥

6.2.1.1. ACE Type 🦋

Here are the constants used for the type field (acetype4):

const ACE4_ACCESS_ALLOWED_ACE_TYPE      = 0x00000000;
const ACE4_ACCESS_DENIED_ACE_TYPE       = 0x00000001;
const ACE4_SYSTEM_AUDIT_ACE_TYPE        = 0x00000002;
const ACE4_SYSTEM_ALARM_ACE_TYPE        = 0x00000003;

Only the ALLOWED and DENIED bits may be used in the dacl attribute, and only the AUDIT and ALARM bits may be used in the sacl attribute. All four are permitted in the acl attribute ✨

+==============================+==============+=====================+
| Value                        | Abbreviation | Description         |
+==============================+==============+=====================+
| ACE4_ACCESS_ALLOWED_ACE_TYPE | ALLOW        | Explicitly grants   |
|                              |              | the access          |
|                              |              | defined in          |
|                              |              | acemask4 to the     |
|                              |              | file or             |
|                              |              | directory.          |
+------------------------------+--------------+---------------------+
| ACE4_ACCESS_DENIED_ACE_TYPE  | DENY         | Explicitly denies   |
|                              |              | the access          |
|                              |              | defined in          |
|                              |              | acemask4 to the     |
|                              |              | file or             |
|                              |              | directory.          |
+------------------------------+--------------+---------------------+
| ACE4_SYSTEM_AUDIT_ACE_TYPE   | AUDIT        | Log (in a system-   |
|                              |              | dependent way)      |
|                              |              | any access          |
|                              |              | attempt to a file   |
|                              |              | or directory that   |
|                              |              | uses any of the     |
|                              |              | access methods      |
|                              |              | specified in        |
|                              |              | acemask4.           |
+------------------------------+--------------+---------------------+
| ACE4_SYSTEM_ALARM_ACE_TYPE   | ALARM        | Generate an alarm   |
|                              |              | (in a system-       |
|                              |              | dependent way)      |
|                              |              | when any access     |
|                              |              | attempt is made     |
|                              |              | to a file or        |
|                              |              | directory for the   |
|                              |              | access methods      |
|                              |              | specified in        |
|                              |              | acemask4.           |
+------------------------------+--------------+---------------------+

                                  Table 6

The "Abbreviation" column is the shorthand we'll use for these types for the rest of this section 💖

6.2.1.2. Attribute 13: aclsupport 🌟

A server doesn't need to support all of the ACE types above. This attribute tells you which ACE types are supported for the current file system 🍓 The bitmask constants used to represent those definitions in the aclsupport attribute are:

const ACL4_SUPPORT_ALLOW_ACL    = 0x00000001;
const ACL4_SUPPORT_DENY_ACL     = 0x00000002;
const ACL4_SUPPORT_AUDIT_ACL    = 0x00000004;
const ACL4_SUPPORT_ALARM_ACL    = 0x00000008;

Servers that support either the ALLOW or DENY ACE type SHOULD support both ALLOW and DENY ACE types 💜

Clients should not attempt to set an ACE unless the server claims support for that ACE type 🙈 If the server gets a request to set an ACE that it can't store, it MUST reject the request with NFS4ERR_ATTRNOTSUPP 🚫 If the server gets a request to set an ACE that it can store but can't enforce, the server SHOULD reject the request with NFS4ERR_ATTRNOTSUPP 😭

Support for any of the ACL attributes is optional (albeit RECOMMENDED). Still, a server that supports either of the new ACL attributes (dacl or sacl) MUST allow use of those new attributes to access all of the ACE types it supports 👑 In other words, if such a server supports ALLOW or DENY ACEs, then it MUST support the dacl attribute, and if it supports AUDIT or ALARM ACEs, then it MUST support the sacl attribute. literally the same energy as "if you offer it, you gotta deliver it" 💅

6.2.1.3. ACE Access Mask 🎀

Here are the bitmask constants for the access mask field:

const ACE4_READ_DATA            = 0x00000001;
const ACE4_LIST_DIRECTORY       = 0x00000001;
const ACE4_WRITE_DATA           = 0x00000002;
const ACE4_ADD_FILE             = 0x00000002;
const ACE4_APPEND_DATA          = 0x00000004;
const ACE4_ADD_SUBDIRECTORY     = 0x00000004;
const ACE4_READ_NAMED_ATTRS     = 0x00000008;
const ACE4_WRITE_NAMED_ATTRS    = 0x00000010;
const ACE4_EXECUTE              = 0x00000020;
const ACE4_DELETE_CHILD         = 0x00000040;
const ACE4_READ_ATTRIBUTES      = 0x00000080;
const ACE4_WRITE_ATTRIBUTES     = 0x00000100;
const ACE4_WRITE_RETENTION      = 0x00000200;
const ACE4_WRITE_RETENTION_HOLD = 0x00000400;

const ACE4_DELETE               = 0x00010000;
const ACE4_READ_ACL             = 0x00020000;
const ACE4_WRITE_ACL            = 0x00040000;
const ACE4_WRITE_OWNER          = 0x00080000;
const ACE4_SYNCHRONIZE          = 0x00100000;

Okay plot twist 😭 some of these masks have coincident values, like ACE4_READ_DATA and ACE4_LIST_DIRECTORY, which are both 0x00000001. The mask entries ACE4_LIST_DIRECTORY, ACE4_ADD_FILE, and ACE4_ADD_SUBDIRECTORY are meant for directory objects, while ACE4_READ_DATA, ACE4_WRITE_DATA, and ACE4_APPEND_DATA are meant for non-directory objects 🧚 same bits, different vibes, iykyk ✨

6.2.1.3.1. Discussion of Mask Attributes ✨

okay so this is where we go through each ACE access mask bit one at a time 💅 every bit gets its own entry with the operations it affects and then the discussion, and it's giving full main character energy 🦋

ACE4_READ_DATA 📖

Operation(s) affected:
- READ
- OPEN
Discussion: permission to read the data of the file. Servers SHOULD allow a user the ability to read the data of the file when only the ACE4_EXECUTE access mask bit is allowed.

ACE4_LIST_DIRECTORY 🌸

Operation(s) affected:
- READDIR
Discussion: permission to list the contents of a directory.

ACE4_WRITE_DATA ✏️

Operation(s) affected:
- WRITE
- OPEN
- SETATTR of size
Discussion: permission to modify a file's data.

ACE4_ADD_FILE 📁

Operation(s) affected:
- CREATE
- LINK
- OPEN
- RENAME
Discussion: permission to add a new file in a directory. The CREATE operation is affected when nfs_ftype4 is NF4LNK, NF4BLK, NF4CHR, NF4SOCK, or NF4FIFO. (NF4DIR is not listed because it's covered by ACE4_ADD_SUBDIRECTORY.) OPEN is affected when it's used to create a regular file. LINK and RENAME are always affected.

ACE4_APPEND_DATA 🧷

Operation(s) affected:
- WRITE
- OPEN
- SETATTR of size
Discussion: the ability to modify a file's data, but only starting at EOF. This makes append-only files possible, by allowing ACE4_APPEND_DATA and denying ACE4_WRITE_DATA to the same user or group. If a file has an ACL such as the one described above and a WRITE request is made for somewhere other than EOF, the server SHOULD return NFS4ERR_ACCESS.

ACE4_ADD_SUBDIRECTORY 🗂️

Operation(s) affected:
- CREATE
- RENAME
Discussion: permission to create a subdirectory in a directory. The CREATE operation is affected when nfs_ftype4 is NF4DIR. The RENAME operation is always affected.

ACE4_READ_NAMED_ATTRS 🔍

Operation(s) affected:
- OPENATTR
Discussion: permission to read the named attributes of a file or to look up the named attribute directory. OPENATTR is affected when it isn't used to create a named attribute directory. That's when 1) createdir is TRUE, but a named attribute directory already exists, or 2) createdir is FALSE.

ACE4_WRITE_NAMED_ATTRS 🏷️

Operation(s) affected:
- OPENATTR
Discussion: permission to write the named attributes of a file or to create a named attribute directory. OPENATTR is affected when it's used to create a named attribute directory. That's when createdir is TRUE and no named attribute directory exists. The ability to check whether or not a named attribute directory exists depends on the ability to look it up, so users also need the ACE4_READ_NAMED_ATTRS permission in order to create a named attribute directory.

ACE4_EXECUTE 🏃‍♀️ (files)

Operation(s) affected:
- READ
- OPEN
- REMOVE
- RENAME
- LINK
- CREATE
Discussion: permission to execute a file.

Servers SHOULD allow a user the ability to read the data of the file when only the ACE4_EXECUTE access mask bit is allowed. That's because there's no way to execute a file without reading its contents. Though a server may treat ACE4_EXECUTE and ACE4_READ_DATA bits identically when deciding to permit a READ operation, it SHOULD still allow the two bits to be set independently in ACLs, and it MUST distinguish between them when replying to ACCESS operations. In particular, servers SHOULD NOT silently turn on one of the two bits when the other is set, because that would make it impossible for the client to correctly enforce the distinction between read and execute permissions.

For example, after a SETATTR of the following ACL:

nfsuser:ACE4_EXECUTE:ALLOW

A later GETATTR of ACL for that file SHOULD return:

nfsuser:ACE4_EXECUTE:ALLOW

Rather than:

nfsuser:ACE4_EXECUTE/ACE4_READ_DATA:ALLOW

ACE4_EXECUTE 🚶‍♀️ (directories)

Operation(s) affected:
- LOOKUP
Discussion: permission to traverse or search a directory.

ACE4_DELETE_CHILD 🗑️

Operation(s) affected:
- REMOVE
- RENAME
Discussion: permission to delete a file or directory within a directory. See Section 6.2.1.3.2 for information on how ACE4_DELETE and ACE4_DELETE_CHILD interact.

ACE4_READ_ATTRIBUTES 👀

Operation(s) affected:
- GETATTR of file system object attributes
- VERIFY
- NVERIFY
- READDIR
Discussion: the ability to read basic attributes (non-ACLs) of a file. On a UNIX system, basic attributes can be thought of as the stat-level attributes. Allowing this access mask bit would mean that the entity can run ls -l and stat. If a READDIR operation requests attributes, this mask must be allowed for the READDIR to succeed.

ACE4_WRITE_ATTRIBUTES ⏰

Operation(s) affected:
- SETATTR of time_access_set, time_backup, time_create, time_modify_set, mimetype, hidden, system
Discussion: permission to change the times associated with a file or directory to an arbitrary value. Also permission to change the mimetype, hidden, and system attributes. A user having ACE4_WRITE_DATA or ACE4_WRITE_ATTRIBUTES will be allowed to set the times associated with a file to the current server time.

ACE4_WRITE_RETENTION 🔒

Operation(s) affected:
- SETATTR of retention_set, retentevt_set
Discussion: permission to modify the durations of event and non-event-based retention. Also permission to enable event and non-event-based retention. A server MAY behave such that setting ACE4_WRITE_ATTRIBUTES allows ACE4_WRITE_RETENTION.

ACE4_WRITE_RETENTION_HOLD 🧊

Operation(s) affected:
- SETATTR of retention_hold
Discussion: permission to modify the administration retention holds. A server MAY map ACE4_WRITE_ATTRIBUTES to ACE_WRITE_RETENTION_HOLD.

ACE4_DELETE 💥

Operation(s) affected:
- REMOVE
Discussion: permission to delete the file or directory. See Section 6.2.1.3.2 for information on how ACE4_DELETE and ACE4_DELETE_CHILD interact.

ACE4_READ_ACL 📜

Operation(s) affected:
- GETATTR of acl, dacl, or sacl
- NVERIFY
- VERIFY
Discussion: permission to read the ACL.

ACE4_WRITE_ACL 🖊️

Operation(s) affected:
- SETATTR of acl and mode
Discussion: permission to write the acl and mode attributes.

ACE4_WRITE_OWNER 👑

Operation(s) affected:
- SETATTR of owner and owner_group
Discussion: permission to write the owner and owner_group attributes. On UNIX systems, this is the ability to execute chown() and chgrp().

ACE4_SYNCHRONIZE 🔄

Operation(s) affected:
- NONE
Discussion: permission to use the file object as a synchronization primitive for interprocess communication. This permission is not enforced or interpreted by the NFSv4.1 server on behalf of the client.

Typically, the ACE4_SYNCHRONIZE permission only means something on local file systems, i.e., file systems not accessed via NFSv4.1. The permission bit exists because some operating environments, such as Windows, use ACE4_SYNCHRONIZE.

For example, if a client copies a file that has ACE4_SYNCHRONIZE set from a local file system to an NFSv4.1 server, and then later copies the file from the NFSv4.1 server to a local file system, then if ACE4_SYNCHRONIZE was set in the original file, the client will probably want it set in the second copy. The first copy won't have the permission set unless the NFSv4.1 server has the means to set the ACE4_SYNCHRONIZE bit. The second copy won't have the permission set unless the NFSv4.1 server has the means to retrieve the ACE4_SYNCHRONIZE bit.

Server implementations don't need to provide the granularity of control that's implied by this list of masks. For example, POSIX-based systems might not distinguish ACE4_APPEND_DATA (the ability to append to a file) from ACE4_WRITE_DATA (the ability to modify existing contents). Both masks would be tied to a single "write" permission [17]. When a server like that returns attributes to the client, it would show both ACE4_APPEND_DATA and ACE4_WRITE_DATA if and only if the write permission is enabled.

If a server receives a SETATTR request that it cannot accurately implement, it should err in the direction of more restricted access, except in the previously discussed cases of execute and read. For example, suppose a server can't distinguish overwriting data from appending new data, as described in the previous paragraph. If a client submits an ALLOW ACE where ACE4_APPEND_DATA is set but ACE4_WRITE_DATA is not (or vice versa), the server should either turn off ACE4_APPEND_DATA or reject the request with NFS4ERR_ATTRNOTSUPP.

6.2.1.3.2. ACE4_DELETE vs. ACE4_DELETE_CHILD 🧹

two access mask bits control the ability to delete a directory entry: ACE4_DELETE on the object itself (the "target") and ACE4_DELETE_CHILD on the containing directory (the "parent").

Lots of systems also use the "sticky bit" (MODE4_SVTX) on a directory to allow unlink only to a user that owns either the target or the parent. On some of those systems, the decision also depends on whether the target is writable.

Servers SHOULD allow unlink if either ACE4_DELETE is permitted on the target, or ACE4_DELETE_CHILD is permitted on the parent. (Note that this is true even if the parent or target explicitly denies one of these permissions.)

If the ACLs in question neither explicitly ALLOW nor DENY either of the above, and if MODE4_SVTX isn't set on the parent, then the server SHOULD allow the removal if and only if ACE4_ADD_FILE is permitted. When MODE4_SVTX is set, the server may also require the remover to own either the parent or the target, or may require the target to be writable.

This lets servers support something close to traditional UNIX-like semantics, with ACE4_ADD_FILE taking the place of the write bit. 🌈

6.2.1.4. ACE flag ✨

okay so here are the bitmask constants for the flag field, bestie 💖

const ACE4_FILE_INHERIT_ACE             = 0x00000001;
const ACE4_DIRECTORY_INHERIT_ACE        = 0x00000002;
const ACE4_NO_PROPAGATE_INHERIT_ACE     = 0x00000004;
const ACE4_INHERIT_ONLY_ACE             = 0x00000008;
const ACE4_SUCCESSFUL_ACCESS_ACE_FLAG   = 0x00000010;
const ACE4_FAILED_ACCESS_ACE_FLAG       = 0x00000020;
const ACE4_IDENTIFIER_GROUP             = 0x00000040;
const ACE4_INHERITED_ACE                = 0x00000080;

a server doesn't have to support any of these flags 🫶 but if it has flags that are similar to these (not exactly the same), the implementation is allowed to define a mapping between the protocol flags and its own flags. that's the flexible vibes we love to see 🦋

here's an example that's giving real-life drama 🎀 say a client tries to set an ACE with ACE4_FILE_INHERIT_ACE on, but not ACE4_DIRECTORY_INHERIT_ACE. if the server doesn't support any form of ACL inheritance, it should reject the request with NFS4ERR_ATTRNOTSUPP 💀. if the server has a single "inherit ACE" flag that covers both files and directories, it may reject the request, which means the client has to set both the file and directory inheritance flags. or it may just accept the request and quietly flip on ACE4_DIRECTORY_INHERIT_ACE for the client 🧚 no notes.

6.2.1.4.1. Discussion of Flag Bits 🌸

ACE4_FILE_INHERIT_ACE: any non-directory file in any sub-directory will get this ACE inherited ✨

ACE4_DIRECTORY_INHERIT_ACE: this one can be placed on a directory, and it says that this ACE should be added to each new directory that gets created 📁

If this flag is set in an ACE inside an ACL attribute that's meant for a non-directory file system object, the operation trying to set the ACL SHOULD fail with NFS4ERR_ATTRNOTSUPP. not it 🙅‍♀️

ACE4_NO_PROPAGATE_INHERIT_ACE: also placed on a directory. it tells the server that inheritance of this ACE should stop at newly created child directories. a little stopper, very responsible 🛑

ACE4_INHERIT_ONLY_ACE: this one can be placed on a directory but it doesn't actually apply to that directory 🙈. ALLOW and DENY ACEs with this bit set don't affect access to the directory, and AUDIT and ALARM ACEs with this bit set don't trigger log or alarm events. these ACEs only take effect once they're applied (with this bit cleared) to newly created files and directories, as specified by ACE4_FILE_INHERIT_ACE and ACE4_DIRECTORY_INHERIT_ACE. it's a whole waiting-in-the-wings situation 👑

If this flag is present on an ACE, but neither ACE4_DIRECTORY_INHERIT_ACE nor ACE4_FILE_INHERIT_ACE is present, then an operation trying to set that attribute SHOULD fail with NFS4ERR_ATTRNOTSUPP. no inheritance target, no vibes 💅

ACE4_SUCCESSFUL_ACCESS_ACE_FLAG and ACE4_FAILED_ACCESS_ACE_FLAG: these are the SUCCESS and FAILED flag bits, and they may only be set on ACE4_SYSTEM_AUDIT_ACE_TYPE (AUDIT) and ACE4_SYSTEM_ALARM_ACE_TYPE (ALARM) ACE types. okay so here's the tea ☕ while the server is processing the file's ACL, if it runs into an AUDIT or ALARM ACE that matches the principal trying to do the OPEN, the server takes note of that fact, plus whether the SUCCESS and FAILED flags are present on that ACE. once the server finishes processing the ACL, it then checks whether the operation succeeded or failed. if it succeeded and the SUCCESS flag was set on the matching AUDIT or ALARM ACE, the appropriate AUDIT or ALARM event happens. if it failed and the FAILED flag was set on the matching AUDIT or ALARM ACE, the appropriate event happens too. you can set either or both of SUCCESS and FAILED, but if you set neither, the AUDIT or ALARM ACE isn't useful at all 🫠

This processing also applies to ACCESS operations even when they return NFS4_OK. for AUDIT and ALARM purposes, an ACCESS operation counts as a "failure" if it fails to return a bit that was requested and supported. the way it's giving petty but precise 😭

ACE4_IDENTIFIER_GROUP: this indicates that the "who" refers to a GROUP as defined under UNIX or a GROUP ACCOUNT as defined under Windows 🧑‍🤝‍🧑. clients and servers MUST ignore this flag on ACEs whose who value is equal to one of the special identifiers outlined in Section 6.2.1.5. no exceptions, periodt 💜

ACE4_INHERITED_ACE: this indicates that the ACE was inherited from a parent directory. a server that supports automatic inheritance will put this flag on any ACEs inherited from the parent directory when it creates a new object, and client apps use it to do automatic inheritance 🌈. clients and servers MUST clear this bit in the acl attribute, since it may only be used in the dacl and sacl attributes. only in the dacl and sacl lanes, bestie 🎀

6.2.1.5. ACE Who 💖

okay the "who" field of an ACE is an identifier that says which principal or principals the ACE applies to ✨. it can refer to a user or a group, and the ACE4_IDENTIFIER_GROUP flag bit is what tells you which one 🦋

there are several special identifiers that need to be understood everywhere, not just inside one particular DNS domain 🌍. some of these can't be understood when an NFS client accesses the server, but they do have meaning when a local process accesses the file. being able to display and modify these permissions is allowed over NFS, even if none of the server's access methods understands the identifiers. that's giving universal translator energy 🛸

+===============+==================================================+
| Who           | Description                                      |
+===============+==================================================+
| OWNER         | The owner of the file.                           |
+---------------+--------------------------------------------------+
| GROUP         | The group associated with the file.              |
+---------------+--------------------------------------------------+
| EVERYONE      | The world, including the owner and owning group. |
+---------------+--------------------------------------------------+
| INTERACTIVE   | Accessed from an interactive terminal.           |
+---------------+--------------------------------------------------+
| NETWORK       | Accessed via the network.                        |
+---------------+--------------------------------------------------+
| DIALUP        | Accessed as a dialup user to the server.         |
+---------------+--------------------------------------------------+
| BATCH         | Accessed from a batch job.                       |
+---------------+--------------------------------------------------+
| ANONYMOUS     | Accessed without any authentication.             |
+---------------+--------------------------------------------------+
| AUTHENTICATED | Any authenticated user (opposite of ANONYMOUS).  |
+---------------+--------------------------------------------------+
| SERVICE       | Access from a system service.                    |
+---------------+--------------------------------------------------+

                             Table 7

to avoid conflicts, these special identifiers get an appended "@" and should show up in the form "xxxx@" (with no domain name after the "@"), like ANONYMOUS@ 🙈

The ACE4_IDENTIFIER_GROUP flag MUST be ignored on entries with these special identifiers. when you're encoding entries with these special identifiers, the ACE4_IDENTIFIER_GROUP flag SHOULD be set to zero. zero drama, zero group energy 🍓

6.2.1.5.1. Discussion of EVERYONE@ 🌈

this part is so important ⭐. EVERYONE@ is not the same as the UNIX "other" entity! 😱 that's because, by definition, UNIX "other" doesn't include the owner or the owning group of a file. EVERYONE@ literally means everyone, including the owner and the owning group. it's the friend who shows up to the party even when they weren't invited 💅

6.2.2. Attribute 58: dacl 🎀

the dacl attribute is like the acl attribute, but dacl only allows ALLOW and DENY ACEs 🧁. the dacl attribute supports automatic inheritance (see Section 6.4.3.2). lowkey the cleaner, more focused sibling 💜

6.2.3. Attribute 59: sacl 🦋

the sacl attribute is like the acl attribute, but sacl only allows AUDIT and ALARM ACEs 👀. the sacl attribute supports automatic inheritance too (see Section 6.4.3.2). the sacl is basically the paparazzi of the ACL family 📸

6.2.4. Attribute 33: mode 🌸

the NFSv4.1 mode attribute is based on the UNIX mode bits ✨. here are the bits that are defined:

const MODE4_SUID = 0x800;  /* set user id on execution */
const MODE4_SGID = 0x400;  /* set group id on execution */
const MODE4_SVTX = 0x200;  /* save text even after use */
const MODE4_RUSR = 0x100;  /* read permission: owner */
const MODE4_WUSR = 0x080;  /* write permission: owner */
const MODE4_XUSR = 0x040;  /* execute permission: owner */
const MODE4_RGRP = 0x020;  /* read permission: group */
const MODE4_WGRP = 0x010;  /* write permission: group */
const MODE4_XGRP = 0x008;  /* execute permission: group */
const MODE4_ROTH = 0x004;  /* read permission: other */
const MODE4_WOTH = 0x002;  /* write permission: other */
const MODE4_XOTH = 0x001;  /* execute permission: other */

okay so MODE4_RUSR, MODE4_WUSR, and MODE4_XUSR apply to the principal identified in the owner attribute 👑. MODE4_RGRP, MODE4_WGRP, and MODE4_XGRP apply to principals identified in the owner_group attribute, but only if they're not identified in the owner attribute. MODE4_ROTH, MODE4_WOTH, and MODE4_XOTH apply to any principal that doesn't match the owner attribute and doesn't have a group matching the owner_group attribute. it's a whole tiered-access situation 🧚

bits within a mode other than the ones listed above aren't defined by this protocol 🙅‍♀️. a server MUST NOT return bits other than those defined above in a GETATTR or READDIR operation. and it MUST return NFS4ERR_INVAL if bits other than those defined above are set in a SETATTR, CREATE, OPEN, VERIFY, or NVERIFY operation. no undefined bits allowed, full stop 🚫

6.2.5. Attribute 74: mode_set_masked 🫶

the mode_set_masked attribute is a write-only attribute that lets you set or reset individual bits in the mode attribute without changing the others 🎀. for example, you can modify MODE4_SUID, MODE4_SGID, and MODE4_SVTX while leaving any of the nine low-order mode bits that cover permissions totally alone. surgical precision, I'm obsessed 🔥

when those nine low-order bits are left unmodified, then neither the acl nor the dacl attribute should be automatically modified, as discussed in Section 6.4.1 🧁

the mode_set_masked attribute is made up of two words, and each one is in the form of a mode4. the first word is the value to apply to the current mode value, and the second is a mask. only bits set to one in the mask word get changed (set or reset) in the file's mode, and everything else stays the same ✨. bits in the first word that line up with zero bits in the mask get ignored, except that undefined bits still get checked for validity, and they can result in NFS4ERR_INVAL as described below 🙈

the mode_set_masked attribute is only valid in a SETATTR operation. if it's used in a CREATE or OPEN operation, the server MUST return NFS4ERR_INVAL 💀

bits that aren't defined as valid in the mode attribute aren't valid in either word of the mode_set_masked attribute either. the server MUST return NFS4ERR_INVAL if any such bits are set to one in a SETATTR. and if the mode and mode_set_masked attributes are both specified in the same SETATTR, the server MUST also return NFS4ERR_INVAL. two attributes in one SETATTR? absolutely not, the server said no 🌈

6.3. Common Methods 🧚

the requirements in this section are going to get referenced in future sections, especially Section 6.4 💖

6.3.1. Interpreting an ACL ✨

6.3.1.1. Server Considerations ✨

okay so the server uses the algorithm from Section 6.2.1 to figure out whether an ACL lets someone into an object 💅. but the ACL is not always the final boss. there are other things that can change the answer, bestie 🫶

in a file system exported as read-only, the server may deny write access even when the object's ACL says yes 🙈
server implementations MAY grant ACE4_WRITE_ACL and ACE4_READ_ACL so nobody ends up stuck in a spot where there's no valid way to ever change the ACL again 😭
all servers will let a user read a file's data when only the execute permission is granted. so if the ACL denies ACE4_READ_DATA but allows ACE4_EXECUTE, the user can still read the data of the file ✨
lots of servers have owner-override, where the owner of the object can get past accesses the ACL denies. this can be really helpful for letting people keep access to files they already have open after the permissions changed 🦋
lots of servers also have a "superuser" who has powers beyond a regular user. the superuser may be able to read or write data or metadata in ways the ACL wouldn't allow 👑
a retention attribute can also block access that the ACLs would otherwise allow (see Section 5.13) 🔒

6.3.1.2. Client Considerations 🌸

clients SHOULD NOT do their own access checks based on their own reading of the ACL 🙅‍♀️ instead, they should use the OPEN and ACCESS operations, so the server does the checking based on its own interpretation of the ACL. that way the client can just act on the server's answer, which is so much less drama 💖

clients must be aware that an object's ACL can define a certain access that the server won't actually enforce. honestly this is a situation where the client has to pull its weight in enforcing access as the ACL defines it 💪. to do that, the client MAY send the appropriate ACCESS operation before it services a request from a user or application, so it can check whether that user or application should get the access they're asking for. for examples of ACLs defining accesses the server doesn't enforce, see Section 6.3.1.1 🧚

6.3.2. Computing a Mode Attribute from an ACL 🎀

you can use the following method to work out the MODE4_R*, MODE4_W*, and MODE4_X* bits of a mode attribute from an ACL. it's giving math, but it's easy math 🍓

first, for each of the special identifiers OWNER@, GROUP@, and EVERYONE@, go through the ACL in order and only look at the ALLOW and DENY ACEs for EVERYONE@ and for the identifier you're currently working on. the result is an NFSv4 ACL mask that shows exactly which bits are permitted to that identifier ✨

then turn the mask you calculated for OWNER@, GROUP@, and EVERYONE@ into mode bits for the user, group, and other, like this:

set the read bit (MODE4_RUSR, MODE4_RGRP, or MODE4_ROTH) if and only if ACE4_READ_DATA is set in the matching mask 📖
set the write bit (MODE4_WUSR, MODE4_WGRP, or MODE4_WOTH) if and only if both ACE4_WRITE_DATA and ACE4_APPEND_DATA are set in the matching mask ✍️
set the execute bit (MODE4_XUSR, MODE4_XGRP, or MODE4_XOTH) if and only if ACE4_EXECUTE is set in the matching mask 🏃‍♀️

6.3.2.1. Discussion 💬

some server implementations also add the bits permitted to named users and groups into the group bits (MODE4_RGRP, MODE4_WGRP, and MODE4_XGRP) 🤔

implementations are discouraged from doing this 🙅‍♀️ because it confuses users. they see members of a file's group get denied access even though the mode bits seem to allow it. (DENY ACEs can cause that same weirdness too, but DENY ACEs are expected to show up less often 🌈)

the same confusion shows up when setting the mode doesn't actually control permissions for the owner, group, and other users. that's the reason behind some of the requirements that come next 🫶

6.4. Requirements 📜

a server that supports both mode and ACL has to be careful to sync the MODE4_*USR, MODE4_*GRP, and MODE4_*OTH bits with the ACEs whose who fields are OWNER@, GROUP@, and EVERYONE@. that way, a client can see the same permissions whether it asks for the owner, owner_group, and mode attributes or just the ACL 🦋

this section leans heavily on the methods in Section 6.3.2, and lots of requirements point back to it. but heads up: those methods have behaviors specified with "SHOULD". that's on purpose, so existing implementations that compute the mode from the withdrawn POSIX ACL draft (1003.1e draft 17), instead of from real permissions on owner, group, and other, don't get broken 🥹

6.4.1. Setting the Mode and/or ACL Attributes 🛠️

if a server supports the sacl or dacl attribute as well as the acl attribute, the server MUST fail a request that sets the acl attribute at the same time as a dacl or sacl attribute. the error to return is NFS4ERR_ATTRNOTSUPP ❌

6.4.1.1. Setting Mode and not ACL 🎀

when any of the nine low-order mode bits might change, either because the mode attribute was set or because the mode_set_masked attribute was set and its mask included one or more of those nine bits, and no ACL attribute was explicitly set, the acl and dacl attributes must be updated to match the new values of those bits. this must happen even if the low-order bits end up the same after the mode is set as they were before 💅

note that any AUDIT or ALARM ACEs (so any ACEs in the sacl attribute) are not affected by mode changes 🙌

whenever the permission bits are subject to change, the acl and dacl attributes MUST be modified so that the mode computed with the method in Section 6.3.2 gives the low-order nine bits (MODE4_R*, MODE4_W*, MODE4_X*) of the mode attribute as changed by the attribute update. the ACL attributes SHOULD also be modified so that:

if MODE4_RGRP is not set, entities explicitly listed in the ACL other than OWNER@ and EVERYONE@ SHOULD NOT be granted ACE4_READ_DATA 🚫
if MODE4_WGRP is not set, entities explicitly listed in the ACL other than OWNER@ and EVERYONE@ SHOULD NOT be granted ACE4_WRITE_DATA or ACE4_APPEND_DATA 🚫
if MODE4_XGRP is not set, entities explicitly listed in the ACL other than OWNER@ and EVERYONE@ SHOULD NOT be granted ACE4_EXECUTE 🚫

access mask bits other than the ones listed above, when they show up in ALLOW ACEs, MAY also be disabled 🧹

heads up that ACEs with the flag ACE4_INHERIT_ONLY_ACE set don't affect the permissions of the ACL itself, and neither do ACEs of type AUDIT and ALARM. so it's a good idea to leave those ACEs alone when you change the ACL attributes 🥰

also, the requirement can be met by throwing out the acl and dacl and replacing them with an ACL that represents the mode and only the mode. that's allowed, but it's better for a server to keep as much of the ACL as it can without breaking the requirements above ✨. throwing the ACL away means a file created with a mode attribute basically can't inherit an ACL (see Section 6.4.3) 😢

6.4.1.2. Setting ACL and Not Mode 🌷

when you set the acl or dacl and don't set the mode or mode_set_masked attributes, the mode's permission bits have to be worked out from the ACL. in this case the ACL attribute SHOULD be set as given. the nine low-order bits of the mode attribute (MODE4_R*, MODE4_W*, MODE4_X*) MUST be changed to match the result of the method in Section 6.3.2. the three high-order bits of the mode (MODE4_SUID, MODE4_SGID, MODE4_SVTX) SHOULD stay unchanged 🧁

6.4.1.3. Setting Both ACL and Mode 💕

when you set both the mode (using either the mode attribute or the mode_set_masked attribute) and the acl or dacl attributes in the same operation, the attributes MUST be applied in this order: mode (or mode_set_masked), then ACL. the mode-related attribute is set as given, and then the ACL attribute is set as given, which might change the final mode, as described above in Section 6.4.1.2 🔁

6.4.2. Retrieving the Mode and/or ACL Attributes 🔍

this section only applies to servers that support both the mode and ACL attributes 🫶

some server implementations have a concept of "objects without ACLs." that means all permissions are granted and denied according to the mode attribute, and no ACL attribute is stored for that object. if an ACL attribute is requested from a server like that, the server SHOULD return an ACL that doesn't conflict with the mode. in other words, the ACL returned SHOULD represent the nine low-order bits of the mode attribute (MODE4_R*, MODE4_W*, MODE4_X*) as described in Section 6.3.2 🎀

for other server implementations, the ACL attribute is always present for every object. those servers SHOULD store at least the three high-order bits of the mode attribute (MODE4_SUID, MODE4_SGID, MODE4_SVTX). the server SHOULD return a mode attribute if one is requested, and the low-order nine bits of the mode (MODE4_R*, MODE4_W*, MODE4_X*) MUST match the result of applying the method in Section 6.3.2 to the ACL attribute 🦋

6.4.3. Creating New Objects ✨

okay so bestie, this is where the server gets to decide what a brand new object looks like the moment it's born 🍼💅 if the server supports any ACL attributes, it MAY use the ACL attributes on the parent directory to cook up an initial ACL for the new object. we're calling that the inherited ACL in this section 🦋 and the act of copying over ACEs from the parent directory's ACL into it? that's inheriting an ACE 🌸

implementors should standardize what CREATE and OPEN are supposed to do depending on whether the mode and ACL attributes show up (or don't) 🎀 it's giving very "choose your own adventure" energy, so let's go through all four paths 👑

if just the mode is given in the call: inheritance SHOULD happen 💖 but the mode MUST be applied to the inherited ACL as described in Section 6.4.1.1, which means the ACL gets modified 🫶
if just the ACL is given in the call: inheritance SHOULD NOT happen 🙈 the ACL from the CREATE or OPEN gets set without any changes, and then the mode gets modified like in Section 6.4.1.2 ✨
if both mode and ACL are given in the call: inheritance SHOULD NOT happen 🚫 and both attributes get set the way Section 6.4.1.3 describes 💜
if neither mode nor ACL is given in the call: okay this one's a whole thing 😭 if the object is created with no initial attributes at all (like an OPEN with an opentype4 of OPEN4_CREATE and a createmode4 of EXCLUSIVE4), inheritance SHOULD NOT happen 🙅‍♀️ and the server SHOULD set permissions to deny all access to the new object. the appropriate client is expected to set the good stuff in a later SETATTR, and the server SHOULD let that SETATTR succeed no matter what permissions the object was born with 🫠 for example, an empty ACL denies everything, but the server should still let the owner's SETATTR go through even though WRITE_ACL is implicitly denied 💀

ngl, EXCLUSIVE4_1 is the better createmode4 choice here, since it DOES permit initial attributes 🧚

in every other case, inheritance SHOULD happen and nothing about the ACL gets modified 🌈 the mode attribute, if supported, MUST be as computed in Section 6.3.2, with the MODE4_SUID, MODE4_SGID, and MODE4_SVTX bits cleared 🍓 if there are no inheritable ACEs on the parent directory, the rules for creating acl, dacl, or sacl attributes are implementation defined 🤷‍♀️ and if either the dacl or sacl attribute is supported, the ACL4_DEFAULTED flag SHOULD be set on the newly created attributes 🔥

6.4.3.1. The Inherited ACL ✨

if the object being created is NOT a directory, the inherited ACL SHOULD NOT inherit ACEs from the parent directory ACL unless the ACE4_FILE_INHERIT_FLAG is set 🙅‍♀️💅

if the object being created IS a directory, the inherited ACL should inherit all the inheritable ACEs from the parent directory, meaning the ones with ACE4_FILE_INHERIT_ACE or ACE4_DIRECTORY_INHERIT_ACE set 🌸 now if the inheritable ACE has ACE4_FILE_INHERIT_ACE set but ACE4_DIRECTORY_INHERIT_ACE is clear, the inherited ACE on the new directory MUST have ACE4_INHERIT_ONLY_ACE set 🎀 that stops it from messing with the directory itself, since it was meant for non-directories 🙈

when a new directory gets created, the server MAY split any inherited ACE that is both inheritable and effective (so, one with neither ACE4_INHERIT_ONLY_ACE nor ACE4_NO_PROPAGATE_INHERIT_ACE set) into two ACEs 🦋 one has no inheritance flags, and the other has ACE4_INHERIT_ONLY_ACE set 💖 (for a dacl or sacl attribute, both of those ACEs SHOULD also have ACE4_INHERITED_ACE set ✨) this makes it way easier to tweak the directory's effective permissions without touching the ACE that gets passed down to the new directory's children 👑

6.4.3.2. Automatic Inheritance ✨

the acl attribute is just an array of ACEs, but the sacl (Section 6.2.3) and dacl (Section 6.2.2) attributes also come with an extra flag field 🧚

struct nfsacl41 {
        aclflag4        na41_flag;
        nfsace4         na41_aces<>;
};

the flag field applies to the whole sacl or dacl 💅 three flag values are defined 🌈

const ACL4_AUTO_INHERIT         = 0x00000001;
const ACL4_PROTECTED            = 0x00000002;
const ACL4_DEFAULTED            = 0x00000004;

and all other bits must be cleared 🫶 the ACE4_INHERITED_ACE flag MAY be set in the ACEs of the sacl or dacl (whereas it MUST always be cleared in the acl) 🍓

together these features let a server support automatic inheritance, which we're about to break down 🔥

normally, inheritable ACEs only get inherited by child objects at the moment the children are created 😭 later changes to inheritable ACEs do NOT update the inherited ACEs on descendants 💀

BUT the dacl and sacl give you an OPTIONAL mechanism that lets a client application push changes to inheritable ACEs down an entire directory hierarchy 🦋✨

a server that supports this does inheritance at creation time in the normal way, and SHOULD set ACE4_INHERITED_ACE on any inherited ACEs as they get added to the new object 🌸

then a client application, like an ACL editor, can propagate changes by recursively walking the directory's descendants and changing each ACL it finds 🎀 it removes any ACEs with ACE4_INHERITED_ACE and replaces them with the new inheritable ACEs (also with ACE4_INHERITED_ACE set) 💜 it uses the existing ACE inheritance flags in the obvious way to decide which ACEs get passed along 👑 (fyi it might run into more inheritable ACEs while going down the hierarchy, and those need to be counted too when propagating to further descendants 🙈)

the reach of this propagation can be limited in two ways 🫶 first, automatic inheritance is NOT performed from any directory ACL that has ACL4_AUTO_INHERIT cleared 🚫 second, automatic inheritance stops wherever an ACL has ACL4_PROTECTED set 🛑 that blocks changes to that ACL, and also (if the ACL is on a directory) to the ACL on any of the object's descendants 🔒

this propagation runs independently for the sacl and the dacl 🌈 so ACL4_AUTO_INHERIT and ACL4_PROTECTED can be set separately for each one 💖 and propagation of one kind can keep going down a hierarchy even after propagation of the other has stopped 🧚

new objects SHOULD be created with a dacl and a sacl that both have ACL4_PROTECTED cleared 🎀 and the ACL4_AUTO_INHERIT flag set to the same value as on the parent object's sacl or dacl, respectively ✨

both the dacl and sacl attributes are RECOMMENDED 💅 and a server MAY support one without supporting the other 🌸

a server that supports both the old acl attribute AND one or both of the new dacl or sacl attributes MUST keep all three consistent with each other 🔥 so the ACEs reported in the acl attribute SHOULD be the union of the ACEs reported in the dacl and sacl, except ACE4_INHERITED_ACE MUST be cleared from the ACEs in the acl 🫶 and of course, a client that only queries the acl can't figure out the dacl or sacl flag fields 🙈

when a client does a SETATTR on the acl attribute, the server SHOULD set ACL4_PROTECTED to true on both the sacl and the dacl 🛑 using acl instead of dacl or sacl tells the server the client may not understand automatic inheritance, so it can't be trusted to set an ACL where automatic inheritance would make sense 😭

when a client app queries an ACL, changes it, and sets it again, it SHOULD leave any ACEs marked ACE4_INHERITED_ACE unchanged, in their original order, at the end of the ACL 💖 if the app can't do that, it SHOULD set ACL4_PROTECTED 🧚 servers don't enforce this, but breaking the rule may lead to unexpected results when apps do automatic inheritance 💀

if a server also supports the mode attribute, it SHOULD set the mode in a way that leaves inherited ACEs unchanged, in their original order, at the end of the ACL 🌈 if it can't, it SHOULD set ACL4_PROTECTED on the file's dacl 🎀

last one, and it's a big one 👑 if the request that creates a new file or directory doesn't set permissions AND there are no ACEs to inherit from the parent directory, then the server's choice of ACL is implementation-dependent 🤷‍♀️ in that case the server SHOULD set ACL4_DEFAULTED on the ACL it picks for the new object 💜 an app doing automatic inheritance reads ACL4_DEFAULTED as a sign that the ACL should be completely replaced by one generated using the automatic inheritance rules 🦋

7. Single-Server Namespace ✨

this section is about the NFSv4 single-server namespace 🌸 single-server namespaces can be shown straight to clients, or used as a base to build bigger multi-server namespaces (like site-wide or organization-wide ones) that get presented to clients, as described in Section 11 🧚

7.1. Server Exports 💖

on a UNIX server, the namespace describes all the files reachable by pathnames under the root directory or "/" 🌈 on a Windows server, the namespace is all the files on disks named by mapped disk letters 💅 NFS server admins rarely share the entire server's file system with NFS clients 🙈 more often, portions of the namespace get made available through an "export" feature 🎀 in previous NFS versions, the root filehandle for each export was obtained through the MOUNT protocol: the client sent a string identifying the export name within the namespace, and the server sent back the root filehandle for that export 🦋 the MOUNT protocol also had an EXPORTS procedure that listed the server's exports 👑

7.2. Browsing Exports 🗂️✨

okay so NFSv4.1 gives you a root filehandle 🌸, and clients can use it to grab filehandles for a server's exports with a chain of LOOKUP operations inside a COMPOUND. it's basically how you walk a path 🧚 the everyday version of this is a file "Open" dialog where you click through folders one at a time 📁💅. the client has to be able to hop from one export to another using single-component, progressive LOOKUPs. no skipping, no teleporting, just step by step 🦋

this style of browsing is not really it in NFSv3 🙈 in NFSv3, the client expects every LOOKUP to stay inside one server file system. the device attribute, for example, will not change. so a client can't follow a namespace path that crosses from one export into another 💀

with NFSv3, an automounter on the client can take a snapshot of the server's namespace with the EXPORTS procedure of the MOUNT protocol 📸. if it understands the server's pathname syntax, it can build a copy of that namespace on the client. the parts that aren't exported get filled in with directories, kind of like an NFSv4.1 "pseudo file system" (see Section 7.3) that lets the user browse from one mounted file system to the next 🎀. the catch is that this copy is static 😭. if the server admin adds a new export, the client has no idea it exists.

7.3. Server Pseudo File System 🧚‍♀️🌈

NFSv4.1 servers dodge that namespace inconsistency by showing every export for a server inside one single namespace 🫶. an NFSv4.1 client uses LOOKUP and READDIR to move seamlessly from one export to another. it's giving smooth ✨

sometimes parts of the server namespace aren't exported, but clients still need a way through them to reach the real exported file systems 🗺️. one technique servers can use is a "pseudo file system" that bridges the unexported part and only shows exported directories. a pseudo file system gets its own unique fsid and acts like a normal, read-only file system 🔒

depending on how the server's namespace is built, there can be more than one pseudo file system 👑 for example:

/a              pseudo file system
/a/b            real file system
/a/b/c          pseudo file system
/a/b/c/d        real file system

each pseudo file system counts as its own separate entity, so each one MUST have its own fsid, and that fsid has to be unique among all the fsids for that server 💖

7.4. Multiple Roots 💅

some operating environments are described as having "multiple roots" 🌷. in those setups, individual file systems usually show up as disk or volume names. NFSv4 servers on these platforms can build a pseudo file system above those root names, so disk letters or volume names just become directory names in the pseudo root 🍓

7.5. Filehandle Volatility 🦋💭

the server's pseudo file system is a logical view of the file system(s) available from the server 🪞. so it's most likely built dynamically when the server first starts up 🚀. it's expected that the pseudo file system might not have an on-disk counterpart to build persistent filehandles from. even though it's preferable for the server to give persistent filehandles for the pseudo file system, the NFS client should expect pseudo file system filehandles to be volatile 😤. you can check this with the fh_expire_type attribute for the filehandles you're looking at 🔍. if the filehandles are volatile, the NFS client must be prepared to recover a filehandle value (e.g., with a series of LOOKUP operations) when it gets an NFS4ERR_FHEXPIRED error 🛠️

because servers will probably implement pseudo file systems with volatile filehandles, clients need to be ready for that 🥹. they shouldn't assume every filehandle will stick around forever 🙅‍♀️

7.6. Exported Root 👑

if the server's root file system is exported, you might think a pseudo file system isn't needed 🤔. that's not necessarily true 🙈. say the server has these file systems:

/       fs1  (exported)
/a      fs2  (not exported)
/a/b    fs3  (exported)

because fs2 isn't exported, fs3 can't be reached with simple LOOKUPs 🚧. the server has to bridge that gap with a pseudo file system 🌉

7.7. Mount Point Crossing 🪄

the server's file system environment can be set up so that one file system has a directory that is 'covered' or mounted upon by a second file system 🎭. for example:

/a/b            (file system 1)
/a/b/c/d        (file system 2)

the pseudo file system for this server might be built to look like this:

/               (place holder/not exported)
/a/b            (file system 1)
/a/b/c/d        (file system 2)

it's the server's job to present a complete pseudo file system to the client 🫡. if the client sends a LOOKUP for the path /a/b/c/d, the server answers with the filehandle for the root of file system /a/b/c/d. in earlier NFS protocol versions, the server would have answered with the filehandle for the directory /a/b/c/d inside file system /a/b instead. big difference, bestie 😳

the NFS client can tell it has crossed a server mount point when the value of the "fsid" attribute changes 🔁✨

7.8. Security Policy and Namespace Presentation 🔐

NFSv4 clients can change the security mechanisms they use, after figuring out what's allowed with SECINFO and SECINFO_NONAME 🔄. because of that, the server SHOULD NOT show a different view of the namespace based on which security mechanism a client is using 🙅‍♀️. instead, it should show one consistent view, and return NFS4ERR_WRONGSEC if someone tries to access data with the wrong security mechanism 🚫

sometimes security means you need to hide that a file system exists at all, not just the data inside it 🕵️‍♀️. in that case, the server can apply the security policy of a shared resource in the server's namespace to the resource's ancestors. for example:

/                           (place holder/not exported)
/a/b                        (file system 1)
/a/b/MySecretProject        (file system 2)

the /a/b/MySecretProject directory is a real file system and it's the shared resource 📂. say the security policy for /a/b/MySecretProject is Kerberos with integrity, and you want to limit who even knows that file system exists 🤫. then the server should apply that same security policy to /a/b. this way, knowing a file system exists can be locked down when you need it to be 🔒💜

here's the catch 🙃. if the server's resources use multiple, disjoint security mechanisms, applying that kind of policy would mean the higher-level file system can't be accessed with any security flavor at all 💀. so that kind of setup doesn't work with hiding the existence (as opposed to the contents) from clients that use multiple disjoint sets of security flavors.

other times, a good policy is for an object's security to include the union of all security mechanisms of its direct descendants 🌳. a common, convenient choice, unless strong security requirements say otherwise, is to make the entire the pseudo file system accessible by all of the valid security mechanisms 🌈

if there are concerns about data security on the network, clients should use strong security mechanisms to access the pseudo file system, so they can prevent man-in-the-middle attacks 🛡️😭

8. State Management 🧠💫

locking has to be built into the NFS protocol, and that makes the protocol stateful 🫠. with things like share reservations, file and directory delegations, recallable layouts, and mandatory byte-range locking, the protocol depends way more on properly managing state than the old NFS and NLM (Network Lock Manager) [54] combo did 📈. these features bring in expanded locking, which gives some level of inter-client exclusion, but the state also does stuff a stateless model just can't do 🤯. three things make that state manageable:

clear division between client and server 🤝
ability to reliably detect inconsistency in state between client and server 🔎
simple and robust recovery mechanisms 🛟

in this model, the server owns the state 👑. the client asks for changes to locks, and the server replies with the changes it made 📬. changes to locking state that the client didn't start are rare 🌙. when they do happen, the client gets prompt notification and can update its view of locking state to match the server's changes 💌

individual pieces of state that the server creates and hands to the client on request are represented by 128-bit stateids 🧩. a stateid can represent an open file, a set of byte-range locks held by a particular owner, or a recallable delegation to access a file in particular ways or at a particular location 🎯

in every case, you go from the most general info, which represents the client as a whole, down to the lightweight stateid that's used for most client and server locking interactions 🪜. the details vary by object type, but it always starts with a client ID 🆔✨

8.1. Client and Session ID 🪪

a client must establish a client ID (see Section 2.4) and then one or more sessionids (see Section 2.10) before it does anything to open, byte-range lock, delegate, or get a layout for a file object 📝. each session ID is tied to one specific client ID, so it works as a shorthand way to refer to an NFSv4.1 client 💅

for some kinds of locking, the client represents a bunch of internal locking entities called "owners", which normally line up with processes inside the client 🧑‍💻. other locking objects, like delegations and layouts, don't get those middle-layer entities. those objects are passed directly between the server and one single client 🫶

8.2. Stateid Definition ✨

okay so bestie, whenever the server grants a lock of any type (opens, byte-range locks, delegations, and layouts), it hands back a unique stateid 🦋 that stands for a set of locks (often just one lock) on the same file, of the same type, with the same ownership vibes. So opens of the same file by different open-owners each get their own identifying stateid 💖 and each set of byte-range locks on a file owned by a specific lock-owner gets one too. Delegations and layouts have stateids you can reference as well 🎀. The stateid is basically a shorthand nickname for a lock or a group of locks. Give the server a stateid and it can figure out the associated state-owner (or the open-owner/lock-owner pair) and the filehandle 🧚. When you use stateids, the current filehandle has to be the one tied to that stateid 💅.

All the stateids linked to one client ID share a common lease 🌸. That lease is the server's promise to keep those stateids and the objects they cover alive. Check out Section 8.3 for the lease tea ☕.

The server can assign stateids independently for different clients 🫶. That means the same bit pattern can point to a totally different set of locks for another client, so no cap, always read the stateid in context. The stateid is always interpreted with respect to the client ID tied to the current session 🔥. Stateids work across every session linked to that client ID, so a client can grab a stateid from one session and use it on another session with the same client ID 👑.

8.2.1. Stateid Types 🌈

okay so with the exception of special stateids (see Section 8.2.3), each stateid stands for locking objects from one of the types defined by the NFSv4.1 protocol 💜. Quick heads up: whenever we say "guarantee" here, just know there are situations like a client restart or lock revocation that can void that guarantee 💀.

Stateids may represent opens of files.

Each stateid here is the OPEN state for a given client ID/open-owner/filehandle triple 🍓. These stateids can change (and their seqid gets incremented) when OPENs cause an upgrade or when OPEN_DOWNGRADE operations happen ✨.

Stateids may represent sets of byte-range locks.

All the locks a particular owner holds on a particular file, gotten under a particular open file, share one stateid 🎀. The seqid goes up whenever LOCK and LOCKU operations change that set of locks 💖.

Stateids may represent file delegations, which are recallable guarantees by the server to the client that other clients will not reference or modify a particular file, until the delegation is returned.

In NFSv4.1, file delegations can be obtained on both regular and non-regular files 🦋. A stateid represents a single delegation held by a client for a particular filehandle 🫶.

Stateids may represent directory delegations, which are recallable guarantees by the server to the client that other clients will not modify the directory, until the delegation is returned.

A stateid represents a single delegation held by a client for a particular directory filehandle 🌸.

Stateids may represent layouts, which are recallable guarantees by the server to the client that particular files may be accessed via an alternate data access protocol at specific locations.

Access like this is limited to particular sets of byte-ranges, and it can keep going until those byte-ranges shrink or the layout gets returned 🧚. A stateid represents the set of all layouts a particular client holds for a particular filehandle with a given layout type 👑. The seqid updates as the layouts for that set of byte-ranges change, through layout stateid changing operations like LAYOUTGET and LAYOUTRETURN 🔥.

8.2.2. Stateid Structure 💅

okay so stateids are split into two fields: a 96-bit "other" field that identifies the specific set of locks, and a 32-bit "seqid" sequence value 🍓. Except for special stateids (see Section 8.2.3), a particular value of the "other" field marks a set of locks of the same type (byte-range locks, opens, delegations, or layouts), for one specific file or directory, sharing the same ownership vibes 💜. The seqid picks out one specific instance of that set and gets incremented whenever the set changes 🎀. That can happen by adding or deleting locks, changing the byte-range they cover, or upgrading or downgrading the type of one or more locks ✨.

When a set of locks is first created, the server returns a stateid with a seqid value of one 🌈. On later operations that change the set, the server is required to increment the "seqid" field by one whenever it returns a stateid for the same state-owner/file/type combination AND there's some actual change in the set of locks 🦋. In that case the server returns a stateid whose "other" field matches what was used before for that combination, but with an incremented "seqid" 💖. This pattern keeps going until the seqid passes NFS4_UINT32_MAX, and then one (not zero) is the next seqid value 🫶.

The whole point of incrementing the seqid is to let the server tell the client what order its locking operations were processed in 🌸, and to make it possible for the client to send requests that only go through if the set of locks hasn't changed since the stateid was returned 🧚.

Except for layout stateids (Section 12.5.3), when a client sends a stateid to the server, it gets two choices for the seqid 👑. It can set the seqid to zero, which tells the server "give me the most up-to-date seqid for that stateid's 'other' field" 🔥. That's the usual move for a stateid sent with a READ or WRITE operation 💅. Or it can set a non-zero value, and then the server checks whether that seqid is the correct one 🍓. In that case the server is required to return NFS4ERR_OLD_STATEID if the seqid is lower than the most current value, and NFS4ERR_BAD_STATEID if the seqid is greater than the most current value 💀. That's the common choice for stateids sent with a CLOSE or OPEN_DOWNGRADE 💜. Here's why it matters 🙈: OPENs can be sent in parallel for the same owner, so a client might close a file without knowing the server had done an OPEN upgrade and changed the lock in question 😭. If CLOSE got sent with a zero seqid, the OPEN upgrade would get canceled before the client even heard that an upgrade happened 💀.

When the server sends a stateid to the client as part of a callback operation, that stateid isn't checked for a current seqid, and NFS4ERR_OLD_STATEID doesn't come into play 🎀. The reason is that the client isn't in a position to know the most up-to-date seqid, so it can't verify it ✨. Unless specifically noted, the seqid for a stateid the server sends as part of a callback is required to be zero, and if it isn't, NFS4ERR_BAD_STATEID gets returned 🌈.

When comparing seqids, both the client (figuring out the order of operations) and the server (deciding whether to return NFS4ERR_OLD_STATEID) have to account for the seqid wrapping around past NFS4_UINT32_MAX 🦋. To do that, the total count of slots for all sessions associated with the current client is used 🫶. When one seqid value is less than this total slot count and another seqid value is greater than NFS4_UINT32_MAX minus the total slot count, the former is treated as lower than the latter, even though it's numerically bigger 🤯💖.

8.2.3. Special Stateids 🧚

Stateid values whose "other" field is all zeros or all ones are reserved 💜. The server can't assign them, but they have special meanings the protocol defines 👑. The meaning depends on whether "other" is all zeros or all ones, plus the specific value of "seqid" 🔥.

The following combinations of "other" and "seqid" are defined in NFSv4.1 🍓:

When "other" and "seqid" are both zero, the stateid is treated as a special anonymous stateid, which can be used in READ, WRITE, and SETATTR requests to say there's no OPEN state tied to the request 💅. When an anonymous stateid is used and an existing open denies the form of access requested, access gets denied to the request 🙅‍♀️. This stateid MUST NOT be used on operations to data servers (Section 13.6) 💀.
When "other" and "seqid" are both all ones, the stateid is a special READ bypass stateid 🌈. When it's used in WRITE or SETATTR, it's treated like the anonymous value 🫶. When it's used in READ, the server MAY grant access, even if access would normally be denied to READ operations ✨. This stateid MUST NOT be used on operations to data servers 🎀.
When "other" is zero and "seqid" is one, the stateid represents the current stateid, which is whatever the last stateid returned by an operation within the COMPOUND was 🦋. For an OPEN, the stateid used is the one returned for the open file, not the delegation 💖. The stateid passed to the operation in place of the special value has its "seqid" set to zero, except when the current stateid is used by the operation CLOSE or OPEN_DOWNGRADE 🌸. If no operation in the COMPOUND has returned a stateid value, the server MUST return the error NFS4ERR_BAD_STATEID 😭. As illustrated in Figure 6, if the current stateid is a special stateid and the stateid in an operation's arguments has "other" set to zero and "seqid" set to one, the server MUST return the error NFS4ERR_BAD_STATEID 🚨.
When "other" is zero and "seqid" is NFS4_UINT32_MAX, the stateid represents a reserved stateid value that's defined to be invalid 🙈. When this stateid is used, the server MUST return the error NFS4ERR_BAD_STATEID 💀.

If a stateid value is used that has all zeros or all ones in the "other" field but doesn't match one of the cases above, the server MUST return the error NFS4ERR_BAD_STATEID. No exceptions, bestie 🚫💅.

Special stateids are different from other stateids 🧚: they aren't tied to individual client IDs or filehandles, so they can be used with all valid client IDs and filehandles 👑. But when a special stateid stands for the current stateid, the value substituted in is tied to a particular client ID and filehandle 🔥. So if it's used where the current filehandle doesn't match the one tied to the current stateid, the operation it's passed to returns NFS4ERR_BAD_STATEID 💔.

8.2.4. Stateid Lifetime and Validation ✨

Okay bestie, stateids must stick around until either a client restart or a server restart happens, or until the client gives back all the locks tied to the stateid with something like CLOSE or DELEGRETURN 🔐. If the locks get yanked by revocation, the stateid still counts as a valid way to point at that revoked state, as long as the client ID is valid. It stays valid until the client frees it with FREE_STATEID 🧹. Byte-range lock stateids are the exception 🙃. They stay valid even when a LOCKU frees every remaining lock, so long as the open file they belong to is still open, unless the client frees them with FREE_STATEID.

Heads up, locks can go invalid without the client ever asking to return them 😬. That includes lease expiration and a bunch of revocation types inside the lease period. Even then, the stateid stays valid, and the client can still use it to figure out what happened to those lost locks 🕵️‍♀️.

An "other" value must never be reused for a different purpose (different filehandle, owner, or type of locks) within a single client ID 🚫. A server can keep an "other" value for the same purpose even past the point where it could otherwise free it, but if it does, it must keep "seqid" continuity with the previous values 🧵.

One way a server can meet the need to spot invalid and out-of-date stateids is to split the "other" field into two pieces 🧩:

an index into a table of locking-state structures.
a generation number that gets incremented on each allocation of a table entry for a particular use.

Then each table entry stores 💾:

the client ID the stateid is associated with.
the current generation number for the (at most one) valid stateid sharing this index value.
the filehandle of the file the locks are taken on.
an indication of the type of stateid (open, byte-range lock, file delegation, directory delegation, layout).
the last "seqid" value returned for the current "other" value.
an indication of the current status of the locks for this stateid, especially whether they've been revoked and, if so, why.

With all that info, a server can validate an incoming stateid and send back the right error when necessary 🎯. Special and non-special stateids get handled separately. (Check Section 8.2.3 for the tea on special stateids 💅.)

Stateids are implicitly qualified by the current client ID, which comes from the client ID associated with the current session. But here's the twist: the semantics of the session stop stateids from a previous client or server instance from being analyzed by this procedure 🙅‍♀️.

If a server restart has resulted in an invalid client ID or an invalid session ID, SEQUENCE will return an error, and the operation that takes a stateid as an argument will never be processed 💀.

If a server restart happened where there's a persistent session and all the leased state is gone, the session is still valid but gets marked as dead 😭. Any operation not satisfied by the reply cache will receive the error NFS4ERR_DEADSESSION, so it won't be processed the way described below.

When a stateid is being tested and the "other" field is all zeros or all ones, the server checks whether "other" and "seqid" match a defined combination for a special stateid, and the results go like this 🔍:

If the "other" and "seqid" fields don't match a defined combination associated with a special stateid, the error NFS4ERR_BAD_STATEID is returned.
If the special stateid designates the current stateid and there is a current stateid, then the current stateid gets swapped in for the special stateid, and the checks for non-special stateids get performed.
If the combination is valid in general but isn't appropriate to the context where the stateid is used (like an all-zero stateid when an OPEN stateid is required in a LOCK operation), the error NFS4ERR_BAD_STATEID is also returned.
Otherwise, the check is completed and the special stateid is accepted as valid ✅.

When a stateid is being tested and the "other" field is neither all zeros nor all ones, the following procedure could be used to validate an incoming stateid and return an appropriate error when necessary 🛠️. This assumes the "other" field gets divided into a table index and an entry generation.

If the table index is outside the range of the associated table, return NFS4ERR_BAD_STATEID.
If the selected table entry has a different generation than the one specified in the incoming stateid, return NFS4ERR_BAD_STATEID.
If the selected table entry doesn't match the current filehandle, return NFS4ERR_BAD_STATEID.
If the client ID in the table entry doesn't match the client ID associated with the current session, return NFS4ERR_BAD_STATEID.
If the stateid represents revoked state, return NFS4ERR_EXPIRED, NFS4ERR_ADMIN_REVOKED, or NFS4ERR_DELEG_REVOKED, as appropriate.
If the stateid type isn't valid for the context in which the stateid appears, return NFS4ERR_BAD_STATEID. A stateid might be valid in general (the kind TEST_STATEID would report) but still be invalid for a particular operation. For example, that happens when a stateid that doesn't represent byte-range locks gets passed to the non-from_open case of LOCK or to LOCKU, or when a stateid that doesn't represent an open gets passed to CLOSE or OPEN_DOWNGRADE. In those cases, the server MUST return NFS4ERR_BAD_STATEID.
If the "seqid" field isn't zero and it's greater than the current sequence value for the current "other" field, return NFS4ERR_BAD_STATEID.
If the "seqid" field isn't zero and it's less than the current sequence value for the current "other" field, return NFS4ERR_OLD_STATEID.
Otherwise, the stateid is valid, and the table entry should hold any extra info about the stateid type and the stuff tied to that type 🗂️. That includes the associated set of locks (open-owner and lock-owner info) and the specifics on those locks, like open modes and byte-ranges.

8.2.5. Stateid Use for I/O Operations 📦

Clients doing I/O operations need to pick an appropriate stateid based on the locks they hold (opens and delegations included) and the different types of state-owners sending the I/O requests 🧐. SETATTR operations that change the file size get treated like I/O operations in this regard.

The rules below get applied in order of decreasing priority to choose the right stateid. While following them, the client only considers locks it has actually been notified about through an appropriate operation response or callback 📬. Note that the rules are a little different for I/O to data servers when file layouts are being used (see Section 13.9.1).

If the client holds a delegation for the file in question, the delegation stateid SHOULD be used.
Otherwise, if the entity corresponding to the lock-owner (like a process) sending the I/O has a byte-range lock stateid for the associated open file, then the byte-range lock stateid for that lock-owner and open file SHOULD be used.
If there is no byte-range lock stateid, then the OPEN stateid for the open file in question SHOULD be used.
Finally, if none of the above apply, then a special stateid SHOULD be used.

Ignoring these rules may leave the server without the info it needs to process the request properly 🙈. For example, when mandatory byte-range locks are in effect, if the stateid doesn't indicate the proper lock-owner through a lock stateid, a request might get avoidably rejected.

The server however should not try to enforce these ordering rules, and should use whatever information is available to properly process I/O requests 🤷‍♀️. In particular, when a client has a delegation for a given file, it SHOULD take note of this fact while processing a request, even if the request is sent with a special stateid 🌟.

8.2.6. Stateid Use for SETATTR Operations 🎀

Each operation is associated with a session ID, and from that the clientid can be determined. So operations don't need to include a stateid for the server to figure out whether they should cause a delegation to be recalled or are to be treated as done within the scope of the delegation 🧠.

For SETATTR operations, a stateid is present. In cases other than those that set the file size, the client may send either a special stateid or, when a delegation is held for the file in question, a delegation stateid 💜. The server SHOULD validate the stateid, and it may use the stateid to optimize the determination of whether a delegation is held. Even when a special stateid is sent, it SHOULD note the presence of a delegation, and it MUST accept a valid delegation stateid when one is sent 👑.

8.3. Lease Renewal 💖

okay so each client/server pair, as represented by a client ID, gets a single lease 🎀 The whole point is to let the client say, in a super low-overhead way, "hey, I'm still active!" 🦋 That tells the server to keep the client's locks. It also means the server can clean up stale locking objects held by a client that crashed or is otherwise unreachable, once that lease expires. Other clients can then grab conflicting locks without being stuck waiting forever on inactive or unreachable clients 💅 Leases are not a cache consistency mechanism, and lease renewals may not be denied if the lease interval hasn't expired yet.

Since each session belongs to one specific client (identified by its client ID), any operation sent on that session shows the client is reachable ✨ When a request goes out on a session, a successful SEQUENCE operation (or a successful retrieval of the SEQUENCE result from the reply cache) on an unexpired lease implicitly renews it for the standard renewal period, which is equal to the lease_time attribute. Slay 💜

If the client ID's lease hasn't expired when the server gets a SEQUENCE operation, then the server MUST renew the lease 🫶 If the client ID's lease has expired when the server gets a SEQUENCE operation, the server MAY renew it. Whether it does depends on whether any state was revoked because the client failed to renew before expiration 🙈

Absent other activity that would renew the lease, a COMPOUND consisting of just one SEQUENCE operation will do the job. The client should also think about communication-related delays and make sure its renewal messages actually reach the server in good time. For example:

When trunking is in effect, the client should consider sending multiple requests on different connections, so renewal still happens even if the path used for one of those connections gets blocked 🌈
Transport retransmission delays might get so big that they approach or go past the length of the lease period 😭 This is especially likely when the server is unresponsive because it restarted (see Section 8.4.2.1). If the client implementation isn't careful, those retransmission delays can make the client miss a server restart before the grace period ends. Here's the nightmare scenario: the client is using a transport with exponential backoff, so the maximum retransmission timeout is bigger than both the grace period and the lease_time attribute. A network partition makes the client's connection retransmission interval back off, and even after the partition heals, the next transport-level retransmission gets sent after the server has restarted and its grace period has ended 💀

The client MUST either recover from the ensuing NFS4ERR_NO_GRACE errors, or it MUST ensure that, despite transport-level retransmission intervals that exceed lease_time, a SEQUENCE operation gets sent that renews the lease before it expires. The client can manage this by associating a new connection with the session and sending a SEQUENCE operation on it. However, if setting up that new connection is delayed for some reason (for example, exponential backoff of the connection establishment packets), the client will have to abort the connection establishment attempt before the lease expires and then try to reconnect 🍓

If the server renews the lease upon receiving a SEQUENCE operation, the server MUST NOT allow the lease to expire while the rest of the operations in that COMPOUND request are still executing 👑 Once the last operation has finished and the response to COMPOUND has been sent, the server MUST set the lease to expire no sooner than the sum of the current time and the value of the lease_time attribute ✨

A client ID's lease can expire when at least the lease interval (lease_time) has passed since the last lease-renewing SEQUENCE operation was sent on any of that client ID's sessions, and there are no active COMPOUND operations on any of those sessions 🧚

Because SEQUENCE is the basic mechanism for renewing a lease, and because it has to happen at least once per lease period, it's the natural way for the server to tell the client about lease status changes it needs to know about 🔥 The client should inspect the status flags (sr_status_flags) returned by SEQUENCE and take the appropriate action (see Section 18.46.3 for details). Here's the tea:

The status bits SEQ4_STATUS_CB_PATH_DOWN and SEQ4_STATUS_CB_PATH_DOWN_SESSION mean there are problems with the backchannel that the client may need to fix so it can receive callback requests 🛠️
The status bits SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING and SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED mean there are problems with GSS contexts or RPCSEC_GSS handles for the backchannel. The client might have to address these so callback requests can be sent 🌸
The status bits SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED, SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, SEQ4_STATUS_ADMIN_STATE_REVOKED, and SEQ4_STATUS_RECALLABLE_STATE_REVOKED tell the client about lock revocation events 😭 When any of these are set, the client should use TEST_STATEID to find out which stateids have been revoked, and use FREE_STATEID to acknowledge the loss of the associated state 💅
The status bit SEQ4_STATUS_LEASE_MOVE means responsibility for lease renewal has been handed off to one or more new servers 🦋
The status bit SEQ4_STATUS_RESTART_RECLAIM_NEEDED means that, because the server restarted, the client must reclaim its locking state 🎀
The status bit SEQ4_STATUS_BACKCHANNEL_FAULT means the server has hit an unrecoverable fault with the backchannel (for example, it has lost track of a sequence ID for a slot in the backchannel) 🙈

8.4. Crash Recovery 🌈

A critical requirement in crash recovery is that both the client and the server know when the other one has failed 🫶 It's also required that a client sees a consistent view of data across server restarts. All READ and WRITE operations that might be sitting in client or network buffers have to wait until the client has successfully recovered the locks protecting those READ and WRITE operations. Any that reach the server before the server can safely determine that the client has recovered enough locking state to be sure they can be processed safely must be rejected. This happens because either:

The state presented is no longer valid, since it's tied to a client ID that's now invalid. In this case, the client gets either an NFS4ERR_BADSESSION or NFS4ERR_DEADSESSION error, and any attempt to attach a new session to that invalid client ID results in an NFS4ERR_STALE_CLIENTID error 💀
Subsequent recovery of locks might make executing the operation inappropriate (NFS4ERR_GRACE) 😭

8.4.1. Client Failure and Recovery 🧸

In the event that a client fails, the server may release that client's locks once the associated lease has expired. Conflicting locks from another client can only be granted after that lease expiration 🌸 As discussed in Section 8.3, if a client hasn't failed and re-establishes its lease before expiration occurs, requests for conflicting locks won't be granted 🔒

To minimize client delay upon restart, lock requests are associated with an instance of the client by a client-supplied verifier 💖 This verifier is part of the client_owner4 sent in the initial EXCHANGE_ID call the client makes. The server returns a client ID as the result of the EXCHANGE_ID operation. The client then confirms the use of that client ID by establishing a session associated with it (see Section 18.36.3 for a description of how this is done). All locks, including opens, byte-range locks, delegations, and layouts obtained by sessions using that client ID, are associated with that client ID ✨

Since the client changes the verifier upon each initialization, the server can compare a new verifier to the verifier associated with currently held locks and see that they don't match 🦋 This signals the client's new instantiation and the subsequent loss (upon confirmation of the new client ID) of its locking state. As a result, the server is free to release all held locks associated with the old client ID that was derived from the old verifier. At that point, conflicting locks from other clients, which were kept waiting while the lease hadn't yet expired, can finally be granted 🎉 In addition, all stateids associated with the old client ID can also be freed, since they're no longer reference-able 🧚

Note that the verifier must have the same uniqueness properties as the verifier for the COMMIT operation 🎀

8.4.2. Server Failure and Recovery 🌷

If the server loses locking state (usually because it restarted), it must give clients time to discover this and re-establish the lost locking state 🕰️ The client must be able to re-establish its locking state without the server denying valid requests because it has granted conflicting access to another client. The same goes for when there's a possibility that clients haven't yet re-established their locking state for a file, and that this locking state might make it invalid to perform READ or WRITE operations. For example, if mandatory locks are a possibility, the server must disallow READ and WRITE operations for that file 🔐

A client can figure out that loss of locking state has happened in several ways 👀

When a SEQUENCE (the most common) or other operation returns NFS4ERR_BADSESSION, this may mean the session has been destroyed but the client ID is still valid. The client sends a CREATE_SESSION request with the client ID to re-establish the session. If CREATE_SESSION fails with NFS4ERR_STALE_CLIENTID, the client MUST establish a new client ID (see Section 8.1) and re-establish its lock state with the new client ID, after the CREATE_SESSION operation succeeds (see Section 8.4.2.1) 💅
When a SEQUENCE (the most common) or other operation on a persistent session returns NFS4ERR_DEADSESSION, this indicates the session is no longer usable for new operations (that is, operations not satisfied from the reply cache) 🥺 Once all pending operations are determined to be either performed before the retry or not performed, the client sends a CREATE_SESSION request with the client ID to re-establish the session. If CREATE_SESSION fails with NFS4ERR_STALE_CLIENTID, the client must establish a new client ID (see Section 8.1) and re-establish its lock state after the CREATE_SESSION with the new client ID succeeds (Section 8.4.2.1) 🌈
When an operation that is neither SEQUENCE nor preceded by SEQUENCE (for example, CREATE_SESSION or DESTROY_SESSION) returns NFS4ERR_STALE_CLIENTID, the client MUST establish a new client ID (Section 8.1) and re-establish its lock state (Section 8.4.2.1) 🫶

8.4.2.1. State Reclaim ✨

okay so when the server restarts and its state and locks get wiped, the protocol needs a way to put that state back 💖 The plan is that for most kinds of locking state (layouts are the exception, fr), there's a request whose whole job is letting the client re-establish a lock it got from an earlier server instance. These are usually souped-up versions of the normal lock-creating requests. We call them "reclaim-type" requests, and redoing a lock this way is called "reclaiming" it 🦋

Every client needs a fair shot at reclaiming all of its locks without some other client sneaking in a conflicting lock, so there's a "grace period" set aside just for reclaiming 👑 During that window, requests that create client IDs and sessions get handled as usual, but locking requests have special rules. Only reclaim-type locking requests are allowed, UNLESS the server can reliably figure out (using state it kept persistently across restarts) that granting a given lock can't possibly clash with a later reclaim. If someone asks for a brand new lock (so not reclaim-type) during the grace period and the server can't make that call, the server must return the error NFS4ERR_GRACE 🙈

Once a session exists on the new client ID, the client uses reclaim-type locking requests (like LOCK operations with reclaim set to TRUE, and OPEN operations with a claim type of CLAIM_PREVIOUS; see Section 9.11) to rebuild its locking state. When that's done, or if there was no locking state to reclaim at all, the client sends a global RECLAIM_COMPLETE operation (one with rca_one_fs set to FALSE) to say "I've reclaimed everything I'm going to reclaim" 💅 After that, it can try non-reclaim locking operations, though it might still get NFS4ERR_GRACE back from each one until the special period is over. See Section 11.11.9 for the analogous lock reclamation when file systems move from server to server 🌸

During the grace period, the server must reject READ and WRITE operations and non-reclaim locking requests (that's the other LOCK and OPEN operations) with NFS4ERR_GRACE, unless it can guarantee they're safe, as described below 🫶

The grace period can last until every client that might have had locks has sent a global RECLAIM_COMPLETE, saying it's done reclaiming the locks it held before the restart. So a client that already sent RECLAIM_COMPLETE still has to be ready to get NFS4ERR_GRACE when it asks for new locks ✨ For the server to know that all clients with possible prior lock state are done, it must keep a list of those clients in stable storage. The server may also end the grace period early, before every client has sent a global RECLAIM_COMPLETE. The server SHOULD NOT end the grace period before a time equal to the lease period. That gives clients a chance to find out about the restart through their requests on associated sessions, which go out at a frequency governed by the lease time. Heads up: if a client doesn't send those requests (or they're sent but the server never receives them), the grace period could expire before the client even notices the restart 😭

Some additional time may be added to the lease time so a client can establish a new client ID and session and do its lock reclaims. Analogous rules apply to file system-specific grace periods discussed in Section 11.11.9 🌈

If the server can reliably tell that granting a non-reclaim request won't conflict with other clients reclaiming their locks, NFS4ERR_GRACE doesn't have to be returned even during the grace period. Still, NFS4ERR_GRACE must always go to clients attempting a non-reclaim lock request before they've done their own global RECLAIM_COMPLETE. For the server to handle READ and WRITE during the grace period, it must again be able to guarantee that no possible conflict could arise between a potential reclaim locking request and the READ or WRITE. If it can't offer that guarantee, NFS4ERR_GRACE must be returned to the client 🧚

For a server that wants simple, valid handling during the grace period, the easiest method is to reject every non-reclaim locking request and every READ and WRITE by returning NFS4ERR_GRACE. However, a server may keep information about granted locks in stable storage. With that info, the server can decide whether a locking, READ, or WRITE operation can be safely processed 🍓

For example, if the server kept summary info in stable storage on whether mandatory locks exist (either mandatory byte-range locks or share reservations specifying deny modes), lots of requests could be allowed during the grace period. If it's known that no such share reservations exist, OPEN requests that don't specify deny modes may safely be granted. If it's also known that no mandatory byte-range locks exist (either from info in stable storage or simply because the server doesn't support such locks), READ and WRITE operations may safely be processed during the grace period. Another important case is where it's known that no mandatory byte-range locks exist, either because the server doesn't support them or because their absence is recorded in persistent data. Then READ and WRITE operations specifying stateids derived from reclaim-type operations may validly be processed during the grace period. The valid reclaim ensures that no lock granted later can prevent the I/O 💜

To reiterate, for a server that allows non-reclaim lock and I/O requests to be processed during the grace period, it MUST determine that no lock subsequently reclaimed will be rejected, and that no lock subsequently reclaimed would have prevented any I/O operation processed during the grace period. 🔥

Clients should be prepared for the return of NFS4ERR_GRACE errors for non-reclaim lock and I/O requests. In this case, the client should employ a retry mechanism for the request. A delay (on the order of several seconds) between retries should be used to avoid overwhelming the server. Further discussion of the general issue is included in [55]. The client must account for the server that can perform I/O and non-reclaim locking requests within the grace period, as well as those that cannot do so 🦋

A reclaim-type locking request outside the server's grace period can only succeed if the server can guarantee that no conflicting lock or I/O request has been granted since restart 🌸

A server may, upon restart, establish a new value for the lease period. Therefore, clients should, once a new client ID is established, refetch the lease_time attribute and use it as the basis for lease renewal for the lease associated with that server. However, the server must establish, for this restart event, a grace period at least as long as the lease period for the previous server instantiation. This allows the client state obtained during the previous server instance to be reliably re-established 🎀

The possibility exists that, because of server configuration events, the client will be communicating with a server different from the one on which the locks were obtained, as shown by the combination of eir_server_scope and eir_server_owner. This raises the question of if and when the client should attempt to reclaim locks previously obtained on what is being reported as a different server. The rules to resolve this are as follows:

If the server scope is different, the client should not attempt to reclaim locks. In this situation, no lock reclaim is possible. Any attempt to re-obtain the locks with non-reclaim operations is problematic, since there is no guarantee that the existing filehandles will be recognized by the new server, or that, if recognized, they denote the same objects. It is best to treat the locks as having been revoked by the reconfiguration event. 🙅‍♀️
If the server scope is the same, the client should attempt to reclaim locks, even if the eir_server_owner value is different. In this situation, it is the responsibility of the server to return NFS4ERR_NO_GRACE if it cannot provide correct support for lock reclaim operations, including the prevention of edge conditions. 🧠

The eir_server_owner field is not used in making this determination. Its function is to specify trunking possibilities for the client (see Section 2.10.5), not to control lock reclaim 💅

8.4.2.1.1. Security Considerations for State Reclaim ✨

okay so bestie, during the grace period a client can reclaim state that it believes or asserts it had before the server restarted 🥺. unless the server kept a complete record of every single bit of state that client had, the server basically has little choice but to trust the client 🫠. (and yes, if the server had a complete record, it wouldn't need to make the client reclaim anything after the restart in the first place, lol 💀)

the server has to trust the client to tell the truth, but the security fallout is pretty limited. it mostly just opens the door to denial-of-service attacks, and only in situations where AUTH_SYS is supported 🚪. the main rule for the server when it processes reclaim requests is giving a firm no to the reclaim if the same non-reclaim request would've been denied during steady state because of access control or access conflict issues. it MUST NOT grant the reclaim in that case 🙅‍♀️. for example, an OPEN reclaim gets refused with NFS4ERR_ACCESS if the principal making the request doesn't have access to open the file according to the discretionary ACL (Section 6.2.2) on that file 💅.

BUT. it's still possible for a client that's acting up, either by accident or on purpose, to block another client from reclaiming its state during reclaim 😤. picture an attacker sending an OPEN reclaim with a deny mode that stops another client from getting back the OPEN state it had before the restart. the attacker could pull the exact same denial-of-service move during steady state before the restart too, as long as they had the permissions 🕵️‍♀️. so the attack vectors are basically the same, which means the grace period doesn't hand anyone a new way to cause denial of service ✨. any worries about this, whether it's during grace or steady state, get handled the same way: use RPCSEC_GSS for authentication, and only let principals that the owner of the file trusts access that file 🔐🫶.

one more thing, and it's a big one 🚨. if the server had client IDs with the EXCHGID4_FLAG_BIND_PRINC_STATEID capability set before the restart (Section 18.35), then the server SHOULD record in stable storage the client owner and the principal that established the client ID via EXCHANGE_ID 💾. if it doesn't, a client could end up unable to reclaim its state if it doesn't have a credential for the principal that was originally authorized to set up that state. that's a whole vibe-killer, so don't skip it 😭💜

8.4.3. Network Partitions and Recovery 🌸

okay so bestie, this section is about what happens when the network splits the client and server apart for a while, and it is such a drama 😭🫶 The big question is: if the client can't renew its lease, does the server just drop everything? Not necessarily! It's giving "the server gets to choose," but with some strict rules attached 💅

If the partition lasts longer than the lease period the server gave out, the server won't have gotten any lease renewal from the client. At that point the server may free all of the client's locks, or it may let the lock state hang around for a good while 🦋 BUT, and this is the important part, if someone makes a request for a conflicting lock, the locks tied to that expired lease don't block it from being granted. They MUST be revoked as needed so they don't get in the way of the conflicting request 🔥

Now if the server waits to free things until there's an actual conflict, it has two flavours. It can free all of the client's locks once a conflict shows up, or it can only revoke the minimum set needed for the conflicting requests 🎀 If it picks the finer-grained option, it must revoke every lock associated with a given stateid, even when the conflict only touches part of them. No cherry-picking within a stateid, bestie 🙈

When the server decides to free all of a client's lock state, whether right when the lease expires or because of the first conflicting lock attempt, it can tell the client about the loss in a few different ways 🧚

Option one: the server can invalidate the session and the client ID. Then, once the client can talk to the server again, it gets NFS4ERR_BADSESSION. If it tries to create a new session, it gets NFS4ERR_STALE_CLIENTID. Once the client makes a brand-new client ID and session, it'll try to reclaim its locks. But usually the server won't allow that, because it won't be in its recovery grace period anymore 💀

Option two: the server keeps the session and client ID, but every stateid the client had becomes invalid or stale. Once the partition heals, the SEQUENCE operation's status will show the loss of locking state, meaning the flag SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED gets set in sr_status_flags. Any I/O the client sends with those now-invalid stateids will fail with NFS4ERR_EXPIRED 😭 Once the client finds out about the loss, it should let the applications that held those locks know. Then it should clean up the invalidated stateids, either by setting up a new client ID with a new verifier or by sending a FREE_STATEID for each one ✨

When the server takes the finer-grained route, only some stateids usually go invalid during the partition. After the partition heals, SEQUENCE will show a partial loss, using SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED 🌈 Operations with the now-invalid stateids, I/O included, will fail with NFS4ERR_EXPIRED. The client then uses TEST_STATEID on all of its stateids to figure out which locks were lost, and tells the affected applications. After that, it can release the invalidated locking state and acknowledge the revocations by sending a FREE_STATEID for each invalidated stateid 🍓

Next up is the scary bit: a network partition plus a server restart creates edge cases that could cause silent data corruption if the server isn't careful. Two of these are known, and we'll go through both 👑

Edge condition one goes like this:

Client A acquires a lock.
Client A and the server get into a mutual network partition, so client A can't renew its lease.
Client A's lease expires, and the server releases the lock.
Client B acquires a lock that would have conflicted with client A's.
Client B releases its lock.
The server restarts.
The partition between client A and the server heals.
Client A connects to the new server instance and learns about the restart.
Client A reclaims its lock within the server's grace period.

See the problem? At that last step, the server has wrongly granted client A's reclaim 😭 If client B changed the object the lock was protecting, client A ends up with corrupted data 💔

Edge condition two goes like this:

Client A acquires one or more locks.
The server restarts.
Client A and the server get into a mutual network partition, so client A can't reclaim all of its locks within the grace period.
The server's reclaim grace period ends. Client A has either no locks or only some of them, from the server's point of view.
Client B acquires a lock that would have conflicted with one of client A's unreclaimed locks.
Client B releases the lock.
The server restarts a second time.
The partition between client A and the server heals.
Client A connects to the new server instance and learns about the restart.
Client A reclaims its lock within the server's grace period.

Same deal as before: at the final step, the server has wrongly granted client A's reclaim 💅

So how do you fix these? Solving both edge conditions means the server either always assumes after a restart that one of these edge conditions happened and returns NFS4ERR_NO_GRACE to every reclaim attempt, or it records some information in stable storage. The more info it saves, the less harsh it has to be whenever these edge conditions pop up 🎀 A server that's totally tolerant of every edge condition would record every acquired lock in stable storage, and only remove that record when the lock is released. For the two edge conditions above, the harshest a server can be while still supporting a reclaim grace period is to save some minimal info in stable storage. For example, an implementation could save, for each client, a record containing:

the co_ownerid field from the client_owner4 presented in the EXCHANGE_ID operation.
a boolean that says whether the client's lease expired, or whether there was administrative intervention (see Section 8.5) to revoke a byte-range lock, share reservation, or delegation, with no acknowledgment of that revocation via FREE_STATEID yet.
a boolean that says whether the client might have locks it thinks are reclaimable, in situations where the grace period ended and the server's view of reclaimability might be shaky. The server sets this for any client record in stable storage where the client hasn't done a suitable RECLAIM_COMPLETE (global, or file system-specific depending on the lock request's target) before the server grants any new (not reclaimed) lock to any client 🌸

With that record-keeping in place, here's how the edge conditions get handled. In the first one, after the restart, the record showing that client A's lease expired means another client could have grabbed a conflicting byte-range lock, share reservation, or delegation. So the server must reject client A's reclaim with NFS4ERR_NO_GRACE 🙅‍♀️

In the second one, after the second restart, the sign that client A hadn't finished its reclaims when the grace period ended means the server must reject client A's reclaim with NFS4ERR_NO_GRACE too 😤

When either edge condition happens, the client's reclaim attempt gets NFS4ERR_NO_GRACE. When that comes back, or when the client restarts with no lock state, the client sends a global RECLAIM_COMPLETE. Once the server receives it, the server and client agree again on which locks are reclaimable, and both booleans in persistent storage can be reset. They only get set again when another event makes lock reclaims questionable ✨

No matter how much or how little record-keeping the server does, the server MUST implement one of the following strategies, which apply to reclaims of share reservations, byte-range locks, and delegations 🦋

Reject all reclaims with NFS4ERR_NO_GRACE. This is super unforgiving, but it's necessary if the server doesn't record lock state in stable storage.
Record enough state in stable storage that every known edge condition involving a server restart, including the two covered in this section, gets detected. It's fine to wrongly spot an edge condition and refuse a reclaim that would've been allowed with more info. The error the server returns in that case is NFS4ERR_NO_GRACE. Heads up though: it isn't known whether there are other edge conditions 👀

If, after a restart, the server finds unrecoverable damage or corruption in its stable storage information, then for every affected client and/or lock, the server MUST return NFS4ERR_NO_GRACE 💀

How the client handles NFS4ERR_NO_GRACE is outside the scope of this specification, since it depends a lot on the client's operating environment. But here's one possible approach 🧚

When the client gets NFS4ERR_NO_GRACE, it could look at the change attribute of the objects it's trying to reclaim state for, and use that to decide whether to re-establish the state with normal OPEN or LOCK operations. That's fine as long as the client's environment allows it. Basically, the client implementor is encouraged to document this behaviour for their users 📝 The client could also tell the application that its byte-range lock or share reservations (delegated or not) were lost, through something like a UNIX signal, a GUI pop-up window, and so on. For how the client should deal with unreclaimed delegations on client state, see Section 10.5 💜

For more on lock revocation, see Section 8.5 🌈

8.5. Server Revocation of Locks ✨

okay so bestie, the server can revoke locks a client is holding at ANY point 😭 and the client has to be ready for that. no cap. when the client finds out its locks have been revoked (or might have been), it's on the client to validate the locking state between itself and the server 💅. "validating" here just means it has to verify or reclaim the state for every lock it currently holds. main character energy for the client, tbh 🫶

there are three main ways this can happen, and it's giving "three plot twists" 🎀

The first one is at server restart 🔄. this includes the situations where sessions stick around but the locking state gets lost anyway ✨. in that case the client gets an error, NFS4ERR_STALE_CLIENTID, on an operation that takes a client ID (usually while it's trying to recover from some problem with the current session). then it just goes through normal crash recovery, as described in Section 8.4.2.1. simple, iconic 💖

The second one is when the client can't renew its lease before it expires, as covered in Section 8.4.3 🥺. the lock revocation is considered rare or unusual, but the client still has to be prepared to recover 💪. the server gets to decide exactly what the lease expiration means and tells the client how far the revocation reaches 👑. the client then checks the status info the server sends back in the SEQUENCE results (the sr_status_flags field, see Section 18.46.3) so it can sync its locking state with the server and recover 🦋

The third one can happen within the lease period 🙈, either because of administrative intervention, or because a recallable lock (a delegation or layout) didn't get returned in time after being recalled. again, rare, but totally possible, so the client has to be ready 💀. when this happens, the client finds out through the status that comes back from the SEQUENCE operation. any use of stateids tied to locks that got revoked during the lease period will get back NFS4ERR_ADMIN_REVOKED or NFS4ERR_DELEG_REVOKED, depending on what happened 🌸

now, whenever a subset of locking state might have been revoked (and that includes every case where it gets revoked inside the lease period), the client has to figure out which locks are gone and which are still good 🔍. it does this with the TEST_STATEID operation, run on the right set of stateids 🧚. once it knows which locks were revoked, the apps can be notified, the dead stateids can be freed, and the revocation gets acknowledged with FREE_STATEID 🎀✨

8.6. Short and Long Leases 🍓

okay, lease length is a whole trade-off situation, bestie 😤. when the server picks the lease time, the usual lease trade-offs apply. a short lease means fast server recovery 🏃‍♀️, but it costs more operations to renew the lease (unless there are other operations in that period that renew it as a side effect). a long lease is way kinder and gentler to servers that are handling a ton of clients 🫶🌈. the number of extra requests needed to renew locks drops in inverse proportion to how long the lease is 📉

the downside of a long lease? recovery after certain failures can be slower 🐢. after a server failure, you might need a longer grace period when some clients don't reclaim their locks quickly and do a global RECLAIM_COMPLETE 😭. and if a client fails, the longer wait for the lease to expire means conflicting requests have to wait longer too 💅

BUT a long lease is practical if the server can keep its lease state in stable storage 💾✨. then, when it recovers, the server can rebuild the lease state from that storage and keep going with its clients like nothing happened 🦋

8.7. Clocks, Propagation Delay, and Calculating Lease Expiration ⏰

okay so the whole point is to avoid needing synchronized clocks 🙌. that's why lease times get granted by the server as a time delta instead of an actual timestamp 💖. BUT the client and server clocks still can't drift too much over the life of the lease 😬. and then there's propagation delay across the network, which can easily be several hundred milliseconds 🐌, plus the chance that requests get lost and have to be retransmitted 💀

so to account for propagation delay, the client should subtract it from the lease times 🧮. like, if the client estimates the one-way propagation delay as 200 milliseconds, it can assume the lease is already 200 milliseconds old when it receives it 🫠. then it takes another 200 milliseconds for a response to get back to the server. so the client MUST send a lease renewal or write data back to the server at least 400 milliseconds before the lease would expire 🚨 (yes, that's the actual number, bestie 💅). if the propagation delay changes over the life of the lease (say the client is on a mobile host 📱), the client needs to keep subtracting the increase in propagation delay from the lease times 🔁

the server's lease period config should also take into account the network distance of the clients that are going to be accessing its resources 🗺️. it's expected that the lease period accounts for the network propagation delays and other network delay factors for the whole client population ✨. and since the protocol doesn't have an automatic way to figure out a good lease period, the server's administrator may have to tune it by hand 🛠️🎀

8.8. Obsolete Locking Infrastructure from NFSv4.0 🗑️

okay so, there are a bunch of operations and fields from existing operations that just don't do anything in NFSv4.1 anymore 🥀. honestly, it's all because of sessions 💖. sessions give you client context and exactly once semantics as a base feature of the protocol, separate from locking itself, so a lot of the old locking stuff is just... not needed 💅

the following NFSv4.0 operations MUST NOT be implemented in NFSv4.1 🚫. and if one of them shows up in an NFSv4.1 COMPOUND, the server MUST return NFS4ERR_NOTSUPP 😭. here's the list of the bye-bye ones:

SETCLIENTID, since its job got taken over by EXCHANGE_ID 🔄
SETCLIENTID_CONFIRM, since client ID confirmation now happens through CREATE_SESSION ✨
OPEN_CONFIRM, because state-owner-based seqids got replaced by the sequence ID in the SEQUENCE operation 🎀
RELEASE_LOCKOWNER, because lock-owners with no associated locks don't have any sequence-related state, so the server can delete them whenever it wants 🧹
RENEW, because every SEQUENCE operation for a session already causes lease renewal, so a separate operation is superfluous 🙄

also, there are a bunch of fields in existing operations, related to locking, that have no use in minor version 1 🥲. they were used in minor version 0 to do stuff that's now handled in a different way:

Sequence ids used to sequence requests for a given state-owner and to provide retry protection. those are now handled via sessions 💖
Client IDs used to identify the client for a given request. client identification now comes from the client ID tied to the current session, so there's no explicit client ID field needed 🦋

those vestigial fields have no function in NFSv4.1 and the server ignores them 🫶. just a heads up though: client IDs in operations that are new to NFSv4.1 (like CREATE_SESSION and DESTROY_CLIENTID) are NOT ignored 🚨 so don't get it twisted, bestie 💅

okay so to support Win32 share reservations, the protocol needs operations that atomically open or create files ✨. a separate share/unshare operation just wouldn't let you implement the Win32 OpenFile API correctly 😤. so to get share semantics right, the old NFS mechanisms used when opening or creating a file (LOOKUP, CREATE, ACCESS) had to be replaced 🔄. NFSv4.1 defines an OPEN operation that can atomically look up, create, AND lock a file on the server 🎀🔥

9.1. Opens and Byte-Range Locks 🔒

it's assumed that manipulating a byte-range lock is rare compared to READ and WRITE operations 📊. it's also assumed that server restarts and network partitions are relatively rare 🌧️. so it's important that READ and WRITE have a lightweight mechanism to indicate whether they hold a lock 🪶. a LOCK operation, on the other hand, carries the heavyweight info needed to establish a byte-range lock and uniquely define its owner 🏋️‍♀️💅

9.1.1. State-Owner Definition 👑

when opening a file or requesting a byte-range lock, the client MUST specify an identifier that represents the owner of the requested lock 🫶. this identifier is a state-owner, represented in the protocol by a state_owner4, which is a variable-length opaque array. when it's concatenated with the current client ID, it uniquely defines the owner of a lock managed by the client ✨. this could be a thread ID, a process ID, or some other unique value 🧵

owners of opens and owners of byte-range locks are separate entities 🙅‍♀️, and they stay separate even if the same opaque arrays are used to designate owners of each. the protocol tells them apart with open-owners (open_owner4 structures) and lock-owners (lock_owner4 structures) 🦋

each open is associated with one specific open-owner 💖. each byte-range lock is associated with a lock-owner AND an open-owner, and that open-owner is the one tied to the open file under which the LOCK operation was done 🎀. delegations and layouts, on the other hand, aren't tied to a specific owner at all 🙈. they're associated with the client as a whole (identified by a client ID) 🌸

this is the part where you realize the rules are sorted by who holds what, bestie 🧚

9.1.2. Use of the Stateid and Locking ✨

Okay bestie, so every READ, WRITE, and SETATTR op carries a stateid 🦋. For this section, SETATTRs that change the size attribute get treated like they're writing the area between the old and new sizes (the byte range that gets chopped off or tacked on by the SETATTR) 💅. That's true even when SETATTR isn't mentioned by name in the text. The stateid you pass to one of these ops must represent an open, a set of byte-range locks, or a delegation. Or it may be a special stateid for anonymous access or the special bypass stateid 🌸.

If the state-owner does a READ or WRITE while it's holding a byte-range lock or share reservation on the server (and yep, any OPEN counts as a share reservation 🙈), the stateid it got back from the server earlier must be used. That stateid says which locks are held by the state-owner, both byte-range locks and share reservations 🔐. If the client hasn't established any state, neither a byte-range lock nor a share reservation, then a special stateid for anonymous state gets used. That one has zero as the value for other and seqid. (Check Section 8.2.3 for the deets on "special" stateids in general 🧚.) Whether you send an anonymous stateid or a server-issued one, if there's a conflicting share reservation or a mandatory byte-range lock on the file, the server MUST refuse to service the READ or WRITE operation 🚫.

Share reservations come from OPEN operations, and they're mandatory by nature. When an OPEN denies READ or WRITE, those operations get rejected with the error NFS4ERR_LOCKED 🔒. Byte-range locks can be implemented as mandatory or advisory. Or the server can decide based on the file being accessed. For example, some UNIX-based servers have a "mandatory lock bit" on the mode attribute, and if it's set, byte-range locks are required on the file before any I/O can happen 🎀. When byte-range locks are advisory, they only stop conflicting lock requests and have zero effect on READs or WRITEs. Mandatory byte-range locks are the ones that actually stop conflicting I/O, and when that I/O gets attempted, it's rejected with NFS4ERR_LOCKED 💥. Now, if the client gets NFS4ERR_LOCKED on a file where it knows it has the proper share reservation, it'll need to send a LOCK operation. That LOCK has to cover the byte range the I/O was headed for, with an appropriate locktype field in its arguments: READ*_LT for a READ, and WRITE*_LT for a WRITE 🦋.

Note that for UNIX environments that support mandatory byte-range locking, the difference between advisory and mandatory locking is subtle 🤓. Honestly, advisory and mandatory byte-range locks are exactly the same as far as the APIs and implementation requirements go. If the mandatory lock attribute is set on the file, the server checks whether the lock-owner has an appropriate shared (READ_LT) or exclusive (WRITE_LT) byte-range lock on the range it wants to READ from or WRITE to. If there's no appropriate lock, the server checks for a conflicting lock. It can do that by trying to acquire the conflicting lock on behalf of the lock-owner, and if that works, releasing it once the READ or WRITE is done. And if there is a conflicting lock, the server returns NFS4ERR_LOCKED 🫶.

For Windows environments, byte-range locks are always mandatory, so the server always checks for byte-range locks during I/O requests 🪟.

So the LOCK operation doesn't need to tell advisory and mandatory byte-range locks apart. It's the server's processing of the READ and WRITE operations that creates the distinction ✨.

Every stateid that's validly passed to READ, WRITE, or SETATTR defines an access mode for the file, with the exception of special stateid values 🌈. The access modes are OPEN4_SHARE_ACCESS_READ, OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH. Here's how each kind of stateid sets it:

For stateids associated with opens, the access mode is the one defined by the original OPEN that caused the allocation of the OPEN stateid. It also gets modified by subsequent OPENs and OPEN_DOWNGRADEs for the same open-owner/file pair. 👑
For stateids returned by byte-range LOCK operations, the appropriate mode is the access mode for the OPEN stateid associated with the lock set that the stateid represents. 🍓
For delegation stateids, the access mode is based on the type of delegation. 💜

When a READ, WRITE, or SETATTR (one that specifies the size attribute) gets done, the operation is checked against the access mode. The point is to verify the operation is appropriate given the stateid it's associated with 🔍.

For WRITE-type operations (that's WRITEs and SETATTRs that set size), the server MUST verify that the access mode allows writing. If it doesn't, the server MUST return an NFS4ERR_OPENMODE error 🙅‍♀️. For READ, the server may do the corresponding check on the access mode. Or it may choose to allow READ on OPENs for OPEN4_SHARE_ACCESS_WRITE, which accommodates clients whose WRITE implementation may unavoidably do reads, like because of buffer cache constraints 🧁. Even so, if READs are allowed in those circumstances, the server MUST still check for locks that conflict with the READ. An example would be another OPEN that specified OPEN4_SHARE_DENY_READ or OPEN4_SHARE_DENY_BOTH 🚧. Now, a server that does enforce the access mode check on READs doesn't need to explicitly check for conflicting share reservations. The reason is that an OPEN for OPEN4_SHARE_ACCESS_READ already guarantees that no conflicting share reservation can exist 💅.

The READ bypass special stateid (all bits of other and seqid set to one) basically says "I'd like to skip the locking checks, please" 🥺. The server MAY allow READ operations to bypass locking checks when this special stateid is used. But WRITE operations with this special stateid value MUST NOT bypass locking checks. They're treated exactly the same as if a special stateid for anonymous state had been used. Not on my watch 🙅‍♀️.

A lock may not be granted while a READ or WRITE using one of the special stateids is being performed, if the scope of the lock to be granted would conflict with that READ or WRITE. This can happen when:

A mandatory byte-range lock is requested with a byte range that conflicts with the byte range of the READ or WRITE. For this purpose, a conflict happens when a shared lock is requested while a WRITE is being performed, or when an exclusive lock is requested while either a READ or a WRITE is being performed. 🔐
A share reservation is requested that denies reading and/or writing, and the corresponding operation is being performed. 🚷
A delegation is about to be granted, and the delegation type would prevent the I/O operation. READ and WRITE both conflict with an OPEN_DELEGATE_WRITE delegation, and WRITE conflicts with an OPEN_DELEGATE_READ delegation. 🌸

When a client holds a delegation, it needs to make sure the stateid it sends conveys that the operation is associated with the delegation. That way the delegation doesn't get recalled when it totally didn't have to 😭. When the delegation stateid is used, or a stateid open associated with that delegation, or a stateid representing byte-range locks derived from such an open, the server knows the READ, WRITE, or SETATTR doesn't conflict with the delegation but is sent under its aegis 🦋. The server could even work out from the client ID (via the session ID) that the client really does have a delegation. But the server isn't obliged to check that. So using a special stateid can result in an avoidable recall of the delegation 💀.

9.2. Lock Ranges 🌈

Okay so the protocol lets a lock-owner request a lock with a byte range, and then upgrade, downgrade, or unlock a sub-range of that initial lock. It can also request a byte range that overlaps, fully or partially, with that initial lock or with a combo of existing locks for the same lock-owner 🧩. This is expected to be an uncommon kind of request, so don't panic. Still, servers or server file systems may not be able to support sub-range lock semantics at all 🫠.

If a server gets a locking request that represents a sub-range of the current locking state for the lock-owner, the server is allowed to return the error NFS4ERR_LOCK_RANGE. That error signifies that it doesn't support sub-range lock operations. So the client should be prepared to receive this error and, if appropriate, report it to the requesting application 📣.

The client is discouraged from combining multiple independent locking ranges that just happen to be adjacent into a single request 🙅‍♀️. The reason is that the server may not support sub-range requests for reasons related to recovering byte-range locking state after a server failure. As discussed in Section 8.4.2, the server may use certain optimizations during recovery. Those only work effectively when the client's behavior during lock recovery is similar to its locking behavior before the server failure 🔧.

9.3. Upgrading and Downgrading Locks 🦋

If a client has a WRITE_LT lock on a byte range, it can ask for an atomic downgrade of that lock to a READ_LT lock. It does this through the LOCK operation by setting the type to READ_LT 💫. If the server supports atomic downgrade, the request succeeds. If not, it returns NFS4ERR_LOCK_NOTSUPP. The client should be prepared to receive this error and, if appropriate, report it to the requesting application 🥹.

If a client has a READ_LT lock on a byte range, it can ask for an atomic upgrade of that lock to a WRITE_LT lock. It does this through the LOCK operation by setting the type to WRITE_LT or WRITEW_LT 🚀. If the server does not support atomic upgrade, it returns NFS4ERR_LOCK_NOTSUPP. If the upgrade can be achieved without an existing conflict, the request succeeds. Otherwise, the server returns either NFS4ERR_DENIED or NFS4ERR_DEADLOCK. The error NFS4ERR_DEADLOCK gets returned if the client sent the LOCK operation with the type set to WRITEW_LT and the server has detected a deadlock 🌀. The client should be prepared to receive these errors too and, if appropriate, report them to the requesting application 💅.

9.4. Stateid Seqid Values and Byte-Range Locks 🔢

When a LOCK or LOCKU operation is performed, the stateid it returns has the same "other" value as the argument's stateid. It also has a "seqid" value that gets incremented, relative to the argument's stateid, to reflect that the LOCK or LOCKU happened 📈. The server MUST increment the value of the "seqid" field whenever there's any change to the locking status of any byte offset described by any of the locks covered by the stateid. A change in locking status includes a change from locked to unlocked (or the reverse), or a change from being locked for READ_LT to being locked for WRITE_LT (or the reverse) 🔒.

When there's no such change, like when a range already locked for WRITE_LT gets locked again for WRITE_LT, the server MAY increment the "seqid" value 🤷‍♀️.

9.5. Issues with Multiple Open-Owners 🧩

okay so when the same file gets opened by multiple open-owners, a client ends up with multiple OPEN stateids for that one file, and each one is tied to a different open-owner 🦋 that means you can get several LOCK and LOCKU requests for the same lock-owner, sent through different OPEN stateids. so there can be a bunch of stateids all representing byte-range locks on the same file, all held by the same lock-owner, but each one attached to a different open-owner. bestie it's a whole situation 💀

here's the vibe check: the locking status of each byte (is it locked, is it READ_LT or WRITE_LT, and which lock-owner is holding it) MUST reflect the last LOCK or LOCKU operation done for that lock-owner, no matter which stateid the request came through 👑

when a byte is locked by that lock-owner, the open-owner the byte-range lock is assigned to SHOULD be the open-owner tied to the stateid used for the last LOCK on that byte 💅 when the open-owner tied to locks on a stateid changes because of a LOCK or LOCKU done through it, the seqid field of that stateid MUST be incremented, even if the lock-owners haven't changed ✨ but when the set of locked bytes moves over to a different stateid for the same lock-owner (so a different open-owner), that other stateid's seqid value MUST NOT be incremented 🙈

9.6. Blocking Locks 🚦

some clients need blocking locks. NFSv4.1 gives you a callback when a lock you wanted becomes free, but that's an OPTIONAL feature, so clients can't count on it being there 😭 clients need to be ready to keep polling for the lock, which is kind of a fairness problem. two of the lock types, READW_LT and WRITEW_LT, basically tell the server "hey, I want a blocking lock" 🌸 when the callback isn't used, the server should maintain an ordered list of pending blocking locks. when the conflicting lock gets released, the server may wait for a period equal to lease_time for the first waiting client to re-request the lock. once that lease period runs out, the next waiting client's request is allowed the lock. clients are required to poll at an interval small enough that they're likely to get the lock in a timely way. the server isn't required to keep a list of pending blocked locks, since that list is there for fairness and not for correct operation 🧚 because crash recovery is unordered, you'd need to store lock state in stable storage to guarantee ordered granting of blocking locks.

servers may also note the lock types and delay returning a denial of the request, giving a conflicting lock extra time to be released so the request can succeed. that way clients avoid the burden of pointless, super frequent polling for blocking locks 🙌 the server should be careful about how long that delay is in case the client retransmits the request.

If a server gets a blocking LOCK, denies it, and then later gets a nonblocking request for the same lock that's also denied, it should take that lock off its list of pending blocking locks 🗑️ clients should send that nonblocking request to say "this is the last time I'm polling for this lock," like when the process asking for it gets interrupted. it's a courtesy so the server doesn't wait a whole lease period for nothing before granting other LOCK operations 🫶 clients aren't required to do this courtesy, and servers must not depend on them doing it. also, clients have to be ready for the chance that this final request gets accepted 😳

when a server sets the flag OPEN4_RESULT_MAY_NOTIFY_LOCK to say that CB_NOTIFY_LOCK callbacks might be done for the current open file, the client should take note 🔔 but since it's just a hint, it can't rely on a CB_NOTIFY_LOCK always showing up. a client can reasonably poll less often for a denied lock, since the extra latency is likely wiped out by a prompt callback, but it still needs to poll. when it gets a CB_NOTIFY_LOCK, it should promptly try to get the lock, but it should remember that other clients may be polling too, and the server has no obligation to hold the lock for that particular client 🏃‍♀️

a share reservation is a way to control access to a file. it's separate and independent from byte-range locking, so don't mix them up 🙅 when a client opens a file, it sends an OPEN operation to the server saying what kind of access it needs (READ, WRITE, or BOTH) and what access to deny everyone else (OPEN4_SHARE_DENY_NONE, OPEN4_SHARE_DENY_READ, OPEN4_SHARE_DENY_WRITE, or OPEN4_SHARE_DENY_BOTH). if the OPEN fails, the client fails the application's open request 💔

pseudo-code for the semantics 🧠:

           if (request.access == 0) {
             return (NFS4ERR_INVAL)
           } else {
             if ((request.access & file_state.deny)) ||
                (request.deny & file_state.access)) {
               return (NFS4ERR_SHARE_DENIED)
           }
           return (NFS4ERR_OK);

when the server does this share reservation check on OPEN, the current file_state includes bits that reflect all current opens, including the ones for the open-owner making the new OPEN request 🌈

the constants used for the access and deny fields of the OPEN and OPEN_DOWNGRADE operations look like this 💖:

   const OPEN4_SHARE_ACCESS_READ   = 0x00000001;
   const OPEN4_SHARE_ACCESS_WRITE  = 0x00000002;
   const OPEN4_SHARE_ACCESS_BOTH   = 0x00000003;

   const OPEN4_SHARE_DENY_NONE     = 0x00000000;
   const OPEN4_SHARE_DENY_READ     = 0x00000001;
   const OPEN4_SHARE_DENY_WRITE    = 0x00000002;
   const OPEN4_SHARE_DENY_BOTH     = 0x00000003;

9.8. OPEN/CLOSE Operations 🔐

to get correct share semantics, a client MUST use the OPEN operation to get the initial filehandle and say what access it wants and what access, if any, to deny 💅 even if the client wants a special stateid for anonymous state or READ bypass, it still has to get the filehandle for the regular file with OPEN, so the right share semantics can be applied. clients that don't have a deny mode built into their programming interfaces for opening a file should request a deny mode of OPEN4_SHARE_DENY_NONE.

the OPEN operation with the CREATE flag also takes over the CREATE operation for regular files from earlier NFS versions. that means a create with a share can happen atomically ⚡

the CLOSE operation removes all share reservations held by the open-owner on that file. if byte-range locks are held, the client SHOULD release all locks before sending a CLOSE operation. the server MAY free all outstanding locks on CLOSE, but some servers may not support closing a file that still has byte-range locks held. the server MUST return failure, NFS4ERR_LOCKS_HELD, if any locks would exist after the CLOSE 🚫

the LOOKUP operation returns a filehandle without setting up any lock state on the server. without a valid stateid, the server assumes the client has the least access 🫣 for example, if one client opened a file with OPEN4_SHARE_DENY_BOTH and another client accesses the file through a filehandle from LOOKUP, the second client could only read the file using the special read bypass stateid. the second client could not WRITE the file at all, because it wouldn't have a valid stateid from OPEN and the special anonymous stateid wouldn't be allowed access 🙅‍♀️

9.9. Open Upgrade and Downgrade ⬆️⬇️

when an OPEN is done for a file and the open-owner it's for already has the file open, it upgrades the open file status on the server to include the access and deny bits from the new OPEN plus the ones from the existing OPEN 🆙 the result: as far as the protocol's concerned there's one open file, and it has the union of the access and deny bits for all the OPEN requests that completed. that open is represented by a single stateid whose "other" value matches the original open's, and whose seqid value gets incremented to show the upgrade happened. the increment is required even when the upgrade doesn't change the open mode (like an OPEN for read when the existing open is already OPEN4_SHARE_ACCESS_BOTH) 🤷 only a single CLOSE is needed to reset the effects of both OPENs. the client can use the stateid returned by the OPEN that did the upgrade, or a stateid with the same "other" field and a seqid of zero, but care needs to be taken with upgrades that happen while the CLOSE is still pending 👀 also, the client might not know, when it sends the OPEN, that the same file is actually being opened. the above only applies if both OPENs result in the OPENed object being designated by the same filehandle 💫

when the server chooses to export multiple filehandles for the same file object and returns different filehandles on two different OPENs of that file object, the server MUST NOT "OR" together the access and deny bits and coalesce the two open files 🚫 instead, the server must maintain separate OPENs with separate stateids, and it will need separate CLOSEs to free them.

when multiple open files on the client get merged into a single OPEN file object on the server, closing one of the client's open files may mean the access and deny status of that open file on the server has to change. that's because the union of the access and deny bits for the remaining opens may be smaller (a proper subset) than before. OPEN_DOWNGRADE is how you make that change, and the client should use it to update the server so other clients' share reservation requests get handled properly 🪄 the stateid returned has the same "other" field as the one passed to the server. the "seqid" value in the returned stateid MUST be incremented, even when the access and deny bits for the file don't change at all ✨💖✨

9.10. Parallel OPENs ✨

Okay bestie, this one is kinda wild 😭. Unlike NFSv4.0, where OPEN operations for the same open-owner are inherently serialized because of the owner-based seqid 🔒, multiple OPENs for the same open-owner may be done in parallel 🏃‍♀️💨. And when clients do this, they can end up in a situation where, because of hard links, two OPEN operations turn out to open the same file 🔗. A later OPEN is then an upgrade of the first one, and the client only finds this out once the operations finish 🙈.

So how do clients figure out the order? They look at the stateids the OPENs gave back 👀. Stateids that share a common value of the other field have opened the same file 🫶. The order of the operations can be figured out from the seqid fields, modulo any possible wraparound of the 32-bit field 🔄. Periodt 💅.

Now here's the tea ☕. If the client might send multiple OPENs for the same open-owner in parallel, an open upgrade could happen without the client ever knowing it was coming 😳. Because of that, CLOSEs and OPEN_DOWNGRADEs should generally be sent with a non-zero seqid in the stateid 🌸. That way, the status change from an open upgrade doesn't get accidentally lost 🙌.

9.11. Reclaim of Open and Byte-Range Locks 🔁

Sometimes the server crashes, and then you need to get your stuff back 🥺. That's when the special forms of the LOCK and OPEN operations come in 🧚. They exist so you can re-establish byte-range locks or opens after a server failure 💖.

To reclaim existing opens, you do an OPEN operation using CLAIM_PREVIOUS 🎀. The client already opened the file and already has the filehandle for it, so this operation needs the current filehandle to be the target file, not a directory, and no file name gets specified 📄.
To reclaim byte-range locks, you use a LOCK operation with the reclaim parameter set to true 🔐.

Reclaims of opens tied to delegations get their own discussion in Section 10.2.1 🦋.

10. Client-Side Caching 💜

Okay so client-side caching of data, file attributes, and file names is basically essential if you want good performance with NFS 🚀. Distributed cache coherence is a seriously hard problem 😵‍💫, and previous NFS versions didn't even try 🙃. Instead, NFS client implementers came up with a bunch of techniques to make the lack of coherence hurt less for users 🫠. The thing is, earlier protocol specs never really defined these techniques, so it was often unclear what counted as valid or invalid client behavior 🤷‍♀️.

NFSv4.1 uses a lot of the same techniques as previous versions ✨. It still does not provide distributed cache coherence 🙅‍♀️. What it does do is define a more limited set of caching guarantees, so locks and share reservations can be used without destructive interference from client-side caching 🔒💅.

NFSv4.1 also introduces a delegation mechanism 🎁, which lets a bunch of decisions that would normally be made by the server get made locally by clients 🧠. This is super efficient for the common cases where sharing is infrequent or where sharing is read-only 🌈.

10.1. Performance Challenges for Client-Side Caching 🏎️

Caching techniques from previous NFS versions were honestly really successful at giving good performance 👑. BUT, several scalability challenges can show up when those techniques are used with a huge number of clients 😬. It gets especially bad when clients are geographically spread out, because that classically increases the latency for cache revalidation requests 🌍.

Previous NFS versions repeat their file data cache validation requests every time the file gets opened 🙄. That can cause serious performance drawbacks 📉. The most common case is when a file is only accessed by a single client 🧍‍♀️, so sharing is infrequent 💤.

In that case, asking the server again and again to confirm that no conflicts exist is just expensive 💸. A better option is to let a client that keeps opening a file do so without checking with the server at all 🙌. The client keeps doing this until potentially conflicting operations from another client actually happen ⚡.

A similar thing happens with byte-range locking 🔐. Sending LOCK and LOCKU operations, plus the READ and WRITE operations needed to keep data caching consistent with the locking semantics (see Section 10.3.2), can seriously limit performance 🐌. When locking is used to protect against infrequent conflicts, the penalty is huge 😩, and that can discourage applications from using byte-range locking at all 💔.

NFSv4.1 provides more aggressive caching strategies with these design goals 🎯:

Compatibility with a large range of server semantics 🤝.
Providing the same caching benefits as previous versions of the NFS protocol when unable to support the more aggressive model 🛟.
Requirements for aggressive caching are organized so that a large portion of the benefit can be obtained even when not all of the requirements can be met 💪.

The appropriate requirements for the server are covered in later sections where specific forms of caching are discussed (see Section 10.4) 📚.

10.2. Delegation and Callbacks 📬

Okay this part is SO good 🥹. A recallable delegation of server responsibilities for a file to a client improves performance, because it avoids repeated requests to the server when there's no inter-client conflict 🏆. The server uses a "callback" RPC to talk back to the client, and when another client starts sharing a delegated file, the server recalls those delegated responsibilities 📞.

A delegation gets passed from the server to the client and specifies the object of the delegation and the type of delegation 🎀. There are different types, but each one has a stateid that represents the delegation whenever operations depend on it 🪪. This stateid is similar to the ones tied to locks and share reservations, but it's different because the stateid for a delegation is associated with a client ID and can be used on behalf of all the open-owners for that client 👯‍♀️. The delegation goes to the client as a whole, not to any specific process or thread inside it 🧵.

The backchannel is set up by CREATE_SESSION and BIND_CONN_TO_SESSION, and the client is required to maintain it 🛠️. The backchannel can be down, even temporarily, so correct protocol operation does not depend on it 🫧. Preliminary testing of backchannel functionality can be done with a CB_COMPOUND procedure containing a single operation, CB_SEQUENCE, to check that the backchannel is continuous 🔍. A server avoids delegating responsibilities until it has determined that the backchannel exists 🧐. Since granting a delegation is always conditional on the absence of conflicting access, clients MUST NOT assume that a delegation will be granted 🚫, and they MUST always be prepared for OPENs, WANT_DELEGATIONs, and GET_DIR_DELEGATIONs to be processed without any delegations being granted 🫠. Be ready, bestie 💅.

Unlike locks, an operation by a second client on a delegated file will cause the server to recall a delegation through a callback 📣. For individual operations, the IMPLEMENTATION sections describe when such operations are required to effect a recall 📖. A few points are worth noting, though 👇:

The server is free to recall a delegation whenever it feels it is desirable, and it may do so even if no operations requiring recall are being done 🎲.
Operations done outside the NFSv4.1 protocol, for example through access by other protocols or by local access, also need to result in delegation recall when they make analogous changes to file system data 🗂️. What's crucial is whether the change would invalidate the guarantees the delegation provides 🛡️. When that's possible, the delegation needs to be recalled and MUST be returned or revoked before the operation is allowed to proceed 🛑.
The semantics of the file system are crucial in deciding when delegation recall is required 🧩. If a particular change within a specific implementation causes a change to a file attribute, then delegation recall is required, even if that operation was never specifically listed as requiring it 🫢. Again, the critical question is whether the guarantees the delegation provides are being invalidated ⚠️.

Despite those caveats, the implementation sections for a bunch of operations describe situations where delegation recall would be required under some common circumstances 📝:

For GETATTR, see Section 18.7.4.
For OPEN, see Section 18.16.4.
For READ, see Section 18.22.4.
For REMOVE, see Section 18.25.4.
For RENAME, see Section 18.26.4.
For SETATTR, see Section 18.30.4.
For WRITE, see Section 18.32.4.

When a recall happens, the client holding the delegation needs to flush modified state (like modified data) to the server and return the delegation 📦. The conflicting request won't be acted on until the recall is complete ⏳. The recall counts as complete when the client returns the delegation, or when the server times out waiting for the delegation to come back and revokes it because of that timeout ⏰. In the meantime, the server will either delay responding to conflicting requests or respond to them with NFS4ERR_DELAY 😤. Once the recall is resolved, the server has the info it needs to grant or deny the second client's request ✅.

When a client gets a delegation recall, it might be holding a ton of state that needs flushing to the server 🌊. So the server should allow enough time for the delegation to be returned, since that might take a bunch of RPCs to the server 🕰️. If the server can tell that the client is diligently flushing state as a result of the recall, the server may extend the usual time allowed for a recall 💖. Still, the time allowed for recall completion should not be unbounded 🚧.

An example of this is when responsibility for mediating opens on a given file has been delegated to a client (see Section 10.4) 🧾. The server won't know what opens are in effect on the client 🫥. Without that knowledge, the server can't determine whether the access and deny states for the file allow any particular open until the delegation for the file has been returned 😵.

A client failure or a network partition can result in a failure to respond to a recall callback 📵. In that case, the server will revoke the delegation, and that will render any modified state still on the client useless 💀. Yikes, bestie 😭.

10.2.1. Delegation Recovery 🦋

okay so bestie, delegation recovery is about what happens when things go sideways 😭 and there are three situations we need to handle:

client restart 🔄
server restart 🖥️
network partition (full or backchannel-only) 🌐

so if the client restarts, a failed lease renewal means byte-range locks and share reservations get revoked 💀 but delegations? they can be treated a bit differently ✨

there are totally situations where delegations need to be re-established after a client restarts 🫶 because the client might have file data stored locally that was tied to the delegations it held. so the client has to re-establish that file state on the server, no cap 💅

to make this possible, the server MAY extend the period for delegation recovery past the usual lease expiration period 🌸 which means requests from other clients that conflict with these delegations have to wait ⏳ and honestly other clients need to be ready for those delays, because recalling a delegation can take real time while the client flushes changed state to the server 🙈 this longer window gives clients time to restart and check stable storage so the delegations can be reclaimed 💖 for OPEN delegations, you reclaim them with OPEN using a claim type of CLAIM_DELEGATE_PREV or CLAIM_DELEG_PREV_FH (check Sections 10.5 and 18.16 for the OPEN delegation discussion and the OPEN details) 🎀

A server MAY support the claim types CLAIM_DELEGATE_PREV and CLAIM_DELEG_PREV_FH, and if it does, it MUST NOT remove delegations upon a CREATE_SESSION that confirms a client ID created by EXCHANGE_ID. Instead, the server MUST, for a period of time no less than the value of the lease_time attribute, maintain the client's delegations so the client has time to send CLAIM_DELEGATE_PREV and/or CLAIM_DELEG_PREV_FH requests. A server that supports CLAIM_DELEGATE_PREV and/or CLAIM_DELEG_PREV_FH MUST support the DELEGPURGE operation. 🚨

when the server restarts, delegations get reclaimed (using OPEN with CLAIM_PREVIOUS) kinda like byte-range locks and share reservations 🧚 but there's a slight semantic twist ✨ normally, if the server decides a delegation shouldn't be granted, it just does the requested action (like OPEN) and grants no delegation 🙅‍♀️ but for reclaim, the server DOES grant the delegation, except it gets a special designation so the client treats it as granted but already recalled by the server 😳 so the client has the duty to write all modified state back to the server and then return the delegation 📦

this whole reclaim process balances three principles of NFSv4.1 💜

upon reclaim, a client reporting resources it got from an earlier server instance MUST be granted those resources 👑
the server has unquestionable authority to decide whether delegations are granted and, once granted, whether they continue 💅
callbacks shouldn't be depended upon until the client has proven it can receive them 📞

when a client needs to reclaim a delegation and there's no associated open, it can use the CLAIM_PREVIOUS variant of the WANT_DELEGATION operation 🌈 but the server isn't required to support that operation, so the alternative is a dummy OPEN together with the delegation using an OPEN of type CLAIM_PREVIOUS 🍓 the dummy open file can then be released with a CLOSE, which re-establishes the original state being reclaimed: a delegation without an associated open ✨

when a client has more than one open tied to a delegation, the extra opens get set up using OPEN operations of type CLAIM_DELEGATE_CUR 🦋 and when those are used to establish opens for reclaimed delegations, the server MUST allow them when they're made within the grace period 💖

when a network partition happens, delegations are subject to being freed by the server once the lease renewal period expires 🌐 which is similar to how locks and share reservations behave 💅 for delegations though, the server may extend the period in which conflicting requests are held off 🫶 eventually, a conflicting request from another client will cause the delegation to be revoked ⚡ and a loss of the backchannel (say, from a later network config change) has the same effect 😭 a recall request will fail and revocation will result 💀

a client usually finds out a delegation was revoked when it uses a stateid tied to a delegation and gets one of these errors: NFS4ERR_EXPIRED, NFS4ERR_ADMIN_REVOKED, or NFS4ERR_DELEG_REVOKED 🚨 it can also find out after a client restart, when it tries to reclaim a delegation and gets that same error 🙈 heads up: with a revoked OPEN_DELEGATE_WRITE delegation there are issues, because data may have been modified by the client whose delegation got revoked AND separately by other clients 😬 see Section 10.5.1 for that discussion ✨ also, when delegations are revoked, the server writes information about the revoked delegation to stable storage (see Section 8.4.3) 🧠 this covers the case where a server restarts after revoking a delegation but before the client holding it is told about the revocation 💖

10.3. Data Caching 🌸

okay so when applications share access to a set of files, they have to be built to handle the possibility of conflicting access from another application 👀 and that's true whether those apps run on different clients or on the same client 💅

share reservations and byte-range locks are the tools NFSv4.1 gives applications to coordinate access using mutual exclusion 🔐 so NFSv4.1 data caching has to be implemented in a way that doesn't break the assumptions people rely on when using those tools 🫶

10.3.1. Data Caching and OPENs 🎀

so to avoid breaking the sharing assumptions apps depend on, NFSv4.1 clients SHOULD NOT hand cached data to applications or modify it on an application's behalf when that wouldn't be valid to get or change via a READ or WRITE operation 🙅‍♀️

hold up, there's more 👀 in the absence of an OPEN delegation (see Section 10.4), two extra rules apply ✨ and these rules are actually followed in practice by a lot of NFSv3 clients 💖

first, cached data on a client MUST be revalidated after doing an OPEN 🔍 revalidating means the client fetches the change attribute from the server, compares it with the cached change attribute, and if they differ, declares the cached data (and the cached attributes) invalid 💀 this makes sure the data for the OPENed file still matches what's in the client's cache ✨ this validation MUST be done at least when the client's OPEN includes a deny of OPEN4_SHARE_DENY_WRITE or OPEN4_SHARE_DENY_BOTH 🚨 that ends a window where other clients may have opened the file with OPEN4_SHARE_ACCESS_WRITE/OPEN4_SHARE_ACCESS_BOTH access 😳 clients MAY choose to revalidate more often (like at OPENs with a deny mode of OPEN4_SHARE_DENY_NONE) to match NFSv3's practice for users who expect that level of cache revalidation 🌈

okay but here's the tea ☕ the change attribute updates for both data AND metadata modifications, so some implementors might be tempted to use time_modify instead, so metadata changes don't spuriously invalidate clean data 🙈 the original says the implementor is cautioned about this approach 🫢 the change attribute is guaranteed to change for each update to the file, whereas time_modify is only guaranteed to change at the granularity of the time_delta attribute ⏱️ using time_modify and not change for cache validation risks the client incorrectly marking stale data as valid 💀 so any cache validation approach by the client MUST include the use of the change attribute 🔥

second, modified data MUST be flushed to the server before closing a file that was OPENed for OPEN4_SHARE_ACCESS_WRITE 📦 this complements the first rule 💕 if the data isn't flushed at CLOSE, the revalidation done after the client OPENs a file can't do its job 😭 the other part is that the data must be committed to stable storage at the server before the client requests the CLOSE operation 🧱 in the case of a server restart and a CLOSEd file, it may not be possible to retransmit the data to be written to the file, which is why this requirement exists 🥺

End of part 48. Keep going next time! 🦋💖

10.3.2. Data Caching and File Locking 🔐✨

Okay bestie, so apps that choose byte-range locking instead of share reservations to keep file access from getting messy 🫶 get their own matching set of rules for client-side data caching 💅. These rules only work if your byte-range locking matches the actual READ and WRITE operations you run 🎀. A vibe-based "we all agree" convention doesn't count 🙄.

Here's an example. You've got a two-megabyte file 📂, and you split it into two one-megabyte ranges. Then you guard each range with a byte-range lock, one on byte zero and one on byte one. A WRITE_LT lock on byte zero means you get to READ and WRITE the first range 💖. A WRITE_LT lock on byte one means the second range is yours 🌸. As long as every app touching the file follows that convention, it works on a local file system ✨. Over NFSv4.1 though, it might not work unless clients skip data caching altogether 😭.

The rules for data caching in the byte-range locking world are:

First, when a client gets a byte-range lock for a particular byte-range, the data cache for that byte-range (if any cached data exists) must be revalidated. If the change attribute says the file may have been updated since that cached data was obtained, the client must flush or invalidate the cached data for the newly locked byte-range. A client might choose to invalidate all of the non-modified cached data it has for the file 💅, but the only thing correct operation requires is invalidating all of the data in the newly locked byte-range 🧠.
Second, before releasing a WRITE_LT lock for a byte-range, all modified data for that byte-range must be flushed to the server. The modified data must also be written to stable storage 💾.

Flushing data to the server and invalidating cached data have to match the actual byte-ranges locked or unlocked 🎯. Rounding up or down to fit client cache block boundaries causes problems if you're not super careful 😬. For example, writing a modified block when only half of it is inside the area being unlocked might change bytes outside the unlocked area 💀. Those bytes could be part of a byte-range locked by another client 🙈. Clients can dodge this by synchronously doing the parts of WRITE operations that overlap the initial or final portion that isn't a full block 🛡️. Invalidating a locked area that isn't a whole number of full buffer blocks is similar. If revalidation shows the client's data may not be valid, the client would have to read one or two partial blocks from the server 🧩.

The data written to the server before unlocking a byte-range must be written to stable storage on the server 💾🔒. The client can do this with synchronous writes, or with asynchronous writes followed by a COMMIT operation 🍓. This is required because retransmitting the modified data after a server restart might conflict with a lock held by another client 😱.

A client implementation may choose to accommodate applications that use byte-range locking in non-standard ways (like using a byte-range lock as a global semaphore) 🧠. It can do this by flushing more data to the server upon a LOCKU than the locked range covers 🌊. That can include modified data inside files other than the one being unlocked 📁. In that case, the client must not interfere with applications whose READs and WRITEs are only done within the bounds of byte-range locks that the application holds 🚧. For example, an application locks a single byte of a file and writes that single byte. A client that handles a LOCKU by flushing all modified data could validly write that single byte in response to an unrelated LOCKU operation ✅. It would not be valid to write the entire block containing that byte, though, since the block includes an unlocked area that might be locked by another client 🚫. Client implementations can avoid this by splitting files with modified data into two groups: files where every modification is inside an appropriate byte-range lock, and files with modifications not covered by any byte-range lock 🗂️. Any writes for the first group must not include areas that aren't locked and therefore weren't modified on the client 🎀.

10.3.3. Data Caching and Mandatory File Locking 🔒💅

Client-side data caching needs to respect mandatory byte-range locking when it's in effect 🫡. Mandatory locking is indicated when the client gets back NFS4ERR_LOCKED from a READ or WRITE operation on a file for which it holds an appropriate share reservation 🚨. When mandatory locking is in effect for a file, the client must check for an appropriate byte-range lock on the data being read or written 🔍. If a byte-range lock exists for the range being read or written, the client may satisfy the request from its validated cache ✨. If there's no appropriate byte-range lock for that range, the request must not be satisfied from the client's cache. It has to be sent to the server for processing 📨. When a read or write request partially overlaps a locked byte-range, the request should be split into multiple pieces, with each byte-range (locked or not) handled appropriately 🧩.

10.3.4. Data Caching and File Identity 🆔🦋

When clients cache data, the file data needs to be organized by the file system object it belongs to 📦. For NFSv3 clients, the usual practice has been to assume, for caching purposes, that distinct filehandles represent distinct file system objects 🗃️. The client can then choose to organize and maintain its data cache on that basis 💖.

NFSv4.1 changed things 👀. There's now the possibility of major deviations from a "one filehandle per object" model, because a filehandle may be built from the object's pathname 🛤️. So clients need a reliable way to figure out whether two filehandles designate the same file system object 🕵️‍♀️. If clients just assumed that all distinct filehandles denote distinct objects and cached data on that basis, caching inconsistencies would pop up between distinct client-side objects that map to the same server-side object 💀.

NFSv4.1 gives you a way to tell filehandles apart, which fixes a potential functional regression compared to NFSv3 🙌. Without this method, caching inconsistencies could happen within the same client, and earlier NFS versions never had that problem 😭. Inconsistencies can also happen with applications running on multiple clients, but that's not what's being addressed here 🙃.

For data caching purposes, the following steps let an NFSv4.1 client determine whether two distinct filehandles denote the same server-side object 🔎:

If GETATTR directed to two filehandles returns different values of the fsid attribute, then the filehandles represent distinct objects.
If GETATTR for any file with an fsid that matches the fsid of the two filehandles in question returns a unique_handles attribute with a value of TRUE, then the two objects are distinct.
If GETATTR directed to the two filehandles does not return the fileid attribute for both of the handles, then it cannot be determined whether the two objects are the same. Therefore, operations that depend on that knowledge (e.g., client-side data caching) cannot be done reliably. Note that if GETATTR does not return the fileid attribute for both filehandles, it will return it for neither of them, since the fsid for both filehandles is the same 🫠.
If GETATTR directed to the two filehandles returns different values for the fileid attribute, then they are distinct objects.
Otherwise, they are the same object 🎉.

10.4. Open Delegation 🦋

okay so bestie, this is where the server gets to be like "hey client, you can handle this one yourself 💅" 🎀 when a file is being OPENed, the server MAY delegate further handling of opens and closes for that file to the opening client ✨ but here's the thing: the delegation is recallable 🙈 since the circumstances that allowed it can totally change. like, if the server gets a conflicting OPEN from another client, it MUST recall the delegation before deciding whether that other OPEN can be granted 😭 no cap, that's the rule.

making a delegation is 100% up to the server 💜 so clients should not assume that any particular OPEN either will or will not get a delegation. the server has its own vibes. 🌸 here's the typical checklist servers might use to decide:

the client MUST be able to respond to the server's callback requests 🫶 if a backchannel has been established, the server will send a CB_COMPOUND request, containing a single operation, CB_SEQUENCE, for a test of backchannel availability. ✨
the client MUST have responded properly to previous recalls 👑
there MUST be no current OPEN conflicting with the requested delegation 🔥
there SHOULD be no current delegation that conflicts with the delegation being requested 🍓
the probability of future conflicting open requests SHOULD be low based on the recent history of the file 🧚
the existence of any server-specific semantics of OPEN/CLOSE that would make the required handling incompatible with the prescribed handling that the delegated client would apply (see below) 🙈

there are two types of OPEN delegations, and they are giving opposite energy 💖

OPEN_DELEGATE_READ lets a client handle, on its own, requests to open a file for reading that do not deny OPEN4_SHARE_ACCESS_READ access to others. multiple OPEN_DELEGATE_READ delegations MAY be outstanding simultaneously and they do not conflict 🦋
OPEN_DELEGATE_WRITE lets the client handle, on its own, all opens 👑 only one OPEN_DELEGATE_WRITE delegation MAY exist for a given file at a given time, and it is inconsistent with any OPEN_DELEGATE_READ delegations 💀

when a client has an OPEN_DELEGATE_READ delegation, it is assured that neither the contents, the attributes (with the exception of time_access), nor the names of any links to the file will change without its knowledge, so long as the delegation is held 🌈 it's literally the safest little bubble.

when a client has an OPEN_DELEGATE_WRITE delegation, it MAY modify the file data locally since no other client will be accessing the file's data ✨ the client holding that delegation MAY only locally affect the file attributes that are intimately connected with the file data: size, change, time_access, time_metadata, and time_modify 💅 all other attributes MUST be reflected on the server. no sneaky local edits to ownership stuff, bestie 🙅‍♀️

when a client has an OPEN delegation, it does not need to send OPENs or CLOSEs to the server 🧚 instead it can just update the status internally. but for an OPEN_DELEGATE_READ delegation, opens that cannot be handled locally (opens that are for OPEN4_SHARE_ACCESS_WRITE/OPEN4_SHARE_ACCESS_BOTH or that deny OPEN4_SHARE_ACCESS_READ access) MUST be sent to the server 😭

when an OPEN delegation is made, the reply to the OPEN contains an OPEN delegation structure that specifies the following:

the type of delegation (OPEN_DELEGATE_READ or OPEN_DELEGATE_WRITE).
space limitation information to control flushing of data on close (OPEN_DELEGATE_WRITE delegation only; see Section 10.4.1) 🎀
an nfsace4 specifying read and write permissions 🫶
a stateid to represent the delegation 🌸

the delegation stateid is separate and distinct from the stateid for the OPEN proper 💖 the standard stateid is associated with a particular lock-owner and will continue to be valid after the delegation is recalled and the file remains open. the delegation stateid, on the other hand, is its own thing. ✨

when a request internal to the client is made to open a file and an OPEN delegation is in effect, it gets accepted or rejected solely on the basis of these two conditions 👑

the access and deny bits for the request and the file as described in Section 9.7.
the read and write permissions as determined below.

any requirement for other checks to be made by the delegate SHOULD result in the OPEN delegation being denied so that the checks can be made by the server itself 🙈 this part is so important!

the nfsace4 passed with delegation can be used to avoid frequent ACCESS calls 🍓 the permission check goes like this:

if the nfsace4 indicates that the open may be done, then it SHOULD be granted without reference to the server. ✨
if the nfsace4 indicates that the open may not be done, then an ACCESS request MUST be sent to the server to obtain the definitive answer. 🦋

the server MAY return an nfsace4 that is more restrictive than the actual ACL of the file 🌈 this includes an nfsace4 that specifies denial of all access. ngl, some common practices like mapping the traditional user "root" to the user "nobody" (see Section 5.9) MAY make it incorrect to return the actual ACL of the file in the delegation response 💀

the use of a delegation together with other forms of caching creates the possibility that no server authentication and authorization will ever be performed for a given user, since all of the user's requests might be satisfied locally 😭 so where the client is depending on the server for authentication and authorization, the client SHOULD be sure authentication and authorization occurs for each user by use of the ACCESS operation 🫶 this SHOULD be the case even if an ACCESS operation would not be required otherwise. as mentioned before, the server MAY enforce frequent authentication by returning an nfsace4 denying all access with every OPEN delegation. 🔥

10.4.1. Open Delegation and Data Caching 💜

an OPEN delegation allows much of the message overhead associated with opening and closing files to be eliminated ✨ an open when an OPEN delegation is in effect does not require that a validation message be sent to the server. the continued endurance of the "OPEN_DELEGATE_READ delegation" provides a guarantee that no OPEN for OPEN4_SHARE_ACCESS_WRITE/OPEN4_SHARE_ACCESS_BOTH, and thus no write, has occurred 🦋 similarly, when closing a file opened for OPEN4_SHARE_ACCESS_WRITE/OPEN4_SHARE_ACCESS_BOTH and if an OPEN_DELEGATE_WRITE delegation is in effect, the data written does not have to be written to the server until the OPEN delegation is recalled. the continued endurance of the OPEN delegation provides a guarantee that no open, and thus no READ or WRITE, has been done by another client. 🌈

for the purposes of OPEN delegation, READs and WRITEs done without an OPEN are treated as the functional equivalents of a corresponding type of OPEN 🧚 although a client SHOULD NOT use special stateids when an open exists, delegation handling on the server can use the client ID associated with the current session to determine if the operation has been done by the holder of the delegation (in which case, no recall is necessary) or by another client (in which case, the delegation MUST be recalled and I/O not proceed until the delegation is returned or revoked). 👑

with delegations, a client is able to avoid writing data to the server when the CLOSE of a file is serviced 🎀 the file close system call is the usual point where the client is notified of a lack of stable storage for the modified file data generated by the application. at the close, file data is written to the server and, through normal accounting, the server is able to determine if the available file system space for the data has been exceeded (i.e., the server returns NFS4ERR_NOSPC or NFS4ERR_DQUOT). this accounting includes quotas. the introduction of delegations requires that an alternative method be in place for the same type of communication to occur between client and server. 💅

in the delegation response, the server provides either the limit of the size of the file or the number of modified blocks and associated block size 🫶 the server MUST ensure that the client will be able to write modified data to the server of a size equal to that provided in the original delegation. the server MUST make this assurance for all outstanding delegations 🔥 therefore, the server MUST be careful in its management of available space for new or modified data, taking into account available file system space and any applicable quotas. the server MAY recall delegations as a result of managing the available file system space. the client SHOULD abide by the server's state space limits for delegations. if the client exceeds the stated limits for the delegation, the server's behavior is undefined. 😭

based on server conditions, quotas, or available file system space, the server MAY grant OPEN_DELEGATE_WRITE delegations with very restrictive space limitations 🌸 the limitations MAY be defined in a way that will always force modified data to be flushed to the server on close. ✨

with respect to authentication, flushing modified data to the server after a CLOSE has occurred MAY be problematic 🙈 for example, the user of the application may have logged off the client, and unexpired authentication credentials may not be present. in this case, the client MAY need to take special care to ensure that local unexpired credentials will in fact be available. this MAY be accomplished by tracking the expiration time of credentials and flushing data well in advance of their expiration, or by making private copies of credentials to assure their availability when needed. 🔐💖

10.4.2. Open Delegation and File Locks 🎀

when a client holds an OPEN_DELEGATE_WRITE delegation, lock operations are performed locally 🦋 this includes those required for mandatory byte-range locking. this can be done since the delegation implies that there can be no conflicting locks. similarly, all of the revalidations that would normally be associated with obtaining locks and the flushing of data associated with the releasing of locks need not be done. ✨

when a client holds an OPEN_DELEGATE_READ delegation, lock operations are not performed locally 😭 all lock operations, including those requesting non-exclusive locks, MUST be sent to the server for resolution. 👑

10.4.3. Handling of CB_GETATTR ✨

okay so the server needs special handling for a GETATTR when the target file has an OPEN_DELEGATE_WRITE delegation in effect 👀 the client holding that delegation may have changed the data, and the server has to show that change to the second client that sent the GETATTR. so the server has to go ask the delegation holder first, using the CB_GETATTR operation 💅 ngl, the only attributes the server can reliably query with CB_GETATTR are size and change.

Since CB_GETATTR is only there to answer someone else's GETATTR, the server just needs to know whether the delegation holder has a modified version of the file 🫶 If the holder's copy isn't modified (data or size), the server can answer from the attributes it already has stored locally. If it is modified, the server only needs to know that much, and then it responds to the second client as if the file had been modified locally at the server 🔥

The change attribute is chosen by the server and is opaque to the client, so the two of them need to agree on a way to say "this file is modified" 🦋 For size, the client just reports its current view of the file size. For change, it's a little more involved, so let's break it down.

For the client, these are the steps when it receives an OPEN_DELEGATE_WRITE delegation:

The value of change is obtained from the server and cached. Let this value be represented by c.
The client creates a value greater than c that it uses to communicate that modified data is held at the client. Let this value be represented by d.
When the client is queried via CB_GETATTR for the change attribute, it checks whether it holds modified data. If the file is modified, the value d is returned for the change attribute value. If the file is not currently modified, the client returns the value c for the change attribute.

For simplicity of implementation, the client MAY for each CB_GETATTR return the same value d 🌸 This is true even if, between successive CB_GETATTR operations, the client modifies the file's data or metadata in its cache again. The client can return the same value because the only requirement is that it be able to indicate to the server that it holds modified data. Therefore, the value of d may always be c + 1.

While the change attribute is opaque to the client in the sense that it has no idea what units of time, if any, the server is counting change with, it is not opaque in that the client has to treat it as an unsigned integer, and the server has to be able to see the results of the client's changes to that integer. Therefore, the server MUST encode the change attribute in network order when sending it to the client. The client MUST decode it from network order to its native order when receiving it, and the client MUST encode it in network order when sending it to the server. For this reason, change is defined as an unsigned integer rather than an opaque array of bytes 🧚

For the server, these are the steps when it provides an OPEN_DELEGATE_WRITE delegation:

Upon providing an OPEN_DELEGATE_WRITE delegation, the server caches a copy of the change attribute in the data structure it uses to record the delegation. Let this value be represented by sc.
When a second client sends a GETATTR operation on the same file to the server, the server obtains the change attribute from the first client. Let this value be cc.
If the value cc is equal to sc, the file is not modified, and the server returns the current values for change, time_metadata, and time_modify (for example) to the second client.
If the value cc is NOT equal to sc, the file is currently modified at the first client and most likely will be modified at the server at a future time. The server then uses its current time to construct attribute values for time_metadata and time_modify. A new value of sc, which we will call nsc, is computed by the server, such that nsc >= sc + 1. The server then returns the constructed time_metadata, time_modify, and nsc values to the requester. The server replaces sc in the delegation record with nsc. To prevent the possibility of time_modify, time_metadata, and change from appearing to go backward (which would happen if the client holding the delegation fails to write its modified data to the server before the delegation is revoked or returned), the server SHOULD update the file's metadata record with the constructed attribute values. For reasons of reasonable performance, committing the constructed attribute values to stable storage is OPTIONAL.

As discussed earlier in this section, the client MAY return the same cc value on subsequent CB_GETATTR calls, even if the file was modified in the client's cache yet again between successive CB_GETATTR calls. Therefore, the server has to assume that the file has been modified yet again, and MUST take care to ensure that the new nsc it constructs and returns is greater than the previous nsc it returned. An example implementation's delegation record would satisfy this mandate by including a boolean field (let us call it "modified") that is set to FALSE when the delegation is granted, and an sc value set at the time of grant to the change attribute value. The modified field would be set to TRUE the first time cc != sc, and would stay TRUE until the delegation is returned or revoked. The processing for constructing nsc, time_modify, and time_metadata would use this pseudo code:

    if (!modified) {
        do CB_GETATTR for change and size;

        if (cc != sc)
            modified = TRUE;
    } else {
        do CB_GETATTR for size;
    }

    if (modified) {
        sc = sc + 1;
        time_modify = time_metadata = current_time;
        update sc, time_modify, time_metadata into file's metadata;
    }

This would return to the client (that sent GETATTR) the attributes it requested, but make sure size comes from what CB_GETATTR returned. The server would not update the file's metadata with the client's modified size.

In the case that the file attribute size is different than the server's current value, the server treats this as a modification regardless of the value of the change attribute retrieved via CB_GETATTR, and responds to the second client as in the last step. 🙈

This methodology resolves issues of clock differences between client and server and other scenarios where the use of CB_GETATTR breaks down 🌈 It should be noted that the server is under no obligation to use CB_GETATTR, and therefore the server MAY simply recall the delegation to avoid its use 💜

10.4.4. Recall of Open Delegation ✨

okay so these are the moments when the server is gonna recall an OPEN delegation, iykyk 💅

potentially conflicting OPEN request (or a READ or WRITE operation done with a special stateid)
SETATTR sent by another client
REMOVE request for the file
RENAME request for the file as either the source or target of the RENAME

Whether a RENAME of a directory in the path leading to the file sets off a recall of an OPEN delegation depends on the semantics of the server's file system 🌸 If that file system denies such RENAMEs when a file is open, the recall must be performed so the server can figure out whether the file is actually open, bestie.

In addition to the situations above, the server may choose to recall OPEN delegations at any time if resource constraints make it advisable to do so. Clients should always be prepared for the possibility of recall 🫶

When a client gets a recall for an OPEN delegation, it needs to update state on the server before returning the delegation. These same updates must be done whenever a client chooses to return a delegation voluntarily. Here's the state that needs to be dealt with:

If the file associated with the delegation is no longer open and no previous CLOSE operation has been sent to the server, a CLOSE operation must be sent to the server.
If a file has other open references at the client, then OPEN operations must be sent to the server. The appropriate stateids will be provided by the server for subsequent use by the client, since the delegation stateid will no longer be valid. These OPEN requests are done with the claim type of CLAIM_DELEGATE_CUR. This lets the client present the delegation stateid so it can establish the appropriate rights to perform the OPEN. (See Section 18.16, which describes the OPEN operation, for details.)
If there are granted byte-range locks, the corresponding LOCK operations need to be performed. This applies to the OPEN_DELEGATE_WRITE delegation case only.
For an OPEN_DELEGATE_WRITE delegation, if at the time of recall the file is not open for OPEN4_SHARE_ACCESS_WRITE/OPEN4_SHARE_ACCESS_BOTH, all modified data for the file must be flushed to the server. If the delegation had not existed, the client would have done this data flush before the CLOSE operation.
For an OPEN_DELEGATE_WRITE delegation when a file is still open at the time of recall, any modified data for the file needs to be flushed to the server.
With the OPEN_DELEGATE_WRITE delegation in place, it is possible that the file was truncated during the duration of the delegation. For example, the truncation could have happened as a result of an OPEN UNCHECKED with a size attribute value of zero. Therefore, if a truncation of the file has occurred and this operation has not been propagated to the server, the truncation must occur before any modified data is written to the server.

In the case of an OPEN_DELEGATE_WRITE delegation, byte-range locking adds some extra requirements 🎀 To precisely maintain the associated invariant, it is required to flush any modified data in any byte-range for which a WRITE_LT lock was released while the OPEN_DELEGATE_WRITE delegation was in effect. However, because the OPEN_DELEGATE_WRITE delegation implies no other locking by other clients, a simpler implementation is to flush all modified data for the file (as described just above) if any WRITE_LT lock has been released while the OPEN_DELEGATE_WRITE delegation was in effect.

An implementation doesn't need to wait until delegation recall (or the decision to voluntarily return a delegation) to perform any of the above actions, if implementation considerations (e.g., resource availability constraints) make that desirable. Generally, though, the fact that the actual OPEN state of the file may keep changing makes it not worthwhile to send information about opens and closes to the server, except as part of delegation return. The exception is when the client has no more internal opens of the file. In that case, sending a CLOSE is useful because it reduces resource utilization on the client and server ✨ Regardless of the client's choices on scheduling these actions, all of them must be performed before the delegation is returned, including (when applicable) the close that corresponds to the OPEN that resulted in the delegation. These actions can be performed either in previous requests or in previous operations in the same COMPOUND request.

10.4.5. Clients That Fail to Honor Delegation Recalls 🙈

A client may fail to respond to a recall for various reasons, such as a failure of the backchannel from the server to the client. The client may be unaware of a failure in the backchannel. That lack of awareness could mean the client finds out long after the failure that its delegation has been revoked, and that another client has modified the data the client had a delegation for. This is especially a problem for the client that held an OPEN_DELEGATE_WRITE delegation 😭

Status bits returned by SEQUENCE operations help give the client an alternate way to hear about issues with the status of the backchannel and of recalled delegations. When the backchannel isn't available, the server returns the status bit SEQ4_STATUS_CB_PATH_DOWN on SEQUENCE operations. The client can react by trying to re-establish the backchannel, and by returning recallable objects if a backchannel can't be successfully re-established.

Whether the backchannel is functioning or not, the recalled delegation might still not be returned. Note that the client's lease might still be renewed, even though the recalled delegation isn't returned. In this situation, servers SHOULD revoke delegations that are not returned in a period of time equal to the lease period. That period of time should give the client time to notice the backchannel-down status and re-establish the backchannel.

When delegations are revoked, the server will return the SEQ4_STATUS_RECALLABLE_STATE_REVOKED status bit set on subsequent SEQUENCE operations. The client should note this and then use TEST_STATEID to find out which delegations have been revoked 👑

10.4.6. Delegation Revocation 💔

At the point a delegation is revoked, if there are associated opens on the client, those opens may or may not be revoked. If no byte-range lock or open that's inconsistent with the existing open is granted, the stateid for the open may remain valid and be disconnected from the revoked delegation, just like would happen if the delegation were returned.

For example, if an OPEN for OPEN4_SHARE_ACCESS_BOTH with a deny of OPEN4_SHARE_DENY_NONE is associated with the delegation, granting another such OPEN to a different client will revoke the delegation but doesn't need to revoke the OPEN, since the two OPENs are consistent with each other. On the other hand, if an OPEN denying write access is granted, then the existing OPEN must be revoked 🍓

When opens and/or locks are revoked, the applications holding them need to be notified. This notification usually happens by returning errors for READ/WRITE operations, or when a close is attempted for the open file.

If no opens exist for the file at the point the delegation is revoked, then notification of the revocation is unnecessary. However, if there is modified data present at the client for the file, the user of the application should be notified. Unfortunately, it may not be possible to notify the user, since active applications may not be present at the client. See Section 10.5.1 for additional details.

10.4.7. Delegations via WANT_DELEGATION 🧚

In addition to providing delegations as part of the reply to OPEN operations, servers MAY provide delegations separately from open, via the OPTIONAL WANT_DELEGATION operation. This lets delegations be obtained in advance of an OPEN that might benefit from them, for objects that aren't a valid target of OPEN, or to deal with cases where a delegation has been recalled and the client wants to try to re-establish it, if the absence of use by other clients allows that 💜

The WANT_DELEGATION operation may be performed on any type of file object other than a directory.

When a delegation is obtained using WANT_DELEGATION, any open files for the same filehandle held by that client are to be treated as subordinate to the delegation, just as if they had been created using an OPEN of type CLAIM_DELEGATE_CUR. Otherwise they stay unchanged as to seqid, access and deny modes, and the relationship with byte-range locks. Similarly, because existing byte-range locks are subordinate to an open, those byte-range locks also become indirectly subordinate to that new delegation.

The WANT_DELEGATION operation provides for delivery of delegations via callbacks, for when the delegations aren't immediately available. When a requested delegation is available, it is delivered to the client via a CB_PUSH_DELEG operation. When this happens, open files for the same filehandle become subordinate to the new delegation at the point the delegation is delivered, just as if they had been created using an OPEN of type CLAIM_DELEGATE_CUR. Similarly, this happens for existing byte-range locks subordinate to an open.

10.5. Data Caching and Revocation 🌈

When locks and delegations are revoked, the assumptions that successful caching depends on are no longer guaranteed. For any locks or share reservations that have been revoked, the corresponding state-owner needs to be notified. This notification includes applications with a file open that has a corresponding delegation that has been revoked. Cached data associated with the revocation must be removed from the client. In the case of modified data existing in the client's cache, that data must be removed from the client without being written to the server. As mentioned, the assumptions made by the client are no longer valid at the point when a lock or delegation has been revoked. For example, another client may have been granted a conflicting byte-range lock after the revocation of the byte-range lock at the first client. Therefore, the data within the lock range may have been modified by the other client. Obviously, the first client is unable to guarantee to the application what has happened to the file in the case of revocation 💀

Notifying a state-owner will in many cases just mean returning an error on the next and all subsequent READs/WRITEs to the open file, or on the close. Where the methods available to a client make such notification impossible because errors for certain operations may not be returned, more drastic action such as signals or process termination may be appropriate. The justification here is that an invariant an application depends on may be violated. Depending on how errors are typically handled in the client's operating environment, further levels of notification, including logging, console messages, and GUI pop-ups, may be appropriate.

10.5.1. Revocation Recovery for Write Open Delegation 💌

Okay bestie, this is the spicy one 😬. Revocation recovery for an OPEN_DELEGATE_WRITE delegation gets tricky because the client can be holding modified data in its cache while the file isn't open anymore. So any client that doesn't flush modified data to the server on every close has to make sure the user actually gets told when a revocation causes a failure 🚨. Some of these messes need a human to fix them, so notification schemes that reach the right user or administrator may be necessary. Logs and console messages are the classic examples 🫶.

If there's modified data on the client, it must not be flushed normally to the server 🚫. A client may try to save a copy of the file data as it was modified during the delegation, under a different name in the file system namespace, to make recovery easier 💾. Note that when the client can tell no other client modified the file, or when it has a complete cached copy of the file, that saved copy of its view can be super valuable 💅. But if recovery means mixing the client's cached data with the server's copy as other clients modified it, it's going to be anything but straightforward 💀. So clients may skip saving contents in those cases, or specially mark the results to warn users about possible problems ⭐.

Saving modified data in delegation revocation situations may be limited to files of a certain size, or only used when there's enough disk space on the target file system 🍓. It may also be restricted to when the client has enough buffering resources to keep the cached copy around until it's properly stored on the target file system 🧚.

10.6. Attribute Caching ✨

This section is all about caching a file's attributes on a client that does not hold a delegation on the file 🦋.

The attributes here don't include named attributes. Each named attribute is basically its own file, so caching its data works just like it does for ordinary files 📂. Also, LOOKUP results from an OPENATTR directory (plus the directory's contents) get cached on the same basis as any other pathnames 🗺️.

Clients may cache file attributes they got from the server and use them to skip later GETATTR requests 🙌. This caching is write through: changes to attributes always go through requests to the server. They should not happen locally, and they should not be cached. The exception is attribute changes that are tightly connected to data caching 💖. So if you extend a file by writing to the local data cache, the size on the client reflects it right away, even though the server hasn't heard about it yet 📏. Normally those changes aren't sent straight to the server, but when the modified data gets flushed, matching attribute changes are made on the server too. When OPEN delegation is in effect, the modified attributes may be sent back to the server in reaction to a CB_RECALL call 🔁.

The result? Attribute caches on different clients won't be coherent 😵‍💫. Changes made in one order on the server might be seen in a different order on one client, and in a third order on another 🌀.

File system APIs don't give you a way to atomically modify or check attributes for multiple files at the same time 🙃. These rules keep those potential incoherencies reasonably manageable, and they're derived from the practice of previous NFS protocols 📚.

All attributes for a given file (per-fsid fsid attributes excepted ✋) are cached as a unit at the client, so no non-serializability can sneak in within the context of a single file 🔒.
An upper time boundary is maintained on how long a client cache entry can be kept without being refreshed from the server ⏰.
When operations change attributes at the server, the updated attribute set gets requested as part of the containing RPC. This includes directory operations that update attributes indirectly. It's done by following the modifying operation with a GETATTR, then using the GETATTR results to update the client's cached attributes 🎀.

Note that if the full set of attributes to be cached is requested by READDIR, the results can be cached by the client on the same basis as attributes obtained via GETATTR 🫧.

To validate its cached attributes for a file, a client can fetch both change and time_access, and assume that if change has the same value it had when the attributes were cached, then no attributes other than time_access have changed 🕵️. Why fetch time_access too? Because many servers operate in environments where the operation that updates change does not update time_access. For example, POSIX file semantics do not update access time when a file is modified by the write system call [15] 🙈. So a client that wants a current time_access value should fetch it along with change during attribute cache validation and update its cached time_access.

The client may maintain a cache of modified attributes for those attributes intimately connected with data of modified regular files: size, time_modify, and change. Other than those three attributes, the client MUST NOT maintain a cache of modified attributes 🚫. Instead, attribute changes are immediately sent to the server 💨.

In some operating environments, the equivalent of time_access is implicitly updated by each read of the content of the file object. If an NFS client is caching the content of a file object, whether it's a regular file, directory, or symbolic link, the client SHOULD NOT update the time_access attribute (via SETATTR or a small READ or READDIR request) on the server with each read that is satisfied from cache 🙅‍♀️. Why? Because that can defeat the performance benefits of caching content, especially since an explicit SETATTR of time_access may alter the change attribute on the server. If change changes, clients caching the content will think the content has changed, and they'll re-read unmodified data from the server 😩. The client isn't encouraged to keep a modified version of time_access in its cache either. It would eventually have to write the access time to the server (with bad performance effects) or never update the server's time_access, which would mean an application caching access time between a close and open of the same file sees the access time oscillating between the past and present 🌊. The time_access attribute always means the time of last access to a file by a read that was satisfied by the server, so clients will tend to see only time_access changes that go forward in time ⏩.

10.7. Data and Metadata Caching and Memory Mapped Files 🗺️✨

Okay so some operating environments let an app map a file's content right into its own address space, which is giving direct-access main character energy 💅 Every time the app touches a memory location that corresponds to a block that hasn't been loaded yet, a page fault happens and the file gets read. (Or, if the block doesn't exist in the file, the block gets allocated and then instantiated in the app's address space 🧚)

As long as each memory-mapped access to the file requires a page fault, the attributes used to detect access and modification (time_access, time_metadata, time_modify, and change) will get updated. 🫶 However, in many operating environments, when page faults aren't required, these attributes will NOT get updated on reads or updates through memory access, whether the file is local or accessed remotely 🙈 A client or server MAY fail to update attributes of a file that is being accessed via memory-mapped I/O. This has a few implications:

If there's an application on the server that has memory mapped a file that a client is also accessing, the client may not be able to get a consistent value of the change attribute to figure out whether its cache is stale 😭 A server that knows the file is memory-mapped could always be pessimistic and return updated values for change, forcing the app to always get the freshest data and metadata for the file. However, due to the negative performance implications, such behavior is OPTIONAL. 🐢
If the memory-mapped file isn't being modified on the server, and is just being read by an app through the memory-mapped interface, the client won't see an updated time_access attribute. However, in many operating environments, neither will any process running on the server. So NFS clients are at no disadvantage compared to local processes 💖
If there's another client memory mapping the file, and that client is holding an OPEN_DELEGATE_WRITE delegation, the same set of issues from the previous two bullets apply 🌀 So when a server does a CB_GETATTR on a file the client has modified in its cache, the reply from CB_GETATTR won't necessarily be accurate. As discussed earlier, the client's obligation is to report that the file has been modified since the delegation was granted, NOT whether it has been modified again between successive CB_GETATTR calls. The server MUST assume that any file the client has modified in cache has been modified again between successive CB_GETATTR calls 👑 Depending on the client's memory management system, this weak obligation may not be possible. A client MAY return stale information in CB_GETATTR whenever the file is memory-mapped.
The mixture of memory mapping and byte-range locking on the same file is problematic 🚩 Consider the following scenario, where a page size on each client is 8192 bytes:
- Client A memory maps the first page (8192 bytes) of file X.
- Client B memory maps the first page (8192 bytes) of file X.
- Client A WRITE_LT locks the first 4096 bytes.
- Client B WRITE_LT locks the second 4096 bytes.
- Client A, via a STORE instruction, modifies part of its locked byte-range.
- Simultaneous to client A, client B executes a STORE on part of its locked byte-range.

Here the challenge is for each client to resynchronize and get a correct view of the first page 🥲 In many operating environments, the virtual memory management systems on each client only know that a page is modified, NOT that a subset of the page corresponding to each lock's byte-range has been modified. So it isn't possible for each client to do the right thing, which would be writing only the locked portion of the page to the server. For example, if client A simply writes out the page, and then client B writes out the page, client A's data is lost 💀

Moreover, if mandatory locking is enabled on the file, we have a different problem 😤 When clients A and B execute the STORE instructions, the resulting page faults require a byte-range lock on the entire page. Each client then tries to extend its locked range to the entire page, which results in a deadlock. Communicating the NFS4ERR_DEADLOCK error to a STORE instruction is difficult at best 🙃

If a client is locking the entire memory-mapped file, there's no problem with advisory or mandatory byte-range locking, at least until the client unlocks a byte-range in the middle of the file. ✨

Given the above issues, the following are permitted:

Clients and servers MAY deny memory mapping a file for which they know there are byte-range locks.
Clients and servers MAY deny a byte-range lock on a file they know is memory-mapped.
A client MAY deny memory mapping a file that it knows requires mandatory locking for I/O. If mandatory locking is enabled after the file is opened and mapped, the client MAY deny the application further access to its mapped file. 🔒

10.8. Name and Directory Caching without Directory Delegations 🗂️

The NFSv4.1 directory delegation facility (described in Section 10.9 below) is OPTIONAL for servers to implement. 🌸 Even where it's implemented, it may not always be functional because of resource availability issues or other constraints. Thus, it's important to understand how name and directory caching work in the absence of directory delegations. These topics are discussed in the next two subsections.

10.8.1. Name Caching 🏷️

The results of LOOKUP and READDIR operations may be cached so the client doesn't have to pay the cost of repeated LOOKUP operations 💅 Just like with attribute caching, inconsistencies may arise among the various client caches 🙃 To mitigate these effects, and given the context of typical file system APIs, there's an upper time boundary on how long a client name cache entry can be kept without verifying that a directory change made by another client hasn't invalidated it.

When a client isn't making changes to a directory that has name cache entries, the client needs to periodically fetch attributes for that directory to make sure it isn't being modified 👀 After determining that no modification has occurred, the expiration time for the associated name cache entries may be updated to the current time plus the name cache staleness bound. 🫧

When a client IS making changes to a given directory, it needs to determine whether other clients have changed the directory too. It does this by using the change attribute as reported before and after the directory operation, in the associated change_info4 value returned for the operation. The server is able to communicate to the client whether the change_info4 data is provided atomically with respect to the directory operation. If the change values are provided atomically, the client has a basis for determining, given proper care, whether other clients are modifying the directory in question. 🧠

The simplest way for the client to make this determination is to serialize all changes made to a specific directory. When this is done, and the server provides before and after values of the change attribute atomically, the client can simply compare the after value of the change attribute from one operation on a directory with the before value on the subsequent operation modifying that directory. When these are equal, the client is assured that no other client is modifying the directory in question. 💖

When such serialization isn't used, and there may be multiple simultaneous outstanding operations modifying a single directory sent from a single client, making this determination gets more complicated 🌀 If two such operations complete in a different order than they were actually performed, that might give the appearance of modification by another client. Where this appears to happen, the client needs to await completion of all such modifications that were started previously, to see if the outstanding before and after change numbers can be sorted into a chain. In that chain, the before value of one change number matches the after value of a previous one, consistent with this client being the only one modifying the directory. 🔗

In either of these cases, the client can determine whether the directory is being modified by another client. If the comparison indicates the directory was updated by another client, the name cache associated with the modified directory is purged from the client 🧹 If the comparison indicates no modification, the name cache can be updated on the client to reflect the directory operation, and the associated timeout can be extended. The post-operation change value needs to be saved as the basis for future change_info4 comparisons. ⭐

As demonstrated by the scenario above, name caching requires that the client revalidate name cache data by inspecting the change attribute of a directory at the point when the name cache item was cached. This requires that the server update the change attribute for directories when the contents of the corresponding directory are modified 📝 For a client to use the change_info4 information appropriately and correctly, the server must report the pre- and post-operation change attribute values atomically. When the server is unable to report the before and after values atomically with respect to the directory operation, the server must indicate that fact in the change_info4 return value. When the information is not atomically reported, the client should not assume that other clients have not changed the directory. 🚨

10.8.2. Directory Caching 📂

okay so the results of READDIR operations can be reused to skip later READDIR operations 🙌 but just like with attribute and name caching, the different client caches can end up out of sync 😬. To cut down on those inconsistencies (and given how typical file system APIs work), the following rules should be followed 💅:

Cached READDIR information for a directory that is not obtained in a single READDIR operation must always be a consistent snapshot of directory contents. This is determined by using a GETATTR before the first READDIR and after the last READDIR that contributes to the cache.
An upper time boundary is maintained to indicate the length of time a directory cache entry is considered valid before the client must revalidate the cached information. ⏰

The revalidation technique is basically the same vibe as name caching 🫶. When the client is not changing the directory in question, checking the change attribute of the directory with GETATTR is adequate. The lifetime of the cache entry can be extended at these checkpoints. When a client is modifying the directory, the client needs to use the change_info4 data to figure out whether other clients are modifying it too. If it's determined that no other client modifications are happening, the client may update its directory cache to reflect its own changes ✨.

As demonstrated previously, directory caching requires that the client revalidate directory cache data by inspecting the change attribute of a directory at the point when the directory was cached. This requires that the server update the change attribute for directories when the contents of the corresponding directory is modified. For a client to use the change_info4 information appropriately and correctly, the server must report the pre- and post-operation change attribute values atomically. When the server is unable to report the before and after values atomically with respect to the directory operation, the server must indicate that fact in the change_info4 return value. When the information is not atomically reported, the client should not assume that other clients have not changed the directory. 🙈

10.9. Directory Delegations 🎀

10.9.1. Introduction to Directory Delegations

Directory caching in NFSv4.1 works like file caching did in earlier versions 💕. Clients usually cache directory information for a duration the client picks. When that timeout ends, the client asks the server whether the directory has been updated. Caching attributes means clients make fewer GETATTR calls to validate them 📉. Frequently used files and directories, like the current working directory, get their attributes cached on the client too, so some NFS operations can happen without making an RPC call 🏃‍♀️. And by caching name and inode info for recently looked-up entries in a Directory Name Lookup Cache (DNLC), clients don't have to send LOOKUP calls to the server every single time those files get accessed.

This approach works pretty well at cutting network traffic in lots of environments 👍. But it doesn't help when there are tons of queries for files that don't exist. Those "misses" still have the client sending requests to the server, so the application gets reasonable semantics and new directory entries are noticed quickly. Compilation in software development environments is a classic example of high-miss activity 🛠️. Right now NFS's behavior limits its scalability and how well it works for sharing across wide areas in these kinds of environments 🌍. Other distributed stateful file system architectures, like AFS and DFS, have shown that adding state around directory contents can greatly reduce network traffic in high-miss environments 🔥.

Delegation of directory contents is an OPTIONAL feature of NFSv4.1 🎁. Directory delegations give similar traffic-reduction benefits to file delegations. Clients can cache directory contents in a read-only way while being notified of changes, so they don't need to keep poking the server to check slowly-changing directories. That cuts network traffic and boosts client performance 🚀. It can also make it simpler to figure out whether other clients are changing the directory when this client is making lots of changes itself and those changes aren't serialized 🧠.

Directory delegations let namespace cache consistency get better through delegations and synchronous recalls, without needing notifications 🤝. And if time-based consistency is good enough, asynchronous notifications can give the client (and possibly the server) performance benefits in common situations like slowly-changing or very large directories 💖.

10.9.2. Directory Delegation Design

NFSv4.1 brings in the GET_DIR_DELEGATION (Section 18.39) operation so a client can ask for a directory delegation 🙋‍♀️. The delegation covers directory attributes and all entries in the directory. If either of those changes, the delegation gets recalled synchronously, and the operation that caused the recall has to wait until the recall is complete ⏳. Changes to directory entry attributes won't cause the delegation to be recalled 🙌.

Besides asking for delegations, a client can also ask for notifications about certain events. These events include changes to the directory's attributes and/or its contents 📣. If a client asks to be notified about a certain event, the server will notify it when that event happens. That doesn't result in the delegation being recalled for that client 🫧. Notifications are asynchronous, and they let the server avoid recalls when a directory is changing so much that the pure recall model wouldn't work well, while the client still gets a lot of benefit 💅. Without notifications, once the delegation is recalled the client has to refresh its directory cache, which might not be very efficient for very large directories 😩.

The delegation is read-only, and the client may not make changes to the directory except by doing NFSv4.1 operations that modify the directory or its associated file attributes, so the server knows about those changes 🔒. To keep the client's namespace in sync with the server's, the server will notify the delegation-holding client (assuming it has asked for notifications) about changes made as a result of that client's own directory-modifying operations. That way the client doesn't have to send follow-up GETATTR or READDIR operations to the server 🧚. If a single client holds the delegation and makes any changes to the directory (meaning the changes are made through operations sent on a session tied to the client ID that holds the delegation), the delegation will not be recalled 💜. Multiple clients may hold a delegation on the same directory, but if any one of them modifies the directory, the server MUST recall the delegation from the other clients, unless those clients have made arrangements to be notified about that kind of modification 🚨.

Delegations can be recalled by the server at any time ⏰. Normally, the server recalls the delegation when the directory changes in a way the notification doesn't cover, or when the directory changes and no notifications were requested. If another client removes the directory that a delegation was granted for, the server will recall the delegation 💀.

10.9.3. Attributes in Support of Directory Notifications

Check out Section 5.11 for a description of the attributes associated with directory notifications 📖.

10.9.4. Directory Delegation Recall

The server recalls a directory delegation by sending a callback to the client 📞. It uses the same callback procedure that it uses for recalling file delegations 🔁. The server recalls the delegation when the directory changes in a way the notification doesn't cover. However, the server need not recall the delegation if attributes of an entry within the directory change 🫶.

If the server notices that handing out a delegation for a directory is causing too many notifications to go out, it may decide to stop handing out delegations for that directory and/or recall the ones already granted 😵‍💫. If a client tries to remove the directory a delegation was granted for, the server will recall all associated delegations 🙅‍♀️.

The implementation sections for a bunch of operations describe situations where notification or delegation recall would be required under some common circumstances. A similar set of caveats to those listed in Section 10.2 applies here 📝.

For CREATE, see Section 18.4.4.
For LINK, see Section 18.9.4.
For OPEN, see Section 18.16.4.
For REMOVE, see Section 18.25.4.
For RENAME, see Section 18.26.4.
For SETATTR, see Section 18.30.4.

10.9.5. Directory Delegation Recovery

Recovering state after a client or server restart, for state on regular files, has two main goals: avoiding the need to break application guarantees about locked files, and delivering updates cached at the client 🔐. Neither of those goals applies to directories protected by OPEN_DELEGATE_READ delegations and notifications 🙌. So no provision is made for reclaiming directory delegations after a client or server restart. The client can just set up a directory delegation the same way it did at first 🌸.

11. Multi-Server Namespace 🌐

NFSv4.1 supports attributes that let a namespace stretch beyond the boundaries of a single server 🧩. It's desirable for clients and servers to support building these multi-server namespaces 💫. Using them is OPTIONAL, though, and for lots of purposes single-server namespaces are perfectly fine 👌. Multi-server namespaces can bring a lot of advantages by separating a file system's logical position in a namespace from the logistical and administrative stuff (which can change) that decides where a particular file system lives on a particular server. Normally that server is reached through a single network access path that has to be known ahead of time or figured out using DNS 🗺️.

11.1. Terminology

Within all of Section 11, the phrase "client ID" always means the 64-bit shorthand identifier the server assigns (a clientid4). It never refers to the structure the client uses to identify itself to the server, which is called an nfs_client_id4 in NFSv4.0 and client_owner in NFSv4.1 💅. The opaque identifier inside those structures is called a "client id string" 🎀.

okay bestie, this part is SO important 💅 the difference between trunking detection and trunking discovery is the whole vibe, so pay attention 👀 The definitions here apply to all minor versions of NFSv4, but we're gonna focus on how they hit for NFS version 4.1 🦋

Trunking detection is about figuring out whether two specific network addresses are connected to the same NFSv4 server 🔍 Which way you can do that depends on the protocol version and, in some cases, on the client implementation, ngl.

For NFS version 4.1 and later minor versions, the trunking detection methods are the ones described in this document, and every client has access to them 💖 Two network addresses connected to the same server can always be used together to access a particular server, but they can't necessarily be used together to access a single session 🙈 Peep the definitions of "server-trunkable" and "session-trunkable" below.

Trunking discovery is a process where a client using one network address can grab other addresses connected to the same server ✨ It usually builds on a trunking detection facility by providing one or more ways for candidate addresses to show up, so the client can use trunking detection to filter them 🧚

Even though trunking detection was already supported, RFC 5661 [66] didn't describe trunking discovery at all, so this document had to provide those means 😭

The combination of a server network address and a particular connection type used by a connection is called a "server endpoint" 🎀 Different connection types might use different ports, but that port difference isn't the essence of the distinction between the two endpoints 💅 Port-specific endpoints are different, though: there, explicit port numbers inside network addresses let a single server node support multiple NFS servers 🌸

Two network addresses connected to the same server are server-trunkable 🧚 They support client ID trunking, as described in Section 2.10.5 💜

Two network addresses connected to the same server such that they can support a single common session are session-trunkable 🦋 Note that two addresses can be server-trunkable without being session-trunkable 👀 And when two connections of different connection types are made to the same network address and are based on a single file system location entry, they're always session-trunkable, no matter the connection type 🫶 As specified by Section 2.10.5, their derivation from the same file system location entry, plus the identity of their network addresses, assures that both connections go to the same server and return server-owner information, so session trunking can be used ✨

Here's the lowdown on building multi-server namespaces out of a bunch of local per-server namespaces 🌈

Each server has a set of exported file systems that NFSv4 clients may access 🍓 Typically, each one gets a name within the pseudo-fs associated with the server, although the pseudo-fs can be skipped if there's only a single exported file system. Each one is part of the server's local namespace and can be considered a file system instance within a larger multi-server namespace ✨
All the exported file systems for a given server make up that server's local namespace 💅
Sometimes a server has a namespace bigger than its local one, thanks to features tied to attributes that provide file system location information 👑 These features, which allow a multi-server namespace to be built, are all described in individual sections below and include referrals (Section 11.5.6), migration (Section 11.5.5), and replication (Section 11.5.4) 🦋
A file system in a server's pseudo-fs may have multiple file system instances on different servers associated with it 🫶 All of them count as replicas of one another. Whether replicas can be used simultaneously is discussed in Section 11.11.1, and the level of coordination between them (important when switching between them) is discussed in Sections 11.11.2 through 11.11.8 below 💜
When a file system is present in a server's pseudo-fs but there's no corresponding local file system, it's "absent" 🙈 In that case, all its associated instances get accessed on other servers.

Now for the attribute stuff used in trunking discovery and other multi-server namespace features 💖

File system location attributes include the fs_locations and fs_locations_info attributes 🎀
File system location entries are the individual file system locations within the file system location attributes 🧚 Each one specifies a server, as a hostname or an address, and an fs name, which designates the location of the file system within the server's local namespace. A file system location entry designates a set of server endpoints the client may establish connections to 🌸 There can be several because a hostname may map to multiple network addresses and because multiple connection types may be used with a single network address. But except where explicit port numbers are used to designate a set of servers within a single server node, all such endpoints MUST designate a way of connecting to a single server 💅 The exact form of the location entry varies with the particular file system location attribute used, as described in Section 11.2.

Network addresses in file system location entries typically show up without port numbers and are used to designate a server at one of the standard ports for NFS access, like 2049 for TCP or 20049 for use with RPC-over-RDMA 🌈 Port numbers may be used in location entries to designate servers (typically user-level ones) accessed using other port numbers. When network addresses indicate trunking relationships, an explicit port number is inappropriate, since trunking is a relationship between network addresses. See Section 11.5.2 for details 👀

File system location elements are derived from location entries, and each one describes a particular network access path made up of a network address and a location within the server's local namespace 🦋 They don't have to appear within a file system location attribute, but each one's existence derives from a corresponding location entry. When a location entry specifies an IP address, there's only a single corresponding location element. Location entries that contain a hostname get resolved using DNS and may result in one or more location elements 🔍 All location elements consist of a location address, which includes the IP address of an interface to a server, and an fs name, which is the location of the file system within the server's local namespace. The fs name can be empty if the server has no pseudo-fs and only a single exported file system at the root filehandle 🍓
Two file system location elements are server-trunkable if they specify the same fs name and their location addresses are server-trunkable. When the corresponding network paths are used, the client will always be able to use client ID trunking, but it will only be able to use session trunking if the paths are also session-trunkable 💖
Two file system location elements are session-trunkable if they specify the same fs name and their location addresses are session-trunkable. When the corresponding network paths are used, the client will be able to use either client ID trunking or session trunking ✨

Discussion of the term "replica" is a bit complicated 😵 because RFC 5661 [66] used the term with a meaning different from the one used in this document. In short, [66] identifies each replica by a single network access path, while in the current document, a set of network access paths that have server-trunkable network addresses and the same root-relative file system pathname is considered a single replica with multiple network access paths 🫶

Each set of server-trunkable location elements defines a set of available network access paths to a particular file system. When there are multiple such file systems, each containing the same data, they're considered replicas of one another 🌈 Logically, that replication is symmetric, since the fs currently in use and an alternate fs are replicas of each other. Other documents often don't apply the term "replica" to the fs currently in use, even though the replication relation is inherently symmetric 💅

11.2. File System Location Attributes ✨

okay so NFSv4.1 has attributes that tell you how a file system can be reached, meaning what network address and where in the namespace 💅. That means a file system in one server's namespace can be linked to one or more copies of it on other servers. These attributes hold file system location entries, and each entry has a server address target (either a DNS name that stands for one or more IP addresses, or one specific IP address) plus the pathname of that file system inside its single-server namespace 🗺️.

The fs_locations_info RECOMMENDED attribute lets you specify one or more file system instance locations where the data for a given file system can be found 🔥. Besides listing those locations, it helps you do a few things:

Guide choices among the various file system instances provided (e.g., priority for use, writability, currency, etc.).
Help the client efficiently effect as seamless a transition as possible among multiple file system instances, when and if that should be necessary.
Guide the selection of the appropriate connection type to be used when establishing a connection.

Inside fs_locations_info, each fs_locations_server4 entry matches up with a file system location entry. The fls_server field names the server, and the fl_rootpath field of the enclosing fs_locations_item4 gives the location pathname within the server's pseudo-fs 🧚.

The fs_locations attribute from NFSv4.0 is also part of NFSv4.1 🌸. It only lets you specify where the data for a file system can be found. Servers SHOULD make this attribute available whenever fs_locations_info is supported, but using fs_locations_info is preferable on the client side because it packs in more information 🫶.

Within fs_locations, each fs_location4 holds a file system location entry. The server field names the server, and the rootpath field gives the location pathname within the server's pseudo-fs 💜.

11.3. File System Presence or Absence 🦋

A given location in an NFSv4.1 namespace (usually a multi-server namespace, but not always) can have a bunch of file system instance locations attached to it via fs_locations or fs_locations_info. There might also be an actual current file system right there, reachable through normal namespace operations like LOOKUP. In that case the file system is "present" at that spot. Clients will typically just use it and save the other location-related attributes for when the main spot stops being available 😭.

When there's no actual file system at that namespace location, it's "absent" 🫥. An absent file system has nothing in it except the root. Any reference to it will error out with NFS4ERR_MOVED, except for a small set of attributes that help figure out alternate locations. Note that if the server ever returns NFS4ERR_MOVED, it MUST support the fs_locations attribute and SHOULD support the fs_locations_info and fs_status attributes 💀.

The error name kinda suggests the file system was present once and then went absent, but that's only one possibility 🙈. A spot can be permanently absent, with the file system(s) from the location attributes being the only real version. The name NFS4ERR_MOVED reflects an older, narrower idea of what it does, but the error comes back whenever the file system is absent, moved or not 💅.

Except for GETATTR-type operations (more on those later), when the current filehandle at the start of an operation is inside an absent file system, that operation doesn't run. You get NFS4ERR_MOVED instead, which tells you the file system is absent on the current server 🚫.

Since GETFH can't succeed when the current filehandle is inside an absent file system, filehandles in an absent file system can't be transferred to the client. If a client does have filehandles in an absent file system, it got them while the file system was present, and then it went absent later 🥲.

The check for the current filehandle being inside an absent file system happens at the start of every operation. So operations that change the current filehandle to somewhere inside an absent file system won't error out! That's what lets combos like PUTFH-GETATTR and LOOKUP-GETATTR grab attribute info, especially location attribute info, as discussed below 👑.

The RECOMMENDED file system attribute fs_status can be used to check whether a given file system is present or absent 🔍.

11.4. Getting Attributes for an Absent File System 🎀

When a file system is absent, most attributes aren't available. But the client still needs access to the small set that are, especially the ones that tell it the correct current locations: fs_locations and fs_locations_info 🗺️.

11.4.1. GETATTR within an Absent File System ✨

As mentioned above, GETATTR gets an exception: attributes can be fetched for a filehandle inside an absent file system. This exception only applies if the attribute mask has at least one bit for an attribute that cares about an absent file system's result: fs_locations, fs_locations_info, or fs_status. If none of those is requested, GETATTR returns NFS4ERR_MOVED 🙅.

When GETATTR runs on an absent file system, the set of supported attributes is super limited. Lots of attributes, including the normally REQUIRED ones, won't be available there. Besides fs_locations, fs_locations_info, and fs_status, the following SHOULD be available on absent file systems. For RECOMMENDED attributes, they should be available at least to the same degree they are on present file systems 🧁.

change_policy: This attribute is useful for absent file systems and can be helpful in summarizing to the client when any of the location-related attributes change.
fsid: This attribute should be provided so that the client can determine file system boundaries, including, in particular, the boundary between present and absent file systems. This value must be different from any other fsid on the current server and need have no particular relationship to fsids on any particular destination to which the client might be directed.
mounted_on_fileid: For objects at the top of an absent file system, this attribute needs to be available. Since the fileid is within the present parent file system, there should be no need to reference the absent file system to provide this information.

Other attributes SHOULD NOT be made available for absent file systems, even when it's possible to provide them 🙃. The server shouldn't assume more information is always better, and it should avoid gratuitously handing out extra info 🛑.

When a GETATTR includes a bit mask for one of fs_locations, fs_locations_info, or fs_status, but the mask also includes attributes that aren't supported, GETATTR won't error. Instead it returns the mask of the actual supported attributes along with the results 🫶.

VERIFY/NVERIFY handling is similar to GETATTR in that if the attribute mask doesn't include fs_locations, fs_locations_info, or fs_status, you get NFS4ERR_MOVED. It's different in that any attribute in the mask that isn't supported for an absent file system (and that includes some normally REQUIRED attributes) will also cause an NFS4ERR_MOVED result 😤.

11.4.2. READDIR and Absent File Systems 🍓

A READDIR run while the current filehandle is inside an absent file system gives you NFS4ERR_MOVED. Unlike GETATTR, READDIR doesn't get that exception 🚫.

You can still fetch attributes for an absent file system through a READDIR on a directory in a present file system, as long as that directory contains the root directories of one or more absent file systems 🧚. In that case, it works like this:

If the attribute set requested includes one of fs_locations, fs_locations_info, or fs_status, then fetching attributes goes through normally, and no NFS4ERR_MOVED indication is returned, even when rdattr_error is requested.
If the attribute set requested doesn't include one of fs_locations, fs_locations_info, or fs_status, then if rdattr_error is requested, each directory entry for the root of an absent file system reports NFS4ERR_MOVED as the value of the rdattr_error attribute.
If the attribute set requested doesn't include any of fs_locations, fs_locations_info, fs_status, or rdattr_error, then the root of an absent file system showing up in the directory makes the READDIR fail with an NFS4ERR_MOVED error 💥.
An attribute being unavailable because the file system is absent doesn't cause any error indication, even if it's normally REQUIRED. The set of attributes returned for the root directory of the absent file system is simply limited to the ones actually available 🌈.

11.5. Uses of File System Location Information 🗺️

okay so this is where the file system location stuff gets really fun 🦋. The fs_locations and fs_locations_info attributes, together with the whole absent file systems thing, unlock a bunch of iconic features for reliable, manageable, and scalable data access 💅.

When a file system is present, these attributes can give you:

The locations of alternative replicas to use for accessing the same data if a server goes down, there are communication problems, or other stuff makes continuing to use the current replica impossible or just impractical 😭. Setting up and using those backup replicas is called "replication", and we'll get into it in Section 11.5.4 below 🌸.
The network address(es) to use for reaching the current file system instance or its replicas. How clients use this info is in Section 11.5.2 below ✨.

Under some circumstances, multiple replicas can be used at the same time for faster access to the file system, although the lack of state sharing between servers can be a blocker for that 🙈.

When a file system is present but then becomes absent, clients can get the chance to keep accessing their data through a different replica 🦋. In that case, if someone keeps trying to use data in the now-absent file system, they'll get an NFS4ERR_MOVED error 💀. Then the successor replica (or the set of possible replica choices) can be fetched and used to keep access going 🏃‍♀️. Moving access over to the new replica location is called "migration", and it's covered in Section 11.5.4 below 👑.

When a file system is currently absent, the file system location info gives a way to link file systems living on one server to a namespace defined by a different server 🌈. This is what allows a general multi-server namespace facility. A designation of such a remote instance, standing in for a file system that wasn't previously present, is called a "pure referral" and is discussed in Section 11.5.6 below 🧚.

Because client support for attributes related to file system location is OPTIONAL, a server may choose to hide migration and referral events from clients, for example by acting as a proxy 🫶. The server can figure out whether a client supports this from the arguments of the EXCHANGE_ID operation (see Section 18.35.3) 🔍.

11.5.1. Combining Multiple Uses in a Single Attribute 🧩

A file system location attribute will sometimes hold info about the locations of multiple replicas, and those can be used in different ways 💖:

File system location entries that relate to the file system instance currently in use provide trunking information, letting the client find additional network addresses it can use to access that instance 🛤️.
File system location entries that provide information about replicas to which access is to be transferred 🚚.
Other file system location entries that relate to replicas available to use if access to the current replica becomes unsatisfactory 🪂.

To keep client handling simple and let the client pick the best replica, the server should follow these guidelines 📝:

All file system location entries that relate to a single file system instance should be adjacent 🧵.
File system location entries that relate to the instance currently in use should appear first 🥇.
File system location entries that relate to replica(s) to which migration is occurring should appear before replicas that are available for later use if the current replica should become inaccessible 🏁.

11.5.2. File System Location Attributes and Trunking 🚇

Trunking is using multiple connections between a client and server to speed up data transfer 🏎️💨. A client can figure out the set of network addresses to use for a given file system in a few ways:

When the client knows the server's name, it can use DNS to get a set of network addresses for reaching the server 🔎.
The client can fetch the file system location attribute for the file system. This gives either the server's name (which can be turned into a set of network addresses with DNS) or a set of server-trunkable location entries. With the latter, the server can hand over addresses it thinks are good to use for accessing that file system 🎁. Even though these entries can contain port numbers, those port numbers aren't used when deciding trunking relationships 🙅‍♀️. Once the candidate addresses are determined and EXCHANGE_ID is done with the proper server, only the value of the so_major_id field returned by those servers decides whether a trunking relationship actually exists 👀.

When the client fetches a location attribute for a file system, it may run into multiple entries for a bunch of reasons, so when working out trunking info it might need to skip over addresses that can't be trunked with one it already knows 🙃.

The server can provide location entries with either names or network addresses 📛. It might use addresses because of DNS-related security concerns, or because the set of addresses to be used might need active management by the server 🛡️.

Location entries used to find candidate addresses for trunking can change, as discussed in Section 11.5.7 below 🔄. The client may respond to those changes by using additional addresses once they're verified, or by ceasing to use existing ones ✅. The server can force the client to stop using an address by returning NFS4ERR_MOVED when that address is used to access a file system 🚫. This is a kind of access transfer that's similar to migration, although the same file system instance is accessed throughout ✨.

11.5.3. File System Location Attributes and Connection Type Selection 🔌

Because there are multiple types of connections, clients have to figure out which connection type to use when connecting to a given server network address 🤔. Sometimes this can be handled with the connection "step-up" facility described in Section 18.36. But since there are cases where that facility isn't available, the client may have to pick a connection type that can't be changed within a single connection 🙅‍♀️.

The two file system location attributes differ in the info they give about this. The fs_locations attribute gives no info to support connection type selection, so clients supporting multiple connection types would need to keep trying to establish connections with different types until the one they prefer is successfully set up 🥲.

The fs_locations_info attribute includes the FSLI4TF_RDMA flag, which is super convenient for a client that wants to use RDMA 🤩. When this flag is set, it means RPC-over-RDMA support is available using the specified location entry. A client can establish a TCP connection and then convert it to use RDMA with the step-up facility 🪄.

Irrespective of the particular attribute used, when there's no indication that a step-up operation can be performed, a client supporting RDMA operation can establish a new RDMA connection and bind it to the session already established by the TCP connection 🔗. That lets the TCP connection be dropped and the session converted to further use in RDMA mode, if the server supports that 💫.

11.5.4. File System Replication 🪞

The fs_locations and fs_locations_info attributes provide alternative file system locations, to be used to access data in place of or in addition to the current file system instance 🌷. On first access to a file system, the client should get the set of alternate locations by interrogating the fs_locations or fs_locations_info attribute, with the latter being preferred 🥇.

If server failures, communication problems, or other difficulties make continued access to the current file system impossible or impractical, the client can use the alternate locations to keep getting access to its data 🆘.

The alternate locations may be physical replicas of the (typically read-only) file system data, plus possibly asynchronous propagation of updates 📚. Or they might provide for various forms of server clustering, where multiple servers offer alternate ways of accessing the same physical file system 🖥️🖥️. How the difference between replicas affects file system transitions can be represented within the fs_locations and fs_locations_info attributes, and how the client deals with transition issues is going to be discussed in detail in later sections 📖.

Although the location attributes give some info about the nature of the inter-replica transition, many aspects of the semantics of possible asynchronous updates aren't currently described by the protocol 😵‍💫. That means clients using replication to switch among replicas that are changing need to get familiar with the semantics of the update approach used 🧐. Because of this lack of specificity, lots of applications may find migration more appropriate, since a server can propagate all updates made before an established point in time to the new replica as part of the migration event 🦋💜.

11.5.4.1. File System Trunking Presented as Replication

Okay bestie, so sometimes a file system location entry points at an access path that's meant to be an alternate location, but the move is actually trunking, not replication 🦋. This only happens when BOTH of these are true, no cap:

The two file system locations (the one where the location attribute was obtained, and the one in the file system location entry) point to the same spots inside their own single-server namespaces ✨.
The two server network addresses (the one used to get the location attribute, and the one in the file system location entry) point to the same server, which you can tell by matching the same so_major_id in the eir_server_owner field returned from EXCHANGE_ID 💅.

When those conditions hold, operations over both access paths are generally trunked 🫶. Trunking can still be disallowed when fs_locations_info is in the picture:

When fs_locations_info shows the two entries as not having the same simultaneous-use class, trunking is inhibited and the two access paths can't be used together 💀.

In this case, the two paths can be used serially with no transition activity needed from the client, and any switch between them is transparent 🌸. When moving access from one to the other, the client acts as if communication got interrupted. It sets up a new connection and possibly a new session to keep accessing the same file system 🧚.

Heads up for two location entries like this: if fs_locations_info has anything saying special transition activity is needed, like the two entries showing up with different handle, fileid, write-verifier, change, and readdir classes, that's a serious problem 😭. The client, if it lets transition to that file system instance at all, must not treat any transition as a transparent one. The server SHOULD NOT indicate that these two entries (for the same file system on the same server) belong to different handle, fileid, write-verifier, change, and readdir classes, whether or not the two entries are shown in the same simultaneous-use class 🙈.

These situations were recognized by [66], even though that document never explicitly mentioned trunking 🍓:

It treated the trunking situation as simultaneous use of two distinct file system instances. But in the framework we use now, it's one file system being accessed through two trunked addresses, which is giving plot twist 👑.
It treated serial use as a special sort of "transparent transition". In the current framework for sorting transition situations, though, this counts as a "network endpoint transition" (see Section 11.9) 💜.

11.5.5. File System Migration

When a file system is present but becomes inaccessible through the current access path, NFSv4.1 gives clients a way to keep access to their data 🌈. This may involve using a different access path to the existing replica, or providing a path to a different replica. The new access path or the location of the new replica is specified by a file system location attribute. The migration that follows includes the ability to retain locks across the transition. Depending on circumstances, this can involve:

The continued use of the existing clientid when accessing the current replica using a new access path.
Use of lock reclaim, taking advantage of a per-fs grace period.
Use of Transparent State Migration.

Typically, a client is accessing the file system, gets an NFS4ERR_MOVED error, and then uses a file system location attribute to find the new access path for the data 🧚. When fs_locations_info is used, extra information is available that defines how the client handles the transition to a new server 🫶.

In most instances, servers choose to migrate all clients using a particular file system to a successor replica at the same time. That way, different clients aren't updating different replicas 😭. Migrating an individual client can still be helpful for load balancing, as long as the replicas in question represent the same data as described in Section 11.11.8 ✨.

If there's no transition between replicas (only a change in access path), there are no special difficulties in using this mechanism for load balancing 🌸.
If the two replicas are sufficiently coordinated to let a single client have coherent, simultaneous access to both, there is, in general, no obstacle to migrating particular clients for load balancing 💅. Usually, this kind of simultaneous use involves cooperation between servers to make sure locks granted on two coordinated replicas can't conflict and can stay effective when transferred to a common replica.
If a large set of clients is accessing a file system in a read-only fashion, it can help to migrate all clients with writable access at the same time, while load balancing across the set of read-only copies. This is fine as long as the rules in Section 11.11.8, which are designed to prevent data reversion, are followed 🔥.

In other cases, the client might not have enough guarantees of data similarity or coherence to function properly. For example, the data in the two replicas might be similar but not identical. If different clients are updating different replicas, that can make the difficulties worse, so load balancing in those situations is a perilous enterprise 💀.

The protocol does not specify how the file system will be moved between servers, or how updates to multiple replicas will be coordinated 🙈. A number of different server-to-server coordination mechanisms might be used, and the choice is left to the server implementer. NFSv4.1 does specify the method used to communicate the migration event between client and server 🦋.

With various forms of server clustering, the new location may be another server providing access to the same physical file system ✨. The client's responsibilities in dealing with this transition depend on whether a switch between replicas has occurred, and on the means the server has chosen to provide continuity of locking state. These issues will be discussed in detail below 🍓.

A single successor location is typical, but multiple locations may be provided 👑. When multiple locations are provided, the client will typically use the first one. If that one is inaccessible for some reason, later ones can be used. In such cases, the client might consider the transition to the new replica to be a migration event, even though some of the servers involved might not be aware of the use of the inaccessible server. Then a client might lose access to locking state as a result of the access transfer 😭.

When an alternate location is designated as the target for migration, it must designate the same data, with metadata being the same to the degree indicated by the fs_locations_info attribute 💜. Where file systems are writable, a change made on the original file system must be visible on all migration targets. Where a file system is not writable but represents a read-only copy (possibly periodically updated) of a writable file system, similar requirements apply to the propagation of updates 🌈. Any change visible in the original file system must already be effected on all migration targets, so that there's no possibility of a client seeing any reversion in file system state when it transitions to a migration target 🫶.

11.5.6. Referrals ✨

okay so referrals are how the server links a spot in the namespace to a file system that's actually sitting on a different server 🗺️. with pure referrals, the server can slot a file system into a place in the namespace basically without caring where it physically lives on any one server 💅. so one server, or a whole group of them, can present a multi-server namespace covering file systems spread across a wider range of servers 🌈. the likely uses are site-wide or organization-wide namespaces, with the eventual possibility of combining those into a truly global namespace, kinda like the one AFS (the Andrew File System) [65] gives us 🫶

referrals happen when a client, upon first referencing a position in the current namespace, figures out that it's part of a new file system and that the file system is absent 👀. this usually goes down when the server throws NFS4ERR_MOVED, and then the client can find the actual location or locations of the file system by fetching a locations attribute 🔍.

the file system location attribute can point to a single location or to multiple locations, and the client picks based on what it needs 🎀. in fs_locations_info, the server can attach priorities to the different location choices. it can give different clients different priorities to adapt to where they are physically or to balance the load ⚖️. when there are both read-only and read-write file systems around, some read-only locations might not be totally up-to-date, which is different from replication and migration 🙈. servers can also use client-substituted variables in their locations, so different clients get sent to different file systems (with different data contents) based on client attributes like CPU architecture 🧠

if fs_locations_info lists multiple possible targets, the relationships between them can matter a lot to the client when it's choosing 🤔. the same rules from Section 11.5.5 below about multiple migration targets apply to these replicas too. like, the client might prefer a writable target on a server that also has more writable replicas it could switch to later 🔄. and unlike replication, there's no need to worry about propagating updates made by the current client, since that client hasn't accessed the file system in question yet 💖

multi-server namespaces are enabled by NFSv4.1 but they aren't required 🙌. how much you use them and their scope depends on your apps and what your sysadmins prefer 🧚

you can set up multi-server namespaces with a single server handing out a big pile of pure referrals to all of the included file systems 🍓. or you can split a multi-server namespace into segments, with separate referral file systems on separate servers for each separately administered portion of the namespace 🧩. the top-level referral file system, or any segment, can use replicated referral file systems for higher availability 💪

honestly, multi-server namespaces are mostly uniform, meaning the same data a client sees at a given location is what every client sees at that location 👯. BUT there are facilities that let different clients get directed to different sets of data, like adapting to things such as CPU architecture 🦋. those facilities are described in Section 11.17.3 ✨

also, when you're providing a uniform namespace, you can still hand different location entries to different clients so each one gets the copy of the data physically closest to it, or otherwise optimize access (for example, load balancing) 🌸

11.5.7. Changes in a File System Location Attribute 🔄

clients will typically fetch a file system location attribute when they first access a file system and whenever NFS4ERR_MOVED gets returned 📥. but a client can also choose to fetch it periodically, in which case the value might change over time ⏰.

for clients that aren't prepared to access multiple replicas simultaneously (see Section 11.11.1), here's how each kind of location change gets handled 💅:

changes in the list of replicas, or in the network addresses associated with replicas, don't require immediate action. the client will typically update its list of replicas to reflect the new information 📝
additions to the list of network addresses for the current file system instance don't need to be acted on promptly. but to get ready for a later migration event, the client can take note of the new address and then use it whenever it needs to switch access to a new replica 🧳
deletions from the list of network addresses for the current file system instance don't make the client immediately stop using existing access paths, although new connections must not be established on addresses that have been deleted 🚫. the client can choose to act on those deletions by preparing for an eventual shift in access, which becomes unavoidable as soon as the server returns NFS4ERR_MOVED to say a particular network access path can't be used to access the current file system 😭

for clients that are prepared to access several replicas at once, there are some extra cases to cover 🫧. same as above, changes in the set of replicas don't need to be acted on right away, though the client can still adjust its access even when there aren't any difficulties that would lead to picking a new replica 🌈

when a new replica is added, which can be accessed simultaneously with one that's already in use, the client is free to use the new replica immediately ✨
when a replica that's currently in use gets deleted from the list, the client doesn't have to stop using it right away. however, since the server may later force that use to stop (by returning NFS4ERR_MOVED), clients might decide to limit the need for a later state transfer 🧠. for example, new opens might be done on other replicas instead of one that's no longer in the list 🙈

11.6. Trunking without File System Location Information 🦋

when a file system is accessed using two server-trunkable addresses (indicated by the same value of the so_major_id field of the eir_server_owner field returned in response to EXCHANGE_ID), trunked access is allowed even if there aren't any location entries specifically saying that trunking is used for that file system 💜

this situation was recognized by [66], although that document didn't explicitly mention trunking and treated it as simultaneous use of two distinct file system instances 🤷‍♀️. in the explanatory framework used now, it's described as a single file system being accessed by two different trunked addresses 🎀

11.7. Users and Groups in a Multi-Server Namespace 👑

just like in a single-server environment (see Section 5.9), when an owner or group name of the form id@domain gets assigned to a file, there's an implicit promise to return that same string when the attribute is interrogated later 🤞. in a multi-server namespace, that promise still applies even when server boundaries get crossed 🌉. similarly, when a file's owner attribute is derived from the security principal that created the file, that attribute should have the same value even if the interrogation happens on a different server from where the file was created 🫶

the set of security principals recognized by all the participating servers also needs to be the same, with each principal having the same credentials no matter which server is being accessed 🔐

to meet all of this, whoever sets up a multi-server namespace needs to limit the servers included so that: 💅

in all cases where more than a single domain is supported, the requirements stated in RFC 8000 [31] are respected
all servers support a common set of domains that includes all of the domains clients use and expect to see returned as the domain portion of an owner or group in the form id@domain. this set most often has just one domain, but it's possible for multiple domains to be supported 🌍
all servers, for each domain they support, accept the same set of user and group ids as valid 🪪
all servers recognize the same set of security principals. for each principal, the same credential is required no matter which server is accessed. the group membership for each principal also has to be the same, regardless of the server accessed 👯

there's no general requirement that the users behind particular security principals have the same local representation on each server, even though it's most often the case that they do 🧚

when AUTH_SYS is used, these extra requirements must be met too: 🔥

only a single NFSv4 domain can be supported through the use of AUTH_SYS
the "local" representation of all owners and groups must be the same on all servers. "local" is used here since that's how numeric user and group ids are described in Section 5.9. however, when AUTH_SYS or stringified numeric owners or groups are used, those identifiers aren't truly local, since they're known to the clients as well as to the server 🙈

similarly, when stringified numeric user and group ids are used, the "local" representation of all owners and groups must be the same on all servers, even when AUTH_SYS isn't used 💖

11.8. Additional Client-Side Considerations 🎀

Okay bestie, this one is all about the client side 💅. When clients use servers that do referrals, replication, and migration, they need to make sure a user who mounts a file system with a referral or a relocated file system still sees a coherent picture of it 🧚. Even though it's secretly made up of a bunch of server-side file systems that might live on totally different servers, the user should not be able to tell 🫶.

One big thing is upward navigation 🪜. That means going from the root of a server-side file system up to its parent (written ".." in UNIX). This gets tricky when the client landed in that file system because of a referral, a migration, or a transition from replication. When the client is at that point and needs to ascend to the parent, it must go back to the parent as seen within the multi-server namespace. It must not send a LOOKUPP operation to the server, because that would return the parent inside that server's single-server namespace 🙅‍♀️. To pull this off, the client needs to remember the filehandles that represent those file system roots, and use them instead of sending LOOKUPP to the current server. This is what lets the client show apps one consistent namespace where going up and going down match up perfectly ✨.

Another issue is refreshing referral locations 🔄. When referrals get used a lot, they may change as server configurations change. Clients are expected to cache information about traversing referrals, so future requests can be resolved locally without talking to the server. This is usually rooted in client-side name lookup caching. Clients should periodically purge this data for referral points so they can detect changes in location information. When the change_policy attribute changes for directories that hold referral entries, or for the referral entries themselves, clients should consider any associated cached referral information to be out of date 💀.

11.9. Overview of File Access Transitions 🦋

File access transitions come in two types:

Those that involve moving from accessing the current replica to another one, in connection with either replication or migration. How these get handled is discussed in Section 11.11.
Those where access to the current file system instance is kept, but the network path used to access that instance changes. This case is discussed in Section 11.10.

11.10. Effecting Network Endpoint Transitions 🌈

The endpoints used to access a particular file system instance can change in a bunch of different ways, as listed below. In every one of these cases, the same fsid, client IDs, filehandles, and stateids get used to continue access, with continuity of lock state 🔐. In lots of cases, the same sessions can be reused too, which is so efficient 😭.

The right move depends on the set of replacement addresses that are available for use (i.e., server endpoints that are server-trunkable with one previously being used) 👑.

When use of a particular address is about to cease, and there is also another address currently in use that is server-trunkable with it, requests that would have gone out on the address being dropped can be issued on the remaining address(es) ✨. When an address is server-trunkable but not session-trunkable with the address being discontinued, the request might need to be modified to reflect the fact that a different session will be used 🎀.
When use of a particular connection is about to cease, as indicated by receiving NFS4ERR_MOVED when using that connection, but that address is still indicated as accessible according to the appropriate file system location entries, it is likely that requests can be issued on a new connection of a different connection type once that connection is established 🔌. Since any two non-port-specific server endpoints that share a network address are inherently session-trunkable, the client can use BIND_CONN_TO_SESSION to access the existing session with the new connection 🫶.
When there are no potential replacement addresses in use, but there are valid addresses that are session-trunkable with the one being discontinued, the client can use BIND_CONN_TO_SESSION to access the existing session using the new address 🧚. The target session will generally be accessible, but there may be rare situations where it's no longer accessible when someone tries to bind the new connection to it 😬. In that case, the client can create a new session to keep enabling continued access to the existing instance over the new connection. This means existing filehandles, stateids, and client ids still work, and continuity of locking state is still supplied 🔥.
When there is no potential replacement address in use, and there are no valid addresses that are session-trunkable with the one being discontinued, other server-trunkable addresses may be used to provide continued access 🌸. Using CREATE_SESSION is available to provide continued access to the existing instance, but servers have the option of providing continued access to the existing session through the new network access path, in a fashion similar to what session migration provides (see Section 11.12) 💜. To take advantage of this possibility, clients can perform an initial BIND_CONN_TO_SESSION, as in the previous case, and only use CREATE_SESSION if that fails 🙈.

11.11. Effecting File System Transitions 🔁

There are a range of situations where a change needs to be effected in the set of replicas used to access a particular file system 🍓. Some of these may involve expanding or contracting the set of replicas used, as discussed in Section 11.11.1 below.

For reasons explained in that section, most transitions will involve moving from a single replica to a corresponding replacement replica 💖. When effecting a replica transition, some types of sharing between the replicas may affect how the transition is handled, as described in Sections 11.11.2 through 11.11.8 below. The attribute fs_locations_info provides helpful information that lets the client figure out the degree of inter-replica sharing 🔍.

With regard to some types of state, how much continuity you get across the transition depends on what prompted it 🤔. Transitions initiated by the servers (i.e., migration) offer much more scope for a nondisruptive transition than cases where the client on its own shifts its access to another replica (i.e., replication) 🌈. This issue potentially applies to locking state and to session state, which are dealt with below as follows:

An introduction to the possible means of providing continuity in these areas appears in Section 11.11.9 below 📖.
Transparent State Migration is introduced in Section 11.12. The possible transfer of session state is addressed there as well 🦋.
The client handling of transitions, including how to deal with the various means the server might take to supply effective continuity of locking state, is discussed in Section 11.13 🎀.
The source and destination servers' responsibilities in effecting Transparent State Migration of locking and session state are discussed in Section 11.14 👑.

11.11.1. File System Transitions and Simultaneous Access 👯

The fs_locations_info attribute (described in Section 11.17) may indicate that two replicas can be used simultaneously. Some situations where that simultaneous access is permitted are more accurately described as instances of trunking (see Section 11.5.4.1) 🧩. Situations where multiple replicas may be accessed at the same time are kind of similar to ones where a single replica is accessed by multiple network addresses, but there are important differences, since locking state is not shared among multiple replicas 🚫.

Because of this difference in state handling, many clients won't be able to take advantage of the fact that these replicas represent the same data 😤. Those clients won't be prepared to use multiple replicas simultaneously. Instead they'll access each file system using only a single replica, although the replica they pick might make multiple server-trunkable addresses available 🌸.

Clients who are prepared to use multiple replicas simultaneously can divide opens among replicas however they choose 💅. Once that choice is made, any subsequent transitions will treat the set of locking state associated with each replica as a single entity 🔗.

For example, if one of the replicas becomes unavailable, access will be transferred to a different replica that is also capable of simultaneous access with the one still in use 🫶.

When there is no such replica, the transition may be to the replica already in use. At that point, the client has a choice: it can merge the locking state for the two replicas under the aegis of the sole replica in use, or it can treat them separately until another replica capable of simultaneous access shows up 🤷‍♀️.

11.11.2. Filehandles and File System Transitions 🧾

There are a number of ways filehandles can be handled across a file system transition 🎀. These can be divided into two broad classes, depending on whether the two file systems involved in the transition share enough state to give some sort of continuity of file system handling 🧠.

When there is no such cooperation in filehandle assignment, the two file systems are reported as being in different handle classes 🚧. In this case, all filehandles are assumed to expire as part of the file system transition. Note that this behavior does not depend on the fh_expire_type attribute, and it supersedes the specification of the FH4_VOL_MIGRATION bit, which only affects behavior when fs_locations_info is not available 💥.

When there is cooperation in filehandle assignment, the two file systems are reported as being in the same handle classes ✨. In this case, persistent filehandles remain valid after the file system transition, while volatile filehandles (excluding those that are only volatile due to the FH4_VOL_MIGRATION bit) are subject to expiration on the target server 🌟.

11.11.3. Fileids and File System Transitions ✨

okay so in NFSv4.0, nobody really dealt with whether fileids stay the same when a file system transitions 😭. The vibe was that if one vendor made both copies with some kind of image copy, the fileids would match, but in a multi-vendor transition they probably wouldn't. That's a mess, especially for a client that doesn't know how the server handles transitions 🫠. And ngl, fileid is not a REQUIRED attribute, but tons of servers support it and a bunch of client APIs lean on it, so it matters 💅.

Here's the tea 🍵. Clients themselves might be totally fine with a fileid changing during a transition, but applications can see the fileid (through stat, for example) 👀. So an app could work perfectly when nothing transitions, or when it only moves between instances from one vendor, and then totally fall apart if a multi-vendor transition lands at the wrong moment 💀.

Keeping the same fileids across multiple server vendors has been seen as pretty hard. There's still work to do, but honestly, part of the difficulty is self-inflicted 🙈. Servers have usually tied the fileid to the inode number, which is the thing used to look up the file. That gets messy when you migrate between vendors, because giving the same index to a file might just not be possible 😤. Here's the bestie-level insight: a fileid doesn't have to help you find the file. It only has to be unique within its file system ✨. So servers that store the fileid as its own piece of metadata, separate from the index value, can keep the fileid steady during a migration, and that makes a truly transparent migration event possible 🦋.

Wherever servers can keep fileids continuous, they should, and the client should be able to figure out that continuity is there and act on it 🌸. The info about whether fileids carry over (or don't) across a transition is shown by whether the file systems are in the same fileid class 🎀.

Now the not-so-fun part 😬. If consistent fileids don't exist across a transition (either because they aren't continuous or because fileid isn't supported on one of the instances), AND there are no reliable filehandles across the transition (either no filehandle continuity or volatile filehandles), the client literally cannot verify that the files it was using before are the same objects 😱. It's forced to assume that nothing got renamed. Unless something guarantees that (like the file system being read-only), apps may run into problems 💀. So only use setups like this when the problems they could cause are things you can live with 🫶.

11.11.4. Fsids and File System Transitions 🧚

fsids are generally only unique on a single server, so they'll probably change during a file system transition 🌈. Clients should not show the fsids they get from the server to applications, since they may not be globally unique and they may change during a transition event 🙅‍♀️. Apps are happiest when they're shielded from transitions as much as possible 💖.

Usually, one source file system turns into one target file system. But there's a special provision for splitting one source file system into several target file systems, and you turn that on with the FSLI4F_MULTI_FS flag 🍓.

11.11.4.1. File System Splitting 🔥

When a file system transition happens and fs_locations_info says the file system might be split into multiple file systems (through the FSLI4F_MULTI_FS flag), the client SHOULD do GETATTRs to find the fsid of every known object inside the file system that's transitioning, so it can figure out the new file system boundaries 🗺️.

Clients might decide to keep the fsids that existing apps already see by mapping all the fsids of the descendant file systems back to the common fsid of the original file system 🎀.

Splitting can happen on a transition between file systems in the same fileid class, because fileids are unique within the source file system, which means they'll stay unique in each target file system too 💜.

11.11.5. The Change Attribute and File System Transitions 💅

The change attribute is defined as server-specific, so change attributes fetched from one server are normally assumed to be invalid on another ✋. That assumption is annoying because it would invalidate every cached change attribute and force a refetch of everything 😩. Even worse, since there's no guaranteed continuity for the change attribute, getting the same value back on a refetch doesn't tell you anything about whether the object changed. That identical value could just be a side effect of a modified file using a different change attribute construction algorithm, where the new algorithm happens to produce the same value by coincidence 🤯.

When the two file systems have consistent change attribute formats, and that's reported to the client by putting them in the same change class, the client may assume continuity of change attribute construction and handle the situation exactly as it would without any file system transition ✨🦋.

11.11.6. Write Verifiers and File System Transitions 🌸

During a file system transition, the two file systems might be working together to handle unstably written data 🤝. Clients can tell if that's happening by checking whether the two file systems belong to the same write-verifier class 🔍. If they do, write verifiers returned by one system can be compared with the ones returned by the other, and you can skip superfluous writes 🙌.

If two file systems are in different write-verifier classes, a verifier generated by one must not be compared to one provided by the other 🚫. Instead, the two verifiers should be treated as not equal, even when their values are identical 🙅‍♀️.

11.11.7. READDIR Cookies and Verifiers and File System Transitions 🦋

During a file system transition, the two file systems might handle READDIR cookies and verifiers in the same way 🫶. Clients can check this by seeing whether the two file systems belong to the same readdir class ✨. If they do, a readdir class, READDIR cookies, and verifiers from one system will be recognized by the other, and a READDIR operation that started on one server can be validly continued on the other just by handing the second file system the cookie and verifier that a READDIR done on the first one returned 🎀.

If two file systems are in different readdir classes, any READDIR cookie and verifier generated by one is not valid on the second and must not be presented to that server by the client 🙈. The client should act as if the verifier were rejected 💀.

11.11.8. File System Data and File System Transitions ✨

okay so when a client is using multiple replicas at the same time or one after another, the apps using them are gonna expect either the same data or data that looks like the normal changes other clients make to a file system 💅 (metadata matching to the degree that fs_locations_info says). ngl, the NFSv4.1 protocol doesn't generally spell out how the data in one replica relates to the data in another 🙈. It's totally possible to present file systems as replicas when their data is so different that some apps have a hard time switching between them 😭. So the namespace is usually built so apps can pick their own level of support: one spot might list a varied mix of replicas, while another spot only counts the up-to-date ones 🌸. The protocol does define three special cases that the server specifies and clients can rely on 🫶:

When multiple replicas are used simultaneously by a client (see the FSLIB4_CLSIMUL definition within fs_locations_info), they must designate the same data. Where file systems are writable, a change made on one instance must be visible on all instances at the same time, no matter whether the instance being asked is the one where the change was made 🦋. This lets a client use these replicas together without any special adaptation to there being multiple replicas, except adapting to the fact that locks obtained on one replica are kept separately (i.e., under a different client ID) 💜. In this case, locks (whether share reservations or byte-range locks) and delegations obtained on one replica are immediately reflected on all replicas, in the sense that access from all other servers is prevented no matter which replica is used 🔥. However, because the servers are not required to treat two associated client IDs as representing the same client, it is best to access each file using only a single client ID ⭐.
When one replica is designated as the successor instance to another existing instance after the return of NFS4ERR_MOVED (i.e., the case of migration), the client may depend on the fact that all changes written to stable storage on the original instance are written to stable storage of the successor (uncommitted writes are dealt with in Section 11.11.6 above) 🌈.
Where a file system is not writable but represents a read-only copy (possibly periodically updated) of a writable file system, clients have similar requirements about how updates get propagated 🍓. They may need a guarantee that any change visible on the original file system instance must be immediately visible on any replica before the client transitions access to that replica, so that a client switching to a replica never sees any reversion in file system state 👑. The specific means of this guarantee varies based on the value of the fss_type field that is reported as part of the fs_status attribute (see Section 11.18) ✨. Since these file systems are presumed to be unsuitable for simultaneous use, there is no specification of how locking is handled; in general, locks obtained on one file system will be separate from those on others 🧚. Since these are expected to be read-only file systems, this is not likely to pose an issue for clients or applications 💖.

When none of these special situations applies, there is no basis within the protocol for the client to make assumptions about the contents of a replica file system or its relationship to previous file system instances 🙈. So switching between nominally identical read-write file systems wouldn't be possible, because either the client doesn't use the fs_locations_info attribute, or the server doesn't support it 💀.

11.11.9. Lock State and File System Transitions 🔒

while a client is accessing a file system, it obtains locks that the server enforces, and those locks may stop other clients from doing actions that conflict with them 🦋.

When access is transferred between replicas, clients need to be assured that the actions disallowed by holding these locks cannot have happened during the transition 😤. This can be ensured by the methods below. Unless at least one of these is implemented, clients won't be assured of continuity of lock possession across a migration event 🚨:

Providing the client an opportunity to re-obtain his locks via a per-fs grace period on the destination server, which denies all clients using the destination file system the chance to obtain new locks that conflict with those held by the transferred client, for as long as that client has not completed its per-fs grace period 🎀. Because the lock reclaim mechanism was originally defined to support server reboot, it implicitly assumes that filehandles will, upon reclaim, be the same as those at open 🧠. In the case of migration, this requires that the source and destination servers use the same filehandles, as evidenced by using the same server scope (see Section 2.10.4) or by showing this agreement using fs_locations_info (see Section 11.11.2 above) 💅.

Note that such a grace period can be implemented without interfering with the ability of non-transferred clients to obtain new locks while it's going on 🌸. As long as the destination server is aware of the transferred locks, it can tell apart requests for new locks that contrast with existing locks from those that don't, which lets it handle such client requests without referring to the ongoing grace period 🧚.

Locking state can be transferred as part of the transition by providing Transparent State Migration, as described in Section 11.12 🦋.

Of these, Transparent State Migration gives clients the smoother experience, since there's no need to go through a reclaim process before new locks can be obtained 🍓. However, it requires a greater degree of inter-server coordination 😮‍💨. In general, the servers taking part in migration are free to provide either facility 💜. However, when the filehandles can differ across the migration event, Transparent State Migration is the only available means of providing the needed functionality 🔥.

It should be noted that these two methods are not mutually exclusive, and a server might well provide both 👑. In particular, if some circumstance prevents a specific lock from being transferred transparently, the destination server can allow it to be reclaimed by implementing a per-fs grace period for the migrated file system ✨.

okay bestie, this one is serious 😭. Although it's possible for a client reclaiming state to misrepresent its state in the same fashion as described in Section 8.4.2.1.1, most implementations that provide for such reclamation in the case of file system transitions will have the ability to detect such misrepresentations 🛡️. This limits the ability of unauthenticated clients to execute denial-of-service attacks in these circumstances 💖. Nevertheless, the rules stated in Section 8.4.2.1.1 regarding principal verification for reclaim requests apply in this situation as well 🫶.

Typically, implementations that support file system transitions will have extensive information about the locks to be transferred 🌈. This is because of the following:

Since failure is not involved, there is no need to store locking information in persistent storage 💅.
There is no need, as there is in the failure case, to update multiple repositories containing locking state to keep them in sync 🙌. Instead, there is a one-time communication of locking state from the source to the destination server 🦋.
Providing this information avoids potential interference with existing clients using the destination file system, by denying them the ability to obtain new locks during the grace period 🌸.

When such detailed locking information (not necessarily including the associated stateids) is available:

It is possible to detect reclaim requests that attempt to reclaim locks that did not exist before the transfer, rejecting them with NFS4ERR_RECLAIM_BAD (Section 15.1.9.4) 💀.
It is possible, when dealing with non-reclaim requests, to determine whether they conflict with existing locks, eliminating the need to return NFS4ERR_GRACE (Section 15.1.9.2) on non-reclaim requests 🧠.

It is possible for implementations of grace periods in connection with file system transitions not to have detailed locking information available at the destination server. In that case, the security situation is exactly as described in Section 8.4.2.1.1 🥺.

11.11.9.2. Leases and File System Transitions ✨

okay so, when a client renews its lease it might not be sending requests for a file system that already got moved to another server 👀. That happens because of how lease renewal works. The client renews the lease tied to all file systems whenever it sends a request on an associated session, no matter which file system that request is actually about 🧚.

Now, if some locking state might've moved to the new server, the client needs to find out about the lease relocation before that lease expires 😭. Here's how it finds out: SEQUENCE returns the status bit SEQ4_STATUS_LEASE_MOVED if responsibility for any of the renewed locking state got transferred to a new server. That bit keeps showing up until the client gets an NFS4ERR_MOVED error for each file system where locking state was relocated 🦋.

When the client sees SEQ4_STATUS_LEASE_MOVED, it should go through each file system on that server that has locking state for it and do an operation on it. Simplest move? Reference all the file systems ✨. The real requirement is that it must reference every file system that had locking state where that state has moved. Once each of those file systems returns NFS4ERR_MOVED, the server clears the indication. The client can stop checking once the indication is cleared 💅, but only if it has gotten a reply to every outstanding SEQUENCE request on all of its sessions with that server. After that, there are no other file systems with moved locking state, so it's all good.

Also, a client can use GETATTR of fs_status (or fs_locations_info) on all of the file systems to grab absence indications in one request or a few 🙈, since absent file systems won't cause an error in this context. It still has to do an operation that gets NFS4ERR_MOVED on each file system, though, in order to clear SEQ4_STATUS_LEASE_MOVED.

Once the set of file systems with transferred locking state is known, the client can follow the normal process to get the new server info (through the fs_locations and fs_locations_info attributes) and renew that lease on the new server. The exception is when fs_locations_info shows that no state could have been transferred. If the state wasn't transferred transparently, the client gets NFS4ERR_STALE_CLIENTID from the new server, as described above, and then it can reclaim locks the same way it does after a server failure 🔥.

11.11.9.3. Transitions and the Lease_time Attribute 🎀

So the client can manage its lease properly during a file system transition, the destination server has to set up the right values for the lease_time attribute.

When state is transferred transparently, that state should include the correct lease_time value. The lease_time on the destination server must never be less than the one on the source, because a smaller value would make a lease the source server granted expire too early 💀. When the transition is transparent, the client doesn't have to refetch lease_time and can keep using the value it fetched from the source server ✨.

If state hasn't been transferred transparently, which happens either because the servers show different eir_server_scope strings or because the client ID gets rejected by the new server, the client should fetch the value of lease_time on the new (i.e., destination) server and use it for later locking requests. The server still has to respect a grace period at least as long as the lease_time on the source server 😤. That gives clients enough time to reclaim their locks before potentially conflicting non-reclaimed locks get granted.

11.12. Transferring State upon Migration 🍓

When the transition is the result of a server-initiated decision to move access, and the source and destination servers have set up the right cooperation, the following is possible:

Transfer locking state from the source to the destination server in a way similar to Transparent State Migration in NFSv4.0, as described in [69]. Server responsibilities are described in Section 11.14.2.
Transfer session state from the source to the destination server. Server responsibilities for making that transfer happen are described in Section 11.14.3.

How the client figures out which of these transfer events happened is covered in Section 11.13.

11.12.1. Transparent State Migration and pNFS 🌈

When pNFS is involved, the protocol can support:

Migration of the Metadata Server (MDS), leaving the Data Servers (DSs) in place.
Migration of the file system as a whole, including the MDS and associated DSs.
Replacement of one DS by another.
Migration of a pNFS file system to one where pNFS isn't used.
Migration of a file system not using pNFS to one where layouts are available.

Note that migration, per se, only involves transferring the MDS function. Servicing of a layout can move from one data server to another, but that doesn't happen through the file system location attributes. The MDS can make those transfers by recalling or revoking existing layouts and granting new ones on a different data server 🧚.

Migration of the MDS function is directly supported by Transparent State Migration. Layout state will normally be transparently transferred, just like the other state 💜. So Transparent State Migration gives you a framework where, with the right inter-MDS data transfer, one MDS can be swapped in for another.

Migration of the file system function as a whole can be done by recalling all layouts as part of the initial phase of the migration. That means I/O goes through the MDS during the migration, and new layouts can be granted once the client is talking to the new MDS. An MDS can also pull this off by revoking all layouts as part of Transparent State Migration, as long as the client gets notified about the loss of locking state 🫶.

To allow migration to a file system where pNFS isn't supported, clients need to be ready for a situation where layouts aren't available or supported on the destination file system. In that case they send direct I/O requests to the destination server instead of depending on layouts being available 💅.

Replacing one DS with another isn't addressed by migration as such, but an MDS can effect it by recalling the layouts for the DS being replaced and issuing new ones to be served by the successor DS.

Migration may move a file system from a server that doesn't support pNFS to one that does. To adapt to that, clients that support pNFS but work fine without it should check for pNFS support when a file system is migrated, and be ready to use pNFS when support is available on the destination 🦋.

11.13. Client Responsibilities When Access Is Transitioned 👑

For a client to respond to an access transition, it must become aware of it. Section 11.13.1 covers the ways that can happen: indications that a specific file system access path has transitioned, and situations where extra activity is needed to determine the set of file systems that have been migrated. Section 11.13.2 finishes the discussion of how that set of migrated file systems might be determined. Sections 11.13.3 through 11.13.5 cover how the client should deal with each transition it becomes aware of, either directly or through migration discovery 🧠.

These terms describe client activities:

"Transition recovery" refers to the process of restoring access to a file system on which NFS4ERR_MOVED was received.
"Migration recovery" refers to the subset of transition recovery that applies when the file system has migrated to a different replica.
"Migration discovery" refers to the process of determining which file system(s) have been migrated. It's necessary to avoid a situation where leases could expire when a file system isn't accessed for a long period of time 😰, since a client unaware of the migration might be referencing an unmigrated file system and not renewing the lease associated with the migrated file system.

11.13.1. Client Transition Notifications ✨

okay bestie, so when the network path a client uses to reach a file system changes, there are a few status signals the client has to deal with 🦋

If someone tries to use or return a filehandle inside a file system that's no longer reachable at the address it was previously reached through, the error NFS4ERR_MOVED comes back 🙈

filehandles are allowed to be used anyway when the client is interrogating a file system location attribute. that way the client can figure out where a new replica lives or what its new network access path is 🧚

this condition sticks around on later attempts to access that file system. the only way the client can dodge the error is to stop accessing the file system at its old server location and go use it at a different address where it's now available 😭

whenever a client sends a SEQUENCE operation to a server that generated state held on that client and tied to a file system that's no longer accessible on that server, the response carries the status bit SEQ4_STATUS_LEASE_MOVED, which means there's been a lease migration 👑

this continues until the client acknowledges the notification by fetching a file system location attribute for the file system whose network access path is changing. if there are multiple such file systems, a location attribute needs to be fetched for each one. the location attribute for all migrated file systems needs to be fetched to clear the condition. even after the condition is cleared, the client still needs to use the location information to access the file system at its new location, so leases don't get needlessly expired 🫶

Unlike NFSv4.0, where these two conditions are both errors and therefore mutually exclusive, in NFSv4.1 the client can (and often will) get both indications on the same request. so implementations need to figure out how to coordinate the necessary recovery actions when both show up in the response to the same request 🔥 it should be noted that when processing an NFSv4 COMPOUND, the server will normally decide whether SEQ4_STATUS_LEASE_MOVED is set before it determines which file system will be referenced or whether NFS4ERR_MOVED is returned 💅

since these indications aren't mutually exclusive in NFSv4.1, these are the possible combos when a COMPOUND is issued:

the COMPOUND status is NFS4ERR_MOVED, and SEQ4_STATUS_LEASE_MOVED is asserted.

transition recovery is required here. it's possible migration discovery is needed too, but it's likely only the accessed file system has transitioned. either way, addressing NFS4ERR_MOVED is necessary so the rejected requests can be processed on the target, so dealing with it will typically take priority over migration discovery 🌸

the COMPOUND status is NFS4ERR_MOVED, and SEQ4_STATUS_LEASE_MOVED is clear.

transition recovery is also required in this case. it's clear that migration discovery isn't needed to find file systems that have migrated, other than the one returning NFS4ERR_MOVED. this result can show up with a referral, or with a migration that has no associated locking state. it can also happen when an access path transition other than migration occurs within the same server. there's no need to set SEQ4_STATUS_LEASE_MOVED there, since the lease stays associated with the current server even though the access path changed 🦋

the COMPOUND status is not NFS4ERR_MOVED, and SEQ4_STATUS_LEASE_MOVED is asserted.

no transition recovery activity is required on the file system(s) the request accessed. however, to prevent avoidable lease expiration, migration discovery needs to be done 🍓

the COMPOUND status is not NFS4ERR_MOVED, and SEQ4_STATUS_LEASE_MOVED is clear.

neither transition-related activity nor migration discovery is required. easy breezy 🌈

note that the specified actions only need to be taken if they aren't already going on 🧠 for example, when NFS4ERR_MOVED is received while accessing a file system that's already undergoing transition recovery, the client just waits for that recovery to finish. the receipt of SEQ4_STATUS_LEASE_MOVED only needs to start migration discovery for a server if discovery isn't already underway for that server 💜

the fact that a lease-migrated condition doesn't cause an error in NFSv4.1 has a bunch of important consequences ✨ besides the two indications not being mutually exclusive (as we just covered), there are a number of issues that matter when you're implementing migration discovery, which is discussed in Section 11.13.2 🎀

because SEQ4_STATUS_LEASE_MOVED isn't an error condition, file systems whose access paths haven't changed can still be successfully accessed on a given server, even when recovery is necessary for other file systems on that same server. so access can happen while:

the migration discovery process is happening for that server.
the transition recovery process is happening for other file systems connected to that server.

11.13.2. Performing Migration Discovery ✨

okay so migration discovery can totally happen in the same context as transition recovery 💅. That means recovery for each migrated file system can get kicked off as soon as it's discovered 🦋. Or it can run in its own separate migration discovery thread 🧵, so discovery can happen in parallel with one or more instances of transition recovery 🌸.

Either way, a lease-migrated indication doesn't come back as an error, so other access to file systems on the server can keep going normally 💖. That means more of these indications might show up, which raises the question of how to deal with them 🤔. In general:

No action needs to be taken for indications received by any threads that are doing migration discovery, since continuing that work will address the issue 🫶.
In other cases where migration discovery is already happening, nothing more needs to be done to respond to these lease migration indications, as long as you can be certain that the migration discovery process would deal with them. More on that below 👀.
For indications received in all other contexts, the right move is to start migration discovery (or make sure it's running) for the file systems associated with the server IP address that returned the indication 🚀.

Now here's the potential drama 😬. Things get tricky when the migration discovery process is near completion but still operating. You should not ignore a SEQ4_STATUS_LEASE_MOVED indication if the discovery process isn't able to respond to additional migrating file systems without extra help 🙅. Another complication is that a lease-migrated indication may reflect the server's state at the time the SEQUENCE operation was processed, which can be different from what's in effect when the response arrives ⏰. New migration events can happen at any time, and a SEQ4_STATUS_LEASE_MOVED indication can reflect a situation from a considerable time before it's received. So special care needs to be taken to make sure SEQ4_STATUS_LEASE_MOVED indications don't get inappropriately ignored 😭.

A useful approach is to use separate externally-visible migration discovery states for each server 🗂️. These values could represent the possible states of the migration discovery process for a server:

Non-operation, where migration discovery is not being performed 💤.
Normal operation, where there is an ongoing scan for migrated file systems 🔍.
Completion/verification of migration discovery processing, where the possible completion of migration discovery processing needs to be verified ✅.

With that framework, migration discovery processing goes like this:

While in the normal-operation state, the discovery thread fetches, for successive file systems known to the client on the server being worked on, a file system location attribute plus the fs_status attribute 📥.
If the fs_status attribute says the file system is a migrated one (i.e., fss_absent is true, and fss_type != STATUS4_REFERRAL), then a migrated file system has been found 🎯. In this situation, the fetch of the file system location attribute has likely cleared one of the file systems contributing to the lease-migrated indication 🧹.
When that happened, the thread can't know whether the lease-migrated indication has actually been cleared. So it enters the completion/verification state and issues a COMPOUND to see if the SEQ4_STATUS_LEASE_MOVED indication has been cleared 🔎.
When the discovery process is in the completion/verification state, if other requests get a lease-migrated indication, they note that it was received 📝. Later, the existence of such indications is used when the request completes, as described below 🔜.

When the request used in the completion/verification state completes:

If a lease-migrated indication is returned, the discovery continues normally 🏃‍♀️. Note that this is so even if all file systems have been traversed, since new migrations could have occurred while the process was going on 🌀.
Otherwise, if there is any record that other requests saw a lease-migrated indication while the request was occurring, that record is cleared, and the verification request is retried 🔁. The discovery process remains in the completion/verification state 🫧.
If there have been no lease-migrated indications, the work of migration discovery is considered completed, and it enters the non-operating state 🎉. Once it enters this state, subsequent lease-migrated indications will trigger a new migration discovery process 💫.

It should be noted that the process described above is not guaranteed to terminate 🙈. A long series of new migration events might keep delaying the clearing of the SEQ4_STATUS_LEASE_MOVED indication 😵. To prevent unnecessary lease expiration, it is appropriate for clients to use the discovery of migrations to effect lease renewal immediately 💞, rather than waiting for the clearing of the SEQ4_STATUS_LEASE_MOVED indication when the complete set of migrations is available.

Lease discovery needs to be provided as described above 📣. This ensures that the client discovers file system migrations soon enough to renew its leases on each destination server before they expire ⏳. Non-renewal of leases can lead to loss of locking state 💀. While the consequences of such loss can be ameliorated through implementations of courtesy locks, servers are under no obligation to do so, and a conflicting lock request may mean that a lock is revoked unexpectedly 😱. Clients should be aware of this possibility 👀.

11.13.3. Overview of Client Response to NFS4ERR_MOVED 🦋

this section lays out a way that a client receiving NFS4ERR_MOVED can do transition recovery by using a new server or server endpoint, if one is available 🌈. As part of that process, it will determine:

Whether the NFS4ERR_MOVED indicates migration has occurred, or whether it indicates another sort of file system access transition as discussed in Section 11.10 above 🤔.
In the case of migration, whether Transparent State Migration has occurred 🧳.
Whether any state has been lost during the process of Transparent State Migration 💔.
Whether sessions have been transferred as part of Transparent State Migration 🎀.

During the first phase of this process, the client goes through file system location entries to find the initial network address it will use to continue access to the file system or its replacement 🗺️. For each location entry that the client examines, the process has five steps:

Performing an EXCHANGE_ID directed at the location address. This operation is used to register the client owner (in the form of a client_owner4) with the server, to obtain a client ID to be used subsequently to communicate with it, to obtain that client ID's confirmation status, and to determine server_owner4 and scope for the purpose of determining if the entry is trunkable with the address previously being used to access the file system (i.e., that it represents another network access path to the same file system and can share locking state with it) 🧠.
Making an initial determination of whether migration has occurred. The initial determination will be based on whether the EXCHANGE_ID results indicate that the current location element is server-trunkable with that used to access the file system when access was terminated by receiving NFS4ERR_MOVED. If it is, then migration has not occurred 🙅‍♀️. In that case, the transition is dealt with, at least initially, as one involving continued access to the same file system on the same server through a new network address 🛣️.
Obtaining access to existing session state or creating new sessions 🔐. How this is done depends on the initial determination of whether migration has occurred, and can be done as described in Section 11.13.4 below in the case of migration, or as described in Section 11.13.5 below in the case of a network address transfer without migration 📚.
Verifying the trunking relationship assumed in step 2 as discussed in Section 2.10.5.1 ✔️. Although this step will generally confirm the initial determination, it is possible for verification to invalidate the initial determination of network address shift (without migration) and instead determine that migration had occurred 😮. There is no need to redo step 3 above, since it will be possible to continue use of the session established already 🙌.
Obtaining access to existing locking state and/or re-obtaining it 🔑. How this is done depends on the final determination of whether migration has occurred, and can be done as described below in Section 11.13.4 in the case of migration, or as described in Section 11.13.5 in the case of a network address transfer without migration 🗝️.

Once the initial address has been determined, clients are free to apply an abbreviated process to find additional addresses trunkable with it (clients may seek session-trunkable or server-trunkable addresses depending on whether they support client ID trunking) 🧩. During this later phase of the process, further location entries are examined using the abbreviated procedure specified below 🏃:

A: Before the EXCHANGE_ID, the fs name of the location entry is examined, and if it does not match that currently being used, the entry is ignored. Otherwise, one proceeds as specified by step 1 above.
B: In the case that the network address is session-trunkable with one used previously, a BIND_CONN_TO_SESSION is used to access that session using the new network address. Otherwise, or if the bind operation fails, a CREATE_SESSION is done 🔗.
C: The verification procedure referred to in step 4 above is used. However, if it fails, the entry is ignored and the next available entry is used 🙃.

11.13.4. Obtaining Access to Sessions and State after Migration ✨

okay bestie, so when migration has happened, recovery starts with figuring out whether Transparent State Migration happened too 🔍 and that call is based on the client ID that EXCHANGE_ID hands back plus its reported confirmation status 💅

If the client ID is an unconfirmed client ID not previously known to the client, then Transparent State Migration has not occurred 🙅‍♀️
If the client ID is a confirmed client ID previously known to the client, then any transferred state would have been merged with an existing client ID representing the client to the destination server. this is the state merger case 🌀 Transparent State Migration might or might not have occurred here, and figuring that out gets put off until sessions are established and the client is ready to start state recovery
If the client ID is a confirmed client ID not previously known to the client, then the client can conclude that the client ID was transferred as part of Transparent State Migration 🦋 this is the transferred client ID case. Transparent State Migration has occurred, although some state might have been lost 😭

Once the client ID is in hand, the client needs access to sessions to keep talking to the new server 🫶 In any case where Transparent State Migration has occurred, a session might have been transferred too. so after doing the EXCHANGE_ID, clients can issue a BIND_CONN_TO_SESSION to hook the transferred session up to a connection to the new server. if that fails, it's the sign that the session wasn't transferred and a new session needs to be created to take its place 🧚

Sometimes a BIND_CONN_TO_SESSION can succeed even when no session migration happened 👀 If state merger took place, the associated client ID may already have had a set of existing sessions, and the session ID of one of them could be the same as one that might have been migrated. in that case, the BIND_CONN_TO_SESSION might succeed anyway. then the client will get sequence errors when the slot sequence values it's using don't fit the new session. when that happens, the client can create a new session and stop using the existing one 💖

Once the client has figured out the initial migration status and knows there was a shift to a new server, it needs to re-establish its locking state, if possible 🔐 To make that work without losing the guarantees locking normally provides, the destination server needs to implement a per-fs grace period in every case where lock state was lost, including the cases where Transparent State Migration wasn't implemented. each client whose locking state was transferred to the new server gets the full length of the grace period to reclaim its locks, counting from the moment its locks were transferred ⏳

Clients need to deal with these cases:

In the state merger case, the server might not have attempted Transparent State Migration, which means state could've been lost without it showing up in the SEQ4_STATUS bits 😬 To check whether that happened, the client can use TEST_STATEID to see whether the stateids created on the source server are still accessible on the destination server. once a single stateid is found to have transferred successfully, the client can conclude that Transparent State Migration was begun, and any failure to transport all of the stateids will show up in the SEQ4_STATUS bits. otherwise, Transparent State Migration has not occurred.
In a case where Transparent State Migration has not occurred, the client can use the per-fs grace period the destination server provides to reclaim the locks that were held on the source server 🔄
In a case where Transparent State Migration has occurred and no lock state was lost (as shown by the SEQ4_STATUS flags), no lock reclaim is necessary 🎉
In a case where Transparent State Migration has occurred and some lock state was lost (as shown by the SEQ4_STATUS flags), existing stateids need to be checked for validity using TEST_STATEID, and reclaim gets used to re-establish any that weren't transferred 🛠️

For all of the cases above, RECLAIM_COMPLETE with an rca_one_fs value of TRUE needs to happen before the file system gets normal use, including before obtaining new locks for it. this applies even if no locks were lost and nothing needed reclaiming 💅✨

11.13.5. Obtaining Access to Sessions and State after Network Address Transfer

The case where there's a transfer to a new network address without migration is kinda similar to Section 11.13.4 above, since you still need to get access to the sessions and locking state you need 🦋 The details are simpler though, and they depend on what type of trunking exists between the address that returned NFS4ERR_MOVED and the one the transfer is going to 🌸

To make a session available for use, a BIND_CONN_TO_SESSION should be used to get access to the session that was in use before. only if that fails should a CREATE_SESSION be done. this mirrors what happens in Section 11.13.4 above, but there's a big difference: keeping the session isn't purely optional here. it depends on the type of trunking 👑

Access to the right locking state will generally need no actions beyond access to the session. however, the SEQ4_STATUS bits still need checking for lost locking state, including whether locks need reclaiming after a server reboot, because locking state can always get lost 🙈

11.14. Server Responsibilities Upon Migration

When a file system migrates and the client connects to the destination server, that server needs to be able to give the client continued access to the files it had open on the source server 🎀 There are two ways to do that:

By providing an fs-specific grace period, which lets the client reclaim its locks in a way that's similar to what would've happened when recovering from a server restart. see Section 11.14.1 for the full scoop 🧁
By implementing Transparent State Migration, possibly alongside session migration, so the server can give the client immediate access to the state that was built up on the source server, now on the destination server ⭐

These features get discussed separately in Sections 11.14.2 and 11.14.3, which cover Transparent State Migration and session migration, respectively 📚

All of these features can involve transferring lock-related information between the source and destination servers 🔁 In some cases that transfer is a necessary part of the implementation, and in others it's a helpful implementation aid that servers might or might not use 🤷‍♀️ The subsections below talk about the information that would be transferred, but they don't define the specifics of the transfer protocol. that's left as an implementation choice, although standards in this area could get developed later 🌈

11.14.1. Server Responsibilities in Effecting State Reclaim after Migration

In this case, the destination server doesn't need to know anything about the locks held on the source server 🧠 It relies on the clients to accurately report (through reclaim operations) the locks they previously held, and it doesn't allow new locks to be granted on migrated file systems until the grace period expires. the ban on new locks applies to every client accessing these file systems, but grace period expiration happens separately for each migrated client 💜

During this grace period, clients get the chance to use reclaim operations to obtain locks for file system objects within the migrated file system, the same way they do when recovering from a server restart ✨ Servers typically rely on clients to accurately report their locks, though they have the option of checking these requests. if the clients only reclaim locks that were held on the source server, no conflict can arise 🫧 Once the client has reclaimed its locks, it signals that it's done by performing a RECLAIM_COMPLETE that specifies rca_one_fs as TRUE 🏁

It isn't necessary for the source and destination servers to cooperate to transfer information about locks, but implementations are well advised to consider transferring these useful bits of info:

If information about the set of clients that have locking state for the transferred file system is made available, the destination server can end the grace period once all of those clients have reclaimed their locks. that lets normal locking activity resume earlier than it otherwise would have 🏃‍♀️💨
Locking summary information for individual clients (at various possible levels of detail) can catch some cases where clients don't accurately represent the locks they held on the source server 🕵️‍♀️🔎

11.14.2. Server Responsibilities in Effecting Transparent State Migration ✨

okay so the main job of the source server in Transparent State Migration is handing the destination server a full description of every piece of locking state tied to the file system being moved 🧚. Along with the client id string and verifier, the source server needs to provide this for each stateid:

The stateid, including the current sequence value.
The associated client ID.
The handle of the associated file.
The type of the lock, such as open, byte-range lock, delegation, or layout.
For locks such as opens and byte-range locks, there will be information about the owner(s) of the lock.
For recallable/revocable lock types, the current recall status needs to be included.
For each lock type, there will be associated type-specific information. For opens, this will include share and deny mode, while for byte-range locks and layouts, there will be a type and a byte-range.

This info will most probably be organized by client id string on the destination server, so each client gets the right context when it shows up to say hi 🫶. Impersonation drama (a client pretending to be another by presenting that client's client id string) can be handled with NFSv4.1 state protection features, as described in Section 21 💅.

Next up, locks that get revoked or lost during file system migration. Locks that appear lost during migration will be reclaimed by the client, so the servers have to make sure that locks revoked right before or right after migration don't sneakily get reclaimed when continuity of possession can't be assured 😭.

For locks lost on the source but whose loss has not yet been acknowledged by the client (by using FREE_STATEID), the destination must be aware of this loss so that it can deny a request to reclaim them.
For locks lost on the destination after the state transfer but before the client's RECLAIM_COMPLETE is done, the destination server should note these and not allow them to be reclaimed.

One more server responsibility: a stateid can't be transferred transparently if it conflicts with an existing stateid the client holds that's associated with a different file system. There are two valid choices here 💜:

Treat the transfer, as in NFSv4.0, as one without Transparent State Migration. In this case, conflicting locks cannot be granted until the client does a RECLAIM_COMPLETE, after reclaiming the locks it had, with the exception of reclaims denied because they were attempts to reclaim locks that had been lost.
Implement Transparent State Migration, except for the lock with the conflicting stateid. In this case, the client will be aware of a lost lock (through the SEQ4_STATUS flags) and be allowed to reclaim it.

When transferring state between the source and destination, the issues discussed in Section 7.2 of [69] still need attention. In this case, NFS4ERR_DELAY may still be necessary in NFSv4.1, as it was in NFSv4.0, to stop locking state from changing while it's being transferred. See Section 15.1.1.3 for info on appropriate client retry approaches if NFS4ERR_DELAY is returned 🔁.

There are a number of important differences in the NFS4.1 context 👑:

The absence of RELEASE_LOCKOWNER means that the one case in which an operation could not be deferred by use of NFS4ERR_DELAY no longer exists.
Sequencing of operations is no longer done using owner-based operation sequence numbers. Instead, sequencing is session-based.

So when sessions are not transferred, the techniques discussed in Section 7.2 of [69] are adequate and won't be discussed further 🌸.

11.14.3. Server Responsibilities in Effecting Session Transfer

The basic job of the source server in effecting session transfer is handing the destination server a description of the current state of each slot within the session, including the following:

The last sequence value received for that slot.
Whether there is cached reply data for the last request executed and, if so, the cached reply.

When sessions are transferred, there are some tricky issues around making the transferred state unmodifiable while it's being gathered up and sent to the destination server 🙈:

A single session may be used to access multiple file systems, not all of which are being transferred.
Requests made on a session may, even if rejected, affect the state of the session by advancing the sequence number associated with the slot used.

So when the file system state might otherwise be considered unmodifiable, the client might have any number of in-flight requests, each capable of changing session state. These can be a few different types:

Those requests that were processed on the migrating file system before migration began.
Those requests that received the error NFS4ERR_DELAY because the file system being accessed was in the process of being migrated.
Those requests that received the error NFS4ERR_MOVED because the file system being accessed had been migrated.
Those requests that accessed the migrating file system in order to obtain location or status information.
Those requests that did not reference the migrating file system.

Note that the history of any particular slot is likely to include a number of these request classes. If a migrated session is used by file systems other than the one migrated, class 5 requests may be common and may be the last request processed for many slots 🍓.

Session state can change even after the locking state has been fixed as part of the migration process. That means the session state the client knows about could be different from what the destination server has, which necessarily reflects the session state on the source server at an earlier time. To figure out how to deal with this, it helps to separate two sorts of behavioral consequences of the choice of initial sequence ID values 🧠:

The error NFS4ERR_SEQ_MISORDERED is returned when the sequence ID in a request is neither equal to the last one seen for the current slot nor the next greater one.

Since it's hard to agree on the correct last sequence value at the point of migration, the server may need to show some forbearance when the sequence ID is one that would be considered unacceptable if session migration weren't involved.

Returning the cached reply for a previously executed request when the sequence ID in the request matches the last value recorded for the slot.

In cases where an error is returned and there's no chance any non-idempotent operation was executed, it may not be necessary to stick to this as strictly as would be proper without session migration. For example, the fact that NFS4ERR_DELAY was returned may not help the client in any real way, while the fact that NFS4ERR_MOVED was returned by the source server may not matter once the request is reissued and sent to the destination server.

An important point is that the spec needs to take note of all potential COMPOUNDs, even if they're unlikely in practice 🤓. For example, a COMPOUND is allowed to access multiple file systems and might perform non-idempotent operations in some of them before accessing a file system being migrated. A COMPOUND may also return considerable data in the response before being rejected with NFS4ERR_DELAY or NFS4ERR_MOVED, and may in addition be marked as sa_cachethis. However, if the client and server follow the rules in Section 15.1.1.3, there's no possibility of non-idempotent operations being spuriously reissued after receiving an NFS4ERR_DELAY response.

To deal with all of this, a destination server MAY do any of the following when implementing session transfer 💖:

Avoid enforcing any sequencing semantics for a particular slot until the client has established the starting sequence for that slot on the destination server.
For each slot, avoid returning a cached reply returning NFS4ERR_DELAY or NFS4ERR_MOVED until the client has established the starting sequence for that slot on the destination server.
Until the client has established the starting sequence for a particular slot on the destination server, avoid reporting NFS4ERR_SEQ_MISORDERED or returning a cached reply that contains either NFS4ERR_DELAY or NFS4ERR_MOVED and consists solely of a series of operations where the response is NFS4_OK until the final error.

Because of all the stuff above, including the rules for handling NFS4ERR_DELAY in Section 15.1.1.3, the destination server can respond properly to SEQUENCE operations from the client by using the three policies below 🎀:

Not responding with NFS4ERR_SEQ_MISORDERED for the initial request on a slot within a transferred session, because the destination server cannot be aware of requests the client made after the server handoff but before the client became aware of the shift. In cases where NFS4ERR_SEQ_MISORDERED would normally have been reported, the request is to be processed normally as a new request.
Replying as it would for a retry whenever the sequence matches the one transferred by the source server, even though this wouldn't provide retry handling for requests issued after the server handoff. This works under the assumption that, when such requests are issued, they will never be responded to in a state-changing fashion, which makes retry support for them unnecessary 🦋.
Once a non-retry SEQUENCE is received for a given slot, using that as the basis for further sequence checking, with no further reference to the sequence value transferred by the source server.

11.15. Effecting File System Referrals 🧚

Okay so bestie, here's where things get spicy. A referral kicks in when the client runs into an absent file system and the server hands over one or more alternate spots through the fs_locations or fs_locations_info attributes 🦋. The client will usually get an NFS4ERR_MOVED error, go grab the location info, and then head over to access the file system on a different server. Even so, it keeps its logical spot in the original namespace, which is giving main character energy 👑✨. Referrals are different from migration events because they only happen when the client has never touched that file system before, so there's nothing to transition 🌸. And referrals can only kick in when an absent file system is found at its root 💅.

The examples coming up are kinda artificial ngl 🙈. A real client won't usually do a multi-component lookup, since it'll have cached info about the upper levels of the name hierarchy. But these examples are picked so the required behavior is crystal clear and fits in a small number of requests, without getting lost in how specific clients choose to cache stuff 🎀.

11.15.1. Referral Example (LOOKUP) 🔍

Let's say the following COMPOUND gets sent in an environment where /this/is/the/path is absent from the target server 🌈. There could be a few reasons for that. The file system might have moved, or the target server might be mostly, or only, around to refer clients to the servers where the various file systems actually live 🧚.

PUTROOTFH
LOOKUP "this"
LOOKUP "is"
LOOKUP "the"
LOOKUP "path"
GETFH
GETATTR (fsid, fileid, size, time_modify)

So under these circumstances, here's what you get 💖:

PUTROOTFH --> NFS_OK. The current fh is now the root of the pseudo-fs.
LOOKUP "this" --> NFS_OK. The current fh is for /this and is within the pseudo-fs.
LOOKUP "is" --> NFS_OK. The current fh is for /this/is and is within the pseudo-fs.
LOOKUP "the" --> NFS_OK. The current fh is for /this/is/the and is within the pseudo-fs.
LOOKUP "path" --> NFS_OK. The current fh is for /this/is/the/path and is within a new, absent file system, but... the client will never see the value of that fh 🙈.
GETFH --> NFS4ERR_MOVED. Fails because the current fh is in an absent file system at the start of the operation, and the specification makes no exception for GETFH 💀.
GETATTR (fsid, fileid, size, time_modify). Not executed, because the failure of the GETFH stops processing of the COMPOUND 😭.

Okay so, with the GETFH failing, the client has the job of figuring out where the root of the absent file system is and where to find that file system, meaning the server and the path relative to that server's root fh 🔥. Here's the tea: in this example, the client didn't get filehandles and attribute info (like fsid) for the intermediate directories, so it can't be sure where the absent file system actually starts 🫶. It could totally be that /this/is/the is the root of the moved file system, and the reason the lookup of "path" succeeded is that the file system wasn't absent during that operation but got moved between the last LOOKUP and the GETFH, since COMPOUND is not atomic 🦋. Even if we had the fsids for all the intermediate directories, we'd still have no way of knowing that /this/is/the/path was the root of a new file system, since we don't have its fsid yet 🌸.

So to get the info we need, let's re-send the chain of LOOKUPs with GETFHs and GETATTRs to at least grab the fsids, so we can be sure where the file system boundaries are ✨. The client could fetch fs_locations_info at the same time, but in most cases the client will already have a good guess about where the boundaries are (because of where NFS4ERR_MOVED was and wasn't received), so grabbing fs_locations_info isn't necessary 💅.

OP01: PUTROOTFH --> NFS_OK
- Current fh is the root of the pseudo-fs.
OP02: GETATTR(fsid) --> NFS_OK
- Just for completeness. Normally, clients will know the fsid of the pseudo-fs as soon as they establish communication with a server 🎀.
OP03: LOOKUP "this" --> NFS_OK
OP04: GETATTR(fsid) --> NFS_OK
- Get the current fsid to see where the file system boundaries are. The fsid will be the one for the pseudo-fs in this example, so no boundary 🧚.
OP05: GETFH --> NFS_OK
- Current fh is for /this and is within the pseudo-fs.
OP06: LOOKUP "is" --> NFS_OK
- Current fh is for /this/is and is within the pseudo-fs.
OP07: GETATTR(fsid) --> NFS_OK
- Get the current fsid to see where the file system boundaries are. The fsid will be the one for the pseudo-fs in this example, so no boundary 🌈.
OP08: GETFH --> NFS_OK
- Current fh is for /this/is and is within the pseudo-fs.
OP09: LOOKUP "the" --> NFS_OK
- Current fh is for /this/is/the and is within the pseudo-fs.
OP10: GETATTR(fsid) --> NFS_OK
- Get the current fsid to see where the file system boundaries are. The fsid will be the one for the pseudo-fs in this example, so no boundary 💜.
OP11: GETFH --> NFS_OK
- Current fh is for /this/is/the and is within the pseudo-fs.
OP12: LOOKUP "path" --> NFS_OK
- Current fh is for /this/is/the/path and is within a new, absent file system, but...
- The client will never see the value of that fh 🙈.
OP13: GETATTR(fsid, fs_locations_info) --> NFS_OK
- We're getting the fsid so we know where the file system boundaries are. In this operation, the fsid will be different than that of the parent directory (which was retrieved in OP10). Note that the fsid we're given won't necessarily be preserved at the new location. That fsid might be different, and in fact the fsid we have for this file system might be a valid fsid of a different file system on that new server 🦋.
- In this particular case, we're pretty sure that what moved is /this/is/the/path rather than /this/is/the, since we have the fsid of the latter and it's the one for the pseudo-fs, which presumably can't move. However, in other examples we might not have this kind of info to rely on (e.g., /this/is/the might be a non-pseudo file system separate from /this/is/the/path), so we need some other reliable source of info on the boundary of the file system that moved. If, for example, the file system /this/is had moved, we'd have a case of migration rather than referral, and once the boundaries of the migrated file system were clear we could fetch fs_locations_info 💅.
- We're fetching fs_locations_info because getting an NFS4ERR_MOVED at this point means it's most likely a referral and we need the destination. Even if /this/is/the is a file system that has migrated, we'd still need the location info for that file system 🫶.
OP14: GETFH --> NFS4ERR_MOVED
- Fails because the current fh is in an absent file system at the start of the operation, and the specification makes no exception for GETFH. Note that this means the server will never send the client a filehandle from within an absent file system 💀.

Okay so, given all that, the client knows where the root of the absent file system is (/this/is/the/path) by noting where the change of fsid happened (between "the" and "path") ✨. The fs_locations_info attribute also tells the client the actual location of the absent file system, so the referral can go ahead 👑. The server gives the client the bare minimum of info about the absent file system, so there's very little room for conflicts between what the referring server says and what the file system's home says 🌸. No filehandles and very few attributes come from the referring server, and the client can treat whatever it does receive as transient info whose whole job is to enable the referral 🎀.

Next up: the rest of the referral stuff and onward 🦋💜

11.15.2. Referral Example (READDIR) 🎀

okay so another place a client can bump into referrals is when it does a READDIR on a directory where some of the subdirectories are the roots of absent file systems 👀 this one's lowkey sneaky 🫶

Suppose that directory gets read like this:

PUTROOTFH
LOOKUP "this"
LOOKUP "is"
LOOKUP "the"
READDIR (fsid, size, time_modify, mounted_on_fileid)

In this case, because rdattr_error is not requested, fs_locations_info is not requested, and some of the attributes cannot be provided, the result will be an NFS4ERR_MOVED error on the READDIR 😭 here's the detailed tea:

PUTROOTFH --> NFS_OK. The current fh is at the root of the pseudo-fs. 🌸
LOOKUP "this" --> NFS_OK. The current fh is for /this and is within the pseudo-fs. 🦋
LOOKUP "is" --> NFS_OK. The current fh is for /this/is and is within the pseudo-fs. ✨
LOOKUP "the" --> NFS_OK. The current fh is for /this/is/the and is within the pseudo-fs. 💜
READDIR (fsid, size, time_modify, mounted_on_fileid) --> NFS4ERR_MOVED. Note that the same error would have been returned if /this/is/the had migrated, but it is returned because the directory contains the root of an absent file system. 💀

So now suppose we re-send with rdattr_error:

PUTROOTFH
LOOKUP "this"
LOOKUP "is"
LOOKUP "the"
READDIR (rdattr_error, fsid, size, time_modify, mounted_on_fileid)

The results will be:

PUTROOTFH --> NFS_OK. The current fh is at the root of the pseudo-fs. 🌸
LOOKUP "this" --> NFS_OK. The current fh is for /this and is within the pseudo-fs. 🦋
LOOKUP "is" --> NFS_OK. The current fh is for /this/is and is within the pseudo-fs. ✨
LOOKUP "the" --> NFS_OK. The current fh is for /this/is/the and is within the pseudo-fs. 💜
READDIR (rdattr_error, fsid, size, time_modify, mounted_on_fileid) --> NFS_OK. The attributes for the directory entry with the component named "path" will only contain rdattr_error with the value NFS4ERR_MOVED, together with an fsid value and a value for mounted_on_fileid. 🙈

Suppose we do another READDIR to get fs_locations_info (although we could have used a GETATTR directly, as in Section 11.15.1). 🧚

PUTROOTFH
LOOKUP "this"
LOOKUP "is"
LOOKUP "the"
READDIR (rdattr_error, fs_locations_info, mounted_on_fileid, fsid, size, time_modify)

The results would be:

PUTROOTFH --> NFS_OK. The current fh is at the root of the pseudo-fs. 🌸
LOOKUP "this" --> NFS_OK. The current fh is for /this and is within the pseudo-fs. 🦋
LOOKUP "is" --> NFS_OK. The current fh is for /this/is and is within the pseudo-fs. ✨
LOOKUP "the" --> NFS_OK. The current fh is for /this/is/the and is within the pseudo-fs. 💜
READDIR (rdattr_error, fs_locations_info, mounted_on_fileid, fsid, size, time_modify) --> NFS_OK. The attributes will be as shown below. 👑

The attributes for the directory entry with the component named "path" will only contain:

rdattr_error (value: NFS_OK)
fs_locations_info
mounted_on_fileid (value: unique fileid within referring file system)
fsid (value: unique value within referring server)

The attributes for entry "path" will not contain size or time_modify because these attributes are not available within an absent file system. 🥺

11.16. The Attribute fs_locations ✨

The fs_locations attribute is structured in the following way:

struct fs_location4 {
        utf8str_cis     server<>;
        pathname4       rootpath;
};

struct fs_locations4 {
        pathname4       fs_root;
        fs_location4    locations<>;
};

The fs_location4 data type is used to represent the location of a file system by providing a server name and the path to the root of the file system within that server's namespace. When a set of servers have corresponding file systems at the same path within their namespaces, an array of server names may be provided. An entry in the server array is a UTF-8 string and represents one of a traditional DNS host name, IPv4 address, IPv6 address, or a zero-length string. An IPv4 or IPv6 address is represented as a universal address (see Section 3.3.9 and [12]), minus the netid, and either with or without the trailing ".p1.p2" suffix that represents the port number. If the suffix is omitted, then the default port, 2049, SHOULD be assumed. A zero-length string SHOULD be used to indicate the current address being used for the RPC call. It is not a requirement that all servers that share the same rootpath be listed in one fs_location4 instance. The array of server names is provided for convenience. Servers that share the same rootpath may also be listed in separate fs_location4 entries in the fs_locations attribute. 🌈

The fs_locations4 data type and the fs_locations attribute each contain an array of such locations. Since the namespace of each server may be constructed differently, the "fs_root" field is provided. The path represented by fs_root represents the location of the file system in the current server's namespace, i.e., that of the server from which the fs_locations attribute was obtained. The fs_root path is meant to aid the client by clearly referencing the root of the file system whose locations are being reported, no matter what object within the current file system the current filehandle designates. The fs_root is simply the pathname the client used to reach the object on the current server (i.e., the object to which the fs_locations attribute applies). 🧭

When the fs_locations attribute is interrogated and there are no alternate file system locations, the server SHOULD return a zero-length array of fs_location4 structures, together with a valid fs_root. 💅

As an example, suppose there is a replicated file system located at two servers (servA and servB). At servA, the file system is located at path /a/b/c. At servB, the file system is located at path /x/y/z. If the client were to obtain the fs_locations value for the directory at /a/b/c/d, it might not necessarily know that the file system's root is located in servA's namespace at /a/b/c. When the client switches to servB, it will need to determine that the directory it first referenced at servA is now represented by the path /x/y/z/d on servB. To facilitate this, the fs_locations attribute provided by servA would have an fs_root value of /a/b/c and two entries in fs_locations. One entry in fs_locations will be for itself (servA) and the other will be for servB with a path of /x/y/z. With this information, the client is able to substitute /x/y/z for the /a/b/c at the beginning of its access path and construct /x/y/z/d to use for the new server. 🦋

Note that there is no requirement that the number of components in each rootpath be the same; there is no relation between the number of components in rootpath or fs_root, and none of the components in a rootpath and fs_root have to be the same. In the above example, we could have had a third element in the locations array, with server equal to "servC" and rootpath equal to "/I/II", and a fourth element in locations with server equal to "servD" and rootpath equal to "/aleph/beth/gimel/daleth/he". 🍓

The relationship between fs_root to a rootpath is that the client replaces the pathname indicated in fs_root for the current server for the substitute indicated in rootpath for the new server. 🔥

For an example of a referred or migrated file system, suppose there is a file system located at serv1. At serv1, the file system is located at /az/buky/vedi/glagoli. The client finds that object at glagoli has migrated (or is a referral). The client gets the fs_locations attribute, which contains an fs_root of /az/buky/vedi/glagoli, and one element in the locations array, with server equal to serv2, and rootpath equal to /izhitsa/fita. The client replaces /az/buky/vedi/glagoli with /izhitsa/fita, and uses the latter pathname on serv2. 🫶

Thus, the server MUST return an fs_root that is equal to the path the client used to reach the object to which the fs_locations attribute applies. Otherwise, the client cannot determine the new path to use on the new server. 🚨

Since the fs_locations attribute lacks information defining various attributes of the various file system choices presented, it SHOULD only be interrogated and used when fs_locations_info is not available. When fs_locations is used, information about the specific locations should be assumed based on the following rules. 🧠

The following rules are general and apply irrespective of the context. ✨

All listed file system instances should be considered as of the same handle class, if and only if, the current fh_expire_type attribute does not include the FH4_VOL_MIGRATION bit. Note that in the case of referral, filehandle issues do not apply since there can be no filehandles known within the current file system, nor is there any access to the fh_expire_type attribute on the referring (absent) file system. 🌸
All listed file system instances should be considered as of the same fileid class if and only if the fh_expire_type attribute indicates persistent filehandles and does not include the FH4_VOL_MIGRATION bit. Note that in the case of referral, fileid issues do not apply since there can be no fileids known within the referring (absent) file system, nor is there any access to the fh_expire_type attribute. 🦋
All file system instances servers should be considered as of different change classes. 💜

For other class assignments, handling of file system transitions depends on the reasons for the transition:

When the transition is due to migration, that is, the client was directed to a new file system after receiving an NFS4ERR_MOVED error, the target should be treated as being of the same write-verifier class as the source. 🧚
When the transition is due to failover to another replica, that is, the client selected another replica without receiving an NFS4ERR_MOVED error, the target should be treated as being of a different write-verifier class from the source. 👑

The specific choices reflect typical implementation patterns for failover and controlled migration, respectively. Since other choices are possible and useful, this information is better obtained by using fs_locations_info. When a server implementation needs to communicate other choices, it MUST support the fs_locations_info attribute. 🎀

See Section 21 for a discussion on the recommendations for the security flavor to be used by any GETATTR operation that requests the fs_locations attribute. 🔐

11.17. The Attribute fs_locations_info ✨

okay so the fs_locations_info attribute is basically the glow-up of fs_locations 💅. fs_locations is going to keep existing and keep being supported, but fs_locations_info gives clients way more tea about alternative file system locations, including extra network paths to replicas that are already in use and extra replicas. If the server doesn't support fs_locations_info, the client can still use fs_locations to get a smaller slice of the info 🎀. And a server that supports fs_locations_info MUST support fs_locations as well 🫶.

There's extra data in fs_locations_info that fs_locations just doesn't have 👀:

Attribute continuity information. This lets a client pick a replica that matches what the apps accessing the data need for transparency, and it unlocks optimizations because the server promises attribute continuity 🦋. Like, if a file's change attribute is continuous across multiple replicas, the client doesn't have to throw out that file's cache when it switches to another replica. Iconic efficiency, no notes 💜.
File system identity information. This shows when several replicas look like the same target file system from the client's point of view, so they can be used interchangeably, without disruption, as separate synchronized replicas of the same file data 🌸.

Note that having two replicas with common identity information is totally different from having two (trunked) paths to the same replica. Don't mix those up, bestie 🙈.

Suitability info. This is stuff that tells you which replicas fit what the client is doing. Lots of apps need a completely up-to-date copy (think anything that writes ✍️), while others are fine with the most up-to-date copy that's reasonably available 🌈.
Server-derived preference info. The server can say which replicas it prefers, which helps with load balancing, and the client still gets the whole file system list to fall back on if the primary fails 🧚.

Structurally, fs_locations_info is kind of a mirror of fs_locations ⭐. The top-level structure (fs_locations_info4) holds the whole attribute, including the root pathname of the file system, plus an array of lower-level structures. Those lower-level structures (fs_locations_item4) describe replicas that share a common rootpath on their respective servers. Each fs_locations_item4 has a specific pathname and info about one or more individual network access paths. At the lowest level, fs_locations_server4 holds per-server-replica info alongside the file system location entry 💖. That per-server-replica info includes a mostly opaque array, fls_info, where specific pieces of data live at the specific indices listed below.

Two fs_location_server4 entries inside different fs_location_item4 structures are never trunkable 🚫. Two entries inside the same fs_location_item4 might or might not be trunkable 🤷‍♀️. Entries that are trunkable will have identical identity information, but as mentioned above, that doesn't work in reverse.

The attribute will always have at least one fs_locations_server entry ✨. Usually there's an entry with the FS4LIGF_CUR_REQ flag set, but in the case of a referral, there won't be any entry with that flag set 🫠.

Also, be aware that the fs_locations_info attributes that different servers return for various replicas may not match 😤. One server might know about replicas that other servers don't know about. Compatibility attributes might differ too. Filehandles might be the same class going from replica A to replica B but not going the other way around. That can happen because the filehandles really are the same, but replica B's server implementation may not have a way to note and report that equivalence 💅.

The fs_locations_info attribute is made of a root pathname (fli_fs_root, just like fs_root in the fs_locations attribute) together with an array of fs_location_item4 structures. Each fs_location_item4 is made of a root pathname (fli_rootpath) and an array (fli_entries) of fs_locations_server4 elements. Here's the whole thing defined in one place 👑:

   /*
    * Defines an individual server access path
    */
   struct  fs_locations_server4 {
           int32_t         fls_currency;
           opaque          fls_info<>;
           utf8str_cis     fls_server;
   };

   /*
    * Byte indices of items within
    * fls_info: flag fields, class numbers,
    * bytes indicating ranks and orders.
    */
   const FSLI4BX_GFLAGS            = 0;
   const FSLI4BX_TFLAGS            = 1;

   const FSLI4BX_CLSIMUL           = 2;
   const FSLI4BX_CLHANDLE          = 3;
   const FSLI4BX_CLFILEID          = 4;
   const FSLI4BX_CLWRITEVER        = 5;
   const FSLI4BX_CLCHANGE          = 6;
   const FSLI4BX_CLREADDIR         = 7;

   const FSLI4BX_READRANK          = 8;
   const FSLI4BX_WRITERANK         = 9;
   const FSLI4BX_READORDER         = 10;
   const FSLI4BX_WRITEORDER        = 11;

   /*
    * Bits defined within the general flag byte.
    */
   const FSLI4GF_WRITABLE          = 0x01;
   const FSLI4GF_CUR_REQ           = 0x02;
   const FSLI4GF_ABSENT            = 0x04;
   const FSLI4GF_GOING             = 0x08;
   const FSLI4GF_SPLIT             = 0x10;

   /*
    * Bits defined within the transport flag byte.
    */
   const FSLI4TF_RDMA              = 0x01;

   /*
    * Defines a set of replicas sharing
    * a common value of the rootpath
    * within the corresponding
    * single-server namespaces.
    */
   struct  fs_locations_item4 {
           fs_locations_server4    fli_entries<>;
           pathname4               fli_rootpath;
   };

   /*
    * Defines the overall structure of
    * the fs_locations_info attribute.
    */
   struct  fs_locations_info4 {
           uint32_t                fli_flags;
           int32_t                 fli_valid_for;
           pathname4               fli_fs_root;
           fs_locations_item4      fli_items<>;
   };

   /*
    * Flag bits in fli_flags.
    */
   const FSLI4IF_VAR_SUB           = 0x00000001;

   typedef fs_locations_info4 fattr4_fs_locations_info;

When it's supported, fs_locations_info can be requested for absent file systems without causing NFS4ERR_MOVED to be returned 🙌. It's generally expected to be available for both present and absent file systems, even if there's only one fs_locations_server4 entry (the current, present file system), or two entries (the previous location of an absent file system, which was just referenced, plus its successor location) 🧚‍♀️. Servers are strongly urged to support this attribute on all file systems if they support it on any file system 📣.

The server can get the data for fs_locations_info in lots of ways, like the administrator specifying it, current protocols for moving data between replicas, or protocols that haven't been developed yet 🔮. NFSv4.1 only defines how the server presents this information to the client 💜.

11.17.1. The fs_locations_server4 Structure

The fs_locations_server4 structure has the following items, in addition to the fls_server field. That field specifies a network address (or set of addresses) to use for accessing the file system 🌸. Both fls_currency and fls_info describe the file system replica itself, so they should not be different when there are multiple fs_locations_server4 structures, each specifying a network path to the chosen replica, for the same replica 🎀.

When these values are different in two fs_locations_server4 structures, the client has no basis for picking one over the other, so it's best off ignoring both entries, whether they apply to migration, replication, or referral 🤷‍♀️. When there are more than two such entries, majority voting can be used to exclude one erroneous entry from consideration 🗳️. If trunking info is provided for a replica that's currently being accessed, the extra trunked addresses can be ignored while access continues on the address currently in use, even if the entry for that path might be considered invalid 🧼.

Currency (fls_currency). This is how up-to-date the file system is, in seconds, relative to the master copy ⏰. A negative value means the server can't give any reasonably useful value here. A value of zero means the file system is the actual writable data, or a reliably coherent and fully up-to-date copy 💯. Positive values show how out-of-date this copy can normally be before it's considered for update. That value isn't a guarantee that updates will always happen on the required schedule. It's just a hint about how far behind the most up-to-date copy this data is expected to be 🫶.
Instance info (fls_info). This is a counted array of one-byte values with info about the particular file system instance 🧩. It covers general flags, transport capability flags, file system equivalence class info, and selection priority info. The encoding is discussed below 📚.
Server string (fls_server). For the replica currently being accessed (via GETATTR), a zero-length string MAY be used to indicate the current address being used for the RPC call 🦋. The fls_server field can also be an IPv4 or IPv6 address, formatted the same way as an IPv4 or IPv6 address in the "server" field of the fs_location4 data type (see Section 11.16) 🌈.

With the exception of the transport-flag field (at offset FSLI4BX_TFLAGS within the fls_info array), all of the data defined in this specification applies to the replica specified by the entry, not to the specific network path used to access it 🛣️. How data in extensions is classified is discussed below 💅.

The data in the fls_info array is made of 8-bit data items, with constants that give the offsets within the array for the various values describing this particular file system instance 🔢. This style of definition was picked over explicit XDR structure definitions for these values, for a few reasons 🧠.

The kinds of data in fls_info, which represent flags, file system classes, and priorities among sets of file systems holding the same data, fit fine in 8 bits 🙌. Even where there might be more than 256 such file system instances, having more than 256 distinct classes or priorities is pretty unlikely 😌.
Defining each specific data item in XDR would limit expandability. Any extension would need yet another attribute, which leads to clumsy specification and implementation 😩. In the NFSv4 extension model that was in effect when fs_locations_info was designed (the one described in RFC 5661 [66]), any Standards Track extension to the data in fls_info would have needed a brand-new minor version 🙅‍♀️. Nobody wants that, periodt 💀.

The set of fls_info data is allowed to grow in a future minor version, or in a Standards Track RFC within a single minor version 💖. The server SHOULD NOT send, and the client MUST NOT use, any indices inside the fls_info array or flag bits that aren't defined in Standards Track RFCs. No sneaking in new stuff, periodt 💅

Because of the new extension model from RFC 8178 [67], and because nobody references the individual fls_info items in the XDR, these practices should be followed when you extend or change the fls_info data within one minor version ✨

All extensions need to be described by Standards Track documents. Those documents don't have to be marked as updating RFC 5661 [66] or this document 🎀
It needs to be clear whether the info in any new data items applies to the replica named in the entry, or to the specific network paths named in the entry 🧚
There needs to be a reliable way to tell whether the server knows about the extension 🔍. That could be based on the length of the fls_info array, but it's way more flexible to provide fs-scope or server-scope attributes that say which extensions are provided 💜

This encoding scheme can also work for multi-byte numeric values, even though none are defined yet 🌸. If extensions come through Standards Track RFCs, multi-byte quantities get encoded as a range of bytes with a range of indices, and each byte is read in big-endian byte order 🍓. The index assignments also have to make sure the quantities don't cross XDR word boundaries, so no sneaky straddling 🙈

The fls_info array currently holds:

Two 8-bit flag fields. One is for general file system characteristics, and the other is reserved for transport-related capabilities 🦋
Six 8-bit class values that define the file system equivalence classes (more on those below) 🌈
Four 8-bit priority values that control file system selection (also explained below) ⭐

The general file system characteristics flag, found at byte index FSLI4BX_GFLAGS, has these bits defined:

FSLI4GF_WRITABLE says the file system target is writable, so clients that need to write can select it. When the current file system instance is writable and is in the same simultaneous use class (the value at index FSLI4BX_CLSIMUL) that the client was previously writing to, it must incorporate any committed write made on the source file system instance into its data. Check out Section 11.11.6 for the write-verifier class stuff 🔥. Not setting this flag on a file system that turns out to be writable is harmless. But turning it on for a read-only file system can cause problems, because clients that pick a migration or replication target based on this flag might end up unable to write 😭
FSLI4GF_CUR_REQ says this replica is the one the request was made on. Only one server entry can have this flag set, and in a referral no entry has it set. This flag might be set even if the request came in on a network access path that's different from any of the ones in the current entry 👑
FSLI4GF_ABSENT says this entry corresponds to an absent file system replica. It can only be set if FSLI4GF_CUR_REQ is also set. When both bits are set, the file system instance isn't usable, but the entry still tells you what kinds of continuity you get when switching from this replica to other possible replicas 🧚. Since this bit can only be true when FSLI4GF_CUR_REQ is true, you could work it out from the fs_status attribute. It's included here anyway as a convenience for the client 💅. An entry with this bit is still a true file system (just absent), so it doesn't show up in a referral. It only shows up when a file system has been accessed at this location and then migrated away 🫶
FSLI4GF_GOING says a replica is still available but shouldn't be used anymore. If the client is using it, the client should move to another file system instance in an orderly way, as quickly as it can 🏃‍♀️. File systems that are going out of service are expected to be announced as FSLI4GF_GOING some time before the service is actually lost 📣. The fli_valid_for value is expected to be small enough that clients can catch and act on scheduled events, but big enough that fetching the fs_locations_info values doesn't get expensive. Values on the order of ten minutes seem reasonable ⏰

When this flag shows up as part of a transition into a new file system, a client might move to another replica right away, or it might keep using the current file system and only transition when a migration event happens 🌸. Likewise, when this flag appears on a replica in a referral, clients would probably avoid being referred there whenever they have another choice 🙃

This flag, like the other fls_info items, applies to the replica and not to a particular path to that replica. When it's set, the indication is about a transition to a new replica, not to a different path to the same replica 💜

FSLI4GF_SPLIT says that when a transition happens from the current file system instance to this one, the replacement might be made up of multiple file systems. In that case, the client has to be ready for objects that were on the same file system before migration to end up on different ones afterward 🪄. FSLI4GF_SPLIT isn't incompatible with the file systems sharing the same fileid class. If a file system has a set of fileids that are unique within it, then each subset assigned to a smaller file system after migration won't have any conflicts inside that file system 🦋

When a file system gets split, a client will check the files it's still connected to (it's free to just forget its cached filehandles) 🧐. If the client remembers the directory filehandle for each open file, it can go upward with LOOKUPP to find the new file system boundaries 🗺️. In the case of a referral there won't be any such files, so these steps don't happen. Instead, a reference to part of the original file system that has now split off into other file systems will run into an fsid change and maybe another referral 🔁

Once the client realizes one file system has split into two, it can keep running applications from breaking by presenting the two file systems as one until a convenient point to recognize the change, like a restart 🛟. That requires a mapping from the server's fsids to the fsids the client sees, but the client already needs that for other reasons 💅. As noted above, the existing fileids in the two descendant file systems won't conflict. Making sure newly created files on the split file systems get non-conflicting fileids is the server's job (or the servers', if they're working together) 🤝. The server can encode filehandles so that filehandles made before the split can be told apart from the ones made after it, which lets the server figure out when the need to fake two file systems as one is over 🎀

Even though this flag can show up in a referral, it's usually not very interesting to the client, since the client isn't expected to know what's currently in the absent file system 🙈

The transport-flag field, at byte index FSLI4BX_TFLAGS, holds these bits related to the transport capabilities of the specific network path(s) in the entry 🚀:

FSLI4TF_RDMA says any specified network paths give NFSv4.1 clients access through an RDMA-capable transport ⚡

Attribute continuity and file system identity info get expressed by defining equivalence relations on the sets of file systems shown to the client 🧩. Each relation is written as a set of file system equivalence classes. For each relation, a file system has an 8-bit class number. Two file systems are in the same class if they both have identical non-zero class numbers, and zero counts as non-matching 🚫. Usually the client wants to know whether a given replica is identical to, or continuous with, the current one in some respect. But the info should also be available for whether two other replicas match in that respect 💖

The following fields give the file system's class numbers for the equivalence relations used to figure out the kind of file system transition. Check out Sections 11.9 through 11.14 and their subsections for the details on how to use this info 📚. Servers can assign these values however they like, as long as file system instances with the same value have the specified relationship to each other. Also, file systems with the specified relationship to each other share a common class value 💅. As each instance entry gets added, you can check its relationships to the entries already there. If you find one with the specified relationship, you can copy that entry's class value to the new entry. If there's no such previous entry, pick a new value for that byte index that hasn't been used yet, most likely by incrementing the last class value assigned for that index 🔢

The field at byte index FSLI4BX_CLSIMUL defines the simultaneous-use class for the file system 🫧
The field at byte index FSLI4BX_CLHANDLE defines the handle class for the file system 🎟️
The field at byte index FSLI4BX_CLFILEID defines the fileid class for the file system 🪪
The field at byte index FSLI4BX_CLWRITEVER defines the write-verifier class for the file system ✍️
The field at byte index FSLI4BX_CLCHANGE defines the change class for the file system 🔄
The field at byte index FSLI4BX_CLREADDIR defines the readdir class for the file system 📖

Server-specified preference info is also given through 8-bit values inside the fls_info array. These values provide a rank and an order (see below), and there's a separate set for read-only and writable file systems 🏆. The values get compared across different file systems to set the server-specified preference, and lower values mean "more preferred" 🥇✨

Okay so rank is basically the server's strict VIP list for clients 👑 Lower number means more preferred, bestie. Clients should try every replica with a given rank before they even look at a higher-rank one 💅 Only if all of those file systems are unavailable should the client move on to the higher ranks. Here's the tea though: setting a rank overrides what the client wants, so servers should be super conservative with it 🫶 especially when client communication characteristics aren't tightly controlled or even visible to the server 👀

Within a rank, the order value is the server's preference, and it guides the client when the client's own preferences aren't in charge 🌸 Lower order also means more preferred. If replicas are basically identical in every way, clients should defer to the server's order ✨ A client that looks at server latency as part of its choice is free to use that, but when the latency differences aren't significant, the server-specified order should guide the selection 💖

The field at byte index FSLI4BX_READRANK gives the rank value to be used for read-only access 📖
The field at byte index FSLI4BX_READORDER gives the order value to be used for read-only access 📖
The field at byte index FSLI4BX_WRITERANK gives the rank value to be used for writable access ✍️
The field at byte index FSLI4BX_WRITEORDER gives the order value to be used for writable access ✍️

Depending on how likely a given client is to need write access, you pick one of the rank and order pairs 🦋 The read rank and order should only be used if the client knows that only reading will ever be done, or if it's prepared to switch to a different replica the moment any write access is needed later 🙈

11.17.2. The fs_locations_info4 Structure ✨

The fs_locations_info4 structure is what encodes the fs_locations_info attribute, and it holds the following goodies 💜

The fli_flags field holds general flags that affect how this structure, and every fs_locations_item4 structure inside it, gets interpreted. The only flag defined so far is FSLI4IF_VAR_SUB. Any bits in fli_flags that aren't defined should always be returned as zero 🔢
The fli_fs_root field holds the pathname of the root of the current file system on the current server, just like it does in the fs_locations4 structure 🌳
fli_items is an array of fs_locations4_item structures with info about replicas of the current file system 🧺 Where the current file system is actually present (or has been present, so not a referral situation), one of the fs_locations_item4 structures will contain a fs_locations_server4 for the current server. That structure will have FSLI4GF_ABSENT set if the current file system is absent, which means normal access to it will return NFS4ERR_MOVED 🚪
The fli_valid_for field is a time in seconds for which it's reasonable for a client to use the fs_locations_info attribute without refetching ⏰ It's not a guarantee of validity, since servers can unexpectedly go out of service or become inaccessible for any number of reasons 😬 Clients are well-advised to refetch this info for an actively accessed file system every fli_valid_for seconds. This is especially important when replicas may go out of service in a controlled way, using the FSLI4GF_GOING flag to communicate the change 🚧 The server should set fli_valid_for to a value that lets well-behaved clients notice the FSLI4GF_GOING flag and make an orderly switch before the loss of service takes effect. If this value is zero, then no refetch interval is appropriate and the client doesn't need to refetch on any schedule 🫠 In the event of a transition to a new file system instance, a new value of the fs_locations_info attribute will be fetched at the destination. That may have a different fli_valid_for value, which the client should then use the same way as the previous one 🔄 Because a refetch causes information from all component entries to be refetched, the server will typically provide a low value for this field if any of the replicas are likely to go out of service in a short time frame 📉 Also, because the server can return NFS4ERR_MOVED to trigger the use of different paths, when alternate trunked paths are available there's generally no need to use low values of fli_valid_for for managing alternate paths to the same replica 🛤️

The FSLI4IF_VAR_SUB flag within fli_flags controls whether variable substitution is enabled. See Section 11.17.3 for an explanation of variable substitution 🧩

11.17.3. The fs_locations_item4 Structure 🎀

The fs_locations_item4 structure contains a pathname (in the field fli_rootpath) that encodes the path of the target file system replicas on the set of servers designated by the included fs_locations_server4 entries 🗺️ How exactly that target location gets specified depends on the value of the FSLI4IF_VAR_SUB flag within the associated fs_locations_info4 structure.

If this flag isn't set, then fli_rootpath simply designates the location of the target file system within each server's single-server namespace, just like the rootpath in the fs_location4 structure does 📍 When this bit is set, though, component entries of a certain form are subject to client-specific variable substitution. This allows a degree of namespace non-uniformity so the selection of client-specific file system targets can adapt to different client architectures or other characteristics 🧑‍💻

When substitution is in effect, a variable that begins with the string "${" and ends with the string "}" and contains a colon is replaced by the client-specific value associated with that variable 🪄 The client should use the string "unknown" when it has no value for such a variable. The pathname that results from these substitutions designates the target file system, so different clients may have different file systems corresponding to that location in the multi-server namespace 👯

As mentioned above, substituted pathname variables contain a colon. The part before the colon is to be a DNS domain name, and the part after is to be a case-insensitive alphanumeric string 🔤

Where the domain is "ietf.org", only variable names defined in this document or subsequent Standards Track RFCs are subject to substitution 📜 Organizations are free to use their own domain names to create their own sets of client-specific variables, to be subject to substitution. In cases where those variables are meant to be used more broadly than a single organization, publishing an Informational RFC that defines them is RECOMMENDED 📢

The variable ${ietf.org:CPU_ARCH} is used to denote the CPU architecture that object files are compiled for 🖥️ This specification doesn't limit the acceptable values (except that they must be valid UTF-8 strings), but values such as "x86", "x86_64", and "sparc" would be expected to be used in line with industry practice 💻

The variable ${ietf.org:OS_TYPE} is used to denote the operating system, and therefore the kernel and library APIs, that code might be compiled for 🐧 Again, this specification doesn't limit the acceptable values (except that they must be valid UTF-8 strings), but values such as "linux" and "freebsd" would be expected to be used in line with industry practice 😎

The variable ${ietf.org:OS_VERSION} is used to denote the operating system version, and therefore the specific details of versioned interfaces, that code might be compiled for 🏷️ This specification doesn't limit the acceptable values (except that they must be valid UTF-8 strings). However, combinations of numbers and letters with interspersed dots would be expected to be used in line with industry practice, and the details of the version format depend on the specific value of the variable ${ietf.org:OS_TYPE} it's used with 🔍

Using these variables could direct different clients to different file systems on the same server, as appropriate for particular clients 🎯 In cases where the target file systems are on different servers, a single server could act as a referral point, so that each valid combination of variable values designates a referral hosted on a single server, with the targets of those referrals spread across a number of different servers 🌐

Because namespace administration is affected by the values selected to substitute for the variables, clients should provide convenient means of determining what variable substitutions a client will implement, and where appropriate, provide means to control the substitutions to be used 🎛️ The exact means of doing this is outside the scope of this specification.

Although variable substitution is most suitable in the context of referrals, it may be used in the context of replication and migration too 🔁 If it's used in those contexts, the server must ensure that no matter what values the client presents for the substituted variables, the result is always a valid successor file system instance to the one the transition is occurring from. In other words, the data is identical or represents a later image of a writable file system 🫶

Note that when fli_rootpath is a null pathname (that is, one with zero components), the file system designated is at the root of the specified server, whether or not the FSLI4IF_VAR_SUB flag within the associated fs_locations_info4 structure is set 🌈

11.18. The Attribute fs_status ✨

okay bestie, picture this 🎀 you've got a bunch of copies of the same basic data floating around, and the info about where each copy came from and how they're related is literally the tea for giving apps consistent data 💅 no cap.

enum fs4_status_type {
        STATUS4_FIXED = 1,
        STATUS4_UPDATED = 2,
        STATUS4_VERSIONED = 3,
        STATUS4_WRITABLE = 4,
        STATUS4_REFERRAL = 5
};

struct fs4_status {
        bool            fss_absent;
        fs4_status_type fss_type;
        utf8str_cs      fss_source;
        utf8str_cs      fss_current;
        int32_t         fss_age;
        nfstime4        fss_version;
};

okay so fss_absent is the bool that tells you whether the file system is currently absent 🙈 It gets set if the file system was present before and now it's absent, or if it has never been present and the type is STATUS4_REFERRAL. When this boolean is set and the type is not STATUS4_REFERRAL, the rest of the fs4_status is showing the last valid info from when the file system was still present ✨

fss_type is the field that tells you what kind of file system image you're looking at 🦋 It's super important when you're using the version values to figure out the right succession of file system images. When fss_absent is set and the file system was previously present, the fss_type you get is whatever it was when the file was last present. There are five values, and they're all giving different vibes:

STATUS4_FIXED is a read-only image, meaning it will never change 🧊 It's allowed that after a migration or a switch to a different image you could access changed data, but inside this specific instance no change is allowed. The client can use this fact to cache aggressively 🧚
STATUS4_VERSIONED is updated externally, just like the STATUS4_UPDATED case, but here the server gives you a guarantee that it'll carefully update an associated version value. That way the client can protect itself from reading data from one version of the file system and then later reading data from an earlier version of that same file system 🫶 Check below for how this can be done.
STATUS4_UPDATED is an image that users can't update by writing to it, but it may be changed externally, usually because it's a periodically updated copy of another writable file system somewhere else 🌈 No version info is provided here, and the client doesn't have the responsibility of making sure the version only advances on a file system instance transition. Instead, it's the server's job to make sure the data presented after a file system instance transition is a proper successor image, and that it includes all changes the client has seen and any change made before all of those changes 👑
STATUS4_WRITABLE means the file system is an actual writable one ✍️ The client doesn't have to actually write to it, but once it does, it should not accept a transition to anything other than a writable instance of that same file system 💜
STATUS4_REFERRAL means the file system is absent and has never been present on this server 🍓

Note that in the STATUS4_UPDATED and STATUS4_VERSIONED cases, the server is responsible for properly handling locks that are inconsistent with external changes to delegations. If a server gives out delegations, they SHOULD be recalled before an inconsistent change is made to the data, and MUST be revoked if this is not possible. Similarly, if an OPEN is inconsistent with data that is changed (the OPEN has OPEN4_SHARE_DENY_WRITE/OPEN4_SHARE_DENY_BOTH and the data is changed), that OPEN SHOULD be considered administratively revoked. 🚨

The opaque strings fss_source and fss_current are a way of presenting info about where the file system image you're looking at came from 🕵️‍♀️ The client isn't supposed to do anything with this info except make it available to administrative tools. It's meant to be helpful when you're digging into possible problems with a file system image, especially when it's unclear whether the correct image is being accessed and, if not, how that image came to be made. This kind of diagnostic info matters because, as seems likely, copies of file systems get made in tons of different ways (simple user-level copies, file-system-level point-in-time copies, clones of the underlying storage) under all sorts of administrative arrangements. In those environments, figuring out how a set of data was constructed can be a total lifesaver 🧠

The opaque string fss_source is used to indicate the source of a given file system, and the expectation is that tools capable of creating a file system image propagate this info whenever possible 📦 It's understood that this might not always be possible, since a user-level copy can be thought of as creating a brand new data set and the tools used might have no way to pass this data along. When a file system is first created, it's desirable to associate it with data about how it was created, where it was created, who created it, and so on. Putting this in a human-readable string in this attribute helps applications and system admins, and it also makes the info available when the original file system is used to make later copies 🎀

The opaque string fss_current should provide whatever info is available about the source of the current copy 💖 That includes the tool that created it, any relevant parameters for that tool, the time the copy was made, the user who made the change, the server where the change was made, and so on. All of this info should be in a human-readable string 📝

The field fss_age gives an indication of how out-of-date the file system currently is compared to its ultimate data source (in case of cascading data updates) ⏳ This complements the fls_currency field of fs_locations_server4 (see Section 11.17) in the following way: the info in fls_currency gives a bound for how out of date the data in a file system might typically get, while the value in fss_age gives a bound on how out-of-date that data actually is. Negative values imply that no info is available. Zero means this data is known to be current ✅ A positive value means this data is known to be no older than that many seconds compared to the ultimate data source. Using this value, the client may be able to decide a data copy is too old, so it can go search for a newer version to use 🔍

The field fss_version provides a version identification in the form of a time value, so successive versions always have later time values ⌛ When the fs_type is anything other than STATUS4_VERSIONED, the server may provide such a value, but there's no guarantee it's valid, and clients won't use it except to add extra info to fss_source and fss_current 🤷‍♀️

When fss_type is STATUS4_VERSIONED, servers SHOULD provide a value of fss_version that progresses monotonically whenever any new version of the data is established 📈 This lets the client, if reliable image progression matters to it, fetch this attribute as part of each COMPOUND where data or metadata from the file system is used 🫶

When it's important to the client to make sure only valid successor images are accepted, it must make sure that it does not read data or metadata from the file system without updating its sense of the current state of the image 🙅‍♀️ This avoids the possibility that the fs_status the client holds is for an earlier image. That would cause the client to accept a new file system instance that is later than that but still earlier than the updated data the client read 😭

In order to accept valid images reliably, the client must do a GETATTR of the fs_status attribute that follows any interrogation of data or metadata within the file system in question. Often the most convenient way is to append such a GETATTR after all the other operations that reference a given file system 🧁 When errors occur between reading file system data and performing such a GETATTR, care must be exercised to make sure the data in question isn't used before the proper fs_status value is obtained. Along those lines, when an OPEN is done within such a versioned file system and the associated GETATTR of fs_status is not successfully completed, the open file in question must not be accessed until that fs_status is fetched 🚫

The procedure above ensures that, before using any data from the file system, the client has in hand a newly-fetched current version of the file system image 💅 When there are multiple values for multiple requests in flight, they can be resolved by assembling them into the required partial order (the elements should form a total order within that partial order) and using the last one 🧩 The client may then, when switching among file system instances, decline to use an instance that does not have an fss_type of STATUS4_VERSIONED, or whose fss_version field is earlier than the last one obtained from the predecessor file system instance 🦋

12. Parallel NFS (pNFS) 💖

12.1. Introduction ✨

okay so pNFS is an OPTIONAL feature within NFSv4.1 💅 and it lets clients go straight to the storage devices that actually hold the file data 🦋. when one NFSv4 server's file data is spread across multiple and/or higher-throughput storage devices (compared to what the server can push itself), file access can get way faster 🚀. the whole setup of multiple clients, one server, and multiple storage devices (where the server and clients can all reach every storage device) is shown in Figure 1 👇

       +-----------+
       |+-----------+                                 +-----------+
       ||+-----------+                                |           |
       |||           |        NFSv4.1 + pNFS          |           |
       +||  Clients  |<------------------------------>|   Server  |
        +|           |                                |           |
         +-----------+                                |           |
              |||                                     +-----------+
              |||                                           |
              |||                                           |
              ||| Storage        +-----------+              |
              ||| Protocol       |+-----------+             |
              ||+----------------||+-----------+  Control   |
              |+-----------------|||           |    Protocol|
              +------------------+||  Storage  |------------+
                                  +|  Devices  |
                                   +-----------+

                                  Figure 1

in this model, the clients, server, and storage devices all share the job of managing file access 🫶. compare that to NFSv4 without pNFS, where it's mostly the server's responsibility (with some of it handed to the client under strictly specified conditions) 👑. check out Section 12.2.5 for the Storage Protocol and Section 12.2.6 for the Control Protocol 🧚

pNFS shows up as OPTIONAL operations that manage protocol objects called 'layouts' (Section 12.2.7) 💜. each layout holds a byte-range and storage location info. layouts are managed kind of like NFSv4.1 data delegations: they're leased, recallable, and revocable 🫠. but layouts are their own thing and get manipulated with new operations. when a client holds a layout, it gets the ability to directly access the byte-range at the storage location in that layout 🎯

layouts also interact with other NFSv4.1 stuff like data delegations and byte-range locking. delegation issues are covered in Section 12.5.5, and byte-range locking issues are in Sections 12.2.9 and 12.5.1 🔐

12.2. pNFS Definitions 🌸

NFSv4.1's pNFS feature gives parallel data access to a file system that stripes its content across multiple storage servers 🌈. the first version of pNFS (part of NFSv4.1) splits file system protocol processing into two parts: metadata processing and data processing. data is the contents of regular files that get striped across storage servers. striping happens in at least two ways: file-by-file and, inside sufficiently large files, block-by-block 🧱. striped access to metadata by pNFS clients isn't provided in NFSv4.1, even though the file system back end of a pNFS server might stripe metadata. metadata is everything else, including the contents of non-regular files (like directories); see Section 12.2.1 🗂️. the metadata functionality is handled by an NFSv4.1 server that supports pNFS and the operations described in Section 18, and that server is called a metadata server (Section 12.2.2) 👑

the data functionality is handled by one or more storage devices, each accessed by the client via a storage protocol. a subset of NFSv4.1 (defined in Section 13.6) is one such storage protocol. new terms are added to the NFSv4.1 nomenclature and existing terms get clarified so pNFS can be described properly 📚

12.2.1. Metadata ✨

info about a file system object, like its name, location within the namespace, owner, ACL, and other attributes 📝. metadata can also include storage location info, and that will vary based on the underlying storage mechanism being used.

12.2.2. Metadata Server 🏠

an NFSv4.1 server that supports the pNFS feature 💻. there are a variety of architectural choices for the metadata server and how it uses the file system info held at the server. some servers only contain metadata for file objects living on the metadata server, while the file data sits on associated storage devices. other metadata servers hold both metadata and a varying degree of file data 🫧

12.2.3. pNFS Client 📲

an NFSv4.1 client that supports pNFS operations and at least one storage protocol for doing I/O to storage devices 🎧

12.2.4. Storage Device 💾

a storage device stores a regular file's data but leaves metadata management to the metadata server 🙌. it could be another NFSv4.1 server, an object-based storage device (OSD), a block device accessed over a System Area Network (SAN, e.g., either FiberChannel or iSCSI SAN), or some other entity 🛸

12.2.5. Storage Protocol 🎯

as noted in Figure 1, the storage protocol is the method the client uses to store and retrieve data directly from the storage devices 🪄

the NFSv4.1 pNFS feature is structured so a variety of storage protocols can be defined and used 🌷. one example is NFSv4.1 itself (documented in Section 13). other options are described elsewhere and include:

Block/volume protocols such as Internet SCSI (iSCSI) [56] and FCP [57]. The block/volume protocol support can be independent of the addressing structure of the block/volume protocol used, allowing more than one protocol to access the same file data and enabling extensibility to other block/volume protocols. See [48] for a layout specification that allows pNFS to use block/volume storage protocols.
Object protocols such as OSD over iSCSI or Fibre Channel [58]. See [47] for a layout specification that allows pNFS to use object storage protocols.

it's possible that various storage protocols are available to both client and server, but a client and server might not have a matching storage protocol available to them 🙈. because of this, the pNFS server MUST support normal NFSv4.1 access to any file accessible by the pNFS feature. this keeps NFSv4.1 client and server interoperable 🫶

12.2.6. Control Protocol 🔧

as noted in Figure 1, the control protocol is used by the exported file system between the metadata server and storage devices 🛠️. specifying those protocols is outside the scope of NFSv4.1. control protocols would handle things like allocating and deallocating storage, managing the state storage devices need for client access control, and (depending on the storage protocol) enforcing authentication and authorization so that the restrictions the metadata server enforces are also enforced by the storage device 🔒

no particular control protocol is REQUIRED by NFSv4.1 (no pressure 😌), but there are requirements placed on the control protocol for keeping attributes like modify time, the change attribute, and the end-of-file (EOF) position in line ⏰. note that if pNFS is layered over a clustered, parallel file system (e.g., PVFS [59]), the mechanisms that enable clustering and parallelism in that file system can be considered the control protocol 🧩

12.2.7. Layout Types 🏷️

a layout describes how a file's data maps to the storage devices that hold it 🗺️. a layout belongs to a specific layout type (data type layouttype4, see Section 3.3.13). the layout type allows variants to handle different storage protocols, like the block/volume [48], object [47], and file (Section 13) layout types. a metadata server, along with its control protocol, MUST support at least one layout type 💪. there's also a private sub-range of the layout type namespace defined, and values from that private range MAY be used for internal testing or experimentation (see Section 3.3.13) 🧪

as an example, the organization of the file layout type could be an array of tuples (e.g., device ID, filehandle), along with a definition of how the data is stored across the devices (e.g., striping) 🧵. a block/volume layout might be an array of tuples that store <device ID, block number, block count>, along with info about the block size and the associated file offset of the block number. an object layout might be an array of tuples <device ID, object ID> plus an extra structure (the aggregation map) that defines how the logical byte sequence of the file data gets serialized into the different objects. the actual layouts are typically way more complex than these simple examples 🤓

requests for pNFS-related operations will often specify a layout type 🎟️. examples of those operations are GETDEVICEINFO and LAYOUTGET. the responses include structures like a device_addr4 or a layout4, and each of those includes a layout type inside it. the layout type sent by the server MUST always be the same one requested by the client 🙅. when a server sends a response with a different layout type, the client SHOULD ignore the response and behave as if the server had returned an error response 💀

12.2.8. Layout 🧩

a layout defines how a file's data is organized on one or more storage devices 🪺. there are many potential layout types, and each one is differentiated by the storage protocol used to access data and by the aggregation scheme that lays out the file data on the underlying storage devices. a layout is precisely identified by the tuple <client ID, filehandle, layout type, iomode, range>, where filehandle refers to the filehandle of the file on the metadata server 🪪

it's important to define when layouts overlap and/or conflict with each other 🫧. for two layouts with overlapping byte-ranges to actually overlap each other, both layouts must be of the same layout type, correspond to the same filehandle, and have the same iomode. layouts conflict when they overlap and differ in the content of the layout (i.e., the storage device/file mapping parameters differ) ⚔️. note that differing iomodes do not lead to conflicting layouts 💅. it is permissible for layouts with different iomodes, pertaining to the same byte-range, to be held by the same client. an example of this would be copy-on-write functionality for a block/volume layout type 🦋

12.2.9. Layout Iomode 🦋

okay so the layout iomode (data type layoutiomode4, see Section 3.3.20) is how the client tells the metadata server what it plans to do. Is it just going to READ 📖, or is it going to mix READ and WRITE ✍️? For some layout types, it's super useful to say this right when you send LAYOUTGET (Section 18.43). Like, for block/volume-based protocols, block allocation could happen when you ask for LAYOUTIOMODE4_RW 🧱. There's also a special LAYOUTIOMODE4_ANY iomode, and it can ONLY be used with LAYOUTRETURN and CB_LAYOUTRECALL, never LAYOUTGET 🚫. It basically means "I'm returning (or the server is recalling) layouts for both LAYOUTIOMODE4_READ and LAYOUTIOMODE4_RW," respectively. 🫶

A storage device may validate I/O against the iomode, and whether it does depends on the storage device implementation and the layout type 🤓. So if the client's iomode doesn't match the I/O it's doing, the storage device may reject that I/O with an error saying the client should get a new layout with the right iomode via LAYOUTGET. For example, if a client gets a layout with a LAYOUTIOMODE4_READ iomode and then does a WRITE to a storage device, the storage device is allowed to reject that WRITE. Not cool, but fair 💀.

The iomode doesn't clash with OPEN share modes or byte-range LOCK operations, though 🙅‍♀️. Open share mode and byte-range lock conflicts still get enforced exactly like they would without pNFS, and they're logically separate from the pNFS layout level. Open share modes and byte-range locks are the preferred way to restrict user access to data files 🔐. For example, an OPEN of OPEN4_SHARE_ACCESS_WRITE does not conflict with a LAYOUTGET containing an iomode of LAYOUTIOMODE4_RW performed by another client. And if apps need to write into the same file at the same time, they can use byte-range locking to serialize their accesses 🎀.

12.2.10. Device IDs ✨

The device ID (data type deviceid4, see Section 3.3.14) identifies a group of storage devices 🧚. Its scope is the pair <client ID, layout type>. Here's the thing: a LOT of info can be needed to fully address a storage device, so instead of stuffing all of that into a layout, layouts just embed device IDs 💅. The NFSv4.1 operation GETDEVICEINFO (Section 18.40) is what gets the full address info (including all the device addresses for that device ID) for the storage device, based on its layout type and device ID. For example, the address of an NFSv4.1 data server or an object-based storage device could be an IP address and port 🌐. The address of a block storage device could be a volume label 🏷️.

Clients cannot expect the mapping between a device ID and its storage device address(es) to persist across a metadata server restart 😭. See Section 12.7.4 for how recovery works in that situation.

A device ID lives as long as there's a layout referring to it. If no layouts refer to the device ID, the server is free to delete the device ID any time 🗑️. Once the server deletes a device ID, the server MUST NOT reuse the device ID for the same layout type and client ID again. This one's a hard rule 🚨. It's totally doable because the device ID is 16 bytes long, which leaves plenty of room to store a generation number if the server's implementation needs to reuse most of the rest of the device ID's content. It's required because otherwise the race conditions between asynchronous notifications of device ID addition and deletion would be way too messy to sort out 🌀.

Device ID to device address mappings are not leased, and they can be changed at any time 🔄. (Btw, while those mappings are likely to change after the metadata server restarts, the server is not required to change them.) A server has two choices for changing mappings. It can recall all layouts referring to the device ID, or it can use a notification mechanism 🔔.

The NFSv4.1 protocol doesn't have an optimal way to recall all layouts that referred to a particular device ID. The exception is when the server ties a single device ID to a single fsid or a single client ID. In that case, CB_LAYOUTRECALL has options for recalling all layouts associated with the fsid, client ID pair, or just the client ID 💖.

With a notification mechanism (see Section 20.12), device ID to device address mappings can change over the course of server operation without recalling or revoking the layouts that refer to the device ID ✨. The notification mechanism can also delete a device ID, but only if the client has no layouts referring to it. A notification of a change to a device ID to device address mapping will immediately or eventually invalidate some or all of that device ID's mappings ⏳. The server MUST support notifications, and the client must request them before they can be used. For more on the notification types, see Section 20.12 🫶.

12.3. pNFS Operations 🌸

NFSv4.1 has several operations that pNFS servers need, no matter the layout type or storage protocol 🧠. All of these get sent to a metadata server, and we'll summarize them here. pNFS is an OPTIONAL feature, but if you do implement pNFS, some operations are REQUIRED to comply with it. See Section 17 📋.

These are the fore channel pNFS operations:

GETDEVICEINFO (Section 18.40), as noted previously (Section 12.2.10), returns the mapping of device ID to storage device address 🗺️.
GETDEVICELIST (Section 18.41) lets clients fetch all device IDs for a specific file system 📂.
LAYOUTGET (Section 18.43) is how a client gets a layout for a file 📄.
LAYOUTCOMMIT (Section 18.42) tells the metadata server the client's intent to commit data that has been written to the storage device (the storage device as originally indicated in the return value of LAYOUTGET) 💾.
LAYOUTRETURN (Section 18.44) returns layouts for a file, a file system ID (FSID), or a client ID 🔙.

These are the backchannel pNFS operations:

CB_LAYOUTRECALL (Section 20.3) recalls a layout, all layouts belonging to a file system, or all layouts belonging to a client ID 📣.
CB_RECALL_ANY (Section 20.6) tells a client that it needs to return some number of recallable objects, including layouts, to the metadata server 🔁.
CB_RECALLABLE_OBJ_AVAIL (Section 20.7) tells a client that a recallable object it was denied (for pNFS, a layout denied by LAYOUTGET) due to resource exhaustion is now available 🎉.
CB_NOTIFY_DEVICEID (Section 20.12) notifies the client of changes to device IDs 🔔.

12.4. pNFS Attributes 💜

A bunch of attributes that are specific to pNFS are listed and described in Section 5.12 ✨.

12.5. Layout Semantics 👑

12.5.1. Guarantees Provided by Layouts 🛡️

Layouts give the client the ability to access data located at a storage device, using the appropriate storage protocol 💅. The client is guaranteed that the layout will be recalled when one of two things happens: either a conflicting layout is requested, or the state the layout encapsulates becomes invalid (this can happen when an event directly or indirectly modifies the layout) 😬. When a layout is recalled and returned by the client, the client can still access file data with normal NFSv4.1 operations through the metadata server. Only the ability to access the storage devices is affected 🫶.

NFSv4.1's rule that all user access rights MUST be obtained through the appropriate OPEN, LOCK, and ACCESS operations doesn't change because layouts exist 🙌. Layouts are handed out to NFSv4.1 clients, and user access still follows the protocol's rules as if layouts didn't exist. To access a storage device, a client MUST hold a layout (it's a requirement, not a suggestion) 🚫. If a storage device gets an I/O request for a byte-range the client doesn't hold a layout for, the storage device SHOULD reject that I/O request. Also, modifying a file that has a layout held doesn't necessarily conflict with holding the layout that describes the file being modified. So the storage protocol or layout type is the one that decides the behavior it needs. For example, block/volume layout types require that the layout's iomode agree with the type of I/O being performed 🧱.

Depending on the layout type and storage protocol in use, storage device access permissions may be granted by LAYOUTGET and encoded inside the type-specific layout 🔑. For an example of storage device access permissions, see an object-based protocol such as [58]. If access permissions are encoded within the layout, the metadata server SHOULD recall the layout when those permissions become invalid for any reason, like when a file becomes unwritable or inaccessible to a client 🚷. Keep in mind that clients are still required to perform the appropriate OPEN, LOCK, and ACCESS operations as described above. How much a client could get around those operations, and what happens if it does, must be clearly specified by the individual layout type specifications. Those specifications must also be clear about the requirements and non-requirements for the checking the server performs 📝.

With pNFS in the picture, mandatory byte-range locks MUST behave exactly as they would without pNFS 🔒. So if mandatory file locks and layouts are provided at the same time, the storage device MUST be able to enforce the mandatory byte-range locks. For example, if one client gets a mandatory byte-range lock and a second client accesses the storage device, the storage device MUST appropriately restrict I/O for the range of that mandatory byte-range lock 🚧. If the storage device can't provide this check in the presence of mandatory byte-range locks, then the metadata server MUST NOT grant layouts and mandatory byte-range locks simultaneously 🙅‍♀️💖.

12.5.2. Getting a Layout ✨

okay so bestie, this is how a client actually goes and gets a layout 🎀 it uses the LAYOUTGET operation 💅 the metadata server is the one who grants layouts of a particular type (e.g., block/volume, object, or file) 🧚 and the client picks a layout type that the server supports AND that the client is prepared to use 🫶 the layout that comes back might not exactly match the byte-range that was asked for (see Section 18.43.3) 🙈 and if it needs to, a client can send multiple LAYOUTGET operations 🦋 these might end up as multiple overlapping, non-conflicting layouts (see Section 12.2.8) ✨

btw, to even get a layout, the client has to have opened the file first with the OPEN operation 🔥 so when a client has no layout on a file, it MUST present an open stateid, a delegation stateid, or a byte-range lock stateid in the loga_stateid argument 💖 a successful LAYOUTGET result includes a layout stateid 🌸 the very first successful LAYOUTGET the server processes using a non-layout stateid MUST have the "seqid" field of the layout stateid in the response set to one 🍓 after that, the client MUST use a layout stateid (see Section 12.5.3) on future LAYOUTGET calls for that file, and the "seqid" MUST NOT be set to zero 👑 once the layout is retrieved, it can be held across multiple OPEN and CLOSE sequences 😭 so a client can hold a layout for a file that isn't currently open by any user on the client 💜 this is lowkey how layouts get cached beyond CLOSE 🌈

the storage protocol the client uses to access data on the storage device is determined by the layout's type ✨ the client is responsible for matching the layout type with an available method to interpret and use the layout 💅 how that layout type selection works is outside the scope of the pNFS functionality 🫶

the metadata server is in control of the layout for a file 👑 BUT the pNFS client can give hints to the server when a file is opened or created about the preferred layout type and aggregation schemes 🎀 pNFS introduces a layout_hint attribute (Section 5.12.4) that the client can set at file creation time to hint the server about new files 🦋 setting this attribute separately, after the file has been created, might make it difficult, or impossible, for the server implementation to comply 💀

okay and this one is a whole thing 😭 the EXCLUSIVE4 createmode4 doesn't allow attributes to be set at file creation time, so NFSv4.1 introduces the EXCLUSIVE4_1 createmode4, which DOES allow attributes to be set then 🌸 but if the session is created with persistent reply caches, EXCLUSIVE4_1 is neither necessary nor allowed 🙈 instead, GUARDED4 both works better and is prescribed 💖 Table 18 in Section 18.16.3 summarizes how a client is allowed to send an exclusive create 🍓

Table 18 (Section 18.16.3): see the RFC for the table contents

12.5.3. Layout Stateid 🦋

the layout stateid has a "seqid" and "other" field, just like every other stateid 🧚 once a layout stateid is established, the "other" field stays constant UNLESS the stateid is revoked or the client returns all layouts on the file and the server disposes of the stateid 💅 the "seqid" field starts at one and is never zero on any NFSv4.1 operation that uses layout stateids, whether it's a fore channel or backchannel operation ✨ after the layout stateid is established, the server increments the "seqid" by one in each subsequent LAYOUTGET and LAYOUTRETURN response, and in each CB_LAYOUTRECALL request 🔥

the design goal of pNFS is parallelism, so the layout stateid is different from other stateid types: the client is expected to send LAYOUTGET and LAYOUTRETURN operations in parallel 🌈 the "seqid" is how the client sorts out responses to LAYOUTGET and LAYOUTRETURN 🫶 it also prevents race conditions between LAYOUTGET and CB_LAYOUTRECALL 💜 since the processing rules differ from other stateid types, only the pNFS sections of this document should be considered to figure out proper layout stateid handling 👑

once the client gets a layout stateid, it MUST use the correct "seqid" for later LAYOUTGET or LAYOUTRETURN operations 💖 the correct "seqid" is the highest "seqid" value from responses of fully processed LAYOUTGET or LAYOUTRETURN operations, or from arguments of a fully processed CB_LAYOUTRECALL operation 🍓 since the server increments "seqid" on each layout operation, the client can work out the order of operation processing by looking at it 🎀 for overlapping layout ranges, that ordering info tells the client which layout ranges it holds 🦋 overlapping ranges can happen because of the client's own requests, OR because the server is allowed to expand the range of a requested layout and tell the client in the LAYOUTRETURN results 😭 more layout stateid sequencing requirements are in Section 12.5.5.2 ✨

getting a "seqid" is NOT enough to use it 🙈 the client has to fully process the operations first 💅 for LAYOUTGET results, if the client is NOT using the forgetful model (Section 12.5.5.1), it MUST first update its record of what ranges of the file's layout it has before using the seqid 🌸 for LAYOUTRETURN results, the client MUST delete the range from its record of what ranges of the file's layout it had before using the seqid 💜 for CB_LAYOUTRECALL arguments, the client MUST send a response to the recall before using the seqid 🧚 the fundamental rule is that "seqid" is used to provide the order of processing 👑 LAYOUTGET results may be processed in parallel 🌈 LAYOUTRETURN results may be processed in parallel 💖 LAYOUTGET and LAYOUTRETURN responses may be processed in parallel as long as the ranges do not overlap 🫶 CB_LAYOUTRECALL request processing MUST be processed in "seqid" order at all times 🔥

once a client has no more layouts on a file, the layout stateid is no longer valid and MUST NOT be used 💀 any attempt to use such a layout stateid will result in NFS4ERR_BAD_STATEID 😭

12.5.4. Committing a Layout 💅

so allowing for the different storage protocol capabilities, the pNFS protocol does NOT require the metadata server and storage devices to have a consistent view of file attributes and data location mappings 🦋 data location mapping is stuff like which offsets store data versus which store holes (see Section 13.4.4 for a discussion) 🧚 related issues come up for storage protocols where a layout may hold provisionally allocated blocks, and that allocation does not survive a complete restart of both the client and server 🙈 because of this inconsistency, it's necessary to resynchronize the client with the metadata server and its storage devices and make any potential changes available to other clients 💖 this is done with the LAYOUTCOMMIT operation ✨

the LAYOUTCOMMIT operation is responsible for committing a modified layout to the metadata server 🎀 the data should be written and committed to the appropriate storage devices BEFORE the LAYOUTCOMMIT happens 🔥 the scope of LAYOUTCOMMIT depends on the storage protocol in use 🌸 the level of synchronization is from the point of view of the client that sent the LAYOUTCOMMIT 💜 the updated state on the metadata server only needs to reflect the state as of the client's last operation before the LAYOUTCOMMIT 🫶 the metadata server is NOT REQUIRED to keep a global view that accounts for other clients' I/O that may have happened in the same time frame 👑

for block/volume-based layouts, LAYOUTCOMMIT may need to update the block list that makes up the file and commit this layout to stable storage 🌈 for file-based layouts, the attributes between the metadata and storage devices need to be synced, mainly the size attribute 🍓

the control protocol is free to sync the attributes before it gets a LAYOUTCOMMIT 💅 BUT upon successful completion of a LAYOUTCOMMIT, state on the metadata server that describes the file MUST be synchronized with the state on the storage devices that make up that file, as of the client's last sent operation 😭 so a client that checks the size of a file between a WRITE to a storage device and the LAYOUTCOMMIT might see a size that doesn't reflect the actual data written 💀 the client MUST have a layout in order to send a LAYOUTCOMMIT operation 🧚

12.5.4.1. LAYOUTCOMMIT and change/time_modify 🌸

the change and time_modify attributes may be updated by the server when LAYOUTCOMMIT is processed 💖 the reason is that some layout types don't support updating these attributes when the storage devices process I/O 🦋 if a client has a layout with the LAYOUTIOMODE4_RW iomode on the file, the client MAY provide a suggested value to the server for time_modify in the LAYOUTCOMMIT arguments ✨ depending on the layout type, the provided value may or may not be used 🫶 the server SHOULD sanity-check client-provided values before using them 🙈 for example, the server SHOULD ensure that time does not flow backwards 🔥 the client always has the option to set time_modify through an explicit SETATTR operation 👑

for some layout protocols, the storage device can tell the metadata server that an I/O happened, so change and time_modify may be updated at the metadata server 🌈 for a metadata server that can monitor updates to change and time_modify, LAYOUTCOMMIT processing is NOT required to update the change attribute 🎀 in this case, the metadata server MUST ensure that no further update to the data has happened since the last update of the attributes 💜 file-based protocols may have enough info to figure that out, or may update the change attribute on each file modification 🧚 this also applies to the time_modify attribute 💅 if the server can determine that the file hasn't been modified since the last time_modify update, the server need NOT update time_modify at LAYOUTCOMMIT 🍓 at LAYOUTCOMMIT completion, the updated attributes SHOULD be visible if that file was modified since the latest previous LAYOUTCOMMIT or LAYOUTGET 🌈

12.5.4.2. LAYOUTCOMMIT and size ✨

okay so this one is about how the file's size gets updated when the client sends LAYOUTCOMMIT 💅 the big field here is loca_last_write_offset, and it tells the server the highest byte offset that got written but hasn't been committed with LAYOUTCOMMIT yet 🦋 its data type is newoffset4 and it's switched on a boolean called no_newoffset. if no_newoffset is FALSE, there's no offset given at all 🙈 bestie, this part is so important: offset is NOT the same thing as length, even though they're related. like, a loca_last_write_offset of zero means one byte got written at offset zero, so the file is at least one byte long 🌸

the server has two options here 👑

option 1: it updates the file's size using the last write offset the client gave it, either as the true file size or as a hint of the file size. if the server has a way to check, any new size should be sanity-checked 💖 for example, the file must not get truncated if the client's last write offset is less than the file's current size 🫶
option 2: it ignores the client's last write offset, and the server MUST have enough knowledge from other sources to figure out the file's size. for example, the server asks the storage devices over the control protocol 🍓

which option is picked depends on what the storage devices and/or the control protocol can do 🔥 like, if the storage devices are block devices that don't know anything about file size, the server MUST rely on the client to set the last write offset properly 😭

the LAYOUTCOMMIT results carry a new size value as a newsize4 union 💜 if the file's size changes because of LAYOUTCOMMIT, the server MUST reply with the new size. otherwise the new size isn't provided at all 🌈 and if the file size does get updated, the server SHOULD update the storage devices so the new file size is reflected once LAYOUTCOMMIT processing is done ✨ like, the client should be able to read up to the new file size, obviously 💅

now the client can also change a file's length by sending a SETATTR to the server with the size attribute specified 🦋 if the size is bigger than the current size, the file gets "zero extended," which just means zeros are implicitly added between the old EOF and the new EOF 🎀 (in lots of implementations, that zero-extended range is made of unallocated holes in the file 🧚). and when the client writes past EOF with WRITE, it does NOT need to use SETATTR 💖 no cap, it's giving efficiency 💅

12.5.4.3. LAYOUTCOMMIT and layoutupdate 🌸

the LAYOUTCOMMIT argument has a loca_layoutupdate field (see Section 18.42.1) of data type layoutupdate4 (see Section 3.3.18) 🦋 this argument is layout-type-specific, meaning it's a structure that can carry any layout-type-specific info from the client to the server at LAYOUTCOMMIT time ✨ for example, with a block/volume layout, the client can tell the server which reserved or allocated blocks it used or didn't use 💜

the content of loca_layoutupdate (field lou_body) doesn't have to be the same layout-type-specific content that LAYOUTGET returned in the loc_body field of the lo_content field of the logr_layout field 🙈 and the content of loca_layoutupdate is defined by the layout type spec and is opaque to LAYOUTCOMMIT, so LAYOUTCOMMIT itself doesn't look inside it 🔥

12.5.5. Recalling a Layout 👑

okay this is the recall part, and bestie, the main thing is that a layout protects a client's access to a file through a direct client-storage-device path 💅 so a layout only needs to get recalled when it can't do its job anymore 😭 that usually happens when the layout no longer matches where the file actually lives over the byte-range it covers 🫶 any operation that changes the layout, like server-driven restriping or load balancing, results in a recall 🌈

a layout gets recalled with the CB_LAYOUTRECALL callback operation (see Section 20.3) and gets returned with LAYOUTRETURN (see Section 18.44) ✨ CB_LAYOUTRECALL can recall a layout identified by a byte-range, all layouts for a file system ID (FSID), or all layouts for a client ID 💖 section 12.5.5.2 talks about the sequencing issues around getting, returning, and recalling layouts, so go check that out 🎀

the iomode matters for recalls too 🦋 generally, the iomode in the recall request MUST match the layout being returned. like, a recall with an iomode of LAYOUTIOMODE4_RW should make the client return only LAYOUTIOMODE4_RW layouts, NOT LAYOUTIOMODE4_READ ones 💜 but there's a special LAYOUTIOMODE4_ANY enumeration for recalling a layout of any iomode 🌸 in other words, the client MUST return both LAYOUTIOMODE4_READ and LAYOUTIOMODE4_RW layouts 🔥

a REMOVE operation SHOULD make the server recall the layout 🙈 this stops the client from touching a file that doesn't exist anymore and lets the client reclaim state it stored 🧚 since a REMOVE can be delayed until the last close of the file, the recall can get delayed too 🫶 after the last reference on the file is released and the file is removed, the client should no longer be able to do I/O using the layout ✨ for a file-based layout, the data server SHOULD return NFS4ERR_STALE in response to any operation on the removed file 💀

once a layout has been returned, the client MUST NOT send I/Os to the storage devices for the file, byte-range, and iomode that the returned layout covered 🚫 bestie, if a client does send an I/O to a storage device it doesn't hold a layout for, the storage device SHOULD reject that I/O 😭

pNFS doesn't change how clients cache file data or what those caching semantics are 🌈 but it knows some clients might do more aggressive write-behind caching to get the most out of pNFS 💅 the catch is that write-behind caching can make returning a layout in response to a CB_LAYOUTRECALL take longer 🙈 this is kind of like file delegations and how file data caching affects DELEGRETURN 🦋 so client implementations SHOULD limit how much unwritten data they have outstanding at any one time, to prevent super long responses to CB_LAYOUTRECALL 💖

once a layout is recalled, a server MUST wait one lease period before taking further action 👑 as soon as a lease period has passed, the server MAY choose to fence the client's access to the storage devices if it thinks the client has taken too long to return the layout 💀 but just like with data delegation and DELEGRETURN, the server MAY also choose to wait, since the client is showing forward progress toward returning the layout ✨ that forward progress can look like successful interaction with the storage devices, or like sub-portions of the layout being returned 🎀 and the server can limit exposure to these problems by limiting the byte-ranges it initially gives in layouts, which limits how much modified data is outstanding 🫶

12.5.5.1. Layout Recall Callback Robustness

okay so up until now we've been vibing on the idea that the pNFS client's state (layout ranges and iomode) for a file is EXACTLY the same as what the pNFS server has for that file ✨. since they agree on everything, any callback leads to a LAYOUTRETURN (or a set of them) that matches the range in the callback perfectly 💅. but it can be lowkey useful if that assumption doesn't always hold, tbh. here are some examples:

If conflicts that require callbacks are very rare, and a server can use a multi-file callback to recover per-client resources (e.g., via an FSID recall or a multi-file recall within a single CB_COMPOUND), the result may be significantly less client-server pNFS traffic 🦋
It may be useful for servers to maintain information about what ranges are held by a client on a coarse-grained basis, leading to the server's layout ranges being beyond those actually held by the client. in the extreme, a server could manage conflicts on a per-file basis, only sending whole-file callbacks even though clients may request and be granted sub-file ranges 🌸
It may be useful for clients to "forget" details about what layouts and ranges the client actually has, leading to the server's layout ranges being beyond those that the client "thinks" it has. as long as the client does not assume it has layouts that are beyond what the server has granted, this is a safe practice 🧚 when a client forgets what ranges and layouts it has, and it receives a CB_LAYOUTRECALL operation, the client MUST follow up with a LAYOUTRETURN for what the server recalled, or alternatively return the NFS4ERR_NOMATCHING_LAYOUT error if it has no layout to return in the recalled range 🙈
In order to avoid errors, it is vital that a client not assign itself layout permissions beyond what the server has granted, and that the server not forget layout permissions that have been granted 💀 on the other hand, if a server believes that a client holds a layout that the client does not know about, it is useful for the client to cleanly indicate completion of the requested recall, either by sending a LAYOUTRETURN operation for the entire requested range or by returning an NFS4ERR_NOMATCHING_LAYOUT error to the CB_LAYOUTRECALL 🍓

so, in light of all that, it is useful for a server to be able to send callbacks for layout ranges it has not granted to a client, and for a client to return ranges it does not hold 🫶 a pNFS client MUST always return layouts that comprise the full range specified by the recall. note, the full recalled layout range need not be returned as part of a single operation, but may be returned in portions. this lets the client stage the flushing of dirty data and commits and returns of layouts 🌈 it also tells the metadata server that the client is making progress ✨

when a layout is returned, the client MUST NOT have any outstanding I/O requests to the storage devices involved in the layout. in other words, the client MUST NOT return the layout while it has outstanding I/O requests to the storage device 🔥

even with this requirement on the client, it is possible that I/O requests may be presented to a storage device that is no longer allowed to perform them 😭 the server has no strict control over when the client will return the layout, so the server may later decide to unilaterally revoke the client's access to the storage devices as provided by the layout. when choosing to revoke access, the server must deal with the possibility of lingering I/O requests, i.e., I/O requests that are still in flight to storage devices identified by the revoked layout 👑 All layout type specifications MUST define whether unilateral layout revocation by the metadata server is supported; if it is, the specification must also describe how lingering writes are processed. for example, storage devices identified by the revoked layout could be fenced off from the client that held the layout 🔒

in order to ensure client/server convergence with regard to layout state, the final LAYOUTRETURN operation in a sequence of LAYOUTRETURN operations for a particular recall MUST specify the entire range being recalled, echoing the recalled layout type, iomode, recall/return type (FILE, FSID, or ALL), and byte-range, even if layouts pertaining to partial ranges were previously returned 💖 in addition, if the client holds no layouts that overlap the range being recalled, the client should return the NFS4ERR_NOMATCHING_LAYOUT error code to CB_LAYOUTRECALL. this lets the server update its view of the client's layout state ✨

12.5.5.2. Sequencing of Layout Operations

as with other stateful operations, pNFS requires the correct sequencing of layout operations 🎀 pNFS uses the "seqid" in the layout stateid to provide the correct sequencing between regular operations and callbacks. it's the server's responsibility to avoid inconsistencies regarding the layouts provided, and the client's responsibility to properly serialize its layout requests and layout returns 💅

12.5.5.2.1. Layout Recall and Return Sequencing

okay this is a big one 😭 one critical issue with layout operations sequencing concerns callbacks. the protocol must defend against races between the reply to a LAYOUTGET or LAYOUTRETURN operation and a subsequent CB_LAYOUTRECALL. A client MUST NOT process a CB_LAYOUTRECALL that implies one or more outstanding LAYOUTGET or LAYOUTRETURN operations to which the client has not yet received a reply. the client detects such a CB_LAYOUTRECALL by examining the "seqid" field of the recall's layout stateid. if the "seqid" is not exactly one higher than what the client currently has recorded, and the client has at least one LAYOUTGET and/or LAYOUTRETURN operation outstanding, the client knows the server sent the CB_LAYOUTRECALL after sending a response to an outstanding LAYOUTGET or LAYOUTRETURN. The client MUST wait before processing such a CB_LAYOUTRECALL until it processes all replies for outstanding LAYOUTGET and LAYOUTRETURN operations for the corresponding file with a seqid less than the seqid given by CB_LAYOUTRECALL (lor_stateid; see Section 20.3.) 🧠

on top of the seqid thing, Section 2.10.6.3 describes the sessions mechanism, which also lets the client detect callback race conditions and delay processing such a CB_LAYOUTRECALL 🌸 The server MAY reference conflicting operations in the CB_SEQUENCE that precedes the CB_LAYOUTRECALL. because the server has already sent replies for these operations before sending the callback, those replies may race with the CB_LAYOUTRECALL. The client MUST wait for all the referenced calls to complete and update its view of the layout state before processing the CB_LAYOUTRECALL 🦋

12.5.5.2.1.1. Get/Return Sequencing

the protocol lets the client send concurrent LAYOUTGET and LAYOUTRETURN operations to the server 🍓 the protocol doesn't give the server any way to process the requests in the same order they were created. however, through the use of the "seqid" field in the layout stateid, the client can figure out the order in which parallel outstanding operations were processed by the server 🕵️‍♀️ so when a layout retrieved by an outstanding LAYOUTGET operation intersects with a layout returned by an outstanding LAYOUTRETURN on the same file, the order in which the two conflicting operations are processed determines the final state of the overlapping layout. the order is determined by the "seqid" returned in each operation: the operation with the higher seqid was executed later 💜

it's permissible for the client to send multiple parallel LAYOUTGET operations for the same file, multiple parallel LAYOUTRETURN operations for the same file, or a mix of both 🌈

it's permissible for the client to use the current stateid (see Section 16.2.3.1.2) for LAYOUTGET operations, for example when compounding LAYOUTGETs or compounding OPEN and LAYOUTGETs. it's also permissible to use the current stateid when compounding LAYOUTRETURNs 🫶

it's permissible for the client to use the current stateid when combining LAYOUTRETURN and LAYOUTGET operations for the same file in the same COMPOUND request, since the server MUST process these in order. however, if a client does send such COMPOUND requests, it MUST NOT have more than one outstanding for the same file at the same time, and it MUST NOT have other LAYOUTGET or LAYOUTRETURN operations outstanding at the same time for that same file 🙅‍♀️✨

12.5.5.2.1.2. Client Considerations

picture a pNFS client that has sent a LAYOUTGET, and before it gets the reply, it receives a CB_LAYOUTRECALL for the same file with an overlapping range 👀 there are two possibilities, and the client can tell them apart using the layout stateid in the recall:

The server processed the LAYOUTGET before sending the recall, so the LAYOUTGET must be waited for, because it may be carrying layout information that will need to be returned to deal with the CB_LAYOUTRECALL 💖
The server sent the callback before receiving the LAYOUTGET. The server will not respond to the LAYOUTGET until the CB_LAYOUTRECALL is processed 🙃

if these possibilities can't be distinguished, a deadlock could result 😱 the client must wait for the LAYOUTGET response before processing the recall in the first case, but that response won't arrive until after the recall is processed in the second case. note that in the first case, the "seqid" in the layout stateid of the recall is two greater than what the client has recorded; in the second case, the "seqid" is one greater than what the client has recorded. this lets the client disambiguate between the two cases, so it knows precisely which possibility applies 🧩

in case 1, the client knows it needs to wait for the LAYOUTGET response before processing the recall (or the client can return NFS4ERR_DELAY) 🌷

in case 2, the client will not wait for the LAYOUTGET response before processing the recall, because waiting would cause deadlock 🚫 so the action at the client only requires waiting in the case that the client has not yet seen the server's earlier responses to the LAYOUTGET operation(s) ✨

the recall process can be considered complete when the final LAYOUTRETURN operation for the recalled range is completed 🎉 the LAYOUTRETURN uses the layout stateid (with seqid) specified in CB_LAYOUTRECALL. if the client uses multiple LAYOUTRETURNs in processing the recall, the first LAYOUTRETURN will use the layout stateid as specified in CB_LAYOUTRECALL, and subsequent LAYOUTRETURNs will use the highest seqid, as is the usual case 🔁

12.5.5.2.1.3. Server Considerations ✨

Okay bestie, let's look at this race from the metadata server's side 👀 The metadata server has sent a CB_LAYOUTRECALL and then gets an overlapping LAYOUTGET for the same file before the LAYOUTRETURN(s) that answer that CB_LAYOUTRECALL. There are three cases, and each one is a little drama 🎭

The client sent the LAYOUTGET before it processed the CB_LAYOUTRECALL. The seqid in the layout stateid inside the LAYOUTGET arguments is one less than the seqid in CB_LAYOUTRECALL. The server returns NFS4ERR_RECALLCONFLICT, which tells the client there's a pending recall 🙈
The client sent the LAYOUTGET after processing the CB_LAYOUTRECALL, but the LAYOUTGET showed up before the LAYOUTRETURN and the response to CB_LAYOUTRECALL that finished that processing. The seqid in the layout stateid of LAYOUTGET is equal to or greater than the seqid in CB_LAYOUTRECALL. The server hasn't gotten a response to the CB_LAYOUTRECALL yet, so it returns NFS4ERR_RECALLCONFLICT 💅
The client sent the LAYOUTGET after processing the CB_LAYOUTRECALL, and the server did get the CB_LAYOUTRECALL response, but the LAYOUTGET arrived before the LAYOUTRETURN that finished that processing. The seqid in the layout stateid of LAYOUTGET is equal to the seqid in CB_LAYOUTRECALL. Since the server already has a response to the CB_LAYOUTRECALL, it returns NFS4ERR_RETURNCONFLICT 🔥

12.5.5.2.1.4. Wraparound and Validation of Seqid 🌀

Layout stateids play by their own rules, unlike the other stateids in the protocol. Two reasons: the seqid value can never be zero, and the stateid's seqid changes during a CB_LAYOUTRECALL operation. Because the non-zero rule meets the naturally parallel nature of layout operations, a bunch of LAYOUTGET and LAYOUTRETURN operations can end up carrying the same seqid value 🫶

The server uses a slightly tweaked version of the modulo arithmetic from Section 2.10.6.1 when it increments a layout stateid's seqid. The twist is that zero is never a valid seqid. When a seqid is 0xFFFFFFFF, the next valid value is 0x00000001. The modulo arithmetic also handles the seqid comparisons when processing CB_LAYOUTRECALL events, as described in Section 12.5.5.2.1.3.

Just like the server validates the seqid during CB_LAYOUTRECALL usage (Section 12.5.5.2.1.3), it also checks that the seqid is in an appropriate range. That range is the degree of parallelism the server supports for layout stateids. If the client sends multiple layout operations in parallel, the seqid in the supplied stateid won't be the current seqid the server holds, by definition 🧠 The range of parallelism runs from the highest, or current, seqid back to some seqid value in the past. Here's the naming so we can talk about it:

SERVER_CURRENT_SEQID is the server's current seqid value for a layout stateid.
PAST_SEQID is the lowest seqid value the server will accept.
VALID_SEQID_RANGE is the range of valid seqid values, or the range of parallelism.

So the following holds, and all arithmetic here is the modulo arithmetic from above:

VALID_SEQID_RANGE = SERVER_CURRENT_SEQID - PAST_SEQID

The server MUST support a minimum VALID_SEQID_RANGE. That minimum is defined as:

VALID_SEQID_RANGE = summation over 1..N of (ca_maxoperations(i) - 1)

Here N is the number of session fore channels, and ca_maxoperations(i) is the value of ca_maxoperations returned from CREATE_SESSION for the i'th session. The - 1 is there to leave room for the required SEQUENCE operation. The server MAY support a VALID_SEQID_RANGE larger than that minimum. The maximum VALID_SEQID_RANGE is 2^(32) - 2, since zero doesn't count as a valid seqid value 🌸

If the server finds that the seqid is zero, it returns NFS4ERR_BAD_STATEID to the client. The server then checks that the seqid is inside the range of parallelism, VALID_SEQID_RANGE. If the seqid falls outside that range, the server returns NFS4ERR_OLD_STATEID. When the client gets NFS4ERR_OLD_STATEID, it updates the stateid in the layout request based on what happened with its other layout requests, and then it re-sends the operation to the server 🔁

12.5.5.2.1.5. Bulk Recall and Return 📦

pNFS can recall and return all the layouts for files that belong to a particular fsid (LAYOUTRECALL4_FSID, LAYOUTRETURN4_FSID) or to a particular client ID (LAYOUTRECALL4_ALL, LAYOUTRETURN4_ALL). Bulk operations don't have "bulk" stateids, so you can't catch races with the seqid in this case 😬 The server MUST NOT initiate a bulk recall while another recall is in progress, or while the matching LAYOUTRETURN is in progress or pending. If the server sends a bulk recall while the client has a pending or in-progress LAYOUTRETURN, CB_LAYOUTRECALL, or LAYOUTGET, the client returns NFS4ERR_DELAY. If the client sends a LAYOUTGET or LAYOUTRETURN while a bulk recall is in progress, the server returns NFS4ERR_RECALLCONFLICT. If the client sends a LAYOUTGET or LAYOUTRETURN after the server gets NFS4ERR_DELAY from a bulk recall, then, to keep things moving forward, the server MAY return NFS4ERR_RECALLCONFLICT.

Once a CB_LAYOUTRECALL of LAYOUTRECALL4_ALL is sent, the server MUST NOT allow the client to use any layout stateid except for LAYOUTCOMMIT operations. Once the client receives a CB_LAYOUTRECALL of LAYOUTRECALL4_ALL, it MUST NOT use any layout stateid except for LAYOUTCOMMIT operations. Once a LAYOUTRETURN of LAYOUTRETURN4_ALL is sent, every layout stateid granted to the client ID is freed. The client MUST NOT use those layout stateids again. It MUST use LAYOUTGET to get new layout stateids 🧹

Once a CB_LAYOUTRECALL of LAYOUTRECALL4_FSID is sent, the server MUST NOT allow the client to use any layout stateid that refers to a file with the specified fsid, except for LAYOUTCOMMIT operations. Once the client receives a CB_LAYOUTRECALL of LAYOUTRECALL4_ALL, it MUST NOT use any layout stateid that refers to a file with the specified fsid, except for LAYOUTCOMMIT operations. Once a LAYOUTRETURN of LAYOUTRETURN4_FSID is sent, every layout stateid granted for the referenced fsid is freed. The client MUST NOT use those freed layout stateids for files with the referenced fsid again. After that, for any file with the referenced fsid, if the client wants to use a layout, it MUST first send a LAYOUTGET to get a new layout stateid for that file 💖

If the server has sent a bulk CB_LAYOUTRECALL and then receives a LAYOUTGET, or a LAYOUTRETURN with a stateid, the server MUST return NFS4ERR_RECALLCONFLICT. If the server has sent a bulk CB_LAYOUTRECALL and receives a LAYOUTRETURN whose lr_returntype isn't equal to the lor_recalltype of that CB_LAYOUTRECALL, the server MUST return NFS4ERR_RECALLCONFLICT too 🚫

12.5.6. Revoking Layouts ⚠️

Parallel NFS lets servers revoke layouts from clients that don't respond to recalls, or that don't renew their lease in time. Depending on the layout type, the server might revoke the layout, and it might take certain actions regarding the client's I/O to the data servers. It's a big deal, so handle with care 🥺

12.5.7. Metadata Server Write Propagation 🪄

Asynchronous writes that go through the metadata server may be propagated lazily to the storage devices. For data written asynchronously through the metadata server, a client that reads from the appropriate storage device isn't guaranteed to see the new data until a COMMIT happens at the metadata server. While the write is pending, reads to the storage device may return the old data, the new data, or a mixture of new and old, so it's a little chaotic 😵‍💫 When a synchronous WRITE or a COMMIT (for asynchronously written data) completes, the metadata server MUST ensure that the storage devices return the new data and that the data has been written to stable storage. If the server implements its storage in any way that it cannot handle these constraints, then it MUST recall the layouts, so no reads happen that can't be handled correctly. Note that the layouts MUST be recalled before the server responds to the associated WRITE operations 👑

12.6. pNFS Mechanics ✨

okay so this section is the step-by-step flow of how a pNFS client talks to a metadata server and a storage device 🫶

when a pNFS client sees a brand new FSID, it sends a GETATTR to the NFSv4.1 server asking for the fs_layout_type (Section 5.12.1) attribute 👀. if that comes back with at least one layout type, and those types are among the ones the client supports, the client knows pNFS is a possibility for that file system 💖. next, if the client doesn't already have a client ID from an EXCHANGE_ID result that returned EXCHGID4_FLAG_USE_PNFS_MDS (from the same server that gave back the new FSID), it MUST send an EXCHANGE_ID to the server with the EXCHGID4_FLAG_USE_PNFS_MDS bit set.

plot twist 💀: if the server's response doesn't have EXCHGID4_FLAG_USE_PNFS_MDS, then even though fs_layout_type said otherwise, the server doesn't support pNFS. the client can't use pNFS with that server, and the server MUST return NFS4ERR_NOTSUPP in response to any pNFS operation.

the client then creates a session and asks for a persistent one 🎀. why? so exclusive creates can be done in a single round trip using the createmode4 of GUARDED4. if the session ends up not being persistent, the client will use EXCLUSIVE4_1 for exclusive creates instead.

if a file is going to be created on a pNFS-enabled file system, the client uses the OPEN operation. among the normal set of attributes that can be provided on OPEN for creation, there's an OPTIONAL layout_hint attribute 🧚. this is how the client says "hey, this is the layout type and layout details I'd prefer." using a createmode4 of UNCHECKED4, GUARDED4, or EXCLUSIVE4_1 lets the client send the layout_hint attribute at create time. the client MUST NOT use EXCLUSIVE4 (see Table 18) 🙅‍♀️. the client is RECOMMENDED to put a GETATTR right after the OPEN in the same COMPOUND 💅. that GETATTR can then grab the layout_type for the newly created file, so the client knows which layout type the server picked and therefore which storage protocol it has to speak 🔥.

if the client wants to open an existing file instead, it also includes a GETATTR to find out what layout type the file supports 🍓.

either way (creating or just opening), that GETATTR can also ask for the layout_blksize and layout_alignment attributes, so the client can figure out the best offsets and lengths for I/O on the file 📏.

assuming the client supports the layout type that GETATTR returned and decides to use pNFS for data access, it sends LAYOUTGET with the filehandle and stateid that OPEN returned, specifying the range it wants to do I/O on 🦋. the response is a layout, which might be a subset of the range the client asked for. it also includes device IDs and a description of how the data is organized (or, when writing, how it's going to be organized) across those devices. the device IDs and data description are encoded in a format specific to the layout type, but the client is expected to understand it 🧠.

when the client wants to send an I/O, it looks at the data description in the layout to figure out which device ID the I/O command should go to 👀. then it sends a GETDEVICEINFO to find the device address(es) for that device ID. after that, it sends the I/O request to one of the device ID's device addresses, using the storage protocol defined for the layout type ✨. quick bonus: if a client has multiple I/Os to send, those requests may be done in parallel 🏃‍♀️💨.

if the I/O was a WRITE, then at some point the client may want to use LAYOUTCOMMIT 💾. that's how it tells the metadata server about the new modification time and the new size of the file (if it thinks it extended the file), and how it commits the modified data to the file system 🫡.

12.7. Recovery 💜

recovery is complicated (sorry bestie 😭) because pNFS is distributed. in general, crash recovery for layouts is pretty similar to crash recovery for delegations in base NFSv4.1. BUT the client being able to do I/O without contacting the metadata server adds some subtleties that have to be handled correctly, or you risk file system corruption 💀. so pay attention to this part!!

12.7.1. Recovery from Client Restart

client recovery for layouts is similar to client recovery for other lock and delegation state 🔄. when a pNFS client restarts, it loses all information about the layouts it previously owned. there are two ways the server can reclaim those resources and let otherwise conflicting layouts go to other clients 🌈.

the first way is through lease expiry. if the client's recovery time is longer than the lease period, the lease expires and the server knows the state may be released. for layouts, the server can release the state right away when the lease expires, OR it can let the layout hang around in case the lease gets revived, as long as no other layout conflicts with it 🕰️.

the second way is when the client restarts in less time than it takes for the lease period to run out ⚡. in that case, the client contacts the server through the standard EXCHANGE_ID protocol. the server finds that the client's co_ownerid matches the co_ownerid of the previous client invocation, but the verifier is different. the server reads that as a signal to release all layout state tied to the client's previous invocation 🚨. in this scenario, any data the client wrote that isn't covered by a successful LAYOUTCOMMIT is in an undefined state: it may have been written, or it may now be lost 😬. this is acceptable behavior, and it's the client's responsibility to use LAYOUTCOMMIT to get the level of stability it wants 💅.

12.7.2. Dealing with Lease Expiration on the Client

⚠️ if a client thinks its lease has expired, it MUST NOT send I/O to the storage device until it has validated its lease. no exceptions bestie 🙅‍♀️

the client can send a SEQUENCE operation to the metadata server to check. if the SEQUENCE is successful but sr_status_flag has SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED, SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, or SEQ4_STATUS_ADMIN_STATE_REVOKED set, the client MUST NOT use its currently held layouts 🛑. the client has two choices to recover from the lease expiration 🪄. choice one: for all modified but uncommitted data, the client writes it to the metadata server, using the FILE_SYNC4 flag on the WRITEs, or WRITE and COMMIT. choice two: the client re-establishes a client ID and session with the server, gets new layouts and device-ID-to-device-address mappings for the modified data ranges, and then writes the data to the storage devices using those newly obtained layouts 💖.

if sr_status_flags from the metadata server has SEQ4_STATUS_RESTART_RECLAIM_NEEDED set (or SEQUENCE returns NFS4ERR_BAD_SESSION and CREATE_SESSION returns NFS4ERR_STALE_CLIENTID), then the metadata server has restarted 🔁. in that case the client SHOULD recover using the methods described in Section 12.7.4.

if sr_status_flags from the metadata server has SEQ4_STATUS_LEASE_MOVED set, the client recovers by following the procedure described in Section 11.11.9.2 🚚. after that, the client may get an indication that the layout state wasn't moved along with the file system. it then recovers the same way as in the other applicable situations discussed in the first two paragraphs of this section.

if sr_status_flags reports no loss of state, then the leases for the layouts the client has are valid and renewed, and the client can happily send I/O requests to the storage devices again 🥰.

now, the client SHOULD NOT send I/Os to storage devices that might extend past the lease expiration time period, but that isn't always possible 😩. for example, an extended network partition could start after the I/O is sent and not heal until the storage device receives the request. so the metadata server and/or the storage devices are responsible for protecting themselves from I/Os that were sent before the lease expired but arrive after it did 🛡️. see Section 12.7.3 for the details.

12.7.3. Dealing with Loss of Layout State on the Metadata Server

this covers the case where ALL of the following are true 👇

the metadata server has not restarted
a pNFS client's layouts have been discarded (usually because the client's lease expired) and are invalid
an I/O from the pNFS client arrives at the storage device

🚨 the metadata server and its storage devices MUST solve this by fencing the client. in other words, they MUST solve this by preventing the execution of I/O operations from the client to the storage devices after layout state loss.

the details of how fencing is done are specific to the layout type 🔐. the solution for NFSv4.1 file-based layouts is described in (Section 13.11), and solutions for other layout types are in their respective external specification documents 📚.

12.7.4. Recovery from Metadata Server Restart 🔄✨

okay bestie, so this is the part where the metadata server restarts and the pNFS client has to figure out what happened 😭🫶 The client will notice the restart the way we talked about in Section 8.4.2, and the pNFS-specific version lives in Section 12.7.2, Paragraph 2. 💅 First thing: the client MUST stop using layouts and delete the device ID to device address mappings it got from the metadata server. 🚫 Then, if it wrote data to the storage device without committing the layouts with LAYOUTCOMMIT, it has extra cleanup to do so that the client, the metadata server, and the storage devices are all on the same page about the data. 🧚

* If the client still has data that's modified and unwritten sitting in its memory, it only has two choices. 💖

The client can get a layout with LAYOUTGET after the server's grace period ends, then write the data to the storage devices. 🦋
The client can WRITE that data through the metadata server using the WRITE (Section 18.32) operation, and then get layouts whenever it wants. 🌸

* If the client wrote data to the storage device asynchronously but still has a copy in its memory, it has the recovery options from the bullet above. 🫶 And if the metadata server is also in its grace period, the client gets the options from the next bullet too. ✨

* The client does not have a copy of the data in its memory and the metadata server is still in its grace period. 😭 The client can't use LAYOUTGET (inside or outside the grace period) to reclaim a layout, because what LAYOUTGET returns may not match what the client had before. 🙈 The range might be different, or the client might get the same range but a different layout content. 💀 Even if the layout looks the same, the device IDs could map to different device addresses, and even if the addresses match, they could have been handed to a different storage device. 🔥 Retrieving the data from the storage device and writing it to the metadata server, like in the recovery scenario above, is also off the table, because the mappings of range to device ID, device ID to device address, and device address to physical device are all stale, and fresh LAYOUTGET results don't fix that. 🙅‍♀️

The only recovery option for this one is to send a LAYOUTCOMMIT in reclaim mode, and the metadata server will accept it as long as it's still in its grace period. 👑 Using LAYOUTCOMMIT in reclaim mode tells the metadata server that the layout has changed. 🌈 It's critical that the metadata server gets this info before its grace period ends, because that's the moment it starts allowing updates to the file system. ⏰

To send LAYOUTCOMMIT in reclaim mode, the client sets the loca_reclaim field of the operation's arguments (Section 18.42.1) to TRUE. 💜 During the metadata server's recovery grace period (and only then), the metadata server is ready to accept LAYOUTCOMMIT requests where loca_reclaim is TRUE. 🎀

When loca_reclaim is TRUE, the client is trying to commit changes to the layout that happened before the metadata server restarted. 🧠 The metadata server runs some consistency checks on the loca_layoutupdate field of the arguments to figure out whether the client can commit the data it wrote to the storage device to the file system. 🔍 The loca_layoutupdate field has the data type layoutupdate4 and holds layout-type-specific content (in the lou_body field of loca_layoutupdate). 🍓 The layout-type-specific info it might contain is covered in Section 12.5.4.3. 📚 If the consistency checks pass, the metadata server MUST commit the data (described by the loca_offset, loca_length, and loca_layoutupdate fields of the arguments) that was written to the storage device. ✅ If the checks fail, the metadata server rejects the LAYOUTCOMMIT and makes no changes to the file system. 🚫 But here's the tea: any time a LAYOUTCOMMIT with loca_reclaim TRUE fails, the pNFS client has lost all the data in the range <loca_offset, loca_length>. 😱 A client can protect itself from this by caching all data in memory, whether it was written synchronously or asynchronously, and not releasing that cache until a LAYOUTCOMMIT succeeds. 🛡️ This doesn't hold for every layout type, though; for example, file-based storage devices don't have to suffer this limitation. 🥰

* The client does not have a copy of the data in its memory and the metadata server is no longer in its grace period, meaning the metadata server returns NFS4ERR_NO_GRACE. 💀 Just like the bullet above, if LAYOUTCOMMIT fails, the data in the range <loca_offset, loca_length> is lost. 😭 The protection is the same: cache all written data on the client until a LAYOUTCOMMIT succeeds. 🫶

12.7.5. Operations during Metadata Server Grace Period 🌷

Some of the recovery scenarios we've covered mention that some operations (WRITE and LAYOUTGET, specifically) might be allowed during the metadata server's grace period. ✨ The metadata server may allow these operations during its grace period. 🌸 For LAYOUTGET, the metadata server MUST reliably determine that handling such a request won't conflict with an incoming LAYOUTCOMMIT reclaim request. 🔒 For WRITE, the metadata server MUST reliably determine that handling the request won't conflict with an incoming OPEN, or with a LOCK on a file that has mandatory byte-range locking enabled. 🔐

As I mentioned before, to keep things simple, the metadata server might reject some operations (WRITE and LAYOUTGET, specifically) during its grace period. 🙃 The simplest correct approach is to reject every non-reclaim pNFS request and every WRITE operation by returning the NFS4ERR_GRACE error. 🚧 However, depending on the storage protocol (which is specific to the layout type) and the metadata server implementation, the metadata server may be able to tell that a particular request is safe. 🧐 For example, a metadata server may save provisional allocation mappings for each file to stable storage, along with info about potentially conflicting OPEN share modes and mandatory byte-range locks that might have been in effect at restart time. 💾 It can then use that info during the recovery grace period to decide that a WRITE request is safe. 🦋

12.7.6. Storage Device Recovery 💾

Recovery from a storage device restart mostly depends on the layout type in use. 🧩 That said, there are a few general techniques a client can use if it finds out a storage device crashed while holding modified, uncommitted data that was written asynchronously. 😬 First and foremost, it's important to realize that the client is the only one with the info needed to recover uncommitted data, since it holds the modified data and probably nothing else does. 🫵 Second, the best solution is for the client to err on the side of caution and try to rewrite the modified data through another path. 🛤️

The client SHOULD immediately WRITE the data to the metadata server, with the stable field in the WRITE4args set to FILE_SYNC4. 🌟 Once it does that, there's no need to wait for the original storage device. 🏃‍♀️

12.8. Metadata and Storage Device Roles 🎭

If the same physical hardware is used to implement both a metadata server and a storage device, then that hardware entity is understood to be playing two distinct roles, and it's really important to be clear about which role the hardware is acting as at any given time. 🤹‍♀️

There are two sub-cases to tell apart. 👯‍♀️

The storage device uses NFSv4.1 as the storage protocol, meaning the same physical hardware implements both a metadata server and a data server. 💖 See Section 13.1 for how multiple roles are handled. 📖
The storage device does not use NFSv4.1 as the storage protocol, and the same physical hardware implements both a metadata server and a storage device. 🌈 Whether distinct network addresses are used to reach the metadata server and the storage device doesn't matter. 🙌 That's because it's always clear to the pNFS client and server, from the upper-layer protocol in use (NFSv4.1 or non-NFSv4.1), which role a request sent to the common server network address is aimed at. 🎯

12.9. Security Considerations for pNFS ✨

okay so pNFS splits the file system into metadata and data and then gives you access to both 🫶 The pNFS-specific operations (listed in Section 12.3) are how you get at the metadata, and all the normal NFSv4.1 security stuff applies to that metadata access, no cap. The whole combo of pieces in a pNFS system (check Figure 1) has to keep the NFSv4.1 security properties when some entity is poking at a storage device from a client. That includes the countermeasures against the threats NFSv4.1 defends against, in places where those threats actually matter 🛡️

sometimes the protection for storage device connections looks like physical isolation, or just a recommendation to skip pNFS in that environment 🙈 For example, it can be impractical to encrypt some storage protocols so nobody can eavesdrop. If eavesdropping is a serious worry somewhere, you might go with physical isolation of the channel (like wiring clients straight to storage devices) and/or ditch pNFS and fall back to regular NFSv4.1 💅

if communication with storage devices faces the same threats as client-to-metadata-server traffic, the protocols used for it need security at least as strong as what RPCSEC_GSS gives NFSv4.1 💖 Other than the storage protocol for the LAYOUT4_NFSV4_1_FILES layout (see Section 13), so other than NFSv4.1 itself, specifying security for storage access protocols is out of scope for this document 🧚

pNFS implementations MUST NOT remove NFSv4.1's access controls. Clients, storage devices, and the metadata server are together responsible for making sure every client-to-storage-device file data access respects NFSv4.1's ACLs and file open modes. That means doing both checks on every access, in the client, the storage device, or both, depending on what applies (and when the storage device is an NFSv4.1 server, that storage device is ultimately responsible for access control, as described in Section 13.9.2) 👑

If a pNFS configuration only does these checks in the client, a misbehaving client getting unauthorized access is a big deal, and it should factor into whether that configuration is appropriate. Such layout types SHOULD NOT be used when client-only access checks don't give enough assurance that NFSv4.1 access control is being applied correctly. 💀

(this isn't a problem for the file layout type in Section 13, because the storage access protocol for LAYOUT4_NFSV4_1_FILES is NFSv4.1 itself, so the security model for storage device access through LAYOUT4_NFSV4_1_FILES is the same as the metadata server's 🫶) If you need access control details for a specific layout, look at that layout's spec, like the NFSv4.1/file-based layout (Section 13) in this document, the blocks layout [48], and the objects layout [47] 📚

13. NFSv4.1 as a Storage Protocol in pNFS: the File Layout Type 🦋

this section is all about the semantics and format of NFSv4.1 file-based layouts for pNFS 🌸 NFSv4.1 file-based layouts use the LAYOUT4_NFSV4_1_FILES layout type, and that type stripes data across multiple NFSv4.1 data servers. so cute, right? 🎀

13.1. Client ID and Session Considerations 🔑

Sessions are a REQUIRED feature of NFSv4.1, and that extends to both the metadata server and the file-based (NFSv4.1-based) data servers 💜

the role a server plays in pNFS is decided by what it returns from EXCHANGE_ID. The roles are:

Metadata server (EXCHGID4_FLAG_USE_PNFS_MDS is set in the result eir_flags).
Data server (EXCHGID4_FLAG_USE_PNFS_DS).
Non-metadata server (EXCHGID4_FLAG_USE_NON_PNFS). This is an NFSv4.1 server that doesn't support operations (like LAYOUTGET) or attributes that are about pNFS.

the client MAY request zero or more of EXCHGID4_FLAG_USE_NON_PNFS, EXCHGID4_FLAG_USE_PNFS_DS, or EXCHGID4_FLAG_USE_PNFS_MDS, even though some combos (like EXCHGID4_FLAG_USE_NON_PNFS | EXCHGID4_FLAG_USE_PNFS_MDS) are contradictory 🙃 Still, the server MUST only return the following acceptable combinations:

     +========================================================+
     | Acceptable Results from EXCHANGE_ID                    |
     +========================================================+
     | EXCHGID4_FLAG_USE_PNFS_MDS                             |
     +--------------------------------------------------------+
     | EXCHGID4_FLAG_USE_PNFS_MDS | EXCHGID4_FLAG_USE_PNFS_DS |
     +--------------------------------------------------------+
     | EXCHGID4_FLAG_USE_PNFS_DS                              |
     +--------------------------------------------------------+
     | EXCHGID4_FLAG_USE_NON_PNFS                             |
     +--------------------------------------------------------+
     | EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_NON_PNFS |
     +--------------------------------------------------------+

                              Table 8

as the table shows, a server can have one or two roles 🧚 It can be both a metadata server and a data server, or both a data server and a non-metadata server. Besides returning two roles in the EXCHANGE_ID results (and serving both through one common client ID), a server can also serve two roles by handing back a unique client ID and server owner for each role across two EXCHANGE_ID results, with each result showing one role 💅

when a server has concurrent pNFS roles served by one common client ID, and the EXCHANGE_ID request from the client has zero or some combo of bits set in eia_flags, the server result should set the bits for the highest acceptable combination of server roles, with a preference for matching the roles the client asked for. So if a client request has (EXCHGID4_FLAG_USE_NON_PNFS | EXCHGID4_FLAG_USE_PNFS_MDS | EXCHGID4_FLAG_USE_PNFS_DS) set, and the server is both a metadata server and a data server serving both roles under one common client ID, the server SHOULD return (EXCHGID4_FLAG_USE_PNFS_MDS | EXCHGID4_FLAG_USE_PNFS_DS) set 🔥

when a server has multiple concurrent pNFS roles, each one served by a unique client ID, and the client asks for zero or some combo of roles, the server results SHOULD return only one role from the combo the client requested. If the role in the server's result doesn't match what the client intended, the client should send the EXCHANGE_ID specifying just the pNFS role it's interested in 🌈

if a pNFS metadata client gets a layout that points it at an NFSv4.1 data server, it needs a client ID on that data server 🫶 If it doesn't already have one from the server that had EXCHGID4_FLAG_USE_PNFS_DS set in its EXCHANGE_ID results, then the client needs to send an EXCHANGE_ID to the data server, using the same co_ownerid it sent to the metadata server, with EXCHGID4_FLAG_USE_PNFS_DS set in the arguments. If the server's EXCHANGE_ID results have EXCHGID4_FLAG_USE_PNFS_DS set, the client MAY use that client ID to create sessions that exchange pNFS data operations. The client ID returned by the data server has no relationship with the client ID returned by a metadata server, unless the client IDs are equal and the server owners and server scopes of the data server and metadata server are equal 🦋

here's the tricky part 😭 In NFSv4.1, the session ID in the SEQUENCE operation implies the client ID, and the server might use that to map the stateid to the right client/server pair. But when a data server gets a READ or WRITE with a stateid, the stateid is tied to a client ID on a metadata server, while the session ID in the preceding SEQUENCE is tied to the data server's client ID. So the data server has no obvious way to figure out the metadata server from the COMPOUND procedure, which means it has no way to validate the stateid 💀 One RECOMMENDED approach is for pNFS servers to encode metadata server routing and/or identity information in the data server filehandles they return in the layout 🎀

if metadata server routing and/or identity information is encoded in data server filehandles, then when the metadata server's identity or location changes, the data server filehandles it handed out become invalid (stale), so the metadata server MUST first recall the layouts. Invalidating a data server filehandle doesn't make the NFS client's data cache invalid, though. The client's cache should map a data server filehandle to a metadata server filehandle, and a metadata server filehandle to cached data 🧠

if a server is both a metadata server and a data server, it might need to tell apart operations aimed at files on the metadata server from ones aimed at the data server. It is RECOMMENDED that the filehandles returned by the LAYOUTGET operation have different values from the filehandle returned by the OPEN of the same file 🌸

another scenario: the metadata server and the storage device might look distinct from one client's point of view but have their roles reversed from another client's point of view. For example, in the cluster file system model, a metadata server to one client might be a data server to another client. If NFSv4.1 is used as the storage protocol, then pNFS servers need to encode filehandle values according to their specific roles 🍓

13.1.1. Sessions Considerations for Data Servers 🌙

Section 2.10.11.2 says a client has to keep its lease renewed so the server doesn't delete its session 💅 If the reply to EXCHANGE_ID has just the EXCHGID4_FLAG_USE_PNFS_DS role set, then (as noted in Section 13.6) the client can't work out the data server's lease_time attribute, because GETATTR won't be permitted 😩 Instead, the rule is that any time a client gets a layout referring it to a data server that returns just the EXCHGID4_FLAG_USE_PNFS_DS role, the client MAY assume that the lease_time attribute from the metadata server that returned the layout applies to the data server. So the data server MUST be aware of the lease_time attribute values of all the metadata servers it's providing I/O for, and it MUST use the maximum of all those lease_time values as the lease interval for all client IDs and sessions established on it 👑

for example, say one metadata server has a lease_time of 20 seconds and a second metadata server has a lease_time of 10 seconds. If both servers return layouts that refer to an EXCHGID4_FLAG_USE_PNFS_DS-only data server, the data server MUST renew a client's lease if the interval between two SEQUENCE operations on different COMPOUND requests is less than 20 seconds ⏰✨

13.2. File Layout Definitions ✨

okay so these definitions are for the LAYOUT4_NFSV4_1_FILES layout type, but they might apply to other layout types too 🎀 let's go through them:

Unit. A unit is a fixed-size chunk of data written to a data server 🧁
Pattern. A pattern is a way of spreading one or more equal-sized units across a set of data servers 🌸 The pattern gets iterated one or more times.
Stripe. A stripe is a set of data spread across a set of data servers in a pattern before that pattern starts over again 💜
Stripe Count. The stripe count is how many units are in a pattern 🔥
Stripe Width. The stripe width is the size of a stripe in bytes. The stripe width = the stripe count * the size of the stripe unit ✨

From here on, this document will call a unit that is written in a pattern a "stripe unit" 🦋

A pattern can have more stripe units than there are data servers 👑 If that happens, some data servers will hold more than one stripe unit per stripe. A data server with multiple stripe units per stripe MAY store each unit in a different data file. Depending on the implementation, it might also give each data file its own unique data filehandle 🫶

13.3. File Layout Data Types 💅

the high-level NFSv4.1 layout types are nfsv4_1_file_layouthint4, nfsv4_1_file_layout_ds_addr4, and nfsv4_1_file_layout4 🧚

The SETATTR operation supports a layout hint attribute (Section 5.12.4). When the client sets a layout hint (data type layouthint4) with a layout type of LAYOUT4_NFSV4_1_FILES (the loh_type field), the loh_body field holds a value of data type nfsv4_1_file_layouthint4 🌈

const NFL4_UFLG_MASK            = 0x0000003F;
const NFL4_UFLG_DENSE           = 0x00000001;
const NFL4_UFLG_COMMIT_THRU_MDS = 0x00000002;
const NFL4_UFLG_STRIPE_UNIT_SIZE_MASK
                                = 0xFFFFFFC0;

typedef uint32_t nfl_util4;

enum filelayout_hint_care4 {
        NFLH4_CARE_DENSE        = NFL4_UFLG_DENSE,

        NFLH4_CARE_COMMIT_THRU_MDS
                                = NFL4_UFLG_COMMIT_THRU_MDS,

        NFLH4_CARE_STRIPE_UNIT_SIZE
                                = 0x00000040,

        NFLH4_CARE_STRIPE_COUNT = 0x00000080
};

/* Encoded in the loh_body field of data type layouthint4: */

struct nfsv4_1_file_layouthint4 {
        uint32_t        nflh_care;
        nfl_util4       nflh_util;
        count4          nflh_stripe_count;
};

The generic layout hint structure is described in Section 3.3.19. The client uses the layout hint in the layout_hint (Section 5.12.4) attribute to say which type of layout it prefers for a newly created file 🎀 The LAYOUT4_NFSV4_1_FILES layout-type-specific content of the layout hint has three fields. The first one, nflh_care, is a set of flags that tell the server which hint values the client cares about. If the NFLH4_CARE_DENSE flag is set, the client uses the second field, nflh_util, to state a preference for how the data file is packed (Section 13.4.4). That preference is controlled by the value of the expression nflh_util & NFL4_UFLG_DENSE ("&" is the bitwise AND operator) 💖 If the NFLH4_CARE_COMMIT_THRU_MDS flag is set, the client states a preference for whether it should send COMMIT operations to the metadata server or the data server (Section 13.7). That's controlled by nflh_util & NFL4_UFLG_COMMIT_THRU_MDS. If the NFLH4_CARE_STRIPE_UNIT_SIZE flag is set, the client states its preferred stripe unit size, which is found in nflh_util & NFL4_UFLG_STRIPE_UNIT_SIZE_MASK. So the stripe unit size MUST be a multiple of 64 bytes, and the minimum stripe unit size is 64 bytes 📏 If the NFLH4_CARE_STRIPE_COUNT flag is set, the client puts the stripe count in the third field, nflh_stripe_count. The stripe count multiplied by the stripe unit size is the stripe width 🧮

When LAYOUTGET returns a LAYOUT4_NFSV4_1_FILES layout (shown in the loc_type field of the lo_content field), the loc_body field of lo_content holds a value of data type nfsv4_1_file_layout4. Among other things, nfsv4_1_file_layout4 has a storage device ID (field nfl_deviceid) of data type deviceid4 🍓 The GETDEVICEINFO operation maps a device ID to a storage device address (type device_addr4). When GETDEVICEINFO returns a device address with a layout type of LAYOUT4_NFSV4_1_FILES (the da_layout_type field), the da_addr_body field holds a value of data type nfsv4_1_file_layout_ds_addr4 💜

typedef netaddr4 multipath_list4<>;

/*
 * Encoded in the da_addr_body field of
 * data type device_addr4:
 */
struct nfsv4_1_file_layout_ds_addr4 {
        uint32_t        nflda_stripe_indices<>;
        multipath_list4 nflda_multipath_ds_list<>;
};

The nfsv4_1_file_layout_ds_addr4 data type represents the device address 🌟 It has two fields:

nflda_multipath_ds_list: An array of lists of data servers. Each list can have one or more elements, and each element is a data server address that can equally serve as the target of I/O operations (see Section 13.5). This array's length might be different from the stripe count 🫧
nflda_stripe_indices: An array of indices used to index into nflda_multipath_ds_list. Each element of nflda_stripe_indices MUST be less than the number of elements in nflda_multipath_ds_list. Each element of nflda_multipath_ds_list SHOULD be referred to by one or more elements of nflda_stripe_indices. The number of elements in nflda_stripe_indices is always equal to the stripe count 🔥

/*
 * Encoded in the loc_body field of
 * data type layout_content4:
 */
struct nfsv4_1_file_layout4 {
         deviceid4      nfl_deviceid;
         nfl_util4      nfl_util;
         uint32_t       nfl_first_stripe_index;
         offset4        nfl_pattern_offset;
         nfs_fh4        nfl_fh_list<>;
};

The nfsv4_1_file_layout4 data type represents the layout 🎀 It has these fields:

nfl_deviceid: The device ID that maps to a value of type nfsv4_1_file_layout_ds_addr4 🧚
nfl_util: Like the nflh_util field of data type nfsv4_1_file_layouthint4, this is a compact way to show how the data in a file on each data server is packed, whether the client should send COMMIT operations to the metadata server or the data server, and the stripe unit size. If a server returns two or more overlapping layouts, each stripe unit size in each overlapping layout MUST be the same 📐
nfl_first_stripe_index: The index into the first element of the nflda_stripe_indices array to use 🌸
nfl_pattern_offset: The logical offset into the file where the striping pattern starts. It is needed to turn the client's logical I/O offset (for example, the current offset in a POSIX file descriptor before the read() or write() system call is sent) into the stripe unit number (see Section 13.4.1) 🦋

If dense packing is used, nfl_pattern_offset is also needed to turn the client's logical I/O offset into an offset in the file on the data server that matches the stripe unit number (see Section 13.4.4) 💖

Note that nfl_pattern_offset is not always the same as lo_offset. For example, a client might use LAYOUTGET to ask for a layout starting at offset 1000 of a file whose striping pattern starts at offset zero 🙈

nfl_fh_list: An array of data server filehandles, one for each list of data servers in each element of the nflda_multipath_ds_list array. The number of elements in nfl_fh_list depends on whether sparse or dense packing is being used 🍓
- If sparse packing is being used, the number of elements in nfl_fh_list MUST be one of three values:
  - Zero. This means the filehandle used for each data server is the same as the filehandle returned by the OPEN operation from the metadata server 🌈
  - One. This means every data server uses the same filehandle: whatever is in nfl_fh_list[0] 🫶
  - The same number of elements as nflda_multipath_ds_list. In this case, when sending an I/O operation to any data server in nflda_multipath_ds_list[X], the filehandle in nfl_fh_list[X] MUST be used.

See the discussion on sparse packing in Section 13.4.4.

If dense packing is being used, the number of elements in nfl_fh_list MUST be the same as the number of elements in nflda_stripe_indices. So when sending an I/O operation to any data server in nflda_multipath_ds_list[nflda_stripe_indices[Y]], the filehandle in nfl_fh_list[Y] MUST be used. Also, any time there are i and j (i != j) such that the intersection of nflda_multipath_ds_list[nflda_stripe_indices[i]] and nflda_multipath_ds_list[nflda_stripe_indices[j]] is not empty, then nfl_fh_list[i] MUST NOT equal nfl_fh_list[j]. In other words, when dense packing is used and a data server appears in two or more units of a striping pattern, each reference to that data server MUST use a different filehandle 🔥

Indeed, if there are multiple striping patterns (shown by multiple objects of data type layout4, returned in one or several LAYOUTGET operations), and a data server is the target of a unit in one pattern and a unit in another pattern, then each reference to each data server MUST use a different filehandle 💅

See the discussion on dense packing in Section 13.4.4.

The details on how to interpret the layout are in Section 13.4 ✨

13.4. Interpreting the File Layout 🧁

13.4.1. Determining the Stripe Unit Number 🌸

to find the stripe unit number that matches the client's logical file offset, the pattern offset is also used 💜 The i'th stripe unit (SUi) is:

    relative_offset = file_offset - nfl_pattern_offset;
    SUi = floor(relative_offset / stripe_unit_size);

13.4.2. Interpreting the File Layout Using Sparse Packing

When sparse packing is used, the algorithm for determining the filehandle and set of data-server network addresses to write stripe unit i (SUi) to is:

stripe_count = number of elements in nflda_stripe_indices;

j = (SUi + nfl_first_stripe_index) % stripe_count;

idx = nflda_stripe_indices[j];

fh_count = number of elements in nfl_fh_list; ds_count = number of elements in nflda_multipath_ds_list;

switch (fh_count) { case ds_count: fh = nfl_fh_list[idx]; break;

case 1: fh = nfl_fh_list[0]; break;

case 0: fh = filehandle returned by OPEN; break;

default: throw a fatal exception; break; }

address_list = nflda_multipath_ds_list[idx];

The client would then select a data server from address_list, and send a READ or WRITE operation using the filehandle specified in fh.

Consider the following example:

Suppose we have a device address consisting of seven data servers, arranged in three equivalence (Section 13.5) classes:

{ A, B, C, D }, { E }, { F, G }

where A through G are network addresses.

Then

nflda_multipath_ds_list<> = { A, B, C, D }, { E }, { F, G }

i.e.,

nflda_multipath_ds_list[0] = { A, B, C, D }

nflda_multipath_ds_list[1] = { E }

nflda_multipath_ds_list[2] = { F, G }

Suppose the striping index array is:

nflda_stripe_indices<> = { 2, 0, 1, 0 }

Now suppose the client gets a layout that has a device ID that maps to the above device address. The initial index contains

nfl_first_stripe_index = 2,

and the filehandle list is

nfl_fh_list = { 0x36, 0x87, 0x67 }.

If the client wants to write to SU0, the set of valid { network address, filehandle } combinations for SUi are determined by:

nfl_first_stripe_index = 2

idx = nflda_stripe_indices[(0 + 2) % 4]

= nflda_stripe_indices[2]

= 1

nflda_multipath_ds_list[1] = { E }

and

nfl_fh_list[1] = { 0x87 }

The client can thus write SU0 to { 0x87, { E } }.

The destinations of the first 13 storage units are:

+=====+============+==============+

SUi

filehandle

data servers

+=====+============+==============+

+-----+------------+--------------+

A,B,C,D

+-----+------------+--------------+

F,G

+-----+------------+--------------+

A,B,C,D

+-----+------------+--------------+ +-----+------------+--------------+

+-----+------------+--------------+

A,B,C,D

+-----+------------+--------------+

F,G

+-----+------------+--------------+

A,B,C,D

+-----+------------+--------------+ +-----+------------+--------------+

+-----+------------+--------------+

A,B,C,D

+-----+------------+--------------+

F,G

+-----+------------+--------------+

A,B,C,D

+-----+------------+--------------+ +-----+------------+--------------+

+-----+------------+--------------+

Table 9

13.4.3. Interpreting the File Layout Using Dense Packing

When dense packing is used, the algorithm for determining the filehandle and set of data server network addresses to write stripe unit i (SUi) to is:

stripe_count = number of elements in nflda_stripe_indices;

j = (SUi + nfl_first_stripe_index) % stripe_count;

idx = nflda_stripe_indices[j];

fh_count = number of elements in nfl_fh_list; ds_count = number of elements in nflda_multipath_ds_list;

switch (fh_count) { case stripe_count: fh = nfl_fh_list[j]; break;

default: throw a fatal exception; break; }

address_list = nflda_multipath_ds_list[idx];

The client would then select a data server from address_list, and send a READ or WRITE operation using the filehandle specified in fh.

Consider the following example (which is the same as the sparse packing example, except for the filehandle list):

Suppose we have a device address consisting of seven data servers, arranged in three equivalence (Section 13.5) classes:

{ A, B, C, D }, { E }, { F, G }

where A through G are network addresses.

Then

nflda_multipath_ds_list<> = { A, B, C, D }, { E }, { F, G }

i.e.,

nflda_multipath_ds_list[0] = { A, B, C, D }

nflda_multipath_ds_list[1] = { E }

nflda_multipath_ds_list[2] = { F, G }

Suppose the striping index array is:

nflda_stripe_indices<> = { 2, 0, 1, 0 }

Now suppose the client gets a layout that has a device ID that maps to the above device address. The initial index contains

nfl_first_stripe_index = 2,

and

nfl_fh_list = { 0x67, 0x37, 0x87, 0x36 }.

The interesting examples for dense packing are SU1 and SU3 because each stripe unit refers to the same data server list, yet each stripe unit MUST use a different filehandle. If the client wants to write to SU1, the set of valid { network address, filehandle } combinations for SUi are determined by:

nfl_first_stripe_index = 2

j = (1 + 2) % 4 = 3

idx = nflda_stripe_indices[j]

= nflda_stripe_indices[3]

= 0

nflda_multipath_ds_list[0] = { A, B, C, D }

and

nfl_fh_list[3] = { 0x36 }

The client can thus write SU1 to { 0x36, { A, B, C, D } }.

For SU3, j = (3 + 2) % 4 = 1, and nflda_stripe_indices[1] = 0. Then nflda_multipath_ds_list[0] = { A, B, C, D }, and nfl_fh_list[1] = 0x37. The client can thus write SU3 to { 0x37, { A, B, C, D } }.

The destinations of the first 13 storage units are:

+=====+============+==============+

SUi

filehandle

data servers

+=====+============+==============+

+-----+------------+--------------+

A,B,C,D

+-----+------------+--------------+

F,G

+-----+------------+--------------+

A,B,C,D

+-----+------------+--------------+ +-----+------------+--------------+

+-----+------------+--------------+

A,B,C,D

+-----+------------+--------------+

F,G

+-----+------------+--------------+

A,B,C,D

+-----+------------+--------------+ +-----+------------+--------------+

+-----+------------+--------------+

A,B,C,D

+-----+------------+--------------+

F,G

+-----+------------+--------------+

A,B,C,D

+-----+------------+--------------+ +-----+------------+--------------+

+-----+------------+--------------+

Table 10

13.4.4. Sparse and Dense Stripe Unit Packing 🧩

okay bestie, so the flag NFL4_UFLG_DENSE in the nfl_util4 data type decides how the data is packed inside each data file on a data server ✨ You'll find it as field nflh_util of nfsv4_1_file_layouthint4 and as field nfl_util of nfsv4_1_file_layout_ds_addr4. There are two packing styles, sparse and dense, and the one in use determines the calculation that maps the client-visible file offset to the offset inside the data file on the data server 💅

If nfl_util & NFL4_UFLG_DENSE is zero, sparse packing is being used 🌸 That means the logical offsets a client sees when it sends READs and WRITEs directly to the metadata server are the same offsets each data server uses when it stores a stripe unit. So for a striping pattern with at least two stripe units, each data server file ends up sparse, or "holey" 🕳️

Here's an example. Say there's a pattern with three stripe units, the stripe unit size is 4096 bytes, and there are three data servers in the pattern. Then the file on data server 1 has stripe units 0, 3, 6, 9, ... filled. Data server 2's file has stripe units 1, 4, 7, 10, ... filled. Data server 3's file has stripe units 2, 5, 8, 11, ... filled. The unfilled stripe units in each file are holes, which is why those files are sparse 🦋

If sparse packing is being used and a client attempts I/O to one of the holes, then an error MUST be returned by the data server 🚫 Using the example above, if data server 3 received a READ or WRITE for block 4, it would return NFS4ERR_PNFS_IO_HOLE. So data servers need to understand the striping pattern in order to support sparse packing 🧠

If nfl_util & NFL4_UFLG_DENSE is one, dense packing is being used and the data server files have no holes ✨ A data server might pick dense packing because it doesn't (efficiently) support holey files, or because it can't recognize read-ahead unless there are no holes. If dense packing is indicated in the layout, the data files get packed. Using the same striping pattern and stripe unit size as the sparse example, the dense version has all stripe units of all data files filled like this:

Logical stripe units 0, 3, 6, ... of the file would live on stripe units 0, 1, 2, ... of the file of data server 1.
Logical stripe units 1, 4, 7, ... of the file would live on stripe units 0, 1, 2, ... of the file of data server 2.
Logical stripe units 2, 5, 8, ... of the file would live on stripe units 0, 1, 2, ... of the file of data server 3.

Because dense packing doesn't leave holes on the data servers, the pNFS client is allowed to write to any offset of any data file of any data server in the stripe 💜 That means the data servers don't need to know the file's striping pattern, which is honestly so clean 🫶

The calculation to determine the byte offset within the data file for dense data server layouts is:

      stripe_width = stripe_unit_size * N;
         where N = number of elements in nflda_stripe_indices.

      relative_offset = file_offset - nfl_pattern_offset;

      data_file_offset = floor(relative_offset / stripe_width)
         * stripe_unit_size
         + relative_offset % stripe_unit_size

If dense packing is being used, and a data server appears more than once in a striping pattern, then the data server MUST use a different filehandle to tell one stripe unit from another 🙈 Let's say there are two data servers. Logical stripe units 0, 3, 6 are served by data server 1. Logical stripe units 1, 4, 7 are served by data server 2. Logical stripe units 2, 5, 8 are also served by data server 2. Unless data server 2 has two filehandles (each referring to a different data file), then, for example, a write to logical stripe unit 1 overwrites the write to logical stripe unit 2, because both logical stripe units are located in the same stripe unit (0) of data server 2 😭

Two filehandles, or the writes collide. Don't skip this one, bestie 💀

13.5. Data Server Multipathing 🛣️

The NFSv4.1 file layout supports multipathing to multiple data server addresses 🦋 Data-server-level multipathing is used for bandwidth scaling via trunking (Section 2.10.5), and for higher availability in the case of a data-server failure. Multipathing lets the client switch to another data server address, which may belong to another data server exporting the same data stripe unit, without having to contact the metadata server for a new layout ✨

To support data server multipathing, each element of the nflda_multipath_ds_list contains an array of one or more data server network addresses. This array (data type multipath_list4) represents a list of data servers, each identified by a network address, with the possibility that some data servers will appear in the list multiple times 💖

The client is free to use any of the network addresses as a destination to send data server requests. If some network addresses are less optimal paths to the data than others, then the MDS SHOULD NOT include those network addresses in an element of nflda_multipath_ds_list 🙅‍♀️ If less optimal network addresses exist to provide failover, the RECOMMENDED method to offer the addresses is to provide them in a replacement device-ID-to-device-address mapping, or a replacement device ID 🎀

When a client finds that no data server in an element of nflda_multipath_ds_list responds, it SHOULD send a GETDEVICEINFO to attempt to replace the existing device-ID-to-device-address mappings 🔥 If the MDS detects that all data servers represented by an element of nflda_multipath_ds_list are unavailable, the MDS SHOULD send a CB_NOTIFY_DEVICEID (if the client has indicated it wants device ID notifications for changed device IDs) to change the device-ID-to-device-address mappings to the available data servers 🌈 If the device ID itself will be replaced, the MDS SHOULD recall all layouts with the device ID, and thus force the client to get new layouts and device ID mappings via LAYOUTGET and GETDEVICEINFO 👑

Generally, if two network addresses appear in an element of nflda_multipath_ds_list, they will designate the same data server, and the two data server addresses will support the implementation of client ID or session trunking (the latter is RECOMMENDED) as defined in Section 2.10.5 🧚 The two data server addresses will share the same server owner or major ID of the server owner. It is not always necessary for the two data server addresses to designate the same server with trunking being used. For example, the data could be read-only, and the data consist of exact replicas 🍓

13.6. Operations Sent to NFSv4.1 Data Servers 💅

Okay bestie, listen up 👑 If you're a client poking at data on an NFSv4.1 data server, you MUST send only the NULL procedure and COMPOUND procedures built from operations in two restricted groups of valid NFSv4.1 operations. ngl, it's a strict guest list 🙈 You MUST use the filehandle the layout gives you whenever you access data on NFSv4.1 data servers ✨

The first group is the management crew 🧚 It's BACKCHANNEL_CTL, BIND_CONN_TO_SESSION, CREATE_SESSION, DESTROY_CLIENTID, DESTROY_SESSION, EXCHANGE_ID, SECINFO_NO_NAME, SET_SSV, and SEQUENCE. The client may use these to set up and maintain the client IDs, sessions, and security contexts it needs for talking to the data server. From here on, we'll call these data-server housekeeping operations 🧹

The second group is COMMIT, READ, WRITE, and PUTFH 🦋 These MUST be used with a current filehandle specified by the layout. For PUTFH, the new current filehandle MUST be one taken from the layout. We'll call these data-server I/O operations. As described in Section 12.5.1, a client MUST NOT send an I/O to a data server for which it does not hold a valid layout, and the data server MUST reject such an I/O 🚫

Unless the server has a concurrent non-data-server personality (aka, EXCHANGE_ID results returned with EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_PNFS_MDS or EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_NON_PNFS, see Section 13.1), any attempted use of operations against a data server other than the ones in those two groups MUST return NFS4ERR_NOTSUPP to the client. That's the vibe, no exceptions 💀

When the server has concurrent data-server and non-data-server personalities, each COMPOUND sent by the client MUST be constructed so that it fits one of the two personalities, and it MUST NOT contain operations aimed at a mix of those personalities. The server MUST enforce this 🔥 To understand the constraints, operations within a COMPOUND get sorted into three classes:

An operation that is ambiguous about its personality. This covers all of the data-server housekeeping operations. Also, if the server has assigned filehandles so that the ones defined by the layout match the ones the metadata server uses, all operations using those filehandles land in this class, with one exception: if the operation uses a stateid that doesn't work with a data-server personality (like a special stateid, or a stateid whose seqid field is non-zero, see Section 13.9.1), it's in class 3, as described below. A COMPOUND containing multiple class 1 operations (and nothing from any other class) MAY be sent to a server with multiple concurrent data server and non-data-server personalities.
An operation that is unambiguously for the data-server personality. This covers data-server I/O operations where the filehandle can only be validly sent to the data-server personality.
An operation that is unambiguously for the non-data-server personality. This covers all COMPOUND operations that are neither data-server housekeeping nor data-server I/O operations, plus data-server I/O operations where the current filehandle (or the one about to become current via PUTFH) is only valid on the metadata server, or where a stateid is used that doesn't work with the data server, meaning it's a special stateid or has a non-zero seqid value.

When a COMPOUND first runs an operation from class 3 above, it acts like a normal COMPOUND on any other server, and the data-server personality stops mattering 😌 There are no special restrictions on the operations in the COMPOUND that limit them to the ones for a data server. When a PUTFH is done, filehandles derived from the layout are not valid. If their format is not normally acceptable, then NFS4ERR_BADHANDLE MUST result. Similarly, current filehandles for other operations don't accept filehandles derived from layouts and aren't normally usable on the metadata server. Using these will result in NFS4ERR_STALE.

When a COMPOUND first runs an operation from class 2, which would be a PUTFH where the filehandle comes from a layout, the COMPOUND from then on gets interpreted with respect to the data-server personality ✨ Operations outside the two classes discussed above MUST result in NFS4ERR_NOTSUPP. Filehandles are validated using the rules of the data server, which can result in NFS4ERR_BADHANDLE and/or NFS4ERR_STALE even when they wouldn't normally do so when addressed to the non-data-server personality. Stateids must obey the rules of the data server, so any use of special stateids, or stateids with non-zero seqid values, must result in NFS4ERR_BAD_STATEID.

Until the server first runs an operation from class 2 or class 3, the client MUST NOT depend on the operation being run by either the data-server or the non-data-server personality 🫣 The server MUST pick one personality consistently for a given COMPOUND, and the only possible switch is a single one, happening when the first operation from class 2 or class 3 runs.

Because assigning filehandles so they work on both a data server and a metadata server gets really complicated, it is RECOMMENDED that, where the same server can have both personalities, the server assigns separate unique filehandles to each personality. This makes it unambiguous which server a given request is meant for 🎀

GETATTR and SETATTR MUST be directed to the metadata server 💜 For a SETATTR of the size attribute, the control protocol is in charge of propagating size updates and truncations to the data servers. For extending WRITEs to the data servers, the new size must be visible on the metadata server once a LAYOUTCOMMIT has completed (see Section 12.5.4.2). Section 13.10 describes how the client handles data-server files that don't reflect the metadata server's size.

13.7. COMMIT through Metadata Server 🌸

The file layout gives you two alternate ways to commit data that was written through data servers 🧁 The flag NFL4_UFLG_COMMIT_THRU_MDS in the field nfl_util of the file layout (data type nfsv4_1_file_layout4) is the metadata server telling the client the REQUIRED way to do COMMIT, either by sending it to the data server or to the metadata server. These two methods match up with broad styles of implementation for a pNFS server that supports the file layout type.

When the flag is FALSE, COMMIT operations MUST to be sent to the data server that the corresponding WRITE operations were sent to. This approach is sometimes useful when file striping is implemented within the pNFS server (instead of the file system), with each individual data server running its own file system.
When the flag is TRUE, COMMIT operations MUST be sent to the metadata server, rather than to the individual data servers. This approach is sometimes useful when file striping is implemented within the clustered file system that backs the pNFS server. In that kind of setup, each COMMIT to each data server might cause repeated writes of metadata blocks, which hurts write performance 😩 Sending one COMMIT to the metadata server can be more efficient when there's a clustered file system that can handle a coordinated COMMIT.

If nfl_util & NFL4_UFLG_COMMIT_THRU_MDS is TRUE, then, to keep the current NFSv4.1 commit and recovery model, the data servers MUST return a common writeverf verifier in all WRITE responses for a given file layout, and the metadata server's COMMIT implementation must return the same writeverf. The value of the writeverf verifier MUST be changed at the metadata server or at any data server referenced in the layout, whenever there's a server event that could possibly lead to loss of uncommitted data. The scope of the verifier can be per file or for the entire pNFS server. It might be harder for the server to keep the verifier at the file level, but the upside is that only events affecting a given file will need recovery action 🔥

Note that if the layout specified dense packing, the offset used for a COMMIT to the MDS may differ from the offset used for a COMMIT to the data server 🙈

The single COMMIT to the metadata server returns a verifier, and the client should compare it to all the verifiers from the WRITEs and fail the COMMIT if there are any mismatched verifiers. If COMMIT to the metadata server fails, the client should re-send WRITEs for all the modified data in the file. The client should treat modified data with a mismatched verifier as a WRITE failure, and try to recover by resending the WRITEs to the original data server, or by using another path to that data if the layout hasn't been recalled. Alternatively, the client can get a new layout, or it could rewrite the data directly to the metadata server. If nfl_util & NFL4_UFLG_COMMIT_THRU_MDS is FALSE, sending a COMMIT to the metadata server might have no effect. If nfl_util & NFL4_UFLG_COMMIT_THRU_MDS is FALSE, a COMMIT sent to the metadata server should be used only to commit data that was written to the metadata server. See Section 12.7.6 for recovery options 🧚

13.8. The Layout Iomode 🦋

The layout iomode doesn't need to be used by the metadata server when it's servicing NFSv4.1 file-based layouts, although in some situations it can be useful 🤷‍♀️ For example, if the server implementation supports reading from read-only replicas or mirrors, it would be useful for the server to hand back a layout that lets the client do that. So the client SHOULD set the iomode based on whether it intends to read or write the data 💖 The client may default to an iomode of LAYOUTIOMODE4_RW. The iomode need not be checked by the data servers when clients perform I/O. However, the data servers SHOULD still validate that the client holds a valid layout, and return an error if the client doesn't ✨

13.9. Metadata and Data Server State Coordination 🫶

13.9.1. Global Stateid Requirements ✨

okay bestie, this is the part where we talk about which stateids a client is allowed to wave around when it sends I/O to a data server 💅🎀 the big vibe is: a layout stateid is a total no-go. LAYOUTGET hands it out and CB_LAYOUTRECALL sends it, but it MUST NOT be used for I/O to a data server 🙈 the stateids that ARE allowed come from three places: an OPEN stateid (the stateid field of OPEN4resok returned by OPEN), a delegation stateid (the stateid field of open_read_delegation4 and open_write_delegation4, returned by OPEN or WANT_DELEGATION, or sent by CB_PUSH_DELEG), or a stateid returned by LOCK or LOCKU 🦋 and the stateid sent to the data server MUST have its seqid set to zero, which means "give me the most current version" instead of pinning a specific non-zero seqid 🧚 and special stateid values are never allowed, no exceptions, not even a little bit 💀

the wild part is that a stateid has to act exactly the same on a data server as it would on the metadata server if you weren't using pNFS at all 🌈 so stateids are globally valid everywhere! that means the metadata server has to pass LOCK and OPEN state changes along to the data servers so they can validate I/O 🫶 we dig into that in Section 13.9.2, and depending on timing, having a valid stateid on the data server can basically prove you have a valid layout 👑

clients have to pick their stateid based on the locks, opens, and delegations they hold, plus whatever kind of state-owner is sending the request 💜 and it's not quite the same as the rules in Section 8.2.5 for metadata servers, so pay attention 😭

the selection rules, in order of decreasing priority:

if the client holds a delegation on the file, the delegation stateid SHOULD be used 🔥
otherwise there MUST be an OPEN stateid for the current open-owner, and you use that OPEN stateid for the open file unless mandatory locking gets in the way (more on that below) ✨
if the data server earlier answered NFS4ERR_LOCKED to the OPEN stateid, the client SHOULD use the byte-range lock stateid whenever one exists for that open file with the current lock-owner 🌸
special stateids should never be used, and if they are, the data server MUST reject the I/O with NFS4ERR_BAD_STATEID 🙅‍♀️💀

the big takeaway: special stateids are a hard no, and a data server that sees one MUST throw an NFS4ERR_BAD_STATEID error back at you 😭

13.9.2. Data Server State Propagation 🦋

okay so the metadata server handles byte-range lock and open-mode state changes, plus ACLs, but it might not live on the same box as the data servers where I/O actually gets validated 🥺 so the server implementation MUST take care of propagating those changes to the data servers. once propagation is complete, the full effect of those changes MUST be in effect at the data servers ✨ some state changes don't have to go out right away, but all changes SHOULD be propagated promptly 🎀 these propagation needs shape the control protocol design, even though the control protocol is technically outside this spec 🧚 immediate propagation just means the metadata server pushes state to the data server(s) synchronously, and the propagation has to finish before the server returns to the client 💖

13.9.2.1. Lock State Propagation 🔐

if the pNFS server supports mandatory byte-range locking, any mandatory byte-range locks on a file MUST be in effect at the data servers before the request that creates them returns to the caller ✨ the effect MUST match what you'd get if that lock state were synchronously propagated, even though the control protocol might skip actually moving the state around in some situations 🌈

advisory byte-range lock state is a different story 🙃 data servers don't use it to check I/O, so there's no real semantic reason to propagate it. updates to advisory locks don't grant or take away any privileges, so they don't need to go out immediately, and might not need to go out promptly at all 💅 they only need to be propagated when the data server has to resolve a question about a stateid. and if byte-range locking is advisory (not mandatory), clients are advised to avoid using byte-range lock-based stateids for I/O 🙈 the stateids that OPEN returns are enough, and they save everyone the overhead 💜

if a client gets NFS4ERR_LOCKED back from a data server, that means mandatory byte-range locking is in force 🔥 the client fixes this by getting a byte-range lock covering the affected range, then re-sends the I/O using that lock's stateid 🫶

13.9.2.2. Open and Deny Mode Validation 🚪

open and deny mode validation MUST be performed against the open and deny mode(s) the data servers hold 🎀 when access gets reduced or a deny mode gets stricter (because of CLOSE or OPEN_DOWNGRADE), the data server MUST block any I/O that would have been denied on the metadata server 🙅‍♀️ when access expands, the data server MUST make sure it doesn't reject requests because of open or deny issues that no longer apply after that loosening 🦋

13.9.2.3. File Attributes 📝

the SETATTR operation can change state that's visible on both the metadata and data servers (like size) 👀 so you have to be careful that the state across all the data servers stays consistent, especially when truncating or growing a file 💖

as we covered earlier, LAYOUTCOMMIT is what keeps metadata in sync with changes made on the data servers ✨ for the NFSv4.1-based data storage protocol, state like the size attribute, and the mtime/change/atime settings, needs to be re-synchronized 🧚 see Section 12.5.4 for the full semantics of LAYOUTCOMMIT and attribute sync 📚 it's worth noting that with an NFSv4.1-based layout type, you can sync this state before LAYOUTCOMMIT happens 🌸 for example, the control protocol can be used to query the attributes on the data servers 🫶

any changes to file attributes that control authorization or access (the ones reflected by ACCESS calls, or by READs and WRITEs on the metadata server) MUST be propagated to the data servers so they can be enforced on READ and WRITE I/O calls 🔐 if the changes on the metadata server make access permissions more restrictive for any user, those changes MUST be propagated to the data servers synchronously 🙅‍♀️

the OPEN operation (Section 18.16.4) doesn't require I/O on an open file to use the same credentials as the OPEN itself (unless EXCHGID4_FLAG_BIND_PRINC_STATEID is set when EXCHANGE_ID creates the client ID) 🫣 so the server's READ and WRITE operations have to do their own appropriate access checking 💅 changes to ACLs also require new access checking by READ and WRITE on the server ✨ propagating access-right changes from ACL changes MAY be asynchronous, but only if the server can tell the updated ACL isn't more restrictive for any user listed in the old ACL 🌈 ACL updates are pretty rare, so it's suggested that all changes be propagated synchronously 💖

13.10. Data Server Component File Size 📏

okay this one is kinda spicy 😭 there's a potential problem when a component data file on a particular data server has grown past EOF, and it happens for both dense and sparse layouts 🙈 picture this: a client creates a new file (size == 0), writes to byte 131072, then seeks back to the beginning and reads byte 100 💜 the client should get zeroes back from the READ 🌸 but if striping sends that READ to a data server other than the one that got the original WRITE, the server handling the READ might think the file is still 0 bytes long 💀 then its READ response has zero bytes and an EOF indication 😱 a data server can only return zeroes if it knows the file's size was extended, and telling every data server about the new size right away would be potentially super costly 💸

so the client that started the size extension MUST be ready to deal with these EOF conditions 🫶 when the offset in the READ arguments is less than the client's view of the file size, and the READ response says EOF and/or has fewer bytes than requested, the client WILL interpret that response as a hole in the file, and the NFS client will substitute zeroes for the data ✨

NFSv4.1 only gives close-to-open file data cache semantics 🔁 that means when a file is closed, all modified data gets written to the server 💅 when a later OPEN of the file happens, the change attribute gets checked for a difference from the cached value 👀 in the example above, that means a LAYOUTCOMMIT gets done at close (along with the data WRITEs), which updates the file's size and change attribute 🎀 after that point, access from another client returns the right size 🦋

Part 88 wrapped up, bestie! 🌈 more parts are coming, so stay tuned 💖

13.11. Layout Revocation and Fencing 🔒

okay so, remember Section 12.7? 💅 That's where the layout-type-specific storage protocol gets the job of handling I/Os that started before lease expiration and kept going right through it 🫠. The LAYOUT4_NFSV4_1_FILES layout type can stop all I/Os to data servers from running after lease expiration. That's called "fencing" 🚧, and it doesn't need a super precise client lease timer or data servers keeping their own lease timers. Lowkey genius. The LAYOUT4_NFSV4_1_FILES pNFS server can revoke individual layouts too, so it can fence I/O on a per-file basis 🎯.

Lease expiration isn't the only reason a layout gets revoked 🙈. The others are: the client fails to respond to a CB_LAYOUTRECALL, the metadata server restarts, or there's administrative intervention. No matter the reason, once a client's layout has been revoked, the pNFS server MUST prevent the client from sending I/O for the affected file from and to all data servers. In other words, it MUST fence the client from the affected file on the data servers. No escape hatch, bestie 🚫.

Here's how fencing works ✨. As described in Section 13.1, in COMPOUND requests to the data server, the data filehandle given by the PUTFH operation and the stateid in the READ or WRITE operation are used to make sure the client has a valid layout for the I/O being done. If it doesn't, the I/O gets rejected with NFS4ERR_PNFS_NO_LAYOUT. The server can just check the stateid and, on top of that, make the data filehandle stale if the layout named a data filehandle that's different from the metadata server's filehandle for the file (see the nfl_fh_list description in Section 13.3). Double-check energy 💅.

Before the metadata server takes any action to revoke layout state given out by a previous instance, it has to make sure all layout state from that previous instance is invalidated at the data servers. This has the following implications, and they're a whole vibe 🌸.

The metadata server must not restripe a file until it has contacted all of the data servers to invalidate the layouts from the previous instance. 🧱
The metadata server must not give out mandatory locks that conflict with layouts from the previous instance without either doing a specific layout invalidation (as it would have to do anyway) or doing a global data server invalidation. 🔐

13.12. Security Considerations for the File Layout Type 🛡️

The NFSv4.1 file layout type MUST adhere to the security considerations outlined in Section 12.9. NFSv4.1 data servers MUST make all of the required access checks on each READ or WRITE I/O as determined by the NFSv4.1 protocol. If the metadata server would deny a READ or WRITE operation on a file because of its ACL, mode attribute, open access mode, open deny mode, mandatory byte-range lock state, or any other attributes and state, the data server MUST also deny the READ or WRITE operation. This messes with the control protocol and with how state gets propagated from the metadata server to the data servers, so see Section 13.9.2 for more details. ⚠️

The methods for authentication, integrity, and privacy for data servers based on the LAYOUT4_NFSV4_1_FILES layout type are the same as the ones used by metadata servers. Metadata and data servers use ONC RPC security flavors to authenticate, and SECINFO and SECINFO_NO_NAME to negotiate the security mechanism and services to be used. So when you're using the LAYOUT4_NFSV4_1_FILES layout type, the impact on the RPC-based security model from pNFS (as alluded to in Sections 1.8.1 and 1.8.2.2) is zero. Iconic, honestly 👑.

For a given file object, a metadata server MAY require different security parameters (secinfo4 value) than the data server. For a given file object with multiple data servers, the secinfo4 value SHOULD be the same across all data servers. If the secinfo4 values across a metadata server and its data servers differ for a specific file, the mapping of the principal to the server's internal user identifier MUST be the same in order for the access-control checks based on ACL, mode, open and deny mode, and mandatory locking to be consistent across on the pNFS server. Mapping consistency is everything, no cap 🫶.

If an NFSv4.1 implementation supports pNFS and supports NFSv4.1 file layouts, then the implementation MUST support the SECINFO_NO_NAME operation on both the metadata and data servers. 🔑

14. Internationalization 🌍

The main thing NFSv4.1 has to handle for internationalization, or I18N, is file names and other strings as used within the protocol. The choice of string representation has to allow reasonable name and string access for clients that use all kinds of languages 🗣️. The UTF-8 encoding of the UCS (Universal Multiple-Octet Coded Character Set) as defined by ISO10646 [18] allows for this type of access and follows the policy described in "IETF Policy on Character Sets and Languages", RFC 2277 [19].

RFC 3454 [16], otherwise known as "stringprep", documents a framework for using Unicode/UTF-8 in networking protocols so as "to increase the likelihood that string input and string comparison work in ways that make sense for typical users throughout the world" 🌎. A protocol has to define a profile of stringprep "in order to fully specify the processing options". The rest of this section defines the NFSv4.1 stringprep profiles, and a lot of the terminology comes straight from stringprep 📚.

There are three UTF-8 string types defined for NFSv4.1: utf8str_cs, utf8str_cis, and utf8str_mixed. Each one gets its own profile. Each profile defines the following, as required by stringprep:

The intended applicability of the profile.
The character repertoire that is the input and output to stringprep (which is Unicode 3.2 for the referenced version of stringprep). However, NFSv4.1 implementations are not limited to 3.2.
The mapping tables from stringprep used (as described in Section 3 of stringprep).
Any additional mapping tables specific to the profile.
The Unicode normalization used, if any (as described in Section 4 of stringprep).
The tables from the stringprep listing of characters that are prohibited as output (as described in Section 5 of stringprep).
The bidirectional string testing used, if any (as described in Section 6 of stringprep).
Any additional characters that are prohibited as output specific to the profile.

Stringprep talks about Unicode characters, whereas NFSv4.1 renders UTF-8 characters. Since there's a one-to-one mapping from UTF-8 to Unicode, whenever the rest of this document says Unicode, you should just assume UTF-8 🫠. Much of the text for the profiles comes from RFC 3491 [20].

14.1. Stringprep Profile for the utf8str_cs Type 🔠

Every use of the utf8str_cs type definition in the NFSv4 protocol specification follows the profile named nfs4_cs_prep.

14.1.1. Intended Applicability of the nfs4_cs_prep Profile

The utf8str_cs type is a case-sensitive string of UTF-8 characters. Its main use in NFSv4.1 is for naming components and pathnames. Components and pathnames get stored on the server's file system. Two valid, distinct UTF-8 strings might end up being the same after processing via the utf8str_cs profile. If the strings are two names inside a directory, the NFSv4.1 server will need to either:

disallow the creation of a second name if its post-processed form collides with that of an existing name, or
allow the creation of the second name, but arrange so that after post-processing, the second name is different than the post-processed form of the first name.

14.1.2. Character Repertoire of nfs4_cs_prep

The nfs4_cs_prep profile uses Unicode 3.2, as defined in stringprep's Appendix A.1. However, NFSv4.1 implementations are not limited to 3.2. 🌈

14.1.3. Mapping Used by nfs4_cs_prep

The nfs4_cs_prep profile specifies mapping using the following tables from stringprep:

Table B.1

Table B.2 is normally not part of the nfs4_cs_prep profile, since it's mainly for dealing with case-insensitive comparisons. However, if the NFSv4.1 file server supports the case_insensitive file system attribute, and if case_insensitive is TRUE, the NFSv4.1 server MUST use Table B.2 (in addition to Table B1) when processing utf8str_cs strings, and the NFSv4.1 client MUST assume Table B.2 (in addition to Table B.1) is being used. 🔤

If the case_preserving attribute is present and set to FALSE, then the NFSv4.1 server MUST use Table B.2 to map case when processing utf8str_cs strings. Whether the server maps from lower to upper case or from upper to lower case is an implementation dependency. 🙃

14.1.4. Normalization used by nfs4_cs_prep

The nfs4_cs_prep profile doesn't specify a normalization form. A later revision of this specification may specify a particular normalization form. Therefore, the server and client can expect that they may receive unnormalized characters within protocol requests and responses. If the operating environment requires normalization, then the implementation must normalize utf8str_cs strings within the protocol before presenting the information to an application (at the client) or local file system (at the server). 🧼

14.1.5. Prohibited Output for nfs4_cs_prep

The nfs4_cs_prep profile RECOMMENDS prohibiting the use of the following tables from stringprep:

Table C.5
Table C.6

14.1.6. Bidirectional Output for nfs4_cs_prep

The nfs4_cs_prep profile doesn't specify any checking of bidirectional strings. 🙅‍♀️

14.2. Stringprep Profile for the utf8str_cis Type 🔡

Every use of the utf8str_cis type definition in the NFSv4.1 protocol specification follows the profile named nfs4_cis_prep.

14.2.1. Intended Applicability of the nfs4_cis_prep Profile

The utf8str_cis type is a case-insensitive string of UTF-8 characters. Its main use in NFSv4.1 is for naming NFS servers. 🖥️

14.2.2. Character Repertoire of nfs4_cis_prep

The nfs4_cis_prep profile uses Unicode 3.2, as defined in stringprep's Appendix A.1. However, NFSv4.1 implementations are not limited to 3.2. 🌸

14.2.3. Mapping Used by nfs4_cis_prep

The nfs4_cis_prep profile specifies mapping using the following tables from stringprep:

Table B.1
Table B.2

14.2.4. Normalization Used by nfs4_cis_prep

The nfs4_cis_prep profile specifies using Unicode normalization form KC, as described in stringprep. ✨

14.2.5. Prohibited Output for nfs4_cis_prep

The nfs4_cis_prep profile specifies prohibiting the use of the following tables from stringprep:

Table C.1.2
Table C.2.2
Table C.3
Table C.4
Table C.5
Table C.6
Table C.7
Table C.8
Table C.9

14.2.6. Bidirectional Output for nfs4_cis_prep

The nfs4_cis_prep profile specifies checking bidirectional strings as described in stringprep's Section 6. 🔄

14.3. Stringprep Profile for the utf8str_mixed Type 🎀

Every use of the utf8str_mixed type definition in the NFSv4.1 protocol specification follows the profile named nfs4_mixed_prep.

14.3.1. Intended Applicability of the nfs4_mixed_prep Profile

The utf8str_mixed type is a string of UTF-8 characters, with a prefix that is case sensitive, a separator equal to '@', and a suffix that is a fully qualified domain name. Its main use in NFSv4.1 is for naming principals identified in an Access Control Entry. 🦋

14.3.2. Character Repertoire of nfs4_mixed_prep 🎀

okay so the nfs4_mixed_prep profile is built on Unicode 3.2, which is what stringprep's Appendix A.1 points to. 💅 BUT, and this is important bestie, NFSv4.1 implementations are not stuck at 3.2. You can go newer if you want. no cap 🌈

14.3.3. Mapping Used by nfs4_cis_prep 🦋

for the prefix and the separator of a utf8str_mixed string, the nfs4_mixed_prep profile does its mapping with this table from stringprep:

Table B.1

for the suffix of a utf8str_mixed string, it uses these stringprep tables:

Table B.1
Table B.2

14.3.4. Normalization Used by nfs4_mixed_prep 🧚

the nfs4_mixed_prep profile says to use Unicode normalization form KC, as described in stringprep. it's giving tidy, consistent strings ✨

14.3.5. Prohibited Output for nfs4_mixed_prep 🚫

the nfs4_mixed_prep profile says you're not allowed to use any of these stringprep tables in the output:

Table C.1.2
Table C.2.2
Table C.3
Table C.4
Table C.5
Table C.6
Table C.7
Table C.8
Table C.9

14.3.6. Bidirectional Output for nfs4_mixed_prep 🔄

the nfs4_mixed_prep profile says to check bidirectional strings the way stringprep's Section 6 describes. it's the bidi check, bestie, and it matters 💜

14.4. UTF-8 Capabilities 🌸

here's the vibe. the constants and typedef are:

const FSCHARSET_CAP4_CONTAINS_NON_UTF8  = 0x1;
const FSCHARSET_CAP4_ALLOWS_ONLY_UTF8   = 0x2;

typedef uint32_t        fs_charset_cap4;

okay so some operating environments and file systems don't actually enforce character set encodings, which is wild. 😭 so NFSv4.1 gives the client the fs_charset_cap attribute (Section 5.8.2.11) to say what UTF-8 abilities a file system has. it's an integer holding a pair of flags.

the first flag is FSCHARSET_CAP4_CONTAINS_NON_UTF8. if it's set to one, the client knows the file system has non-UTF-8 characters in it. the server won't convert non-UTF characters to UTF-8 when the client reads a symbolic link or directory, and operations that take component names or pathnames in their arguments won't convert those strings to UTF-8 either. main character energy: the server just doesn't touch them 🙈

the second flag is FSCHARSET_CAP4_ALLOWS_ONLY_UTF8. if it's set to one, the server will only accept (and generate) UTF-8 characters on that file system. 🫶

if FSCHARSET_CAP4_ALLOWS_ONLY_UTF8 is set to one, FSCHARSET_CAP4_CONTAINS_NON_UTF8 MUST be set to zero. this one's a hard rule, no wiggle room 💀

and FSCHARSET_CAP4_ALLOWS_ONLY_UTF8 SHOULD always be set to one. it's the recommended default energy, periodt ✨

when the client sends an invalid UTF-8 string, the server should return NFS4ERR_INVAL (see Table 11). this covers cases where it finds inappropriate prefixes, and where the count includes trailing bytes that don't make up a full UCS character. 😩

if the client-supplied string is valid UTF-8 but has characters the server can't support as a value for that string, the server should return NFS4ERR_BADCHAR. the example in the spec is names with characters outside of Unicode plane 0 on file systems that can't handle them, even though Unicode itself defines them. 🌈

when a UTF-8 string gets used as a file name, and the file system does support every character in the name but still won't allow that particular name, the server should return NFS4ERR_BADNAME (Table 11). this includes a file system that puts a normalization rule on name strings. it also covers stuff like file systems that ban "." and ".." as file names for certain operations, plus other constraints like that. iykyk 🙈

15. Error Values ✨

NFS error numbers get assigned to operations that fail inside a Compound (COMPOUND or CB_COMPOUND) request. a Compound request holds a bunch of NFS operations, and their results get encoded in order in a Compound reply. 🎀 successful operations give you an NFS4_OK status followed by their encoded results. but if an NFS operation fails, an error status goes into the reply and the Compound request ends right there. rip to the rest of the batch 😭

15.1. Error Definitions 🦋

okay bestie, here's the whole error roster 🦋 every name, its number, and where the nitty gritty lives 💅

    +===================================+========+===================+
    | Error                             | Number | Description       |
    +===================================+========+===================+
    | NFS4_OK                           | 0      | Section 15.1.3.1  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_ACCESS                    | 13     | Section 15.1.6.1  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_ATTRNOTSUPP               | 10032  | Section 15.1.15.1 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_ADMIN_REVOKED             | 10047  | Section 15.1.5.1  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BACK_CHAN_BUSY            | 10057  | Section 15.1.12.1 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BADCHAR                   | 10040  | Section 15.1.7.1  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BADHANDLE                 | 10001  | Section 15.1.2.1  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BADIOMODE                 | 10049  | Section 15.1.10.1 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BADLAYOUT                 | 10050  | Section 15.1.10.2 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BADNAME                   | 10041  | Section 15.1.7.2  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BADOWNER                  | 10039  | Section 15.1.15.2 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BADSESSION                | 10052  | Section 15.1.11.1 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BADSLOT                   | 10053  | Section 15.1.11.2 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BADTYPE                   | 10007  | Section 15.1.4.1  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BADXDR                    | 10036  | Section 15.1.1.1  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BAD_COOKIE                | 10003  | Section 15.1.1.2  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BAD_HIGH_SLOT             | 10077  | Section 15.1.11.3 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BAD_RANGE                 | 10042  | Section 15.1.8.1  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BAD_SEQID                 | 10026  | Section 15.1.16.1 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BAD_SESSION_DIGEST        | 10051  | Section 15.1.12.2 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_BAD_STATEID               | 10025  | Section 15.1.5.2  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_CB_PATH_DOWN              | 10048  | Section 15.1.11.4 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_CLID_INUSE                | 10017  | Section 15.1.13.2 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_CLIENTID_BUSY             | 10074  | Section 15.1.13.1 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_COMPLETE_ALREADY          | 10054  | Section 15.1.9.1  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_CONN_NOT_BOUND_TO_SESSION | 10055  | Section 15.1.11.6 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_DEADLOCK                  | 10045  | Section 15.1.8.2  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_DEADSESSION               | 10078  | Section 15.1.11.5 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_DELAY                     | 10008  | Section 15.1.1.3  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_DELEG_ALREADY_WANTED      | 10056  | Section 15.1.14.1 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_DELEG_REVOKED             | 10087  | Section 15.1.5.3  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_DENIED                    | 10010  | Section 15.1.8.3  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_DIRDELEG_UNAVAIL          | 10084  | Section 15.1.14.2 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_DQUOT                     | 69     | Section 15.1.4.2  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_ENCR_ALG_UNSUPP           | 10079  | Section 15.1.13.3 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_EXIST                     | 17     | Section 15.1.4.3  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_EXPIRED                   | 10011  | Section 15.1.5.4  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_FBIG                      | 27     | Section 15.1.4.4  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_FHEXPIRED                 | 10014  | Section 15.1.2.2  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_FILE_OPEN                 | 10046  | Section 15.1.4.5  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_GRACE                     | 10013  | Section 15.1.9.2  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_HASH_ALG_UNSUPP           | 10072  | Section 15.1.13.4 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_INVAL                     | 22     | Section 15.1.1.4  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_IO                        | 5      | Section 15.1.4.6  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_ISDIR                     | 21     | Section 15.1.2.3  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_LAYOUTTRYLATER            | 10058  | Section 15.1.10.3 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_LAYOUTUNAVAILABLE         | 10059  | Section 15.1.10.4 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_LEASE_MOVED               | 10031  | Section 15.1.16.2 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_LOCKED                    | 10012  | Section 15.1.8.4  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_LOCKS_HELD                | 10037  | Section 15.1.8.5  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_LOCK_NOTSUPP              | 10043  | Section 15.1.8.6  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_LOCK_RANGE                | 10028  | Section 15.1.8.7  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_MINOR_VERS_MISMATCH       | 10021  | Section 15.1.3.2  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_MLINK                     | 31     | Section 15.1.4.7  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_MOVED                     | 10019  | Section 15.1.2.4  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_NAMETOOLONG               | 63     | Section 15.1.7.3  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_NOENT                     | 2      | Section 15.1.4.8  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_NOFILEHANDLE              | 10020  | Section 15.1.2.5  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_NOMATCHING_LAYOUT         | 10060  | Section 15.1.10.5 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_NOSPC                     | 28     | Section 15.1.4.9  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_NOTDIR                    | 20     | Section 15.1.2.6  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_NOTEMPTY                  | 66     | Section 15.1.4.10 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_NOTSUPP                   | 10004  | Section 15.1.1.5  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_NOT_ONLY_OP               | 10081  | Section 15.1.3.3  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_NOT_SAME                  | 10027  | Section 15.1.15.3 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_NO_GRACE                  | 10033  | Section 15.1.9.3  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_NXIO                      | 6      | Section 15.1.16.3 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_OLD_STATEID               | 10024  | Section 15.1.5.5  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_OPENMODE                  | 10038  | Section 15.1.8.8  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_OP_ILLEGAL                | 10044  | Section 15.1.3.4  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_OP_NOT_IN_SESSION         | 10071  | Section 15.1.3.5  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_PERM                      | 1      | Section 15.1.6.2  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_PNFS_IO_HOLE              | 10075  | Section 15.1.10.6 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_PNFS_NO_LAYOUT            | 10080  | Section 15.1.10.7 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_RECALLCONFLICT            | 10061  | Section 15.1.14.3 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_RECLAIM_BAD               | 10034  | Section 15.1.9.4  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_RECLAIM_CONFLICT          | 10035  | Section 15.1.9.5  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_REJECT_DELEG              | 10085  | Section 15.1.14.4 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_REP_TOO_BIG               | 10066  | Section 15.1.3.6  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_REP_TOO_BIG_TO_CACHE      | 10067  | Section 15.1.3.7  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_REQ_TOO_BIG               | 10065  | Section 15.1.3.8  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_RESTOREFH                 | 10030  | Section 15.1.16.4 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_RETRY_UNCACHED_REP        | 10068  | Section 15.1.3.9  |
    +-----------------------------------+--------+-------------------+

    | NFS4ERR_RETURNCONFLICT            | 10086  | Section 15.1.10.8 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_ROFS                      | 30     | Section 15.1.4.11 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_SAME                      | 10009  | Section 15.1.15.4 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_SHARE_DENIED              | 10015  | Section 15.1.8.9  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_SEQUENCE_POS              | 10064  | Section 15.1.3.10 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_SEQ_FALSE_RETRY           | 10076  | Section 15.1.11.7 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_SEQ_MISORDERED            | 10063  | Section 15.1.11.8 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_SERVERFAULT               | 10006  | Section 15.1.1.6  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_STALE                     | 70     | Section 15.1.2.7  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_STALE_CLIENTID            | 10022  | Section 15.1.13.5 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_STALE_STATEID             | 10023  | Section 15.1.16.5 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_SYMLINK                   | 10029  | Section 15.1.2.8  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_TOOSMALL                  | 10005  | Section 15.1.1.7  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_TOO_MANY_OPS              | 10070  | Section 15.1.3.11 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_UNKNOWN_LAYOUTTYPE        | 10062  | Section 15.1.10.9 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_UNSAFE_COMPOUND           | 10069  | Section 15.1.3.12 |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_WRONGSEC                  | 10016  | Section 15.1.6.3  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_WRONG_CRED                | 10082  | Section 15.1.6.4  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_WRONG_TYPE                | 10083  | Section 15.1.2.9  |
    +-----------------------------------+--------+-------------------+
    | NFS4ERR_XDEV                      | 18     | Section 15.1.4.12 |
    +-----------------------------------+--------+-------------------+

                   Table 11: Protocol Error Definitions

15.1.1. General Errors ✨

This bit covers errors that apply to, like, a whole bunch of different situations 💖

15.1.1.1. NFS4ERR_BADXDR (Error Code 10036) 🙈

The arguments for this operation don't match what the XDR definition says they should be. That includes the case where the request ends before all the arguments have been seen. Heads up: this error also applies when a fixed enumeration (booleans count too!) has a value in the input stream that isn't valid for that enum. A replier may pre-parse all operations for a Compound procedure before doing any operation execution, and return RPC-level XDR errors in that case. Very organized of it, ngl 🗂️

15.1.1.2. NFS4ERR_BAD_COOKIE (Error Code 10003) 🍪

This one's used for operations that give you a set of info indexed by some quantity. That quantity can be something the client provided, or a cookie the server sent back on an earlier call. If the value can't be used for its intended purpose, this error shows up. Cookies gonna cookie 💀

15.1.1.3. NFS4ERR_DELAY (Error Code 10008) 🫶

Okay, so this one is a whole saga. For any number of reasons, the replier couldn't get through this operation in what it considered a reasonable amount of time. The client should wait, then try the request again with a new slot and sequence value. Here are some situations that might lead to this:

A server that supports hierarchical storage gets a request to process a file that had been migrated 📦
An operation needs a delegation recall to proceed, but waiting for that delegation to be recalled and returned makes it impossible to process this request in a timely way ⏳
A request is being performed on a session that's being migrated from another server, as described in Section 11.14.3, and the lack of full information about the state of the session on the source makes it impossible to process the request right away 🚧

Here's the slay part: returning NFS4ERR_DELAY lets the necessary preparatory operations keep going without holding up requester resources, like a session slot. After delaying for a period of time, the client can re-send the operation, often as part of a nearly identical request. Because we need to avoid spurious reissues of non-idempotent operations, and avoid acting on NFS4ERR_DELAY errors that came back on responses from the replier's reply cache, the session-provided reply cache has to be integrated. There are a few cases to juggle, and each one needs different handling from the requester and the replier 🧩

If NFS4ERR_DELAY is returned on a SEQUENCE operation, the request is retried in full, with the SEQUENCE operation containing the same slot and sequence values. In this case, the replier MUST avoid returning a response containing NFS4ERR_DELAY as the response to SEQUENCE solely because an earlier instance of the same request returned that error and it was stored in the reply cache. If the replier did that, the retries wouldn't be effective, because the replier would have no chance to see whether the condition that caused NFS4ERR_DELAY got fixed in the time between the original request and the retry. Not it 🙅‍♀️
If NFS4ERR_DELAY is returned on an operation other than SEQUENCE that validly appears as the first operation of a request, the handling is similar. The request can be retried in full without modification. Also in this case, the replier MUST avoid returning a response containing NFS4ERR_DELAY as the response to an initial operation of a request solely on the basis of its presence in the reply cache. Same energy as before: otherwise the retries wouldn't work, since the replier wouldn't get to check whether the original condition was rectified between the original request and the retry 🔄
If NFS4ERR_DELAY is returned on an operation other than the first in the request, the request, when retried, MUST contain a SEQUENCE operation that's different from the original one, with either the slot ID or the sequence value different from that in the original request. Because requesters do this, the replier doesn't need to go out of its way to avoid returning an NFS4ERR_DELAY error that came from the reply cache. When no non-idempotent operations have been processed before the NFS4ERR_DELAY was returned, the requester should retry the request in full, with the only difference from the original being the change to the slot ID or sequence value in the reissued SEQUENCE operation 🎀
When NFS4ERR_DELAY is returned on an operation other than the first within a request, and a non-idempotent operation has already been processed before the NFS4ERR_DELAY was returned, reissuing the request as is normally done would incorrectly cause the re-execution of that non-idempotent operation. 😱

To avoid this situation, the client should reissue the request without the non-idempotent operation. The request still must use a SEQUENCE operation with either a different slot ID or sequence value from the SEQUENCE in the original request. Because that's done, the replier has no way to avoid spuriously re-executing the non-idempotent operation, since the different SEQUENCE parameters stop the requester from recognizing that the non-idempotent operation is being retried. Big brain stuff 🧠

Now, quick but important: without the ability to return NFS4ERR_DELAY, and without the requester being willing to re-send when it gets one, deadlock might result. For example, if a recall is done, and the delegation return (or the operations preparing for it) are held up by other operations that need the delegation to be returned, session slots might not be available. The result could be deadlock. Nobody wants that, periodt 🚫

15.1.1.4. NFS4ERR_INVAL (Error Code 22) 🌸

The arguments for this operation aren't valid for some reason, even though they do match what the XDR definition says for the request. Valid shape, wrong vibes 💅

15.1.1.5. NFS4ERR_NOTSUPP (Error Code 10004) 🙅‍♀️

Operation not supported. That happens either because the operation is an OPTIONAL one and this server doesn't support it, or because the operation MUST NOT be implemented in the current minor version. Not everything is for everyone, bestie 🦋

15.1.1.6. NFS4ERR_SERVERFAULT (Error Code 10006) 💀

An error occurred on the server that doesn't map to any of the specific legal NFSv4.1 protocol error values. The client should translate this into an appropriate error. UNIX clients may choose to translate this to EIO. The server really did a little oopsie 😭

15.1.1.7. NFS4ERR_TOOSMALL (Error Code 10005) 📏

This is used where an operation returns a variable amount of data, with a limit the client specified. If the data returned can't fit within the limit the client set, this error results. Too big for the box, sorry 📦

15.1.2. Filehandle Errors 🦋

These errors are all about the situation where the current or saved filehandle, or the filehandle passed to PUTFH that's meant to become the current filehandle, is invalid in some way. That includes situations where the filehandle is valid in general but isn't the right object type for the current operation. Filehandle drama, iykyk 💜

Whenever the error description points to a problem with the current or saved filehandle, it's understood that filehandles only get checked for the condition if they're implicit arguments of the operation in question. Gotta know what's actually being checked, bestie 👑

15.1.2.1. NFS4ERR_BADHANDLE (Error Code 10001) 🚫

This is an illegal NFS filehandle for the current server. The current filehandle failed internal consistency checks. Once it's been accepted as valid (by PUTFH), no later status change can make the filehandle generate this error. Once it's in, it's in 🔒

15.1.2.2. NFS4ERR_FHEXPIRED (Error Code 10014) ⏰

A current or saved filehandle that's an argument to the current operation is volatile and has expired at the server. Expired, not me 💀

15.1.2.3. NFS4ERR_ISDIR (Error Code 21) 📁

The current or saved filehandle designates a directory, but the current operation doesn't allow a directory to be accepted as the target of this operation. Directories are not the vibe here 🙈

15.1.2.4. NFS4ERR_MOVED (Error Code 10019) 🦋

okay bestie, the file system that holds your current filehandle is either missing from the server or can't be reached from the network address you used 😭 maybe it's chilling on a different set of addresses now, or it got relocated or migrated to another server, or it was never there at all 💀. to find its new home, grab the fs_locations or fs_locations_info attribute for the current filehandle 🗺️. for the full tea, see Section 11.3 ✨.

like with NFS4ERR_DELAY, some non-idempotent operations might have already run successfully inside the COMPOUND before NFS4ERR_MOVED shows up 🙈. so once you know the new location, don't re-run the whole original request in full. instead, send a fresh COMPOUND with any successfully executed non-idempotent operations taken out 💅. if you reuse the same session for that new COMPOUND, its SEQUENCE operation should use a different slot ID or sequence 🔥.

15.1.2.5. NFS4ERR_NOFILEHANDLE (Error Code 10020) 🌸

the logical current or saved filehandle value is required by the current operation and it's not set 😩. this can happen when the COMPOUND is malformed, meaning there's no PUTFH or PUTROOTFH before an operation that needs the current filehandle set. oops 🙈.

15.1.2.6. NFS4ERR_NOTDIR (Error Code 20) 📁

the current (or saved) filehandle points at something that isn't a directory, but the operation needs a directory 🙅‍♀️.

15.1.2.7. NFS4ERR_STALE (Error Code 70) 👻

the current or saved filehandle, the one used as an argument to the current operation, is invalid 💀. the file it refers to is gone, or your access to it has been revoked. it's giving ghost energy 👻.

15.1.2.8. NFS4ERR_SYMLINK (Error Code 10029) 🔗

the current filehandle points at a symbolic link, but the current operation doesn't allow a symlink as its target 🙅‍♀️.

15.1.2.9. NFS4ERR_WRONG_TYPE (Error Code 10083) 🎭

the current (or saved) filehandle points at an object with the wrong type for the current operation, and there isn't a more specific error (like NFS4ERR_ISDIR or NFS4ERR_SYMLINK) that fits 🎯. side note: back in NFSv4.0, these situations usually came back as the vaguer NFS4ERR_INVAL 🧐.

15.1.3. Compound Structure Errors 🎀

this section is about errors in the overall structure of a Compound request (that means both COMPOUND and CB_COMPOUND), not about any one particular operation 🧩.

there are some basic rules for which operations can show up in a Compound request 📋. sessions add to those rules by requiring a Sequence operation (either SEQUENCE or CB_SEQUENCE) at the very start of the Compound ✨.

15.1.3.1. NFS_OK (Error code 0) 💖

means the operation finished successfully, because every one of the constituent operations completed without an error 🥳.

15.1.3.2. NFS4ERR_MINOR_VERS_MISMATCH (Error code 10021) 🙅‍♀️

the minor version you asked for isn't one the current listener supports. this shows up in the overall status for the Compound, not on a specific operation, since the results come back with a result count of zero 📉.

15.1.3.3. NFS4ERR_NOT_ONLY_OP (Error Code 10081) ☝️

some operations are allowed outside of a session, but when the Compound doesn't start with a Sequence operation, they MUST be the only operation in that Compound 🫵. if that rule isn't met, you get this error 💅.

15.1.3.4. NFS4ERR_OP_ILLEGAL (Error Code 10044) 🚫

the operation code isn't valid for the current Compound procedure 💀. the opcode in the result stream that goes with this error is the ILLEGAL value, even though the value in the request stream might be something else 🤓. and if the replier pre-parses every operation in a Compound before running any of them, you might get an RPC-level XDR error instead 🧾.

15.1.3.5. NFS4ERR_OP_NOT_IN_SESSION (Error Code 10071) 🔒

most forward operations and all callback operations only work inside the context of a session, so the Compound request in question MUST begin with a Sequence operation 🫶. if you try to run these outside of a session, this error shows up 😭.

15.1.3.6. NFS4ERR_REP_TOO_BIG (Error Code 10066) 📦

the reply to a Compound would be bigger than the channel's negotiated maximum response size 🐘.

15.1.3.7. NFS4ERR_REP_TOO_BIG_TO_CACHE (Error Code 10067) 🗄️

the reply to a Compound would be bigger than the channel's negotiated maximum size for replies cached in the reply cache, when the Sequence for the current request says this request should be cached 🙃.

15.1.3.8. NFS4ERR_REQ_TOO_BIG (Error Code 10065) 🧳

the Compound request is bigger than the channel's negotiated maximum size for requests 💥.

15.1.3.9. NFS4ERR_RETRY_UNCACHED_REP (Error Code 10068) 🔁

the requester tried to retry a Compound that it had earlier asked not to be placed in the reply cache 🙈.

15.1.3.10. NFS4ERR_SEQUENCE_POS (Error Code 10064) 📍

a Sequence operation showed up somewhere other than the very first operation of a Compound request 😵‍💫.

15.1.3.11. NFS4ERR_TOO_MANY_OPS (Error Code 10070) 🧮

the Compound request has too many operations, going over the count that was negotiated when the session was created 🙃.

15.1.3.12. NFS4ERR_UNSAFE_COMPOUND (Error Code 10068) ⚠️

the client sent a COMPOUND request with an unsafe mix of operations. specifically, a non-idempotent operation changes the current filehandle and it isn't followed by a GETFH 🫠.

15.1.4. File System Errors 💾

these errors describe things that went wrong in the underlying file system implementation, rather than in the protocol or any NFSv4.x feature 🛠️.

15.1.4.1. NFS4ERR_BADTYPE (Error Code 10007) 🧱

an attempt was made to create an object with an inappropriate type specified to CREATE 🙅‍♀️. that could be because the type is undefined, because the server doesn't support it, or because it isn't meant to be created by CREATE (like a regular file or a named attribute, which use OPEN to get created) 📂.

15.1.4.2. NFS4ERR_DQUOT (Error Code 69) 💸

resource (quota) hard limit exceeded 😤. the user has gone past their resource limit on the server 🛑.

15.1.4.3. NFS4ERR_EXIST (Error Code 17) 👯

a file with the target name you specified already exists, whether you were creating, renaming, or linking 🙅‍♀️.

15.1.4.4. NFS4ERR_FBIG (Error Code 27) 🐋

the file is too big 🫠. the operation would have made the file grow past the server's limit.

15.1.4.5. NFS4ERR_FILE_OPEN (Error Code 10046) 🚪

the operation isn't allowed because a file involved in it is currently open 🔒. servers may, but aren't required to, block linking-to, removing, or renaming open files 🤷‍♀️.

15.1.4.6. NFS4ERR_IO (Error Code 5) 💥

an I/O error happened, and the file system couldn't recover from it 😭.

15.1.4.7. NFS4ERR_MLINK (Error Code 31) 🔗

the request would have pushed the server's limit on hard links for a file past its max 🙈.

15.1.4.8. NFS4ERR_NOENT (Error Code 2) 🔍

no such file or directory. the file or directory name you gave doesn't exist 👀.

15.1.4.9. NFS4ERR_NOSPC (Error Code 28) 🕳️

there's no space left on the device 💀. the operation would have pushed the server's file system past its limit.

15.1.4.10. NFS4ERR_NOTEMPTY (Error Code 66) 🗑️

you tried to remove a directory that still has stuff in it 🙅‍♀️.

15.1.4.11. NFS4ERR_ROFS (Error Code 30) 📖

read-only file system 📚. you tried a modifying operation on something that's read-only.

15.1.4.12. NFS4ERR_XDEV (Error Code 18) 🌉

an operation, like linking, crossed a boundary it shouldn't have 🚧. this can happen because of boundaries like:

the one between file systems (where the fsids are different).
the one between different named attribute directories, or between a named attribute directory and an ordinary directory.
the one between byte-ranges of a file system that the file system implementation treats as separate (for example, for space accounting purposes), where crossing between those byte-ranges isn't allowed.

15.1.5. State Management Errors 🔐

these errors are about problems with the stateid (or one of the stateids) passed to an operation 🪪. that includes a stateid that's invalid, and also a stateid that's valid but points at locking state that's been revoked 🙃. depending on the operation, a valid stateid might designate opens, byte-range locks, file or directory delegations, layouts, or device maps 🗂️.

15.1.5.1. NFS4ERR_ADMIN_REVOKED (Error Code 10047) 👮‍♀️

a stateid points at locking state of any type that got revoked because of administrative action, which could happen even while the lease is still valid 😤.

15.1.5.2. NFS4ERR_BAD_STATEID (Error Code 10026) ❌

a stateid doesn't properly designate any valid state 🙅‍♀️. for how stateids get validated, see Sections 8.2.4 and 8.2.3 🔎.

15.1.5.3. NFS4ERR_DELEG_REVOKED (Error Code 10087) 🫥

a stateid points at recallable locking state of any type (a delegation or a layout) that got revoked because the client didn't return the lock when it was recalled 📞.

15.1.5.4. NFS4ERR_EXPIRED (Error Code 10011) ⏰

a stateid points at locking state of any type that got revoked because the client's lease expired. that could happen right when the lease expires, or after a later request for a conflicting lock 🫠.

15.1.5.5. NFS4ERR_OLD_STATEID (Error Code 10024) 🕰️

a stateid with a non-zero seqid value doesn't match the current seqid for the state the user designated 🙈.

15.1.6. Security Errors 🛡️

these are the various permission-related errors in NFSv4.1 🫶.

15.1.6.1. NFS4ERR_ACCESS (Error Code 13) 🚷

permission denied 😤. the caller doesn't have the right permission for the requested operation. compare this with NFS4ERR_PERM (Section 15.1.6.2), which only covers owner or privileged-user permission failures, and NFS4ERR_WRONG_CRED (Section 15.1.6.4), which is about having the right permission to delete or change transient objects based on the credentials of whoever created them 🧐.

15.1.6.2. NFS4ERR_PERM (Error Code 1) 👑

the requester isn't the owner 🙅‍♀️. the operation wasn't allowed because the caller is neither a privileged user (root) nor the owner of the target 👑.

15.1.6.3. NFS4ERR_WRONGSEC (Error Code 10016) 🔄

the security mechanism the client is using for the operation doesn't match the server's security policy 🚨. the client should switch the security mechanism and re-send the operation, but not with the same slot ID and sequence ID. one or both of them MUST be different on the re-send 🔀. SECINFO and SECINFO_NO_NAME can be used to figure out the right mechanism 🔍.

15.1.6.4. NFS4ERR_WRONG_CRED (Error Code 10082) 🎟️

an operation that changes state was tried by a principal that wasn't allowed to modify that piece of state 🙅‍♀️.

15.1.7. Name Errors 🏷️

names in NFSv4 are UTF-8 strings 🔤. if a string isn't valid UTF-8 or has a length of zero, you get NFS4ERR_INVAL. beyond that, there are a bunch of other errors for specific name problems 🧩.

15.1.7.1. NFS4ERR_BADCHAR (Error Code 10040) 🙊

a UTF-8 string contains a character that the server doesn't support in the context where it's being used 😭.

15.1.7.2. NFS4ERR_BADNAME (Error Code 10041) 🙈

okay so a name string in a request was made of totally valid UTF-8 characters the server knows about, but the server just doesn't accept that name for the operation you're doing 💅 like, the characters are fine, the name is not. the example they give is making a file or directory called ".." on a server whose file system already uses that name for linking to the parent directory. not it, bestie 💀

15.1.7.3. NFS4ERR_NAMETOOLONG (Error Code 63) 📏

this one pops up when the filename in an operation goes over the server's implementation limit. too long, sorry 🙃

15.1.8. Locking Errors 🔒

this section is all about errors with locking, both share reservations and byte-range locking 🦋 it does NOT cover errors about reclaiming locks though. those live in Section 15.1.9 so go check there 💜

15.1.8.1. NFS4ERR_BAD_RANGE (Error Code 10042) 🍓

the byte-range in a LOCK, LOCKT, or LOCKU operation isn't allowed by the server. like, a server that only does 32-bit ranges gets a range it can't handle, and boom, this error 💥 (see Section 18.10.3 for the deets)

15.1.8.2. NFS4ERR_DEADLOCK (Error Code 10045) 💀

the server figured out there's a byte-range locking deadlock going on for a READW_LT or WRITEW_LT LOCK operation. that's a whole mess, ngl 😭

15.1.8.3. NFS4ERR_DENIED (Error Code 10010) 🌸

so an attempt to lock a file got denied. this might be temporary, so the client is encouraged to re-send the lock request until it gets accepted ✨ BUT the re-send MUST NOT use the same slot ID and sequence ID. one or both MUST be different, no exceptions 🙅‍♀️ check Section 9.6 for the whole re-send tea.

15.1.8.4. NFS4ERR_LOCKED (Error Code 10012) 🔐

a READ or WRITE got attempted on a file where the I/O and an existing lock were in a fight 🥊 that happens when:

there's a share reservation that doesn't match the I/O being done
the range you want to read or write crosses an existing mandatory byte-range lock

15.1.8.5. NFS4ERR_LOCKS_HELD (Error Code 10037) 🫠

an operation got blocked because locks were showing up when they weren't invited 😤

15.1.8.6. NFS4ERR_LOCK_NOTSUPP (Error Code 10043) 🙃

a LOCK operation tried to upgrade or downgrade a byte-range lock the owner already holds, but the server doesn't support atomic upgrade or downgrade of locks. so no glow-up for you, not on this server 💅

15.1.8.7. NFS4ERR_LOCK_RANGE (Error Code 10028) 🧩

this one is a bit of a puzzle 🧩 a LOCK operation is working on a range that partly overlaps a byte-range lock the same lock-owner already has, and it doesn't line up exactly with a single one of those locks. the server doesn't support this type of request, so it doesn't do POSIX locking semantics [21]. see Sections 18.10.4, 18.11.4, and 18.12.4 for how this plays out for LOCK, LOCKT, and LOCKU 🦋

15.1.8.8. NFS4ERR_OPENMODE (Error Code 10038) 🚫

the client tried a READ, WRITE, LOCK, or some other operation that the stateid doesn't allow. for example, writing to a file that was opened read-only. nope nope 🙅‍♀️

an attempt to OPEN a file with a share reservation failed because of a share conflict. sorry bestie, someone got there first 😔

15.1.9. Reclaim Errors 🔄

these errors are about the process of reclaiming locks after a server restart 🔁

15.1.9.1. NFS4ERR_COMPLETE_ALREADY (Error Code 10054) ✅

the client already sent a successful RECLAIM_COMPLETE for the same scope. the scope can be global, or it can be the same file system for a per-fs RECLAIM_COMPLETE. sending another one isn't needed, and it gets this error 🙄

15.1.9.2. NFS4ERR_GRACE (Error Code 10013) ⏳

this error comes back when the server is still in its grace period for the file system object the lock was requested on 🕰️ in that situation, a non-reclaim locking request can't be granted. this can happen because either:

the server doesn't have enough info about locks that might get reclaimed to decide whether this lock can be granted
the request is from a client that's responsible for reclaiming its locks, and it hasn't done the right RECLAIM_COMPLETE yet, so it isn't allowed to get new locks

for a per-fs grace period, there might be clients (aka the ones currently using the destination file system) who don't know why the grace period started. those clients need to keep retrying until the grace period is over, same as everyone else 🫶

15.1.9.3. NFS4ERR_NO_GRACE (Error Code 10033) 🚷

a reclaim of client state got attempted when the server can't guarantee that conflicting state hasn't been handed to another client. this happens in any of these situations:

there's no active grace period for the file system object the request was made on
the client making the request has no current role in reclaiming locks
previous operations created a situation where the server can't tell that a reclaim-interfering edge condition doesn't exist

15.1.9.4. NFS4ERR_RECLAIM_BAD (Error Code 10034) 🗑️

the server decided a reclaim from the client isn't valid. the lock being reclaimed couldn't possibly have existed before the server restart or file system migration event 😬 the server isn't obligated to make that call, and usually it just trusts the client to only reclaim locks it got before the restart. but if the server does have reliable info to tell, then this error means the reclaim got rejected as invalid. that's different from NFS4ERR_RECLAIM_CONFLICT (see Section 15.1.9.5), where the server only knows some reclaim is invalid but can't pin down which request it is 🕵️‍♀️

15.1.9.5. NFS4ERR_RECLAIM_CONFLICT (Error Code 10035) ⚔️

the reclaim from the client ran into a conflict and can't be satisfied. this might mean a misbehaving client, but not necessarily the one getting the error 👀 the misbehavior could be from the client that set up the lock this client bumped into. see also Section 15.1.9.4 for the related error, NFS4ERR_RECLAIM_BAD 💜

15.1.10. pNFS Errors 🧚

this section covers pNFS-related errors, including ones tied to using NFSv4.1 to talk with a data server ✨

15.1.10.1. NFS4ERR_BADIOMODE (Error Code 10049) 🎀

an invalid or inappropriate layout iomode got specified. for example, say a client's LAYOUTGET operation asked for an iomode of LAYOUTIOMODE4_RW, but the server can't or won't let the client send write requests to data servers. the server can reply with NFS4ERR_BADIOMODE. then the client just sends another LAYOUTGET with an iomode of LAYOUTIOMODE4_READ 🥰

15.1.10.2. NFS4ERR_BADLAYOUT (Error Code 10050) 🫥

the layout is invalid in some way. for LAYOUTCOMMIT, that means the layout isn't held by the client or isn't mode LAYOUTIOMODE4_RW. for LAYOUTGET, it means a layout matching the client's minimum length can't be granted 📉

15.1.10.3. NFS4ERR_LAYOUTTRYLATER (Error Code 10058) ⏰

layouts are temporarily unavailable for the file. the client should re-send later, but NOT with the same slot ID and sequence ID. one or both MUST be different on the re-send 🙏

15.1.10.4. NFS4ERR_LAYOUTUNAVAILABLE (Error Code 10059) 🚫

comes back when layouts aren't available for the current file system or that particular file 📁

15.1.10.5. NFS4ERR_NOMATCHING_LAYOUT (Error Code 10060) 🤷‍♀️

comes back when layouts get recalled and the client has no layouts matching the spec of the ones being recalled 🌀

15.1.10.6. NFS4ERR_PNFS_IO_HOLE (Error Code 10075) 🕳️

the pNFS client tried to read from or write to an illegal hole in a file on a data server that's using sparse packing. see Section 13.4.4 🕳️

15.1.10.7. NFS4ERR_PNFS_NO_LAYOUT (Error Code 10080) 🙅‍♀️

the pNFS client tried to read from or write to a file (through a request to a data server) without holding a valid layout. that includes the case where the client had a layout, but the iomode doesn't allow a WRITE 📝

15.1.10.8. NFS4ERR_RETURNCONFLICT (Error Code 10086) 🔀

a layout is unavailable because the client tried to do LAYOUTGET before a pending LAYOUTRETURN on the file got received. see Section 12.5.5.2.1.3 🔁

15.1.10.9. NFS4ERR_UNKNOWN_LAYOUTTYPE (Error Code 10062) ❓

the client specified a layout type the server doesn't support 🫠

15.1.11. Session Use Errors 🎟️

this section is about errors you run into while using sessions. basically, errors that show up when a request uses a Sequence operation (that's either SEQUENCE or CB_SEQUENCE) 💅

15.1.11.1. NFS4ERR_BADSESSION (Error Code 10052) 👻

the session ID you gave isn't known to the server the operation was sent to 🫥

15.1.11.2. NFS4ERR_BADSLOT (Error Code 10053) 🎰

the requester sent a Sequence operation that tried to use a slot the replier doesn't have in its slot table. the slot might have been retired, so that's possible too 🕰️

15.1.11.3. NFS4ERR_BAD_HIGH_SLOT (Error Code 10077) 📈

the highest_slot argument in a Sequence operation goes above the replier's enforced highest_slotid. too high, bestie ⬆️

15.1.11.4. NFS4ERR_CB_PATH_DOWN (Error Code 10048) 📡

there's a problem contacting the client over the callback path. this one's mostly been replaced by status flags in the SEQUENCE reply (see Section 18.46), so it's kinda retired now 🪦

15.1.11.5. NFS4ERR_DEADSESSION (Error Code 10078) 💀

the session is a persistent session that's dead. it won't accept new requests or do new operations on existing requests. this covers the case where a request was partly run before a server restart 😵

15.1.11.6. NFS4ERR_CONN_NOT_BOUND_TO_SESSION (Error Code 10055) 🔌

a Sequence operation got sent on a connection that isn't associated with the specified session. the client asked for connection association to be enforced with SP4_MACH_CRED or SP4_SSV state protection, so this is a no-go 🚫

15.1.11.7. NFS4ERR_SEQ_FALSE_RETRY (Error Code 10076) 🎭

the requester sent a Sequence operation with a slot ID and sequence ID that are in the reply cache. but the replier noticed the retried request isn't the same as the original one 👀 see Section 2.10.6.1.3.1 for the details.

15.1.11.8. NFS4ERR_SEQ_MISORDERED (Error Code 10063) 🔢

the requester sent a Sequence operation with an invalid sequence ID. out of order, sis 😤

15.1.12. Session Management Errors 🗂️

this section covers errors tied to requests used in session management 🎀

15.1.12.1. NFS4ERR_BACK_CHAN_BUSY (Error Code 10057) 📞

someone tried to destroy a session, but it can't be destroyed because the server still has callback requests outstanding 📬

15.1.12.2. NFS4ERR_BAD_SESSION_DIGEST (Error Code 10051) 🔏

the digest used in a SET_SSV request isn't valid. the signature is wrong, babe 🫤

15.1.13. Client Management Errors 🪪

this section covers errors tied to requests used to create and manage client IDs 💖

15.1.13.1. NFS4ERR_CLIENTID_BUSY (Error Code 10074) 🚧

the DESTROY_CLIENTID operation found sessions and/or unexpired state still hanging around for the client ID it was trying to destroy. not yet, bestie, clean up your mess first 🧹

15.1.13.2. NFS4ERR_CLID_INUSE (Error Code 10017) 💅

okay so this one is a collision moment 😭. While the server is processing an EXCHANGE_ID op, it gets a co_ownerid that matches a client already holding valid leased state. BUT the principal sending this EXCHANGE_ID is not the same principal that set up the existing client. So two clients ended up with the same co_ownerid, most likely by pure chance 🎲. The fix is for the client to change its co_ownerid and send EXCHANGE_ID again. Just don't reuse the same slot ID AND sequence ID. One or both MUST be different on the re-send 🫶.

15.1.13.3. NFS4ERR_ENCR_ALG_UNSUPP (Error Code 10079) 🔐

this one is giving "no matching encryption, sorry bestie" 💀. An EXCHANGE_ID asked for state protection via SSV, but the encryption algorithms the client offered didn't include a single one the server supports.

15.1.13.4. NFS4ERR_HASH_ALG_UNSUPP (Error Code 10072) 🧂

same energy, different algorithm 🙈. An EXCHANGE_ID asked for state protection via SSV, but the hashing algorithms the client listed didn't include any the server supports.

15.1.13.5. NFS4ERR_STALE_CLIENTID (Error Code 10022) 👻

this is the ghost client error 👻. A client ID the server doesn't recognize got passed to an operation. Here's the tea: unlike NFSv4.0, client IDs aren't sent explicitly in ordinary locking ops, so this error can't pop up there ✨. Instead, when the server restarts, the first sign is an error on the associated session. The client ID is only found to be stale when the client tries to attach it to a new session 🧚.

15.1.14. Delegation Errors 🦋

this section is all about the errors that show up when you ask for a delegation or hand one back 🎀.

15.1.14.1. NFS4ERR_DELEG_ALREADY_WANTED (Error Code 10056) 🙋‍♀️

the client asked for a delegation it had already said it wanted. Main character energy, but she already put her hand up 🙈.

15.1.14.2. NFS4ERR_DIRDELEG_UNAVAIL (Error Code 10084) 📂

the server returns this when it can't or won't give out the directory delegation that was requested 💔.

15.1.14.3. NFS4ERR_RECALLCONFLICT (Error Code 10061) ⚠️

so a recallable object (a layout or a delegation) is off limits right now because another recall is already happening on that same object 🚫. Traffic jam, but make it state 🚦.

15.1.14.4. NFS4ERR_REJECT_DELEG (Error Code 10085) ❌

the callback that was called to handle a new delegation said "nah, rejected" 💅.

15.1.15. Attribute Handling Errors 🏷️

this section covers the errors that come from handling attributes in NFSv4 🧁.

15.1.15.1. NFS4ERR_ATTRNOTSUPP (Error Code 10032) 🚫

the attribute that got asked for isn't supported by the server. This error MUST NOT be returned by the GETATTR op, so GETATTR is never allowed to throw this one 🙅‍♀️.

15.1.15.2. NFS4ERR_BADOWNER (Error Code 10039) 🪪

this one pops up when an owner or owner_group attribute value, or the who field of an ACE inside an ACL attribute value, can't be translated into a local representation. Lost in translation, literally 🌍.

15.1.15.3. NFS4ERR_NOT_SAME (Error Code 10027) 🙅

the VERIFY op returns this to say the attributes it compared weren't the same as what the client sent. Not the vibe 😤.

15.1.15.4. NFS4ERR_SAME (Error Code 10009) 👯

the NVERIFY op returns this to say the attributes it compared were the same as what the client sent. Twins 👯‍♀️, which is the opposite of the last one.

15.1.16. Obsoleted Errors 🗑️

these errors MUST NOT be generated by any NFSv4.1 op. There are a few reasons this happens 🧐:

The function the error covered has been replaced by one of the status bits that SEQUENCE returns.
The new session structure and the change in locking made the error unnecessary.
Some errors got restructured for NFSv4.1, and that got rid of certain ones.

15.1.16.1. NFS4ERR_BAD_SEQID (Error Code 10026) 🔢

the seqid in a locking request is neither the next expected number nor the last one processed. In NFSv4.1 these seqids just get ignored 🙃.

15.1.16.2. NFS4ERR_LEASE_MOVED (Error Code 10031) 🚚

a lease that's being renewed belongs to a file system that got migrated to a new server. This error has been superseded by the SEQ4_STATUS_LEASE_MOVED status bit (see Section 18.46) 🦋.

15.1.16.3. NFS4ERR_NXIO (Error Code 5) 📟

this is the I/O error that says "no such device or address" 🙃. It's for block and character device access, and NFSv4.1 isn't a device-access protocol, so it doesn't apply here 🙅‍♀️.

15.1.16.4. NFS4ERR_RESTOREFH (Error Code 10030) 📌

the RESTOREFH op has no saved filehandle (the one SAVEFH saved) to work with. In NFSv4.1, NFS4ERR_NOFILEHANDLE has taken over this job 🧃.

15.1.16.5. NFS4ERR_STALE_STATEID (Error Code 10023) 🕰️

a stateid from an earlier server instance got used. This one is moot in NFSv4.1 because any op that takes a stateid MUST be preceded by the SEQUENCE op. The session infrastructure that supports SEQUENCE catches the earlier server instance on its own 🧠✨.

15.2. Operations and Their Valid Errors 📋

this section has a table that lists the valid error returns for each protocol op 💖. The NFS4_OK error code (meaning no error) isn't listed, but you should understand that every op can return it, with two big exceptions 🌸:

The ops that MUST NOT be implemented: OPEN_CONFIRM, RELEASE_LOCKOWNER, RENEW, SETCLIENTID, and SETCLIENTID_CONFIRM 🚫.
The invalid op: ILLEGAL 💀.

Okay bestie, this is the big cheat sheet 📋✨ It lists every operation next to the errors it can return, so you can check any op at a glance 💅

     +======================+========================================+
     | Operation            | Errors                                 |
     +======================+========================================+
     | ACCESS               | NFS4ERR_ACCESS, NFS4ERR_BADXDR,        |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_INVAL,      |
     |                      | NFS4ERR_IO, NFS4ERR_MOVED,             |
     |                      | NFS4ERR_NOFILEHANDLE,                  |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS                   |
     +----------------------+----------------------------------------+
     | BACKCHANNEL_CTL      | NFS4ERR_BADXDR, NFS4ERR_DEADSESSION,   |
     |                      | NFS4ERR_DELAY, NFS4ERR_INVAL,          |
     |                      | NFS4ERR_NOENT,                         |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_TOO_MANY_OPS                   |
     +----------------------+----------------------------------------+
     | BIND_CONN_TO_SESSION | NFS4ERR_BADSESSION, NFS4ERR_BADXDR,    |
     |                      | NFS4ERR_BAD_SESSION_DIGEST,            |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_INVAL, NFS4ERR_NOT_ONLY_OP,    |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT,                   |
     |                      | NFS4ERR_TOO_MANY_OPS                   |
     +----------------------+----------------------------------------+
     | CLOSE                | NFS4ERR_ADMIN_REVOKED, NFS4ERR_BADXDR, |
     |                      | NFS4ERR_BAD_STATEID,                   |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_EXPIRED, NFS4ERR_FHEXPIRED,    |
     |                      | NFS4ERR_LOCKS_HELD, NFS4ERR_MOVED,     |
     |                      | NFS4ERR_NOFILEHANDLE,                  |
     |                      | NFS4ERR_OLD_STATEID,                   |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_WRONG_CRED                     |
     +----------------------+----------------------------------------+
     | COMMIT               | NFS4ERR_ACCESS, NFS4ERR_BADXDR,        |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_IO,         |
     |                      | NFS4ERR_ISDIR, NFS4ERR_MOVED,          |
     |                      | NFS4ERR_NOFILEHANDLE,                  |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_SYMLINK, NFS4ERR_TOO_MANY_OPS, |
     |                      | NFS4ERR_WRONG_TYPE                     |
     +----------------------+----------------------------------------+
     | CREATE               | NFS4ERR_ACCESS, NFS4ERR_ATTRNOTSUPP,   |
     |                      | NFS4ERR_BADCHAR, NFS4ERR_BADNAME,      |
     |                      | NFS4ERR_BADOWNER, NFS4ERR_BADTYPE,     |
     |                      | NFS4ERR_BADXDR, NFS4ERR_DEADSESSION,   |
     |                      | NFS4ERR_DELAY, NFS4ERR_DQUOT,          |
     |                      | NFS4ERR_EXIST, NFS4ERR_FHEXPIRED,      |
     |                      | NFS4ERR_INVAL, NFS4ERR_IO,             |
     |                      | NFS4ERR_MLINK, NFS4ERR_MOVED,          |
     |                      | NFS4ERR_NAMETOOLONG,                   |
     |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOSPC,   |
     |                      | NFS4ERR_NOTDIR,                        |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_PERM, NFS4ERR_REP_TOO_BIG,     |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_ROFS, NFS4ERR_SERVERFAULT,     |
     |                      | NFS4ERR_STALE, NFS4ERR_TOO_MANY_OPS,   |
     |                      | NFS4ERR_UNSAFE_COMPOUND                |
     +----------------------+----------------------------------------+
     | CREATE_SESSION       | NFS4ERR_BADXDR, NFS4ERR_CLID_INUSE,    |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_INVAL, NFS4ERR_NOENT,          |
     |                      | NFS4ERR_NOT_ONLY_OP, NFS4ERR_NOSPC,    |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SEQ_MISORDERED,                |
     |                      | NFS4ERR_SERVERFAULT,                   |
     |                      | NFS4ERR_STALE_CLIENTID,                |
     |                      | NFS4ERR_TOOSMALL,                      |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_WRONG_CRED                     |
     +----------------------+----------------------------------------+
     | DELEGPURGE           | NFS4ERR_BADXDR, NFS4ERR_DEADSESSION,   |
     |                      | NFS4ERR_DELAY, NFS4ERR_NOTSUPP,        |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT,                   |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_WRONG_CRED                     |
     +----------------------+----------------------------------------+
     | DELEGRETURN          | NFS4ERR_ADMIN_REVOKED, NFS4ERR_BADXDR, |
     |                      | NFS4ERR_BAD_STATEID,                   |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_DELEG_REVOKED,                 |
     |                      | NFS4ERR_EXPIRED, NFS4ERR_FHEXPIRED,    |
     |                      | NFS4ERR_INVAL, NFS4ERR_MOVED,          |
     |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTSUPP, |
     |                      | NFS4ERR_OLD_STATEID,                   |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_WRONG_CRED                     |
     +----------------------+----------------------------------------+
     | DESTROY_CLIENTID     | NFS4ERR_BADXDR, NFS4ERR_CLIENTID_BUSY, |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_NOT_ONLY_OP,                   |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT,                   |
     |                      | NFS4ERR_STALE_CLIENTID,                |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_WRONG_CRED                     |
     +----------------------+----------------------------------------+
     | DESTROY_SESSION      | NFS4ERR_BACK_CHAN_BUSY,                |
     |                      | NFS4ERR_BADSESSION, NFS4ERR_BADXDR,    |
     |                      | NFS4ERR_CB_PATH_DOWN,                  |
     |                      | NFS4ERR_CONN_NOT_BOUND_TO_SESSION,     |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_NOT_ONLY_OP,                   |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT,                   |
     |                      | NFS4ERR_STALE_CLIENTID,                |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_WRONG_CRED                     |
     +----------------------+----------------------------------------+
     | EXCHANGE_ID          | NFS4ERR_BADCHAR, NFS4ERR_BADXDR,       |
     |                      | NFS4ERR_CLID_INUSE,                    |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_ENCR_ALG_UNSUPP,               |
     |                      | NFS4ERR_HASH_ALG_UNSUPP,               |
     |                      | NFS4ERR_INVAL, NFS4ERR_NOENT,          |
     |                      | NFS4ERR_NOT_ONLY_OP, NFS4ERR_NOT_SAME, |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT,                   |

okay so we're picking up mid-table, right where the rows keep going 💅🎀

     |                      | NFS4ERR_TOO_MANY_OPS                   |
     +----------------------+----------------------------------------+
     | FREE_STATEID         | NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID,   |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_LOCKS_HELD,                    |
     |                      | NFS4ERR_OLD_STATEID,                   |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT,                   |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_WRONG_CRED                     |
     +----------------------+----------------------------------------+
     | GET_DIR_DELEGATION   | NFS4ERR_ACCESS, NFS4ERR_BADXDR,        |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_DIRDELEG_UNAVAIL,              |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,      |
     |                      | NFS4ERR_INVAL, NFS4ERR_IO,             |
     |                      | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,   |
     |                      | NFS4ERR_NOTDIR, NFS4ERR_NOTSUPP,       |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS                   |
     +----------------------+----------------------------------------+
     | GETATTR              | NFS4ERR_ACCESS, NFS4ERR_BADXDR,        |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,      |
     |                      | NFS4ERR_INVAL, NFS4ERR_IO,             |
     |                      | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,   |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_WRONG_TYPE                     |
     +----------------------+----------------------------------------+
     | GETDEVICEINFO        | NFS4ERR_BADXDR, NFS4ERR_DEADSESSION,   |
     |                      | NFS4ERR_DELAY, NFS4ERR_INVAL,          |
     |                      | NFS4ERR_NOENT, NFS4ERR_NOTSUPP,        |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_TOOSMALL, |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_UNKNOWN_LAYOUTTYPE             |
     +----------------------+----------------------------------------+
     | GETDEVICELIST        | NFS4ERR_BADXDR, NFS4ERR_BAD_COOKIE,    |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_INVAL,      |
     |                      | NFS4ERR_IO, NFS4ERR_NOFILEHANDLE,      |
     |                      | NFS4ERR_NOTSUPP, NFS4ERR_NOT_SAME,     |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT,                   |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_UNKNOWN_LAYOUTTYPE             |
     +----------------------+----------------------------------------+
     | GETFH                | NFS4ERR_FHEXPIRED, NFS4ERR_MOVED,      |
     |                      | NFS4ERR_NOFILEHANDLE,                  |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_STALE                          |
     +----------------------+----------------------------------------+
     | ILLEGAL              | NFS4ERR_BADXDR, NFS4ERR_OP_ILLEGAL     |
     +----------------------+----------------------------------------+
     | LAYOUTCOMMIT         | NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED, |
     |                      | NFS4ERR_ATTRNOTSUPP,                   |
     |                      | NFS4ERR_BADIOMODE, NFS4ERR_BADLAYOUT,  |
     |                      | NFS4ERR_BADXDR, NFS4ERR_DEADSESSION,   |
     |                      | NFS4ERR_DELAY, NFS4ERR_DELEG_REVOKED,  |
     |                      | NFS4ERR_EXPIRED, NFS4ERR_FBIG,         |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,      |
     |                      | NFS4ERR_INVAL, NFS4ERR_IO,             |
     |                      | NFS4ERR_ISDIR NFS4ERR_MOVED,           |
     |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTSUPP, |
     |                      | NFS4ERR_NO_GRACE,                      |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_RECLAIM_BAD,                   |
     |                      | NFS4ERR_RECLAIM_CONFLICT,              |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_SYMLINK, NFS4ERR_TOO_MANY_OPS, |
     |                      | NFS4ERR_UNKNOWN_LAYOUTTYPE,            |
     |                      | NFS4ERR_WRONG_CRED                     |
     +----------------------+----------------------------------------+
     | LAYOUTGET            | NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED, |
     |                      | NFS4ERR_BADIOMODE, NFS4ERR_BADLAYOUT,  |
     |                      | NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID,   |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_DELEG_REVOKED, NFS4ERR_DQUOT,  |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,      |
     |                      | NFS4ERR_INVAL, NFS4ERR_IO,             |
     |                      | NFS4ERR_LAYOUTTRYLATER,                |
     |                      | NFS4ERR_LAYOUTUNAVAILABLE,             |
     |                      | NFS4ERR_LOCKED, NFS4ERR_MOVED,         |
     |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOSPC,   |
     |                      | NFS4ERR_NOTSUPP, NFS4ERR_OLD_STATEID,  |
     |                      | NFS4ERR_OPENMODE,                      |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_RECALLCONFLICT,                |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOOSMALL,                      |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_UNKNOWN_LAYOUTTYPE,            |
     |                      | NFS4ERR_WRONG_TYPE                     |
     +----------------------+----------------------------------------+
     | LAYOUTRETURN         | NFS4ERR_ADMIN_REVOKED, NFS4ERR_BADXDR, |
     |                      | NFS4ERR_BAD_STATEID,                   |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_DELEG_REVOKED,                 |
     |                      | NFS4ERR_EXPIRED, NFS4ERR_FHEXPIRED,    |
     |                      | NFS4ERR_GRACE, NFS4ERR_INVAL,          |
     |                      | NFS4ERR_ISDIR, NFS4ERR_MOVED,          |
     |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTSUPP, |
     |                      | NFS4ERR_NO_GRACE, NFS4ERR_OLD_STATEID, |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_UNKNOWN_LAYOUTTYPE,            |
     |                      | NFS4ERR_WRONG_CRED, NFS4ERR_WRONG_TYPE |
     +----------------------+----------------------------------------+
     | LINK                 | NFS4ERR_ACCESS, NFS4ERR_BADCHAR,       |
     |                      | NFS4ERR_BADNAME, NFS4ERR_BADXDR,       |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_DQUOT, NFS4ERR_EXIST,          |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_FILE_OPEN,  |
     |                      | NFS4ERR_GRACE, NFS4ERR_INVAL,          |
     |                      | NFS4ERR_ISDIR, NFS4ERR_IO,             |
     |                      | NFS4ERR_MLINK, NFS4ERR_MOVED,          |
     |                      | NFS4ERR_NAMETOOLONG,                   |
     |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOSPC,   |
     |                      | NFS4ERR_NOTDIR, NFS4ERR_NOTSUPP,       |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_ROFS, NFS4ERR_SERVERFAULT,     |
     |                      | NFS4ERR_STALE, NFS4ERR_SYMLINK,        |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_WRONGSEC, NFS4ERR_WRONG_TYPE,  |
     |                      | NFS4ERR_XDEV                           |
     +----------------------+----------------------------------------+
     | LOCK                 | NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED, |
     |                      | NFS4ERR_BADXDR, NFS4ERR_BAD_RANGE,     |
     |                      | NFS4ERR_BAD_STATEID, NFS4ERR_DEADLOCK, |

	NFS4ERR_DEADSESSION, NFS4ERR_DELAY,
	NFS4ERR_DENIED, NFS4ERR_EXPIRED,
	NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,
	NFS4ERR_INVAL, NFS4ERR_ISDIR,
	NFS4ERR_LOCK_NOTSUPP,
	NFS4ERR_LOCK_RANGE, NFS4ERR_MOVED,
	NFS4ERR_NOFILEHANDLE,
	NFS4ERR_NO_GRACE, NFS4ERR_OLD_STATEID,
	NFS4ERR_OPENMODE,
	NFS4ERR_OP_NOT_IN_SESSION,
	NFS4ERR_RECLAIM_BAD,
	NFS4ERR_RECLAIM_CONFLICT,
	NFS4ERR_REP_TOO_BIG,
	NFS4ERR_REP_TOO_BIG_TO_CACHE,
	NFS4ERR_REQ_TOO_BIG,
	NFS4ERR_RETRY_UNCACHED_REP,
	NFS4ERR_ROFS, NFS4ERR_SERVERFAULT,
	NFS4ERR_STALE, NFS4ERR_SYMLINK,
	NFS4ERR_TOO_MANY_OPS,
	NFS4ERR_WRONG_CRED, NFS4ERR_WRONG_TYPE

+----------------------+----------------------------------------+

LOCKT	NFS4ERR_ACCESS, NFS4ERR_BADXDR,
	NFS4ERR_BAD_RANGE,
	NFS4ERR_DEADSESSION, NFS4ERR_DELAY,
	NFS4ERR_DENIED, NFS4ERR_FHEXPIRED,
	NFS4ERR_GRACE, NFS4ERR_INVAL,
	NFS4ERR_ISDIR, NFS4ERR_LOCK_RANGE,
	NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,
	NFS4ERR_OP_NOT_IN_SESSION,
	NFS4ERR_REP_TOO_BIG,
	NFS4ERR_REP_TOO_BIG_TO_CACHE,
	NFS4ERR_REQ_TOO_BIG,
	NFS4ERR_RETRY_UNCACHED_REP,
	NFS4ERR_ROFS, NFS4ERR_STALE,
	NFS4ERR_SYMLINK, NFS4ERR_TOO_MANY_OPS,
	NFS4ERR_WRONG_CRED, NFS4ERR_WRONG_TYPE

+----------------------+----------------------------------------+

LOCKU	NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED,
	NFS4ERR_BADXDR, NFS4ERR_BAD_RANGE,
	NFS4ERR_BAD_STATEID,
	NFS4ERR_DEADSESSION, NFS4ERR_DELAY,
	NFS4ERR_EXPIRED, NFS4ERR_FHEXPIRED,
	NFS4ERR_INVAL, NFS4ERR_LOCK_RANGE,
	NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,
	NFS4ERR_OLD_STATEID,
	NFS4ERR_OP_NOT_IN_SESSION,
	NFS4ERR_REP_TOO_BIG,
	NFS4ERR_REP_TOO_BIG_TO_CACHE,
	NFS4ERR_REQ_TOO_BIG,
	NFS4ERR_RETRY_UNCACHED_REP,
	NFS4ERR_SERVERFAULT, NFS4ERR_STALE,
	NFS4ERR_TOO_MANY_OPS,
	NFS4ERR_WRONG_CRED

+----------------------+----------------------------------------+

LOOKUP	NFS4ERR_ACCESS, NFS4ERR_BADCHAR,
	NFS4ERR_BADNAME, NFS4ERR_BADXDR,
	NFS4ERR_DEADSESSION, NFS4ERR_DELAY,
	NFS4ERR_FHEXPIRED, NFS4ERR_INVAL,
	NFS4ERR_IO, NFS4ERR_MOVED,
	NFS4ERR_NAMETOOLONG, NFS4ERR_NOENT,
	NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTDIR,
	NFS4ERR_OP_NOT_IN_SESSION,
	NFS4ERR_REP_TOO_BIG,
	NFS4ERR_REP_TOO_BIG_TO_CACHE,
	NFS4ERR_REQ_TOO_BIG,
	NFS4ERR_RETRY_UNCACHED_REP,
	NFS4ERR_SERVERFAULT, NFS4ERR_STALE,
	NFS4ERR_SYMLINK, NFS4ERR_TOO_MANY_OPS,
	NFS4ERR_WRONGSEC

+----------------------+----------------------------------------+

LOOKUPP	NFS4ERR_ACCESS, NFS4ERR_DEADSESSION,
	NFS4ERR_DELAY, NFS4ERR_FHEXPIRED,
	NFS4ERR_IO, NFS4ERR_MOVED,
	NFS4ERR_NOENT, NFS4ERR_NOFILEHANDLE,
	NFS4ERR_NOTDIR,
	NFS4ERR_OP_NOT_IN_SESSION,
	NFS4ERR_REP_TOO_BIG,
	NFS4ERR_REP_TOO_BIG_TO_CACHE,
	NFS4ERR_REQ_TOO_BIG,
	NFS4ERR_RETRY_UNCACHED_REP,
	NFS4ERR_SERVERFAULT, NFS4ERR_STALE,
	NFS4ERR_SYMLINK, NFS4ERR_TOO_MANY_OPS,
	NFS4ERR_WRONGSEC

+----------------------+----------------------------------------+

NVERIFY	NFS4ERR_ACCESS, NFS4ERR_ATTRNOTSUPP,
	NFS4ERR_BADCHAR, NFS4ERR_BADXDR,
	NFS4ERR_DEADSESSION, NFS4ERR_DELAY,
	NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,
	NFS4ERR_INVAL, NFS4ERR_IO,
	NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,
	NFS4ERR_OP_NOT_IN_SESSION,
	NFS4ERR_REP_TOO_BIG,
	NFS4ERR_REP_TOO_BIG_TO_CACHE,
	NFS4ERR_REQ_TOO_BIG,
	NFS4ERR_RETRY_UNCACHED_REP,
	NFS4ERR_SAME, NFS4ERR_SERVERFAULT,
	NFS4ERR_STALE, NFS4ERR_TOO_MANY_OPS,
	NFS4ERR_UNKNOWN_LAYOUTTYPE,
	NFS4ERR_WRONG_TYPE

+----------------------+----------------------------------------+

OPEN	NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED,
	NFS4ERR_ATTRNOTSUPP, NFS4ERR_BADCHAR,
	NFS4ERR_BADNAME, NFS4ERR_BADOWNER,
	NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID,
	NFS4ERR_DEADSESSION, NFS4ERR_DELAY,
	NFS4ERR_DELEG_ALREADY_WANTED,
	NFS4ERR_DELEG_REVOKED, NFS4ERR_DQUOT,
	NFS4ERR_EXIST, NFS4ERR_EXPIRED,
	NFS4ERR_FBIG, NFS4ERR_FHEXPIRED,
	NFS4ERR_GRACE, NFS4ERR_INVAL,
	NFS4ERR_ISDIR, NFS4ERR_IO,
	NFS4ERR_MOVED, NFS4ERR_NAMETOOLONG,
	NFS4ERR_NOENT, NFS4ERR_NOFILEHANDLE,
	NFS4ERR_NOSPC, NFS4ERR_NOTDIR,
	NFS4ERR_NO_GRACE, NFS4ERR_OLD_STATEID,
	NFS4ERR_OP_NOT_IN_SESSION,
	NFS4ERR_PERM, NFS4ERR_RECLAIM_BAD,
	NFS4ERR_RECLAIM_CONFLICT,
	NFS4ERR_REP_TOO_BIG,
	NFS4ERR_REP_TOO_BIG_TO_CACHE,
	NFS4ERR_REQ_TOO_BIG,
	NFS4ERR_RETRY_UNCACHED_REP,
	NFS4ERR_ROFS, NFS4ERR_SERVERFAULT,
	NFS4ERR_SHARE_DENIED, NFS4ERR_STALE,
	NFS4ERR_SYMLINK, NFS4ERR_TOO_MANY_OPS,
	NFS4ERR_UNSAFE_COMPOUND,
	NFS4ERR_WRONGSEC, NFS4ERR_WRONG_TYPE

+----------------------+----------------------------------------+

OPEN_CONFIRM

NFS4ERR_NOTSUPP

+----------------------+----------------------------------------+

OPEN_DOWNGRADE	NFS4ERR_ADMIN_REVOKED, NFS4ERR_BADXDR,
	NFS4ERR_BAD_STATEID,
	NFS4ERR_DEADSESSION, NFS4ERR_DELAY,
	NFS4ERR_EXPIRED, NFS4ERR_FHEXPIRED,
	NFS4ERR_INVAL, NFS4ERR_MOVED,
	NFS4ERR_NOFILEHANDLE,
	NFS4ERR_OLD_STATEID,
	NFS4ERR_OP_NOT_IN_SESSION,
	NFS4ERR_REP_TOO_BIG,
	NFS4ERR_REP_TOO_BIG_TO_CACHE,
	NFS4ERR_REQ_TOO_BIG,
	NFS4ERR_RETRY_UNCACHED_REP,
	NFS4ERR_ROFS, NFS4ERR_SERVERFAULT,
	NFS4ERR_STALE, NFS4ERR_TOO_MANY_OPS,
	NFS4ERR_WRONG_CRED

+----------------------+----------------------------------------+

OPENATTR	NFS4ERR_ACCESS, NFS4ERR_BADXDR,
	NFS4ERR_DEADSESSION, NFS4ERR_DELAY,
	NFS4ERR_DQUOT, NFS4ERR_FHEXPIRED,
	NFS4ERR_IO, NFS4ERR_MOVED,
	NFS4ERR_NOENT, NFS4ERR_NOFILEHANDLE,
	NFS4ERR_NOSPC, NFS4ERR_NOTSUPP,
	NFS4ERR_OP_NOT_IN_SESSION,
	NFS4ERR_REP_TOO_BIG,
	NFS4ERR_REP_TOO_BIG_TO_CACHE,
	NFS4ERR_REQ_TOO_BIG,
	NFS4ERR_RETRY_UNCACHED_REP,
	NFS4ERR_ROFS, NFS4ERR_SERVERFAULT,
	NFS4ERR_STALE, NFS4ERR_TOO_MANY_OPS,
	NFS4ERR_UNSAFE_COMPOUND,
	NFS4ERR_WRONG_TYPE

+----------------------+----------------------------------------+

PUTFH	NFS4ERR_BADHANDLE, NFS4ERR_BADXDR,
	NFS4ERR_DEADSESSION, NFS4ERR_DELAY,
	NFS4ERR_MOVED,
	NFS4ERR_OP_NOT_IN_SESSION,
	NFS4ERR_REP_TOO_BIG,
	NFS4ERR_REP_TOO_BIG_TO_CACHE,
	NFS4ERR_REQ_TOO_BIG,

okay so this table is basically the cheat sheet for which errors each operation is allowed to throw 💅 keep scrolling bestie 🧚

     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONGSEC |
     +----------------------+----------------------------------------+
     | PUTPUBFH             | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT,                   |
     |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONGSEC |
     +----------------------+----------------------------------------+
     | PUTROOTFH            | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT,                   |
     |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONGSEC |
     +----------------------+----------------------------------------+
     | READ                 | NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED, |
     |                      | NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID,   |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_DELEG_REVOKED,                 |
     |                      | NFS4ERR_EXPIRED, NFS4ERR_FHEXPIRED,    |
     |                      | NFS4ERR_GRACE, NFS4ERR_INVAL,          |
     |                      | NFS4ERR_ISDIR, NFS4ERR_IO,             |
     |                      | NFS4ERR_LOCKED, NFS4ERR_MOVED,         |
     |                      | NFS4ERR_NOFILEHANDLE,                  |
     |                      | NFS4ERR_OLD_STATEID, NFS4ERR_OPENMODE, |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_PNFS_IO_HOLE,                  |
     |                      | NFS4ERR_PNFS_NO_LAYOUT,                |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_SYMLINK, NFS4ERR_TOO_MANY_OPS, |
     |                      | NFS4ERR_WRONG_TYPE                     |
     +----------------------+----------------------------------------+
     | READDIR              | NFS4ERR_ACCESS, NFS4ERR_BADXDR,        |
     |                      | NFS4ERR_BAD_COOKIE,                    |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_INVAL,      |
     |                      | NFS4ERR_IO, NFS4ERR_MOVED,             |
     |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTDIR,  |
     |                      | NFS4ERR_NOT_SAME,                      |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOOSMALL, NFS4ERR_TOO_MANY_OPS |
     +----------------------+----------------------------------------+
     | READLINK             | NFS4ERR_ACCESS, NFS4ERR_DEADSESSION,   |
     |                      | NFS4ERR_DELAY, NFS4ERR_FHEXPIRED,      |
     |                      | NFS4ERR_INVAL, NFS4ERR_IO,             |
     |                      | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,   |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_WRONG_TYPE                     |
     +----------------------+----------------------------------------+
     | RECLAIM_COMPLETE     | NFS4ERR_BADXDR,                        |
     |                      | NFS4ERR_COMPLETE_ALREADY,              |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_INVAL,      |
     |                      | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,   |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_WRONG_CRED, NFS4ERR_WRONG_TYPE |
     +----------------------+----------------------------------------+
     | RELEASE_LOCKOWNER    | NFS4ERR_NOTSUPP                        |
     +----------------------+----------------------------------------+
     | REMOVE               | NFS4ERR_ACCESS, NFS4ERR_BADCHAR,       |
     |                      | NFS4ERR_BADNAME, NFS4ERR_BADXDR,       |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_FILE_OPEN,  |
     |                      | NFS4ERR_GRACE, NFS4ERR_INVAL,          |
     |                      | NFS4ERR_IO, NFS4ERR_MOVED,             |
     |                      | NFS4ERR_NAMETOOLONG, NFS4ERR_NOENT,    |
     |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTDIR,  |
     |                      | NFS4ERR_NOTEMPTY,                      |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_ROFS, NFS4ERR_SERVERFAULT,     |
     |                      | NFS4ERR_STALE, NFS4ERR_TOO_MANY_OPS    |
     +----------------------+----------------------------------------+
     | RENAME               | NFS4ERR_ACCESS, NFS4ERR_BADCHAR,       |
     |                      | NFS4ERR_BADNAME, NFS4ERR_BADXDR,       |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_DQUOT, NFS4ERR_EXIST,          |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_FILE_OPEN,  |
     |                      | NFS4ERR_GRACE, NFS4ERR_INVAL,          |
     |                      | NFS4ERR_IO, NFS4ERR_MLINK,             |
     |                      | NFS4ERR_MOVED, NFS4ERR_NAMETOOLONG,    |
     |                      | NFS4ERR_NOENT, NFS4ERR_NOFILEHANDLE,   |
     |                      | NFS4ERR_NOSPC, NFS4ERR_NOTDIR,         |
     |                      | NFS4ERR_NOTEMPTY,                      |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_ROFS, NFS4ERR_SERVERFAULT,     |
     |                      | NFS4ERR_STALE, NFS4ERR_TOO_MANY_OPS,   |
     |                      | NFS4ERR_WRONGSEC, NFS4ERR_XDEV         |
     +----------------------+----------------------------------------+
     | RENEW                | NFS4ERR_NOTSUPP                        |
     +----------------------+----------------------------------------+
     | RESTOREFH            | NFS4ERR_DEADSESSION,                   |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_MOVED,      |
     |                      | NFS4ERR_NOFILEHANDLE,                  |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONGSEC |
     +----------------------+----------------------------------------+
     | SAVEFH               | NFS4ERR_DEADSESSION,                   |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_MOVED,      |
     |                      | NFS4ERR_NOFILEHANDLE,                  |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS                   |
     +----------------------+----------------------------------------+
     | SECINFO              | NFS4ERR_ACCESS, NFS4ERR_BADCHAR,       |
     |                      | NFS4ERR_BADNAME, NFS4ERR_BADXDR,       |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_INVAL,      |
     |                      | NFS4ERR_MOVED, NFS4ERR_NAMETOOLONG,    |
     |                      | NFS4ERR_NOENT, NFS4ERR_NOFILEHANDLE,   |
     |                      | NFS4ERR_NOTDIR,                        |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS                   |
     +----------------------+----------------------------------------+
     | SECINFO_NO_NAME      | NFS4ERR_ACCESS, NFS4ERR_BADXDR,        |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_INVAL,      |
     |                      | NFS4ERR_MOVED, NFS4ERR_NOENT,          |
     |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTDIR,  |
     |                      | NFS4ERR_NOTSUPP,                       |

okay the table keeps going, so here's the rest of Table 12 💖📋

     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS                   |
     +----------------------+----------------------------------------+
     | SEQUENCE             | NFS4ERR_BADSESSION, NFS4ERR_BADSLOT,   |
     |                      | NFS4ERR_BADXDR, NFS4ERR_BAD_HIGH_SLOT, |
     |                      | NFS4ERR_CONN_NOT_BOUND_TO_SESSION,     |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SEQUENCE_POS,                  |
     |                      | NFS4ERR_SEQ_FALSE_RETRY,               |
     |                      | NFS4ERR_SEQ_MISORDERED,                |
     |                      | NFS4ERR_TOO_MANY_OPS                   |
     +----------------------+----------------------------------------+
     | SET_SSV              | NFS4ERR_BADXDR,                        |
     |                      | NFS4ERR_BAD_SESSION_DIGEST,            |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_INVAL,                         |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_TOO_MANY_OPS                   |
     +----------------------+----------------------------------------+
     | SETATTR              | NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED, |
     |                      | NFS4ERR_ATTRNOTSUPP, NFS4ERR_BADCHAR,  |
     |                      | NFS4ERR_BADOWNER, NFS4ERR_BADXDR,      |
     |                      | NFS4ERR_BAD_STATEID,                   |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_DELEG_REVOKED, NFS4ERR_DQUOT,  |
     |                      | NFS4ERR_EXPIRED, NFS4ERR_FBIG,         |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,      |
     |                      | NFS4ERR_INVAL, NFS4ERR_IO,             |
     |                      | NFS4ERR_LOCKED, NFS4ERR_MOVED,         |
     |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOSPC,   |
     |                      | NFS4ERR_OLD_STATEID, NFS4ERR_OPENMODE, |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_PERM, NFS4ERR_REP_TOO_BIG,     |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_ROFS, NFS4ERR_SERVERFAULT,     |
     |                      | NFS4ERR_STALE, NFS4ERR_TOO_MANY_OPS,   |
     |                      | NFS4ERR_UNKNOWN_LAYOUTTYPE,            |
     |                      | NFS4ERR_WRONG_TYPE                     |
     +----------------------+----------------------------------------+
     | SETCLIENTID          | NFS4ERR_NOTSUPP                        |
     +----------------------+----------------------------------------+
     | SETCLIENTID_CONFIRM  | NFS4ERR_NOTSUPP                        |
     +----------------------+----------------------------------------+
     | TEST_STATEID         | NFS4ERR_BADXDR, NFS4ERR_DEADSESSION,   |
     |                      | NFS4ERR_DELAY,                         |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT,                   |
     |                      | NFS4ERR_TOO_MANY_OPS                   |
     +----------------------+----------------------------------------+
     | VERIFY               | NFS4ERR_ACCESS, NFS4ERR_ATTRNOTSUPP,   |
     |                      | NFS4ERR_BADCHAR, NFS4ERR_BADXDR,       |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,      |
     |                      | NFS4ERR_INVAL, NFS4ERR_IO,             |
     |                      | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,   |
     |                      | NFS4ERR_NOT_SAME,                      |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_UNKNOWN_LAYOUTTYPE,            |
     |                      | NFS4ERR_WRONG_TYPE                     |
     +----------------------+----------------------------------------+
     | WANT_DELEGATION      | NFS4ERR_BADXDR, NFS4ERR_DEADSESSION,   |
     |                      | NFS4ERR_DELAY,                         |
     |                      | NFS4ERR_DELEG_ALREADY_WANTED,          |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,      |
     |                      | NFS4ERR_INVAL, NFS4ERR_IO,             |
     |                      | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,   |
     |                      | NFS4ERR_NOTSUPP, NFS4ERR_NO_GRACE,     |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_RECALLCONFLICT,                |
     |                      | NFS4ERR_RECLAIM_BAD,                   |
     |                      | NFS4ERR_RECLAIM_CONFLICT,              |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,    |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_WRONG_TYPE                     |
     +----------------------+----------------------------------------+
     | WRITE                | NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED, |
     |                      | NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID,   |
     |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,    |
     |                      | NFS4ERR_DELEG_REVOKED, NFS4ERR_DQUOT,  |
     |                      | NFS4ERR_EXPIRED, NFS4ERR_FBIG,         |
     |                      | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,      |
     |                      | NFS4ERR_INVAL, NFS4ERR_IO,             |
     |                      | NFS4ERR_ISDIR, NFS4ERR_LOCKED,         |
     |                      | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,   |
     |                      | NFS4ERR_NOSPC, NFS4ERR_OLD_STATEID,    |
     |                      | NFS4ERR_OPENMODE,                      |
     |                      | NFS4ERR_OP_NOT_IN_SESSION,             |
     |                      | NFS4ERR_PNFS_IO_HOLE,                  |
     |                      | NFS4ERR_PNFS_NO_LAYOUT,                |
     |                      | NFS4ERR_REP_TOO_BIG,                   |
     |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,          |
     |                      | NFS4ERR_REQ_TOO_BIG,                   |
     |                      | NFS4ERR_RETRY_UNCACHED_REP,            |
     |                      | NFS4ERR_ROFS, NFS4ERR_SERVERFAULT,     |
     |                      | NFS4ERR_STALE, NFS4ERR_SYMLINK,        |
     |                      | NFS4ERR_TOO_MANY_OPS,                  |
     |                      | NFS4ERR_WRONG_TYPE                     |
     +----------------------+----------------------------------------+

         Table 12: Valid Error Returns for Each Protocol Operation

15.3. Callback Operations and Their Valid Errors ✨

okay so this section is the cheat sheet 📋 for which errors each callback operation is allowed to throw back at you. bestie, it's giving very organized energy 💅

the table below lists the valid error returns for every callback operation. NFS4_OK (no error) isn't in the list, but it's understood to be returnable by all callback operations except CB_ILLEGAL 🫶

    +=========================+=======================================+
    | Callback Operation      | Errors                                |
    +=========================+=======================================+
    | CB_GETATTR              | NFS4ERR_BADHANDLE, NFS4ERR_BADXDR,    |
    |                         | NFS4ERR_DELAY, NFS4ERR_INVAL,         |
    |                         | NFS4ERR_OP_NOT_IN_SESSION,            |
    |                         | NFS4ERR_REP_TOO_BIG,                  |
    |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,         |
    |                         | NFS4ERR_REQ_TOO_BIG,                  |
    |                         | NFS4ERR_RETRY_UNCACHED_REP,           |
    |                         | NFS4ERR_SERVERFAULT,                  |
    |                         | NFS4ERR_TOO_MANY_OPS,                 |
    +-------------------------+---------------------------------------+
    | CB_ILLEGAL              | NFS4ERR_BADXDR, NFS4ERR_OP_ILLEGAL    |
    +-------------------------+---------------------------------------+
    | CB_LAYOUTRECALL         | NFS4ERR_BADHANDLE, NFS4ERR_BADIOMODE, |
    |                         | NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID,  |
    |                         | NFS4ERR_DELAY, NFS4ERR_INVAL,         |
    |                         | NFS4ERR_NOMATCHING_LAYOUT,            |
    |                         | NFS4ERR_NOTSUPP,                      |
    |                         | NFS4ERR_OP_NOT_IN_SESSION,            |
    |                         | NFS4ERR_REP_TOO_BIG,                  |
    |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,         |
    |                         | NFS4ERR_REQ_TOO_BIG,                  |
    |                         | NFS4ERR_RETRY_UNCACHED_REP,           |
    |                         | NFS4ERR_TOO_MANY_OPS,                 |
    |                         | NFS4ERR_UNKNOWN_LAYOUTTYPE,           |
    |                         | NFS4ERR_WRONG_TYPE                    |
    +-------------------------+---------------------------------------+
    | CB_NOTIFY               | NFS4ERR_BADHANDLE, NFS4ERR_BADXDR,    |
    |                         | NFS4ERR_BAD_STATEID, NFS4ERR_DELAY,   |
    |                         | NFS4ERR_INVAL, NFS4ERR_NOTSUPP,       |
    |                         | NFS4ERR_OP_NOT_IN_SESSION,            |
    |                         | NFS4ERR_REP_TOO_BIG,                  |
    |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,         |
    |                         | NFS4ERR_REQ_TOO_BIG,                  |
    |                         | NFS4ERR_RETRY_UNCACHED_REP,           |
    |                         | NFS4ERR_SERVERFAULT,                  |
    |                         | NFS4ERR_TOO_MANY_OPS                  |
    +-------------------------+---------------------------------------+
    | CB_NOTIFY_DEVICEID      | NFS4ERR_BADXDR, NFS4ERR_DELAY,        |
    |                         | NFS4ERR_INVAL, NFS4ERR_NOTSUPP,       |
    |                         | NFS4ERR_OP_NOT_IN_SESSION,            |
    |                         | NFS4ERR_REP_TOO_BIG,                  |
    |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,         |
    |                         | NFS4ERR_REQ_TOO_BIG,                  |
    |                         | NFS4ERR_RETRY_UNCACHED_REP,           |
    |                         | NFS4ERR_SERVERFAULT,                  |
    |                         | NFS4ERR_TOO_MANY_OPS                  |
    +-------------------------+---------------------------------------+
    | CB_NOTIFY_LOCK          | NFS4ERR_BADHANDLE, NFS4ERR_BADXDR,    |
    |                         | NFS4ERR_BAD_STATEID, NFS4ERR_DELAY,   |
    |                         | NFS4ERR_NOTSUPP,                      |
    |                         | NFS4ERR_OP_NOT_IN_SESSION,            |
    |                         | NFS4ERR_REP_TOO_BIG,                  |
    |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,         |
    |                         | NFS4ERR_REQ_TOO_BIG,                  |
    |                         | NFS4ERR_RETRY_UNCACHED_REP,           |
    |                         | NFS4ERR_SERVERFAULT,                  |
    |                         | NFS4ERR_TOO_MANY_OPS                  |
    +-------------------------+---------------------------------------+
    | CB_PUSH_DELEG           | NFS4ERR_BADHANDLE, NFS4ERR_BADXDR,    |
    |                         | NFS4ERR_DELAY, NFS4ERR_INVAL,         |
    |                         | NFS4ERR_NOTSUPP,                      |
    |                         | NFS4ERR_OP_NOT_IN_SESSION,            |
    |                         | NFS4ERR_REJECT_DELEG,                 |
    |                         | NFS4ERR_REP_TOO_BIG,                  |
    |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,         |
    |                         | NFS4ERR_REQ_TOO_BIG,                  |
    |                         | NFS4ERR_RETRY_UNCACHED_REP,           |
    |                         | NFS4ERR_SERVERFAULT,                  |
    |                         | NFS4ERR_TOO_MANY_OPS,                 |
    |                         | NFS4ERR_WRONG_TYPE                    |
    +-------------------------+---------------------------------------+
    | CB_RECALL               | NFS4ERR_BADHANDLE, NFS4ERR_BADXDR,    |
    |                         | NFS4ERR_BAD_STATEID, NFS4ERR_DELAY,   |
    |                         | NFS4ERR_OP_NOT_IN_SESSION,            |
    |                         | NFS4ERR_REP_TOO_BIG,                  |
    |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,         |
    |                         | NFS4ERR_REQ_TOO_BIG,                  |
    |                         | NFS4ERR_RETRY_UNCACHED_REP,           |
    |                         | NFS4ERR_SERVERFAULT,                  |
    |                         | NFS4ERR_TOO_MANY_OPS                  |
    +-------------------------+---------------------------------------+
    | CB_RECALL_ANY           | NFS4ERR_BADXDR, NFS4ERR_DELAY,        |
    |                         | NFS4ERR_INVAL,                        |
    |                         | NFS4ERR_OP_NOT_IN_SESSION,            |
    |                         | NFS4ERR_REP_TOO_BIG,                  |
    |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,         |
    |                         | NFS4ERR_REQ_TOO_BIG,                  |
    |                         | NFS4ERR_RETRY_UNCACHED_REP,           |
    |                         | NFS4ERR_TOO_MANY_OPS                  |
    +-------------------------+---------------------------------------+
    | CB_RECALLABLE_OBJ_AVAIL | NFS4ERR_BADXDR, NFS4ERR_DELAY,        |
    |                         | NFS4ERR_INVAL, NFS4ERR_NOTSUPP,       |
    |                         | NFS4ERR_OP_NOT_IN_SESSION,            |
    |                         | NFS4ERR_REP_TOO_BIG,                  |
    |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,         |
    |                         | NFS4ERR_REQ_TOO_BIG,                  |
    |                         | NFS4ERR_RETRY_UNCACHED_REP,           |
    |                         | NFS4ERR_SERVERFAULT,                  |
    |                         | NFS4ERR_TOO_MANY_OPS                  |
    +-------------------------+---------------------------------------+
    | CB_RECALL_SLOT          | NFS4ERR_BADXDR,                       |
    |                         | NFS4ERR_BAD_HIGH_SLOT, NFS4ERR_DELAY, |
    |                         | NFS4ERR_OP_NOT_IN_SESSION,            |
    |                         | NFS4ERR_REP_TOO_BIG,                  |
    |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,         |
    |                         | NFS4ERR_REQ_TOO_BIG,                  |
    |                         | NFS4ERR_RETRY_UNCACHED_REP,           |
    |                         | NFS4ERR_TOO_MANY_OPS                  |
    +-------------------------+---------------------------------------+
    | CB_SEQUENCE             | NFS4ERR_BADSESSION, NFS4ERR_BADSLOT,  |
    |                         | NFS4ERR_BADXDR,                       |
    |                         | NFS4ERR_BAD_HIGH_SLOT,                |
    |                         | NFS4ERR_CONN_NOT_BOUND_TO_SESSION,    |
    |                         | NFS4ERR_DELAY, NFS4ERR_REP_TOO_BIG,   |
    |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,         |
    |                         | NFS4ERR_REQ_TOO_BIG,                  |
    |                         | NFS4ERR_RETRY_UNCACHED_REP,           |
    |                         | NFS4ERR_SEQUENCE_POS,                 |
    |                         | NFS4ERR_SEQ_FALSE_RETRY,              |
    |                         | NFS4ERR_SEQ_MISORDERED,               |
    |                         | NFS4ERR_TOO_MANY_OPS                  |
    +-------------------------+---------------------------------------+
    | CB_WANTS_CANCELLED      | NFS4ERR_BADXDR, NFS4ERR_DELAY,        |
    |                         | NFS4ERR_NOTSUPP,                      |
    |                         | NFS4ERR_OP_NOT_IN_SESSION,            |
    |                         | NFS4ERR_REP_TOO_BIG,                  |
    |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,         |
    |                         | NFS4ERR_REQ_TOO_BIG,                  |
    |                         | NFS4ERR_RETRY_UNCACHED_REP,           |
    |                         | NFS4ERR_SERVERFAULT,                  |
    |                         | NFS4ERR_TOO_MANY_OPS                  |
    +-------------------------+---------------------------------------+

     Table 13: Valid Error Returns for Each Protocol Callback Operation

15.4. Errors and the Operations That Use Them 🎀

I can't continue this one in the rewrite voice, because the RFC text you've pasted is a fragment that starts partway through a table. Nothing in it covers the section headings you listed, and there's no earlier text here for me to carry on from. Could you paste part 102 itself, starting at the beginning of the section it covers? Once I have that, I'll rewrite it in the bestie voice, keeping every requirement, error name, operation list and table entry exact.

Okay the table keeps going, bestie, and every error code still has its exact operation list 🦋

   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, SECINFO,              |
   |                                   | SECINFO_NO_NAME, SEQUENCE,    |
   |                                   | SETATTR, SET_SSV,             |
   |                                   | TEST_STATEID, VERIFY,         |
   |                                   | WANT_DELEGATION, WRITE        |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_DELEG_ALREADY_WANTED      | OPEN, WANT_DELEGATION         |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_DELEG_REVOKED             | DELEGRETURN, LAYOUTCOMMIT,    |
   |                                   | LAYOUTGET, LAYOUTRETURN,      |
   |                                   | OPEN, READ, SETATTR, WRITE    |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_DENIED                    | LOCK, LOCKT                   |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_DIRDELEG_UNAVAIL          | GET_DIR_DELEGATION            |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_DQUOT                     | CREATE, LAYOUTGET, LINK,      |
   |                                   | OPEN, OPENATTR, RENAME,       |
   |                                   | SETATTR, WRITE                |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_ENCR_ALG_UNSUPP           | EXCHANGE_ID                   |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_EXIST                     | CREATE, LINK, OPEN, RENAME    |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_EXPIRED                   | CLOSE, DELEGRETURN,           |
   |                                   | LAYOUTCOMMIT, LAYOUTRETURN,   |
   |                                   | LOCK, LOCKU, OPEN,            |
   |                                   | OPEN_DOWNGRADE, READ,         |
   |                                   | SETATTR, WRITE                |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_FBIG                      | LAYOUTCOMMIT, OPEN, SETATTR,  |
   |                                   | WRITE                         |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_FHEXPIRED                 | ACCESS, CLOSE, COMMIT,        |
   |                                   | CREATE, DELEGRETURN,          |
   |                                   | GETATTR, GETDEVICELIST,       |
   |                                   | GETFH, GET_DIR_DELEGATION,    |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SETATTR, VERIFY,              |
   |                                   | WANT_DELEGATION, WRITE        |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_FILE_OPEN                 | LINK, REMOVE, RENAME          |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_GRACE                     | GETATTR, GET_DIR_DELEGATION,  |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, NVERIFY, OPEN, READ,   |
   |                                   | REMOVE, RENAME, SETATTR,      |
   |                                   | VERIFY, WANT_DELEGATION,      |
   |                                   | WRITE                         |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_HASH_ALG_UNSUPP           | EXCHANGE_ID                   |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_INVAL                     | ACCESS, BACKCHANNEL_CTL,      |
   |                                   | BIND_CONN_TO_SESSION,         |
   |                                   | CB_GETATTR, CB_LAYOUTRECALL,  |
   |                                   | CB_NOTIFY,                    |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_PUSH_DELEG,                |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_RECALL_ANY, CREATE,        |
   |                                   | CREATE_SESSION, DELEGRETURN,  |
   |                                   | EXCHANGE_ID, GETATTR,         |
   |                                   | GETDEVICEINFO,                |
   |                                   | GETDEVICELIST,                |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | NVERIFY, OPEN,                |
   |                                   | OPEN_DOWNGRADE, READ,         |
   |                                   | READDIR, READLINK,            |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, SECINFO,              |
   |                                   | SECINFO_NO_NAME, SETATTR,     |
   |                                   | SET_SSV, VERIFY,              |
   |                                   | WANT_DELEGATION, WRITE        |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_IO                        | ACCESS, COMMIT, CREATE,       |
   |                                   | GETATTR, GETDEVICELIST,       |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LINK, LOOKUP, LOOKUPP,        |
   |                                   | NVERIFY, OPEN, OPENATTR,      |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | REMOVE, RENAME, SETATTR,      |
   |                                   | VERIFY, WANT_DELEGATION,      |
   |                                   | WRITE                         |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_ISDIR                     | COMMIT, LAYOUTCOMMIT,         |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, OPEN, READ, WRITE      |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_LAYOUTTRYLATER            | LAYOUTGET                     |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_LAYOUTUNAVAILABLE         | LAYOUTGET                     |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_LOCKED                    | LAYOUTGET, READ, SETATTR,     |
   |                                   | WRITE                         |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_LOCKS_HELD                | CLOSE, FREE_STATEID           |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_LOCK_NOTSUPP              | LOCK                          |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_LOCK_RANGE                | LOCK, LOCKT, LOCKU            |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_MLINK                     | CREATE, LINK, RENAME          |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_MOVED                     | ACCESS, CLOSE, COMMIT,        |
   |                                   | CREATE, DELEGRETURN,          |
   |                                   | GETATTR, GETFH,               |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, READ, READDIR,         |
   |                                   | READLINK, RECLAIM_COMPLETE,   |
   |                                   | REMOVE, RENAME, RESTOREFH,    |
   |                                   | SAVEFH, SECINFO,              |
   |                                   | SECINFO_NO_NAME, SETATTR,     |
   |                                   | VERIFY, WANT_DELEGATION,      |
   |                                   | WRITE                         |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_NAMETOOLONG               | CREATE, LINK, LOOKUP, OPEN,   |
   |                                   | REMOVE, RENAME, SECINFO       |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_NOENT                     | BACKCHANNEL_CTL,              |
   |                                   | CREATE_SESSION, EXCHANGE_ID,  |
   |                                   | GETDEVICEINFO, LOOKUP,        |
   |                                   | LOOKUPP, OPEN, OPENATTR,      |
   |                                   | REMOVE, RENAME, SECINFO,      |
   |                                   | SECINFO_NO_NAME               |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_NOFILEHANDLE              | ACCESS, CLOSE, COMMIT,        |
   |                                   | CREATE, DELEGRETURN,          |
   |                                   | GETATTR, GETDEVICELIST,       |
   |                                   | GETFH, GET_DIR_DELEGATION,    |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SETATTR, VERIFY,              |
   |                                   | WANT_DELEGATION, WRITE        |

okay so this table is a whole map 🗺️✨ Each error code sits on the left, and the operations that can throw it are listed on the right. Let's go, bestie 💅

   +-----------------------------------+-------------------------------+
   | NFS4ERR_NOMATCHING_LAYOUT         | CB_LAYOUTRECALL               |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_NOSPC                     | CREATE, CREATE_SESSION,       |
   |                                   | LAYOUTGET, LINK, OPEN,        |
   |                                   | OPENATTR, RENAME, SETATTR,    |
   |                                   | WRITE                         |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_NOTDIR                    | CREATE, GET_DIR_DELEGATION,   |
   |                                   | LINK, LOOKUP, LOOKUPP, OPEN,  |
   |                                   | READDIR, REMOVE, RENAME,      |
   |                                   | SECINFO, SECINFO_NO_NAME      |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_NOTEMPTY                  | REMOVE, RENAME                |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_NOTSUPP                   | CB_LAYOUTRECALL, CB_NOTIFY,   |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG,                |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_WANTS_CANCELLED,           |
   |                                   | DELEGPURGE, DELEGRETURN,      |
   |                                   | GETDEVICEINFO,                |
   |                                   | GETDEVICELIST,                |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK,           |
   |                                   | OPENATTR, OPEN_CONFIRM,       |
   |                                   | RELEASE_LOCKOWNER, RENEW,     |
   |                                   | SECINFO_NO_NAME,              |
   |                                   | SETCLIENTID,                  |
   |                                   | SETCLIENTID_CONFIRM,          |
   |                                   | WANT_DELEGATION               |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_NOT_ONLY_OP               | BIND_CONN_TO_SESSION,         |
   |                                   | CREATE_SESSION,               |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION, EXCHANGE_ID  |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_NOT_SAME                  | EXCHANGE_ID, GETDEVICELIST,   |
   |                                   | READDIR, VERIFY               |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_NO_GRACE                  | LAYOUTCOMMIT, LAYOUTRETURN,   |
   |                                   | LOCK, OPEN, WANT_DELEGATION   |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_OLD_STATEID               | CLOSE, DELEGRETURN,           |
   |                                   | FREE_STATEID, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LOCK, LOCKU,    |
   |                                   | OPEN, OPEN_DOWNGRADE, READ,   |
   |                                   | SETATTR, WRITE                |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_OPENMODE                  | LAYOUTGET, LOCK, READ,        |
   |                                   | SETATTR, WRITE                |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_OP_ILLEGAL                | CB_ILLEGAL, ILLEGAL           |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_OP_NOT_IN_SESSION         | ACCESS, BACKCHANNEL_CTL,      |
   |                                   | CB_GETATTR, CB_LAYOUTRECALL,  |
   |                                   | CB_NOTIFY,                    |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG, CB_RECALL,     |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_RECALL_ANY,                |
   |                                   | CB_RECALL_SLOT,               |
   |                                   | CB_WANTS_CANCELLED, CLOSE,    |
   |                                   | COMMIT, CREATE, DELEGPURGE,   |
   |                                   | DELEGRETURN, FREE_STATEID,    |
   |                                   | GETATTR, GETDEVICEINFO,       |
   |                                   | GETDEVICELIST, GETFH,         |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SETATTR, SET_SSV,             |
   |                                   | TEST_STATEID, VERIFY,         |
   |                                   | WANT_DELEGATION, WRITE        |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_PERM                      | CREATE, OPEN, SETATTR         |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_PNFS_IO_HOLE              | READ, WRITE                   |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_PNFS_NO_LAYOUT            | READ, WRITE                   |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_RECALLCONFLICT            | LAYOUTGET, WANT_DELEGATION    |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_RECLAIM_BAD               | LAYOUTCOMMIT, LOCK, OPEN,     |
   |                                   | WANT_DELEGATION               |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_RECLAIM_CONFLICT          | LAYOUTCOMMIT, LOCK, OPEN,     |
   |                                   | WANT_DELEGATION               |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_REJECT_DELEG              | CB_PUSH_DELEG                 |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_REP_TOO_BIG               | ACCESS, BACKCHANNEL_CTL,      |
   |                                   | BIND_CONN_TO_SESSION,         |
   |                                   | CB_GETATTR, CB_LAYOUTRECALL,  |
   |                                   | CB_NOTIFY,                    |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG, CB_RECALL,     |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_RECALL_ANY,                |
   |                                   | CB_RECALL_SLOT, CB_SEQUENCE,  |
   |                                   | CB_WANTS_CANCELLED, CLOSE,    |
   |                                   | COMMIT, CREATE,               |
   |                                   | CREATE_SESSION, DELEGPURGE,   |
   |                                   | DELEGRETURN,                  |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION,              |
   |                                   | EXCHANGE_ID, FREE_STATEID,    |
   |                                   | GETATTR, GETDEVICEINFO,       |
   |                                   | GETDEVICELIST,                |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SEQUENCE, SETATTR, SET_SSV,   |
   |                                   | TEST_STATEID, VERIFY,         |
   |                                   | WANT_DELEGATION, WRITE        |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_REP_TOO_BIG_TO_CACHE      | ACCESS, BACKCHANNEL_CTL,      |
   |                                   | BIND_CONN_TO_SESSION,         |
   |                                   | CB_GETATTR, CB_LAYOUTRECALL,  |
   |                                   | CB_NOTIFY,                    |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG, CB_RECALL,     |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_RECALL_ANY,                |
   |                                   | CB_RECALL_SLOT, CB_SEQUENCE,  |
   |                                   | CB_WANTS_CANCELLED, CLOSE,    |
   |                                   | COMMIT, CREATE,               |
   |                                   | CREATE_SESSION, DELEGPURGE,   |
   |                                   | DELEGRETURN,                  |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION,              |
   |                                   | EXCHANGE_ID, FREE_STATEID,    |
   |                                   | GETATTR, GETDEVICEINFO,       |
   |                                   | GETDEVICELIST,                |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |

that's where we're at for this part, and the table keeps going next time 🦋💜

okay the table is NOT done being iconic yet, keep scrolling 💅✨

   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SEQUENCE, SETATTR, SET_SSV,   |
   |                                   | TEST_STATEID, VERIFY,         |
   |                                   | WANT_DELEGATION, WRITE        |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_REQ_TOO_BIG               | ACCESS, BACKCHANNEL_CTL,      |
   |                                   | BIND_CONN_TO_SESSION,         |
   |                                   | CB_GETATTR, CB_LAYOUTRECALL,  |
   |                                   | CB_NOTIFY,                    |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG, CB_RECALL,     |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_RECALL_ANY,                |
   |                                   | CB_RECALL_SLOT, CB_SEQUENCE,  |
   |                                   | CB_WANTS_CANCELLED, CLOSE,    |
   |                                   | COMMIT, CREATE,               |
   |                                   | CREATE_SESSION, DELEGPURGE,   |
   |                                   | DELEGRETURN,                  |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION,              |
   |                                   | EXCHANGE_ID, FREE_STATEID,    |
   |                                   | GETATTR, GETDEVICEINFO,       |
   |                                   | GETDEVICELIST,                |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SEQUENCE, SETATTR, SET_SSV,   |
   |                                   | TEST_STATEID, VERIFY,         |
   |                                   | WANT_DELEGATION, WRITE        |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_RETRY_UNCACHED_REP        | ACCESS, BACKCHANNEL_CTL,      |
   |                                   | BIND_CONN_TO_SESSION,         |
   |                                   | CB_GETATTR, CB_LAYOUTRECALL,  |
   |                                   | CB_NOTIFY,                    |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG, CB_RECALL,     |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_RECALL_ANY,                |
   |                                   | CB_RECALL_SLOT, CB_SEQUENCE,  |
   |                                   | CB_WANTS_CANCELLED, CLOSE,    |
   |                                   | COMMIT, CREATE,               |
   |                                   | CREATE_SESSION, DELEGPURGE,   |
   |                                   | DELEGRETURN,                  |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION,              |
   |                                   | EXCHANGE_ID, FREE_STATEID,    |
   |                                   | GETATTR, GETDEVICEINFO,       |
   |                                   | GETDEVICELIST,                |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SEQUENCE, SETATTR, SET_SSV,   |
   |                                   | TEST_STATEID, VERIFY,         |
   |                                   | WANT_DELEGATION, WRITE        |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_ROFS                      | CREATE, LINK, LOCK, LOCKT,    |
   |                                   | OPEN, OPENATTR,               |
   |                                   | OPEN_DOWNGRADE, REMOVE,       |
   |                                   | RENAME, SETATTR, WRITE        |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_SAME                      | NVERIFY                       |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_SEQUENCE_POS              | CB_SEQUENCE, SEQUENCE         |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_SEQ_FALSE_RETRY           | CB_SEQUENCE, SEQUENCE         |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_SEQ_MISORDERED            | CB_SEQUENCE, CREATE_SESSION,  |
   |                                   | SEQUENCE                      |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_SERVERFAULT               | ACCESS,                       |
   |                                   | BIND_CONN_TO_SESSION,         |
   |                                   | CB_GETATTR, CB_NOTIFY,        |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG, CB_RECALL,     |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_WANTS_CANCELLED, CLOSE,    |
   |                                   | COMMIT, CREATE,               |
   |                                   | CREATE_SESSION, DELEGPURGE,   |
   |                                   | DELEGRETURN,                  |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION,              |
   |                                   | EXCHANGE_ID, FREE_STATEID,    |
   |                                   | GETATTR, GETDEVICEINFO,       |
   |                                   | GETDEVICELIST,                |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKU, LOOKUP, LOOKUPP,       |
   |                                   | NVERIFY, OPEN, OPENATTR,      |
   |                                   | OPEN_DOWNGRADE, PUTFH,        |
   |                                   | PUTPUBFH, PUTROOTFH, READ,    |
   |                                   | READDIR, READLINK,            |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SETATTR, TEST_STATEID,        |
   |                                   | VERIFY, WANT_DELEGATION,      |
   |                                   | WRITE                         |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_SHARE_DENIED              | OPEN                          |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_STALE                     | ACCESS, CLOSE, COMMIT,        |
   |                                   | CREATE, DELEGRETURN,          |
   |                                   | GETATTR, GETFH,               |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, READ, READDIR,         |
   |                                   | READLINK, RECLAIM_COMPLETE,   |
   |                                   | REMOVE, RENAME, RESTOREFH,    |
   |                                   | SAVEFH, SECINFO,              |
   |                                   | SECINFO_NO_NAME, SETATTR,     |
   |                                   | VERIFY, WANT_DELEGATION,      |
   |                                   | WRITE                         |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_STALE_CLIENTID            | CREATE_SESSION,               |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION               |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_SYMLINK                   | COMMIT, LAYOUTCOMMIT, LINK,   |
   |                                   | LOCK, LOCKT, LOOKUP,          |
   |                                   | LOOKUPP, OPEN, READ, WRITE    |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_TOOSMALL                  | CREATE_SESSION,               |
   |                                   | GETDEVICEINFO, LAYOUTGET,     |
   |                                   | READDIR                       |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_TOO_MANY_OPS              | ACCESS, BACKCHANNEL_CTL,      |
   |                                   | BIND_CONN_TO_SESSION,         |
   |                                   | CB_GETATTR, CB_LAYOUTRECALL,  |
   |                                   | CB_NOTIFY,                    |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG, CB_RECALL,     |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_RECALL_ANY,                |
   |                                   | CB_RECALL_SLOT, CB_SEQUENCE,  |
   |                                   | CB_WANTS_CANCELLED, CLOSE,    |
   |                                   | COMMIT, CREATE,               |
   |                                   | CREATE_SESSION, DELEGPURGE,   |
   |                                   | DELEGRETURN,                  |
   |                                   | DESTROY_CLIENTID,             |

   |                                   | DESTROY_SESSION,              |
   |                                   | EXCHANGE_ID, FREE_STATEID,    |
   |                                   | GETATTR, GETDEVICEINFO,       |
   |                                   | GETDEVICELIST,                |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SEQUENCE, SETATTR, SET_SSV,   |
   |                                   | TEST_STATEID, VERIFY,         |
   |                                   | WANT_DELEGATION, WRITE        |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_UNKNOWN_LAYOUTTYPE        | CB_LAYOUTRECALL,              |
   |                                   | GETDEVICEINFO,                |
   |                                   | GETDEVICELIST, LAYOUTCOMMIT,  |
   |                                   | LAYOUTGET, LAYOUTRETURN,      |
   |                                   | NVERIFY, SETATTR, VERIFY      |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_UNSAFE_COMPOUND           | CREATE, OPEN, OPENATTR        |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_WRONGSEC                  | LINK, LOOKUP, LOOKUPP, OPEN,  |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | RENAME, RESTOREFH             |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_WRONG_CRED                | CLOSE, CREATE_SESSION,        |
   |                                   | DELEGPURGE, DELEGRETURN,      |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION,              |
   |                                   | FREE_STATEID, LAYOUTCOMMIT,   |
   |                                   | LAYOUTRETURN, LOCK, LOCKT,    |
   |                                   | LOCKU, OPEN_DOWNGRADE,        |
   |                                   | RECLAIM_COMPLETE              |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_WRONG_TYPE                | CB_LAYOUTRECALL,              |
   |                                   | CB_PUSH_DELEG, COMMIT,        |
   |                                   | GETATTR, LAYOUTGET,           |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, NVERIFY, OPEN,         |
   |                                   | OPENATTR, READ, READLINK,     |
   |                                   | RECLAIM_COMPLETE, SETATTR,    |
   |                                   | VERIFY, WANT_DELEGATION,      |
   |                                   | WRITE                         |
   +-----------------------------------+-------------------------------+
   | NFS4ERR_XDEV                      | LINK, RENAME                  |
   +-----------------------------------+-------------------------------+

             Table 14: Errors and the Operations That Use Them

16. NFSv4.1 Procedures ✨

Okay bestie, both procedures, NULL and COMPOUND, MUST be implemented. No skipping either one, no cap! 💅

16.1. Procedure 0: NULL - No Operation 🌸

16.1.1. ARGUMENTS

   void;

16.1.2. RESULTS

   void;

16.1.3. DESCRIPTION

So NULL is literally the procedure that does absolutely nothing, and that's the whole point 😭🫶 It takes a void argument and sends back a void response, no extra functionality, nothing. Because it's so simple, people sometimes use it to measure how much overhead it takes to process a request, which is iconic. The vibe is that the server SHOULD make sure it doesn't do any unnecessary work while handling this procedure. Lowkey the most minimal thing ever. 🦋

16.1.4. ERRORS

None. Zero drama, zero errors, pure main character energy as a no-op 👑

16.2. Procedure 1: COMPOUND - Compound Operations 🎀

16.2.1. ARGUMENTS

Okay so COMPOUND is where the magic happens, and this is the enum of every operation number you can bundle together. It's giving a whole menu of ops 🍓

   enum nfs_opnum4 {
    OP_ACCESS              = 3,
    OP_CLOSE               = 4,
    OP_COMMIT              = 5,
    OP_CREATE              = 6,
    OP_DELEGPURGE          = 7,
    OP_DELEGRETURN         = 8,
    OP_GETATTR             = 9,
    OP_GETFH               = 10,
    OP_LINK                = 11,
    OP_LOCK                = 12,
    OP_LOCKT               = 13,
    OP_LOCKU               = 14,
    OP_LOOKUP              = 15,
    OP_LOOKUPP             = 16,
    OP_NVERIFY             = 17,
    OP_OPEN                = 18,
    OP_OPENATTR            = 19,
    OP_OPEN_CONFIRM        = 20, /* Mandatory not-to-implement */
    OP_OPEN_DOWNGRADE      = 21,
    OP_PUTFH               = 22,
    OP_PUTPUBFH            = 23,
    OP_PUTROOTFH           = 24,
    OP_READ                = 25,
    OP_READDIR             = 26,
    OP_READLINK            = 27,
    OP_REMOVE              = 28,
    OP_RENAME              = 29,
    OP_RENEW               = 30, /* Mandatory not-to-implement */
    OP_RESTOREFH           = 31,
    OP_SAVEFH              = 32,
    OP_SECINFO             = 33,
    OP_SETATTR             = 34,
    OP_SETCLIENTID         = 35, /* Mandatory not-to-implement */
    OP_SETCLIENTID_CONFIRM = 36, /* Mandatory not-to-implement */
    OP_VERIFY              = 37,
    OP_WRITE               = 38,
    OP_RELEASE_LOCKOWNER   = 39, /* Mandatory not-to-implement */

   /* new operations for NFSv4.1 */

    OP_BACKCHANNEL_CTL     = 40,
    OP_BIND_CONN_TO_SESSION = 41,
    OP_EXCHANGE_ID         = 42,
    OP_CREATE_SESSION      = 43,
    OP_DESTROY_SESSION     = 44,
    OP_FREE_STATEID        = 45,
    OP_GET_DIR_DELEGATION  = 46,
    OP_GETDEVICEINFO       = 47,
    OP_GETDEVICELIST       = 48,
    OP_LAYOUTCOMMIT        = 49,
    OP_LAYOUTGET           = 50,
    OP_LAYOUTRETURN        = 51,
    OP_SECINFO_NO_NAME     = 52,
    OP_SEQUENCE            = 53,
    OP_SET_SSV             = 54,
    OP_TEST_STATEID        = 55,
    OP_WANT_DELEGATION     = 56,
    OP_DESTROY_CLIENTID    = 57,
    OP_RECLAIM_COMPLETE    = 58,
    OP_ILLEGAL             = 10044
   };

   union nfs_argop4 switch (nfs_opnum4 argop) {
    case OP_ACCESS:        ACCESS4args opaccess;
    case OP_CLOSE:         CLOSE4args opclose;
    case OP_COMMIT:        COMMIT4args opcommit;
    case OP_CREATE:        CREATE4args opcreate;
    case OP_DELEGPURGE:    DELEGPURGE4args opdelegpurge;
    case OP_DELEGRETURN:   DELEGRETURN4args opdelegreturn;
    case OP_GETATTR:       GETATTR4args opgetattr;
    case OP_GETFH:         void;
    case OP_LINK:          LINK4args oplink;
    case OP_LOCK:          LOCK4args oplock;
    case OP_LOCKT:         LOCKT4args oplockt;
    case OP_LOCKU:         LOCKU4args oplocku;
    case OP_LOOKUP:        LOOKUP4args oplookup;
    case OP_LOOKUPP:       void;
    case OP_NVERIFY:       NVERIFY4args opnverify;
    case OP_OPEN:          OPEN4args opopen;
    case OP_OPENATTR:      OPENATTR4args opopenattr;

    /* Not for NFSv4.1 */
    case OP_OPEN_CONFIRM:  OPEN_CONFIRM4args opopen_confirm;

    case OP_OPEN_DOWNGRADE:
                           OPEN_DOWNGRADE4args opopen_downgrade;

    case OP_PUTFH:         PUTFH4args opputfh;
    case OP_PUTPUBFH:      void;
    case OP_PUTROOTFH:     void;
    case OP_READ:          READ4args opread;
    case OP_READDIR:       READDIR4args opreaddir;
    case OP_READLINK:      void;
    case OP_REMOVE:        REMOVE4args opremove;
    case OP_RENAME:        RENAME4args oprename;

    /* Not for NFSv4.1 */
    case OP_RENEW:         RENEW4args oprenew;

    case OP_RESTOREFH:     void;
    case OP_SAVEFH:        void;
    case OP_SECINFO:       SECINFO4args opsecinfo;
    case OP_SETATTR:       SETATTR4args opsetattr;

    /* Not for NFSv4.1 */
    case OP_SETCLIENTID: SETCLIENTID4args opsetclientid;

    /* Not for NFSv4.1 */
    case OP_SETCLIENTID_CONFIRM: SETCLIENTID_CONFIRM4args
                                   opsetclientid_confirm;
    case OP_VERIFY:        VERIFY4args opverify;
    case OP_WRITE:         WRITE4args opwrite;

    /* Not for NFSv4.1 */
    case OP_RELEASE_LOCKOWNER:
                           RELEASE_LOCKOWNER4args
                           oprelease_lockowner;

    /* Operations new to NFSv4.1 */
    case OP_BACKCHANNEL_CTL:
                           BACKCHANNEL_CTL4args opbackchannel_ctl;

    case OP_BIND_CONN_TO_SESSION:
                           BIND_CONN_TO_SESSION4args
                           opbind_conn_to_session;

    case OP_EXCHANGE_ID:   EXCHANGE_ID4args opexchange_id;

    case OP_CREATE_SESSION:
                           CREATE_SESSION4args opcreate_session;

    case OP_DESTROY_SESSION:
                           DESTROY_SESSION4args opdestroy_session;

    case OP_FREE_STATEID:  FREE_STATEID4args opfree_stateid;

    case OP_GET_DIR_DELEGATION:
                           GET_DIR_DELEGATION4args
                                   opget_dir_delegation;

    case OP_GETDEVICEINFO: GETDEVICEINFO4args opgetdeviceinfo;
    case OP_GETDEVICELIST: GETDEVICELIST4args opgetdevicelist;
    case OP_LAYOUTCOMMIT:  LAYOUTCOMMIT4args oplayoutcommit;
    case OP_LAYOUTGET:     LAYOUTGET4args oplayoutget;
    case OP_LAYOUTRETURN:  LAYOUTRETURN4args oplayoutreturn;

    case OP_SECINFO_NO_NAME:
                           SECINFO_NO_NAME4args opsecinfo_no_name;

    case OP_SEQUENCE:      SEQUENCE4args opsequence;
    case OP_SET_SSV:       SET_SSV4args opset_ssv;
    case OP_TEST_STATEID:  TEST_STATEID4args optest_stateid;

    case OP_WANT_DELEGATION:
                           WANT_DELEGATION4args opwant_delegation;

    case OP_DESTROY_CLIENTID:
                           DESTROY_CLIENTID4args
                                   opdestroy_clientid;

    case OP_RECLAIM_COMPLETE:
                           RECLAIM_COMPLETE4args
                                   opreclaim_complete;

    /* Operations not new to NFSv4.1 */
    case OP_ILLEGAL:       void;
   };

   struct COMPOUND4args {
           utf8str_cs      tag;
           uint32_t        minorversion;
           nfs_argop4      argarray<>;
   };

16.2.2. RESULTS ✨

Okay bestie, this is the results menu 💅 Every possible per-operation result gets its own slot in this union, so the server can hand back whatever each operation produced 🎀

union nfs_resop4 switch (nfs_opnum4 resop) {
 case OP_ACCESS:        ACCESS4res opaccess;
 case OP_CLOSE:         CLOSE4res opclose;
 case OP_COMMIT:        COMMIT4res opcommit;
 case OP_CREATE:        CREATE4res opcreate;
 case OP_DELEGPURGE:    DELEGPURGE4res opdelegpurge;
 case OP_DELEGRETURN:   DELEGRETURN4res opdelegreturn;
 case OP_GETATTR:       GETATTR4res opgetattr;
 case OP_GETFH:         GETFH4res opgetfh;
 case OP_LINK:          LINK4res oplink;
 case OP_LOCK:          LOCK4res oplock;
 case OP_LOCKT:         LOCKT4res oplockt;
 case OP_LOCKU:         LOCKU4res oplocku;
 case OP_LOOKUP:        LOOKUP4res oplookup;
 case OP_LOOKUPP:       LOOKUPP4res oplookupp;
 case OP_NVERIFY:       NVERIFY4res opnverify;
 case OP_OPEN:          OPEN4res opopen;
 case OP_OPENATTR:      OPENATTR4res opopenattr;
 /* Not for NFSv4.1 */
 case OP_OPEN_CONFIRM:  OPEN_CONFIRM4res opopen_confirm;

 case OP_OPEN_DOWNGRADE:
                        OPEN_DOWNGRADE4res
                                opopen_downgrade;

 case OP_PUTFH:         PUTFH4res opputfh;
 case OP_PUTPUBFH:      PUTPUBFH4res opputpubfh;
 case OP_PUTROOTFH:     PUTROOTFH4res opputrootfh;
 case OP_READ:          READ4res opread;
 case OP_READDIR:       READDIR4res opreaddir;
 case OP_READLINK:      READLINK4res opreadlink;
 case OP_REMOVE:        REMOVE4res opremove;
 case OP_RENAME:        RENAME4res oprename;
 /* Not for NFSv4.1 */
 case OP_RENEW:         RENEW4res oprenew;
 case OP_RESTOREFH:     RESTOREFH4res oprestorefh;
 case OP_SAVEFH:        SAVEFH4res opsavefh;
 case OP_SECINFO:       SECINFO4res opsecinfo;
 case OP_SETATTR:       SETATTR4res opsetattr;
 /* Not for NFSv4.1 */
 case OP_SETCLIENTID: SETCLIENTID4res opsetclientid;

 /* Not for NFSv4.1 */
 case OP_SETCLIENTID_CONFIRM:
                        SETCLIENTID_CONFIRM4res
                                opsetclientid_confirm;
 case OP_VERIFY:        VERIFY4res opverify;
 case OP_WRITE:         WRITE4res opwrite;

 /* Not for NFSv4.1 */
 case OP_RELEASE_LOCKOWNER:
                        RELEASE_LOCKOWNER4res
                                oprelease_lockowner;

 /* Operations new to NFSv4.1 */
 case OP_BACKCHANNEL_CTL:
                        BACKCHANNEL_CTL4res
                                opbackchannel_ctl;

 case OP_BIND_CONN_TO_SESSION:
                        BIND_CONN_TO_SESSION4res
                                 opbind_conn_to_session;

 case OP_EXCHANGE_ID:   EXCHANGE_ID4res opexchange_id;

 case OP_CREATE_SESSION:
                        CREATE_SESSION4res
                                opcreate_session;

 case OP_DESTROY_SESSION:
                        DESTROY_SESSION4res
                                opdestroy_session;

 case OP_FREE_STATEID:  FREE_STATEID4res
                                opfree_stateid;

 case OP_GET_DIR_DELEGATION:
                        GET_DIR_DELEGATION4res
                                opget_dir_delegation;

 case OP_GETDEVICEINFO: GETDEVICEINFO4res
                                opgetdeviceinfo;

 case OP_GETDEVICELIST: GETDEVICELIST4res
                                opgetdevicelist;

 case OP_LAYOUTCOMMIT:  LAYOUTCOMMIT4res oplayoutcommit;
 case OP_LAYOUTGET:     LAYOUTGET4res oplayoutget;
 case OP_LAYOUTRETURN:  LAYOUTRETURN4res oplayoutreturn;

 case OP_SECINFO_NO_NAME:
                        SECINFO_NO_NAME4res
                                opsecinfo_no_name;

 case OP_SEQUENCE:      SEQUENCE4res opsequence;
 case OP_SET_SSV:       SET_SSV4res opset_ssv;
 case OP_TEST_STATEID:  TEST_STATEID4res optest_stateid;

 case OP_WANT_DELEGATION:
                        WANT_DELEGATION4res
                                opwant_delegation;

 case OP_DESTROY_CLIENTID:
                        DESTROY_CLIENTID4res
                                opdestroy_clientid;

 case OP_RECLAIM_COMPLETE:
                        RECLAIM_COMPLETE4res
                                opreclaim_complete;

 /* Operations not new to NFSv4.1 */
 case OP_ILLEGAL:       ILLEGAL4res opillegal;
};

struct COMPOUND4res {
        nfsstat4        status;
        utf8str_cs      tag;
        nfs_resop4      resarray<>;
};

16.2.3. DESCRIPTION 🦋

Alright, so COMPOUND is the procedure that squishes one or more NFSv4 operations into a single RPC request 📦 The server reads each operation in order. If an operation runs and its status is NFS4_OK, the server moves on to the next one in the COMPOUND. It keeps going until either there's nothing left to run or some operation comes back with a status other than NFS4_OK 🌸

Here's a plot twist: while processing a COMPOUND, the server might realize it doesn't have enough resources to run some or all of the operations in the sequence. For the full tea on that, check out Section 2.10.6.4 💜

The server will usually pick between two ways of decoding the client's request. The first is the classic one-pass XDR decode, and if that hits an XDR decoding error, the RPC XDR decode error is what gets returned. The second is a two-step vibe: first decode the basic COMPOUND request, then XDR decode the individual operations. The juiciest part of that is decoding attributes 🙈 With this method, the server might hit an XDR decode error during the second pass, and if it does, it returns NFS4ERR_BADXDR to say "decode error, bestie" 😭

The COMPOUND arguments have a minorversion field. For NFSv4.1, that value is 1. If the server gets a COMPOUND with a minorversion it doesn't support, the server MUST return an error of NFS4ERR_MINOR_VERS_MISMATCH and a zero-length resultdata array. Not negotiable 💀

The COMPOUND results include a status field. If the results array is non-zero in length, this status has to match the status of the last operation that ran inside the COMPOUND. So if some operation fails, the status is the same error value being returned for that failed operation 🔥

Operations zero and one aren't defined for COMPOUND. Operation 2 is undefined too, reserved for future use with minor versioning ✨ If the server gets an operation array containing operation 2 and the minorversion is zero, it returns NFS4ERR_OP_ILLEGAL, which the next paragraph explains. If the operation array contains operation 2, the minorversion is non-zero, and the server doesn't support that minor version, the server returns NFS4ERR_MINOR_VERS_MISMATCH. So NFS4ERR_MINOR_VERS_MISMATCH takes priority over every other error, period 👑

The server might also get a request with an operation that's less than the first legal operation (OP_ACCESS) or greater than the last legal one (OP_RELEASE_LOCKOWNER). In that case, the response encodes OP_ILLEGAL instead of the illegal opcode from the request 🧚 The status field in the ILLEGAL return results gets set to NFS4ERR_OP_ILLEGAL, and the COMPOUND's return results are NFS4ERR_OP_ILLEGAL too.

The definition of the tag in the request is up to the implementor 🎀 It can be used to summarize what the Compound request contains, which helps packet-sniffers and engineers debugging implementations. The tag value in the response SHOULD be the same value that was in the request. That goes for the tag field of the CB_COMPOUND procedure too 💖

16.2.3.1. Current Filehandle and Stateid 🧁

The COMPOUND procedure gives you a simple environment for running the operations the client asks for. The first two things relate to the filehandle, and the last two relate to the current stateid 🌈

16.2.3.1.1. Current Filehandle 📂

The current and saved filehandles show up all over the protocol. Most operations quietly use the current filehandle as an argument, and lots of them set the current filehandle as part of their results. Mixing client-chosen sequences of operations with current and saved filehandle arguments and results gives you a lot of flexibility ✨ The best or easiest example of current filehandle usage looks like this:

      PUTFH fh1              {fh1}
      LOOKUP "compA"         {fh2}
      GETATTR                {fh2}
      LOOKUP "compB"         {fh3}
      GETATTR                {fh3}
      LOOKUP "compC"         {fh4}
      GETATTR                {fh4}
      GETFH

                                Figure 2

Here, the PUTFH (Section 18.19) operation explicitly sets the current filehandle value, and the result of each LOOKUP sets the current filehandle to the resulting file system object 🦋 The client can also slip GETATTR operations in, using the current filehandle as the argument.

PUTROOTFH (Section 18.21) and PUTPUBFH (Section 18.20) also set the current filehandle 👑 So the example above could swap "PUTFH fh1" for PUTROOTFH or PUTPUBFH with no filehandle argument and get the same result, as long as "compA" sits directly below the root of the namespace.

There's also a saved filehandle hanging around with the current one 💜 The current filehandle gets set as a result of operations like LOOKUP, but the saved filehandle has to be set directly with the SAVEFH operation. SAVEFH copies the current filehandle value into the saved value. The saved filehandle gets paired with the current filehandle for the LINK and RENAME operations 🔥 RESTOREFH copies the saved filehandle value back into the current filehandle value, so the saved one works like a sort of "scratch" area for the client's series of operations 🫶

16.2.3.1.2. Current Stateid ✨

okay so with NFSv4.1, the COMPOUND environment got two new bestie-slots: a current stateid and a saved stateid 💅 That means stateids can get passed between operations! There are no changes to the syntax at all, just to what a few operations mean 🦋

A "current stateid" is the stateid associated with the current filehandle 🎀 It can only be changed by an operation that modifies the current filehandle or returns a stateid. If an operation returns a stateid, it MUST set the current stateid to that returned value 😭 If an operation sets the current filehandle but doesn't return a stateid, the current stateid MUST be set to the all-zeros special stateid, aka (seqid, other) = (0, 0) 🫶 And if an operation uses a stateid as an argument but doesn't return one, the current stateid MUST NOT be changed 🙈

Here's the glow-up in action: PUTFH, PUTROOTFH, and PUTPUBFH change the current server state from {ocfh, (osid)} to {cfh, (0, 0)}, while LOCK changes it from {cfh, (osid} to {cfh, (nsid)}. Operations like LOOKUP, which turn a current filehandle and component name into a new current filehandle, also change the current state to {0, 0} 💖 And SAVEFH and RESTOREFH save and restore the current filehandle and the current stateid as a set 👑

The following is the most common case: a simple READ with a normal stateid. You can see PUTFH setting the current stateid to (0, 0), and then the READ with (sid1) leaves it alone 🌸

PUTFH fh1                             - -> {fh1, (0, 0)}
READ (sid1), 0, 1024      {fh1, (0, 0)} -> {fh1, (0, 0)}

Figure 3

Next up, we do an OPEN with the root filehandle, which generates stateid (sid1). The next operation specifies READ with an argument stateid where (seqid, other) equal (1, 0), but the current stateid set by the previous operation is the one actually used when the operation is evaluated. This is what allows correct interaction with any existing, potentially conflicting, locks 🔥

PUTROOTFH                             - -> {fh1, (0, 0)}
OPEN "compA"              {fh1, (0, 0)} -> {fh2, (sid1)}
READ (1, 0), 0, 1024      {fh2, (sid1)} -> {fh2, (sid1)}
CLOSE (1, 0)              {fh2, (sid1)} -> {fh2, (sid2)}

Figure 4

This one is kind of a sequel to the last example 🦋 It passes the stateid sid2, generated by LOCK, along to the next READ. This lets the client explicitly wrap a single I/O operation in a lock and its matching stateid, so it stays correct next to other client locks 🧚 It also shows how SAVEFH and RESTOREFH can save a filehandle and stateid and reuse them later, passing them in as the current filehandle and stateid for a READ 💜

PUTFH fh1                             - -> {fh1, (0, 0)}
LOCK 0, 1024, (sid1)      {fh1, (sid1)} -> {fh1, (sid2)}
READ (1, 0), 0, 1024      {fh1, (sid2)} -> {fh1, (sid2)}
LOCKU 0, 1024, (1, 0)     {fh1, (sid2)} -> {fh1, (sid3)}
SAVEFH                    {fh1, (sid3)} -> {fh1, (sid3)}

PUTFH fh2                 {fh1, (sid3)} -> {fh2, (0, 0)}
WRITE (1, 0), 0, 1024     {fh2, (0, 0)} -> {fh2, (0, 0)}

RESTOREFH                 {fh2, (0, 0)} -> {fh1, (sid3)}
READ (1, 0), 1024, 1024   {fh1, (sid3)} -> {fh1, (sid3)}

Figure 5

The final example is a big no-no 🙅‍♀️ It shows a disallowed use of the current stateid. The client is trying to sneakily pass the anonymous special stateid, (0,0), into the READ implicitly. The server MUST return NFS4ERR_BAD_STATEID in the reply to that READ 💀

PUTFH fh1                             - -> {fh1, (0, 0)}
READ (1, 0), 0, 1024      {fh1, (0, 0)} -> NFS4ERR_BAD_STATEID

Figure 6

16.2.4. ERRORS 🙈

COMPOUND will of course return every error that each operation on the fore channel can return (see Table 12). But if COMPOUND returns zero operations, the error it returns has nothing to do with an error from an operation, obviously 😅 Here's the list of errors COMPOUND can return when it processes zero operations:

    +==============================+==================================+
    | Error                        | Notes                            |
    +==============================+==================================+
    | NFS4ERR_BADCHAR              | The tag argument has a character |
    |                              | the replier does not support.    |
    +------------------------------+----------------------------------+
    | NFS4ERR_BADXDR               |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_DELAY                |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_INVAL                | The tag argument is not in UTF-8 |
    |                              | encoding.                        |
    +------------------------------+----------------------------------+
    | NFS4ERR_MINOR_VERS_MISMATCH  |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_SERVERFAULT          |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_TOO_MANY_OPS         |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_REP_TOO_BIG          |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_REP_TOO_BIG_TO_CACHE |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_REQ_TOO_BIG          |                                  |
    +------------------------------+----------------------------------+

Table 15: COMPOUND Error Returns

17. Operations: REQUIRED, RECOMMENDED, or OPTIONAL 🎀

okay so this section is the cheat sheet 📝 It sums up the operations in NFSv4.1 and whether each one is REQUIRED, RECOMMENDED, or OPTIONAL to implement, or MUST NOT be implemented. The MUST NOT designation is reserved for operations that were defined in NFSv4.0 and MUST NOT be implemented in NFSv4.1 🙅‍♀️

For the most part, the REQUIRED, RECOMMENDED, or OPTIONAL label on client-sent operations is about the server implementation 💅 The client is generally expected to implement whatever its operating environment needs. For example, a read-only NFSv4.1 client has no reason to implement WRITE, so it isn't required to 🌈

For callback operations sent by the server, the REQUIRED or OPTIONAL label applies to both the client and the server 🫶 Usually, the client gets to choose whether to create the backchannel and whether to send the fore channel operations that will be the catalyst for the server sending callbacks. The one partial exception is CB_RECALL_SLOT: the only way a client can avoid supporting it is by not creating a backchannel 🧚

This is only a summary, so there are subtleties that aren't shown here. So if you have a question about implementation requirements, you need to read the operation descriptions themselves, along with the other relevant explanatory text in this specification 📖

The abbreviations used in the second and third columns of the table are:

REQ: REQUIRED to implement
REC: RECOMMEND to implement
OPT: OPTIONAL to implement
MNI: MUST NOT implement

For the NFSv4.1 features that are OPTIONAL, the operations supporting those features are OPTIONAL too, and the server would return NFS4ERR_NOTSUPP if the client used those operations 😭 If an OPTIONAL feature is supported, a set of operations related to that feature may become REQUIRED to implement. The third column of the table tells you the feature(s) and whether the operation is REQUIRED or OPTIONAL when that feature is supported ✨

The OPTIONAL features and their abbreviations are:

pNFS: Parallel NFS
FDELG: File Delegations
DDELG: Directory Delegations

Okay bestie, here's the big operations scoreboard 💅✨ Every operation gets a REQ, REC, OPT, or MNI label, plus its feature tag and where it's defined. Let's gooo 🦋

    +======================+=============+============+===============+
    | Operation            | REQ, REC,   | Feature    | Definition    |
    |                      | OPT, or MNI | (REQ, REC, |               |
    |                      |             | or OPT)    |               |
    +======================+=============+============+===============+
    | ACCESS               | REQ         |            | Section 18.1  |
    +----------------------+-------------+------------+---------------+
    | BACKCHANNEL_CTL      | REQ         |            | Section 18.33 |
    +----------------------+-------------+------------+---------------+
    | BIND_CONN_TO_SESSION | REQ         |            | Section 18.34 |
    +----------------------+-------------+------------+---------------+
    | CLOSE                | REQ         |            | Section 18.2  |
    +----------------------+-------------+------------+---------------+
    | COMMIT               | REQ         |            | Section 18.3  |
    +----------------------+-------------+------------+---------------+
    | CREATE               | REQ         |            | Section 18.4  |
    +----------------------+-------------+------------+---------------+
    | CREATE_SESSION       | REQ         |            | Section 18.36 |
    +----------------------+-------------+------------+---------------+
    | DELEGPURGE           | OPT         | FDELG      | Section 18.5  |
    |                      |             | (REQ)      |               |
    +----------------------+-------------+------------+---------------+
    | DELEGRETURN          | OPT         | FDELG,     | Section 18.6  |
    |                      |             | DDELG,     |               |
    |                      |             | pNFS (REQ) |               |
    +----------------------+-------------+------------+---------------+
    | DESTROY_CLIENTID     | REQ         |            | Section 18.50 |
    +----------------------+-------------+------------+---------------+
    | DESTROY_SESSION      | REQ         |            | Section 18.37 |
    +----------------------+-------------+------------+---------------+
    | EXCHANGE_ID          | REQ         |            | Section 18.35 |
    +----------------------+-------------+------------+---------------+
    | FREE_STATEID         | REQ         |            | Section 18.38 |
    +----------------------+-------------+------------+---------------+
    | GETATTR              | REQ         |            | Section 18.7  |
    +----------------------+-------------+------------+---------------+
    | GETDEVICEINFO        | OPT         | pNFS (REQ) | Section 18.40 |
    +----------------------+-------------+------------+---------------+
    | GETDEVICELIST        | OPT         | pNFS (OPT) | Section 18.41 |
    +----------------------+-------------+------------+---------------+
    | GETFH                | REQ         |            | Section 18.8  |
    +----------------------+-------------+------------+---------------+
    | GET_DIR_DELEGATION   | OPT         | DDELG      | Section 18.39 |
    |                      |             | (REQ)      |               |
    +----------------------+-------------+------------+---------------+
    | LAYOUTCOMMIT         | OPT         | pNFS (REQ) | Section 18.42 |
    +----------------------+-------------+------------+---------------+
    | LAYOUTGET            | OPT         | pNFS (REQ) | Section 18.43 |
    +----------------------+-------------+------------+---------------+
    | LAYOUTRETURN         | OPT         | pNFS (REQ) | Section 18.44 |
    +----------------------+-------------+------------+---------------+
    | LINK                 | OPT         |            | Section 18.9  |
    +----------------------+-------------+------------+---------------+
    | LOCK                 | REQ         |            | Section 18.10 |
    +----------------------+-------------+------------+---------------+
    | LOCKT                | REQ         |            | Section 18.11 |
    +----------------------+-------------+------------+---------------+
    | LOCKU                | REQ         |            | Section 18.12 |
    +----------------------+-------------+------------+---------------+
    | LOOKUP               | REQ         |            | Section 18.13 |
    +----------------------+-------------+------------+---------------+
    | LOOKUPP              | REQ         |            | Section 18.14 |
    +----------------------+-------------+------------+---------------+
    | NVERIFY              | REQ         |            | Section 18.15 |
    +----------------------+-------------+------------+---------------+
    | OPEN                 | REQ         |            | Section 18.16 |
    +----------------------+-------------+------------+---------------+
    | OPENATTR             | OPT         |            | Section 18.17 |
    +----------------------+-------------+------------+---------------+
    | OPEN_CONFIRM         | MNI         |            | N/A           |
    +----------------------+-------------+------------+---------------+
    | OPEN_DOWNGRADE       | REQ         |            | Section 18.18 |
    +----------------------+-------------+------------+---------------+
    | PUTFH                | REQ         |            | Section 18.19 |
    +----------------------+-------------+------------+---------------+
    | PUTPUBFH             | REQ         |            | Section 18.20 |
    +----------------------+-------------+------------+---------------+
    | PUTROOTFH            | REQ         |            | Section 18.21 |
    +----------------------+-------------+------------+---------------+
    | READ                 | REQ         |            | Section 18.22 |
    +----------------------+-------------+------------+---------------+
    | READDIR              | REQ         |            | Section 18.23 |
    +----------------------+-------------+------------+---------------+
    | READLINK             | OPT         |            | Section 18.24 |
    +----------------------+-------------+------------+---------------+
    | RECLAIM_COMPLETE     | REQ         |            | Section 18.51 |
    +----------------------+-------------+------------+---------------+
    | RELEASE_LOCKOWNER    | MNI         |            | N/A           |
    +----------------------+-------------+------------+---------------+
    | REMOVE               | REQ         |            | Section 18.25 |
    +----------------------+-------------+------------+---------------+
    | RENAME               | REQ         |            | Section 18.26 |
    +----------------------+-------------+------------+---------------+
    | RENEW                | MNI         |            | N/A           |
    +----------------------+-------------+------------+---------------+
    | RESTOREFH            | REQ         |            | Section 18.27 |
    +----------------------+-------------+------------+---------------+
    | SAVEFH               | REQ         |            | Section 18.28 |
    +----------------------+-------------+------------+---------------+
    | SECINFO              | REQ         |            | Section 18.29 |
    +----------------------+-------------+------------+---------------+
    | SECINFO_NO_NAME      | REC         | pNFS file  | Section       |
    |                      |             | layout     | 18.45,        |
    |                      |             | (REQ)      | Section 13.12 |
    +----------------------+-------------+------------+---------------+
    | SEQUENCE             | REQ         |            | Section 18.46 |
    +----------------------+-------------+------------+---------------+
    | SETATTR              | REQ         |            | Section 18.30 |
    +----------------------+-------------+------------+---------------+
    | SETCLIENTID          | MNI         |            | N/A           |
    +----------------------+-------------+------------+---------------+
    | SETCLIENTID_CONFIRM  | MNI         |            | N/A           |
    +----------------------+-------------+------------+---------------+
    | SET_SSV              | REQ         |            | Section 18.47 |
    +----------------------+-------------+------------+---------------+
    | TEST_STATEID         | REQ         |            | Section 18.48 |
    +----------------------+-------------+------------+---------------+
    | VERIFY               | REQ         |            | Section 18.31 |
    +----------------------+-------------+------------+---------------+
    | WANT_DELEGATION      | OPT         | FDELG      | Section 18.49 |
    |                      |             | (OPT)      |               |
    +----------------------+-------------+------------+---------------+
    | WRITE                | REQ         |            | Section 18.32 |
    +----------------------+-------------+------------+---------------+

                            Table 16: Operations

Table 16 is the whole lineup, so you can see where every operation lives and how much it matters 🌸💜

    +=========================+=============+============+============+
    | Operation               | REQ, REC,   | Feature    | Definition |
    |                         | OPT, or MNI | (REQ, REC, |            |
    |                         |             | or OPT)    |            |
    +=========================+=============+============+============+
    | CB_GETATTR              | OPT         | FDELG      | Section    |
    |                         |             | (REQ)      | 20.1       |
    +-------------------------+-------------+------------+------------+
    | CB_LAYOUTRECALL         | OPT         | pNFS (REQ) | Section    |
    |                         |             |            | 20.3       |
    +-------------------------+-------------+------------+------------+
    | CB_NOTIFY               | OPT         | DDELG      | Section    |
    |                         |             | (REQ)      | 20.4       |
    +-------------------------+-------------+------------+------------+
    | CB_NOTIFY_DEVICEID      | OPT         | pNFS (OPT) | Section    |
    |                         |             |            | 20.12      |
    +-------------------------+-------------+------------+------------+
    | CB_NOTIFY_LOCK          | OPT         |            | Section    |
    |                         |             |            | 20.11      |
    +-------------------------+-------------+------------+------------+
    | CB_PUSH_DELEG           | OPT         | FDELG      | Section    |
    |                         |             | (OPT)      | 20.5       |
    +-------------------------+-------------+------------+------------+
    | CB_RECALL               | OPT         | FDELG,     | Section    |
    |                         |             | DDELG,     | 20.2       |
    |                         |             | pNFS (REQ) |            |
    +-------------------------+-------------+------------+------------+
    | CB_RECALL_ANY           | OPT         | FDELG,     | Section    |
    |                         |             | DDELG,     | 20.6       |
    |                         |             | pNFS (REQ) |            |
    +-------------------------+-------------+------------+------------+
    | CB_RECALL_SLOT          | REQ         |            | Section    |
    |                         |             |            | 20.8       |
    +-------------------------+-------------+------------+------------+
    | CB_RECALLABLE_OBJ_AVAIL | OPT         | DDELG,     | Section    |
    |                         |             | pNFS (REQ) | 20.7       |
    +-------------------------+-------------+------------+------------+
    | CB_SEQUENCE             | OPT         | FDELG,     | Section    |
    |                         |             | DDELG,     | 20.9       |
    |                         |             | pNFS (REQ) |            |
    +-------------------------+-------------+------------+------------+
    | CB_WANTS_CANCELLED      | OPT         | FDELG,     | Section    |
    |                         |             | DDELG,     | 20.10      |
    |                         |             | pNFS (REQ) |            |
    +-------------------------+-------------+------------+------------+

                       Table 17: Callback Operations

18. NFSv4.1 Operations ✨

18.1. Operation 3: ACCESS - Check Access Rights 🔑

18.1.1. ARGUMENTS

okay so here are the ACCESS args, the access bits are all just flags you OR together 🎀

   const ACCESS4_READ      = 0x00000001;
   const ACCESS4_LOOKUP    = 0x00000002;
   const ACCESS4_MODIFY    = 0x00000004;
   const ACCESS4_EXTEND    = 0x00000008;
   const ACCESS4_DELETE    = 0x00000010;
   const ACCESS4_EXECUTE   = 0x00000020;

   struct ACCESS4args {
           /* CURRENT_FH: object */
           uint32_t        access;
   };

18.1.2. RESULTS

bestie, the response is giving two fields, supported and access, and only the good path (NFS4_OK) carries the resok 🦋

   struct ACCESS4resok {
           uint32_t        supported;
           uint32_t        access;
   };

   union ACCESS4res switch (nfsstat4 status) {
    case NFS4_OK:
            ACCESS4resok   resok4;
    default:
            void;
   };

18.1.3. DESCRIPTION ✨

okay bestie, ACCESS is the op that answers "can this user actually do the thing?" 🔐 The user is whoever's identified in the RPC request's credentials 🧚, and the object is whatever the current filehandle points at. The client packs the rights it wants checked into the bit mask access, and the server checks them. If you get NFS4_OK back, you get two masks: supported (the rights the server can verify reliably 🫶) and access (what this user has on that filehandle). On success, the current filehandle keeps its value 💅

Note that the reply's supported and access fields MUST NOT contain more values than the client originally put in the request's access field. ngl this is so strict 😭 Like, if you only ask for ACCESS4_READ and the server supports it, the server MUST NOT set anything beyond ACCESS4_READ in supported, even if it could have reliably checked other stuff 🙈

The reply's access field MUST NOT contain more values than the supported field. 💜

The results are necessarily advisory 🔮 An NFS4_OK status plus the right bit set does NOT mean that access will still be allowed later, because the server can revoke access rights at any time. not me being dramatic, but this is the lore 💀

The following access permissions may be requested:

ACCESS4_READ — Read data from a file or read a directory 📖
ACCESS4_LOOKUP — Look up a name in a directory (no meaning for non-directory objects) 🔍
ACCESS4_MODIFY — Rewrite existing file data or modify existing directory entries ✏️
ACCESS4_EXTEND — Write new data or add directory entries ➕
ACCESS4_DELETE — Delete an existing directory entry 🗑️
ACCESS4_EXECUTE — Execute a regular file (no meaning for a directory) ⚡

On success, the current filehandle retains its value. 🎀

ACCESS4_EXECUTE is a challenging semantic to implement, because NFS gives you remote file access, not remote execution 😵‍💫 This leads to the following:

Whether or not a regular file is executable ought to be the responsibility of the NFS client and not the server. And yet the ACCESS operation is specified to seemingly require a server to own that responsibility 🤡
When a client executes a regular file, it has to read the file from the server. Strictly speaking, the server should not allow the client to read a file being executed unless the user has read permissions on the file. Requiring explicit read permissions on executable files in order to access them over NFS is not going to be acceptable to some users and storage administrators. Historically, NFS servers have allowed a user to READ a file if the user has execute access to the file. 🥹

As a practical example, the UNIX specification [60] says that an implementation claiming conformance to UNIX may indicate in the access() programming interface's result that a privileged user has execute rights, even if no execute permission bits are set on the regular file's attributes. It's also possible to claim conformance to the UNIX specification and instead not indicate execute rights in that situation, which is true for some operating environments. Now suppose the operating environments of the client and server implement the access() semantics for privileged users differently, and the ACCESS operation implementations of the client and server follow their respective access() semantics. This can cause undesired behavior 🙃

Suppose the client's access() interface returns X_OK if the user is privileged and no execute permission bits are set on the regular file's attribute, but the server's access() interface does not return X_OK in that situation. Then the client will be unable to execute files stored on the NFS server that could be executed if stored on a non-NFS file system. 😩
Suppose the client's access() interface does not return X_OK if the user is privileged and no execute permission bits are set on the regular file's attribute, but the server's access() interface does return X_OK in that situation. Then:
- The client will be able to execute files stored on the NFS server that could be executed if stored on a non-NFS file system, unless the client's execution subsystem also checks for execute permission bits.
- Even if the execution subsystem is checking for execute permission bits, there are more potential issues. For example, suppose the client is invoking access() to build a "path search table" of all executable files in the user's "search path", where the path is a list of directories each containing executable files. Suppose there are two files in separate directories of the search path, with the same component name. In the first directory the file has no execute permission bits set, and in the second directory the file has execute bits set. The path search table will indicate that the first directory has the executable file, but the execute subsystem will fail to execute it. The command shell might fail to try the second file in the second directory. And even if it did, this is a potential performance issue. Clearly, the desired outcome for the client is for the path search table to not contain the first file. 🫠

To deal with the problems described above, the "smart client, stupid server" principle is used 🧠 The client owns overall responsibility for determining execute access and relies on the server to parse the execution permissions within the file's mode, acl, and dacl attributes. The rules for the client and server follow:

If the client is sending ACCESS in order to determine if the user can read the file, the client SHOULD set ACCESS4_READ in the request's access field. 📚
If the client's operating environment only grants execution to the user if the user has execute access according to the execute permissions in the mode, acl, and dacl attributes, then if the client wants to determine execute access, the client SHOULD send an ACCESS request with the ACCESS4_EXECUTE bit set in the request's access field. ⚡
If the client's operating environment grants execution to the user even if the user does not have execute access according to the execute permissions in the mode, acl, and dacl attributes, then if the client wants to determine execute access, it SHOULD send an ACCESS request with both the ACCESS4_EXECUTE and ACCESS4_READ bits set in the request's access field. This way, if any read or execute permission grants the user read or execute access (or if the server interprets the user as privileged), as indicated by the presence of ACCESS4_EXECUTE and/or ACCESS4_READ in the reply's access field, the client will be able to grant the user execute access to the file. 🎉
If the server supports execute permission bits, or some other method for denoting executability (e.g., the suffix of the name of the file might indicate execute), it MUST check only execute permissions, not read permissions, when determining whether or not the reply will have ACCESS4_EXECUTE set in the access field. The server MUST NOT also examine read permission bits when determining whether or not the reply will have ACCESS4_EXECUTE set in the access field. Even if the server's operating environment would grant execute access to the user (e.g., the user is privileged), the server MUST NOT reply with ACCESS4_EXECUTE set in the reply's access field unless there is at least one execute permission bit set in the mode, acl, or dacl attributes. In the case of acl and dacl, the "one execute permission bit" MUST be an ACE4_EXECUTE bit set in an ALLOW ACE. 🛡️
If the server does not support execute permission bits or some other method for denoting executability, it MUST NOT set ACCESS4_EXECUTE in the reply's supported and access fields. If the client set ACCESS4_EXECUTE in the ACCESS request's access field, and ACCESS4_EXECUTE is not set in the reply's supported field, then the client will have to send an ACCESS request with the ACCESS4_READ bit set in the request's access field. 🔁
If the server supports read permission bits, it MUST only check for read permissions in the mode, acl, and dacl attributes when it receives an ACCESS request with ACCESS4_READ set in the access field. The server MUST NOT also examine execute permission bits when determining whether the reply will have ACCESS4_READ set in the access field or not. 🧼

Note that if the ACCESS reply has ACCESS4_READ or ACCESS_EXECUTE set, then the user also has permissions to OPEN (Section 18.16) or READ (Section 18.22) the file. In other words, if the client sends an ACCESS request with the ACCESS4_READ and ACCESS_EXECUTE set in the access field (or two separate requests, one with ACCESS4_READ set and the other with ACCESS4_EXECUTE set), and the reply has just ACCESS4_EXECUTE set in the access field (or just one reply has ACCESS4_EXECUTE set), then the user has authorization to OPEN or READ the file. and that's the tea, periodt 🌈👑

18.1.4. IMPLEMENTATION ✨

okay bestie, first things first 🙈 clients can't just guess whether they're allowed to do something by peeking at the uid, gid, and mode fields in the file attributes, or by trying to read the ACL attribute. the server may do uid or gid mapping or add extra access-control restrictions, and it's also possible the server isn't even in the same ID space as the client. in these cases (and maybe others), the client can't reliably do an access check with only the current file attributes 💅

back in NFSv2, the only reliable way to find out if an operation was allowed was to just try it and see if it worked or failed 😭 but in NFSv4.1 the ACCESS operation lets the client ask the server whether one or more classes of operations are permitted 🦋 ACCESS exists so clients can check before doing a whole series of operations that would end in an access failure. the OPEN operation is also a spot where the server can verify access to the file object and hand that info back to the client. ACCESS is still super useful for directory operations, or when the UNIX interface access() is used on the client 🌸

the info the server sends back in an ACCESS reply isn't permanent, babe 🥺 it was correct at the exact moment the server did the checks, but not necessarily after that. the server can revoke access permission at any time, so don't get too attached 💀

the client should use the effective credentials of the user to build the authentication information in the ACCESS request used to determine access rights. it's the effective user and group credentials that get used in later READ and WRITE operations, so those are the ones that matter 👑

ngl the ACCESS4_DELETE bit is the messy one. lots of implementations don't directly support it 😭

here's the tea: operating systems like UNIX ignore the ACCESS4_DELETE bit if it's set on an access request for a non-directory object. on those systems, delete permission on a file is decided by the access permissions on the directory the file lives in, not by the file's own permissions 🙃 so the mask returned enumerating which access rights can be determined will have the ACCESS4_DELETE value set to 0. that's the server's way of saying "i couldn't check that particular access right" 🫶 and then the client will ignore the ACCESS4_DELETE bit in the access mask that comes back. iconic, honestly 🎀

18.2. Operation 4: CLOSE - Close File 🌈

18.2.1. ARGUMENTS

struct CLOSE4args {
        /* CURRENT_FH: object */
        seqid4          seqid;
        stateid4        open_stateid;
};

18.2.2. RESULTS

union CLOSE4res switch (nfsstat4 status) {
 case NFS4_OK:
         stateid4       open_stateid;
 default:
         void;
};

18.2.3. DESCRIPTION

okay so CLOSE is the one that lets go of share reservations for the regular or named attribute file that's sitting in the current filehandle 🦋 the share reservations and other state info that get released at the server because of this CLOSE are only the ones tied to the stateid you hand over. any state from other OPENs? totally untouched, they're not invited to this party 🎉

now the locks situation, bestie 💅

if byte-range locks are held, the client SHOULD release all locks before sending a CLOSE. the server MAY free all outstanding locks on CLOSE, but some servers may not support closing a file that still has byte-range locks held 😬 and the server MUST return failure if any locks would exist after the CLOSE. no exceptions, no vibes-based exceptions 🙅‍♀️

the argument seqid MAY have any value, and the server MUST ignore seqid. literally doesn't matter what you put there 🤷‍♀️

on success, the current filehandle retains its value ✨

the server MAY require that the combination of principal, security flavor, and, if applicable, GSS mechanism that sent the OPEN request also be the one to CLOSE the file. this might not be possible if credentials for the principal are no longer available 😭 the server MAY allow the machine credential or SSV credential (see Section 18.35) to send CLOSE too, so there's a backup plan 🫶

18.2.4. IMPLEMENTATION

ok so even though CLOSE hands back a stateid, that stateid is not useful to the client and should be treated as deprecated 💀 CLOSE "shuts down" the state associated with all OPENs for the file by a single open-owner. like we said above, CLOSE will either release all file-locking state or return an error. so the stateid it returns isn't useful for the operations that follow 🙈 to help find any uses of this stateid by clients, the server SHOULD return the invalid special stateid (the "other" value is zero and the "seqid" field is NFS4_UINT32_MAX, see Section 8.2.3). that's the sneaky tripwire, honestly iconic 🧚

and one more thing: a CLOSE operation may make delegations grantable where they weren't before 🍓 servers may choose to respond immediately if there are pending delegation want requests, or they may respond to the situation at a later time. your call, server 👑

18.3. Operation 5: COMMIT - Commit Cached Data 💜

18.3.1. ARGUMENTS

struct COMMIT4args {
        /* CURRENT_FH: file */
        offset4         offset;
        count4          count;
};

18.3.2. RESULTS

struct COMMIT4resok {
        verifier4       writeverf;
};

union COMMIT4res switch (nfsstat4 status) {
 case NFS4_OK:
         COMMIT4resok   resok4;
 default:
         void;
};

18.3.3. DESCRIPTION

the COMMIT operation forces or flushes uncommitted, modified data out to stable storage for the file in the current filehandle 🌊 the data being flushed is whatever was previously written with one or more WRITE operations whose committed field in their results was set to UNSTABLE4.

the offset says where in the file the flush should begin 📍 an offset value of zero means start flushing from the very beginning of the file. the count says how many bytes of data to flush 🧮 and if the count is zero, the flush goes from the offset all the way to the end of the file. big energy 🌟

the server returns a write verifier when the COMMIT finishes successfully 🫶 the write verifier is how the client figures out if the server restarted between the initial WRITE operations and the COMMIT. the client does this by comparing the write verifier returned from the initial WRITE operations with the one returned by the COMMIT operation 🕵️‍♀️ the server must vary the value of the write verifier at each server event or instantiation that may lead to a loss of uncommitted data. most commonly that's when the server gets restarted, but other events at the server can cause uncommitted data loss too 😬

on success, the current filehandle retains its value ✨

18.3.4. IMPLEMENTATION

okay, the COMMIT operation is basically the cousin of the POSIX fsync() [22] system interface, which syncs a file's state with the disk (file data and metadata get flushed to disk or stable storage) 🧠 COMMIT does the same thing for a client, flushing any unsynchronized data and metadata on the server to the server's disk or stable storage for the specified file. like fsync(), there might be some modified data or no modified data to synchronize at all 🤷‍♀️ the data may have already been synced by the server's normal periodic buffer synchronization activity. COMMIT should return NFS4_OK, unless there's been an unexpected error 🎀

COMMIT is not exactly the same as fsync() though! 💅 the client can flush just a range of the file, which is most likely triggered by a buffer-reclamation scheme on the client before the file has been completely written 🌸

the server side of COMMIT is pretty simple, bestie 🧁 if the server gets a full file COMMIT request (starting at offset zero with count zero), it should do the equivalent of applying fsync() to the entire file. otherwise, it should arrange for the modified data in the range given by offset and count to be flushed to stable storage. in both cases, any metadata associated with the file must be flushed to stable storage before returning ✅ and it's not an error for there to be nothing to flush on the server. that just means the data and metadata that needed flushing were already flushed, or got lost during the last server failure 💀

the client side is a little more complex, no cap 🌀 there are two reasons a client would want to commit a client buffer to stable storage. the first is that the client wants to reuse a buffer. in that case, the offset and count of the buffer get sent to the server in the COMMIT request 📦 the server then flushes any modified data based on that offset and count, and flushes any modified metadata associated with the file. then it returns the status of the flush and the write verifier. the second reason is a full file flush, like what might happen at close. the client gathers all of the buffers for this file that contain uncommitted data, does the COMMIT with an offset of zero and count of zero, and then frees all of those buffers 🧹 any other dirty buffers get sent to the server in the normal fashion.

bestie, this next part is so important 😭 after a buffer is written by the client with WRITE and the committed field in the WRITE result is UNSTABLE4, the buffer must be considered as modified by the client until it's been flushed via a COMMIT operation, or written via a WRITE operation with the committed field in the result set to FILE_SYNC4 or DATA_SYNC4. this stops the buffer from getting freed and reused before the data can be flushed to stable storage on the server 🛑

when a response comes back from either a WRITE or a COMMIT and it has a write verifier that's different from the one the server returned before, the client will need to retransmit all of the buffers containing uncommitted data to the server 😱 how to do that is up to the implementor 🛠️ if there's only one buffer of interest, it should be sent in a WRITE request with the FILE_SYNC4 stable parameter. if there's more than one buffer, it might be worthwhile to retransmit all of them in WRITE operations with the stable parameter set to UNSTABLE4, and then retransmit the COMMIT operation to flush all of the data on the server to stable storage 🚀 however, if the server repeatably returns a verifier from COMMIT that differs from the one returned by WRITE, the only way to ensure progress is to retransmit all of the buffers with WRITE requests using the FILE_SYNC4 stable parameter 🫠

the above description applies to page-cache-based systems as well as buffer-cache-based systems 🌈 in the former, the virtual memory system will need to be modified instead of the buffer cache.

18.4. Operation 6: CREATE - Create a Non-Regular File Object 🦋

18.4.1. ARGUMENTS

union createtype4 switch (nfs_ftype4 type) {
 case NF4LNK:
        linktext4 linkdata;
 case NF4BLK:
 case NF4CHR:
        specdata4 devdata;
 case NF4SOCK:
 case NF4FIFO:
 case NF4DIR:
        void;
 default:
        void;  /* server should return NFS4ERR_BADTYPE */
};

struct CREATE4args {
        /* CURRENT_FH: directory for creation */
        createtype4     objtype;
        component4      objname;
        fattr4          createattrs;
};

18.4.2. RESULTS 💅

Okay so the reply is a union keyed on status 💖. If the status is NFS4_OK, you get back a CREATE4resok with the change info (cinfo) and the attrset (the attributes that were set). For any other status, there's just void. 🎀

   struct CREATE4resok {
           change_info4    cinfo;
           bitmap4         attrset;        /* attributes set */
   };

   union CREATE4res switch (nfsstat4 status) {
    case NFS4_OK:
            /* new CURRENTFH: created object */
            CREATE4resok resok4;
    default:
            void;
   };

18.4.3. DESCRIPTION ✨

Okay bestie, this is the big one 😭. CREATE makes a file object that isn't an ordinary file (think directories, symlinks, and so on) inside a directory, under a given name. If you want a regular file or a named attribute, OPEN is the one you MUST use, no exceptions 🙅‍♀️.

The current filehandle has to be a directory, meaning an object of type NF4DIR. If it's an attribute directory (type NF4ATTRDIR), you get NFS4ERR_WRONG_TYPE. If it's any other type of object, you get NFS4ERR_NOTDIR. Rude 💀.

The objname is the name for the new object, and objtype decides what kind of object gets made: a directory, a symlink, etc. If the type you ask for is an ordinary file, a named attribute, or a named attribute directory, you get NFS4ERR_BADTYPE 🙈.

If something with that same name is already in the directory, the server returns NFS4ERR_EXIST. Iconic behavior, no notes 🌸.

For the directory where the new object was created, the server sends back change_info4 information in cinfo. The atomic field of that data type tells you whether the before and after change attributes were obtained atomically with respect to the file object creation. Very detail-oriented, I'm obsessed 🤓.

If objname has a length of zero, or if objname doesn't follow the UTF-8 definition, the server returns NFS4ERR_INVAL 🚫.

Once it works, the current filehandle gets replaced by the new object's filehandle 🦋.

The createattrs field sets the starting attributes for the object. It can include any writable attribute that's valid for that object type. When the operation succeeds, the server sends the client an attribute mask showing which attributes it actually managed to set 💜.

Now the ownership drama. If createattrs has neither the owner attribute nor an ACL with an ACE for the owner, and the server's file system both supports and requires an owner attribute (or owner ACE), then the server MUST derive the owner (or the owner ACE). Usually that comes from the principal in the RPC credentials of the call, but the server's operating environment or file system semantics might call for a different way of deriving it 🔍. The same goes for the group: if createattrs has neither the group attribute nor a group ACE, and the file system both supports and requires a group attribute (or group ACE), the server MUST derive the group attribute (or the corresponding owner ACE) for the file. This could come from the RPC call's credentials, like the group principal if the credentials include one (such as with AUTH_SYS). It could also come from the group identifier tied to the principal in the credentials (for example, POSIX systems have a user database [23] that has a group identifier for every user identifier). It could be inherited from the directory the object is created in, or whatever else the server's operating environment or file system semantics dictate 🌈. This applies to the OPEN operation too, periodt 💅.

Flip side: the client might put an owner attribute, group attribute, or ACL in createattrs that the principal in the RPC call's credentials doesn't have permission to create files for. In that case, the error is NFS4ERR_PERM 😬. This also applies to the OPEN operation.

Now the delegation part, which is kind of a lot 🥺. If the current filehandle is a directory that another client holds a directory delegation for, then the delegation MUST be recalled, unless the delegation is set up so the situation can be fixed by sending a notification instead. And the CREATE operation MUST NOT proceed until that delegation is returned or revoked. Unless it happens super fast, you'll get one or more NFS4ERR_DELAY errors for requests made while the delegation is still outstanding ⏳.

When the current filehandle is a directory with one or more directory delegations, and those delegations asked for these notifications, a NOTIFY4_ADD_ENTRY gets generated as a result of this operation 🔔.

Last thing: if the capability FSCHARSET_CAP4_ALLOWS_ONLY_UTF8 is set (Section 14.4) and a symbolic link is being created, then the content of the symbolic link MUST be in UTF-8 encoding 🔤.

18.4.4. IMPLEMENTATION 🧚

If the client wants to set attribute values after the create, you can add a SETATTR operation to the COMPOUND request so the right attributes get set. Easy peasy 🍓.

18.5. Operation 7: DELEGPURGE - Purge Delegations Awaiting Recovery 🌸

18.5.1. ARGUMENTS

   struct DELEGPURGE4args {
           clientid4       clientid;
   };

18.5.2. RESULTS

   struct DELEGPURGE4res {
           nfsstat4        status;
   };

18.5.3. DESCRIPTION 🧹

DELEGPURGE purges all of the delegations waiting for recovery for a given client. It's handy for clients that don't commit their delegation info to stable storage. It tells the server that conflicting requests don't need to be held up while it waits for delegation info to be recovered 🙌.

The client is NOT specified by the clientid field of the request. The client SHOULD set the client field to zero, and the server MUST ignore the clientid field. Instead, the server MUST derive the client ID from the value of the session ID in the arguments of the SEQUENCE operation that comes right before DELEGPURGE in the COMPOUND request 🔑.

Clients that record delegation info on their own stable storage should use DELEGPURGE. After the client recovers all the delegations it knows about, it should immediately send a DELEGPURGE operation. That tells the server no more delegations for this client will be recovered, so the server can free resources and stop delaying other clients whose requests conflict with the unrecovered delegations 🧠. The set of delegations the server knows about and the set the client knows about might not match. The reason is that after a request that resulted in a delegation, the client could fail before it both received the delegation and committed it to its stable storage 😵‍💫.

The server MAY support DELEGPURGE. But if it doesn't, it MUST NOT support CLAIM_DELEGATE_PREV and MUST NOT support CLAIM_DELEG_PREV_FH 🚫.

18.6. Operation 8: DELEGRETURN - Return Delegation 🦋

18.6.1. ARGUMENTS

   struct DELEGRETURN4args {
           /* CURRENT_FH: delegated object */
           stateid4        deleg_stateid;
   };

18.6.2. RESULTS

   struct DELEGRETURN4res {
           nfsstat4        status;
   };

18.6.3. DESCRIPTION 👋

DELEGRETURN gives back the delegation represented by the current filehandle and stateid. That's the whole vibe 💅.

Delegations can be returned voluntarily (before the server recalls them) or when they get recalled. Either way, the client must properly propagate any state changes made under the context of the delegation to the server before returning it. No skipping that step, bestie 📦.

The server MAY require that the principal, security flavor, and, if applicable, the GSS mechanism combination that acquired the delegation is also the one that sends DELEGRETURN on the file. That might not be possible if the credentials for that principal aren't around anymore 🥲. The server MAY allow the machine credential or SSV credential (see Section 18.35) to send DELEGRETURN instead 🤖.

18.7. Operation 9: GETATTR - Get Attributes 🔍

18.7.1. ARGUMENTS

   struct GETATTR4args {
           /* CURRENT_FH: object */
           bitmap4         attr_request;
   };

18.7.2. RESULTS

   struct GETATTR4resok {
           fattr4          obj_attributes;
   };

   union GETATTR4res switch (nfsstat4 status) {
    case NFS4_OK:
            GETATTR4resok  resok4;
    default:
            void;
   };

18.7.3. DESCRIPTION 📋

GETATTR gets the attributes for the file system object the current filehandle points to. The client sets a bit in the bitmap argument for each attribute value it wants back. The server replies with an attribute bitmap showing which values it was able to return. That will include every attribute the client asked for that the server supports on the target file system. The bitmap is followed by the attribute values, ordered from the lowest attribute number up ✨.

The server MUST return a value for each attribute the client requests, as long as the server supports that attribute on the target file system. If the server doesn't support a particular attribute on the target file system, it MUST NOT return that attribute value and MUST NOT set the attribute bit in the result bitmap. If the server supports an attribute on the target but can't get its value, the server MUST return an error. In that case, no attribute values come back at all 😱.

Absent file systems should be treated as supporting a very small set of attributes, as described in Section 11.4.1, even if more attributes were supported back when the file system was present 🫥.

All servers MUST support the REQUIRED attributes as specified in Section 5.6, for all file systems, except absent file systems. Non-negotiable 🫶.

On success, the current filehandle keeps its value 🌟.

18.7.4. IMPLEMENTATION 🧩

Picture this: there's an OPEN_DELEGATE_WRITE delegation held by another client for the file in question, and size and/or change are among the attributes being asked about. The server has two choices. First, it can get the real current values of those attributes from the client holding the delegation, using the CB_GETATTR callback. Second, the server can recall the delegation in question, especially when the delegated client isn't responding 📞.

The GETATTR MUST NOT proceed until one of the following happens:

The requested attribute values come back in the response to CB_GETATTR.
The OPEN_DELEGATE_WRITE delegation is returned.
The OPEN_DELEGATE_WRITE delegation is revoked.

Unless one of those happens very quickly, you'll get one or more NFS4ERR_DELAY errors while a delegation is outstanding ⏳.

18.8. Operation 10: GETFH - Get Current Filehandle 🎀

18.8.1. ARGUMENTS

   /* CURRENT_FH: */
   void;

18.8.2. RESULTS

   struct GETFH4resok {
           nfs_fh4         object;
   };

   union GETFH4res switch (nfsstat4 status) {
    case NFS4_OK:
           GETFH4resok     resok4;
    default:
           void;
   };

18.8.3. DESCRIPTION 🪪

GETFH returns the current filehandle value. Simple, but iconic 💖.

On success, the current filehandle keeps its value.

Like Section 2.10.6.4 explains, GETFH is REQUIRED or RECOMMENDED to immediately follow certain operations. Servers are free to reject those operations if the client doesn't put GETFH in the request the way it's REQUIRED or RECOMMENDED to. And Section 18.16.4.1 gives more reasons why GETFH MUST follow OPEN 🧐.

18.8.4. IMPLEMENTATION 🧭

Operations that change the current filehandle, like LOOKUP or CREATE, don't automatically send back the new filehandle as a result. So if a client needs to look up a directory entry and get its filehandle, it needs a request like this:

      PUTFH (directory filehandle)

      LOOKUP (entry name)

      GETFH

18.9. Operation 11: LINK - Create Link to a File 🔗

18.9.1. ARGUMENTS ✨

LINK takes just one argument, and the rest comes from the filehandles 🧚

struct LINK4args {
        /* SAVED_FH: source object */
        /* CURRENT_FH: target directory */
        component4      newname;
};

18.9.2. RESULTS 🎀

Here's what comes back when LINK works 💖

struct LINK4resok {
        change_info4    cinfo;
};

union LINK4res switch (nfsstat4 status) {
 case NFS4_OK:
         LINK4resok resok4;
 default:
         void;
};

18.9.3. DESCRIPTION 💅

okay so LINK makes an extra newname for the file the saved filehandle points to (that's the one you stash with SAVEFH 🦋), and it drops that name into the directory the current filehandle is pointing at. The existing file and the target directory must reside within the same file system on the server, bestie. If all goes well, the current filehandle stays right where it is, pointing at the target directory ✨. And if something already lives in the target directory under the name newname, the server must return NFS4ERR_EXIST 💀.

For the target directory, the server sends back change_info4 info in cinfo. The atomic field of that change_info4 data type is how the server says whether the before and after change attributes were grabbed atomically with respect to the link creation 🍓.

Also, if newname has a length of zero, or if newname doesn't follow the UTF-8 definition, you get NFS4ERR_INVAL back 🙈.

18.9.4. IMPLEMENTATION 🔥

Okay, implementation time, and this part is kind of a lot 😭

The server MAY impose restrictions on the LINK operation, so LINK might not be allowed when the file is open, or when that open was done by particular protocols, or with particular options or access modes. When LINK gets rejected for that kind of reason, the error NFS4ERR_FILE_OPEN is returned 🫶.

Now here's the tricky bit. If a server does have restrictions like that, and those restrictions include NFSv4 opens that would stop a link from succeeding, then the server needs to recall any delegations that could be hiding the existence of those opens 🕵️‍♀️. Why? Because when a client holds a delegation, the server might not have an accurate picture of the opens for that client, since the client can do OPENs and CLOSEs locally 👀. The LINK operation must be delayed only until a definitive result can be obtained. For example, say there are multiple delegations, and one of them sets up an open that would block the link. Given the server's semantics, NFS4ERR_FILE_OPEN may be returned to the caller as soon as that one delegation comes back, without waiting for the other delegations to be returned ⭐. Similarly, if those opens aren't tied to any delegations, NFS4ERR_FILE_OPEN can be returned right away, with no delegation recall being done at all 🌸.

If the current filehandle points to a directory that another client holds a directory delegation for, then the delegation MUST be recalled, and the operation can't succeed until that delegation is returned or revoked. The only exception is when the situation can be fixed by just sending a notification instead, so keep that in mind 🧠. Unless it all happens super quickly, one or more NFS4ERR_DELAY errors will be sent back to any requests made while the delegation is still outstanding 😩.

When the current filehandle is a directory with one or more directory delegations, and those delegations ask for notifications, the server doesn't recall anything. Instead, a NOTIFY4_ADD_ENTRY gets generated as the result of the LINK operation 🦋.

If the current file system supports the numlinks attribute, and other clients have delegations to the file being linked, then those delegations MUST be recalled, and the LINK operation MUST NOT proceed until every one of them is returned or revoked 🚫. Again, unless it's super quick, one or more NFS4ERR_DELAY errors will be returned to requests made while the delegation is outstanding 💜.

Changes to any property of the "hard" linked files show up in all of the linked files, which is honestly so iconic 👑. When a link gets made to a file, the attributes for that file should have a numlinks value that's one greater than it was before the LINK operation ✨.

Let's talk about "file and the target directory must reside within the same file system on the server." That basically means the fsid fields in the attributes for the objects are the same 🍓. If they live on different file systems, you get NFS4ERR_XDEV. Some servers might also send that error if there's an internal partitioning of a file system that the LINK operation would break 🙈.

On some servers, "." and ".." aren't allowed as newname values, and if you send them, NFS4ERR_BADNAME comes back 😭.

Now named attributes. When the current filehandle designates a named attribute directory, and the object being linked (the saved filehandle) isn't a named attribute for that same object, the error NFS4ERR_XDEV MUST be returned 🚨. And when the saved filehandle designates a named attribute, but the current filehandle isn't the right named attribute directory, NFS4ERR_XDEV MUST also be returned 🚨.

When the current filehandle designates a named attribute directory, and the object being linked (the saved filehandle) is a named attribute inside that directory, the server may return NFS4ERR_NOTSUPP 🌈.

If newname is already linked to the file the saved filehandle represents, the server will return NFS4ERR_EXIST 💅.

One last thing: symbolic links aren't made with LINK. They're created with the CREATE operation 🧚.

18.10. Operation 12: LOCK - Create Lock 🔒

18.10.1. ARGUMENTS ✨

LOCK is where the locking drama really starts 🎭. The locker4 union handles two cases: either the lock-owner is brand new for this open file, or it already has lock state going 💫.

/*
 * For LOCK, transition from open_stateid and lock_owner
 * to a lock stateid.
 */
struct open_to_lock_owner4 {
        seqid4          open_seqid;
        stateid4        open_stateid;
        seqid4          lock_seqid;
        lock_owner4     lock_owner;
};

/*
 * For LOCK, existing lock stateid continues to request new
 * file lock for the same lock_owner and open_stateid.
 */
struct exist_lock_owner4 {
        stateid4        lock_stateid;
        seqid4          lock_seqid;
};

union locker4 switch (bool new_lock_owner) {
 case TRUE:
        open_to_lock_owner4     open_owner;
 case FALSE:
        exist_lock_owner4       lock_owner;
};

/*
 * LOCK/LOCKT/LOCKU: Record lock management
 */
struct LOCK4args {
        /* CURRENT_FH: file */
        nfs_lock_type4  locktype;
        bool            reclaim;
        offset4         offset;
        length4         length;
        locker4         locker;
};

18.10.2. RESULTS 🎀

Here are the possible outcomes. You get a stateid if it works, the details of the conflicting lock if it doesn't, or just a bare status otherwise 💖.

struct LOCK4denied {
        offset4         offset;
        length4         length;
        nfs_lock_type4  locktype;
        lock_owner4     owner;
};

struct LOCK4resok {
        stateid4        lock_stateid;
};

union LOCK4res switch (nfsstat4 status) {
 case NFS4_OK:
         LOCK4resok     resok4;
 case NFS4ERR_DENIED:
         LOCK4denied    denied;
 default:
         void;
};

18.10.3. DESCRIPTION 💜

So LOCK requests a byte-range lock for the byte range set by the offset and length parameters, with the lock type from the locktype parameter 🦋. If it's a reclaim request, the reclaim parameter will be TRUE ✨.

You can lock bytes in a file even if those bytes aren't actually allocated to the file yet 🌸. To lock from a specific offset all the way through end-of-file (no matter how long the file actually is), set length to NFS4_UINT64_MAX 🌈. The server MUST return NFS4ERR_INVAL for these combinations of length and offset 🚨:

Length is equal to zero.
Length is not equal to NFS4_UINT64_MAX, and the sum of length and offset exceeds NFS4_UINT64_MAX.

32-bit servers are servers that support locking for byte offsets that fit within 32 bits (that is, less than or equal to NFS4_UINT32_MAX) 🧠. If the client asks for a range that overlaps one or more bytes beyond offset NFS4_UINT32_MAX but doesn't end at offset NFS4_UINT64_MAX, then that 32-bit server MUST return the error NFS4ERR_BAD_RANGE 🙈.

If the server returns NFS4ERR_DENIED, it sends back the owner, offset, and length of a conflicting lock 😭.

The locker argument says which lock-owner goes with the LOCK operation 💅. The locker4 structure is a switched union that tells the server whether the client has already set up byte-range locking state for the current open file and lock-owner. If it has, the argument is just a stateid for the set of locks tied to that open file and lock-owner, along with a lock_seqid value that MAY be any value and MUST be ignored by the server 🫶. If no byte-range locking state exists yet, or the client doesn't have the stateid available, the argument instead holds the stateid of the open file the lock is going to be tied to, plus the lock-owner it's tied to 🧚. The open_to_lock_owner case covers the very first lock a lock-owner makes on a given open file, and it's the way to use the established open_stateid state to switch over to a lock stateid 🦋.

The following fields of the locker parameter MAY be set to any value by the client, and MUST be ignored by the server 🙈:

The clientid field of the lock_owner field of the open_owner field (locker.open_owner.lock_owner.clientid). The reason the server MUST ignore the clientid field is that the server MUST derive the client ID from the session ID from the SEQUENCE operation of the COMPOUND request 🔑.
The open_seqid and lock_seqid fields of the open_owner field (locker.open_owner.open_seqid and locker.open_owner.lock_seqid).
The lock_seqid field of the lock_owner field (locker.lock_owner.lock_seqid).

Here's a sneaky detail: the client ID showing up in a LOCK4denied structure is the actual client connected to the conflicting lock, whether that's the client ID tied to the current session or a different one 👀. So if the server returns NFS4ERR_DENIED, it MUST set the clientid field of the owner field of the denied field 💖.

If the current filehandle isn't an ordinary file, an error comes back to the client. If the current filehandle represents an object of type NF4DIR, NFS4ERR_ISDIR is returned 📁. If the current filehandle designates a symbolic link, NFS4ERR_SYMLINK is returned 🔗. In all other cases, NFS4ERR_WRONG_TYPE is returned 💀.

On success, the current filehandle keeps its value ✨.

18.10.4. IMPLEMENTATION ✨

If the server can't figure out the exact offset and length of the conflicting byte-range lock, the same offset and length that were provided in the arguments should be returned in the denied results. 🎯

LOCK operations get permission checks and checks against the access type of the file, but the specific rights and modes needed for each kind of lock reflect how the server-exported file system works. The protocol doesn't specify them. For example, Windows 2000 allows a write lock on a file that's open for read access, while a POSIX-compliant system doesn't. 🙈

When the client sends a LOCK operation for a range the lock-owner has already locked (with the same or a different lock type), or for a sub-range of such a range, or for a byte-range that includes multiple locks already granted to that lock-owner (in whole or in part), and the server doesn't support that kind of locking (meaning it doesn't support POSIX locking semantics), the server will return the error NFS4ERR_LOCK_RANGE. 😭 In that case, the client may return an error, or it may emulate the required operations. To do the emulation, it can only use LOCK for ranges that don't include any bytes already locked by that lock-owner, plus LOCKU on the locks held by that lock-owner (specifying an exactly matching range and type). 💅

It's a similar story for upgrades (changing from a READ_LT lock to a WRITE_LT lock) and downgrades (changing from a WRITE_LT lock to a READ_LT lock). If the server doesn't support that kind of lock on an existing byte-range lock, the server will return NFS4ERR_LOCK_NOTSUPP. These operations may not perfectly reflect the required semantics when there are conflicting LOCK operations from other clients. Not perfect, bestie, but that's the deal. 🥲

When a client holds an OPEN_DELEGATE_WRITE delegation, it's assured that there are no opens by other clients. That means there can't be any conflicting LOCK operations from those clients. So the client may handle locking requests locally, without doing LOCK operations on the server. If it does that, it must be prepared to update the lock status on the server, by sending the appropriate LOCK and LOCKU operations before returning the delegation. Local handling, but make sure the server catches up before you give it back! 🧚

When one or more clients hold OPEN_DELEGATE_READ delegations, any LOCK operation where the server is implementing mandatory locking semantics MUST result in the recall of all such delegations. The LOCK operation may not be granted until all of those delegations are returned or revoked. Unless that happens super quickly, one or more NFS4ERR_DELAY errors will be returned to requests made while the delegation is still outstanding. Main character energy, but also a lot of waiting. ⏳🎀

18.11. Operation 13: LOCKT - Test for Lock 🔍

18.11.1. ARGUMENTS

struct LOCKT4args {
        /* CURRENT_FH: file */
        nfs_lock_type4  locktype;
        offset4         offset;
        length4         length;
        lock_owner4     owner;
};

18.11.2. RESULTS

union LOCKT4res switch (nfsstat4 status) {
 case NFS4ERR_DENIED:
         LOCK4denied    denied;
 case NFS4_OK:
         void;
 default:
         void;
};

18.11.3. DESCRIPTION

Okay so LOCKT is basically the "is anyone already holding this lock?" check. 👀 It tests the lock described in the arguments. If there's a conflicting lock, the owner, offset, length, and type of that conflicting lock get sent back. The owner field in the results includes the client ID of the owner of the conflicting lock, whether that's the client ID tied to the current session or a different client ID. If no lock is held, you get back nothing other than NFS4_OK. Lock types READ_LT and READW_LT are handled the same way: the conflicting lock test is done without regard to blocking or non-blocking. Same goes for WRITE_LT and WRITEW_LT. 🌸

The ranges are specified just like they are for LOCK. The NFS4ERR_INVAL and NFS4ERR_BAD_RANGE errors get returned under the same circumstances as they do for LOCK. ✨

The clientid field of the owner MAY be set to any value by the client and MUST be ignored by the server. The reason the server MUST ignore the clientid field is that the server MUST derive the client ID from the session ID in the SEQUENCE operation of the COMPOUND request. So the server basically does its own detective work and doesn't trust what's in that field. 🕵️‍♀️💖

If the current filehandle isn't an ordinary file, an error will be returned to the client. If the current filehandle represents an object of type NF4DIR, NFS4ERR_ISDIR is returned. If the current filehandle designates a symbolic link, NFS4ERR_SYMLINK is returned. In all other cases, NFS4ERR_WRONG_TYPE is returned. 🙅‍♀️

On success, the current filehandle keeps its value. 🫶

18.11.4. IMPLEMENTATION

If the server can't determine the exact offset and length of the conflicting lock, the same offset and length that were provided in the arguments should be returned in the denied results. 🎀

LOCKT uses a lock_owner4 rather than a stateid4 (which is what LOCK uses) to identify the owner. This is because the client doesn't have to open the file to test for the existence of a lock, so a stateid might not be available. Smart, right? 🧠

As noted in Section 18.10.4, some servers may return NFS4ERR_LOCK_RANGE to certain LOCK operations that otherwise wouldn't conflict but overlap ranges already granted to the current lock-owner. 🙃

The LOCKT operation's test for conflicting locks SHOULD exclude locks for the current lock-owner, so it should return NFS4_OK in such cases. Note that this means a server might return NFS4_OK to a LOCKT request even though a LOCK operation for the same range and lock-owner would fail with NFS4ERR_LOCK_RANGE. Yes, it's a little plot twist, but it's consistent. 🦋

When a client holds an OPEN_DELEGATE_WRITE delegation, it may choose (see Section 18.10.4) to handle LOCK requests locally. In that case, LOCKT requests will similarly be handled locally. 🌈

18.12. Operation 14: LOCKU - Unlock File 🔓

18.12.1. ARGUMENTS

struct LOCKU4args {
        /* CURRENT_FH: file */
        nfs_lock_type4  locktype;
        seqid4          seqid;
        stateid4        lock_stateid;
        offset4         offset;
        length4         length;
};

18.12.2. RESULTS

union LOCKU4res switch (nfsstat4 status) {
 case   NFS4_OK:
         stateid4       lock_stateid;
 default:
         void;
};

18.12.3. DESCRIPTION

LOCKU is the unlock move, and it unlocks the byte-range lock described by the parameters. 🔓✨ The client can set the locktype field to any value that's legal for the nfs_lock_type4 enumerated type, and the server MUST accept any legal value for locktype. Any legal value for locktype has no effect on whether LOCKU succeeds or fails. So the server can't be picky about it. 💅

The ranges are specified just like they are for LOCK. The NFS4ERR_INVAL and NFS4ERR_BAD_RANGE errors get returned under the same circumstances as they do for LOCK. 🌸

The seqid parameter MAY be any value, and the server MUST ignore it. Ignored, bestie! 🙈

On success, the current filehandle keeps its value. 🫶

The server MAY require that the principal, security flavor, and (if applicable) the GSS mechanism combination that sent a LOCK operation also be the one that sends LOCKU on the file. That might not be possible if credentials for the principal are no longer available, which is kind of a bummer. 😭 The server MAY allow the machine credential or SSV credential (see Section 18.35) to send LOCKU. 🔐

18.12.4. IMPLEMENTATION

If the area to be unlocked doesn't correspond exactly to a lock actually held by the lock-owner, the server may return the error NFS4ERR_LOCK_RANGE. This includes the case where the area isn't locked at all, where the area is a sub-range of the locked area, where it overlaps the locked area without matching exactly, or where the specified area includes multiple locks held by the lock-owner. All of these cases are allowed by POSIX locking [21] semantics. So if a client gets this error and wants support for such operations, it should simulate the operation: use LOCKU on ranges that match locks it actually holds, possibly followed by LOCK operations for the sub-ranges that aren't being unlocked. Little puzzle pieces, honestly iconic. 🧩💜

When a client holds an OPEN_DELEGATE_WRITE delegation, it may choose (see Section 18.10.4) to handle LOCK requests locally. In that case, LOCKU operations will similarly be handled locally. 🍓

18.13. Operation 15: LOOKUP - Lookup Filename 🔎

18.13.1. ARGUMENTS

struct LOOKUP4args {
        /* CURRENT_FH: directory */
        component4      objname;
};

18.13.2. RESULTS

struct LOOKUP4res {
        /* New CURRENT_FH: object */
        nfsstat4        status;
};

18.13.3. DESCRIPTION

LOOKUP is how you find a file system object, using the directory specified by the current filehandle. 🗺️ LOOKUP evaluates the component, and if the object exists, the current filehandle gets replaced with the component's filehandle. Easy peasy. ✨

If the component can't be evaluated, either because it doesn't exist or because the client doesn't have permission to evaluate it, then an error will be returned and the current filehandle stays unchanged. 🙈

If the component is a zero-length string, or if any component doesn't obey the UTF-8 definition, the error NFS4ERR_INVAL will be returned. Rude, but fair. 💀

18.13.4. IMPLEMENTATION

If the client wants a multi-component lookup, it can build a COMPOUND request like this one (and collect each filehandle along the way):

PUTFH  (directory filehandle)
LOOKUP "pub"
GETFH
LOOKUP "foo"
GETFH
LOOKUP "bar"
GETFH

Unlike NFSv3, NFSv4.1 lets LOOKUP requests cross mountpoints on the server. 🌍 The client can detect a mountpoint crossing by comparing the fsid attribute of the directory with the fsid attribute of the directory that was looked up. If the fsids are different, the new directory is a server mountpoint. UNIX clients that detect a mountpoint crossing will need to mount the server's file system. That needs to happen to keep the file object identity checking mechanisms that UNIX clients rely on working. Hold onto that, it matters. 🦋

Servers that limit NFS access to "shared" or "exported" file systems should provide a pseudo file system where the exported file systems can be integrated, so clients can browse the server's namespace. 🗂️ A client's view of a pseudo file system is limited to paths that lead to exported file systems. Peeking only where it's allowed, very respectful. 🫶

Note: previous versions of the protocol gave special meaning to the names "." and "..". NFSv4.1 assigns no special semantics to these names. To look up a parent directory, the LOOKUPP operator must be used. 📌

Note that this operation does not follow symbolic links. The client is responsible for all parsing of filenames, including filenames modified by symbolic links encountered during the lookup process. Client does the heavy lifting here. 💪

If the current filehandle supplied isn't a directory but a symbolic link, the error NFS4ERR_SYMLINK is returned. For all other non-directory file types, the error NFS4ERR_NOTDIR is returned. 🙅‍♀️

18.14. Operation 16: LOOKUPP - Lookup Parent Directory ⬆️

18.14.1. ARGUMENTS

/* CURRENT_FH: object */
void;

18.14.2. RESULTS

struct LOOKUPP4res {
        /* new CURRENT_FH: parent directory */
        nfsstat4        status;
};

18.14.3. DESCRIPTION 🎀

Okay so the current filehandle is supposed to be a regular directory or a named attribute directory 📁. LOOKUPP then sets the current filehandle to the filehandle of that directory's parent, bestie. If there's no parent, the server has to return NFS4ERR_NOENT 💀. So the server returns NFS4ERR_NOENT whenever the current filehandle is already at the root or top of the server's file tree 🌳.

LOOKUPP will also cross mountpoints, just like LOOKUP does 🚪✨.

If the current filehandle isn't a directory or a named attribute directory, you get NFS4ERR_NOTDIR 🙈.

Security flavor check time 🔐. If the requester's security flavor doesn't match the one configured for the parent directory, the server SHOULD return NFS4ERR_WRONGSEC in the LOOKUPP response (a future minor revision of NFSv4 may upgrade this to MUST) 🫶. BUT if the server does send that, it MUST support the SECINFO_NO_NAME operation (Section 18.45) 💖. That way the client can figure out the right security flavor without drama.

Last one! If the current filehandle is a named attribute directory that's associated with a file system object through OPENATTR (so, not a sub-directory of a named attribute directory), LOOKUPP SHOULD return the filehandle of that associated file system object 🦋.

18.14.4. IMPLEMENTATION 🧚

Heads up, this one's a little tricky 👀. Upward navigation out of named attribute directories is the thing to watch. Named attribute directories are basically detached from the namespace, and the client operating environment should represent that safely 🌸. LOOKUPP on a named attribute directory can hand back the filehandle of the associated file, and passing that along to applications can be unsafe, because lots of apps assume an object's parent is always a directory 😭.

So the client may want to hide the parent of named attribute directories (shown as .. in UNIX) 🙈, or treat the named attribute directory as its own parent, which is what UNIX typically does for the file system root directory 🔄✨.

18.15. Operation 17: NVERIFY - Verify Difference in Attributes 💅

18.15.1. ARGUMENTS

   struct NVERIFY4args {
           /* CURRENT_FH: object */
           fattr4          obj_attributes;
   };

18.15.2. RESULTS

   struct NVERIFY4res {
           nfsstat4        status;
   };

18.15.3. DESCRIPTION 💖

NVERIFY is used to prefix a sequence of operations 🔗. Those operations only run if one or more attributes have changed on some file system object 🔄. If ALL the attributes match, the error NFS4ERR_SAME MUST be returned 🙅‍♀️.

On success, the current filehandle keeps its value 🎀.

18.15.4. IMPLEMENTATION 🦋

This one's a total cache validation superpower 🧠✨. If the object's attributes changed, the operations that follow can grab fresh data for it. For example, to check whether a file has changed and get new data if it has:

         SEQUENCE
         PUTFH fh
         NVERIFY attrbits attrs
         READ 0 32767

Compare that to NFSv3, which would send a GETATTR in one request/reply round trip first, and then, if the attributes showed the client's cache was stale, send a READ in a second round trip 😩. NVERIFY lets you do it all in one go 🏃‍♀️💨.

If a RECOMMENDED attribute is specified in the NVERIFY operation and the server doesn't support that attribute for the file system object, the client gets the error NFS4ERR_ATTRNOTSUPP 🥲.

When the attribute rdattr_error or any set-only attribute (e.g., time_modify_set) is specified, the error NFS4ERR_INVAL is returned to the client. Don't include those, bestie 🚫.

18.16. Operation 18: OPEN - Open a Regular File 🌈

18.16.1. ARGUMENTS

   /*
    * Various definitions for OPEN
    */
   enum createmode4 {
           UNCHECKED4      = 0,
           GUARDED4        = 1,
           /* Deprecated in NFSv4.1. */
           EXCLUSIVE4      = 2,
           /*
            * New to NFSv4.1. If session is persistent,
            * GUARDED4 MUST be used.  Otherwise, use
            * EXCLUSIVE4_1 instead of EXCLUSIVE4.
            */
           EXCLUSIVE4_1    = 3
   };

   struct creatverfattr {
            verifier4      cva_verf;
            fattr4         cva_attrs;
   };

   union createhow4 switch (createmode4 mode) {
    case UNCHECKED4:
    case GUARDED4:
            fattr4         createattrs;
    case EXCLUSIVE4:
            verifier4      createverf;
    case EXCLUSIVE4_1:
            creatverfattr  ch_createboth;
   };

   enum opentype4 {
           OPEN4_NOCREATE  = 0,
           OPEN4_CREATE    = 1
   };

   union openflag4 switch (opentype4 opentype) {
    case OPEN4_CREATE:
            createhow4     how;
    default:
            void;
   };

   /* Next definitions used for OPEN delegation */
   enum limit_by4 {
           NFS_LIMIT_SIZE          = 1,
           NFS_LIMIT_BLOCKS        = 2
           /* others as needed */
   };

   struct nfs_modified_limit4 {
           uint32_t        num_blocks;
           uint32_t        bytes_per_block;
   };

   union nfs_space_limit4 switch (limit_by4 limitby) {
    /* limit specified as file size */
    case NFS_LIMIT_SIZE:
            uint64_t               filesize;
    /* limit specified by number of blocks */
    case NFS_LIMIT_BLOCKS:
            nfs_modified_limit4    mod_blocks;
   } ;

   /*
    * Share Access and Deny constants for open argument
    */
   const OPEN4_SHARE_ACCESS_READ   = 0x00000001;
   const OPEN4_SHARE_ACCESS_WRITE  = 0x00000002;
   const OPEN4_SHARE_ACCESS_BOTH   = 0x00000003;

   const OPEN4_SHARE_DENY_NONE     = 0x00000000;
   const OPEN4_SHARE_DENY_READ     = 0x00000001;
   const OPEN4_SHARE_DENY_WRITE    = 0x00000002;
   const OPEN4_SHARE_DENY_BOTH     = 0x00000003;


   /* new flags for share_access field of OPEN4args */
   const OPEN4_SHARE_ACCESS_WANT_DELEG_MASK        = 0xFF00;
   const OPEN4_SHARE_ACCESS_WANT_NO_PREFERENCE     = 0x0000;
   const OPEN4_SHARE_ACCESS_WANT_READ_DELEG        = 0x0100;
   const OPEN4_SHARE_ACCESS_WANT_WRITE_DELEG       = 0x0200;
   const OPEN4_SHARE_ACCESS_WANT_ANY_DELEG         = 0x0300;
   const OPEN4_SHARE_ACCESS_WANT_NO_DELEG          = 0x0400;
   const OPEN4_SHARE_ACCESS_WANT_CANCEL            = 0x0500;

   const
    OPEN4_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_RESRC_AVAIL
    = 0x10000;

   const
    OPEN4_SHARE_ACCESS_WANT_PUSH_DELEG_WHEN_UNCONTENDED
    = 0x20000;

   enum open_delegation_type4 {
           OPEN_DELEGATE_NONE      = 0,
           OPEN_DELEGATE_READ      = 1,
           OPEN_DELEGATE_WRITE     = 2,
           OPEN_DELEGATE_NONE_EXT  = 3 /* new to v4.1 */
   };

   enum open_claim_type4 {
           /*
            * Not a reclaim.
            */
           CLAIM_NULL              = 0,

           CLAIM_PREVIOUS          = 1,
           CLAIM_DELEGATE_CUR      = 2,
           CLAIM_DELEGATE_PREV     = 3,

           /*
            * Not a reclaim.
            *
            * Like CLAIM_NULL, but object identified
            * by the current filehandle.
            */
           CLAIM_FH                = 4, /* new to v4.1 */

           /*
            * Like CLAIM_DELEGATE_CUR, but object identified
            * by current filehandle.
            */
           CLAIM_DELEG_CUR_FH      = 5, /* new to v4.1 */

           /*
            * Like CLAIM_DELEGATE_PREV, but object identified
            * by current filehandle.
            */
           CLAIM_DELEG_PREV_FH     = 6 /* new to v4.1 */
   };

   struct open_claim_delegate_cur4 {
           stateid4        delegate_stateid;
           component4      file;
   };

   union open_claim4 switch (open_claim_type4 claim) {
    /*
     * No special rights to file.
     * Ordinary OPEN of the specified file.
     */
    case CLAIM_NULL:
           /* CURRENT_FH: directory */
           component4      file;
    /*
     * Right to the file established by an
     * open previous to server reboot.  File
     * identified by filehandle obtained at
     * that time rather than by name.
     */
    case CLAIM_PREVIOUS:
           /* CURRENT_FH: file being reclaimed */
           open_delegation_type4   delegate_type;

    /*
     * Right to file based on a delegation
     * granted by the server.  File is
     * specified by name.
     */
    case CLAIM_DELEGATE_CUR:
           /* CURRENT_FH: directory */
           open_claim_delegate_cur4        delegate_cur_info;

    /*
     * Right to file based on a delegation
     * granted to a previous boot instance
     * of the client.  File is specified by name.
     */
    case CLAIM_DELEGATE_PREV:
            /* CURRENT_FH: directory */
           component4      file_delegate_prev;

    /*
     * Like CLAIM_NULL.  No special rights
     * to file.  Ordinary OPEN of the
     * specified file by current filehandle.
     */
    case CLAIM_FH: /* new to v4.1 */
           /* CURRENT_FH: regular file to open */
           void;

    /*
     * Like CLAIM_DELEGATE_PREV.  Right to file based on a
     * delegation granted to a previous boot
     * instance of the client.  File is identified
     * by filehandle.
     */
    case CLAIM_DELEG_PREV_FH: /* new to v4.1 */
           /* CURRENT_FH: file being opened */
           void;

    /*
     * Like CLAIM_DELEGATE_CUR.  Right to file based on
     * a delegation granted by the server.
     * File is identified by filehandle.
     */
    case CLAIM_DELEG_CUR_FH: /* new to v4.1 */
            /* CURRENT_FH: file being opened */
            stateid4       oc_delegate_stateid;

   };

   /*
    * OPEN: Open a file, potentially receiving an OPEN delegation
    */
   struct OPEN4args {
           seqid4          seqid;
           uint32_t        share_access;
           uint32_t        share_deny;
           open_owner4     owner;
           openflag4       openhow;
           open_claim4     claim;
   };

18.16.2. RESULTS ✨

Okay, here's what the server hands back when OPEN works out 💌

   struct open_read_delegation4 {
    stateid4 stateid;    /* Stateid for delegation*/
    bool     recall;     /* Pre-recalled flag for
                            delegations obtained
                            by reclaim (CLAIM_PREVIOUS) */

    nfsace4 permissions; /* Defines users who don't
                            need an ACCESS call to
                            open for read */
   };

   struct open_write_delegation4 {
    stateid4 stateid;      /* Stateid for delegation */
    bool     recall;       /* Pre-recalled flag for
                              delegations obtained
                              by reclaim
                              (CLAIM_PREVIOUS) */

    nfs_space_limit4
              space_limit; /* Defines condition that
                              the client must check to
                              determine whether the
                              file needs to be flushed
                              to the server on close.  */

    nfsace4   permissions; /* Defines users who don't
                              need an ACCESS call as
                              part of a delegated
                              open. */
   };


   enum why_no_delegation4 { /* new to v4.1 */
           WND4_NOT_WANTED         = 0,
           WND4_CONTENTION         = 1,
           WND4_RESOURCE           = 2,
           WND4_NOT_SUPP_FTYPE     = 3,
           WND4_WRITE_DELEG_NOT_SUPP_FTYPE = 4,
           WND4_NOT_SUPP_UPGRADE   = 5,
           WND4_NOT_SUPP_DOWNGRADE = 6,
           WND4_CANCELLED          = 7,
           WND4_IS_DIR             = 8
   };

   union open_none_delegation4 /* new to v4.1 */
   switch (why_no_delegation4 ond_why) {
           case WND4_CONTENTION:
                   bool ond_server_will_push_deleg;
           case WND4_RESOURCE:
                   bool ond_server_will_signal_avail;
           default:
                   void;
   };

   union open_delegation4
   switch (open_delegation_type4 delegation_type) {
           case OPEN_DELEGATE_NONE:
                   void;
           case OPEN_DELEGATE_READ:
                   open_read_delegation4 read;
           case OPEN_DELEGATE_WRITE:
                   open_write_delegation4 write;
           case OPEN_DELEGATE_NONE_EXT: /* new to v4.1 */
                   open_none_delegation4 od_whynone;
   };

   /*
    * Result flags
    */

   /* Client must confirm open */
   const OPEN4_RESULT_CONFIRM      = 0x00000002;
   /* Type of file locking behavior at the server */
   const OPEN4_RESULT_LOCKTYPE_POSIX = 0x00000004;
   /* Server will preserve file if removed while open */
   const OPEN4_RESULT_PRESERVE_UNLINKED = 0x00000008;

   /*
    * Server may use CB_NOTIFY_LOCK on locks
    * derived from this open
    */
   const OPEN4_RESULT_MAY_NOTIFY_LOCK = 0x00000020;

   struct OPEN4resok {
    stateid4       stateid;      /* Stateid for open */
    change_info4   cinfo;        /* Directory Change Info */
    uint32_t       rflags;       /* Result flags */
    bitmap4        attrset;      /* attribute set for create*/
    open_delegation4 delegation; /* Info on any open
                                    delegation */
   };

   union OPEN4res switch (nfsstat4 status) {
    case NFS4_OK:
           /* New CURRENT_FH: opened file */
           OPEN4resok      resok4;
    default:
           void;
   };

18.16.3. DESCRIPTION 💅

Okay bestie, OPEN is the operation that opens a regular file inside a directory, using either a name or a filehandle 📂. It can also make a brand-new file if you hand it a name and say you want one created ✨ Whether a file gets created, and how, is decided by the openhow parameter. That's a switched union (data type opengflag4) that switches on opentype, which is either OPEN4_NOCREATE or OPEN4_CREATE. If you go with OPEN4_CREATE, you get another switched union (data type createhow4) with four creation methods to pick from: UNCHECKED4, GUARDED4, EXCLUSIVE4, or EXCLUSIVE4_1 🎀. And if opentype is OPEN4_CREATE, then the claim field of the claim field MUST be one of CLAIM_NULL, CLAIM_DELEGATE_CUR, or CLAIM_DELEGATE_PREV, because those claim methods include a component of a file name (yes, claim of claim, I know 🙈).

Upon success, which might mean a new file got created, the current filehandle is replaced by the one for the created or existing object 🦋

If the current filehandle is a named attribute directory, OPEN will then create or open a named attribute file. Note that exclusive create of a named attribute is not supported 🙅‍♀️ If the createmode is EXCLUSIVE4 or EXCLUSIVE4_1 and the current filehandle is a named attribute directory, the server will return EINVAL.

UNCHECKED4 means the file should be created if a file with that name doesn't exist, and running into an existing regular file with that name is not an error 🌸 For this type of create, createattrs sets the initial set of attributes for the file. That set may include any writable attribute that's valid for regular files. When an UNCHECKED4 create hits an existing file, the attributes in createattrs are not used. The one exception is when createattrs sets the size attribute to zero, because then the existing file gets truncated ✂️

If GUARDED4 is specified, the server checks for a duplicate object by name before it creates anything. If a duplicate exists, NFS4ERR_EXIST is returned 🚫 If the object doesn't exist, the request is handled the same way as UNCHECKED4.

For the UNCHECKED4 and GUARDED4 cases, when the operation succeeds, the server sends back an attribute mask showing which attributes were successfully set for the object 🧚

EXCLUSIVE4_1 and EXCLUSIVE4 mean the server is supposed to follow exclusive creation semantics, using the verifier to make sure the target is created exclusively 🔐 The server should check for a duplicate object by name. If the object doesn't exist, the server creates it and stores the verifier with the object. If the object does exist and the stored verifier matches the verifier the client sent, the server uses the existing object as the newly created one. If the stored verifier doesn't match, the server returns NFS4ERR_EXIST 💀

If you're using EXCLUSIVE4, and the server uses attributes to store the exclusive create verifier, the server will say which attributes it used by setting the matching bits in the attribute mask sent back in the results. Unlike UNCHECKED4, GUARDED4, and EXCLUSIVE4_1, EXCLUSIVE4 does not let you set attributes at file creation. So after a successful OPEN via EXCLUSIVE4, the client MUST send a SETATTR to set attributes to a known state 🫶

In NFSv4.1, EXCLUSIVE4 has been deprecated in favor of EXCLUSIVE4_1 🥀 Unlike EXCLUSIVE4, attributes can be provided in the EXCLUSIVE4_1 case. But because the server may use attributes of the target object to store the verifier, the set of allowable attributes may be smaller than the set SETATTR allows. The allowable attributes for EXCLUSIVE4_1 are listed in the suppattr_exclcreat (Section 5.8.1.14) attribute. If the client tries to set an attribute in cva_attrs that is not in suppattr_exclcreat, the server MUST return NFS4ERR_INVAL 🚨 The response field, attrset, shows both which attributes the server set from cva_attrs and which attributes it used to store the verifier. As described in Section 18.16.4, the client can compare cva_attrs.attrmask with attrset to figure out which attributes were used for the verifier 🔍

Now the drama 🎭 With persistent sessions and pNFS in the picture, under some conditions EXCLUSIVE4 MUST NOT be used by the client or supported by the server. Here's the table that sums up the appropriate and mandated exclusive create methods for NFSv4.1 implementations 👑

     +=============+==========+==============+=======================+
     | Persistent  | Server   | Server       | Client Allowed        |
     | Reply Cache | Supports | REQUIRED     |                       |
     | Enabled     | pNFS     |              |                       |
     +=============+==========+==============+=======================+
     | no          | no       | EXCLUSIVE4_1 | EXCLUSIVE4_1 (SHOULD) |
     |             |          | and          | or EXCLUSIVE4 (SHOULD |
     |             |          | EXCLUSIVE4   | NOT)                  |
     +-------------+----------+--------------+-----------------------+
     | no          | yes      | EXCLUSIVE4_1 | EXCLUSIVE4_1          |
     +-------------+----------+--------------+-----------------------+
     | yes         | no       | GUARDED4     | GUARDED4              |
     +-------------+----------+--------------+-----------------------+
     | yes         | yes      | GUARDED4     | GUARDED4              |
     +-------------+----------+--------------+-----------------------+

              Table 18: Required Methods for Exclusive Create

If CREATE_SESSION4_FLAG_PERSIST is set in the results of CREATE_SESSION, the reply cache is persistent (see Section 18.36) 💾 If the EXCHGID4_FLAG_USE_PNFS_MDS flag is set in the results from EXCHANGE_ID, the server is a pNFS server (see Section 18.35) 🌈 If the client tries to use EXCLUSIVE4 on a persistent session, or on a session derived from an EXCHGID4_FLAG_USE_PNFS_MDS client ID, the server MUST return NFS4ERR_INVAL 🙅‍♀️

With persistent sessions, exclusive create semantics are fully achievable with GUARDED4, so EXCLUSIVE4 or EXCLUSIVE4_1 MUST NOT be used 🍓 When pNFS is being used, the layout_hint attribute might not be supported after the file is created. Only the EXCLUSIVE4_1 and GUARDED methods of exclusive file creation allow the atomic setting of attributes ⚛️

For the target directory, the server returns change_info4 information in cinfo. With the atomic field of the change_info4 data type, the server indicates whether the before and after change attributes were obtained atomically with respect to the link creation 🔗

The OPEN operation provides Windows share reservation capability through the share_access and share_deny fields of the OPEN arguments 🪟 The client specifies the required share_access and share_deny modes at OPEN. For clients that don't directly support SHAREs (i.e., UNIX), the expected deny value is OPEN4_SHARE_DENY_NONE. If there's an existing SHARE reservation that conflicts with the OPEN request, the server returns the error NFS4ERR_SHARE_DENIED 🙃 For more on SHARE semantics, see Section 9.7.

For each OPEN, the client provides a value for the owner field of the OPEN argument. The owner field is of data type open_owner4, and contains a field called clientid and a field called owner. The client can set the clientid field to any value, and the server MUST ignore it 🙈 Instead, the server MUST derive the client ID from the session ID of the SEQUENCE operation in the COMPOUND request ✨

The "seqid" field of the request isn't used in NFSv4.1. It MAY be any value, and the server MUST ignore it 🌷

If the client is recovering state after a server failure, the claim field of the OPEN argument is used to signal that the request is meant to reclaim state that was previously held 🔄

The "claim" field of the OPEN argument is used to specify the file to be opened and the state information the client claims to have. There are seven claim types, as follows:

   +======================+============================================+
   | open type            | description                                |
   +======================+============================================+
   | CLAIM_NULL, CLAIM_FH | For the client, this is a new OPEN         |
   |                      | request and there is no previous state     |
   |                      | associated with the file for the           |
   |                      | client.  With CLAIM_NULL, the file is      |
   |                      | identified by the current filehandle       |
   |                      | and the specified component name.          |
   |                      | With CLAIM_FH (new to NFSv4.1), the        |
   |                      | file is identified by just the current     |
   |                      | filehandle.                                |
   +----------------------+--------------------------------------------+
   | CLAIM_PREVIOUS       | The client is claiming basic OPEN          |
   |                      | state for a file that was held             |
   |                      | previous to a server restart.              |
   |                      | Generally used when a server is            |
   |                      | returning persistent filehandles; the      |
   |                      | client may not have the file name to       |
   |                      | reclaim the OPEN.                          |
   +----------------------+--------------------------------------------+
   | CLAIM_DELEGATE_CUR,  | The client is claiming a delegation        |
   | CLAIM_DELEG_CUR_FH   | for OPEN as granted by the server.         |
   |                      | Generally, this is done as part of         |
   |                      | recalling a delegation.  With              |
   |                      | CLAIM_DELEGATE_CUR, the file is            |
   |                      | identified by the current filehandle       |
   |                      | and the specified component name.          |
   |                      | With CLAIM_DELEG_CUR_FH (new to            |
   |                      | NFSv4.1), the file is identified by        |
   |                      | just the current filehandle.               |
   +----------------------+--------------------------------------------+
   | CLAIM_DELEGATE_PREV, | The client is claiming a delegation        |
   | CLAIM_DELEG_PREV_FH  | granted to a previous client instance;     |
   |                      | used after the client restarts.  The       |
   |                      | server MAY support CLAIM_DELEGATE_PREV     |
   |                      | and/or CLAIM_DELEG_PREV_FH (new to         |
   |                      | NFSv4.1).  If it does support either       |
   |                      | claim type, CREATE_SESSION MUST NOT        |
   |                      | remove the client's delegation state,      |
   |                      | and the server MUST support the            |
   |                      | DELEGPURGE operation.                      |
   +----------------------+--------------------------------------------+

                                  Table 19

Okay bestie, Table 19 is the whole open-type cheat sheet 📋✨ CLAIM_NULL and CLAIM_FH are the "brand new OPEN, I have no history with this file" options 🦋. With CLAIM_NULL you find the file by the current filehandle plus the component name, and with CLAIM_FH (new in NFSv4.1) you just use the current filehandle. CLAIM_PREVIOUS is the comeback arc 🌸: the client is reclaiming basic OPEN state from before a server restart, and it's usually a thing when the server hands out persistent filehandles, because the client might not even have the file name anymore. Then we have the delegation claims 👑. CLAIM_DELEGATE_CUR and CLAIM_DELEG_CUR_FH are for claiming a delegation the server already granted, and they're generally part of recalling a delegation. CLAIM_DELEGATE_PREV and CLAIM_DELEG_PREV_FH are for delegations that went to a previous client instance, so they come into play after the client restarts 🧚. The server MAY support those last two, but if it does, CREATE_SESSION MUST NOT wipe the client's delegation state, and the server MUST support the DELEGPURGE operation 💜.

Now the grace period 🕰️. For OPEN requests that reach the server during the grace period, the server returns NFS4ERR_GRACE, but there are some exceptions, so pay attention:

OPEN requests with the claim type CLAIM_PREVIOUS are all about reclaiming opens after a server restart, and they're typically only valid during the grace period 🙈.
OPEN requests with the claim types CLAIM_DELEGATE_CUR and CLAIM_DELEG_CUR_FH are valid both during and after the grace period 🫶. The reason is that the delegation they hang off of already guarantees there's no conflict with locks other clients might need to reclaim, so the server need not return NFS4ERR_GRACE for these.

For any OPEN request, the server may return an OPEN delegation, which lets further opens and closes be handled locally on the client, as described in Section 10.4 🎀. Delegation is totally up to the server, so the client should never assume it will or won't be granted. It should always be ready for either outcome 💅. The partial exception is the reclaim (CLAIM_PREVIOUS) case, where a delegation type is claimed. There, delegation will always be granted, although the server may specify an immediate recall in the delegation structure 😭.

The rflags returned by a successful OPEN let the server tell the client how the open file should be handled ✨:

OPEN4_RESULT_CONFIRM is deprecated and MUST NOT be returned by an NFSv4.1 server 💀.
OPEN4_RESULT_LOCKTYPE_POSIX says the server's byte-range locking behavior supports the complete set of POSIX locking techniques [21]. From that, the client can choose how to manage byte-range locking state so it can handle any mismatch in byte-range locking management 🔥.
OPEN4_RESULT_PRESERVE_UNLINKED says the server will preserve the open file if the client (or any other client) removes the file while it's still open. The server also promises to preserve the file through the grace period after a server restart, which gives the client the chance to reclaim its open 🌈.
OPEN4_RESULT_MAY_NOTIFY_LOCK says the server may attempt CB_NOTIFY_LOCK callbacks for locks on this file. It's only a hint, so the client can safely ignore it 🍓.

If the component has zero length, NFS4ERR_INVAL gets returned 🙅‍♀️. The component also has to pass the normal UTF-8, character support, and name checks. See Section 14.5 for the full tea ☕.

When an OPEN is done and the specified open-owner already has the resulting filehandle open, the new share and deny status get "OR"ed together with the existing status. Only one CLOSE needs to happen in that case, even if several OPENs were completed. Checking of share reservations for the new OPEN still goes through normally, with no exception for the existing OPEN held by the same open-owner. The stateid that comes back has an "other" field matching the previous open, and its "seqid" field gets incremented to reflect the status change from the new open 🔐.

If the underlying file system is only accessible in read-only mode and the OPEN request specifies ACCESS_WRITE or ACCESS_BOTH, the server returns NFS4ERR_ROFS to say it's a read-only file system 📖.

Same energy as CREATE here 🫧. The server MUST derive the owner, owner ACE, group, or group ACE if any of those four attributes are required and supported by the server's file system. For an OPEN with the EXCLUSIVE4 createmode, the server has no choice, since those OPEN calls don't include the createattrs field 🤷‍♀️. On the flip side, if createattrs (UNCHECKED4 or GUARDED4) or cva_attrs (EXCLUSIVE4_1) is specified and includes an owner, owner_group, or ACE that the principal in the RPC call's credentials isn't authorized to create files for, then the server may return NFS4ERR_PERM 🚫.

When an OPEN specifies a size of zero (for example, truncation) and the file has named attributes, the named attributes are left as they are and are not removed 🌷.

NFSv4.1 gives clients way more precise control over getting delegations, using these new flags in the share_access field of OPEN4args 🎯:

OPEN4_SHARE_ACCESS_WANT_READ_DELEG
OPEN4_SHARE_ACCESS_WANT_WRITE_DELEG
OPEN4_SHARE_ACCESS_WANT_ANY_DELEG
OPEN4_SHARE_ACCESS_WANT_NO_DELEG
OPEN4_SHARE_ACCESS_WANT_CANCEL
OPEN4_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_RESRC_AVAIL
OPEN4_SHARE_ACCESS_WANT_PUSH_DELEG_WHEN_UNCONTENDED

If (share_access & OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) is not zero, then the client will have specified one and only one of these: OPEN4_SHARE_ACCESS_WANT_READ_DELEG, OPEN4_SHARE_ACCESS_WANT_WRITE_DELEG, OPEN4_SHARE_ACCESS_WANT_ANY_DELEG, OPEN4_SHARE_ACCESS_WANT_NO_DELEG, or OPEN4_SHARE_ACCESS_WANT_CANCEL 🦄. Otherwise, the client isn't saying it wants a delegation or doesn't want one, so the server MAY or MAY not return a delegation in the OPEN response 🤞.

If the server supports the new _WANT_ flags and the client sends one or more of them, then when the server doesn't return a delegation, it MUST return a delegation type of OPEN_DELEGATE_NONE_EXT 🙌. The field ond_why in the reply says why no delegation came back, and it will be one of these:

WND4_NOT_WANTED: The client specified OPEN4_SHARE_ACCESS_WANT_NO_DELEG.
WND4_CONTENTION: There is a conflicting delegation or open on the file.
WND4_RESOURCE: Resource limitations prevent the server from granting a delegation.
WND4_NOT_SUPP_FTYPE: The server does not support delegations on this file type.
WND4_WRITE_DELEG_NOT_SUPP_FTYPE: The server does not support OPEN_DELEGATE_WRITE delegations on this file type.
WND4_NOT_SUPP_UPGRADE: The server does not support atomic upgrade of an OPEN_DELEGATE_READ delegation to an OPEN_DELEGATE_WRITE delegation.
WND4_NOT_SUPP_DOWNGRADE: The server does not support atomic downgrade of an OPEN_DELEGATE_WRITE delegation to an OPEN_DELEGATE_READ delegation.
WND4_CANCELED: The client specified OPEN4_SHARE_ACCESS_WANT_CANCEL, and now any "want" for this file object is cancelled.
WND4_IS_DIR: The specified file object is a directory, and the operation is OPEN or WANT_DELEGATION, which do not support delegations on directories.

OPEN4_SHARE_ACCESS_WANT_READ_DELEG, OPEN_SHARE_ACCESS_WANT_WRITE_DELEG, or OPEN_SHARE_ACCESS_WANT_ANY_DELEG mean, respectively, that the client wants an OPEN_DELEGATE_READ, an OPEN_DELEGATE_WRITE, or any delegation, no matter which of OPEN4_SHARE_ACCESS_READ, OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH is set 🌟. If the client already has an OPEN_DELEGATE_READ delegation on a file and requests an OPEN_DELEGATE_WRITE delegation, then the client is asking for an atomic upgrade of that delegation 📈. If the client already has an OPEN_DELEGATE_WRITE delegation and requests an OPEN_DELEGATE_READ delegation, then it's asking for an atomic downgrade to OPEN_DELEGATE_READ 📉. A server MAY support atomic upgrade or downgrade. If it does, a returned delegation_type of OPEN_DELEGATE_READ or OPEN_DELEGATE_WRITE that's different from the one the client currently has means the upgrade or downgrade worked 🎉. If the server doesn't support atomic delegation upgrade or downgrade, then ond_why gets set to WND4_NOT_SUPP_UPGRADE or WND4_NOT_SUPP_DOWNGRADE 🥲.

OPEN4_SHARE_ACCESS_WANT_NO_DELEG means the client wants no delegation 🙅‍♀️. OPEN4_SHARE_ACCESS_WANT_CANCEL means the client wants no delegation and also wants to cancel any previously registered "want" for one 🧹.

The client may set one or both of OPEN4_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_RESRC_AVAIL and OPEN4_SHARE_ACCESS_WANT_PUSH_DELEG_WHEN_UNCONTENDED 💫. However, they have no effect unless one of these is also set: OPEN4_SHARE_ACCESS_WANT_READ_DELEG, OPEN4_SHARE_ACCESS_WANT_WRITE_DELEG, or OPEN4_SHARE_ACCESS_WANT_ANY_DELEG 🎀.

okay so if the client sets OPEN4_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_RESRC_AVAIL 🦋, it's basically saying "hey, I want a delegation, and if the OPEN results don't come with one, please remember me!" 💖 If the server says no because it's out of resources, the server MAY later tell the client, using the CB_RECALLABLE_OBJ_AVAIL operation, that the resource crunch has eased ✨. The server lets the client know it's planning to send that future CB_RECALLABLE_OBJ_AVAIL by setting delegation_type in the results to OPEN_DELEGATE_NONE_EXT, ond_why to WND4_RESOURCE, and ond_server_will_signal_avail to TRUE 🎀. And if ond_server_will_signal_avail is TRUE, the server MUST later send a CB_RECALLABLE_OBJ_AVAIL operation. No ghosting, periodt 💅.

If the client sets OPEN4_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_UNCONTENDED 🌸, it's the same vibe: it wants a "want" registered for a delegation, in case the OPEN results don't include one. If the server denies it because of contention, the server MAY later use the CB_PUSH_DELEG operation to say the contention has eased 🧚. The server signals that by setting delegation_type to OPEN_DELEGATE_NONE_EXT, ond_why to WND4_CONTENTION, and ond_server_will_push_deleg to TRUE 💜. If ond_server_will_push_deleg is TRUE, the server MUST later send a CB_PUSH_DELEG operation 🫶.

Now, if the client already registered a want for a delegation on a file, and then asks to register another want on that same file, the server MUST return a brand-new error: NFS4ERR_DELEG_ALREADY_WANTED 🙈. And if the client wants a different type of delegation want for the same file, it MUST cancel the existing delegation WANT first 😭.

18.16.4. IMPLEMENTATION ✨

Okay, this part is giving implementation realness 🔥. Without a persistent session, the client does an exclusive create by setting the how parameter to EXCLUSIVE4 or EXCLUSIVE4_1 🎀. The client brings a verifier that can reasonably be expected to be unique 🦋. Something like a client identifier (maybe the client network address) plus a unique number the client generates (maybe the RPC transaction identifier) can be a great combo 💖.

If the object doesn't exist, the server creates it and stores the verifier in stable storage ✨. For file systems that don't have a way to store arbitrary file attributes, the server MAY use one or more elements of the object's metadata to hold the verifier 🧚. The verifier MUST be stored in stable storage to prevent erroneous failure on retransmission of the request 💅. It's assumed an exclusive create is happening because exclusive semantics are critical to the application 👑. Because of that, exclusive CREATE does not rely solely on the server's reply cache to store the verifier 😭. A nonpersistent reply cache doesn't survive a crash, and the session and reply cache may be deleted after a network partition that exceeds the lease time, which would open failure windows 💀.

An NFSv4.1 server SHOULD NOT store the verifier in any of the file's RECOMMENDED or REQUIRED attributes 🙈. If it does anyway, the server SHOULD use time_modify_set or time_access_set to store the verifier 🌈. The server SHOULD NOT store the verifier in these attributes:

acl (it's desirable for access control to be established at creation) 🔒
dacl (ditto) 🔒
mode (ditto) 🔒
owner (ditto) 🔒
owner_group (ditto) 🔒
retentevt_set (it may be desired to establish retention at creation)
retention_hold (ditto)
retention_set (ditto)
sacl (it's desirable for auditing control to be established at creation) 🔍
size (on some servers, size may have a limited range of values) 📏
mode_set_masked (as with mode)
time_creation (a meaningful file creation should be set when the file is created) 🍓

Another option for the server is to use a named attribute to store the verifier 🦋.

Because the EXCLUSIVE4 create method doesn't specify initial attributes when it's processed, the server:

SHOULD set the owner of the file to the one that matches the credential in the request's RPC header 👑.
SHOULD NOT leave the file's access control to anyone but the owner of the file 🫶.

If the server can't support exclusive create semantics, maybe because it has to commit the verifier to stable storage, it should fail the OPEN request with the error NFS4ERR_NOTSUPP 😭.

During an exclusive CREATE request, if the object already exists, the server rebuilds the object's verifier and compares it to the verifier in the request 🔍. If they match, the server treats the request as a success ✨. The request is presumed to be a duplicate of an earlier, successful request whose reply got lost and that the server's duplicate request cache didn't catch 🙈. If the verifiers don't match, the request gets rejected with the status NFS4ERR_EXIST 💀.

After a successful exclusive create, the attrset response tells you which attributes were used to store the verifier 🎀. If EXCLUSIVE4 was used, the attributes set in attrset were used for the verifier 💖. If EXCLUSIVE4_1 was used, the client figures out which attributes hold the verifier by comparing attrset with cva_attrs.attrmask; any bits set in the former but not the latter identify those attributes 🧚. The client MUST immediately send a SETATTR to set the attributes used to store the verifier 🚨. Until it does, those attributes can't be relied upon 😭. And the subsequent SETATTR MUST NOT occur in the same COMPOUND request as the OPEN 🙅‍♀️.

Unless a persistent session is used, the GUARDED4 attribute does not provide exactly once semantics 💀. If a reply gets lost and the server doesn't detect the retransmission, the operation can fail with NFS4ERR_EXIST even though the create actually worked 😭. The client would use this in the case where the application didn't ask for an exclusive create but did ask for the file to be truncated when it's opened 🌸. And if the client times out and retransmits the create request, the client can use GUARDED4 to prevent a sequence like create, write, create (retransmitted) from happening 🛡️.

For SHARE reservations, the value of the expression (share_access & ~OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) MUST be one of OPEN4_SHARE_ACCESS_READ, OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH 🎀. If it isn't, the server MUST return NFS4ERR_INVAL 🙈. The value of share_deny MUST be one of OPEN4_SHARE_DENY_NONE, OPEN4_SHARE_DENY_READ, OPEN4_SHARE_DENY_WRITE, or OPEN4_SHARE_DENY_BOTH 💅. If it isn't, the server MUST return NFS4ERR_INVAL again 💀.

Based on the share_access value (OPEN4_SHARE_ACCESS_READ, OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH), the client SHOULD check that the requester has the right access to perform the specified operation 🔍. This usually comes from applying the ACL access rules to the file for the current requester 🦋. But just like with the ACCESS operation, the client SHOULD NOT try to second-guess the server's decisions 🙅‍♀️, since access rights can change and may be subject to server administrative controls outside the ACL framework 👑. If the requester's READ or WRITE operation isn't authorized (depending on the share_access value), the server MUST return NFS4ERR_ACCESS 🚫.

Note that if the client ID wasn't created with the EXCHGID4_FLAG_BIND_PRINC_STATEID capability set in the reply to EXCHANGE_ID, the server MUST NOT require that READs and WRITEs sent for an open file have the same credentials as the OPEN itself 🌈. The server is REQUIRED to perform access checking on the READs and WRITEs themselves 🔥. But if the reply to EXCHANGE_ID did have EXCHGID4_FLAG_BIND_PRINC_STATEID set, then, with one exception, the credentials used in the OPEN request MUST match those used in the READs and WRITEs 🫶, and the stateids in the READs and WRITEs MUST match, or be derived from the stateid in the reply to OPEN ✨. The exception is when SP4_SSV or SP4_MACH_CRED state protection is used and the spo_must_allow result of EXCHANGE_ID includes the READ and/or WRITE operations 🧚. In that case, the machine or SSV credential will be allowed to send READ and/or WRITE 🎀. See Section 18.35 📖.

If the component given to OPEN is a symbolic link, the error NFS4ERR_SYMLINK gets returned to the client 🔗. If it's a directory, the error NFS4ERR_ISDIR gets returned 📁. If the component is neither of those but still isn't an ordinary file, the error NFS4ERR_WRONG_TYPE is returned 🙈. And if the current filehandle isn't a directory, the error NFS4ERR_NOTDIR gets returned 💀.

The OPEN4_RESULT_PRESERVE_UNLINKED result flag lets a client skip the super common implementation trick of renaming an open file to ".nfs<unique value>" after it removes the file 🙌. After the server returns OPEN4_RESULT_PRESERVE_UNLINKED, if a client sends a REMOVE operation that would drop the file's link count to zero, the server SHOULD report a value of zero for the numlinks attribute on the file 🍓.

If another client has a delegation on the file being opened that conflicts with the open (sometimes depending on the share_access or share_deny value), the delegation(s) MUST be recalled, and the operation can't proceed until each such delegation is returned or revoked 🚨. Except where this happens super quickly, one or more NFS4ERR_DELAY errors will be returned to requests made while the delegation is still outstanding ⏳. For an OPEN_DELEGATE_WRITE delegation, any open by a different client will conflict 💥. For an OPEN_DELEGATE_READ delegation, only opens with one of these characteristics count as conflicting:

The value of share_access includes the bit OPEN4_SHARE_ACCESS_WRITE.
The value of share_deny specifies OPEN4_SHARE_DENY_READ or OPEN4_SHARE_DENY_BOTH.
OPEN4_CREATE is specified together with UNCHECKED4, the size attribute is specified as zero (for truncation), and an existing file is truncated.

If OPEN4_CREATE is specified and the file doesn't exist, and the current filehandle designates a directory for which another client holds a directory delegation, then, unless the delegation is such that the situation can be resolved by sending a notification, the delegation MUST be recalled 📬. The operation can't proceed until the delegation is returned or revoked. Except where this happens very quickly, one or more NFS4ERR_DELAY errors will be returned to requests made while the delegation remains outstanding 😩.

If OPEN4_CREATE is specified and the file doesn't exist, and the current filehandle designates a directory for which one or more directory delegations exist, then, when those delegations request such notifications, NOTIFY4_ADD_ENTRY will be generated as a result of this operation 🔔✨.

18.16.4.1. Warning to Client Implementors 🚨

OKAY bestie, this one is SO important 😭 OPEN is lowkey like LOOKUP in that it hands the client back a filehandle 🎀 BUT here's the twist: unlike LOOKUP, OPEN creates server state on that filehandle 👀 And in normal circumstances, the only way the client can release that state is with a CLOSE operation. ✨

Now CLOSE figures out which file to close by looking at the current filehandle 💅 So the client MUST follow every OPEN operation with a GETFH operation in the same COMPOUND procedure 🫶 That way the client gets the filehandle and CLOSE can be used appropriately. Non-negotiable, periodt 💀

Also, simply waiting for the lease on the file to expire is NOT enough 🙈 The server may keep that state indefinitely as long as no other client tries to make a conflicting access to the same file. So don't just vibe and wait, you have to actually close it 💖

18.17. Operation 19: OPENATTR - Open Named Attribute Directory 🌸

18.17.1. ARGUMENTS 🧚

struct OPENATTR4args {
        /* CURRENT_FH: object */
        bool    createdir;
};

18.17.2. RESULTS 🍓

struct OPENATTR4res {
        /*
         * If status is NFS4_OK,
         *   new CURRENT_FH: named attribute
         *                   directory
         */
        nfsstat4        status;
};

18.17.3. DESCRIPTION 💜

OPENATTR is the op you use to get the filehandle of the named attribute directory tied to the current filehandle ✨ The result is a filehandle to an object of type NF4ATTRDIR. From there, READDIR and LOOKUP can get you filehandles for the different named attributes on the original file system object. Filehandles returned inside the named attribute directory designate objects of type NF4NAMEDATTR 🔥

The createdir argument lets the client say whether a named attribute directory should be created as a result of OPENATTR. Some clients use OPENATTR with createdir set to FALSE just to check whether any named attributes exist for the object 🙈 If none exist, you get NFS4ERR_NOENT back 💀 If createdir is TRUE and there's no named attribute directory yet, one gets created and its filehandle becomes the current filehandle 🎀 If createdir is TRUE and the directory already exists, no error, and the existing directory's filehandle becomes the current filehandle instead 🫶 Creating a named attribute directory assumes the server has implemented named attribute support in this way, and the spec doesn't require it to do so by this definition 🤷‍♀️

If the current filehandle designates an object of type NF4NAMEDATTR (a named attribute) or NF4ATTRDIR (a named attribute directory), you get an error of NFS4ERR_WRONG_TYPE 😭 Named attributes and named attribute directories MUST NOT have their own named attributes. Not me being shocked that we need a rule for this 💅

18.17.4. IMPLEMENTATION 🌈

If the server doesn't support named attributes for the current filehandle, the client gets an error of NFS4ERR_NOTSUPP ✨

18.18. Operation 21: OPEN_DOWNGRADE - Reduce Open File Access 🦋

18.18.1. ARGUMENTS 🧚

struct OPEN_DOWNGRADE4args {
        /* CURRENT_FH: opened file */
        stateid4        open_stateid;
        seqid4          seqid;
        uint32_t        share_access;
        uint32_t        share_deny;
};

18.18.2. RESULTS 🍓

struct OPEN_DOWNGRADE4resok {
        stateid4        open_stateid;
};

union OPEN_DOWNGRADE4res switch(nfsstat4 status) {
 case NFS4_OK:
        OPEN_DOWNGRADE4resok    resok4;
 default:
         void;
};

18.18.3. DESCRIPTION 💜

OPEN_DOWNGRADE is how you adjust the access and deny states for a given open 🔥 This is needed when one open-owner opens the same file multiple times with different access and deny values. So when you close one of those opens, the share_access and share_deny flags might need to drop bits tied to opens that aren't in effect anymore 🙌

Valid values for the expression (share_access & ~OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) are OPEN4_SHARE_ACCESS_READ, OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH. If the client sends any other values, the server MUST reply with NFS4ERR_INVAL 😭

Valid values for the share_deny field are OPEN4_SHARE_DENY_NONE, OPEN4_SHARE_DENY_READ, OPEN4_SHARE_DENY_WRITE, or OPEN4_SHARE_DENY_BOTH. Anything else and the server MUST reply with NFS4ERR_INVAL 💀

After checking those values, the server replaces the current access and deny modes on the file with share_access and share_deny, but there are constraints 👀

The bits in share_access SHOULD equal the union of the share_access bits (not including OPEN4_SHARE_WANT_* bits) specified for some subset of the OPENs in effect for the current open-owner on the current file.
The bits in share_deny SHOULD equal the union of the share_deny bits specified for some subset of the OPENs in effect for the current open-owner on the current file.

If those constraints aren't respected, the server SHOULD return the error NFS4ERR_INVAL 🙈 Since share_access and share_deny bits should be subsets of what's already granted, unless there's a defect in the client or server implementation, OPEN_DOWNGRADE can't get denied because of conflicting share reservations 🫶

The seqid argument is not used in NFSv4.1 ✨ It MAY be any value, and the server MUST ignore it 🧚

On success, the current filehandle keeps its value 🎀

18.18.4. IMPLEMENTATION 🌈

An OPEN_DOWNGRADE operation may make OPEN_DELEGATE_READ delegations grantable where they weren't before 🦋 Servers can choose to respond right away if there are pending delegation want requests, or they can deal with the situation later 💖

18.19. Operation 22: PUTFH - Set Current Filehandle 🌸

18.19.1. ARGUMENTS 🧚

struct PUTFH4args {
        nfs_fh4         object;
};

18.19.2. RESULTS 🍓

struct PUTFH4res {
        /*
         * If status is NFS4_OK,
         *    new CURRENT_FH: argument to PUTFH
         */
        nfsstat4        status;
};

18.19.3. DESCRIPTION 💜

PUTFH replaces the current filehandle with the filehandle you give it as an argument 💅 It also clears the current stateid ✨

If the security mechanism the requester uses doesn't meet the requirements of the filehandle given to this operation, the server MUST return NFS4ERR_WRONGSEC 😭

See Section 16.2.3.1.1 for more details on the current filehandle 🦋

See Section 16.2.3.1.2 for more details on the current stateid 🫶

18.19.4. IMPLEMENTATION 🌈

This operation is used in an NFS request to set the context for the file accessing operations that follow in the same COMPOUND request 💖

18.20. Operation 23: PUTPUBFH - Set Public Filehandle 🧚

18.20.1. ARGUMENT 🎀

void;

18.20.2. RESULT 🍓

struct PUTPUBFH4res {
        /*
         * If status is NFS4_OK,
         *   new CURRENT_FH: public fh
         */
        nfsstat4        status;
};

18.20.3. DESCRIPTION 💜

PUTPUBFH replaces the current filehandle with the filehandle that represents the public filehandle of the server's namespace 🔥 This one can be different from the "root" filehandle that might be tied to some other directory on the server 👑

PUTPUBFH also clears the current stateid ✨

The public filehandle represents the concepts in RFC 2054 [49], RFC 2055 [50], and RFC 2224 [61] 🦋 The idea for NFSv4.1 is that the public filehandle (set with PUTPUBFH) gets used to give WebNFS server compatibility with NFSv3 🌈

The public filehandle and the root filehandle (set with PUTROOTFH) SHOULD be equivalent 💅 If they aren't equivalent, then the directory for the public filehandle MUST be a descendant of the directory for the root filehandle 🫶

See Section 16.2.3.1.1 for more details on the current filehandle 🦋

See Section 16.2.3.1.2 for more details on the current stateid 💖

18.20.4. IMPLEMENTATION 🌸

This operation is used in an NFS request to set the context for the file accessing operations that follow in the same COMPOUND request 🧚

With the NFSv3 public filehandle, the client can say whether the pathname in the LOOKUP should be evaluated as an absolute path relative to the server's root, or relative to the public filehandle 🙌 RFC 2224 [61] has more discussion on that 📚 With NFSv4.1 that kind of specification isn't directly available in LOOKUP, because the component separators needed to say absolute vs. relative aren't allowed in NFSv4 🙈 So the client has to build its request so that using PUTROOTFH or PUTPUBFH signals absolute or relative evaluation of an NFS URL, respectively 🫶

Note there are warnings in RFC 2224 [61] about absolute evaluation and the restrictions the server may put on it, based on how much of its namespace has been made available 👀 Those same warnings apply to NFSv4.1 too 💅 So because of server implementation details, an NFSv3 absolute public filehandle lookup might act differently than an NFSv4.1 absolute resolution 🤷‍♀️

There's a form of security negotiation in RFC 2755 [62] that uses the public filehandle and an overloaded pathname 😭 That method isn't available in NFSv4.1, since filehandles aren't overloaded with special meaning and so don't give the same framework as NFSv3 💀 So clients should use the security negotiation mechanisms in Section 2.6 instead 🎀

18.21. Operation 24: PUTROOTFH - Set Root Filehandle 🌈

18.21.1. ARGUMENTS 🧚

void;

18.21.2. RESULTS 🍓

struct PUTROOTFH4res {
        /*
         * If status is NFS4_OK,
         *   new CURRENT_FH: root fh
         */
        nfsstat4        status;
};

18.21.3. DESCRIPTION 💜

PUTROOTFH replaces the current filehandle with the filehandle that represents the root of the server's namespace 🌸 From this filehandle, a LOOKUP can find any other filehandle on the server 🔥 This one may be different from the "public" filehandle tied to some other directory on the server ✨

PUTROOTFH also clears the current stateid 💅

See Section 16.2.3.1.1 for more details on the current filehandle 🦋

See Section 16.2.3.1.2 for more details on the current stateid 🫶

18.21.4. IMPLEMENTATION 🌈

This operation is used in an NFS request to set the context for the file accessing operations that follow in the same COMPOUND request 💖

18.22. Operation 25: READ - Read from File 🦋

18.22.1. ARGUMENTS 🧚

struct READ4args {
        /* CURRENT_FH: file */
        stateid4        stateid;
        offset4         offset;
        count4          count;
};

18.22.2. RESULTS 🍓

struct READ4resok {
        bool            eof;
        opaque          data<>;
};

union READ4res switch (nfsstat4 status) {
 case NFS4_OK:
         READ4resok     resok4;
 default:
         void;
};

18.22.3. DESCRIPTION ✨

okay so READ is the op that pulls actual bytes out of a regular file, and it's always aimed at whatever file the current filehandle is pointing to 🦋

the client tells the server where to start (offset) and how many bytes it wants (count) 🎀 an offset of zero means "start at the very beginning of the file" 🌸 if the offset is greater than or equal to the file's size, you get NFS4_OK back with a data length of zero and eof set to TRUE. and bestie, READ still has to pass access permissions checking, no skipping that 💅

if the client asks for a count of zero, the READ still succeeds and gives back zero bytes, but (you guessed it) it's still subject to access permissions checking 🙈 the server MAY also decide to hand back fewer bytes than the client asked for. so the client has to check for that and handle it properly, fr fr 💖

unless special stateids are being used, the stateid on a READ is the one you got back from an earlier byte-range lock or share reservation request, or it's the stateid tied to a delegation ✨ the stateid tells the server who the owners are (if there are any) and lets the server double-check that the locks are still valid, like they haven't been revoked 🔥

now the eof flag 👀 if the read ended right at the end of the file (formally, in a correctly formed READ, offset + count equals the file's size), or if the READ goes past the end of the file (offset + count is greater than the file's size), then eof comes back as TRUE. otherwise it's FALSE. and a successful READ of an empty file will ALWAYS return eof as TRUE 🌈

if the current filehandle isn't an ordinary file, the client gets an error 😭 if it's an object of type NF4DIR, you get NFS4ERR_ISDIR. if the filehandle is a symbolic link, you get NFS4ERR_SYMLINK. and for literally everything else, it's NFS4ERR_WRONG_TYPE 💀

for a READ with a stateid where all the bits are zero, the server MAY let the READ go ahead, subject to mandatory byte-range locks or the file's current share deny modes 🦋 for a READ with a stateid where all the bits are one, the server MAY let READ operations skip the locking checks on the server. iconic behaviour, honestly 👑

and on success, the current filehandle keeps its value 💅

18.22.4. IMPLEMENTATION

okay this part is the real-life survival guide 🫶

if the server sends back a "short read" (meaning fewer data than requested and eof is FALSE), the client should send another READ to grab the rest 💖 servers can return less data than requested for a few reasons. the file might have been truncated by another client, or by the server itself, which changes the file size from what the requesting client thinks it is 😭 that shrinks how much data the client can actually get. it's also possible the server reduces the transfer size and hands back a short read. and server resource exhaustion can cause a short read too 💀

if mandatory byte-range locking is in effect for the file, and the byte-range the READ wants is WRITE_LT locked by an owner that isn't associated with the stateid, the server will return the NFS4ERR_LOCKED error 🚫 the client should try to get the right READ_LT through the LOCK operation before it tries the READ again. and once the READ finishes, the client should release that byte-range lock with LOCKU 🔐

now the delegation drama 🎭 if another client has an OPEN_DELEGATE_WRITE delegation on the file being read, that delegation MUST be recalled, and the operation can't move forward until the delegation is returned or revoked. unless that happens super fast, one or more NFS4ERR_DELAY errors will get sent back to requests made while the delegation is still outstanding ⏳ normally, delegations won't get recalled because of a READ, since the recall happens because of an earlier OPEN ✨ but a READ can be done with a special stateid, so the server needs to check for that case, even though the client should have done an OPEN before. no cap, that's the kind of plot twist that keeps servers up at night 🌙

18.23. Operation 26: READDIR - Read Directory

18.23.1. ARGUMENTS

struct READDIR4args {
        /* CURRENT_FH: directory */
        nfs_cookie4     cookie;
        verifier4       cookieverf;
        count4          dircount;
        count4          maxcount;
        bitmap4         attr_request;
};

18.23.2. RESULTS

struct entry4 {
        nfs_cookie4     cookie;
        component4      name;
        fattr4          attrs;
        entry4          *nextentry;
};

struct dirlist4 {
        entry4          *entries;
        bool            eof;
};

struct READDIR4resok {
        verifier4       cookieverf;
        dirlist4        reply;
};


union READDIR4res switch (nfsstat4 status) {
 case NFS4_OK:
         READDIR4resok  resok4;
 default:
         void;
};

18.23.3. DESCRIPTION

READDIR is the op for grabbing a batch of entries from a directory 📂✨ it returns the attributes the client asked for on each entry, plus a little cookie so the client can ask for more entries in a later READDIR 🍓

the arguments include a cookie that says where the READDIR should start in the directory 🧚 a cookie of zero means "start at the beginning." for later requests, the client uses a cookie that the server gave back on an earlier READDIR 💜

the request's cookieverf field should be set to 0 (zero) when the request's cookie field is zero (that's the first read of the directory) 🌸 on later requests, the cookieverf field must match the cookieverf returned by the READDIR where the cookie was acquired. and if the server decides the cookieverf isn't valid for the directory anymore, it MUST return NFS4ERR_NOT_SAME 😬

the dircount field is a hint for the max number of bytes of directory info the server should send back 💅 that number counts the total length of the directory entry names and the cookie values for those entries. the length is measured as the XDR encoding of that data (names and cookies), not the length in the server's native format 🎀

the maxcount field is the max total size of everything in the READDIR4resok structure, and it includes the XDR overhead 🔥 the server MAY return less data than that. and if the server can't fit even a single directory entry inside the maxcount limit, it MUST return the error NFS4ERR_TOOSMALL to the client 💀

last but not least, attr_request is the list of attributes to be returned for each directory entry the server sends 🦋

a successful reply is a list of directory entries. each one has the entry's name, a cookie for that entry, and the attributes that were asked for ✨ the eof flag is TRUE if there are no more entries in the directory 🌈

the cookie value only means something to the server, and it's basically a cursor for that directory entry 👑 the client uses it on later READDIR operations to keep reading the directory. the cookie is kind of like a READ offset, but the client MUST NOT interpret it as one 🙅‍♀️ ideally, the cookie value SHOULD NOT change if the directory gets modified, since the client might be caching these values 🫶

sometimes the server runs into an error while getting the attributes for a directory entry 😵 instead of failing the whole READDIR, the server can return the rdattr_error attribute (Section 5.8.1.12) instead 🛟 that way it can tell the client about the failure without failing the entire operation, which matters when the problem is maybe just a transient hiccup. obviously, the client has to request the fattr4_rdattr_error attribute for this to work properly. if the client doesn't ask for it, the server has no choice but to return failure for the entire READDIR operation 😭

for some file system environments, the directory entries "." and ".." have special meaning, and for others they don't 🧭 if the server supports these special entries in a directory, they SHOULD NOT be returned to the client as part of the READDIR response 💖 to help some client environments, the cookie values of zero, 1, and 2 are reserved. (the UNIX client uses these values when it combines the server's response with its local representation, so the application gets a fully formed UNIX directory presentation 🐧)

for READDIR arguments, cookie values of one and two SHOULD NOT be used 🙈 and for READDIR results, cookie values of zero, one, and two SHOULD NOT be returned 🚫

and on success, the current filehandle keeps its value 💅

18.23.4. IMPLEMENTATION

directory representations on servers can be super different from each other 🌍 and a client's programming interfaces might be tied to its local OS in a way that doesn't map nicely onto NFS. that's why the dircount and maxcount fields exist: they let the client give the server hints 💡 if the client is aggressive about collecting attributes during a READDIR, the server gets an idea of how to limit the encoded response 🎯

if dircount is zero, the server bounds the size of the reply based on the request's maxcount field ✨

the cookieverf can be used by the server to help manage cookie values that might go stale 🧁 it should be a rare thing for a server to be unable to keep reading a directory with the provided cookie/cookieverf pair. the server SHOULD make every effort to avoid that situation, because the application on the client might not be able to handle that kind of failure well 😬

the cookieverf also protects the client from using READDIR cookie values that might be stale 🛡️ for example, if the file system got migrated, the server might or might not be able to use the same cookie values the previous server used. with the client providing the cookieverf, the server can send back the right response. that stops the awkward case where the server accepts a cookie but the underlying directory has changed, and the response is invalid from the client's point of view of its previous READDIR 🌸

some servers won't return "." and ".." entries the way previous NFS versions did, so a client that needs those entries in its READDIR responses has to fabricate them itself 🛠️✨

18.24. Operation 27: READLINK - Read Symbolic Link

18.24.1. ARGUMENTS

/* CURRENT_FH: symlink */
void;

18.24.2. RESULTS

struct READLINK4resok {
        linktext4       link;
};

union READLINK4res switch (nfsstat4 status) {
 case NFS4_OK:
         READLINK4resok resok4;
 default:
         void;
};

18.24.3. DESCRIPTION

READLINK reads the data attached to a symbolic link 🔗 depending on the value of the UTF-8 capability attribute (Section 14.4), that data is encoded in UTF-8 🌈 whether the link was made by an NFS client or created locally on the server, the data in a symbolic link isn't interpreted when it's created (except maybe to check for proper UTF-8 encoding). it's just stored as-is, no drama 💖

and on success, the current filehandle keeps its value 💅✨

18.24.4. IMPLEMENTATION ✨

Okay so a symlink is basically a pointer to another file 🦋, but the server doesn't actually interpret what's inside it. It just stores the data. That means a client can totally save a pathname that makes zero sense to the server's OS 🙈, and READLINK just hands the data back so the client can figure it out 💅. If different implementations want to share symlinks, they've gotta agree on how to read that data, bestie 🫶.

READLINK is only allowed on objects of type NF4LNK 🎀. If the object isn't NF4LNK, the server should return NFS4ERR_WRONG_TYPE 💀.

18.25. Operation 28: REMOVE - Remove File System Object 🗑️

18.25.1. ARGUMENTS

struct REMOVE4args {
        /* CURRENT_FH: directory */
        component4      target;
};

18.25.2. RESULTS

struct REMOVE4resok {
        change_info4    cinfo;
};

union REMOVE4res switch (nfsstat4 status) {
 case NFS4_OK:
         REMOVE4resok   resok4;
 default:
         void;
};

18.25.3. DESCRIPTION

The REMOVE operation deletes a directory entry named by filename from the directory the current filehandle points to 🧚. If that entry was the last reference to the file system object, the object may be destroyed 💥. The directory can be either NF4DIR or NF4ATTRDIR.

For the directory where the filename got removed, the server sends back change_info4 info in cinfo 📦. With the atomic field of the change_info4 data type, the server tells you whether the before and after change attributes were obtained atomically with respect to the removal ✨.

If the target has a length of zero, or if the target doesn't obey the UTF-8 definition (and the server is enforcing UTF-8 encoding; see Section 14.4), the error NFS4ERR_INVAL gets returned 🙅‍♀️.

On success, the current filehandle keeps its value 🌸.

18.25.4. IMPLEMENTATION

Okay, the history tea ☕: NFSv3 needed a different operator, RMDIR, for removing directories, and REMOVE for everything else 🍓. That way clients could skip checking the file type when they got passed a non-directory delete system call (like unlink() [24] in POSIX) to remove a directory, and the converse (like a rmdir() on a non-directory), because they knew the server would check the file type anyway. NFSv4.1 REMOVE can delete any directory entry no matter its file type 🦋. So if you're the implementor building an NFSv4.1 client's entry points from the unlink() and rmdir() system calls, check the file type against the types that system call is allowed to remove before sending a REMOVE operation 🔍. Or, you can build a COMPOUND call that includes a LOOKUP/VERIFY sequence to verify the file type before a REMOVE operation in the same COMPOUND call 💜.

The concept of "last reference" is server specific 🤷‍♀️. But if the numlinks field in the previous attributes of the object had the value 1, the client should not rely on referring to the object via a filehandle. Same vibe with resources: the client should not rely on the resources (disk space, directory entry, and so on) formerly tied to the object becoming immediately available 🙈. So if a client needs to keep accessing a file after using REMOVE on it, it should take steps to make sure the file will still be accessible 🫶. The traditional move is to RENAME the file from its old name to a new hidden name. But the NFSv4.1 OPEN operation MAY return a result flag, OPEN4_RESULT_PRESERVE_UNLINKED, which tells the client the file will be preserved if it has an outstanding open (see Section 18.16) 🌈.

If the server finds that the file is still open when the REMOVE arrives:

The server SHOULD NOT delete the file's directory entry if the file was opened with OPEN4_SHARE_DENY_WRITE or OPEN4_SHARE_DENY_BOTH.
If the file was not opened with OPEN4_SHARE_DENY_WRITE or OPEN4_SHARE_DENY_BOTH, the server SHOULD delete the file's directory entry. However, until the last CLOSE of the file, the server MAY continue to allow access to the file via its filehandle 🦋.
The server MUST NOT delete the directory entry if the reply from OPEN had the flag OPEN4_RESULT_PRESERVE_UNLINKED set 🛑.

The server MAY implement its own restrictions on removing a file while it's open 👑. It might disallow such a REMOVE (or a removal that happens as part of RENAME). The things that can influence those restrictions include:

Whether certain access protocols (i.e., not just NFS) are holding the file open.
Whether particular options, access modes, or policies on the server are enabled.

If a file has an outstanding OPEN and that prevents removing the file's directory entry, the error NFS4ERR_FILE_OPEN gets returned 🚫.

Here's where it gets spicy 🌶️. Sometimes you can't definitively decide because delegations are being held. Those delegations MUST be recalled so the REMOVE processing can keep going 🔄. When a delegation is held, the server has no reliable knowledge of the status of OPENs for that client. So unless there are files opened with the particular deny modes by clients without delegations, the decision can't be made until delegations get recalled, and the operation can't proceed until each sufficient delegation has been returned or revoked so the server can make a correct call ⏳.

Whenever delegations get recalled, the server is likely to return one or more NFS4ERR_DELAY errors while those delegations are still outstanding. Hang tight, babe 💅.

If the current filehandle designates a directory for which another client holds a directory delegation, then, unless the situation can be resolved by sending a notification, the directory delegation MUST be recalled, and the operation MUST NOT proceed until the delegation is returned or revoked 🧁. Except where this happens super quickly, one or more NFS4ERR_DELAY errors will be returned to requests made while the delegation is still outstanding 😭.

When the current filehandle designates a directory with one or more directory delegations, then, if those delegations request notifications, NOTIFY4_REMOVE_ENTRY will be generated as a result of this operation 📣.

Note that when a remove happens as a result of a RENAME, NOTIFY4_REMOVE_ENTRY will only be generated if the removal happens as a separate operation 🎀. If the removal is integrated and atomic with the RENAME, the notification of the removal gets folded into the notification for the RENAME. See the discussion of the NOTIFY4_RENAME_ENTRY notification in Section 20.4 💖.

18.26. Operation 29: RENAME - Rename Directory Entry 🌷

18.26.1. ARGUMENTS

struct RENAME4args {
        /* SAVED_FH: source directory */
        component4      oldname;
        /* CURRENT_FH: target directory */
        component4      newname;
};

18.26.2. RESULTS

struct RENAME4resok {
        change_info4    source_cinfo;
        change_info4    target_cinfo;
};

union RENAME4res switch (nfsstat4 status) {
 case NFS4_OK:
        RENAME4resok    resok4;
 default:
        void;
};

18.26.3. DESCRIPTION

The RENAME operation renames the object named by oldname in the source directory (the one the saved filehandle points to, as set by the SAVEFH operation) to newname in the target directory (the one the current filehandle points to) 🦋. The operation has to be atomic to the client ⚡. Source and target directories MUST reside on the same file system on the server 🏠. On success, the current filehandle keeps pointing at the target directory 🎯.

If the target directory already has an entry with the name newname, the source object MUST be compatible with the target. Basically, either both are non-directories, or both are directories and the target MUST be empty 🧐. If they're compatible, the existing target gets removed before the rename happens or, even better, the target gets removed atomically as part of the rename ✨. See Section 18.25.4 for the client and server actions whenever a target is removed. But heads up: when the removal happens atomically with the rename, certain parts of that removal get folded into the rename itself 🧩. For example, the notification of the removal won't come as a NOTIFY4_REMOVE_ENTRY. It'll show up as part of the NOTIFY4_ADD_ENTRY or NOTIFY4_RENAME_ENTRY generated by the rename 📬.

If the source object and the target aren't compatible, or if the target is a directory that isn't empty, the server returns the error NFS4ERR_EXIST 🙅‍♀️.

If oldname and newname both refer to the same file (for example, they might be hard links of each other), then, unless the file is open (see Section 18.26.4), RENAME MUST perform no action and return NFS4_OK 🫧.

For both directories involved in the RENAME, the server returns change_info4 information. With the atomic field of the change_info4 data type, the server indicates whether the before and after change attributes were obtained atomically with respect to the rename 🌈.

If oldname refers to a named attribute and the saved and current filehandles refer to different file system objects, the server returns NFS4ERR_XDEV, just as if the saved and current filehandles represented directories on different file systems 🗺️.

If oldname or newname has a length of zero, or if either one doesn't obey the UTF-8 definition, the error NFS4ERR_INVAL gets returned 🙅‍♀️.

18.26.4. IMPLEMENTATION

okay so the server is allowed to put restrictions on RENAME 💅 It MAY say no to a RENAME if the file being renamed is open, or if that open came through particular protocols, or with particular options or access modes. Same deal if a file already exists with the target name and it's open. When RENAME gets rejected for any of those reasons, the error is NFS4ERR_FILE_OPEN 🔒

When oldname and rename refer to the same file, and that file is open in a way where RENAME would normally get rejected with NFS4ERR_FILE_OPEN if they were two different files, then RENAME SHOULD be rejected with NFS4ERR_FILE_OPEN too. Same energy, no cap 🫶

If a server does have restrictions like that, and they include NFSv4 opens that would stop a rename from working, the server needs to recall any delegations that could be hiding those opens 🕵️‍♀️ This is because when a client holds a delegation, the server might not have an accurate picture of the opens for that client, since the client can do OPENs and CLOSEs locally. The RENAME only has to wait until there's a definitive answer. For example, if there are several delegations and one of them has an open that would block the rename, NFS4ERR_FILE_OPEN can go back to the caller as soon as that one delegation comes back, without waiting on the others 🦋 And if those opens aren't tied to delegations at all, NFS4ERR_FILE_OPEN can be returned right away with no delegation recall happening.

If the current filehandle or the saved filehandle points at a directory that another client holds a directory delegation for, then, unless the situation can be fixed by sending a notification, the delegation MUST be recalled 📣 The operation can't move forward until that delegation is returned or revoked. Unless this happens super fast, one or more NFS4ERR_DELAY errors will get sent back to requests made while the delegation is still outstanding. Lowkey stressful 😭

When the current and saved filehandles are the same and they point at a directory with one or more directory delegations, then, if those delegations asked for notifications, a notification of type NOTIFY4_RENAME_ENTRY gets generated because of this operation 🌸 When oldname and rename refer to the same file, no notification is generated, because (as Section 18.26.3 says) the server MUST take no action 🙅‍♀️ When a file gets removed because it has the same name as the target, and that removal happens atomically with the rename, a NOTIFY4_REMOVE_ENTRY notification will not be generated. Instead, the deletion of the file gets reported as part of the NOTIFY4_RENAME_ENTRY notification ✨

When the current and saved filehandles are not the same:

If the current filehandle points at a directory with one or more directory delegations, then, if those delegations asked for notifications, NOTIFY4_ADD_ENTRY gets generated because of this operation 🎀 When a file gets removed because it has the same name as the target, and that removal happens atomically with the rename, a NOTIFY4_REMOVE_ENTRY notification will not be generated. Instead, the deletion of the file gets reported as part of the NOTIFY4_ADD_ENTRY notification.
If the saved filehandle points at a directory with one or more directory delegations, then, if those delegations asked for notifications, NOTIFY4_REMOVE_ENTRY gets generated because of this operation 🧚

If the object being renamed has file delegations held by clients other than the one doing the RENAME, the delegations MUST be recalled 🚨 The operation can't proceed until each of those delegations is returned or revoked. Note that for multiply linked files, the delegation recall requirement applies even if the delegation was obtained through a different name than the one being renamed 👀 In every case where delegations get recalled, the server is likely to return one or more NFS4ERR_DELAY errors while the delegation(s) stay outstanding, although it might not do that if the delegations come back quickly.

The RENAME operation must be atomic to the client 💜 The statement "source and target directories MUST reside on the same file system on the server" means that the fsid fields in the attributes for the directories are the same. If they live on different file systems, the error NFS4ERR_XDEV is returned. Not allowed, bestie 🙅‍♀️

Depending on the value of the fh_expire_type attribute for the object, the filehandle may or may not expire on a RENAME. However, server implementors are strongly encouraged to try to keep filehandles from expiring in this way 🌈

On some servers, the file names "." and ".." are illegal as either oldname or newname, and using them results in the error NFS4ERR_BADNAME 🙈 On lots of servers, the case where oldname or newname is an alias for the source directory also gets checked for. Those servers return the error NFS4ERR_INVAL in these cases.

If either of the source or target filehandles is not a directory, the server returns NFS4ERR_NOTDIR 📁

18.27. Operation 31: RESTOREFH - Restore Saved Filehandle

18.27.1. ARGUMENTS

/* SAVED_FH: */
void;

18.27.2. RESULTS

struct RESTOREFH4res {
        /*
         * If status is NFS4_OK,
         *     new CURRENT_FH: value of saved fh
         */
        nfsstat4        status;
};

18.27.3. DESCRIPTION

The RESTOREFH operation sets the current filehandle and stateid to the values in the saved filehandle and stateid 🔄 If there's no saved filehandle, the server returns the error NFS4ERR_NOFILEHANDLE 😬

See Section 16.2.3.1.1 for more details on the current filehandle.

See Section 16.2.3.1.2 for more details on the current stateid.

18.27.4. IMPLEMENTATION

Operations like OPEN and LOOKUP use the current filehandle to represent a directory and replace it with a new filehandle. Assuming the previous filehandle was saved with a SAVEFH operator, the previous filehandle can be restored as the current filehandle 🪄 This is commonly used to get post-operation attributes for the directory, for example:

      PUTFH (directory filehandle)
      SAVEFH
      GETATTR attrbits     (pre-op dir attrs)
      CREATE optbits "foo" attrs
      GETATTR attrbits     (file attributes)
      RESTOREFH
      GETATTR attrbits     (post-op dir attrs)

18.28. Operation 32: SAVEFH - Save Current Filehandle

18.28.1. ARGUMENTS

/* CURRENT_FH: */
void;

18.28.2. RESULTS

struct SAVEFH4res {
        /*
         * If status is NFS4_OK,
         *    new SAVED_FH: value of current fh
         */
        nfsstat4        status;
};

18.28.3. DESCRIPTION

The SAVEFH operation saves the current filehandle and stateid 💾 If a previous filehandle was saved, it's no longer accessible, so it's gone, bestie 👋 The saved filehandle can be restored as the current filehandle with the RESTOREFH operator.

On success, the current filehandle keeps its value ✨

See Section 16.2.3.1.1 for more details on the current filehandle.

See Section 16.2.3.1.2 for more details on the current stateid.

18.28.4. IMPLEMENTATION

(Nothing in this section yet, it's intentionally blank 🫧)

18.29. Operation 33: SECINFO - Obtain Available Security

18.29.1. ARGUMENTS

struct SECINFO4args {
        /* CURRENT_FH: directory */
        component4      name;
};

18.29.2. RESULTS

/*
 * From RFC 2203
 */
enum rpc_gss_svc_t {
        RPC_GSS_SVC_NONE        = 1,
        RPC_GSS_SVC_INTEGRITY   = 2,
        RPC_GSS_SVC_PRIVACY     = 3
};

struct rpcsec_gss_info {
        sec_oid4        oid;
        qop4            qop;
        rpc_gss_svc_t   service;
};

/* RPCSEC_GSS has a value of '6' - See RFC 2203 */
union secinfo4 switch (uint32_t flavor) {
 case RPCSEC_GSS:
         rpcsec_gss_info        flavor_info;
 default:
         void;
};

typedef secinfo4 SECINFO4resok<>;

union SECINFO4res switch (nfsstat4 status) {
 case NFS4_OK:
        /* CURRENTFH: consumed */
         SECINFO4resok resok4;
 default:
         void;
};

18.29.3. DESCRIPTION

SECINFO is how the client finds out which valid RPC authentication flavors it can use for a specific directory filehandle and file name pair 🔐 SECINFO should apply the same access methodology used for LOOKUP when it evaluates the name. So if the requester doesn't have the right access to LOOKUP the name, SECINFO MUST behave the same way and return NFS4ERR_ACCESS 🚫

The result is an array of the security mechanisms the server has available, in the server's order of preference, with the most preferred one first 🥇 The client is free to pick whatever mechanism it both wants and supports, or to go down the server's preference list and pick the first one it supports 💅 The array entries are represented by the secinfo4 structure. The field 'flavor' will contain a value of AUTH_NONE, AUTH_SYS (as defined in RFC 5531 [3]), or RPCSEC_GSS (as defined in RFC 2203 [4]). The field flavor can also be any other security flavor registered with IANA.

For the flavors AUTH_NONE and AUTH_SYS, no additional security information is returned. The same is true of many (if not most) other security flavors, including AUTH_DH 🌷 For a return value of RPCSEC_GSS, a security triple is returned that contains the mechanism object identifier (OID, as defined in RFC 2743 [7]), the quality of protection (as defined in RFC 2743 [7]), and the service type (as defined in RFC 2203 [4]). It's possible for SECINFO to return multiple entries with flavor equal to RPCSEC_GSS with different security triple values 🎀

On success, the current filehandle is consumed (see Section 2.6.3.1.1.8). If the next operation after SECINFO tries to use the current filehandle, that operation will fail with the status NFS4ERR_NOFILEHANDLE 😵

If the name has a length of zero, or if the name doesn't obey the UTF-8 definition (assuming UTF-8 capabilities are enabled; see Section 14.4), the error NFS4ERR_INVAL gets returned 🙅‍♀️

See Section 2.6 for additional information on the use of SECINFO 💜

18.29.4. IMPLEMENTATION ✨

okay so SECINFO is basically the client's go-to move 🔍 when another NFS operation comes back with NFS4ERR_WRONGSEC. That error is the server going "not with that security setup, bestie" 🙅‍♀️, meaning the server's security policy is different from what the client is currently using. So the client has to ask for the list of security flavors it can use and then pick whichever one fits its policies best 💅.

As I said, the server's security policies decide when a request gets NFS4ERR_WRONGSEC. Check out Table 14 for the list of operations that can return it 📋. Also, when READDIR returns attributes, the rdattr_error (Section 5.8.1.12) can contain NFS4ERR_WRONGSEC 👀. Now the main drama: CREATE and REMOVE MUST NOT return NFS4ERR_WRONGSEC 🚫. For CREATE, the target name can't have its own security policy unless it already exists, and the parent's policy was already checked when its filehandle got injected into the COMPOUND request's operations stream, so it's giving "already handled" 💁‍♀️. An OPEN that creates the target MUST NOT return NFS4ERR_WRONGSEC, for the same kind of reason 🫶. If the target name already exists, CREATE MUST return NFS4ERR_EXIST instead, and any separate policy on that target doesn't matter here 🙃. For REMOVE, the target might have its own policy, but it's getting deleted anyway, so the parent's policy wins 👑.

RENAME and LINK MAY return NFS4ERR_WRONGSEC, but only for the saved filehandle (see Section 2.6.3.1.2) 🦋. Any NFS4ERR_WRONGSEC on the current filehandle that LINK and RENAME run into MUST be returned by the PUTFH, PUTPUBFH, PUTROOTFH, or RESTOREFH operation that injected that current filehandle 🎀. Basically, the error has to come from whoever put the filehandle there in the first place 💖.

Besides LINK and RENAME, the operations that can return NFS4ERR_WRONGSEC are the exact moments when a client can inject a filehandle into the "current filehandle" on the server 🧚. That filehandle can come from the client directly (PUTFH, PUTPUBFH, PUTROOTFH), from a name-to-filehandle translation (LOOKUP and OPEN), or from the saved filehandle via RESTOREFH 🗝️. As Section 2.6.3.1.1.1 explains, a put filehandle operation followed by SAVEFH MUST NOT return NFS4ERR_WRONGSEC 🙅‍♀️. That's why RESTOREFH is allowed to return NFS4ERR_WRONGSEC under certain conditions (see Section 2.6.3.1.1), so security policies can still be enforced 🔐.

READDIR will not directly return NFS4ERR_WRONGSEC 🙈. But if the READDIR request asked for attributes, the request's security triple might not match a directory entry's. If that happens and the client asked for the rdattr_error attribute, the server puts NFS4ERR_WRONGSEC in rdattr_error for that entry instead 🌸.

Now, how the client fixes a NFS4ERR_WRONGSEC error 🛠️:

For LOOKUP and OPEN, the client uses SECINFO with the same current filehandle and name that were given in the original LOOKUP or OPEN, so it can list the available security triples 📝.
For the rdattr_error, the client uses SECINFO with the same current filehandle that was given in the original READDIR. The name passed to SECINFO is the directory entry (as returned from READDIR) that had NFS4ERR_WRONGSEC in its rdattr_error attribute 🍓.
For PUTFH, PUTROOTFH, PUTPUBFH, RESTOREFH, LINK, and RENAME, the client uses SECINFO_NO_NAME { style = SECINFO_STYLE4_CURRENT_FH }. The client puts the appropriate PUTFH, PUTPUBFH, or PUTROOTFH operation in front of SECINFO_NO_NAME, and that operation supplies the filehandle that was originally provided by the PUTFH, PUTPUBFH, PUTROOTFH, or RESTOREFH operation 🧁.
- NOTE: In NFSv4.0, the client was required to use SECINFO, and it had to rebuild the parent of the original filehandle and the component name of the original filehandle 😭. NFSv4.1 introduced SECINFO_NO_NAME, so no more rebuilding needed, which is honestly such a glow-up 🌈.
For LOOKUPP, the client uses SECINFO_NO_NAME { style = SECINFO_STYLE4_PARENT } and provides the filehandle that equals the one originally given to LOOKUPP 🧚‍♀️.

Check out Section 21 for the recommendations on which security flavor SECINFO and SECINFO_NO_NAME should use 🔒.

18.30. Operation 34: SETATTR - Set Attributes 💅

18.30.1. ARGUMENTS

   struct SETATTR4args {
           /* CURRENT_FH: target object */
           stateid4        stateid;
           fattr4          obj_attributes;
   };

18.30.2. RESULTS

   struct SETATTR4res {
           nfsstat4        status;
           bitmap4         attrsset;
   };

18.30.3. DESCRIPTION

SETATTR is the operation that changes one or more attributes of a file system object ✏️. The new attributes come with a bitmap, and the attribute values follow the bitmap in bit order 🎀.

The stateid argument is there to provide the byte-range locking context that's needed for SETATTR requests that set the size attribute 🔐. Changing the size changes the file's data, so it has the same locking requirements as a matching WRITE 📚. Any SETATTR that sets the size attribute is incompatible with a share reservation that specifies OPEN4_SHARE_DENY_WRITE 🚧. The area between the old end-of-file and the new end-of-file counts as modified, just like it would if that area had been the target of a WRITE. This matters when checking for conflicts with byte-range locks, but only where the server is implementing mandatory byte-range locking behavior 🦋. A valid stateid SHOULD always be specified 💖. When the file size attribute is not set, the special stateid made of all zero bits MAY be passed 🫧.

Whether the operation succeeds or fails, the server returns the attrsset bitmask to show which attributes (if any) were actually set 📋. The attrsset in the response is a subset of the attrmask field in the obj_attributes argument 🧩. On success, the current filehandle keeps its value 👑.

18.30.4. IMPLEMENTATION

If the request asks to set the owner attribute, the server SHOULD let the operation succeed when the object's current owner matches the value in the request 🙌. Some servers might prohibit setting the owner attribute unless the requester has the privilege to do so, so it's not one-size-fits-all 🤷‍♀️. If the server is lenient in this one case of matching owner values, the client implementation may be simplified when it creates an object (for example, an exclusive create via OPEN) and then does a SETATTR ✨.

The file size attribute is used to request changes to a file's size 📏. A value of zero truncates the file 🪄. A value smaller than the file's current size throws away the data from the new size to the end of the file 🗑️. A size greater than the current size adds logically zeroed data bytes to the end of the file 🧊. Servers are free to do this with unallocated bytes (holes) or with allocated data bytes set to zero 🌿. Clients should not make any assumptions about how a server implements this, beyond the fact that the bytes in the affected byte-range returned by READ will be zeroed 🙈. Servers MUST support extending the file size via SETATTR 💪.

SETATTR is not guaranteed to be atomic 😬. A failed SETATTR might partially change a file's attributes, which is exactly why the reply always includes the status and the list of attributes that were set 📝.

If the object whose attributes are being changed has a file delegation held by a client other than the one doing the SETATTR, the delegation(s) must be recalled 📞. The operation can't actually change an attribute until each of those delegations has been returned or revoked ⏳. Whenever delegations get recalled, the server is likely to return one or more NFS4ERR_DELAY errors while the delegation(s) are still outstanding 🕰️. It might not do that if the delegations come back quickly 🏃‍♀️.

If the object whose attributes are being set is a directory, and another client holds a directory delegation for that directory, then asynchronous notifications get generated when the set of changed attributes has a non-null intersection with the set of attributes that notification was requested for 🔔. These notifications are of type NOTIFY4_CHANGE_DIR_ATTRS and go to the appropriate client(s), but the SETATTR isn't held up waiting for them to be sent 🚀.

If the object whose attributes are being set is a member of a directory that another client holds a directory delegation for, asynchronous notifications get generated when the set of changed attributes has a non-null intersection with the set of attributes that notification was requested for 📣. These are of type NOTIFY4_CHANGE_CHILD_ATTRS and go to the appropriate clients, but once again the SETATTR doesn't wait around for them 💨.

Changing a file's size with SETATTR also indirectly changes the time_modify and change attributes ⏰. The client has to account for this, since size changes can lead to data deletion 🧨.

The attributes time_access_set and time_modify_set are write-only attributes built as a switched union, so the client can tell the server how to set the time values 🕒. If the switched union specifies SET_TO_CLIENT_TIME4, the client has supplied an nfstime4 to use for the operation 🎀. If the switch union doesn't specify SET_TO_CLIENT_TIME4, the server uses its own current time for the SETATTR operation 🖥️.

If the server's time and the client's time are different, programs that compare client time to file times can break 💀. A time synchronization protocol should be used to limit the client/server time skew ⌚.

Using a COMPOUND that has a VERIFY operation (which only specifies the change attribute), immediately followed by a SETATTR, is a way for a client to make a request that emulates the SETATTR guard mechanism from NFSv3 🛡️. The whole point of the guard is to avoid changing file attributes based on stale information 🙅‍♀️. So any delay between checking the guard condition and setting the attributes could ruin that, and so could the matching delay in the NFSv4 emulation ⚠️. That's why NFSv4.1 servers SHOULD take care to avoid such delays, as much as possible, when they run this kind of request 🧘‍♀️.

If the server doesn't support an attribute the client asked for, the server SHOULD return NFS4ERR_ATTRNOTSUPP 🙃.

SETATTR returns a mask of the attributes it actually set in every case 📋. That mask MUST NOT include attribute bits the client didn't ask to set 🚫. If the attribute masks in the request and the reply are equal, the status field in the reply MUST be NFS4_OK 🎉 periodt.

18.31. Operation 37: VERIFY - Verify Same Attributes 🧐

18.31.1. ARGUMENTS

   struct VERIFY4args {
           /* CURRENT_FH: object */
           fattr4          obj_attributes;
   };

18.31.2. RESULTS

   struct VERIFY4res {
           nfsstat4        status;
   };

18.31.3. DESCRIPTION ✨

okay so VERIFY is literally the "wait, are you sure??" check of NFS 💅 It makes sure the attributes are still what the client thinks they are before the rest of the COMPOUND request runs. If even one attribute doesn't match, the server MUST return NFS4ERR_NOT_SAME 🙈 Also, the current filehandle keeps its value after a successful VERIFY, so nothing gets swapped out from under you 🦋

18.31.4. IMPLEMENTATION 🎀

bestie, here's a super iconic way to use VERIFY 😭 The client wants to be sure the file it's about to delete is the exact file it thinks it is. So it runs this little series:

      PUTFH (directory filehandle)
      LOOKUP (file name)
      VERIFY (filehandle == fh)
      PUTFH (directory filehandle)
      REMOVE (file name)

ngl this is a lifesaver for stopping accidental deletions 🧚 BUT it's not foolproof. A second client could remove the file and make a brand-new one right in the middle of the sequence. It still helps avoid the unintended result though, so it's giving protection 💜

Now, if VERIFY includes a RECOMMENDED attribute and the server doesn't support that attribute for the file system object, the client gets NFS4ERR_ATTRNOTSUPP 🌸

And if VERIFY includes rdattr_error or any set-only attribute (like time_modify_set), the server returns NFS4ERR_INVAL. not me being picky about this one, it's a hard no 🚫💀

18.32. Operation 38: WRITE - Write to File 🍓

18.32.1. ARGUMENTS 🌈

enum stable_how4 {
        UNSTABLE4       = 0,
        DATA_SYNC4      = 1,
        FILE_SYNC4      = 2
};

struct WRITE4args {
        /* CURRENT_FH: file */
        stateid4        stateid;
        offset4         offset;
        stable_how4     stable;
        opaque          data<>;
};

18.32.2. RESULTS 👑

struct WRITE4resok {
        count4          count;
        stable_how4     committed;
        verifier4       writeverf;
};

union WRITE4res switch (nfsstat4 status) {
 case NFS4_OK:
         WRITE4resok    resok4;
 default:
         void;
};

18.32.3. DESCRIPTION ✨

okay the WRITE operation is how you write data to a regular file 📝 The target file is whatever the current filehandle points to. The offset says where the data goes, and an offset of zero means start at the very beginning of the file 🦋 The count is encoded as part of the opaque data parameter and tells you how many bytes to write. If count is zero, the WRITE succeeds and returns a count of zero, as long as permissions check out 💅 The server MAY write fewer bytes than the client asked for, so don't be shocked 😭

Now the stable parameter is the really juicy part 🔥 It tells the server how serious to be about saving the data:

If stable is FILE_SYNC4, the server MUST commit the data written plus all file system metadata to stable storage before it returns results. This matches NFSv2 semantics. Anything else is a protocol violation, so no shortcuts 🙅‍♀️
If stable is DATA_SYNC4, the server MUST commit all of the data to stable storage and enough of the metadata to retrieve the data before returning. The server implementor is free to do DATA_SYNC4 the same way as FILE_SYNC4, just with a possible performance drop 🐢
If stable is UNSTABLE4, the server is free to commit any part of the data and metadata to stable storage, including all or none of it, before replying. There's no guarantee whether or when uncommitted data gets saved later 🫠

The only promises the server makes here are that it won't destroy any data without changing the value of writeverf, and that it won't commit data and metadata at a level lower than the client requested 🫶

Next, stateid. Unless special stateids are being used, the stateid for a WRITE is a value from an earlier byte-range LOCK or OPEN, or the stateid tied to a delegation 🎀 It identifies the owners, if any, and the server uses it to check that the locks are still valid (like, not revoked) ✨

When it succeeds, you get back results. count is how many bytes actually got written to the file 💖 The server might write fewer than you asked for, and if so, the real number of bytes written starting at offset is what comes back 🌸

The server also tells you how committed the data and metadata are via committed. Check out Table 20 for the valid combos:

The server MAY commit the data at a stronger level than requested. 💜
The server MUST commit the data at a level at least as high as the one committed. 🔒

+============+===================================+
| stable     | committed                         |
+============+===================================+
| UNSTABLE4  | FILE_SYNC4, DATA_SYNC4, UNSTABLE4 |
+------------+-----------------------------------+
| DATA_SYNC4 | FILE_SYNC4, DATA_SYNC4            |
+------------+-----------------------------------+
| FILE_SYNC4 | FILE_SYNC4                        |
+------------+-----------------------------------+

    Table 20: Valid Combinations of the Fields
    Stable in the Request and Committed in the
                      Reply

The last piece is writeverf 🧚 It's a cookie the client uses to tell whether the server's instance state changed (say, after a restart) between a WRITE and a later WRITE or COMMIT. This cookie MUST stay unchanged during a single instance of the NFSv4.1 server, and MUST be unique between instances of the NFSv4.1 server 🦋 If the cookie changes, the client MUST assume that any data written with UNSTABLE4 for committed and an old writeverf has been lost and needs to be recovered 😱

So if a client sends UNSTABLE4 and the reply says committed is DATA_SYNC4 or UNSTABLE4, the client will follow up later with a COMMIT to sync the outstanding data and metadata to stable storage, barring client error 🍓 It's possible that a client crash or some other error means that COMMIT never reaches the server 💀

For a WRITE with a stateid of all zero bits, the server MAY let the WRITE go through subject to mandatory byte-range locks or the current share deny modes for the file 🔐 For a WRITE with a stateid of all bits equal to 1, the server MUST NOT let the WRITE bypass locking checks at the server. Otherwise it's treated the same as an all-zero stateid 🌈

And on success, the current filehandle keeps its value 👑✨

18.32.4. IMPLEMENTATION ✨

okay so the server can totally write fewer bytes than the client asked for 🫠 when that happens, the server SHOULD NOT return an error unless it wrote nothing at all. if it writes less than the number of bytes specified, the client has to send another WRITE to handle the leftover data 🎀

it's assumed that writing to a file updates the time_modified and change attributes 💅 but these attributes SHOULD NOT be changed unless the file's contents actually change. so a WRITE with count set to zero SHOULD NOT update time_modified or change either 🙈

stable storage is persistent storage that survives:

Repeated power failures.
Hardware failures (of any board, power supply, etc.).
Repeated software crashes and restarts.

ngl the definition doesn't cover the stable storage module itself failing 💀

the verifier exists so a client can tell apart different instances of an NFSv4.1 protocol server that might lose cached, uncommitted data 🔍 most of the time, it lets the client detect server restarts. this info is required so the client can safely figure out whether the server could have lost cached data. if the server crashes unexpectedly and the client has uncommitted data from earlier WRITE requests (sent with the stable argument set to UNSTABLE4, where the committed result also came back as UNSTABLE4), the server might not have flushed that cached data to stable storage 😱 the burden of recovery is on the client, which will need to retransmit the data to the server.

a suggested verifier is the time the server was last started (if restarting the server results in lost buffers) ⏰

the reply's committed field helps the client cache way more effectively ✨ if the server commits all WRITE requests to stable storage, it SHOULD return committed set to FILE_SYNC4, no matter what the stable field said in the arguments. a server using an NVRAM accelerator might choose this policy 🔥 the client can then discard cached data that's already been committed on the server, which makes its cache work better 💖

some implementations may return NFS4ERR_NOSPC instead of NFS4ERR_DQUOT when a user's quota is exceeded 🍓

if the current filehandle is of type NF4DIR, the server will return NFS4ERR_ISDIR. if the current file is a symbolic link, the error NFS4ERR_SYMLINK gets returned. otherwise, if the current filehandle doesn't designate an ordinary file, the server will return NFS4ERR_WRONG_TYPE 🙅‍♀️

this next part is the locking drama, bestie 🔒😭

if mandatory byte-range locking is in effect for the file, and the matching byte-range of the data being written is READ_LT or WRITE_LT locked by an owner that isn't associated with the stateid, the server MUST return NFS4ERR_LOCKED. when that happens, the client MUST check whether the owner tied to the stateid used with the WRITE operation has a conflicting READ_LT lock that overlaps the byte-range being written. if the stateid's owner has no conflicting READ_LT lock, the client SHOULD try to grab the right write byte-range lock via the LOCK operation before retrying the WRITE 🔁 once the WRITE finishes, the client SHOULD release that byte-range lock via LOCKU ✅

if the stateid's owner did have a conflicting READ_LT lock, the client has no choice but to return an error to the application that tried the WRITE 💀 that's because, since the owner had a READ_LT lock, either the server tried to temporarily upgrade that READ_LT lock to a WRITE_LT lock, or the server simply can't upgrade locks. if the server tried to upgrade and failed, it's pointless for the client to retry the upgrade via the LOCK operation, since another client might be trying to upgrade too 🙃 if two clients are stuck waiting to upgrade the same lock, they deadlock. and if the server has no upgrade capability, trying a LOCK operation to upgrade is pointless as well 🫠

if one or more other clients have delegations for the file being written, those delegations MUST be recalled, and the operation can't go ahead until those delegations are returned or revoked 🚫 unless that happens super quickly, one or more NFS4ERR_DELAY errors will come back for requests made while the delegation is still outstanding. normally, delegations won't get recalled because of a WRITE, since the recall happens as a result of an earlier OPEN. but since a WRITE can be done with a special stateid, the server needs to check for this case even though the client should have done an OPEN first 👀

18.33. Operation 40: BACKCHANNEL_CTL - Backchannel Control 🎀

18.33.1. ARGUMENT

typedef opaque gsshandle4_t<>;

struct gss_cb_handles4 {
        rpc_gss_svc_t           gcbp_service; /* RFC 2203 */
        gsshandle4_t            gcbp_handle_from_server;
        gsshandle4_t            gcbp_handle_from_client;
};

union callback_sec_parms4 switch (uint32_t cb_secflavor) {
case AUTH_NONE:
        void;
case AUTH_SYS:
        authsys_parms   cbsp_sys_cred; /* RFC 5531 */
case RPCSEC_GSS:
        gss_cb_handles4 cbsp_gss_handles;
};

struct BACKCHANNEL_CTL4args {
        uint32_t                bca_cb_program;
        callback_sec_parms4     bca_sec_parms<>;
};

18.33.2. RESULT

struct BACKCHANNEL_CTL4res {
        nfsstat4                bcr_status;
};

18.33.3. DESCRIPTION

BACKCHANNEL_CTL swaps out the backchannel's callback program number, and it adds RPCSEC_GSS handles for the backchannel to use (it doesn't replace them) 💅

the BACKCHANNEL_CTL arguments are basically a subset of the CREATE_SESSION parameters 🧚 in BACKCHANNEL_CTL, the bca_cb_program field and the bca_sec_parms field line up with csa_cb_program and csa_sec_parms in the CREATE_SESSION arguments (see Section 18.36).

BACKCHANNEL_CTL MUST appear in a COMPOUND that starts with SEQUENCE ✨

if the RPCSEC_GSS handle identified by gcbp_handle_from_server doesn't exist on the server, the server MUST return NFS4ERR_NOENT 🙈

if an RPCSEC_GSS handle is using the SSV context (see Section 2.10.9), then, since every SSV RPCSEC_GSS handle shares one common SSV GSS context, there are security considerations specific to this situation. they're covered in Section 2.10.10, so go read those too 🔐

18.34. Operation 41: BIND_CONN_TO_SESSION - Associate Connection with Session 🦋

18.34.1. ARGUMENT

enum channel_dir_from_client4 {
 CDFC4_FORE             = 0x1,
 CDFC4_BACK             = 0x2,
 CDFC4_FORE_OR_BOTH     = 0x3,
 CDFC4_BACK_OR_BOTH     = 0x7
};

struct BIND_CONN_TO_SESSION4args {
 sessionid4     bctsa_sessid;

 channel_dir_from_client4
                bctsa_dir;

 bool           bctsa_use_conn_in_rdma_mode;
};

18.34.2. RESULT

enum channel_dir_from_server4 {
 CDFS4_FORE     = 0x1,
 CDFS4_BACK     = 0x2,
 CDFS4_BOTH     = 0x3
};

struct BIND_CONN_TO_SESSION4resok {
 sessionid4     bctsr_sessid;

 channel_dir_from_server4
                bctsr_dir;

 bool           bctsr_use_conn_in_rdma_mode;
};

union BIND_CONN_TO_SESSION4res
 switch (nfsstat4 bctsr_status) {

 case NFS4_OK:
  BIND_CONN_TO_SESSION4resok
                bctsr_resok4;

 default:       void;
};

18.34.3. DESCRIPTION

BIND_CONN_TO_SESSION is how you link extra connections to a session 🔗 it MUST be sent on the connection being associated with the session, and it MUST be the only operation in the COMPOUND procedure 🙌 if SP4_NONE (Section 18.35) state protection is in use, any principal, security flavor, or RPCSEC_GSS context MAY be used to invoke the operation 🫶 if SP4_MACH_CRED is used, RPCSEC_GSS MUST be used with the integrity or privacy services, using the principal that created the client ID 🔐 and if SP4_SSV is used, RPCSEC_GSS with the SSV GSS mechanism (Section 2.10.9) and integrity or privacy MUST be used 💜

if the client opted for SP4_NONE state protection when the client ID was created, the client isn't required to use BIND_CONN_TO_SESSION to associate the connection with the session, unless it wants to associate the connection with the backchannel 🙂 with SP4_NONE protection, just sending a COMPOUND request with a SEQUENCE operation is enough to associate the connection with the session named in SEQUENCE ✨

the bctsa_dir field tells the server whether the client wants the connection associated with the fore channel, the backchannel, or both 🦋 the value CDFC4_FORE_OR_BOTH means the client wants both channels, but will accept the connection being associated with just the fore channel. the value CDFC4_BACK_OR_BOTH means the client wants both channels, but will accept just the backchannel. the server replies in bctsr_dir with which channel(s) the connection actually ended up on 📣 if the client specified CDFC4_FORE, the server MUST return CDFS4_FORE. if the client specified CDFC4_BACK, the server MUST return CDFS4_BACK. if the client specified CDFC4_FORE_OR_BOTH, the server MUST return CDFS4_FORE or CDFS4_BOTH. and if the client specified CDFC4_BACK_OR_BOTH, the server MUST return CDFS4_BACK or CDFS4_BOTH 🌈

for bctsa_use_conn_in_rdma_mode, check the description of the csa_use_conn_in_rdma_mode argument in the CREATE_SESSION operation (Section 18.36). for bctsr_use_conn_in_rdma_mode, check the description of csr_use_conn_in_rdma_mode in that same section 🧚

invoking BIND_CONN_TO_SESSION on a connection that's already associated with the specified session does nothing, and the server MUST respond with NFS4_OK 🌸 the exception is when the client is demanding changes to the set of channels the connection is associated with. in that case, the server MUST return NFS4ERR_INVAL 💀

18.34.4. IMPLEMENTATION ✨

okay so if a session's channel loses every connection, the client might need BIND_CONN_TO_SESSION to hook up a new one, depending on the client ID's state protection and what type of channel it is 🔗. If the server restarted and didn't keep the reply cache in stable storage, the server won't recognize the session ID 💀. The client will eventually have to call EXCHANGE_ID to make a brand-new client ID and session, no cap.

Now, suppose SP4_SSV state protection is in use and BIND_CONN_TO_SESSION is one of the operations in the spo_must_enforce set when the client ID was created (Section 18.35). Then things get messy if SET_SSV gets sent, no response comes back, and the last connection tied to the client ID drops 😭. The client, per the sessions model, MUST retry the SET_SSV. But it needs a new connection to do that, and it MUST associate that connection with the session through a BIND_CONN_TO_SESSION authenticated with the SSV GSS mechanism. The problem is that the RPCSEC_GSS message integrity codes use a subkey derived from the SSV as the key, and the SSV may have changed. There are multiple recovery strategies, but here's the single, general one I'm walking you through. 🧩

The client reconnects.
The client assumes that the SET_SSV was executed, and so sends BIND_CONN_TO_SESSION with the subkey (derived from the new SSV, i.e., what SET_SSV would have set the SSV to) used as the key for the RPCSEC_GSS credential message integrity codes.
If the request succeeds, this means that the original attempted SET_SSV did execute successfully. The client re-sends the original SET_SSV, which the server will reply to via the reply cache.
If the server returns an RPC authentication error, this means that the server's current SSV was not changed (and the SET_SSV was likely not executed). The client then tries BIND_CONN_TO_SESSION with the subkey derived from the old SSV as the key for the RPCSEC_GSS message integrity codes.
The attempted BIND_CONN_TO_SESSION with the old SSV should succeed. If so, the client re-sends the original SET_SSV. If the original SET_SSV was not executed, then the server executes it. If the original SET_SSV was executed but failed, the server will return the SET_SSV from the reply cache.

18.35. Operation 42: EXCHANGE_ID - Instantiate Client ID 🎀

EXCHANGE_ID swaps long-hand client and server identifiers (owners) and gives you access to a client ID, making one if needed. That client ID gets tied to the connection the operation runs on, so it's ready to go when CREATE_SESSION happens or when the connection is used to issue a request on an existing session for the current client 🦋.

18.35.1. ARGUMENT

const EXCHGID4_FLAG_SUPP_MOVED_REFER    = 0x00000001;
const EXCHGID4_FLAG_SUPP_MOVED_MIGR     = 0x00000002;

const EXCHGID4_FLAG_BIND_PRINC_STATEID  = 0x00000100;

const EXCHGID4_FLAG_USE_NON_PNFS        = 0x00010000;
const EXCHGID4_FLAG_USE_PNFS_MDS        = 0x00020000;
const EXCHGID4_FLAG_USE_PNFS_DS         = 0x00040000;

const EXCHGID4_FLAG_MASK_PNFS           = 0x00070000;

const EXCHGID4_FLAG_UPD_CONFIRMED_REC_A = 0x40000000;
const EXCHGID4_FLAG_CONFIRMED_R         = 0x80000000;

struct state_protect_ops4 {
        bitmap4 spo_must_enforce;
        bitmap4 spo_must_allow;
};

struct ssv_sp_parms4 {
        state_protect_ops4      ssp_ops;
        sec_oid4                ssp_hash_algs<>;
        sec_oid4                ssp_encr_algs<>;
        uint32_t                ssp_window;
        uint32_t                ssp_num_gss_handles;
};

enum state_protect_how4 {
        SP4_NONE = 0,
        SP4_MACH_CRED = 1,
        SP4_SSV = 2
};

union state_protect4_a switch(state_protect_how4 spa_how) {
        case SP4_NONE:
                void;
        case SP4_MACH_CRED:
                state_protect_ops4      spa_mach_ops;
        case SP4_SSV:
                ssv_sp_parms4           spa_ssv_parms;
};

struct EXCHANGE_ID4args {
        client_owner4           eia_clientowner;
        uint32_t                eia_flags;
        state_protect4_a        eia_state_protect;
        nfs_impl_id4            eia_client_impl_id<1>;
};

18.35.2. RESULT

struct ssv_prot_info4 {
    state_protect_ops4     spi_ops;
    uint32_t               spi_hash_alg;
    uint32_t               spi_encr_alg;
    uint32_t               spi_ssv_len;
    uint32_t               spi_window;
    gsshandle4_t           spi_handles<>;
};

union state_protect4_r switch(state_protect_how4 spr_how) {
    case SP4_NONE:
            void;
    case SP4_MACH_CRED:
            state_protect_ops4     spr_mach_ops;
    case SP4_SSV:
            ssv_prot_info4         spr_ssv_info;
};

struct EXCHANGE_ID4resok {
    clientid4        eir_clientid;
    sequenceid4      eir_sequenceid;
    uint32_t         eir_flags;
    state_protect4_r eir_state_protect;
    server_owner4    eir_server_owner;
    opaque           eir_server_scope<NFS4_OPAQUE_LIMIT>;
    nfs_impl_id4     eir_server_impl_id<1>;
};

union EXCHANGE_ID4res switch (nfsstat4 eir_status) {
case NFS4_OK:
    EXCHANGE_ID4resok      eir_resok4;

default:
    void;
};

18.35.3. DESCRIPTION

The client uses EXCHANGE_ID to register this particular instance of itself with the server, as represented by a client_owner4. But if the client_owner4 was already registered some other way (like Transparent State Migration), the client may still use EXCHANGE_ID to grab the client ID that was assigned before 🧚.

The client ID returned from this operation gets associated with the connection the EXCHANGE_ID is received on. It acts as a parent object for sessions the client creates on that connection, or sessions the connection is bound to. When those sessions make requests that create state, that state becomes associated with the client ID that was returned 🌸.

If the client_owner hasn't been registered before, the client ID has to be used first, along with the returned eir_sequenceid, to create an associated session using CREATE_SESSION 🍓.

If the flag EXCHGID4_FLAG_CONFIRMED_R is set in the result eir_flags, that means the client_owner was already registered and a further CREATE_SESSION isn't needed to confirm it. Of course, later CREATE_SESSION operations may still be needed for other reasons 💜.

The value eir_sequenceid sets an initial sequence value for the client ID that comes back. If a CREATE_SESSION has already been done, the client doesn't need this value, since sequencing of those requests is already established, so the client ignores it 🙈.

EXCHANGE_ID MAY be sent in a COMPOUND procedure that starts with SEQUENCE. But when a client communicates with a server for the first time, it won't have a session yet, so using SEQUENCE won't be possible. If EXCHANGE_ID is sent without a preceding SEQUENCE, then it MUST be the only operation in the COMPOUND procedure's request. If it isn't, the server MUST return NFS4ERR_NOT_ONLY_OP 🚫.

The eia_clientowner field is made up of a co_verifier field and a co_ownerid string. As noted in Section 2.4, co_ownerid identifies the client, and co_verifier specifies a particular incarnation of that client. An EXCHANGE_ID sent with a new incarnation of the client will lead to the server removing lock state of the old incarnation. On the other hand, when an EXCHANGE_ID sent with the current incarnation and co_ownerid does not result in an unrelated error, it will potentially update an existing client ID's properties or simply return information about the existing client_id. The latter would happen when this operation is done to the same server using different network addresses as part of creating trunked connections. 🛤️

A server MUST NOT provide the same client ID to two different incarnations of an eia_clientowner.

Along with the client ID and sequence ID, the server returns a server owner (eir_server_owner) and server scope (eir_server_scope). The former is used in connection with network trunking as described in Section 2.10.5. The latter lets clients figure out when client IDs sent by one server may be recognized by another in the event of file system migration (see Section 11.11.9 of the current document) 🌍.

The client ID returned by EXCHANGE_ID is only unique relative to the combination of eir_server_owner.so_major_id and eir_server_scope. So if two servers return the same client ID, the onus is on the client to tell the client IDs apart based on eir_server_owner.so_major_id and eir_server_scope. In the event two different servers claim matching server_owner.so_major_id and eir_server_scope, the client can use the verification techniques discussed in Section 2.10.5.1 to determine if the servers are distinct. If they are distinct, then the client will need to note the destination network addresses of the connections used with each server and use the network address as the final discriminator 🕵️‍♀️.

The server, as defined by the unique identity expressed in the so_major_id of the server owner and the server scope, needs to track several properties of each client ID it hands out. The properties apply to the client ID and all sessions associated with the client ID. They're derived from the arguments and results of EXCHANGE_ID. The client ID properties include:

The capabilities expressed by the following bits, which come from the results of EXCHANGE_ID:
- EXCHGID4_FLAG_SUPP_MOVED_REFER
- EXCHGID4_FLAG_SUPP_MOVED_MIGR
- EXCHGID4_FLAG_BIND_PRINC_STATEID
- EXCHGID4_FLAG_USE_NON_PNFS
- EXCHGID4_FLAG_USE_PNFS_MDS
- EXCHGID4_FLAG_USE_PNFS_DS

These properties may be updated by subsequent EXCHANGE_ID operations on confirmed client IDs, though the server MAY refuse to change them.

The state protection method used, one of SP4_NONE, SP4_MACH_CRED, or SP4_SSV, as set by the spa_how field of the arguments to EXCHANGE_ID. Once the client ID is confirmed, this property cannot be updated by subsequent EXCHANGE_ID operations.
For SP4_MACH_CRED or SP4_SSV state protection:
- The list of operations (spo_must_enforce) that MUST use the specified state protection. This list comes from the results of EXCHANGE_ID.
- The list of operations (spo_must_allow) that MAY use the specified state protection. This list comes from the results of EXCHANGE_ID.

Once the client ID is confirmed, these properties cannot be updated by subsequent EXCHANGE_ID requests.

For SP4_SSV protection:
- The OID of the hash algorithm. This property is represented by one of the algorithms in the ssp_hash_algs field of the EXCHANGE_ID arguments. Once the client ID is confirmed, this property cannot be updated by subsequent EXCHANGE_ID requests.
- The OID of the encryption algorithm. This property is represented by one of the algorithms in the ssp_encr_algs field of the EXCHANGE_ID arguments. Once the client ID is confirmed, this property cannot be updated by subsequent EXCHANGE_ID requests.

okay so this part is all about the SSV, which is basically the secret sauce that keeps a client's state safe 🔐💖

The length of the SSV. This shows up as the spi_ssv_len field in the EXCHANGE_ID results 🎀. Once the client ID is confirmed, later EXCHANGE_ID operations cannot update this property ✨.

There are REQUIRED and RECOMMENDED relationships between the length of the key of the encryption algorithm ("key length"), the length of the output of the hash algorithm ("hash length"), and the length of the SSV ("SSV length") 🧮🌸. Here's the tea:

key length MUST be <= hash length. The encryption keys are really subkeys derived from the SSV, and that derivation goes through the hash algorithm 🔑. If you picked an encryption algorithm whose key length was bigger than the hash output, you'd need padding, and that would weaken the encryption algorithm 💀.
hash length SHOULD be <= SSV length. The SSV is the key used to derive subkeys via an HMAC, and it's recommended that the key going into an HMAC be at least as long as the HMAC's hash output (see Section 3 of [52]) 🦋.
key length SHOULD be <= SSV length. This is a transitive result of the above two invariants 🔁.
key length SHOULD be >= hash length / 2. Subkey derivation is via an HMAC, and it's recommended that if the HMAC has to be truncated, it should not be truncated to less than half the hash length (see Section 4 of RFC 2104 [52]) ✂️.

Number of concurrent versions of the SSV the client and server will support (see Section 2.10.9) 🧚. This is represented by spi_window in the EXCHANGE_ID results, and later EXCHANGE_ID operations may update it 💜.
The client's implementation ID, represented by the eia_client_impl_id field of the arguments 🍓. Subsequent EXCHANGE_ID requests may update it.
The server's implementation ID, represented by the eir_server_impl_id field of the reply 🌈. Replies to subsequent EXCHANGE_ID requests may update it.

The eia_flags passed in the arguments and the eir_flags in the results let the client and server tell each other what they can do, and they also show how the client ID will be used 👑. Whether a bit is set or cleared on the arguments' flags does not force the server to set or clear the same bit on the results' side 🙈. Bits that aren't defined above cannot be set in the eia_flags field. If they are, the server MUST reject the operation with NFS4ERR_INVAL 🚫.

The EXCHGID4_FLAG_UPD_CONFIRMED_REC_A bit can only be set in eia_flags, and it's always off in eir_flags. The EXCHGID4_FLAG_CONFIRMED_R bit can only be set in eir_flags, and it's always off in eia_flags 💅.

If the server recognizes the co_ownerid and co_verifier as mapping to a confirmed client ID, it sets EXCHGID4_FLAG_CONFIRMED_R in eir_flags ✨. That flag is how a client can tell if the client ID it's trying to create already exists and is confirmed 🫶.

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is set in eia_flags, the client is trying to update the properties of an existing confirmed client ID 🔥. (If it wants to update an unconfirmed client ID instead, it MUST NOT set EXCHGID4_FLAG_UPD_CONFIRMED_REC_A 🙅‍♀️.) In that case it's RECOMMENDED that the client send the update EXCHANGE_ID operation in the same COMPOUND as a SEQUENCE, so the EXCHANGE_ID is executed exactly once 🎯. Whether the client can update the properties of its client ID depends on the state protection it picked when the client ID was created, and on the principal and security flavor it used when sending the EXCHANGE_ID operation 🔐. The situations described in items 6, 7, 8, or 9 of the second numbered list of Section 18.35.4 below will apply 📋. And if the operation succeeds and returns a client ID that is already confirmed, the server MUST set the EXCHGID4_FLAG_CONFIRMED_R bit in eir_flags 💖.

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is not set in eia_flags, the client is either establishing a new client ID, trunking data communication to the server (See Section 2.10.5), or updating properties of an unconfirmed client ID 🦋. The situations described in items 1, 2, 3, 4, or 5 of the second numbered list of Section 18.35.4 below will apply 🧚. If the operation succeeds and returns a client ID that was previously confirmed, the server MUST set the EXCHGID4_FLAG_CONFIRMED_R bit in eir_flags 🌸.

When the EXCHGID4_FLAG_SUPP_MOVED_REFER flag bit is set, the client is saying it can handle an NFS4ERR_MOVED error as part of a referral sequence 🧭. If it's not set, it's still legal for the server to do a referral sequence 🤷‍♀️. A server may use the fact that a client can't correctly respond to a referral to avoid referring that particular client. For instance, it may act as a proxy for that file system, at some cost in performance, although it is not obligated to do so 🎀. If the server will potentially perform a referral, it MUST set EXCHGID4_FLAG_SUPP_MOVED_REFER in eir_flags 📣.

When EXCHGID4_FLAG_SUPP_MOVED_MIGR is set, the client is saying it can handle an NFS4ERR_MOVED error as part of a file system migration sequence 🚚. If it's not set, it's still legal for the server to say a file system has moved when that really happens 🙃. A server may use the fact that a client can't respond correctly to a migration when it schedules which file systems to migrate, so it avoids migrating file systems that are actively being used 🌷. It may also hide real migrations from clients that can't deal with them, by acting as a proxy for a migrated file system for particular clients, at some cost in performance, although it is not obligated to do so 🪄. If the server will potentially perform a migration, it MUST set EXCHGID4_FLAG_SUPP_MOVED_MIGR in eir_flags 💜.

When EXCHGID4_FLAG_BIND_PRINC_STATEID is set, the client wants the server to bind the stateid to the principal 🔒. That means a principal that creates a stateid has to be the one that uses it 👑. If the server will do the binding, it returns EXCHGID4_FLAG_BIND_PRINC_STATEID. The server MAY return EXCHGID4_FLAG_BIND_PRINC_STATEID even if the client didn't ask for it 🙈. If an update to the client ID changes the value of the EXCHGID4_FLAG_BIND_PRINC_STATEID client ID property, the change only applies to new stateids 🍓. Existing stateids (and all stateids with the same "other" field) that were created with binding in force will keep binding in force 🫶. Existing stateids (and all stateids with the same "other" field) that were created with binding not in force will keep binding not in force 🌈.

The EXCHGID4_FLAG_USE_NON_PNFS, EXCHGID4_FLAG_USE_PNFS_MDS, and EXCHGID4_FLAG_USE_PNFS_DS bits are described in Section 13.1, and they tell the client which roles the client ID is for in a pNFS environment 🌐. The server MUST set one of the acceptable combinations of these bits (roles) in eir_flags, as specified in that section 🧩. The same client owner and server owner pair can have multiple roles ✨. Multiple roles can be tied to the same client ID or to different client IDs 🔁. So if a client sends EXCHANGE_ID from the same client owner to the same server owner more than once, but asks for different pNFS roles each time, the server might return different client IDs 🎭. Since different pNFS roles might have different client IDs, the client may ask for different properties for each role/client ID 💅.

The spa_how field of the eia_state_protect field says how the client wants to protect its client, locking, and session states from unauthorized changes (Section 2.10.8.3) 🔐🎀:

SP4_NONE. The client doesn't ask the NFSv4.1 server to enforce state protection. The NFSv4.1 server MUST NOT enforce state protection for the returned client ID 🙅‍♀️.
SP4_MACH_CRED. If spa_how is SP4_MACH_CRED, the client MUST send the EXCHANGE_ID operation with RPCSEC_GSS as the security flavor, and with a service of RPC_GSS_SVC_INTEGRITY or RPC_GSS_SVC_PRIVACY 🛡️. If SP4_MACH_CRED is specified, the client wants to use an RPCSEC_GSS-based machine credential to protect its state 💖. The server MUST note the principal the EXCHANGE_ID operation was sent with, and the GSS mechanism used 📝. These notes together make up the machine credential 🧾.

After the client ID is confirmed, as long as the lease associated with the client ID hasn't expired, a later EXCHANGE_ID operation that uses the same eia_clientowner.co_owner as the first EXCHANGE_ID MUST also use the same machine credential as the first EXCHANGE_ID 🔁. The server returns the same client ID for the later EXCHANGE_ID as it returned for the first one 🫶.

SP4_SSV. If spa_how is SP4_SSV, the client MUST send the EXCHANGE_ID operation with RPCSEC_GSS as the security flavor, and with a service of RPC_GSS_SVC_INTEGRITY or RPC_GSS_SVC_PRIVACY 🌸. If SP4_SSV is specified, the client wants to use the SSV to protect its state ✨. The server records the credential used in the request as the machine credential (as defined above) for the eia_clientowner.co_owner 🧚. The CREATE_SESSION operation that confirms the client ID MUST use the same machine credential 👑.

When a client specifies SP4_MACH_CRED or SP4_SSV, it also gives two lists of operations, each written as a bitmap 🗂️. The first list is spo_must_enforce, and it has the operations the client MUST send with the machine credential (if SP4_MACH_CRED protection is specified) or with the SSV-based credential (if SP4_SSV protection is used) 🔑. This is subject to the server confirming the list of operations in the result of EXCHANGE_ID 🦋. The client MUST send these operations with RPCSEC_GSS credentials that specify the RPC_GSS_SVC_INTEGRITY or RPC_GSS_SVC_PRIVACY security service 🛡️. Typically, the first list includes EXCHANGE_ID, CREATE_SESSION, DELEGPURGE, DESTROY_SESSION, BIND_CONN_TO_SESSION, and DESTROY_CLIENTID 📋. The client SHOULD NOT put any operations that require a filehandle in this list 🙅‍♀️, because the server's access policies MAY conflict with the client's choice, and then the client would be unable to access a subset of the server's namespace 😭.

Note that if SP4_SSV protection is specified, and the client indicates that CREATE_SESSION must be protected with SP4_SSV, there's a catch 🚨. The SSV can't exist without a confirmed client ID, so the first CREATE_SESSION MUST instead be sent using the machine credential, and the server MUST accept the machine credential 🫶💜.

okay so the server gets to pick its own enforcement list too 💅 There's a matching result, also called spo_must_enforce, for the operations where the server will require SP4_MACH_CRED or SP4_SSV protection 🔐 Normally the server's result just mirrors the client's argument, but the result MAY be different ✨ If the client asks for one or more operations from { EXCHANGE_ID, CREATE_SESSION, DELEGPURGE, DESTROY_SESSION, BIND_CONN_TO_SESSION, DESTROY_CLIENTID }, then the result spo_must_enforce MUST include the operations the client requested from that set 🫶

If spo_must_enforce in the results has BIND_CONN_TO_SESSION set, then connection binding enforcement is enabled 🔒 and the client MUST use the machine credential (if SP4_MACH_CRED protection is used) or the SSV credential (if SP4_SSV protection is used) on calls to BIND_CONN_TO_SESSION. no skipping that part, bestie 😭

The second list is spo_must_allow 🌸 It's the operations the client wants the option of sending with the machine credential or the SSV-based credential, even if the object those operations act on isn't owned by the machine or SSV credential. The matching result, also called spo_must_allow, is the list of operations the server will let the client use SP4_SSV or SP4_MACH_CRED credentials with. Normally the server's result equals the client's argument, but the result MAY be different 🦋

Why does spo_must_allow even exist? Okay, picture this, it's giving drama 🍿 The client ID is confirmed with EXCHGID4_FLAG_BIND_PRINC_STATEID, and it calls OPEN with the RPCSEC_GSS credentials of a normal user. Then the user's credentials expire and can't be renewed (like when a Kerberos ticket granting ticket expires, and the user has logged off and isn't getting a new ticket granting ticket). Now the client can't send CLOSE without the user's credentials 😩 So it has to either leave the state sitting on the server, or re-send EXCHANGE_ID with a new verifier to clear all the state. Unless, of course, it puts CLOSE on the list in spo_must_allow and the server agrees 💖 main character solution, honestly

The SP4_SSV protection parameters also have a few extra fields 🧚

ssp_hash_algs: This is the set of algorithms the client supports for computing the digests needed for the internal SSV GSS mechanism and for the SET_SSV operation. Each algorithm is specified as an object identifier (OID). The REQUIRED algorithms for a server are id-sha1, id-sha224, id-sha256, id-sha384, and id-sha512 [25] 🔥

Due to known weaknesses in id-sha1, it is RECOMMENDED that the client specify at least one algorithm within ssp_hash_algs other than id-sha1 🙈

The algorithm the server picks from that set comes back in spi_hash_alg, a field of spr_ssv_prot_info. The field spi_hash_alg is an index into the array ssp_hash_algs. And because of the known weaknesses in id-sha1, it is RECOMMENDED that the server not select it as long as ssp_hash_algs contains any other supported algorithm 💅

If the server doesn't support any of the offered algorithms, it returns NFS4ERR_HASH_ALG_UNSUPP. If ssp_hash_algs is empty, the server MUST return NFS4ERR_INVAL 💀

ssp_encr_algs: This is the set of algorithms the client supports for providing privacy protection for the internal SSV GSS mechanism. Each algorithm is specified as an OID. The REQUIRED algorithm for a server is id-aes256-CBC. The RECOMMENDED algorithms are id-aes192-CBC and id-aes128-CBC [26] ✨ The selected algorithm comes back in spi_encr_alg, an index into ssp_encr_algs. If the server doesn't support any of the offered algorithms, it returns NFS4ERR_ENCR_ALG_UNSUPP. If ssp_encr_algs is empty, the server MUST return NFS4ERR_INVAL.

Note that, due to the previously stated requirements and recommendations on the relationships between key length and hash length, some combinations of RECOMMENDED and REQUIRED encryption algorithm and hash algorithm either SHOULD NOT or MUST NOT be used 🙅‍♀️ Table 21 is the cheat sheet for the illegal and discouraged combos 👑

ssp_window: This is the number of SSV versions the client wants the server to maintain (basically, each successful call to SET_SSV makes a brand new version of the SSV 🦋). If ssp_window is zero, the server MUST return NFS4ERR_INVAL. The server responds with spi_window, which MUST NOT exceed ssp_window and MUST be at least one. Any requests on the backchannel or fore channel that use a version of the SSV outside the window will fail with an ONC RPC authentication error, and the requester will have to retry them with the same slot ID and sequence ID 😭

ssp_num_gss_handles: This is the number of RPCSEC_GSS handles the server should create that are based on the GSS SSV mechanism (see Section 2.10.9) 🌈 It is not the total number of RPCSEC_GSS handles for the client ID. Subsequent calls to EXCHANGE_ID will add more RPCSEC_GSS handles, so it's not one-and-done. The server responds with a list of handles in spi_handles. If the client asks for at least one handle and the server can't create it, the server MUST return an error. The handles in spi_handles aren't available for use until the client ID is confirmed, which could happen right away if EXCHANGE_ID returns EXCHGID4_FLAG_CONFIRMED_R, or upon successful confirmation from CREATE_SESSION ⭐

While a client ID can span all the connections connected to a server that shares the same eir_server_owner.so_major_id, the RPCSEC_GSS handles returned in spi_handles can only be used on connections to a server that returns the same eir_server_owner.so_major_id and eir_server_owner.so_minor_id on each connection. It's permissible for the client to set ssp_num_gss_handles to zero, since the client can create more handles with another EXCHANGE_ID call 🎀

Because each SSV RPCSEC_GSS handle shares a common SSV GSS context, there are security considerations specific to this situation, discussed in Section 2.10.10. Please read that section, it's not optional 🙈

The seq_window (see Section 5.2.3.1 of RFC 2203 [4]) of each RPCSEC_GSS handle in spi_handle MUST be the same as the seq_window of the RPCSEC_GSS handle used for the credential of the RPC request that the EXCHANGE_ID operation was sent as a part of 🫶

Here's Table 21, the illegal and discouraged combos, in all their glory 💅

   +======================+===========================+===============+
   | Encryption Algorithm | MUST NOT be combined with | SHOULD NOT be |
   |                      |                           | combined with |
   +======================+===========================+===============+
   | id-aes128-CBC        |                           | id-sha384,    |
   |                      |                           | id-sha512     |
   +----------------------+---------------------------+---------------+
   | id-aes192-CBC        | id-sha1                   | id-sha512     |
   +----------------------+---------------------------+---------------+
   | id-aes256-CBC        | id-sha1, id-sha224        |               |
   +----------------------+---------------------------+---------------+

                                 Table 21

The arguments include an array of up to one element in length called eia_client_impl_id. If eia_client_impl_id is present, it contains the information identifying the implementation of the client 🧑‍💻 Similarly, the results include an array of up to one element in length called eir_server_impl_id that identifies the implementation of the server. Servers MUST accept a zero-length eia_client_impl_id array, and clients MUST accept a zero-length eir_server_impl_id array 🌸

A possible use for implementation identifiers is diagnostic software that pulls this info to figure out interoperability problems, performance workload behaviors, or general usage statistics 📊 Since the whole point of having this info is planning or general diagnosis only, the client and server MUST NOT interpret this implementation identity information in a way that affects how the implementation interacts with its peer 🚫 The client and server aren't allowed to depend on the peer showing a particular behavior based on an implementation identifier, but they are required to interoperate as specified elsewhere in the protocol specification 🤝

Because some implementations might violate the protocol specification and interpret the identity information anyway, implementations MUST provide facilities that let the NFSv4 client and server be configured to set the contents of the nfs_impl_id structures they send to any specified value 🎀 Thank you for coming to my networking TED talk 💜

18.35.4. IMPLEMENTATION ✨

okay bestie, every client record on the server is a 5-tuple 🧚 and each piece has a job:

co_ownerid: the client identifier string, from the eia_clientowner structure of the EXCHANGE_ID4args structure. 🫶
co_verifier: a client-specific value that tracks incarnations (so a client restart counts as a brand new incarnation) 🦋. It comes from the eia_clientowner structure of the EXCHANGE_ID4args structure.
principal: the principal that was defined in the RPC header's credential and/or verifier when the client record was made 💜.
client ID: the shorthand client identifier. The server generates it and sends it back in the eir_clientid field of the EXCHANGE_ID4resok structure 🌸.
confirmed: a private field on the server that says whether the client record has been confirmed 🔥. A record is confirmed if a CREATE_SESSION operation has successfully confirmed it. If not, it's unconfirmed. An unconfirmed record gets made by an EXCHANGE_ID call. Any unconfirmed record that isn't confirmed within a lease period SHOULD be removed 💀.

Next up, these identifiers are special values for the fields in the records 👑:

ownerid_arg: the value of the eia_clientowner.co_ownerid subfield of the EXCHANGE_ID4args structure from the current request.
verifier_arg: the value of the eia_clientowner.co_verifier subfield of the EXCHANGE_ID4args structure from the current request.
old_verifier_arg: a value of the eia_clientowner.co_verifier field from a client record that came in on a previous request. This is distinct from verifier_arg.
principal_arg: the value of the RPCSEC_GSS principal for the current request.
old_principal_arg: a value of the principal of a client record, as defined by the RPC header's credential or verifier on a previous request. This is distinct from principal_arg.
clientid_ret: the value of the eir_clientid field that the server will return in the EXCHANGE_ID4resok structure for the current request.
old_clientid_ret: the value of the eir_clientid field that the server returned in the EXCHANGE_ID4resok structure for a previous request. This is distinct from clientid_ret.
confirmed: the client ID has been confirmed.
unconfirmed: the client ID has not been confirmed.

Okay, real talk, EXCHANGE_ID is non-idempotent, so we have to think about retries 😭. Retries can happen because of a client restart, a network partition, a malfunctioning router, and so on. Retries get identified by the value of the eia_clientowner field of EXCHANGE_ID4args, and the scenarios below explain how to handle each one 🎀.

The scenarios are described in terms of the client record(s) a server has for a given co_ownerid. Also, if the client ID was created with SP4_SSV state protection and EXCHANGE_ID is one of the operations in spo_must_allow, then the server MUST authorize EXCHANGE_IDs with the SSV principal in addition to the principal that created the client ID 🔒.

New Owner ID 🌱

If the server has no client records with eia_clientowner.co_ownerid matching ownerid_arg, and EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is not set in the EXCHANGE_ID, then a new shorthand client ID (let's call it clientid_ret) gets generated, and this unconfirmed record gets added to the server's state:

{ ownerid_arg, verifier_arg, principal_arg, clientid_ret,
unconfirmed }

After that, the server returns clientid_ret ✨.

Non-Update on Existing Client ID 🔁

If the server has this confirmed record, and the request does not have EXCHGID4_FLAG_UPD_CONFIRMED_REC_A set, then the request is a retry caused by a faulty router or a lost connection, or the client is trying to figure out whether it can do trunking 🧐.

{ ownerid_arg, verifier_arg, principal_arg, clientid_ret,
confirmed }

The record has been confirmed, so the client must have gotten the server's reply to the first EXCHANGE_ID request. The server has a confirmed record, and EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is not set, so the server returns the same result it did when the client ID's properties were last updated (or, if they were never updated, the result from when the client ID was created). The one possible exception is eir_server_owner.so_minor_id. The confirmed record stays unchanged 💅.

Client Collision 💥

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is not set, and the server has this confirmed record, then this request is probably the result of a random collision between the eia_clientowner.co_ownerid values of two different clients 👀.

{ ownerid_arg, *, old_principal_arg, old_clientid_ret, confirmed
}

If there's no state tied to old_clientid_ret, or there is state but its lease has expired, then this case works the same as the New Owner ID case of Section 18.35.4, Paragraph 7, Item 1. The confirmed record gets deleted, old_clientid_ret and its lock state get deleted, a new shorthand client ID is generated, and this unconfirmed record gets added to the server's state:

{ ownerid_arg, verifier_arg, principal_arg, clientid_ret,
unconfirmed }

After that, the server returns clientid_ret 🎉.

If old_clientid_ret has an unexpired lease with state, then no state for old_clientid_ret gets changed or deleted. The server returns NFS4ERR_CLID_INUSE to tell the client to retry with a different value for the eia_clientowner.co_ownerid subfield of EXCHANGE_ID4args 🙅‍♀️. The client record doesn't change.

Replacement of Unconfirmed Record 🔄

If the EXCHGID4_FLAG_UPD_CONFIRMED_REC_A flag is not set, and the server has this unconfirmed record, then the client is sending EXCHANGE_ID again for an unconfirmed client ID. It might be a retry, a client restart before the client ID got confirmed (that is, before CREATE_SESSION was called), or something else 🤷‍♀️.

{ ownerid_arg, *, *, old_clientid_ret, unconfirmed }

The properties of old_clientid_ret might be different from the ones in the current EXCHANGE_ID. Whether or not the properties are being updated, the server deletes the unconfirmed record to clear up any ambiguity, generates a new client ID (clientid_ret), and sets up this unconfirmed record:

{ ownerid_arg, verifier_arg, principal_arg, clientid_ret,
unconfirmed }

Client Restart 🏃‍♀️

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is not set, and the server has this confirmed client record, then this request is probably coming from a confirmed client that has restarted 🌈.

{ ownerid_arg, old_verifier_arg, principal_arg, old_clientid_ret,
confirmed }

The previous incarnation of the same client won't be making requests anymore. So once CREATE_SESSION confirms the new client ID, byte-range locks and share reservations should be released right away, instead of making the new incarnation wait for the lease on the previous incarnation to run out ⏰. Session state should also be removed, since if the client had kept that information across the restart, this request wouldn't have been sent. If the server supports neither the CLAIM_DELEGATE_PREV nor the CLAIM_DELEG_PREV_FH claim types, the associated delegations should be purged too. Otherwise, delegations are kept and recovery goes ahead according to Section 10.2.1 🛠️.

After processing, clientid_ret gets returned to the client, and this client record gets added:

{ ownerid_arg, verifier_arg, principal_arg, clientid_ret,
unconfirmed }

The confirmed record from before still exists, so the same ownerid_arg shows up in both a confirmed and an unconfirmed state at the same time 🙃. The number of states can drop to one once the server gets a suitable CREATE_SESSION or EXCHANGE_ID.

If the server later gets a successful CREATE_SESSION that confirms clientid_ret, then the server atomically destroys the confirmed record and makes the unconfirmed record confirmed, as described in Section 18.36.3 💖.
If the server instead later gets an EXCHANGE_ID with the client owner equal to ownerid_arg, one strategy is to just delete the unconfirmed record and process the EXCHANGE_ID as described in the whole of Section 18.35.4 🧁.

Update ✏️

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is set, and the server has this confirmed record, then this request is an attempt at an update.

{ ownerid_arg, verifier_arg, principal_arg, clientid_ret,
confirmed }

The record has been confirmed, so the client must have gotten the server's reply to the first EXCHANGE_ID request. The server allows the update, and the client record stays intact 💪.

Update but No Confirmed Record 🚫

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is set, and the server has no confirmed record for ownerid_arg, then the server returns NFS4ERR_NOENT and leaves any unconfirmed record intact 😬.

Update but Wrong Verifier ❌

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is set, and the server has this confirmed record, then this request is an illegal attempt at an update, maybe because of a retry from a previous client incarnation 🙈.

{ ownerid_arg, old_verifier_arg, *, clientid_ret, confirmed }

The server returns NFS4ERR_NOT_SAME and leaves the client record intact.

Update but Wrong Principal 🚨

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is set, and the server has this confirmed record, then this request is an illegal attempt at an update by an unauthorized principal 🛑.

{ ownerid_arg, verifier_arg, old_principal_arg, clientid_ret,
confirmed }

The server returns NFS4ERR_PERM and leaves the client record intact.

18.36. Operation 43: CREATE_SESSION - Create New Session and Confirm Client ID 🎀

18.36.1. ARGUMENT

Here's the CREATE_SESSION argument struct, the shape of the request you send to create a session and confirm the client ID 🧚‍♀️:

   struct channel_attrs4 {
           count4                  ca_headerpadsize;
           count4                  ca_maxrequestsize;
           count4                  ca_maxresponsesize;
           count4                  ca_maxresponsesize_cached;
           count4                  ca_maxoperations;
           count4                  ca_maxrequests;
           uint32_t                ca_rdma_ird<1>;
   };

   const CREATE_SESSION4_FLAG_PERSIST              = 0x00000001;
   const CREATE_SESSION4_FLAG_CONN_BACK_CHAN       = 0x00000002;
   const CREATE_SESSION4_FLAG_CONN_RDMA            = 0x00000004;

   struct CREATE_SESSION4args {
           clientid4               csa_clientid;
           sequenceid4             csa_sequence;

           uint32_t                csa_flags;

           channel_attrs4          csa_fore_chan_attrs;
           channel_attrs4          csa_back_chan_attrs;

           uint32_t                csa_cb_program;
           callback_sec_parms4     csa_sec_parms<>;
   };

18.36.2. RESULT 🎀

struct CREATE_SESSION4resok {
        sessionid4              csr_sessionid;
        sequenceid4             csr_sequence;

        uint32_t                csr_flags;

        channel_attrs4          csr_fore_chan_attrs;
        channel_attrs4          csr_back_chan_attrs;
};

union CREATE_SESSION4res switch (nfsstat4 csr_status) {
case NFS4_OK:
        CREATE_SESSION4resok    csr_resok4;
default:
        void;
};

18.36.3. DESCRIPTION 💅

okay bestie, this is the op the client uses to make brand-new session objects on the server ✨

CREATE_SESSION can be sent with or without a preceding SEQUENCE operation in the same COMPOUND procedure 🦋. If there's a SEQUENCE in front of it, the session it creates has no direct relation to the session named in that SEQUENCE 🙈. The two sessions might still be associated with the same client ID though, so it's not totally unrelated lol.

Now, if there's no SEQUENCE before it, then CREATE_SESSION MUST be the only operation in the COMPOUND procedure's request 🚨. If it isn't, the server MUST return NFS4ERR_NOT_ONLY_OP 💀. Not a vibe.

Besides making the session, CREATE_SESSION also does these two iconic things:

The first session created with a new client ID confirms the creation of that client's state on the server 👑. The server also returns the parameter values for the new session.
The connection that CREATE_SESSION is sent over gets associated with the session's fore channel 🧚.

Here's how the arguments and results work 🌸:

csa_clientid: This is the client ID the new session will be associated with. The matching result is csr_sessionid, which is the session ID of the new session 🍓.

csa_sequence: Each client ID serializes CREATE_SESSION through a per-client ID sequence number (see Section 18.36.4) 🔢. The matching result is csr_sequence, and it MUST be equal to csa_sequence 💖.

Next up are the three arguments where the client offers a value to be a property of the session. Except where stated otherwise, it is RECOMMENDED that the server accept the value 🫶. If it isn't acceptable, the server MAY use a different value 🦋. Either way, the server MUST return the value the session will actually use to the client, which will be either what the client offered or what the server is insisting on 📣.

csa_flags: This field holds a list of the following flag bits ✨:

CREATE_SESSION4_FLAG_PERSIST: If this flag is set, the client wants the server to provide a persistent reply cache 💾. For sessions where only idempotent operations will be used (like a read-only session), clients SHOULD NOT set CREATE_SESSION4_FLAG_PERSIST 🙅. If the server does not or cannot provide a persistent reply cache, the server MUST NOT set CREATE_SESSION4_FLAG_PERSIST in the field csr_flags 😤.

If the server is a pNFS metadata server, then for the reasons described in Section 12.5.2 it SHOULD support CREATE_SESSION4_FLAG_PERSIST if it supports the layout_hint (Section 5.12.4) attribute 🌈.

CREATE_SESSION4_FLAG_CONN_BACK_CHAN: If this flag is set in csa_flags, the client is asking for the connection that the CREATE_SESSION operation arrived on to be associated with the session's backchannel, in addition to its fore channel 🔁. If the server agrees, it sets CREATE_SESSION4_FLAG_CONN_BACK_CHAN in the result field csr_flags ✅. If the flag is not set in csa_flags, then CREATE_SESSION4_FLAG_CONN_BACK_CHAN MUST NOT be set in csr_flags 🚫.
CREATE_SESSION4_FLAG_CONN_RDMA: If this flag is set in csa_flags, and the connection the CREATE_SESSION operation arrived on is currently in non-RDMA mode but has the capability to operate in RDMA mode, then the client is asking the server to "step up" to RDMA mode on the connection 🚀. If the server agrees, it sets CREATE_SESSION4_FLAG_CONN_RDMA in the result field csr_flags. If the flag is not set in csa_flags, then CREATE_SESSION4_FLAG_CONN_RDMA MUST NOT be set in csr_flags. Also, once the server agrees to step up, it and the client MUST exchange all future traffic on that connection with RPC RDMA framing and not Record Marking ([32]) 🦖. Big plot twist energy.

csa_fore_chan_attrs, csa_back_chan_attrs: These fields apply to the attributes of the fore channel (which carries requests from the client to the server) and the backchannel (the channel that carries callback requests from the server to the client), respectively 🔀. The results come back in matching structures called csr_fore_chan_attrs and csr_back_chan_attrs 📦. The results set the attributes for each channel, and they apply to all later use of each channel in the session 💜. Each structure has these fields:

ca_headerpadsize: This is the maximum amount of padding the requester is willing to apply so that write payloads are aligned on some boundary at the replier 📐. For each channel, the server:
- will reply in ca_headerpadsize with its preferred value, or zero if padding is not in use, and
- MAY decrease this value but MUST NOT increase it 📉.
ca_maxrequestsize: This is the maximum size of a COMPOUND or CB_COMPOUND request that will be sent 📏. This size is the XDR encoded size of the request, including the RPC headers (with the security flavor credentials and verifiers), but it excludes any RPC transport framing headers 🙃. Picture a request coming over a non-RDMA TCP/IP connection with a single Record Marking header in front of it. The maximum allowable count encoded in that header will be ca_maxrequestsize 🧠. If a requester sends a request bigger than ca_maxrequestsize, the error NFS4ERR_REQ_TOO_BIG gets returned, as described in Section 2.10.6.4 💀. For each channel, the server MAY decrease this value but MUST NOT increase it 🙅‍♀️.
ca_maxresponsesize: This is the maximum size of a COMPOUND or CB_COMPOUND reply that the requester will accept from the replier, including RPC headers (see the ca_maxrequestsize definition) 📨. For each channel, the server MAY decrease this value, but MUST NOT increase it 🔽. However, if the client picks a value for ca_maxresponsesize so that a replier on a channel could never send a response, then the server SHOULD return NFS4ERR_TOOSMALL in the CREATE_SESSION reply 🙈. After the session is created, if a requester sends a request whose reply would be bigger than this value, the replier returns NFS4ERR_REP_TOO_BIG, as described in Section 2.10.6.4 🚨.
ca_maxresponsesize_cached: This works like ca_maxresponsesize, but it's the maximum size of a reply that will be stored in the reply cache (Section 2.10.6.1) 🗄️. For each channel, the server MAY decrease this value, but MUST NOT increase it ⬇️. If, in the reply to CREATE_SESSION, the ca_maxresponsesize_cached value for a channel is less than the ca_maxresponsesize value for that same channel, then that's a hint to the requester that it needs to be picky about which replies it tells the replier to cache 🎯. For example, large replies from non-idempotent operations (like COMPOUND requests with a READ operation) shouldn't get cached 🙅. The requester decides which replies to cache with an argument to the SEQUENCE operation (the sa_cachethis field, see Section 18.46) or the CB_SEQUENCE operation (the csa_cachethis field, see Section 20.9) 🎀. After the session is created, if a requester sends a request whose reply would be bigger than ca_maxresponsesize_cached, the replier returns NFS4ERR_REP_TOO_BIG_TO_CACHE, as described in Section 2.10.6.4 💅.
ca_maxoperations: This is the maximum number of operations the replier will accept in a COMPOUND or CB_COMPOUND 🔢. For the backchannel, the server MUST NOT change the value the client offers 🔒. For the fore channel, the server MAY change the requested value 🔄. After the session is created, if a requester sends a COMPOUND or CB_COMPOUND with more operations than ca_maxoperations, the replier MUST return NFS4ERR_TOO_MANY_OPS 🚫.
ca_maxrequests: This is the maximum number of concurrent COMPOUND or CB_COMPOUND requests the requester will send on the session 🏃‍♀️. Later requests each get a slot identifier from the requester, in the range zero to ca_maxrequests - 1 inclusive 🎟️. For the backchannel, the server MUST NOT change the value the client offers 🔒. For the fore channel, the server MAY change the requested value 🔄.
ca_rdma_ird: This array holds a maximum of one element 🧩. If the array has one element, then that element is the inbound RDMA read queue depth (IRD) 📊. For each channel, the server MAY decrease this value, but MUST NOT increase it 📉.

csa_cb_program: This is the ONC RPC program number the server MUST use in any callbacks it sends through the backchannel to the client 📞. The server MUST specify an ONC RPC program number equal to csa_cb_program and an ONC RPC version number equal to 4 in callbacks sent to the client 🎯. If a CB_COMPOUND is sent to the client, the server MUST use a minor version number of 1 🔢. There's no matching result for this one 🤷‍♀️.

csa_sec_parms: This field is an array of acceptable security credentials the server can use on the session's backchannel 🔐. Three security flavors are supported: AUTH_NONE, AUTH_SYS, and RPCSEC_GSS 🛡️. If AUTH_NONE is specified for a credential, then the client is authorizing the server to use AUTH_NONE on all callbacks for the session 🔓. If AUTH_SYS is specified, then the client is authorizing the server to use AUTH_SYS on all callbacks, using the credential specified in cbsp_sys_cred 🪪. If RPCSEC_GSS is specified, then the server is allowed to use the RPCSEC_GSS context specified in cbsp_gss_parms as the RPCSEC_GSS context in the credential of the RPC header of callbacks to the client 🧾. There's no matching result here either ✨.

The RPCSEC_GSS context for the backchannel is specified with a pair of values of data type gsshandle4_t 🗝️. The data type gsshandle4_t represents an RPCSEC_GSS handle, and it's precisely the same as the data type of the "handle" field of the rpc_gss_init_res data type defined in "Context Creation Response - Successful Acceptance", Section 5.2.3.1 of [4] 📚. Fully iconic, no notes 💜.

💖 The Handles Situation (cont.)

The first RPCSEC_GSS handle, gcbp_handle_from_server, is the fore handle the server handed back to the client ✨ (either in the handle field of data type rpc_gss_init_res or as one of the elements of the spi_handles field in the reply to EXCHANGE_ID) when the RPCSEC_GSS context got created on the server. The second handle, gcbp_handle_from_client, is the back handle 🦋 that the client maps the RPCSEC_GSS context to. The server can immediately use the value of gcbp_handle_from_client in the RPCSEC_GSS credential in callback RPCs. Like, that value can be used as the value of the field "handle" in data type rpc_gss_cred_t (see "Elements of the RPCSEC_GSS Security Protocol", Section 5 of [4]) in callback RPCs 🎀. The server MUST use the RPCSEC_GSS security service specified in gcbp_service, i.e., it MUST set the "service" field of the rpc_gss_cred_t data type in RPCSEC_GSS credential to the value of gcbp_service (see "RPC Request Header", Section 5.3.1 of [4]) 🔥.

If the RPCSEC_GSS handle identified by gcbp_handle_from_server does not exist on the server, the server will return NFS4ERR_NOENT 💀.

Within each element of csa_sec_parms, the fore and back RPCSEC_GSS contexts MUST share the same GSS context and MUST have the same seq_window (see Section 5.2.3.1 of RFC 2203 [4]). The fore and back RPCSEC_GSS context state are independent of each other as far as the RPCSEC_GSS sequence number goes (see the seq_num field in the rpc_gss_cred_t data type of Sections 5 and 5.3.1 of [4]) 🌸.

🚨 Security drama alert: If an RPCSEC_GSS handle is using the SSV context (see Section 2.10.9), then because each SSV RPCSEC_GSS handle shares a common SSV GSS context, there are security considerations specific to this situation discussed in Section 2.10.10. Read that section, bestie, it's not optional 😭

Once the session is created, the first SEQUENCE or CB_SEQUENCE received on a slot MUST have a sequence ID equal to 1; if not, the replier MUST return NFS4ERR_SEQ_MISORDERED 🧚.

18.36.4. IMPLEMENTATION ✨

Okay so, to describe a possible implementation, we use the same notation for client records that we introduced in the EXCHANGE_ID description, plus one addition:

clientid_arg: The value of the csa_clientid field of the CREATE_SESSION4args structure of the current request.

Since CREATE_SESSION is a non-idempotent operation, we need to think about retries showing up because of a client restart, a network partition, a malfunctioning router, etc. 🙈 For each client ID created by EXCHANGE_ID, the server keeps a separate reply cache (called the CREATE_SESSION reply cache) that's similar to the session reply cache used for SEQUENCE operations, with two distinctions:

First, this is a reply cache just for detecting and processing CREATE_SESSION requests for a given client ID.
Second, the size of the client ID reply cache is one slot (and as a result, the CREATE_SESSION request does not carry a slot number). This means that at most one CREATE_SESSION request for a given client ID can be outstanding 💅.

As previously stated, CREATE_SESSION can be sent with or without a preceding SEQUENCE operation. Even if a SEQUENCE precedes CREATE_SESSION, the server MUST maintain the CREATE_SESSION reply cache, which is separate from the reply cache for the session associated with a SEQUENCE. If CREATE_SESSION was originally sent by itself, the client MAY send a retry of the CREATE_SESSION operation within a COMPOUND preceded by a SEQUENCE. If CREATE_SESSION was originally sent in a COMPOUND that started with a SEQUENCE, then the client SHOULD send a retry in a COMPOUND that starts with a SEQUENCE that has the same session ID as the SEQUENCE of the original request. However, the client MAY send a retry in a COMPOUND that either has no preceding SEQUENCE, or has a preceding SEQUENCE that refers to a different session than the original CREATE_SESSION. This might be necessary if the client sends a CREATE_SESSION in a COMPOUND preceded by a SEQUENCE with session ID X, and session X no longer exists 🥺. Regardless, any retry of CREATE_SESSION, with or without a preceding SEQUENCE, MUST use the same value of csa_sequence as the original.

After the client gets a reply to an EXCHANGE_ID operation that contains a new, unconfirmed client ID, the server expects the client to follow up with a CREATE_SESSION operation to confirm the client ID. The server expects the value of csa_sequenceid in the arguments to that CREATE_SESSION to equal the value of the field eir_sequenceid that was returned in the results of the EXCHANGE_ID that returned the unconfirmed client ID. Before the server replies to that EXCHANGE_ID operation, it initializes the client ID slot to be equal to eir_sequenceid - 1 (accounting for underflow), and records a contrived CREATE_SESSION result with a "cached" result of NFS4ERR_SEQ_MISORDERED. With the client ID slot set up like that, the processing of the CREATE_SESSION operation is divided into four phases 🌈:

Client record look up. The server looks up the client ID in its client record table. If the server contains no records with client ID equal to clientid_arg, then most likely the client's state was purged during a period of inactivity, possibly due to a loss of connectivity. NFS4ERR_STALE_CLIENTID is returned, and no changes are made to any client records on the server. Otherwise, the server goes to phase 2.
Sequence ID processing. If csa_sequenceid is equal to the sequence ID in the client ID's slot, then this is a replay of the previous CREATE_SESSION request, and the server returns the cached result. If csa_sequenceid is not equal to the sequence ID in the slot, and is more than one greater (accounting for wraparound), then the server returns the error NFS4ERR_SEQ_MISORDERED, and does not change the slot. If csa_sequenceid is equal to the slot's sequence ID + 1 (accounting for wraparound), then the slot's sequence ID is set to csa_sequenceid, and the CREATE_SESSION processing moves on to the next phase. A subsequent new CREATE_SESSION call over the same client ID MUST use a csa_sequenceid that is one greater than the sequence ID in the slot.
Client ID confirmation. If this would be the first session for the client ID, the CREATE_SESSION operation serves to confirm the client ID. Otherwise, the client ID confirmation phase is skipped and only the session creation phase occurs. Any case in which there is more than one record with identical values for client ID represents a server implementation error. The operation in the potential valid cases is summarized as follows:
- Successful Confirmation. If the server has the following unconfirmed record, then this is the expected confirmation of an unconfirmed record: { ownerid, verifier, principal_arg, clientid_arg, unconfirmed }. As noted in Section 18.35.4, the server might also have the following confirmed record: { ownerid, old_verifier, principal_arg, old_clientid, confirmed }. The server schedules the replacement of both records with: { ownerid, verifier, principal_arg, clientid_arg, confirmed }. The processing of CREATE_SESSION continues on to session creation. Once the session is successfully created, the scheduled client record replacement is committed. If the session is not successfully created, then no changes are made to any client records on the server.
- Unsuccessful Confirmation. If the server has the following record, then either the client has changed principals after the previous EXCHANGE_ID request, or there has been a chance collision between shorthand client identifiers: { *, *, old_principal_arg, clientid_arg, * }. Neither of these cases is permissible. Processing stops and NFS4ERR_CLID_INUSE is returned to the client. No changes are made to any client records on the server.
Session creation. The server confirmed the client ID, either in this CREATE_SESSION operation or in a previous CREATE_SESSION operation. The server examines the remaining fields of the arguments.

The server creates the session by recording the parameter values used (including whether the CREATE_SESSION4_FLAG_PERSIST flag is set and has been accepted by the server) and allocating space for the session reply cache (if there is not enough space, the server returns NFS4ERR_NOSPC). For each slot in the reply cache, the server sets the sequence ID to zero, and records an entry containing a COMPOUND reply with zero operations and the error NFS4ERR_SEQ_MISORDERED. This way, if the first SEQUENCE request sent has a sequence ID equal to zero, the server can simply return what is in the reply cache: NFS4ERR_SEQ_MISORDERED. The client initializes its reply cache for receiving callbacks in the same way, and similarly, the first CB_SEQUENCE operation on a slot after session creation MUST have a sequence ID of one.

If the session state is created successfully, the server associates the session with the client ID provided by the client.

When a request that had CREATE_SESSION4_FLAG_CONN_RDMA set needs to be retried, the retry MUST be done on a new connection that is in non-RDMA mode. If properties of the new connection are different enough that the arguments to CREATE_SESSION need to change, then a non-retry MUST be sent. The server will eventually dispose of any session that was created on the original connection 🧹.

On the backchannel, the client and server might wish to have many slots, in some cases perhaps more than the fore channel, so they can deal with situations where the network link has high latency and is the primary bottleneck for responses to recalls. If so, and if the client provides too few slots to the backchannel, the server might limit the number of recallable objects it gives to the client 🐢.

Implementing RPCSEC_GSS callback support requires changes to both the client and server implementations of RPCSEC_GSS. One possible set of changes includes:

Adding a data structure that wraps the GSS-API context with a reference count.
New functions to increment and decrement the reference count. If the reference count is decremented to zero, the wrapper data structure and the GSS-API context it refers to would be freed.
Changing RPCSEC_GSS to create the wrapper data structure upon receiving a GSS-API context from gss_accept_sec_context() and gss_init_sec_context(). The reference count would be initialized to 1.
Adding a function to map an existing RPCSEC_GSS handle to a pointer to the wrapper data structure. The reference count would be incremented.
Adding a function to create a new RPCSEC_GSS handle from a pointer to the wrapper data structure. The reference count would be incremented.
Replacing calls from RPCSEC_GSS that free GSS-API contexts with calls to decrement the reference count on the wrapper data structure.

18.37. Operation 44: DESTROY_SESSION - Destroy a Session 🧚

18.37.1. ARGUMENT 🎀

struct DESTROY_SESSION4args {
        sessionid4      dsa_sessionid;
};

18.37.2. RESULT ✨

struct DESTROY_SESSION4res {
        nfsstat4        dsr_status;
};

18.37.3. DESCRIPTION 💅

okay so DESTROY_SESSION is the operation that closes the session and tosses its reply cache (if it even has one 💀). Any connections still hooked up to that session get immediately disassociated, no drama. If a connection has no sessions left after that, the server MAY close it. That's the server's call 🫶

Locks, delegations, layouts, wants, and the lease are all tied to the client ID, so DESTROY_SESSION doesn't touch any of them 🦋 It's giving very targeted energy.

DESTROY_SESSION MUST be invoked on a connection that's associated with the session being destroyed. bestie, don't send it from a random connection 🙅‍♀️ Also, if SP4_MACH_CRED state protection was set when the client ID was created, the RPCSEC_GSS principal that created the session MUST be the one that destroys it, and it has to use RPCSEC_GSS privacy or integrity. If SP4_SSV state protection was set instead, RPCSEC_GSS using the SSV mechanism (Section 2.10.9) MUST be used, with integrity or privacy. so much security, I'm obsessed 🔐

Now the COMPOUND stuff, which gets a little spicy 🌶️

If the COMPOUND request starts with SEQUENCE, and the sessionids in SEQUENCE and DESTROY_SESSION are the same, then:

DESTROY_SESSION MUST be the final operation in the COMPOUND request.
It is advisable to avoid placing DESTROY_SESSION in a COMPOUND request with other state-modifying operations, because DESTROY_SESSION will destroy the reply cache. 🙈
Because the session and its reply cache are destroyed, a client that retries the request may get an error in reply to the retry, even though the original request was successful. the retry is lowkey cursed 😭

If the COMPOUND request starts with SEQUENCE, and the sessionids are different, then DESTROY_SESSION can appear in any position of the COMPOUND request (except for the first position). The two sessionids can belong to different client IDs. two different clients, one request, zero chill 👑

If the COMPOUND request does not start with SEQUENCE, and DESTROY_SESSION is not the sole operation, then the server MUST return NFS4ERR_NOT_ONLY_OP. no sharing the stage, it has to be solo 🎤

Last one, the backchannel drama 🥀 If there's a backchannel on the session and the server has outstanding CB_COMPOUND operations for that session that haven't been replied to, then the server MAY refuse to destroy the session and return an error. If that happens and the backchannel is down, the server SHOULD return NFS4ERR_CB_PATH_DOWN to tell the client the backchannel needs fixing before the session can be destroyed. Otherwise, the error CB_BACK_CHAN_BUSY SHOULD be returned to say there are CB_COMPOUNDs that still need replies. The client SHOULD reply to all outstanding CB_COMPOUNDs before re-sending DESTROY_SESSION. finish your homework first, then try again 📚✨

18.38. Operation 45: FREE_STATEID - Free Stateid with No Locks ✨

18.38.1. ARGUMENT 🎀

struct FREE_STATEID4args {
        stateid4        fsa_stateid;
};

18.38.2. RESULT 💖

struct FREE_STATEID4res {
        nfsstat4        fsr_status;
};

18.38.3. DESCRIPTION 🌸

FREE_STATEID is how you free a stateid that no longer has any locks attached to it. That covers opens, byte-range locks, delegations, and layouts. This might happen because the client did LOCKU operations, or because the server revoked things. If there are still valid locks of any kind tied to the stateid, you'll get NFS4ERR_LOCKS_HELD and the stateid stays put, no freeing for you 🙅‍♀️

When a stateid that had revoked locks gets freed with FREE_STATEID, the client is basically saying "yep, I acknowledge I lost those locks" 🫶 That lets the server, once all the revoked state has been acknowledged, let that client reclaim locks again, without running into the edge conditions talked about in Section 8.4.2. it's giving closure, honestly 🦋

Once a successful FREE_STATEID is done for a stateid, any later use of that stateid gets you an NFS4ERR_BAD_STATEID error. it's gone gone, periodt 💀

18.39. Operation 46: GET_DIR_DELEGATION - Get a Directory Delegation ✨

18.39.1. ARGUMENT 🎀

typedef nfstime4 attr_notice4;

struct GET_DIR_DELEGATION4args {
        /* CURRENT_FH: delegated directory */
        bool            gdda_signal_deleg_avail;
        bitmap4         gdda_notification_types;
        attr_notice4    gdda_child_attr_delay;
        attr_notice4    gdda_dir_attr_delay;
        bitmap4         gdda_child_attributes;
        bitmap4         gdda_dir_attributes;
};

18.39.2. RESULT 🌈

struct GET_DIR_DELEGATION4resok {
        verifier4       gddr_cookieverf;
        /* Stateid for get_dir_delegation */
        stateid4        gddr_stateid;
        /* Which notifications can the server support */
        bitmap4         gddr_notification;
        bitmap4         gddr_child_attributes;
        bitmap4         gddr_dir_attributes;
};

enum gddrnf4_status {
        GDD4_OK         = 0,
        GDD4_UNAVAIL    = 1
};

union GET_DIR_DELEGATION4res_non_fatal
 switch (gddrnf4_status gddrnf_status) {
 case GDD4_OK:
  GET_DIR_DELEGATION4resok      gddrnf_resok4;
 case GDD4_UNAVAIL:
  bool                          gddrnf_will_signal_deleg_avail;
};

union GET_DIR_DELEGATION4res
 switch (nfsstat4 gddr_status) {
 case NFS4_OK:
  GET_DIR_DELEGATION4res_non_fatal      gddr_res_non_fatal4;
 default:
  void;
};

18.39.3. DESCRIPTION 🧚

GET_DIR_DELEGATION is how a client asks for a directory delegation 📂 The directory is whichever one is the current filehandle. The client can also say it wants the server to ping it when the directory changes in certain ways, by setting bits in a bitmap. The server may refuse to hand one out, and if it does, it returns NFS4ERR_DIRDELEG_UNAVAIL. If the server says yes, it returns a cookie verifier for that directory. If that verifier changes while the client is holding the delegation, the delegation gets recalled, unless the client asked to be notified about that event. the main character drama is real 🎭

The server also returns a directory delegation stateid, gddr_stateid, as part of the result. That stateid shows up in callback messages about the delegation, like notifications and recalls. The client uses it to give the delegation back, either voluntarily or when it gets recalled. Returning a delegation happens by calling the DELEGRETURN operation. bye bye delegation 👋

The server might not be able to support notifications for certain events. If the client asks for one of those, the server MUST inform the client that it can't do it, and it does that in the GET_DIR_DELEGATION reply by leaving the matching bit unset in the supported notifications bitmask, gddr_notification. The server MUST NOT add bits to gddr_notification that the client didn't request. no adding stuff nobody asked for, that's a hard boundary 🙅‍♀️✨

GET_DIR_DELEGATION works for both normal directories and named attribute directories 🌸

If the client sets gdda_signal_deleg_avail to TRUE, it's registering a "want" for a directory delegation with the server. If the delegation isn't available right now, and the server supports and will honor the "want", the results come back with gddrnf_will_signal_deleg_avail set to TRUE, and no error gets indicated. In that case, the client should expect a future CB_RECALLABLE_OBJ_AVAIL operation to say a directory delegation is available. If the server doesn't want to honor the "want", or can't, it returns NFS4ERR_DIRDELEG_UNAVAIL. And if the delegation is available immediately, the server SHOULD return it right in the response to the operation, instead of making the client wait for a callback 🎁

Now the tricky part 👀 When a client asks for a directory delegation while it already holds one for that directory (even if that delegation was recalled but not yet returned, or revoked by the server), the server MUST reply with gddr_status set to NFS4_OK, gddrnf_status set to GDD4_UNAVAIL, and gddrnf_will_signal_deleg_avail set to FALSE. The delegation the client already had stays intact, and its state doesn't change. The current stateid isn't changed either (see Section 16.2.3.1.2 for a description of the current stateid). it's a whole vibe of "nothing happens, and that's on purpose" 💜

18.39.4. IMPLEMENTATION 💖

Okay bestie, so directory delegations give you better cache consistency for namespace info ✨, and that happens through synchronous callbacks. A server has to support synchronous callbacks to support directory delegations at all 💅. On top of that, asynchronous notifications can cut down on network traffic and make things feel snappier for clients in certain situations 🦋.

Notifications are described in terms of potential changes to the directory 🌸. A client asks for events by setting one or more bits in gdda_notification_types. It can ask to hear about entries being added (by setting NOTIFY4_ADD_ENTRY), entries being removed (NOTIFY4_REMOVE_ENTRY), renames (NOTIFY4_RENAME_ENTRY), directory attribute changes (NOTIFY4_CHANGE_DIR_ATTRIBUTES), and cookie verifier changes (NOTIFY4_CHANGE_COOKIE_VERIFIER), and it does that by setting one or more of the matching bits in the gdda_notification_types field.

The client can also ask for notifications about changes to the attributes of directory entries (NOTIFY4_CHANGE_CHILD_ATTRIBUTES) so its attribute cache stays fresh 🧚. The thing to know is that changes to child attributes do not cause the delegation to be recalled. If a client cares about directory entry caching or negative name caching, it can set gdda_notification_types to match exactly what it needs, and the server will tell it about every change that would otherwise invalidate its name cache 💜. What kind of notification a client asks for might depend on the directory size, how fast it changes, and which applications are using the directory 👑. Listing every condition under which a client might ask for a notification is out of scope for this spec, so we're leaving that one alone 🙈.

For attribute notifications, the client sets bits in the gdda_dir_attributes bitmap to say which attributes it wants to hear about. If the server doesn't support notifications for changes to a certain attribute, it SHOULD NOT set that attribute in the supported attribute bitmap in the reply (gddr_dir_attributes) 🚫. The client also sets the attributes of directory entries it wants notifications for in the gdda_child_attributes bitmap, and the server says which of those it will actually notify about in gddr_child_attributes 🎀.

The client also gets a say in timing. It can ask for a notification as soon as an attribute change happens, or after a delay, by setting a delay factor. gdda_child_attr_delay covers attribute changes to directory entries, and gdda_dir_attr_delay covers attribute changes to the directory itself ⏰. If the delay factor is zero, that tells the server the client wants to hear about attribute changes the moment they happen ⚡. If the delay factor is N seconds, the server makes a best-effort guarantee that attribute updates get synchronized within N seconds 🍓. If the client asks for a delay factor the server doesn't support, or one that would make the server burn a lot of resources by sending a flood of notifications, the server should not commit to sending attribute notifications. That means it must not set the matching bits in gddr_child_attributes and gddr_dir_attributes in the response 😭.

Security check, bestie 🔐: the client MUST use a security tuple (Section 2.6.1) that the directory or its applicable ancestor (Section 2.6) is exported with. If it isn't, the server MUST return NFS4ERR_WRONGSEC to the operation that both comes right before GET_DIR_DELEGATION and sets the current filehandle (see Section 2.6.3.1). No skipping this one, periodt 💅.

Last but not least, the directory delegation covers all the entries in the directory except the parent entry 🌈. So if a directory and its parent both hold directory delegations, changes to the parent won't send a notification for the child, even though the child's parent entry points to the parent directory. Easy to miss, so keep it in mind 👀.

18.40. Operation 47: GETDEVICEINFO - Get Device Information ✨

18.40.1. ARGUMENT

struct GETDEVICEINFO4args {
        deviceid4       gdia_device_id;
        layouttype4     gdia_layout_type;
        count4          gdia_maxcount;
        bitmap4         gdia_notify_types;
};

18.40.2. RESULT

struct GETDEVICEINFO4resok {
        device_addr4    gdir_device_addr;
        bitmap4         gdir_notification;
};

union GETDEVICEINFO4res switch (nfsstat4 gdir_status) {
case NFS4_OK:
        GETDEVICEINFO4resok     gdir_resok4;
case NFS4ERR_TOOSMALL:
        count4                  gdir_mincount;
default:
        void;
};

18.40.3. DESCRIPTION

Okay, GETDEVICEINFO is the operation that hands back the pNFS storage device address info for a specific device ID 🦋. The client says which device it means by sending gdia_device_id and gdia_layout_type, which together uniquely identify the device 🔍. It also sends gdia_maxcount to cap how many bytes the result can be 🎀. That limit covers everything inside the GETDEVICEINFO4resok structure, XDR overhead included 📦. The server may return less data than that 💅. If the server can't fit any information under the gdia_maxcount limit, it returns the error NFS4ERR_TOOSMALL. BUT if gdia_maxcount is zero, NFS4ERR_TOOSMALL MUST NOT be returned 🙅‍♀️.

The da_layout_type field of the gdir_device_addr the server returns MUST be equal to the gdia_layout_type the client specified 🫶. If it isn't, the client SHOULD ignore the response as invalid and act as if the server returned an error, even if the client does support the layout type that came back 🙈.

The client also sends a notification bitmap, gdia_notify_types, for the device ID mapping notifications it's interested in 🔔. The server has to support device ID notifications for the request to have affect 🧐. The notification mask is built the same way as the bitmap for file attributes (Section 3.3.7). The bit positions come from the notify_device_type4 enumeration type (Section 20.12). Only two of its enumerated values apply to GETDEVICEINFO right now: NOTIFY_DEVICEID4_CHANGE and NOTIFY_DEVICEID4_DELETE (see Section 20.12) ✨.

The notification bitmap only applies to the device ID you named 🍓. If a client sends GETDEVICEINFO on the same deviceID more than once, the server uses the last notification bitmap for later notifications 💜. If the bitmap is zero or empty, the device ID's notifications are turned off 🔕.

If the client just wants to update or turn off notifications, it MAY send a GETDEVICEINFO with gdia_maxcount set to zero 🌸. In that case, if the device ID is valid, the da_addr_body field of the gdir_device_addr in the reply will have zero length 🫧.

If gdia_device_id is an unknown device ID, the server returns NFS4ERR_NOENT 😬. Otherwise, the device address info comes back in gdir_device_addr. Finally, if the server supports notifications for device ID mappings, the gdir_notification result holds a bitmap of which notifications it will actually send to the client (via CB_NOTIFY_DEVICEID, see Section 20.12) 🔔.

If NFS4ERR_TOOSMALL is returned, the results also include gdir_mincount. That value is the minimum size needed to get the device information 📏.

18.40.4. IMPLEMENTATION

Besides updating or turning off notifications, the other use case for gdia_maxcount set to zero is to validate a device ID ✅.

The client SHOULD request a notification for changes to, or deletion of, a device ID to device address mapping 🔔. That way the server can let the client switch to a new mapping gracefully, without pending I/O failing abruptly, and without forcing layouts that use the device ID to be recalled or revoked 🫶.

Here's where it gets a little dramatic 🎭. GETDEVICEINFO (and GETDEVICELIST) can race with CB_NOTIFY_DEVICEID. That means CB_NOTIFY_DEVICEID could arrive before the client gets and processes the response to GETDEVICEINFO or GETDEVICELIST. The analysis of this race depends on the fact that the server MUST NOT delete a device ID that a layout the client holds refers to 🔒.

CB_NOTIFY_DEVICEID deletes a device ID. If the client believes it has layouts that refer to that device ID, some layouts referring to the deleted device ID may have been revoked 😱. The client should send a TEST_STATEID request using the stateid for each layout that might have been revoked. If TEST_STATEID shows that any layouts were revoked, the client must recover from layout revocation as described in Section 12.5.6. If TEST_STATEID shows that at least one layout has not been revoked, the client should send a GETDEVICEINFO on the supposedly deleted device ID to check that it's really been deleted 🕵️‍♀️.

If GETDEVICEINFO says the device ID doesn't exist, the client assumes the server is faulty and recovers by sending an EXCHANGE_ID operation 🛠️. If GETDEVICEINFO says the device ID does exist, then the server is still faulty for sending a wrong device ID deletion notification, but how faulty it is doesn't require the client to create a new client ID 💅.

If the client doesn't have layouts that refer to the device ID, no harm done 🌷. The client should mark the device ID as deleted. Once GETDEVICEINFO or GETDEVICELIST results show that the device ID has actually been deleted, the device ID should be removed from the client's cache 🧹.

CB_NOTIFY_DEVICEID indicates that a device ID's device addressing mappings have changed 🔄. The client should assume that the results of an in-progress GETDEVICEINFO will be stale for that device ID once they arrive, so it should send another GETDEVICEINFO on the device ID 🔁.

18.41. Operation 48: GETDEVICELIST - Get All Device Mappings for a File System ✨

18.41.1. ARGUMENT

struct GETDEVICELIST4args {
        /* CURRENT_FH: object belonging to the file system */
        layouttype4     gdla_layout_type;

        /* number of deviceIDs to return */
        count4          gdla_maxdevices;

        nfs_cookie4     gdla_cookie;
        verifier4       gdla_cookieverf;
};

18.41.2. RESULT

struct GETDEVICELIST4resok {
        nfs_cookie4             gdlr_cookie;
        verifier4               gdlr_cookieverf;
        deviceid4               gdlr_deviceid_list<>;
        bool                    gdlr_eof;
};

union GETDEVICELIST4res switch (nfsstat4 gdlr_status) {
case NFS4_OK:
        GETDEVICELIST4resok     gdlr_resok4;
default:
        void;
};

18.41.3. DESCRIPTION ✨

okay so this op is how the client asks, "bestie, which device IDs does this server's file system use?" 🦋 It's giving enumeration energy. The client sends a current filehandle for a file object that belongs to the file system (so every file object sharing the same fsid as that filehandle counts) plus the layout type in gdia_layout_type. 💅 Since it might take more than one call to list everything (it's kinda like READDIR, Section 18.23), the client also sends gdia_cookie and gdia_cookieverf to mark where it is in the list. 🔖

Want to start from the very beginning of the device mappings? Set gdla_cookie to zero 🌸. When gdla_cookie is zero, the server MUST ignore gdla_cookieverf, iykyk 🙈. The client uses gdla_maxdevices to cap how many device IDs come back. If gdla_maxdevices is zero, the server MUST return NFS4ERR_INVAL 💀. The server MAY return fewer device IDs than you asked for, so don't panic if the list is short 🫶.

The successful response gives you the cookie gdlr_cookie and the cookie verifier gdlr_cookieverf to use on the next GETDEVICELIST. ⭐ If gdlr_eof is TRUE, that means the server has no more entries in its device list, so you're done 🎉. Each element of gdlr_deviceid_list is a device ID 🧚.

18.41.4. IMPLEMENTATION 🍓

This is especially useful for pNFS clients and servers that use LAYOUT4_BLOCK_VOLUME layouts. In those setups, it can be really helpful for a client to check device accessibility the first time it touches the file system 👑.

18.42. Operation 49: LAYOUTCOMMIT - Commit Writes Made Using a Layout 💖

18.42.1. ARGUMENT 🎀

   union newtime4 switch (bool nt_timechanged) {
   case TRUE:
           nfstime4           nt_time;
   case FALSE:
           void;
   };

   union newoffset4 switch (bool no_newoffset) {
   case TRUE:
           offset4           no_offset;
   case FALSE:
           void;
   };

   struct LAYOUTCOMMIT4args {
           /* CURRENT_FH: file */
           offset4                 loca_offset;
           length4                 loca_length;
           bool                    loca_reclaim;
           stateid4                loca_stateid;
           newoffset4              loca_last_write_offset;
           newtime4                loca_time_modify;
           layoutupdate4           loca_layoutupdate;
   };

18.42.2. RESULT 🦋

   union newsize4 switch (bool ns_sizechanged) {
   case TRUE:
           length4         ns_size;
   case FALSE:
           void;
   };

   struct LAYOUTCOMMIT4resok {
           newsize4                locr_newsize;
   };

   union LAYOUTCOMMIT4res switch (nfsstat4 locr_status) {
   case NFS4_OK:
           LAYOUTCOMMIT4resok      locr_resok4;
   default:
           void;
   };

18.42.3. DESCRIPTION 💅

okay, LAYOUTCOMMIT is the client saying "I'm done writing with this layout, bestie" 🫶. It commits changes to the layout for the current filehandle, the client ID (derived from the session ID in the SEQUENCE op right before it), the byte-range, and the stateid. Layouts can be subdivided, so you can commit a smaller slice of a layout you got through LAYOUTGET 🧩. The byte-range is set by loca_offset and loca_length. This byte-range MUST overlap with one or more existing layouts previously granted via LAYOUTGET (Section 18.43), each with an iomode of LAYOUTIOMODE4_RW. 🔥 If any held layout segment's iomode isn't LAYOUTIOMODE4_RW, the server should return NFS4ERR_BAD_IOMODE 😭. If the client doesn't hold matching layout segments for the byte-range, the server should return NFS4ERR_BAD_LAYOUT.

The client may have only written part of the range it asked for, and that's totally fine. LAYOUTCOMMIT lets it commit or discard provisionally allocated space and update the server with a new end-of-file 📝. The layout is still valid after the op finishes, so the client can keep referencing it by client ID, filehandle, byte-range, layout type, and stateid ✨.

If loca_reclaim is TRUE, the client is trying to commit changes to a layout after the metadata server restarted during its recovery grace period (see Section 12.7.4). 🚨 This can be necessary when the client has uncommitted writes to provisionally allocated byte-ranges that went to the storage devices before the restart. In that case, the layout the client sends MUST be a subset of a writable layout it held immediately before the metadata server restarted. The value of the field loca_stateid MUST be a value that the metadata server returned before it restarted. The metadata server is free to accept or reject this request based on its own internal metadata consistency checks. If the layout fails those checks, the metadata server MUST reject the request with the status NFS4ERR_RECLAIM_BAD 💀.

Here's the tea 🍵: a successful LAYOUTCOMMIT with loca_reclaim set to TRUE does not hand the client a layout for the file. It just commits the changes in the loca_layoutupdate field. To get an actual layout, the client must send a LAYOUTGET request after the server's grace period has expired. And if the metadata server gets a LAYOUTCOMMIT with loca_reclaim set to TRUE while it's not in its recovery grace period, it MUST reject the request with the status NFS4ERR_NO_GRACE 🙅‍♀️.

Setting loca_reclaim to TRUE is required if and only if the committed layout was acquired before the metadata server restart 🎯. If the client is committing a layout that was acquired during the metadata server's grace period, it MUST set the "reclaim" field to FALSE.

loca_stateid is a layout stateid, the kind returned by earlier successful layout operations (see Section 12.5.3) 🦋.

loca_last_write_offset tells the server the offset of the last byte the client wrote before the LAYOUTCOMMIT 🧾. This value is never equal to the file's size (at most it's one byte less than the file's size), and it MUST be less than or equal to NFS4_MAXFILEOFF. Also, loca_last_write_offset MUST overlap the range described by loca_offset and loca_length. The metadata server can use this to figure out whether the file's size needs updating. If the metadata server does update the file's size because of the LAYOUTCOMMIT, it must return the new size (locr_newsize.ns_size) as part of the results 📏.

loca_time_modify is the client's suggestion for the modification time it wants the server to set ⏰. The server can take the suggestion, or it can use the time of the LAYOUTCOMMIT operation instead. If the server uses the client-provided time, it should make sure time doesn't flow backwards 🕰️. Want to force an exact time? Then the client should send a SETATTR in a COMPOUND right after LAYOUTCOMMIT (see Section 12.5.4 for more details). And if the client wants to see the resulting modification time, it should build the COMPOUND so a GETATTR follows the LAYOUTCOMMIT 👀.

loca_layoutupdate is the mechanism for sending layout-specific updates to the metadata server 🧩. For example, the update can say which byte-ranges of the original layout were used and which ones can be deallocated. There is no NFSv4.1 file layout-specific layoutupdate4 structure, so no file-layout version here 🙈.

Block devices need way more detail than objects and files, because those two hide the messy block allocation behind their storage protocols 🧱. At the very minimum, the client needs to tell the server about changes to the end-of-file location and, if it wants, its view of the file's modification time. For block/volume layouts, though, it also needs to say precisely which blocks have been used 💖.

If the layout in the arguments doesn't exist, the error NFS4ERR_BADLAYOUT is returned (yes, no underscore, it's a different one from NFS4ERR_BAD_LAYOUT, so read carefully 🤓). The layout being committed may also be rejected if it doesn't correspond to an existing layout with an iomode of LAYOUTIOMODE4_RW.

On success, the current filehandle keeps its value, and so does the current stateid 🌈.

18.42.4. IMPLEMENTATION 🌸

The client MAY also use LAYOUTCOMMIT with loca_reclaim set to TRUE to pass along hints about modified file attributes, or to report layout-type specific info such as I/O errors for object-based storage layouts, just like it normally does during regular operation 🔍. Doing this may help the metadata server recover files more efficiently after a restart 🚀. For example, some file system implementations may need a big, expensive recovery of file system objects if the metadata server doesn't get a positive signal from every client holding a LAYOUTIOMODE4_RW layout that it finished all its writes. Sending a LAYOUTCOMMIT (if needed) and then following it with LAYOUTRETURN can give that signal and allow for graceful, efficient recovery 💜.

If loca_reclaim is TRUE, the metadata server is free to either examine or ignore the value in loca_stateid 🤷‍♀️. The metadata server implementation might or might not encode information into its layout stateid that lets it run a consistency check on the LAYOUTCOMMIT request 🔐.

18.43. Operation 50: LAYOUTGET - Get Layout Information 🧚

18.43.1. ARGUMENT 🎀

   struct LAYOUTGET4args {
           /* CURRENT_FH: file */
           bool                    loga_signal_layout_avail;
           layouttype4             loga_layout_type;
           layoutiomode4           loga_iomode;
           offset4                 loga_offset;
           length4                 loga_length;
           length4                 loga_minlength;
           stateid4                loga_stateid;
           count4                  loga_maxcount;
   };

18.43.2. RESULT 🦋

   struct LAYOUTGET4resok {
           bool               logr_return_on_close;
           stateid4           logr_stateid;
           layout4            logr_layout<>;
   };

   union LAYOUTGET4res switch (nfsstat4 logr_status) {
   case NFS4_OK:
           LAYOUTGET4resok     logr_resok4;
   case NFS4ERR_LAYOUTTRYLATER:
           bool                logr_will_signal_layout_avail;
   default:
           void;
   };

18.43.3. DESCRIPTION 💅

LAYOUTGET is how the client asks the metadata server for a layout so it can read or write the file given by the filehandle, covering the byte-range set by offset and length 📂. Layouts are identified by the client ID (derived from the session ID in the preceding SEQUENCE op), the current filehandle, the layout type (loga_layout_type), and the layout stateid (loga_stateid) 🔑. How loga_iomode gets used depends on the layout type, but it should reflect the client's data access intent 🎯.

If the metadata server is in a grace period and doesn't persist layouts and device ID to device address mappings, then it MUST return NFS4ERR_GRACE (see Section 8.4.2.1) 🚨.

okay so LAYOUTGET is the op where the client asks the server for a layout for a byte-range 🦋 and it's lowkey wild because the client actually asks for two ranges, both starting at the same spot in loga_offset 💅. The first one runs from loga_offset through loga_offset + loga_length - 1, both ends included. That's the range the client would love to have covered. The second runs from loga_offset through loga_offset + loga_minlength - 1, also inclusive, and that's the range the client needs covered. So loga_minlength MUST be less than or equal to loga_length ✨ no exceptions, bestie.

When a length field is set to NFS4_UINT64_MAX, it's giving "give me everything from loga_offset to the end-of-file, no matter how long the file is" 🌈. If it's loga_length, that's a wish. If it's loga_minlength, that's a hard requirement.

Here are the rules for how loga_length, loga_minlength, and loga_offset relate to each other and what their minimums are 🎀

If loga_length is less than loga_minlength, the metadata server MUST return NFS4ERR_INVAL. Rude 🙅‍♀️
If loga_minlength is zero, that's the client telling the server "I'll take any layout at loga_offset or lower that you have readily available." Readily is subjective 🫠 and depends on the layout type and the pNFS server implementation. For example, some metadata servers might have to pre-allocate stable storage when they get a request for a range that goes past the file's current length. If loga_minlength is zero and loga_length is greater than zero, that tells the server what range the client would prefer to have. If loga_length and loga_minlength are both zero, the client is saying it wants a layout of any length, where the ending offset is no less than loga_offset and the starting offset is at or below loga_offset 🍓. If the metadata server doesn't have a layout that's readily available, it MUST return NFS4ERR_LAYOUTTRYLATER 💀
If the sum of loga_offset and loga_minlength goes over NFS4_UINT64_MAX, and loga_minlength is not NFS4_UINT64_MAX, the error NFS4ERR_INVAL MUST result. Math said no 🙈
If the sum of loga_offset and loga_length goes over NFS4_UINT64_MAX, and loga_length is not NFS4_UINT64_MAX, the error NFS4ERR_INVAL MUST result. Same energy, different field 💜

After the metadata server has done all those checks on loga_offset, loga_minlength, and loga_offset, it MUST return a layout according to the rules in Table 22 📋

Acceptable layouts based on loga_minlength. Here's Table 22 🎀

   Acceptable layouts based on loga_minlength.  Note: u64m =
   NFS4_UINT64_MAX; a_off = loga_offset; a_minlen = loga_minlength.

   +===========+============+==========+==========+===================+
   | Layout    | Layout     | Layout   | Layout   | Layout length of  |
   | iomode of | a_minlen   | iomode   | offset   | reply             |
   | request   | of request | of reply | of reply |                   |
   +===========+============+==========+==========+===================+
   | _READ     | u64m       | MAY be   | MUST be  | MUST be >= file   |
   |           |            | _READ    | <= a_off | length - layout   |
   |           |            |          |          | offset            |
   +-----------+------------+----------+----------+-------------------+
   | _READ     | u64m       | MAY be   | MUST be  | MUST be u64m      |
   |           |            | _RW      | <= a_off |                   |
   +-----------+------------+----------+----------+-------------------+
   | _READ     | > 0 and <  | MAY be   | MUST be  | MUST be >=        |
   |           | u64m       | _READ    | <= a_off | MIN(file length,  |
   |           |            |          |          | a_minlen + a_off) |
   |           |            |          |          | - layout offset   |
   +-----------+------------+----------+----------+-------------------+
   | _READ     | > 0 and <  | MAY be   | MUST be  | MUST be >= a_off  |
   |           | u64m       | _RW      | <= a_off | - layout offset + |
   |           |            |          |          | a_minlen          |
   +-----------+------------+----------+----------+-------------------+
   | _READ     | 0          | MAY be   | MUST be  | MUST be > 0       |
   |           |            | _READ    | <= a_off |                   |
   +-----------+------------+----------+----------+-------------------+
   | _READ     | 0          | MAY be   | MUST be  | MUST be > 0       |
   |           |            | _RW      | <= a_off |                   |
   +-----------+------------+----------+----------+-------------------+
   | _RW       | u64m       | MUST be  | MUST be  | MUST be u64m      |
   |           |            | _RW      | <= a_off |                   |
   +-----------+------------+----------+----------+-------------------+
   | _RW       | > 0 and <  | MUST be  | MUST be  | MUST be >= a_off  |
   |           | u64m       | _RW      | <= a_off | - layout offset + |
   |           |            |          |          | a_minlen          |
   +-----------+------------+----------+----------+-------------------+
   | _RW       | 0          | MUST be  | MUST be  | MUST be > 0       |
   |           |            | _RW      | <= a_off |                   |
   +-----------+------------+----------+----------+-------------------+

                                 Table 22

If loga_minlength is not zero and the metadata server can't return a layout according to Table 22, it MUST return the error NFS4ERR_BADLAYOUT 😭 If loga_minlength is zero and the server can't or won't return a layout according to Table 22, it MUST return the error NFS4ERR_LAYOUTTRYLATER. As long as loga_length is greater than loga_minlength or equal to zero, the metadata server SHOULD return a layout according to the rules in Table 23 🦋

Desired layouts based on loga_length. The rules of Table 22 MUST be applied first ✨ Here's Table 23 💖

    Desired layouts based on loga_length.  The rules of Table 22 MUST be
    applied first.  Note: u64m = NFS4_UINT64_MAX; a_off = loga_offset;
    a_len = loga_length.

    +===============+==========+==========+==========+================+
    | Layout iomode | Layout   | Layout   | Layout   | Layout length  |
    | of request    | a_len of | iomode   | offset   | of reply       |
    |               | request  | of reply | of reply |                |
    +===============+==========+==========+==========+================+
    | _READ         | u64m     | MAY be   | MUST be  | SHOULD be u64m |
    |               |          | _READ    | <= a_off |                |
    +---------------+----------+----------+----------+----------------+
    | _READ         | u64m     | MAY be   | MUST be  | SHOULD be u64m |
    |               |          | _RW      | <= a_off |                |
    +---------------+----------+----------+----------+----------------+
    | _READ         | > 0 and  | MAY be   | MUST be  | SHOULD be >=   |
    |               | < u64m   | _READ    | <= a_off | a_off - layout |
    |               |          |          |          | offset + a_len |
    +---------------+----------+----------+----------+----------------+
    | _READ         | > 0 and  | MAY be   | MUST be  | SHOULD be >=   |
    |               | < u64m   | _RW      | <= a_off | a_off - layout |
    |               |          |          |          | offset + a_len |
    +---------------+----------+----------+----------+----------------+
    | _READ         | 0        | MAY be   | MUST be  | SHOULD be >    |
    |               |          | _READ    | <= a_off | a_off - layout |
    |               |          |          |          | offset         |
    +---------------+----------+----------+----------+----------------+
    | _READ         | 0        | MAY be   | MUST be  | SHOULD be >    |
    |               |          | _READ    | <= a_off | a_off - layout |
    |               |          |          |          | offset         |
    +---------------+----------+----------+----------+----------------+
    | _RW           | u64m     | MUST be  | MUST be  | SHOULD be u64m |
    |               |          | _RW      | <= a_off |                |
    +---------------+----------+----------+----------+----------------+
    | _RW           | > 0 and  | MUST be  | MUST be  | SHOULD be >=   |
    |               | < u64m   | _RW      | <= a_off | a_off - layout |
    |               |          |          |          | offset + a_len |
    +---------------+----------+----------+----------+----------------+
    | _RW           | 0        | MUST be  | MUST be  | SHOULD be >    |
    |               |          | _RW      | <= a_off | a_off - layout |
    |               |          |          |          | offset         |
    +---------------+----------+----------+----------+----------------+

                                  Table 23

The loga_stateid field is your valid stateid 🪪 If the client doesn't currently hold a layout, loga_stateid is a stateid for the matching open, byte-range lock, or delegation. Once the client holds a layout on the file, loga_stateid MUST be a stateid that came back from a previous LAYOUTGET or LAYOUTRETURN operation, or one handed over by a CB_LAYOUTRECALL operation (see Section 12.5.3) 🔥

The loga_maxcount field is the biggest layout (in bytes) the client can handle 🍰 If the layout structure is bigger than maxcount, the metadata server will return the NFS4ERR_TOOSMALL error 😬

The returned layout is an array called logr_layout, and each element is a layout4 🧚 If a file has a single striping pattern, logr_layout SHOULD contain just one entry. But if the requested range overlaps more than one striping pattern, logr_layout will contain as many entries as it needs. The elements of logr_layout MUST be sorted in ascending order by their lo_offset field. There MUST be no gaps or overlaps between two elements next to each other in logr_layout. And the lo_iomode field MUST be the same in every element 💜

Tables 22 and 23 both talk about a returned layout's iomode, offset, and length. Since the layout is encoded in the logr_layout array, these need a bit more explaining 🌸

iomode: The returned layout iomode in Tables 22 and 23 is equal to the lo_iomode value in each element of logr_layout. As the tables show, the metadata server MAY return a layout with an lo_iomode that's different from the requested iomode (the loga_iomode field of the request). If it does, it MUST make sure the lo_iomode is more permissive than the loga_iomode that was requested. For example, this lets an implementation upgrade LAYOUTIOMODE4_READ requests to LAYOUTIOMODE4_RW requests whenever it wants, as long as it stays within the limits of the layout type's specific protocol 🦋 Either LAYOUTIOMODE4_READ or LAYOUTIOMODE4_RW MUST be returned 🫶
offset: The returned layout offset in Tables 22 and 23 is always equal to the lo_offset field of the first element of logr_layout ✨
length: Setting the returned layout length is a little messy because the special layout length value NFS4_UINT64_MAX might be involved 🙃 For a logr_layout array with N elements, the lo_length field in the first N-1 elements MUST NOT be NFS4_UINT64_MAX. The lo_length field of the last element of logr_layout can be NFS4_UINT64_MAX under some conditions, which are listed next 👑
okay so if an applicable rule of Table 22 says the metadata server MUST return a layout of length NFS4_UINT64_MAX, then the lo_length field of the last element of logr_layout MUST be NFS4_UINT64_MAX 💖
and if an applicable rule of Table 22 says the metadata server MUST NOT return a layout of length NFS4_UINT64_MAX, then the lo_length field of the last element of logr_layout MUST NOT be NFS4_UINT64_MAX 🙅‍♀️
if an applicable rule of Table 23 says the metadata server SHOULD return a layout of length NFS4_UINT64_MAX, then the lo_length field of the last element of logr_layout SHOULD be NFS4_UINT64_MAX ✨
when the returned layout length from Table 22 and Table 23 is not NFS4_UINT64_MAX, the returned layout length is equal to the sum of the lo_length fields of each element of logr_layout 🧮

Okay, the logr_return_on_close result field is basically a directive that says "return the layout before closing the file" 🦋 When the metadata server sets this return value to TRUE, it MUST be prepared to recall the layout in case the client fails to return it before close. Bestie, this is the important part 😭 For a metadata server that already knows a layout must be returned before the file is closed, this return value is how it communicates that to the client, which removes one extra step from the client's and the metadata server's interaction. Main character energy for efficiency 💅

The logr_stateid stateid gets returned to the client so it can be used in later layout-related operations 🎀 See Sections 8.2, 12.5.3, and 12.5.5.2 for a further discussion and requirements, iykyk 🧚

The format of the returned layout (lo_content) is specific to the layout type 🌈 The value of the layout type (lo_content.loc_type) for each element of the array of layouts returned by the metadata server (logr_layout) MUST be equal to the loga_layout_type the client specified. If it is not equal, the client SHOULD ignore the response as invalid and behave as if the metadata server returned an error, even if the client does support the layout type that was returned 🙈

Now the error vibes, because this is where things get spicy 🌶️

If neither the requested file nor its containing file system supports layouts, the metadata server MUST return NFS4ERR_LAYOUTUNAVAILABLE 💀
If the layout type is not supported, the metadata server MUST return NFS4ERR_UNKNOWN_LAYOUTTYPE 👑
If layouts are supported but no layout matches the client-provided layout identification, the metadata server MUST return NFS4ERR_BADLAYOUT 😤
If an invalid loga_iomode is specified, or a loga_iomode of LAYOUTIOMODE4_ANY is specified, the metadata server MUST return NFS4ERR_BADIOMODE 🫠

If the layout for the file is unavailable because of transient conditions, for example file sharing prohibits layouts, the metadata server MUST return NFS4ERR_LAYOUTTRYLATER ⏳

If the layout request gets rejected because of an overlapping layout recall, the metadata server MUST return NFS4ERR_RECALLCONFLICT 💅 See Section 12.5.5.2 for details 🦋

If the layout conflicts with a mandatory byte-range lock held on the file, and the storage devices have no method of enforcing mandatory locks other than restricting layouts, the metadata server SHOULD return NFS4ERR_LOCKED 🔒

If the client sets loga_signal_layout_avail to TRUE, then it is registering a "want" for a layout with the client, in case the layout can't be obtained due to resource exhaustion 🫶 If the metadata server supports and will honor the "want", the results will have logr_will_signal_layout_avail set to TRUE. If so, the client should expect a CB_RECALLABLE_OBJ_AVAIL operation to indicate that a layout is available 🔥

On success, the current filehandle keeps its value, and the current stateid is updated to match the value returned in the results ✨

18.43.4. IMPLEMENTATION ✨

okay bestie, LAYOUTGET usually shows up in a COMPOUND request right after an OPEN 💅 and that's how the client ends up with location info for the file 🗺️. For that to work, loga_stateid has to be set to the special stateid that tells the metadata server to use the current stateid, which is the one OPEN just set (see Section 16.2.3.1.2) 🔑. A client can also hold onto a layout across multiple OPENs 🦋. It picks a layout type, which limits what kind of layout the metadata server will hand back, so the server can't give you something you literally can't use 🙅‍♀️.

As Table 22 and Table 23 show, the spec gives a pNFS client and server a lot of wiggle room 🌈. There are several ways a pNFS client can send LAYOUTGET, so here are some examples, iykyk 💖.

If LAYOUTGET comes right after OPEN in the same COMPOUND request and the OPEN asks for OPEN4_SHARE_ACCESS_READ access, the client might go for a _READ layout with loga_offset set to zero, loga_minlength set to zero, and loga_length set to NFS4_UINT64_MAX. If the file has space allocated to it, that space is striped over one or more storage devices, and there is either no conflicting layout or the concept of a conflicting layout doesn't apply to the pNFS server's layout type or implementation, then the metadata server might return a layout with a starting offset of zero and a length equal to the length of the file, if not NFS4_UINT64_MAX. If the length of the file is not a multiple of the pNFS server's stripe width (see Section 13.2 for a formal definition), the metadata server might round up the returned layout's length 📏.
If LAYOUTGET comes right after OPEN in the same COMPOUND request, and the OPEN asks for OPEN4_SHARE_ACCESS_WRITE access and does not truncate the file, the client might go for a _RW layout with loga_offset set to zero, loga_minlength set to zero, and loga_length set to the file's current length (if known), or NFS4_UINT64_MAX. Same as the last one, under some conditions the metadata server might return a layout that covers the entire length of the file or beyond ✍️.
This one is like the above, but the OPEN truncates the file 🪄. The client might expect to write from offset zero, so loga_offset and loga_minlength are set to zero, and loga_length is set to the value of threshold4_write_iosize. The metadata server might return a layout from offset zero with a length at least as long as threshold4_write_iosize.
A process on the client wants to read from offset 10000 for length 50000, and the client is using buffered I/O with 4096-byte buffers 📦. It plans to map that request into a series of READ requests starting at offset 8192. The end offset needs to be higher than 10000 + 50000 = 60000, and the next multiple of 4096 is 61440. The difference between 61440 and the layout's starting offset is 53248 (which is 4096 times 15). Since threshold4_read_iosize is less than 53248, the client sends a LAYOUTGET with loga_offset set to 8192, loga_minlength set to 53248, and loga_length set to the file's length (if known) minus 8192, or NFS4_UINT64_MAX if the file's length isn't known. Because this request goes over the metadata server's threshold, the server grants the layout, possibly with an initial offset of zero and an end offset of at least 8192 + 53248 - 1 = 61439, but preferably a layout with an offset aligned on the stripe width and a length that's a multiple of the stripe width 🎯.
Same deal as above, but the client isn't using buffered I/O, so all internal I/O requests go straight to the server 🚀. The LAYOUTGET has loga_offset equal to 10000 and loga_minlength set to 50000. The value of loga_length is set to the length of the file. The metadata server is free to return a layout that fully overlaps the requested range, with a starting offset and length aligned on the stripe width ✨.
Again, a process on the client wants to read from offset 10000 for length 50000 (so a range from 10000 to 69999), and buffered I/O is in use 🧁. The client expects the server might not be able to return a layout for the full I/O range. It plans to map the process's request into thirteen READ requests starting at offset 8192, each with length 4096, for a total length of 53248 (which equals 13 * 4096), which fully contains the range the process wants. Because threshold4_read_iosize is equal to 4096, it's practical and reasonable for the client to use several LAYOUTGET operations to finish the series of READs 🙌. The client sends a LAYOUTGET with loga_offset set to 8192, loga_minlength set to 4096, and loga_length set to 53248 or higher. The server will grant a layout possibly with an initial offset of zero and an end offset of at least 8192 + 4096 - 1 = 12287, but preferably a layout with an offset aligned on the stripe width and a length that's a multiple of the stripe width. This lets the client keep making forward progress, and it may send more LAYOUTGET operations for the rest of the range 🏃‍♀️.
An NFS client detects a sequential read pattern, so it sends a LAYOUTGET that goes well beyond any current or pending read requests 📖. The server might detect this pattern too and grant the LAYOUTGET request. Once the client reads from an offset that's 50% of the way through the range of the last layout it received, it sends more operations from the offset that's 50% of the way through that last layout, so it doesn't stall I/O waiting on a layout 😤. The client keeps asking for layouts with byte-ranges well ahead of the byte-ranges of recent and/or read requests from processes running on the client 🌸.
This one is like the above, except the client fails to detect the pattern but the server does 👀. The next time the metadata server gets a LAYOUTGET, it returns a layout with a length that's well beyond loga_minlength 🎁.
A client is using buffered I/O, has a long queue of write-behinds to process, and also detects a sequential write pattern ✏️. It sends a LAYOUTGET for a layout that spans the range of the queued write-behinds and well beyond, including ranges past the filer's current length. The client keeps sending LAYOUTGET operations once the write-behind queue reaches 50% of the maximum queue length 💜.

Once the client has obtained a layout referring to a particular device ID, the metadata server MUST NOT delete the device ID until the layout is returned or revoked 🚫. Important, do not skip this one.

CB_NOTIFY_DEVICEID can race with LAYOUTGET 🏎️. In one race, LAYOUTGET returns a device ID for which the client doesn't have device address mappings. The metadata server sends a CB_NOTIFY_DEVICEID to add the device ID to the client's awareness, and meanwhile the client sends GETDEVICEINFO on that device ID. This scenario is discussed in Section 18.40.4 🔍. In another scenario, the client processes CB_NOTIFY_DEVICEID before it processes the results from LAYOUTGET. The client will send a GETDEVICEINFO on the device ID. If the GETDEVICEINFO results arrive before the client gets the LAYOUTGET results, there's no race anymore 🎉. If the LAYOUTGET results arrive before the GETDEVICEINFO results, the client can either wait for the GETDEVICEINFO results or send another one to get possibly more up-to-date device address mappings for the device ID 🤷‍♀️.

18.44. Operation 51: LAYOUTRETURN - Release Layout Information 🦋

18.44.1. ARGUMENT 💅

/* Constants used for LAYOUTRETURN and CB_LAYOUTRECALL */
const LAYOUT4_RET_REC_FILE      = 1;
const LAYOUT4_RET_REC_FSID      = 2;
const LAYOUT4_RET_REC_ALL       = 3;

enum layoutreturn_type4 {
        LAYOUTRETURN4_FILE = LAYOUT4_RET_REC_FILE,
        LAYOUTRETURN4_FSID = LAYOUT4_RET_REC_FSID,
        LAYOUTRETURN4_ALL  = LAYOUT4_RET_REC_ALL
};

struct layoutreturn_file4 {
        offset4         lrf_offset;
        length4         lrf_length;
        stateid4        lrf_stateid;
        /* layouttype4 specific data */
        opaque          lrf_body<>;
};

union layoutreturn4 switch(layoutreturn_type4 lr_returntype) {
        case LAYOUTRETURN4_FILE:
                layoutreturn_file4      lr_layout;
        default:
                void;
};

struct LAYOUTRETURN4args {
        /* CURRENT_FH: file */
        bool                    lora_reclaim;
        layouttype4             lora_layout_type;
        layoutiomode4           lora_iomode;
        layoutreturn4           lora_layoutreturn;
};

18.44.2. RESULT 🎀

union layoutreturn_stateid switch (bool lrs_present) {
case TRUE:
        stateid4                lrs_stateid;
case FALSE:
        void;
};

union LAYOUTRETURN4res switch (nfsstat4 lorr_status) {
case NFS4_OK:
        layoutreturn_stateid    lorr_stateid;
default:
        void;
};

18.44.3. DESCRIPTION ✨

okay bestie, so this operation is how the client hands one or more layouts back to the server 🦋. each layout gets identified by the client ID (which comes from the session ID in the SEQUENCE op right before it 💅), the lora_layout_type, and the lora_iomode. when lr_returntype is LAYOUTRETURN4_FILE, the layout is also pinned down by the current filehandle, lrf_offset, lrf_length, and lrf_stateid. and if lrf_length is NFS4_UINT64_MAX, then every byte of the layout, starting at lrf_offset, gets returned 😭

when lr_returntype is LAYOUTRETURN4_FSID, the current filehandle is used to figure out the file system, and all the layouts matching the client ID, the fsid of that file system, lora_layout_type, and lora_iomode get returned 🌸. when it's LAYOUTRETURN4_ALL, all layouts matching the client ID, lora_layout_type, and lora_iomode are returned, and the current filehandle isn't used at all 🧚. after this call, the client MUST NOT use the returned layout(s) and the associated storage protocol to access the file data. no more poking at that data once it's gone, bestie 💀

if the set of layouts for LAYOUTRETURN4_FSID or LAYOUTRETURN4_ALL is empty, no error happens 🫶. for LAYOUTRETURN4_FILE, the byte-range you name gets returned even if it's only a piece of a layout you got from LAYOUTGET, a mix of several LAYOUTGET layouts, or a mix that includes some LAYOUTGET layouts plus pieces of others 🍓. and if the byte-range doesn't cover any bytes the client actually holds a layout for (for that file, client ID, layout type, and mode), still no error ✨ see Section 12.5.5.2.1.5 for the things to think about with "bulk" return of layouts.

the layout you give back can be a subset or a superset of what CB_LAYOUTRECALL asked for 👑. but if it's a subset, the recall isn't finished until the whole recalled scope has come back. recalled scope means the byte-range for LAYOUTRETURN4_FILE, or the use of LAYOUTRETURN4_FSID, or the use of LAYOUTRETURN4_ALL. there MUST be a LAYOUTRETURN with a matching scope to complete the return, even if every current layout range was already returned one at a time 🔥

for every lr_returntype value, an iomode of LAYOUTIOMODE4_ANY means that all layouts matching the other arguments are being returned. that's the client ID, lora_layout_type, and either the current filehandle plus a range, or the fsid derived from the current filehandle, or LAYOUTRETURN4_ALL 🌈

when lr_returntype is LAYOUTRETURN4_FILE, the lrf_stateid the client sends has to be a layout stateid, like one that came back from earlier layout operations. heads up: the "seqid" field of lrf_stateid MUST NOT be zero 🙈 see Sections 8.2, 12.5.3, and 12.5.5.2 for more discussion and requirements.

returning a layout (or all of them) does NOT wipe out the mapping of a storage device ID to a storage device address 💜 that mapping sticks around until it's changed or deleted through device ID notification callbacks. the one exception: if no remaining layouts point at a device ID, the server is free to delete that device ID without a callback. that's basically the situation when notifications aren't in effect.

if lora_reclaim is set to TRUE, the client is trying to return a layout it got before the metadata server restarted, during the server's grace period ⭐ when returning layouts that were acquired during the grace period, the client MUST set lora_reclaim to FALSE. lora_reclaim MUST be set to FALSE too when lr_layoutreturn is LAYOUTRETURN4_FSID or LAYOUTRETURN4_ALL. see LAYOUTCOMMIT (Section 18.42) for more details.

layouts can come back because they were recalled, or voluntarily, before the server ever recalls them 🦋 either way, the client must properly pass along any state changes made under the layout to the storage device(s) or the metadata server before returning it.

if the client returns the layout because of a CB_LAYOUTRECALL where the lor_recalltype field of the clora_recall field was LAYOUTRECALL4_FILE, then the client should use the lor_stateid value from CB_LAYOUTRECALL as the value for lrf_stateid. otherwise, it should use logr_stateid (from an earlier LAYOUTGET result) or lorr_stateid (from an earlier LAYRETURN result) 🎀 this is how it marks the point in time, in terms of layout stateid transitions, when the recall was sent. the client uses the precise lora_recallstateid value and MUST NOT set the stateid's seqid to zero; otherwise, NFS4ERR_BAD_STATEID MUST be returned. NFS4ERR_OLD_STATEID can be returned if the client is using an old seqid and the server knows the client shouldn't be using it. for example, the client uses the seqid on slot 1 of the session, gets the response with the new seqid, and then uses that slot to send another request with the old seqid. the plot twists, bestie 😭

if a client doesn't return a layout in a timely way, the metadata server SHOULD use its control protocol with the storage devices to fence that client off from the data the layout covers 🔒 see Section 12.5.5 for more details.

if the LAYOUTRETURN request sets lora_reclaim to TRUE after the metadata server's grace period, NFS4ERR_NO_GRACE is returned 💀

if the LAYOUTRETURN request sets lora_reclaim to TRUE and lr_returntype is LAYOUTRETURN4_FSID or LAYOUTRETURN4_ALL, NFS4ERR_INVAL is returned 🙅

if the client sets lr_returntype to LAYOUTRETURN4_FILE, the lrs_stateid field represents the layout stateid as updated by this operation's processing 🌸 the current stateid gets updated to match the returned value too. if the last byte of any layout for the current file, client ID, and layout type is being returned, and there are no remaining pending CB_LAYOUTRECALL operations that need a LAYOUTRETURN, then lrs_present MUST be FALSE and no stateid comes back. on top of that, the COMPOUND request's current stateid gets set to the all-zeroes special stateid (see Section 16.2.3.1.2). the server MUST reject with NFS4ERR_BAD_STATEID any further use of the current stateid in that COMPOUND, until a later stateid-returning operation re-establishes it 🫠

on success, the current filehandle keeps its value 🍓

if the EXCHGID4_FLAG_BIND_PRINC_STATEID capability is set on the client ID (see Section 18.35), the server will require that the principal, security flavor, and (if it applies) the GSS mechanism combination that acquired the layout is also the one sending LAYOUTRETURN 🔐 that might not be possible if the principal's credentials aren't available anymore. the server will allow the machine credential or SSV credential (see Section 18.35) to send LAYOUTRETURN if LAYOUTRETURN's operation code was set in the spo_must_allow result of EXCHANGE_ID 👑

18.44.4. IMPLEMENTATION

the final LAYOUTRETURN operation answering a CB_LAYOUTRECALL callback MUST be serialized with any outstanding, intersecting LAYOUTRETURN operations 🧚 heads up: while a client is returning the layout for some recalled range, the server might recall a superset of that range (like LAYOUTRECALL4_ALL). the final return for the bigger one has to block until the smaller layout recall is done 🎀

returning all layouts in a file system with LAYOUTRETURN4_FSID is typically done in response to a CB_LAYOUTRECALL for that file system, as the final return operation ✨ same deal with LAYOUTRETURN4_ALL, which answers a recall callback for all layouts. the client may have already returned some outstanding layouts with individual LAYOUTRETURN calls, and the call for LAYOUTRETURN4_FSID or LAYOUTRETURN4_ALL marks the end of the LAYOUTRETURN sequence. see Section 12.5.5.1 for more details 🦋

once the client has returned all layouts referring to a particular device ID, the server MAY delete the device ID 🌈

18.45. Operation 52: SECINFO_NO_NAME - Get Security on Unnamed Object

18.45.1. ARGUMENT

enum secinfo_style4 {
        SECINFO_STYLE4_CURRENT_FH       = 0,
        SECINFO_STYLE4_PARENT           = 1
};

/* CURRENT_FH: object or child directory */
typedef secinfo_style4 SECINFO_NO_NAME4args;

18.45.2. RESULT

/* CURRENTFH: consumed if status is NFS4_OK */
typedef SECINFO4res SECINFO_NO_NAME4res;

18.45.3. DESCRIPTION 🔍

okay so SECINFO_NO_NAME is like SECINFO's cooler cousin 💅 the client uses it to get a list of valid RPC authentication flavors for a specific file object. the difference: SECINFO_NO_NAME only works with objects that get accessed by filehandle 🌸

there are two styles, picked by the secinfo_style4 enumeration. if SECINFO_STYLE4_CURRENT_FH is passed, SECINFO_NO_NAME asks what security is required for the current filehandle. if SECINFO_STYLE4_PARENT is passed, it asks what security is required for the current filehandle's parent 🧚 and with that style, SECINFO should apply the same access methodology it uses for LOOKUPP when checking the traversal to the parent directory. so if the requester doesn't have the right access to LOOKUPP the parent, SECINFO_NO_NAME must behave the same way and return NFS4ERR_ACCESS 🙈

if PUTFH, PUTPUBFH, PUTROOTFH, or RESTOREFH returns NFS4ERR_WRONGSEC, the client sorts it out by sending a COMPOUND request with PUTFH, PUTPUBFH, or PUTROOTFH right before SECINFO_NO_NAME, style SECINFO_STYLE4_CURRENT_FH 🫶 see Section 2.6 for how to deal with NFS4ERR_WRONGSEC errors coming back from PUTFH, PUTROOTFH, PUTPUBFH, or RESTOREFH.

if SECINFO_STYLE4_PARENT is specified and there's no parent directory, SECINFO_NO_NAME MUST return NFS4ERR_NOENT 💀

on success, the current filehandle is consumed (see Section 2.6.3.1.1.8). if the next operation after SECINFO_NO_NAME tries to use the current filehandle, that operation fails with NFS4ERR_NOFILEHANDLE 😭

everything else about SECINFO_NO_NAME is the same as SECINFO, so see the discussion on SECINFO (Section 18.29.3) ✨

18.45.4. IMPLEMENTATION

see the discussion on SECINFO (Section 18.29.4) 🦋

18.46. Operation 53: SEQUENCE - Supply Per-Procedure Sequencing and Control

18.46.1. ARGUMENT

struct SEQUENCE4args {
        sessionid4     sa_sessionid;
        sequenceid4    sa_sequenceid;
        slotid4        sa_slotid;
        slotid4        sa_highest_slotid;
        bool           sa_cachethis;
};

18.46.2. RESULT 🎀

Okay bestie, this is the RESULT part, where the server tells you what's going on with the session 💅 Here are the SEQUENCE4 status flags and the reply struct, copied exactly:

   const SEQ4_STATUS_CB_PATH_DOWN                  = 0x00000001;
   const SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING      = 0x00000002;
   const SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED       = 0x00000004;
   const SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED     = 0x00000008;
   const SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED    = 0x00000010;
   const SEQ4_STATUS_ADMIN_STATE_REVOKED           = 0x00000020;
   const SEQ4_STATUS_RECALLABLE_STATE_REVOKED      = 0x00000040;
   const SEQ4_STATUS_LEASE_MOVED                   = 0x00000080;
   const SEQ4_STATUS_RESTART_RECLAIM_NEEDED        = 0x00000100;
   const SEQ4_STATUS_CB_PATH_DOWN_SESSION          = 0x00000200;
   const SEQ4_STATUS_BACKCHANNEL_FAULT             = 0x00000400;
   const SEQ4_STATUS_DEVID_CHANGED                 = 0x00000800;
   const SEQ4_STATUS_DEVID_DELETED                 = 0x00001000;

   struct SEQUENCE4resok {
           sessionid4      sr_sessionid;
           sequenceid4     sr_sequenceid;
           slotid4         sr_slotid;
           slotid4         sr_highest_slotid;
           slotid4         sr_target_highest_slotid;
           uint32_t        sr_status_flags;
   };

   union SEQUENCE4res switch (nfsstat4 sr_status) {
   case NFS4_OK:
           SEQUENCE4resok  sr_resok4;
   default:
           void;
   };

18.46.3. DESCRIPTION 💖

okay bestie, so SEQUENCE is the operation the server uses to run session request control AND the reply cache semantics 🧚 it's literally the gatekeeper of the whole vibe ✨

SEQUENCE MUST appear as the first operation of any COMPOUND it shows up in 💅 if it's found in any spot past the first one, the server gives back NFS4ERR_SEQUENCE_POS 🙈 and operations other than SEQUENCE, BIND_CONN_TO_SESSION, EXCHANGE_ID, CREATE_SESSION, and DESTROY_SESSION MUST NOT appear as the first operation in a COMPOUND. not it, bestie 💀 if they do try to start a COMPOUND, they MUST yield NFS4ERR_OP_NOT_IN_SESSION.

If SEQUENCE gets received on a connection that isn't associated with the session via CREATE_SESSION or BIND_CONN_TO_SESSION, and connection association enforcement is enabled (see Section 18.35), then the server returns NFS4ERR_CONN_NOT_BOUND_TO_SESSION 🚫🦋

The sa_sessionid argument identifies the session this request applies to 🎀 and the sr_sessionid result MUST equal sa_sessionid. no cap, it has to match 🫶

The sa_slotid argument is the index in the reply cache for the request 🌸 the sa_sequenceid field is the sequence number of the request for that reply cache entry (slot). the sr_slotid result MUST equal sa_slotid, and the sr_sequenceid result MUST equal sa_sequenceid. so many matching vibes 😭

The sa_highest_slotid argument is the highest slot ID the client has a request outstanding for, and it could be equal to sa_slotid ✨ the server returns two "highest_slotid" values: sr_highest_slotid and sr_target_highest_slotid. the former is the highest slot ID the server will accept in future SEQUENCE operations, and it SHOULD NOT be less than the value of sa_highest_slotid (but see Section 2.10.6.1 for an exception) 🍓 the latter is the highest slot ID the server would prefer the client use on a future SEQUENCE operation 💜

If sa_cachethis is TRUE, the client is requesting that the server cache the entire reply in the server's reply cache. so the server MUST cache the reply (see Section 2.10.6.1.3) 🔥 the server MAY cache the reply if sa_cachethis is FALSE. and if the server does NOT cache the entire reply, it MUST still record that it executed the request at the specified slot and sequence ID 📝

The response to SEQUENCE has a word of status flags (sr_status_flags) that can give the client info about its lock state and communications paths 👑 note that any status bits relating to lock state MAY be reset when lock state is lost due to a server restart (even if the session is persistent across restarts, because session persistence does not imply lock state persistence) or the establishment of a new client instance 🌈

SEQ4_STATUS_CB_PATH_DOWN

When set, this means the client has no operational backchannel path for ANY session tied to the client ID, so it needs to re-establish one 🥺 this bit stays set on all SEQUENCE responses on all sessions tied to the client ID until at least one backchannel is available on any of them. if the client fails to re-establish a backchannel for the client ID, it's subject to having recallable state revoked 💀

SEQ4_STATUS_CB_PATH_DOWN_SESSION

When set, this means the session has no operational backchannel. there are two reasons this may be set and SEQ4_STATUS_CB_PATH_DOWN not 🧚 first, a callback operation that applies specifically to the session (e.g., CB_RECALL_SLOT, see Section 20.8) needs to be sent. second, the server did send a callback operation, but the connection was lost before the reply. the server can't be sure whether or not the client got the callback operation, so per the rules on request retry, the server MUST retry the callback operation over the same session 🔁 this bit is the indication to the client that it needs to associate a connection to the session's backchannel. it remains set on all SEQUENCE responses of the session until a connection is associated with the session's a backchannel. if the client fails to re-establish a backchannel for the session, it's subject to having recallable state revoked 😭

SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING

When set, this means all GSS contexts or RPCSEC_GSS handles assigned to the session's backchannel will expire within a period equal to the lease time ⏳ this bit remains set on all SEQUENCE replies until at least one of the following is true:

All SSV RPCSEC_GSS handles on the session's backchannel have been destroyed and all non-SSV GSS contexts have expired.
At least one more SSV RPCSEC_GSS handle has been added to the backchannel.
The expiration time of at least one non-SSV GSS context of an RPCSEC_GSS handle is beyond the lease period from the current time (relative to the time when a SEQUENCE response was sent)

SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED

When set, this means all non-SSV GSS contexts AND all SSV RPCSEC_GSS handles assigned to the session's backchannel have expired or been destroyed 🥀 this bit remains set on all SEQUENCE replies until at least one non-expired non-SSV GSS context for the session's backchannel has been established, or at least one SSV RPCSEC_GSS handle has been assigned to the backchannel 🫶

SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED

When set, this means the lease has expired and, as a result, the server released ALL of the client's locking state 💀 this status bit remains set on all SEQUENCE replies until the loss of all such locks has been acknowledged by use of FREE_STATEID (see Section 18.38), or by establishing a new client instance. to do that, you destroy all sessions (via DESTROY_SESSION), then the client ID (via DESTROY_CLIENTID), and then invoke EXCHANGE_ID and CREATE_SESSION to establish a new client ID 🔄

SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED

When set, this means some subset of the client's locks have been revoked due to the lease period expiring, followed by another client's conflicting LOCK operation 😬 this status bit remains set on all SEQUENCE replies until the loss of all such locks has been acknowledged by use of FREE_STATEID.

SEQ4_STATUS_ADMIN_STATE_REVOKED

When set, this means one or more locks have been revoked WITHOUT the lease period expiring, due to administrative action 🛠️ this status bit remains set on all SEQUENCE replies until the loss of all such locks has been acknowledged by use of FREE_STATEID.

SEQ4_STATUS_RECALLABLE_STATE_REVOKED

When set, this means one or more recallable objects have been revoked WITHOUT the lease period expiring, because the client failed to return them when recalled 📦 that may be a consequence of there being no working backchannel and the client failing to re-establish one per the SEQ4_STATUS_CB_PATH_DOWN, SEQ4_STATUS_CB_PATH_DOWN_SESSION, or SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED status flags. this status bit remains set on all SEQUENCE replies until the loss of all such locks has been acknowledged by use of FREE_STATEID.

SEQ4_STATUS_LEASE_MOVED

When set, this means responsibility for lease renewal has been transferred to one or more new servers 🚚 this condition continues until the client receives an NFS4ERR_MOVED error and the server receives the subsequent GETATTR for the fs_locations or fs_locations_info attribute for an access to each file system whose lease has been moved to a new server. see Section 11.11.9.2 🦋

SEQ4_STATUS_RESTART_RECLAIM_NEEDED

When set, this means that due to a server restart, the client must reclaim locking state 🔄 until the client sends a global RECLAIM_COMPLETE (Section 18.51), every SEQUENCE operation will return SEQ4_STATUS_RESTART_RECLAIM_NEEDED.

SEQ4_STATUS_BACKCHANNEL_FAULT

The server has hit an unrecoverable fault with the backchannel (e.g., it has lost track of the sequence ID for a slot in the backchannel) 🚨 the client MUST stop sending more requests on the session's fore channel, wait for all outstanding requests to complete on the fore and back channels, and then destroy the session.

SEQ4_STATUS_DEVID_CHANGED

The client is using device ID notifications, and the server has changed a device ID mapping the client holds 🗺️ this flag stays present until the client has obtained the new mapping with GETDEVICEINFO.

SEQ4_STATUS_DEVID_DELETED

The client is using device ID notifications, and the server has deleted a device ID mapping the client holds 🗑️ this flag stays in effect until the client sends a GETDEVICEINFO on the device ID with a null value in the argument gdia_notify_types.

The value of the sa_sequenceid argument, relative to the cached sequence ID on the slot, falls into one of three cases 👀

If the difference between sa_sequenceid and the server's cached sequence ID at the slot ID is two (2) or more, or if sa_sequenceid is less than the cached sequence ID (accounting for wraparound of the unsigned sequence ID value), then the server MUST return NFS4ERR_SEQ_MISORDERED ❌
If sa_sequenceid and the cached sequence ID are the same, this is a retry, and the server replies with what is recorded in the reply cache. the lease is possibly renewed as described below 🔁
If sa_sequenceid is one greater (accounting for wraparound) than the cached sequence ID, then this is a new request, and the slot's sequence ID is incremented. the operations subsequent to SEQUENCE, if any, are processed. if there are no other operations, the only other effects are to cache the SEQUENCE reply in the slot, maintain the session's activity, and possibly renew the lease ✨

If the client reuses a slot ID and sequence ID for a completely different request, the server MAY treat the request as if it is a retry of what it has already executed. the server MAY however detect the client's illegal reuse and return NFS4ERR_SEQ_FALSE_RETRY 🕵️

If SEQUENCE returns an error, then the state of the slot (sequence ID, cached reply) MUST NOT change, and the associated lease MUST NOT be renewed 🙅‍♀️

If SEQUENCE returns NFS4_OK, then the associated lease MUST be renewed (see Section 8.3), EXCEPT if SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED is returned in sr_status_flags 🌸

18.46.4. IMPLEMENTATION ✨

okay bestie, the server MUST keep a mapping of session ID to client ID 🗂️ because that's how it validates any operations that come after SEQUENCE and take a stateid as an argument and/or result 🔐.

so if the client sets up a persistent session 💾, a SEQUENCE that shows up after a server restart might bump into requests that were already performed and recorded in a persistent reply cache before the restart 🫶. In that case SEQUENCE gets processed successfully, while requests that weren't previously performed and recorded get rejected with NFS4ERR_DEADSESSION 💀.

depending on which operations in the COMPOUND were successfully performed before the restart, those operations will also get their replies sent from the server reply cache ✨. ngl this part is kind of messy 😭: when these operations establish locking state, it's locking state that applies to the previous server instance and the previous client ID, even though the restart (which logically happened after these operations) wiped that state out. for a partially executed COMPOUND, processing might reach an operation that wasn't processed during the earlier server instance. that makes it a brand new operation that can't be performed on the existing session, so NFS4ERR_DEADSESSION gets returned from that operation 🙈.

18.47. Operation 54: SET_SSV - Update SSV for a Client ID 🔑

18.47.1. ARGUMENT

struct ssa_digest_input4 {
        SEQUENCE4args sdi_seqargs;
};

struct SET_SSV4args {
        opaque          ssa_ssv<>;
        opaque          ssa_digest<>;
};

18.47.2. RESULT

struct ssr_digest_input4 {
        SEQUENCE4res sdi_seqres;
};

struct SET_SSV4resok {
        opaque          ssr_digest<>;
};

union SET_SSV4res switch (nfsstat4 ssr_status) {
case NFS4_OK:
        SET_SSV4resok   ssr_resok4;
default:
        void;
};

18.47.3. DESCRIPTION 💖

this operation is how you update the SSV for a client ID 🎀. before SET_SSV gets called the first time on a client ID, the SSV is zero. the SSV is the key used for the SSV GSS mechanism (Section 2.10.9) 🗝️.

SET_SSV MUST be preceded by a SEQUENCE operation in the same COMPOUND. it MUST NOT be used if the client didn't opt for SP4_SSV state protection when the client ID was created (see Section 18.35). in that case the server returns NFS4ERR_INVAL 🙅‍♀️.

the field ssa_digest is computed as the output of the HMAC (RFC 2104 [52]) 🧮. it uses the subkey derived from SSV4_SUBKEY_MIC_I2T and the current SSV as the key (see Section 2.10.9 for a description of subkeys), plus an XDR encoded value of data type ssa_digest_input4. the field sdi_seqargs is equal to the arguments of the SEQUENCE operation for the COMPOUND procedure that SET_SSV is within.

the argument ssa_ssv gets XORed with the current SSV to produce the new SSV ✨. the argument ssa_ssv SHOULD be generated randomly 🎲.

in the response, ssr_digest is the output of the HMAC using the subkey derived from SSV4_SUBKEY_MIC_T2I and the new SSV as the key, plus an XDR encoded value of data type ssr_digest_input4. the field sdi_seqres is equal to the results of the SEQUENCE operation for the COMPOUND procedure that SET_SSV is within.

as noted in Section 18.35, the client and server can maintain multiple concurrent versions of the SSV 🌈. the client and server each MUST maintain an internal SSV version number. it's set to one the first time SET_SSV executes on the server and the client receives the first SET_SSV reply. each subsequent SET_SSV increases the internal SSV version number by one. this version number corresponds to the smpt_ssv_seq, smt_ssv_seq, sspt_ssv_seq, and ssct_ssv_seq fields of the SSV GSS mechanism tokens (see Section 2.10.9) 👑.

18.47.4. IMPLEMENTATION 🧠

when the server receives ssa_digest, it MUST verify the digest by computing the digest the same way the client did and comparing it with ssa_digest 🔍. if the server gets a different result, that's an error: NFS4ERR_BAD_SESSION_DIGEST. this error might happen because another SET_SSV from the same client ID changed the SSV. if so, the client recovers by sending a SET_SSV operation again with a recomputed digest based on the subkey of the new SSV 🔄. if the transport connection drops after the SET_SSV request is sent but before the SET_SSV reply is received, there are special considerations for recovery if the client has no more connections associated with sessions associated with the client ID of the SSV. see Section 18.34.4 📚.

Clients SHOULD NOT send an ssa_ssv that is equal to a previous ssa_ssv, nor equal to a previous or current SSV. that includes an ssa_ssv equal to zero, since the SSV is initialized to zero when the client ID is created 🙅.

Clients SHOULD send SET_SSV with RPCSEC_GSS privacy 🔒. Servers MUST support RPCSEC_GSS with privacy for any COMPOUND that has { SEQUENCE, SET_SSV } 🫶.

A client SHOULD NOT send SET_SSV with the SSV GSS mechanism's credential, because the purpose of SET_SSV is to seed the SSV from non-SSV credentials 🌱. instead, SET_SSV SHOULD be sent with the credential of a user that is accessing the client ID for the first time (Section 2.10.8.3). however, if the client does send SET_SSV with SSV credentials, the digest protecting the arguments uses the value of the SSV before ssa_ssv is XORed in. the digest protecting the results uses the value of the SSV after ssa_ssv is XORed in 🤓.

18.48. Operation 55: TEST_STATEID - Test Stateids for Validity 🧪

18.48.1. ARGUMENT

struct TEST_STATEID4args {
        stateid4        ts_stateids<>;
};

18.48.2. RESULT

struct TEST_STATEID4resok {
        nfsstat4        tsr_status_codes<>;
};

union TEST_STATEID4res switch (nfsstat4 tsr_status) {
    case NFS4_OK:
        TEST_STATEID4resok tsr_resok4;
    default:
        void;
};

18.48.3. DESCRIPTION 🕵️‍♀️

TEST_STATEID is used to check the validity of a set of stateids 💅. it can be used at any time, but the client should definitely use it when it gets an indication that one or more of its stateids have been invalidated due to lock revocation 🚨. that happens when the SEQUENCE operation returns with one of these sr_status_flags set:

SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED
SEQ4_STATUS_EXPIRED_ADMIN_STATE_REVOKED
SEQ4_STATUS_EXPIRED_RECALLABLE_STATE_REVOKED

the client can use TEST_STATEID one or more times to test the validity of its stateids 🧚. each use lets it test a large set of stateids at once, and it avoids problems where earlier stateids in a COMPOUND request interfere with checking later ones. that's what would happen if individual stateids were tested by a series of corresponding by operations in a COMPOUND request 🙃.

for each stateid, the server returns the status code that would be returned if that stateid were used in normal operation. returning that status isn't an error, and it doesn't stop COMPOUND processing 🫧. checks for stateid validity proceed as they would for normal operations, with a few exceptions:

There is no check for the type of stateid object, as would be the case for normal use of a stateid.
There is no reference to the current filehandle.
Special stateids are always considered invalid (they result in the error code NFS4ERR_BAD_STATEID).

all stateids are interpreted as being associated with the client for the current session. any possible association with a previous instance of the client (as stale stateids) is not considered 🚫.

the valid status values in the returned status_code array are NFS4ERR_OK, NFS4ERR_BAD_STATEID, NFS4ERR_OLD_STATEID, NFS4ERR_EXPIRED, NFS4ERR_ADMIN_REVOKED, and NFS4ERR_DELEG_REVOKED 🌸.

18.48.4. IMPLEMENTATION 📖

see Sections 8.2.2 and 8.2.4 for a discussion of stateid structure, lifetime, and validation 💜.

18.49. Operation 56: WANT_DELEGATION - Request Delegation 🎁

18.49.1. ARGUMENT

union deleg_claim4 switch (open_claim_type4 dc_claim) {
/*
 * No special rights to object.  Ordinary delegation
 * request of the specified object.  Object identified
 * by filehandle.
 */
case CLAIM_FH: /* new to v4.1 */
        /* CURRENT_FH: object being delegated */
        void;

/*
 * Right to file based on a delegation granted
 * to a previous boot instance of the client.
 * File is specified by filehandle.
 */
case CLAIM_DELEG_PREV_FH: /* new to v4.1 */
        /* CURRENT_FH: object being delegated */
        void;

/*
 * Right to the file established by an open previous
 * to server reboot.  File identified by filehandle.
 * Used during server reclaim grace period.
 */
case CLAIM_PREVIOUS:
        /* CURRENT_FH: object being reclaimed */
        open_delegation_type4   dc_delegate_type;
};

struct WANT_DELEGATION4args {
        uint32_t        wda_want;
        deleg_claim4    wda_claim;
};

18.49.2. RESULT

union WANT_DELEGATION4res switch (nfsstat4 wdr_status) {
case NFS4_OK:
        open_delegation4 wdr_resok4;
default:
        void;
};

18.49.3. DESCRIPTION ✨

okay bestie, first thing: if this description says a specific error code comes back for a specific condition, and more than one condition applies at once, the server MAY return any of the mandated error codes 🫶

so what can a client actually do with WANT_DELEGATION? three things, and they're all iconic 💅

Get a delegation on all types of files except directories.
Register a "want" for a delegation on a file object and get a callback once the delegation is ready 🔔 The server MAY support notifications of availability via callbacks. If the server doesn't support registering wants, it MUST NOT return an error to say so. Instead it MUST return with ond_why set to WND4_CONTENTION or WND4_RESOURCE, and ond_server_will_push_deleg or ond_server_will_signal_avail set to FALSE. When the server says it'll notify by callback, it'll either hand over the delegation with a CB_PUSH_DELEG operation, or cancel its promise by sending a CB_WANTS_CANCELLED operation.
Cancel a want for a delegation.

the client SHOULD NOT set OPEN4_SHARE_ACCESS_READ and SHOULD NOT set OPEN4_SHARE_ACCESS_WRITE in wda_want. if it does anyway, the server MUST ignore them 🙈

the flags in wda_want mean the same thing they mean in OPEN, except for the notes below. the flags are:

OPEN4_SHARE_ACCESS_WANT_READ_DELEG
OPEN4_SHARE_ACCESS_WANT_WRITE_DELEG
OPEN4_SHARE_ACCESS_WANT_ANY_DELEG
OPEN4_SHARE_ACCESS_WANT_NO_DELEG: this one's the odd duck 🦆 Unlike the OPEN operation, this flag SHOULD NOT be set by the client in the arguments to WANT_DELEGATION, and it MUST be ignored by the server.
OPEN4_SHARE_ACCESS_WANT_CANCEL
OPEN4_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_RESRC_AVAIL
OPEN4_SHARE_ACCESS_WANT_PUSH_DELEG_WHEN_UNCONTENDED

WANT_DELEGATION handles all of these exactly like OPEN does 💖 The info about the delegation, and the promises the server makes about future callbacks, look just like what's in the open_delegation4 structure.

the successful results of WANT_DELEGATION are of type open_delegation4, the same type as the delegation field in OPEN's results (see Section 18.16.3). the server builds wdr_resok4 the same way it builds OPEN's delegation, with one twist: WANT_DELEGATION MUST NOT return a delegation type of OPEN_DELEGATE_NONE. 🚫

if ((wda_want & OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) & ~OPEN4_SHARE_ACCESS_WANT_NO_DELEG) comes out to zero, the client has shown no explicit want or no-want for a delegation, and the server MUST return NFS4ERR_INVAL 🙅‍♀️

the client uses OPEN4_SHARE_ACCESS_WANT_CANCEL in WANT_DELEGATION to cancel a want it made earlier. heads up though: if the server is already in the middle of sending the delegation via CB_PUSH_DELEG when the cancel shows up, the delegation might still get pushed to the client 👀

if WANT_DELEGATION fails to return a delegation but still returns NFS4_OK, the server MUST set the delegation type to OPEN4_DELEGATE_NONE_EXT and set od_whynone, the same way Section 18.16 describes.

write delegations aren't available for file types that aren't writable 📝 That includes file objects of types NF4BLK, NF4CHR, NF4LNK, NF4SOCK, and NF4FIFO. if the client asks for OPEN4_SHARE_ACCESS_WANT_WRITE_DELEG without OPEN4_SHARE_ACCESS_WANT_READ_DELEG on one of those file types, the server must set wdr_resok4.od_whynone.ond_why to WND4_WRITE_DELEG_NOT_SUPP_FTYPE 💀

18.49.4. IMPLEMENTATION

a request for a conflicting delegation isn't normally meant to trigger a recall of the existing one 🙅‍♀️ servers can decide some clients get higher priority, so their wants trigger a recall anyway, but that's expected to be a pretty unusual situation.

servers will generally recall delegations assigned by WANT_DELEGATION the same way they recall ones assigned by OPEN ✨ CB_RECALL is usually only sent when other clients do stuff that clashes with the delegation. and the normal response to aging delegations is CB_RECALL_ANY, which gives the client a chance to keep the delegations that matter most to it 🫶

18.50. Operation 57: DESTROY_CLIENTID - Destroy a Client ID

18.50.1. ARGUMENT

struct DESTROY_CLIENTID4args {
        clientid4       dca_clientid;
};

18.50.2. RESULT

struct DESTROY_CLIENTID4res {
        nfsstat4        dcr_status;
};

18.50.3. DESCRIPTION

DESTROY_CLIENTID destroys the client ID 💅 If there are sessions (both idle and non-idle), opens, locks, delegations, layouts, and/or wants (Section 18.49) tied to the client ID's unexpired lease, the server MUST return NFS4ERR_CLIENTID_BUSY 🙈 DESTROY_CLIENTID MAY be preceded by a SEQUENCE operation, as long as the client ID derived from SEQUENCE's session ID is not the same as the client ID being destroyed. if the client IDs match, then the server MUST return NFS4ERR_CLIENTID_BUSY.

if DESTROY_CLIENTID isn't prefixed by SEQUENCE, it MUST be the only operation in the COMPOUND request. otherwise, the server MUST return NFS4ERR_NOT_ONLY_OP 🚨 And if the operation is sent without a SEQUENCE in front, a client that retransmits the request might get an error back, because the original request may have already been executed successfully 😭

18.50.4. IMPLEMENTATION

DESTROY_CLIENTID lets a server immediately reclaim the resources used by an unused client ID, and also forget that it ever generated that client ID 🧹 Once it forgets, the server can safely reuse the client ID in a future EXCHANGE_ID operation 🦋

18.51. Operation 58: RECLAIM_COMPLETE - Indicates Reclaims Finished

18.51.1. ARGUMENT

struct RECLAIM_COMPLETE4args {
        /*
         * If rca_one_fs TRUE,
         *
         *    CURRENT_FH: object in
         *    file system reclaim is
         *    complete for.
         */
        bool            rca_one_fs;
};

18.51.2. RESULTS

struct RECLAIM_COMPLETE4res {
        nfsstat4        rcr_status;
};

18.51.3. DESCRIPTION

RECLAIM_COMPLETE is how the client says "okay, I've reclaimed every lock I'm going to reclaim" 🎀 That applies when it's recovering state after a server restart, or after a file system migrates to another server. there are two kinds:

When rca_one_fs is FALSE, it's a global RECLAIM_COMPLETE. This means recovery of all the locks the client held on the previous server instance is done. The current filehandle doesn't need to be set in this case.
When rca_one_fs is TRUE, it's a file system-specific RECLAIM_COMPLETE. This means recovery of locks for the single fs (the one designated by the current filehandle) is done after a migration. A current filehandle is required when rca_one_fs is set to TRUE. If the current filehandle points at a file system that isn't being migrated, the operation returns NFS4_OK and is otherwise ignored 🌸

once a RECLAIM_COMPLETE is done, there can be no more reclaims for locks whose scope has finished recovering 🔒 After the client sends RECLAIM_COMPLETE, the server won't let it do any more reclaims of locking state for that scope. if the client tries anyway, the server returns NFS4ERR_NO_GRACE 🙅‍♀️

whenever a client gets a new client ID, it MUST send a RECLAIM_COMPLETE with rca_one_fs set to FALSE before its first non-reclaim operation that obtains a lock, even if there are no locks to reclaim 🫶 If non-reclaim locking operations happen before the RECLAIM_COMPLETE, the server returns an NFS4ERR_GRACE error.

same deal when the client reaches a migrated file system on a new server: before it sends the first non-reclaim operation that obtains a lock on that new server, it MUST send a RECLAIM_COMPLETE with rca_one_fs set to TRUE and the current filehandle inside that file system, even if there are no locks to reclaim 🧚 If non-reclaim locking operations happen on that file system before the RECLAIM_COMPLETE, the server returns an NFS4ERR_GRACE error.

sometimes a client has to send both kinds 🤯 For example, a file system is migrated to a server the client has no clientid for. the client then needs a clientid from the server (which means it takes on the job of sending RECLAIM_COMPLETE with rca_one_fs set to FALSE) and also a RECLAIM_COMPLETE with rca_one_fs set to TRUE, to finish the per-fs grace period for the migration. both can go in either order, as long as all the necessary lock reclaims are done before sending either one 💜

any locks that haven't been reclaimed by the time RECLAIM_COMPLETE is done become non-reclaimable. the client MUST NOT try to reclaim them, not during the current server instance, not in any later server instance, and not on another server that picks up responsibility for that file system. if it did, it would be breaking the protocol by claiming to own locks it doesn't own, so it has no right to reclaim them 🙈 See Section 8.4.3 of [66] for a discussion of edge conditions related to lock reclaim.

by sending RECLAIM_COMPLETE, the client signals that it's ready to move on to normal non-reclaim locking operations 🦋 The client should be aware that those operations might temporarily return NFS4ERR_GRACE errors, until the server is ready to end its grace period ✨

18.51.4. IMPLEMENTATION 🧚

okay so servers usually use the info about when reclaim stuff is done to shorten the grace period 🕰️ if the server keeps a list in persistent storage of clients that might've had locks, then once every single one of them has sent a RECLAIM_COMPLETE, it can end the grace period and start normal life (aka granting brand-new lock requests) way sooner than it otherwise would ✨

latency can be minimized by sending a RECLAIM_COMPLETE as part of the COMPOUND request where the last lock-reclaiming operation happens. 🏃‍♀️💨 when there are no reclaims to do at all, RECLAIM_COMPLETE should be done right away so the grace period can end as soon as possible 💖

RECLAIM_COMPLETE should only be done once for each server instance or each time a file system transitions. if you send it a second time, you'll get the error NFS4ERR_COMPLETE_ALREADY 🙃 but here's the cute part: because of the session feature's retry protection, retries of COMPOUND requests that contain a RECLAIM_COMPLETE operation will NOT trigger that error. the session has your back 🫶

when a RECLAIM_COMPLETE is sent, the client is basically saying "fine, any locks I haven't reclaimed yet are lost" 💅 that lets the server re-enable the client's ability to recover locks, in case some edge conditions (see Section 8.4.3) made the server turn that ability off in the first place 🔓

the drama: previous descriptions of RECLAIM_COMPLETE weren't clear enough about when using it with rca_one_fs set to TRUE was actually appropriate, so some clients misused it, sending rca_one_fs set to TRUE when they shouldn't have. 😭 and some servers didn't handle that misuse the way they should have. they either ignored the rca_one_fs setting (treating the whole thing as a global RECLAIM_COMPLETE) or ignored the entire operation 💀

clients SHOULD NOT misuse this feature, and servers SHOULD respond to misuse as described above. but implementors have to keep these considerations in mind while making trade-offs between interoperating with existing implementations and properly supporting lock recovery when a file system migrates 🦋

when a server has no support for becoming the destination server of a file system being migrated, a per-fs RECLAIM_COMPLETE can never be done legitimately, so occurrences of it SHOULD be ignored. the downside of accepting this mistaken use is pretty limited, as long as the client doesn't send it before all the necessary reclaims are done 🌸
when a server might become the destination for a file system being migrated, misuse of per-fs RECLAIM_COMPLETE is a bigger deal 😬 if the file system named isn't in a per-fs grace period, the per-fs RECLAIM_COMPLETE SHOULD be ignored, and accepting it has limited downsides, just like the no-migration case. but if the server runs into a file system that is undergoing migration, it can't accept the operation as if it were a global RECLAIM_COMPLETE without breaking what it was meant for 🚫

18.52. Operation 10044: ILLEGAL - Illegal Operation 🚫

18.52.1. ARGUMENTS

void;

18.52.2. RESULTS

struct ILLEGAL4res {
        nfsstat4        status;
};

18.52.3. DESCRIPTION

this operation is a placeholder for encoding a result when the client sends an operation code inside a COMPOUND that isn't supported 🙈 check the COMPOUND procedure description for more details.

the status field of ILLEGAL4res MUST be set to NFS4ERR_OP_ILLEGAL. 💅

18.52.4. IMPLEMENTATION

a client will probably never send an operation with code OP_ILLEGAL, but if it does, the response is an ILLEGAL4res, just like it would be for any other invalid operation code 🧚 also, if the server gets an illegal operation code that is NOT OP_ILLEGAL, and the server checks for legal operation codes during the XDR decode phase, then ILLEGAL4res would not be returned. bestie, that's a plot twist 😭

19. NFSv4.1 Callback Procedures 🔁

the procedures used for callbacks are defined in the following sections 📖 to keep things clear, the terms "client" and "server" refer to NFS clients and servers, even though for an individual callback RPC, the meaning of those terms is exactly the opposite. it's a whole mirror-world thing 🪞

Both procedures, CB_NULL and CB_COMPOUND, MUST be implemented. 💖

19.1. Procedure 0: CB_NULL - No Operation 😴

19.1.1. ARGUMENTS

void;

19.1.2. RESULTS

void;

19.1.3. DESCRIPTION

CB_NULL is the standard ONC RPC NULL procedure, with the standard void argument and void response. 🫠 it doesn't do anything directly, but the server uses CB_NULL to confirm that there's a working path for RPCs from the server to the client. it's like a "you there?" text 📱

19.1.4. ERRORS

None. ✨

19.2. Procedure 1: CB_COMPOUND - Compound Operations 🎀

19.2.1. ARGUMENTS

enum nfs_cb_opnum4 {
        OP_CB_GETATTR           = 3,
        OP_CB_RECALL            = 4,
/* Callback operations new to NFSv4.1 */
        OP_CB_LAYOUTRECALL      = 5,
        OP_CB_NOTIFY            = 6,
        OP_CB_PUSH_DELEG        = 7,
        OP_CB_RECALL_ANY        = 8,
        OP_CB_RECALLABLE_OBJ_AVAIL = 9,
        OP_CB_RECALL_SLOT       = 10,
        OP_CB_SEQUENCE          = 11,
        OP_CB_WANTS_CANCELLED   = 12,
        OP_CB_NOTIFY_LOCK       = 13,
        OP_CB_NOTIFY_DEVICEID   = 14,

        OP_CB_ILLEGAL           = 10044
};

union nfs_cb_argop4 switch (unsigned argop) {
 case OP_CB_GETATTR:
      CB_GETATTR4args           opcbgetattr;
 case OP_CB_RECALL:
      CB_RECALL4args            opcbrecall;
 case OP_CB_LAYOUTRECALL:
      CB_LAYOUTRECALL4args      opcblayoutrecall;
 case OP_CB_NOTIFY:
      CB_NOTIFY4args            opcbnotify;
 case OP_CB_PUSH_DELEG:
      CB_PUSH_DELEG4args        opcbpush_deleg;
 case OP_CB_RECALL_ANY:
      CB_RECALL_ANY4args        opcbrecall_any;
 case OP_CB_RECALLABLE_OBJ_AVAIL:
      CB_RECALLABLE_OBJ_AVAIL4args opcbrecallable_obj_avail;
 case OP_CB_RECALL_SLOT:
      CB_RECALL_SLOT4args       opcbrecall_slot;
 case OP_CB_SEQUENCE:
      CB_SEQUENCE4args          opcbsequence;
 case OP_CB_WANTS_CANCELLED:
      CB_WANTS_CANCELLED4args   opcbwants_cancelled;
 case OP_CB_NOTIFY_LOCK:
      CB_NOTIFY_LOCK4args       opcbnotify_lock;
 case OP_CB_NOTIFY_DEVICEID:
      CB_NOTIFY_DEVICEID4args   opcbnotify_deviceid;
 case OP_CB_ILLEGAL:          void;
};

struct CB_COMPOUND4args {
        utf8str_cs      tag;
        uint32_t        minorversion;
        uint32_t        callback_ident;
        nfs_cb_argop4   argarray<>;
};

19.2.2. RESULTS

union nfs_cb_resop4 switch (unsigned resop) {
 case OP_CB_GETATTR:    CB_GETATTR4res  opcbgetattr;
 case OP_CB_RECALL:     CB_RECALL4res   opcbrecall;

 /* new NFSv4.1 operations */
 case OP_CB_LAYOUTRECALL:
                        CB_LAYOUTRECALL4res
                                        opcblayoutrecall;

 case OP_CB_NOTIFY:     CB_NOTIFY4res   opcbnotify;

 case OP_CB_PUSH_DELEG: CB_PUSH_DELEG4res
                                        opcbpush_deleg;

 case OP_CB_RECALL_ANY: CB_RECALL_ANY4res
                                        opcbrecall_any;

 case OP_CB_RECALLABLE_OBJ_AVAIL:
                        CB_RECALLABLE_OBJ_AVAIL4res
                                opcbrecallable_obj_avail;

 case OP_CB_RECALL_SLOT:
                        CB_RECALL_SLOT4res
                                        opcbrecall_slot;

 case OP_CB_SEQUENCE:   CB_SEQUENCE4res opcbsequence;

 case OP_CB_WANTS_CANCELLED:
                        CB_WANTS_CANCELLED4res
                                opcbwants_cancelled;

 case OP_CB_NOTIFY_LOCK:
                        CB_NOTIFY_LOCK4res
                                        opcbnotify_lock;

 case OP_CB_NOTIFY_DEVICEID:
                        CB_NOTIFY_DEVICEID4res
                                        opcbnotify_deviceid;

 /* Not new operation */
 case OP_CB_ILLEGAL:    CB_ILLEGAL4res  opcbillegal;
};

struct CB_COMPOUND4res {
        nfsstat4 status;
        utf8str_cs      tag;
        nfs_cb_resop4   resarray<>;
};

19.2.3. DESCRIPTION

the CB_COMPOUND procedure bundles one or more callback procedures into a single RPC request 📦 the main callback RPC program has two main procedures, CB_NULL and CB_COMPOUND, and every other operation rides along inside CB_COMPOUND as a wrapper 🎁

while CB_COMPOUND is being processed, the client might realize it doesn't have the resources to run some or all of the operations in the sequence 🥺 see Section 2.10.6.4 for the details.

The minorversion field of the arguments MUST be the same as the minorversion of the COMPOUND procedure used to create the client ID and session. For NFSv4.1, minorversion MUST be set to 1. 🔢✨

the results of CB_COMPOUND contain a "status" field, and this status MUST be equal to the status of the last operation executed within CB_COMPOUND 🎯 so if some operation hit an error, the "status" value is that same error, the one being returned for the operation that failed.

the "tag" field is handled exactly the way it is in the COMPOUND procedure (see Section 16.2.3) 🏷️

illegal operation codes are handled the same way they are for the COMPOUND procedure 🙅‍♀️

19.2.4. IMPLEMENTATION

CB_COMPOUND is how individual operations get combined into one RPC request 🧩 the client works through each operation in turn. if an operation runs on the client and its status is NFS4_OK, the next operation in the CB_COMPOUND gets executed 🚀 this keeps going until there are no more operations left, or one of them comes back with a status other than NFS4_OK. 💅

19.2.5. ERRORS ✨

okay so CB_COMPOUND will obviously return every error that each operation on the backchannel can return (see Table 13) 💅. but if CB_COMPOUND returns zero operations, the error returned by COMPOUND has nothing to do with an error returned by an operation 🙈. these are the errors CB_COMPOUND can return when it processes zero operations:

    +==============================+==================================+
    | Error                        | Notes                            |
    +==============================+==================================+
    | NFS4ERR_BADCHAR              | The tag argument has a character |
    |                              | the replier does not support.    |
    +------------------------------+----------------------------------+
    | NFS4ERR_BADXDR               |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_DELAY                |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_INVAL                | The tag argument is not in UTF-8 |
    |                              | encoding.                        |
    +------------------------------+----------------------------------+
    | NFS4ERR_MINOR_VERS_MISMATCH  |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_SERVERFAULT          |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_TOO_MANY_OPS         |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_REP_TOO_BIG          |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_REP_TOO_BIG_TO_CACHE |                                  |
    +------------------------------+----------------------------------+
    | NFS4ERR_REQ_TOO_BIG          |                                  |
    +------------------------------+----------------------------------+

                    Table 24: CB_COMPOUND Error Returns

20. NFSv4.1 Callback Operations 🦋

20.1. Operation 3: CB_GETATTR - Get Attributes 🌸

20.1.1. ARGUMENT

   struct CB_GETATTR4args {
           nfs_fh4 fh;
           bitmap4 attr_request;
   };

20.1.2. RESULT

   struct CB_GETATTR4resok {
           fattr4  obj_attributes;
   };

   union CB_GETATTR4res switch (nfsstat4 status) {
    case NFS4_OK:
            CB_GETATTR4resok       resok4;
    default:
            void;
   };

20.1.3. DESCRIPTION

okay bestie, CB_GETATTR is how the server asks for the current modified state of a file that has been given an OPEN_DELEGATE_WRITE delegation 🧚 Only size and change are guaranteed to be serviced by the client, so those are the ones that really matter here. For the full tea on how the client and server should work together when CB_GETATTR is in play, check out Section 10.4.3 📖.

If the filehandle you pass in is not one the client holds an OPEN_DELEGATE_WRITE delegation for, you get an NFS4ERR_BADHANDLE error back 💀.

20.1.4. IMPLEMENTATION

ngl this one is pretty chill ✨ The client only sends back attrmask bits plus the attribute values for change, and for the attributes it might actually change (time_modify and size) 🎀.

20.2. Operation 4: CB_RECALL - Recall a Delegation 🔥

20.2.1. ARGUMENT

   struct CB_RECALL4args {
           stateid4        stateid;
           bool            truncate;
           nfs_fh4         fh;
   };

20.2.2. RESULT

   struct CB_RECALL4res {
           nfsstat4        status;
   };

20.2.3. DESCRIPTION

CB_RECALL is the operation that kicks off the whole process of recalling a delegation and getting it back to the server 👑

The truncate flag is a little optimization for when the file is a regular file that's about to get truncated to zero 🧹. If it's TRUE, the client doesn't have to bother pushing along any modified data for that file to the server, since that data is totally irrelevant now 🙅‍♀️.

If the handle you specify isn't one the client holds a delegation for, you get an NFS4ERR_BADHANDLE error 💅.

If the stateid you specify doesn't correspond to an OPEN delegation for the file named by the filehandle, you get an NFS4ERR_BAD_STATEID error 🙈.

20.2.4. IMPLEMENTATION

this part is giving main character energy 😭 The client SHOULD reply to the callback right away. Replying doesn't finish the recall, except when the reply's status field is neither NFS4ERR_DELAY nor NFS4_OK. The recall isn't complete until the delegation actually gets handed back with a DELEGRETURN operation 🦋.

20.3. Operation 5: CB_LAYOUTRECALL - Recall Layout from Client ✨

20.3.1. ARGUMENT

   /*
    * NFSv4.1 callback arguments and results
    */

   enum layoutrecall_type4 {
           LAYOUTRECALL4_FILE = LAYOUT4_RET_REC_FILE,
           LAYOUTRECALL4_FSID = LAYOUT4_RET_REC_FSID,
           LAYOUTRECALL4_ALL  = LAYOUT4_RET_REC_ALL
   };

   struct layoutrecall_file4 {
           nfs_fh4         lor_fh;
           offset4         lor_offset;
           length4         lor_length;
           stateid4        lor_stateid;
   };

   union layoutrecall4 switch(layoutrecall_type4 lor_recalltype) {
   case LAYOUTRECALL4_FILE:
           layoutrecall_file4 lor_layout;
   case LAYOUTRECALL4_FSID:
           fsid4              lor_fsid;
   case LAYOUTRECALL4_ALL:
           void;
   };

   struct CB_LAYOUTRECALL4args {
           layouttype4             clora_type;
           layoutiomode4           clora_iomode;
           bool                    clora_changed;
           layoutrecall4           clora_recall;
   };

20.3.2. RESULT

   struct CB_LAYOUTRECALL4res {
           nfsstat4        clorr_status;
   };

20.3.3. DESCRIPTION

okay so CB_LAYOUTRECALL is how the server tells the client to recall layouts 📦 That means the client starts the process of giving layouts back using LAYOUTRETURN 🫶. This operation has three flavours of recall, set by layoutrecall_type4: a specific layout of a specific file (LAYOUTRECALL4_FILE), a whole file system ID (LAYOUTRECALL4_FSID), or literally everything (LAYOUTRECALL4_ALL) 🌈.

The behaviour changes depending on which layoutrecall_type4 you get. Here's the breakdown:

LAYOUTRECALL4_FILE: For a layout to match the recall, the values of clora_type, clora_iomode, lor_fh, and the byte range given by lor_offset and lor_length all have to line up with the layout's. clora_iomode can be the special value LAYOUTIOMODE4_ANY, which matches any iomode originally returned in a layout, so it's basically a wild card 🃏. lor_length has its own special value too: if it's NFS4_UINT64_MAX, it means the maximum possible file size 🧠. If a matching layout is found, it MUST be returned using the LAYOUTRETURN operation (see Section 18.44) 📮. For example, if clora_iomode is LAYOUTIOMODE4_ANY, lor_offset is zero, and lor_length is NFS4_UINT64_MAX, then the entire layout is supposed to come back 🎁.

The NFS4ERR_NOMATCHING_LAYOUT error only shows up when the client doesn't hold layouts for the file, or when it has no overlapping layouts for the spec in the recall 🙃.

LAYOUTRECALL4_FSID and LAYOUTRECALL4_ALL: If you get LAYOUTRECALL4_FSID, the fsid tells you which file system's outstanding layouts MUST be returned 🗂️. If you get LAYOUTRECALL4_ALL, then all outstanding layouts MUST be returned, no exceptions 💥. On top of that, both of these also recall all the storage device ID to storage device address mappings in the affected file system(s). The matching LAYOUTRETURN, using either LAYOUTRETURN4_FSID or LAYOUTRETURN4_ALL, is how the client tells the server it invalidated those device mappings. See Section 12.5.5.2.1.5 for the considerations around "bulk" recall of layouts 🧐.

The NFS4ERR_NOMATCHING_LAYOUT error is only returned when the client holds no layouts and also has no valid deviceid mappings 🙌.

now the clora_changed field is where it gets spicy 🌶️ The client also changes how it behaves based on clora_changed. The server uses this field to give extra context on why the layout is being recalled 🔍. A FALSE value means no change to the layout is expected, so the client is allowed to write modified data to the storage devices involved, and it must do that before it returns the layout with LAYOUTRETURN 📝. A TRUE value means the server is changing the layout. Examples of why that happens: the metadata server is restriping the file, or a permanent error has hit a storage device and the metadata server wants to hand over a new layout for the file 🚧. So a clora_changed value of TRUE tells you there's some level of change coming, and the client SHOULD NOT write and commit modified data to the storage devices. Instead, in that case, the client writes and commits its data through the metadata server 📡.

For how to build the lor_stateid field in the arguments, see Section 12.5.3. Heads up: the "seqid" field of lor_stateid MUST NOT be zero ⚠️. For more on this, see Sections 8.2, 12.5.3, and 12.5.5.2 🤓.

20.3.4. IMPLEMENTATION 🧚

Okay bestie, so the client handles CB_LAYOUTRECALL kinda like CB_RECALL (the file delegation recall 💅). It replies to the server right away, and only after that does it actually hand the layouts back with LAYOUTRETURN. ✨ But the operation isn't finished (it's still pending 🥺) until every affected layout has been returned to the server via LAYOUTRETURN.

Before the client sends a layout back via LAYOUTRETURN, it should wait for any in-process or in-flight READ, WRITE, or COMMIT operations that are using the recalled layout to get their responses. 🫶

If the client is holding modified data that's affected by a recalled layout, it's got a few options for writing that data to the server. 🌸 As always, the client can write the data through the metadata server. Honestly, the client might not have a choice other than the metadata server when the clora_changed argument is TRUE and the server can't give it a new layout. 😭 But if clora_changed is FALSE, the client may be able to write the modified data straight to the storage device, and this needs to happen before the layout gets returned via LAYOUTRETURN. 🔥 Also, if the client were to get a new layout covering the modified data's byte-range, writing to the storage devices becomes an available alternative. Just remember that before grabbing a new layout, the client must first return the original one. 💖

When modified data gets written while the layout is still held, the client must use LAYOUTCOMMIT operations at the right time. As required, LAYOUTCOMMIT must be done before the LAYOUTRETURN. 🎀 If there's a huge pile of modified data outstanding, the client may send LAYOUTRETURNs for portions of the recalled layout. This lets the server keep an eye on the client's progress and on how well it's sticking to the original recall request. 👀 However, the last LAYOUTRETURN in a sequence of returns MUST specify the full range being recalled (see Section 12.5.5.1 for details). No skipping that last one, periodt. 💅

If a server needs to delete a device ID and there are layouts referring to that device ID, CB_LAYOUTRECALL MUST be invoked to make the client return all layouts referring to the device ID. Only then can the server delete the device ID. 🙈 If the client doesn't return the affected layouts, the server MAY revoke the layouts. 💀

20.4. Operation 6: CB_NOTIFY - Notify Client of Directory Changes 🌈

20.4.1. ARGUMENT 🍓

/*
 * Directory notification types.
 */
enum notify_type4 {
        NOTIFY4_CHANGE_CHILD_ATTRS = 0,
        NOTIFY4_CHANGE_DIR_ATTRS = 1,
        NOTIFY4_REMOVE_ENTRY = 2,
        NOTIFY4_ADD_ENTRY = 3,
        NOTIFY4_RENAME_ENTRY = 4,
        NOTIFY4_CHANGE_COOKIE_VERIFIER = 5
};

/* Changed entry information.  */
struct notify_entry4 {
        component4      ne_file;
        fattr4          ne_attrs;
};

/* Previous entry information */
struct prev_entry4 {
        notify_entry4   pe_prev_entry;
        /* what READDIR returned for this entry */
        nfs_cookie4     pe_prev_entry_cookie;
};

struct notify_remove4 {
        notify_entry4   nrm_old_entry;
        nfs_cookie4     nrm_old_entry_cookie;
};

struct notify_add4 {
        /*
         * Information on object
         * possibly renamed over.
         */
        notify_remove4      nad_old_entry<1>;
        notify_entry4       nad_new_entry;
        /* what READDIR would have returned for this entry */
        nfs_cookie4         nad_new_entry_cookie<1>;
        prev_entry4         nad_prev_entry<1>;
        bool                nad_last_entry;
};

struct notify_attr4 {
        notify_entry4   na_changed_entry;
};

struct notify_rename4 {
        notify_remove4  nrn_old_entry;
        notify_add4     nrn_new_entry;
};

struct notify_verifier4 {
        verifier4       nv_old_cookieverf;
        verifier4       nv_new_cookieverf;
};

/*
 * Objects of type notify_<>4 and
 * notify_device_<>4 are encoded in this.
 */
typedef opaque notifylist4<>;

struct notify4 {
        /* composed from notify_type4 or notify_deviceid_type4 */
        bitmap4         notify_mask;
        notifylist4     notify_vals;
};

struct CB_NOTIFY4args {
        stateid4    cna_stateid;
        nfs_fh4     cna_fh;
        notify4     cna_changes<>;
};

20.4.2. RESULT ✨

struct CB_NOTIFY4res {
        nfsstat4    cnr_status;
};

20.4.3. DESCRIPTION 📣

The CB_NOTIFY operation is how the server sends notifications to clients about changes in delegated directories. 🦋 Registering for these notifications happens when the delegation is set up using GET_DIR_DELEGATION. The notifications travel over the backchannel, and they go out once the original request has been processed on the server. The server sends an array of notifications for changes that might have happened in the directory, and they come as a list of pairs of bitmaps and values. For how NFSv4.1 bitmaps work, see Section 3.3.7. 🧠

If the server has more notifications than can fit in one CB_COMPOUND request, it SHOULD send a sequence of serial CB_COMPOUND requests, so the client's view of the directory doesn't get confused. 😵‍💫 For example, say the server reports that a file named "foo" was added and then that "foo" was removed. The client has to receive those notifications in the same order the operations happened on the server. Order matters, bestie. 🕰️

If the client holding the delegation makes changes in the directory that add or remove files or sub-directories, the server will notify that client of the resulting change(s). 📂 If the client holding the delegation is only changing attributes or the cookie verifier, the server doesn't need to send it notifications. The server sends the following information for each operation:

NOTIFY4_ADD_ENTRY

The server sends information about the new directory entry being created, along with the cookie for that entry. The entry information (data type notify_add4) includes the component name of the entry and its attributes. The server sends this type of entry when a file is actually being created, when an entry is being added to a directory because of a rename across directories (see below), and when a hard link is being created to an existing file. 🔗 If the entry gets added to the end of the directory, the server sets the nad_last_entry flag to TRUE. If the file is added so that there's at least one entry before it, the server also returns the previous entry information (nad_prev_entry, a variable-length array of up to one element; if the array is zero length, there's no previous entry), along with its cookie. This helps clients find the right spot in their file name caches and directory caches for this entry. 🗂️ If the new entry's cookie is available, it shows up in the nad_new_entry_cookie field (another variable-length array of up to one element). And if adding the entry causes another entry to be deleted (which can only happen in the rename case) atomically with the addition, then info on that entry gets reported in nad_old_entry. 💥

NOTIFY4_REMOVE_ENTRY

The server sends information about the directory entry being deleted. It also sends the cookie value for the deleted entry, so clients can get to the cached info for that entry. 🗑️

NOTIFY4_RENAME_ENTRY

The server sends information about both the old entry and the new entry, including the name and attributes for each one. 🏷️ If the rename causes an entry to be deleted (the case of a file renamed over), that gets reported in nrn_new_new_entry.nad_old_entry. This notification is only sent if both entries are in the same directory. If the rename is across directories, the server sends a remove notification to one directory and an add notification to the other directory, assuming both have a directory delegation. 🧩

NOTIFY4_CHANGE_CHILD_ATTRS/NOTIFY4_CHANGE_DIR_ATTRS

The client uses the attribute mask to tell the server which attributes it wants notifications for. This change notification can be requested for changes to the directory's own attributes and also for changes to any file's attributes in the directory, using two separate attribute masks. 🎯 The client can't ask for change attribute notification for a specific file, since one attribute mask covers all the files in the directory. Upon any attribute change, the server sends back the values of the changed attributes. Notifications might not make sense for some file system-wide attributes, so it's up to the server to decide which subset it wants to support. 🤷‍♀️ The client can negotiate how often it gets attribute notifications by letting the server know how frequently it wants to hear about a change. The server returns supported notification frequencies, or an indication that no notification is permitted for directory or child attributes, by setting the dir_notif_delay and dir_entry_notif_delay attributes, respectively. ⏱️

NOTIFY4_CHANGE_COOKIE_VERIFIER

If the cookie verifier changes while a client is holding a delegation, the server notifies the client so it can invalidate its cookies and re-send a READDIR to get the new set of cookies. 🍪

20.5. Operation 7: CB_PUSH_DELEG - Offer Previously Requested Delegation to Client 🎁

20.5.1. ARGUMENT 🧾

struct CB_PUSH_DELEG4args {
        nfs_fh4          cpda_fh;
        open_delegation4 cpda_delegation;

};

20.5.2. RESULT 📥

struct CB_PUSH_DELEG4res {
        nfsstat4 cpdr_status;
};

20.5.3. DESCRIPTION 💌

CB_PUSH_DELEG is how the server does two things at once: it tells the client that the delegation it wanted is available, and it offers that delegation to the client. 🎉 The client wanted it earlier, through a want established from an OPEN or WANT_DELEGATION operation. The client has options. It can accept the delegation by returning NFS4_OK to the server. It can delay the decision by returning NFS4ERR_DELAY. Or it can permanently turn down the offer by returning NFS4ERR_REJECT_DELEG. 🙅‍♀️ When a delegation gets rejected like that, the want that was established is permanently deleted, and the delegation becomes available for another client to acquire. 🏃‍♀️💨

20.5.4. IMPLEMENTATION 🛠️

If the client does return NFS4ERR_DELAY and there's a conflicting delegation request, the server MAY process that request at the expense of the client that returned NFS4ERR_DELAY. The client's want isn't cancelled, but it MAY be processed behind other delegation requests or registered wants. 🐢

When a client returns a status other than NFS4_OK, NFS4ERR_DELAY, or NFS4ERR_REJECT_DELAY, the want stays pending. Servers may decide to cancel the want by sending a CB_WANTS_CANCELLED. 🙃

20.6. Operation 8: CB_RECALL_ANY - Keep Any N Recallable Objects 🌟

20.6.1. ARGUMENT

const RCA4_TYPE_MASK_RDATA_DLG          = 0;
const RCA4_TYPE_MASK_WDATA_DLG          = 1;
const RCA4_TYPE_MASK_DIR_DLG            = 2;
const RCA4_TYPE_MASK_FILE_LAYOUT        = 3;
const RCA4_TYPE_MASK_BLK_LAYOUT         = 4;
const RCA4_TYPE_MASK_OBJ_LAYOUT_MIN     = 8;
const RCA4_TYPE_MASK_OBJ_LAYOUT_MAX     = 9;
const RCA4_TYPE_MASK_OTHER_LAYOUT_MIN   = 12;
const RCA4_TYPE_MASK_OTHER_LAYOUT_MAX   = 15;

struct  CB_RECALL_ANY4args      {
        uint32_t        craa_objects_to_keep;
        bitmap4         craa_type_mask;
};

20.6.2. RESULT

struct CB_RECALL_ANY4res {
        nfsstat4        crar_status;
};

20.6.3. DESCRIPTION

okay so the server might realize it can't hold onto all the state for its recallable objects, like delegations and layouts, without running out of resources 😭. when that happens, it's not ideal, but the server is totally free to recall individual objects to lighten the load 🫶

the whole point of stuff like delegations is to cut down on the client talking to the server, so the server literally can't assume an object is useless just because nobody's touched it lately 🙈. no visible use is totally consistent with a delegation quietly handling operations so they never get sent to the server.

layouts are a little different ngl. a layout's usefulness shows up when storage devices get I/O requests, but nothing requires a storage device to tell the metadata server about any past or present use of a layout 💀. so the metadata server usually won't know which layouts are good candidates to recall when resources are low.

to build an effective reclaim scheme for these objects, the server's knowledge of its available resources has to be what decides when objects must be recalled, and the clients pick which actual objects get returned ✨

server implementations can totally differ in how they handle resource allocation 🌸. one server might share resources across every class of recallable object, while another might keep separate pools for layouts and delegations, or even split things further by delegation type.

when a given resource pool is over-utilized, the server can send a CB_RECALL_ANY to clients holding recallable objects of the involved types. this lets the client keep a certain number of those objects and return any extra. a mask says which types of objects are being limited, and the client uses its own knowledge of what's currently useful to decide which objects in that class to give back 💅

a bunch of bits are defined here. for some of them, ranges are defined, and it's up to the definition of the storage protocol to say how those get used. there are ranges reserved for object-based storage protocols and for other experimental storage protocols. an RFC that defines one of those storage protocols needs to spell out how the specific bits in its range get used. for example, it could map layout attributes (read vs. write, size of area) to a bit, or it could define a field in the layout where the server makes the associated bit position available to the client 🦋

RCA4_TYPE_MASK_RDATA_DLG: the client is to return OPEN_DELEGATE_READ delegations on non-directory file objects.
RCA4_TYPE_MASK_WDATA_DLG: the client is to return OPEN_DELEGATE_WRITE delegations on regular file objects.
RCA4_TYPE_MASK_DIR_DLG: the client is to return directory delegations.
RCA4_TYPE_MASK_FILE_LAYOUT: the client is to return layouts of type LAYOUT4_NFSV4_1_FILES.
RCA4_TYPE_MASK_BLK_LAYOUT: see [48] for a description.
RCA4_TYPE_MASK_OBJ_LAYOUT_MIN to RCA4_TYPE_MASK_OBJ_LAYOUT_MAX: see [47] for a description.
RCA4_TYPE_MASK_OTHER_LAYOUT_MIN to RCA4_TYPE_MASK_OTHER_LAYOUT_MAX: this range is reserved for telling the client to recall layouts of experimental or site-specific layout types (see Section 3.3.13). 🧚

if a bit is set in the type mask that matches an undefined type of recallable object, NFS4ERR_INVAL MUST be returned. but if a bit matches a defined type that the client doesn't support, NFS4ERR_INVAL MUST NOT be returned. future minor versions of NFSv4 might expand the set of valid type mask bits 👑

CB_RECALL_ANY gives a count of objects the client MAY keep, not a count it has to give back. this is to dodge a race between a CB_RECALL_ANY carrying a count of objects to free and the client's own operations that return layouts or delegations. if that race happened, the client and server would disagree on how many objects to return, and the client could accidentally free way too many 😭

if resource demands call for it, the server can send another CB_RECALL_ANY with a lower count, even before it gets an acknowledgment for an earlier CB_RECALL_ANY with the same type mask. these might arrive in a different order than they were sent, but any reordering of the callback stream is harmless 🌈. it's the client's job to bring the size of its recallable object set in line with each CB_RECALL_ANY it receives. until that job is done, it can't be cancelled or changed by a later CB_RECALL_ANY for the same type mask. so if the server sends two CB_RECALL_ANYs, the result is the same as if only the lower count was sent, no matter what order they arrive in. this also means a server can't cancel a CB_RECALL_ANY by sending another recall with a higher count. and if a CB_RECALL_ANY arrives with a count that's already within the limit, or above a limit the client is still working to reach, that callback does nothing ✨

servers are generally free to deny recallable objects when they don't have enough resources. just know this policy quietly favors objects that already exist over new requests, so resources might not get used as well as they could 🙈. to avoid that, servers are well advised to start sending CB_RECALL_ANY callbacks a little below the point where they stop handing out new delegations and layouts. that way the client can clear out its less-used objects when it makes sense, and its later requests can keep getting fresh resources freed up by object returns 🍓

20.6.4. IMPLEMENTATION

the client can choose to return any type of object the mask specifies. if a server wants to limit the use of objects of one particular type, it should only put that type in the mask it sends 💅. if the client fails to return requested objects, it's up to the server to handle that, typically by sending specific recalls (that is, sending CB_RECALL operations) to limit resource usage properly. the server should give the client enough time to return objects before it moves on to specific recalls. that time should not be less than the lease period ⏳

20.7. Operation 9: CB_RECALLABLE_OBJ_AVAIL - Signal Resources for Recallable Objects ✨

20.7.1. ARGUMENT

typedef CB_RECALL_ANY4args CB_RECALLABLE_OBJ_AVAIL4args;

20.7.2. RESULT

struct CB_RECALLABLE_OBJ_AVAIL4res {
        nfsstat4        croa_status;
};

20.7.3. DESCRIPTION

CB_RECALLABLE_OBJ_AVAIL is how the server tells the client that it has resources to grant recallable objects that OPEN, WANT_DELEGATION, GET_DIR_DELEG, or LAYOUTGET might have turned down earlier 🎀

the craa_objects_to_keep argument is the total number of recallable objects, of the types in type_mask, that the server thinks it can let the client have. that total includes the ones the client already holds. if a client tries to grab more recallable objects than the server said it can have, it runs the risk of getting objects recalled 😭

the server isn't obligated to reserve the gap between what the client currently has and craa_objects_to_keep. and delaying the reply to CB_RECALLABLE_OBJ_AVAIL doesn't stop the server from using those resources for something else 💀. actually, if a client replies slowly, the server might decide it has reduced ability to manage recallable objects, and then cancel or shrink any reservation it's holding for that client. so if the client wants more recallable objects, it needs to reply quickly and then send the operations that acquire them 🏃‍♀️

20.8. Operation 10: CB_RECALL_SLOT - Change Flow Control Limits 🌸

20.8.1. ARGUMENT

struct CB_RECALL_SLOT4args {
        slotid4       rsa_target_highest_slotid;
};

20.8.2. RESULT

struct CB_RECALL_SLOT4res {
        nfsstat4   rsr_status;
};

20.8.3. DESCRIPTION

CB_RECALL_SLOT asks the client to give back session slots and, where it applies, transport credits (like RDMA credits for connections tied to the operations channel) on the session's fore channel 🦋. it carries rsa_target_highest_slotid, which is the target highest slot ID the server wants for the session. the client MUST then work toward bringing the session's highest slot ID down to that target value 💖

if the only connections tied to the operations channel are non-RDMA, the client just has to wait for all outstanding requests with a slot ID greater than rsa_target_highest_slotid to finish. then it sends one COMPOUND containing a single SEQUENCE operation, with the sa_highestslot field set to rsa_target_highest_slotid ✨. if there are RDMA-based connections tied to the operations channel, the client also needs to send enough zero-length "RDMA Send" messages to bring the total RDMA credit count down to rsa_target_highest_slotid + 1 or lower 🫶

20.8.4. IMPLEMENTATION

if the client doesn't get its highest slot on the fore channel down to what the server asked for, the server can force the issue 😤. it does that by asserting flow control on the receive side of every connection bound to the fore channel, then finishing all the outstanding requests in slots above rsa_target_highest_slotid. once that's done, the server can reopen flow control. after that, if the client sends a new request on a slot above rsa_target_highest_slotid, the server can return NFS4ERR_BADSLOT 🚫

20.9. Operation 11: CB_SEQUENCE - Supply Backchannel Sequencing and Control 🔥

20.9.1. ARGUMENT

struct referring_call4 {
        sequenceid4     rc_sequenceid;
        slotid4         rc_slotid;
};

struct referring_call_list4 {
        sessionid4      rcl_sessionid;
        referring_call4 rcl_referring_calls<>;
};

struct CB_SEQUENCE4args {
        sessionid4           csa_sessionid;
        sequenceid4          csa_sequenceid;
        slotid4              csa_slotid;
        slotid4              csa_highest_slotid;
        bool                 csa_cachethis;
        referring_call_list4 csa_referring_call_lists<>;
};

20.9.2. RESULT ✨

struct CB_SEQUENCE4resok {
        sessionid4         csr_sessionid;
        sequenceid4        csr_sequenceid;
        slotid4            csr_slotid;
        slotid4            csr_highest_slotid;
        slotid4            csr_target_highest_slotid;
};

union CB_SEQUENCE4res switch (nfsstat4 csr_status) {
case NFS4_OK:
        CB_SEQUENCE4resok   csr_resok4;
default:
        void;
};

20.9.3. DESCRIPTION 🎀

okay so bestie, CB_SEQUENCE is the op that keeps the books for the backchannel of whatever session the request rides on 📒✨ it carries the session ID this request belongs to, plus the slot ID and sequence ID the server uses for session request control and exactly once semantics 💅 it also swaps slot ID maxima so the reply cache can resize 🦋 and yes, the rules are strict: CB_SEQUENCE MUST appear once in each CB_COMPOUND request and MUST be the first operation 🫶 if it shows up anywhere past the first spot, the error NFS4ERR_SEQUENCE_POS MUST be returned 😭 and if some other op is sitting in the first position, NFS4ERR_OP_NOT_IN_SESSION MUST be returned 💀

check Section 18.46.3 if you want the full slot processing vibe 🌸

if csa_cachethis is TRUE, the server is basically saying "pls cache this reply" 🥺 and the client MUST cache the reply (see Section 2.10.6.1.3) 💜

the csa_referring_call_lists array is a list of earlier COMPOUND requests the client sent, identified by session ID, slot ID, and sequence ID 🧚 those older requests made state that some ops in this same CB_COMPOUND are now identifying 👑 the session ID is in there because leased state is tied to a client ID, and one client ID can have multiple sessions 🌈 see Section 2.10.6.3 for more tea ☕

now the csa_sequenceid vs. the cached sequence ID at the slot falls into one of three cases, periodt 💅

* The difference is two (2) or more, OR csa_sequenceid is less than the cached sequence ID (wraparound of the unsigned sequence ID counts too 🙈): the client MUST return NFS4ERR_SEQ_MISORDERED 😭

* csa_sequenceid and the cached sequence ID are the same 🎀: this is a retry, and the client returns the CB_COMPOUND request's cached reply 🔁✨

* csa_sequenceid is one greater than the cached sequence ID (wraparound counts) 🌿: this is a brand new request, so the slot's sequence ID gets incremented 📈 the operations after CB_SEQUENCE, if any, get processed 🏃‍♀️ if there are no other ops, the only other stuff that happens is caching the CB_SEQUENCE reply in the slot, keeping the session's activity alive, and when the server gets the CB_SEQUENCE reply, the lease of state tied to the client ID gets renewed 💖

if the server reuses a slot ID and sequence ID for a totally different request, the client MAY treat it as a retry of what it already ran 🤷‍♀️ the client MAY also detect that the server did an illegal reuse and return NFS4ERR_SEQ_FALSE_RETRY 🚨

if CB_SEQUENCE returns an error, the state of the slot (sequence ID, cached reply) MUST NOT change 🙅‍♀️ for when NFS4ERR_RETRY_UNCACHED_REP might come back, see Section 2.10.6.1.3 🧚

the client sends back two "highest_slotid" values, and they are different queens 👑 csr_highest_slotid is the highest slot ID the client will accept in a future CB_SEQUENCE, and it SHOULD NOT be less than csa_highest_slotid (but see Section 2.10.6.1 for an exception 🙈) 🫶 csr_target_highest_slotid is the highest slot ID the client would prefer the server use in a future CB_SEQUENCE 🥹 it's giving "pls be reasonable" energy 💅

20.10. Operation 12: CB_WANTS_CANCELLED - Cancel Pending Delegation Wants 🦋

20.10.1. ARGUMENT ✨

struct CB_WANTS_CANCELLED4args {
        bool cwca_contended_wants_cancelled;
        bool cwca_resourced_wants_cancelled;
};

20.10.2. RESULT 🎀

struct CB_WANTS_CANCELLED4res {
        nfsstat4        cwcr_status;
};

20.10.3. DESCRIPTION 💜

this op tells the client that some or all of the wants it signed up for (recallable delegations and layouts) have been cancelled 🙃 like, the server is basically going "we're not gonna call you back about that one" 📵

if cwca_contended_wants_cancelled is TRUE, the server will NOT be pushing any delegations to the client that become available after contention passes 🚫✨

if cwca_resourced_wants_cancelled is TRUE, the server will NOT notify the client when there are resources on the server to grant delegations or layouts 🔕

after getting a CB_WANTS_CANCELLED, the client is free to try to acquire the delegations or layouts it was waiting for, and possibly re-register wants 🔁🌈 main character reset moment 👑

20.10.4. IMPLEMENTATION 🌸

when a client has an OPEN, WANT_DELEGATION, or GET_DIR_DELEGATION request outstanding and a CB_WANTS_CANCELLED gets sent, the server might need to make it clear whether a promise to signal delegation availability happened before the CB_WANTS_CANCELLED (so it's covered by it) or after (so it wasn't) 🧐 the server can sort this out by putting the right requests into the list of referring calls in the associated CB_SEQUENCE 🗂️💖

20.11. Operation 13: CB_NOTIFY_LOCK - Notify Client of Possible Lock Availability 🔐

20.11.1. ARGUMENT 🎀

struct CB_NOTIFY_LOCK4args {
    nfs_fh4     cnla_fh;
    lock_owner4 cnla_lock_owner;
};

20.11.2. RESULT ✨

struct CB_NOTIFY_LOCK4res {
        nfsstat4        cnlr_status;
};

20.11.3. DESCRIPTION 🦋

the server can use this op to say that a byte-range lock for the given file and lock-owner might be available 🔓 the client had previously asked for it with an unsuccessful LOCK op, so this is like a little "psst, try again" 🤫💅

this callback exists to help cut down the latency of blocking locks 🏃‍♀️💨 when the server notices that a client polling for a blocking byte-range lock might now be able to get it 👀 if the server supports this callback for a given file, it MUST set the OPEN4_RESULT_MAY_NOTIFY_LOCK flag when responding to successful opens for that file 🚩 this does NOT commit the server to ever using CB_NOTIFY_LOCK 🙅‍♀️ it's just a hint the client can use to decide how often to poll for locks from that open 🧠

if an OPEN results in an upgrade, where the stateid returned has an "other" value matching one already allocated, with a new "seqid" showing the lock being represented has changed, then the OPEN4_RESULT_MAY_NOTIFY_LOCK flag in that new OPEN response controls handling from that point on 🔄 when parallel OPENs happen on the same file and open-owner, the ordering of the "seqid" fields of the returned stateids (subject to wraparound) picks the controlling value of the OPEN4_RESULT_MAY_NOTIFY_LOCK flag 🎯

20.11.4. IMPLEMENTATION 🌈

the server MUST NOT grant the byte-range lock to the client unless and until it gets a LOCK op from the client 🙅‍♀️ and the client getting this callback can NOT assume it now has the lock, or that a later LOCK for it will work 😬

the server is not required to implement this callback, and even if it does, it isn't required to use it in any particular case 🤷‍♀️ so the client still has to rely on polling for blocking locks, as described in Section 9.6 🔁 the client isn't required to implement this callback either, and even if it does, it's still free to ignore it 💅 so the server MUST NOT assume the client will act based on the callback 🫶

20.12. Operation 14: CB_NOTIFY_DEVICEID - Notify Client of Device ID Changes 🧚

20.12.1. ARGUMENT 🎀

/*
 * Device notification types.
 */
enum notify_deviceid_type4 {
        NOTIFY_DEVICEID4_CHANGE = 1,
        NOTIFY_DEVICEID4_DELETE = 2
};

/* For NOTIFY4_DEVICEID4_DELETE */
struct notify_deviceid_delete4 {
        layouttype4     ndd_layouttype;
        deviceid4       ndd_deviceid;
};

/* For NOTIFY4_DEVICEID4_CHANGE */
struct notify_deviceid_change4 {
        layouttype4     ndc_layouttype;
        deviceid4       ndc_deviceid;
        bool            ndc_immediate;
};

struct CB_NOTIFY_DEVICEID4args {
        notify4 cnda_changes<>;
};

20.12.2. RESULT 💖

struct CB_NOTIFY_DEVICEID4res {
        nfsstat4        cndr_status;
};

20.12.3. DESCRIPTION 🌸

the CB_NOTIFY_DEVICEID op is how the server tells clients about changes to pNFS device IDs 📣 registering for device ID notifications is optional and happens through GETDEVICEINFO 🙌 these notifications go out over the backchannel once the original request has been processed on the server ✅ the server sends an array, cnda_changes, as a list of pairs of bitmaps and values 🗂️ see Section 3.3.7 for how NFSv4.1 bitmaps work 🧠

just like CB_NOTIFY (Section 20.4.3), the server might have more notifications than fit in one CB_COMPOUND, so multiple CB_COMPOUNDs may be needed 📦 but serialization isn't a problem here, because unlike directory entries, device IDs can't be re-used after being deleted (Section 12.2.10) 🙅‍♀️✨

every device ID notification contains a device ID and a layout type 🎀 the layout type matters because two different layout types can share the same device ID, and that common device ID can have totally different mappings for each layout type 🤯

the server will send these notifications:

NOTIFY_DEVICEID4_CHANGE 🔄: a previously provided device-ID-to-device-address mapping has changed, and the client uses GETDEVICEINFO to grab the updated mapping 📥 this is encoded in a notify_deviceid_change4 value 💌 it also has a boolean field, ndc_immediate, which if TRUE means the change gets enforced right away, so the client might not be able to finish any pending I/O to the device ID 😱 if ndc_immediate is FALSE, the client can finish pending I/O for an indefinite time 🕰️ once pending I/O is done, the client SHOULD get the new device-ID-to-device-address mappings before sending new I/O requests to the storage devices addressed by the device ID 🫶

NOTIFY4_DEVICEID_DELETE 🗑️: deletes a device ID from the mappings 🙈 this notification MUST NOT be sent if the client has a layout that refers to the device ID 🚫 put another way, if the server is sending a delete device ID notification, one of these is true for layouts tied to the layout type:

the client never had a layout referring to that device ID 🫥
the client has returned all layouts referring to that device ID ↩️
the server has revoked all layouts referring to that device ID 🧹

this is encoded in a notify_deviceid_delete4 value 🎁 and after a server deletes a device ID, it MUST NOT reuse that device ID for the same layout type until the client ID is deleted 🔒💜

20.13. Operation 10044: CB_ILLEGAL - Illegal Callback Operation 🙈

20.13.1. ARGUMENT 🌸

void;

20.13.2. RESULT 💅

/*
 * CB_ILLEGAL: Response for illegal operation numbers
 */
struct CB_ILLEGAL4res {
        nfsstat4        status;
};

20.13.3. DESCRIPTION 🎀

okay bestie, so this operation is basically a placeholder 🧚 It exists so there's a result to encode when the server sends an operation code inside CB_COMPOUND that isn't defined in the NFSv4.1 specification 💀 Check out Section 19.2.3 if you want the full tea ☕

The status field of CB_ILLEGAL4res MUST be set to NFS4ERR_OP_ILLEGAL. no cap, that part is non-negotiable 🔥

20.13.4. IMPLEMENTATION 🦋

ngl this section is so relatable 🙈 a server will probably never send OP_CB_ILLEGAL on purpose 😭 but if it does, the response is CB_ILLEGAL4res, just like it would be for any other invalid operation code. it's giving consistency ✨

highkey important detail: if the client gets an illegal operation code that is not OP_ILLEGAL, and the client checks for legal operation codes during the XDR decode phase, then an instance of the CB_ILLEGAL4res data type will not be returned 💜 the decode step catches it first, so the placeholder never shows up. iconic, honestly 👑

21. Security Considerations 🔐💖

Okay bestie, this is the scary part, so let's go slow 😭. Historically, NFS auth was basically: the whole machine IS the NFS client, and the server just trusts that client to vouch for the end-user 🙈. The server only shared files with specific clients, picked out by their source network address. In that world, the AUTH_SYS RPC security flavor just told the NFS server who the end-user was. When the client got responses back, it only checked that they came from the same network address and port number the request went to. Super easy to build and deploy, but ngl it's unsafe 💀.

So NFSv4.1 implementations are REQUIRED to support a security model with end-to-end authentication. An end-user on a client mutually authenticates to a principal on an NFS server, using crypto schemes that don't expose passwords or keys in the clear on the network 🫶. Integrity and privacy of requests and responses matter too 🔒. All of that end-to-end mutual authentication, integrity, and privacy stuff is discussed in Section 2.2.1.1.1 💅, and the specific Kerberos V5 considerations are in Section 2.2.1.1.1.2.1.1 🦋.

Quick plot twist 🌈: being REQUIRED to implement does not mean REQUIRED to use. AUTH_SYS can still be used by NFSv4.1 clients and servers. BUT AUTH_SYS is merely an OPTIONAL security flavor in NFSv4.1, so interoperability via AUTH_SYS is not assured 😬.

For reasons like less admin overhead, better performance, and/or lower CPU use, users might decline security mechanisms that enable integrity protection on each remote procedure call and response 🙃. Using mechanisms without integrity leaves the user vulnerable to a man-in-the-middle of the NFS client and server, one that modifies the RPC request and/or the response. Implementations are free to offer weaker security options, but there are three operations in particular that warrant the implementation overriding user choices 👑.

The first two such operations are SECINFO and SECINFO_NO_NAME. It is RECOMMENDED that the client send both of them protected with a security flavor that has integrity protection, such as RPCSEC_GSS with either the rpc_gss_svc_integrity or rpc_gss_svc_privacy service. Without integrity protection encapsulating SECINFO and SECINFO_NO_NAME and their results, a man-in-the-middle could modify the results so the client might select a weaker algorithm from the set allowed by the server. That would make the client and/or server vulnerable to further attacks 😱.
The third operation that SHOULD use integrity protection is any GETATTR for the fs_locations and fs_locations_info attributes, to mitigate the severity of a man-in-the-middle attack. The attack has two steps. First, the attacker modifies the unprotected results of some operation to return NFS4ERR_MOVED. Second, when the client follows up with a GETATTR for fs_locations or fs_locations_info, the attacker modifies the results to make the client migrate its traffic to a server the attacker controls. With integrity protection, this attack is mitigated ✨.

Compared to previous NFS versions, NFSv4.1 has additional security considerations for pNFS (see Sections 12.9 and 13.12), locking and session state (see Section 2.10.8.3), and state recovery during the grace period (see Section 8.4.2.1.1). For locking and session state, if SP4_SSV state protection is being used, Section 2.10.10 has specific security considerations for the NFSv4.1 client and server 🧠.

Security considerations for lock reclaim are different between the two situations where state reclaim happens. The server failure situation is discussed in Section 8.4.2.1.1, while the per-fs state reclaim done for migration/replication is discussed in Section 11.11.9.1 🌸.

The use of the multi-server namespace features described in Section 11 raises the possibility that requests to determine the set of network addresses for a given server might be interfered with, or have their responses modified in flight. Given that possibility, the following considerations should be noted:

When DNS is used to convert server names to addresses and DNSSEC [29] is not available, the validity of the returned network addresses generally cannot be relied upon. However, when combined with a trusted resolver, DNS over TLS [30] and DNS over HTTPS [34] can be relied upon to provide valid address resolutions.

In situations where the validity of the provided addresses cannot be relied upon and the client uses RPCSEC_GSS to access the designated server, it is possible for mutual authentication to discover invalid server addresses, as long as the RPCSEC_GSS implementation used does not use insecure DNS queries to canonicalize the hostname components of the service principal names, as explained in [28] 🕵️.

The fetching of attributes containing file system location information SHOULD be performed using integrity protection. It's important to know that a client making this sort of request without integrity protection needs to be aware of the negative consequences, which can lead to invalid hostnames or network addresses being returned. These include cases where the client is directed to a server under an attacker's control, and the attacker might get access to data written or provide incorrect values for data read 😨. So the client needs to recognize that using such returned location information to access an NFSv4 server without RPCSEC_GSS (i.e., by using AUTH_SYS) is dangerous, because the client can end up interacting with such an attacker-controlled server without any authentication facilities to verify the server's identity.
Despite the fact that it is a requirement that implementations provide "support" for RPCSEC_GSS, it cannot be assumed that RPCSEC_GSS is always available between any particular client-server pair.
When a client has the network addresses of a server but not the associated hostnames, that would interfere with its ability to use RPCSEC_GSS.

In light of the above, a server SHOULD present file system location entries that correspond to file systems on other servers using a hostname. This would let the client interrogate the fs_locations on the destination server to obtain trunking information (as well as replica information) using integrity protection. That way the name provided gets validated, and the client can be sure the response hasn't been modified in flight 🦋.

When RPCSEC_GSS is not available on a server, the client needs to be aware that the location entries are subject to modification in flight, so they cannot be relied upon. If a client is directed to another server after NFS4ERR_MOVED, this could vitiate the authentication provided by RPCSEC_GSS on the designated destination server. Even when RPCSEC_GSS authentication is available on the destination, the server might still properly authenticate as the server the client was erroneously directed to. Without a way to decide whether the server is a valid one, the client can only determine, using RPCSEC_GSS, that the server corresponds to the name provided, with no basis for trusting that server. As a result, the client SHOULD NOT use such unverified location entries as a basis for migration, even though RPCSEC_GSS might be available on the destination 🚫.

When a file system location attribute is fetched upon connecting with an NFS server, it SHOULD, as stated above, be done with integrity protection. When this is not possible, it is generally best for the client to ignore trunking and replica information or simply not fetch the location information for these purposes.

When location information cannot be verified, it can be subjected to additional filtering to keep the client from being inappropriately directed. For example, if a range of network addresses can be determined that assures the servers and clients using AUTH_SYS are subject to the appropriate set of constraints (e.g., physical network isolation, administrative controls on the operating systems used), then network addresses in that appropriate range can be used, with others discarded or restricted in their use of AUTH_SYS 🧱.

To sum up the considerations for using RPCSEC_GSS when fetching location information, we need to consider the following possibilities for requests to interrogate location information, with interrogation approaches on the referring and destination servers arrived at separately 📋:

The use of integrity protection is RECOMMENDED in all cases, since the absence of integrity protection exposes the client to the possibility of the results being modified in transit.
The use of requests issued without RPCSEC_GSS (i.e., using AUTH_SYS, which has no provision to avoid modification of data in flight), while undesirable and a potential security exposure, may not be avoidable in all cases. Where the use of the returned information cannot be avoided, it is made subject to the filtering described above, to eliminate the possibility that the client would treat an invalid address as if it were an NFSv4 server. The specifics will vary depending on the degree of network isolation and on whether the request is to the referring or destination servers.

Even if such requests are not interfered with in flight, it is possible for a compromised server to direct the client to inappropriate servers, such as those under the attacker's control. It is not clear that being directed to such servers represents a greater threat to the client than the damage that could be done by the compromised server itself. However, it is possible that some sorts of transient server compromises might be exploited to direct a client to a server capable of doing greater damage over a longer time. One useful step to guard against this possibility is to issue requests to fetch location data using RPCSEC_GSS, even if no mapping to an RPCSEC_GSS principal is available. In this case, RPCSEC_GSS would not be used, as it typically is, to identify the client principal to the server, but rather to make sure (via RPCSEC_GSS mutual authentication) that the server being contacted is the one intended 🚨.

Similar considerations apply if the threat to be avoided is the redirection of client traffic to inappropriate (i.e., poorly performing) servers. In both cases, there is no reason for the information returned to depend on the identity of the client principal requesting it, while the validity of the server information, which has the capability to affect all client principals, is of considerable importance 💜.

22. IANA Considerations 🧚

This section uses terms that are defined in [63].

22.1. IANA Actions 🎀

Okay bestie, this update doesn't need any changes to the registry entries or registry rules for NFSv4.1 💅 BUT since it obsoletes RFC 5661, IANA went through and updated every registry entry and rule that pointed at RFC 5661 so they point here now. Iconic housekeeping 🫶

The remaining subsections of Section 22 list the earlier IANA actions related to NFSv4.1 ✨

22.2. Named Attribute Definitions 🦋

IANA made a registry called the "NFSv4 Named Attribute Definitions Registry" 🍓

So here's the tea: NFSv4.1 lets a file have zero or more named attributes, and each one is identified by a string name. The protocol doesn't decide how the namespace for these file attributes gets assigned. The registry is there so everyone stays on the same page where it counts ✨ App developers are totally allowed to define and use attributes as needed, but they're encouraged to register them with IANA 💜

Registered named attributes are presumed to apply to all minor versions of NFSv4, including the ones defined after the registration. If an attribute is meant to be limited to specific minor versions, the registry assignment will say so clearly 💅

All assignments to the registry are made on a First Come First Served basis, per Section 4.4 of [63]. The policy for each assignment is Specification Required, per Section 4.6 of [63].

Now the length drama 😭 In theory, the name of a named attribute can be up to 2^(32) - 1 bytes long, but in practice NFSv4.1 clients and servers can't handle a string that long. So IANA should reject any assignment request with a named attribute that goes over 128 UTF-8 characters. To give the IESG flexibility to set up the bases of assignment for Experimental Use and Standards Action, the prefixes "EXPE" and "STDS" are Reserved. The named attribute with a zero-length name is Reserved too 🙈

The prefix "PRIV" is designated for Private Use. If a site wants to use unregistered named attributes without risking a clash with an assignment in IANA's registry, it should use the "PRIV" prefix on all of its named attributes 🔐

Because some NFSv4.1 clients and servers are case-insensitive, the fifteen additional lower case and mixed case permutations of each of "EXPE", "PRIV", and "STDS" are Reserved (e.g., "expe", "expE", "exPe", etc. are Reserved). Also, IANA must not allow two assignments that would conflict if both named attributes were converted to a common case. Not allowed, periodt 💀

The registry of named attributes is a list of assignments, and each assignment has three fields:

A US-ASCII string name that is the actual name of the attribute. This name must be unique, and it can be 1 to 128 UTF-8 characters long.
A reference to the specification of the named attribute. The reference can consume up to 256 bytes (or more if IANA permits).
The point of contact of the registrant. The point of contact can consume up to 256 bytes (or more if IANA permits).

22.2.1. Initial Registry

There is no initial registry. The list starts out empty, no cap 🧚

22.2.2. Updating Registrations

The registrant is always permitted to update the point of contact field. Any other change will require Expert Review or IESG Approval 💖

22.3. Device ID Notifications 🌸

IANA created a registry called the "NFSv4 Device ID Notifications Registry" ✨

Okay, so new notification types could get added to the CB_NOTIFY_DEVICEID operation (see Section 20.12). That can happen through changes to the operations that register notifications, or by adding new operations to NFSv4. Either way it needs a new minor version of NFSv4, plus a Standards Track document from the IETF. Another route is to specify a new layout type (see Section 22.5).

So all assignments to the registry are made on a Standards Action basis per Section 4.6 of [63], with Expert Review required 🔥

The registry is a list of assignments, and each assignment has five fields:

The name of the notification type. This name must have the prefix "NOTIFY_DEVICEID4_", and it must be unique.
The value of the notification. IANA will assign this number, and the registrant's request will use TBD1 instead of an actual value. IANA MUST use a whole number that can be no higher than 2^(32)-1, and should be the next available value. The value assigned must be unique. A Designated Expert must be used to make sure that when the name of the notification type and its value are added to the NFSv4.1 notify_deviceid_type4 enumerated data type in the NFSv4.1 XDR description [10], the result continues to be a valid XDR description 🫶
The Standards Track RFC(s) that describe the notification. If the RFC(s) haven't been published yet, the registrant will use RFCTBD2, RFCTBD3, etc. instead of an actual RFC number.
How the RFC introduces the notification. This is shown by a single US-ASCII value. If the value is N, it means a minor revision to the NFSv4 protocol. If the value is L, it means a new pNFS layout type. Other values can be used with IESG Approval.
The minor versions of NFSv4 that are allowed to use the notification. These are numeric values, but IANA won't allocate or assign them. The author of the relevant RFCs assigns these numbers, with IESG Approval. Each time a new minor version of NFSv4 is approved, a Designated Expert should review the registry to make recommended updates as needed 👑

22.3.1. Initial Registry

The initial registry is in Table 25. Note that the next available value is zero.

+=========================+=======+==========+=====+================+
| Notification Name       | Value | RFC      | How | Minor Versions |
+=========================+=======+==========+=====+================+
| NOTIFY_DEVICEID4_CHANGE | 1     | RFC      | N   | 1              |
|                         |       | 8881     |     |                |
+-------------------------+-------+----------+-----+----------------+
| NOTIFY_DEVICEID4_DELETE | 2     | RFC      | N   | 1              |
|                         |       | 8881     |     |                |
+-------------------------+-------+----------+-----+----------------+

            Table 25: Initial Device ID Notification Assignments

22.3.2. Updating Registrations

The update of a registration will require IESG Approval on the advice of a Designated Expert 💜

22.4. Object Recall Types 🦋

IANA created a registry called the "NFSv4 Recallable Object Types Registry" 🌈

The potential exists for new object types to be added to the CB_RECALL_ANY operation (see Section 20.6). This can happen through changes to the operations that add recallable types, or by adding new operations to NFSv4. That requires a new minor version of NFSv4, plus a Standards Track document from IETF. Another way to add a new recallable object is to specify a new layout type (see Section 22.5).

All assignments to the registry are made on a Standards Action basis per Section 4.9 of [63], with Expert Review required ✨

Recallable object types are 32-bit unsigned numbers. There are no Reserved values. Values in the range 12 through 15, inclusive, are designated for Private Use 🔐

The registry is a list of assignments, and each assignment has five fields:

The name of the recallable object type. This name must have the prefix "RCA4_TYPE_MASK_". The name must be unique.
The value of the recallable object type. IANA will assign this number, and the registrant's request will use TBD1 instead of an actual value. IANA MUST use a whole number that can be no higher than 2^(32)-1, and should be the next available value. The value must be unique. A Designated Expert must be used to make sure that when the name of the recallable type and its value are added to the NFSv4 XDR description [10], the result continues to be a valid XDR description 🎀
The Standards Track RFC(s) that describe the recallable object type. If the RFC(s) haven't been published yet, the registrant will use RFCTBD2, RFCTBD3, etc. instead of an actual RFC number.
How the RFC introduces the recallable object type. This is shown by a single US-ASCII value. If the value is N, it means a minor revision to the NFSv4 protocol. If the value is L, it means a new pNFS layout type. Other values can be used with IESG Approval.
The minor versions of NFSv4 that are allowed to use the recallable object type. These are numeric values, but IANA won't allocate or assign them. The author of the relevant RFCs assigns these numbers, with IESG Approval. Each time a new minor version of NFSv4 is approved, a Designated Expert should review the registry to make recommended updates as needed 👑

22.4.1. Initial Registry

The initial registry is in Table 26. Note that the next available value is five.

     +===============================+=======+======+=====+==========+
     | Recallable Object Type Name   | Value | RFC  | How | Minor    |
     |                               |       |      |     | Versions |
     +===============================+=======+======+=====+==========+
     | RCA4_TYPE_MASK_RDATA_DLG      | 0     | RFC  | N   | 1        |
     |                               |       | 8881 |     |          |
     +-------------------------------+-------+------+-----+----------+
     | RCA4_TYPE_MASK_WDATA_DLG      | 1     | RFC  | N   | 1        |
     |                               |       | 8881 |     |          |
     +-------------------------------+-------+------+-----+----------+
     | RCA4_TYPE_MASK_DIR_DLG        | 2     | RFC  | N   | 1        |
     |                               |       | 8881 |     |          |
     +-------------------------------+-------+------+-----+----------+
     | RCA4_TYPE_MASK_FILE_LAYOUT    | 3     | RFC  | N   | 1        |
     |                               |       | 8881 |     |          |
     +-------------------------------+-------+------+-----+----------+
     | RCA4_TYPE_MASK_BLK_LAYOUT     | 4     | RFC  | L   | 1        |
     |                               |       | 8881 |     |          |
     +-------------------------------+-------+------+-----+----------+
     | RCA4_TYPE_MASK_OBJ_LAYOUT_MIN | 8     | RFC  | L   | 1        |
     |                               |       | 8881 |     |          |
     +-------------------------------+-------+------+-----+----------+
     | RCA4_TYPE_MASK_OBJ_LAYOUT_MAX | 9     | RFC  | L   | 1        |
     |                               |       | 8881 |     |          |
     +-------------------------------+-------+------+-----+----------+

            Table 26: Initial Recallable Object Type Assignments

22.4.2. Updating Registrations

The update of a registration will require IESG Approval on the advice of a Designated Expert 🥹

22.5. Layout Types 🧚

okay so IANA set up a registry called the "pNFS Layout Types Registry" 🗂️✨ Every single assignment goes through the Standards Action process, with Expert Review required 💅

Layout types are 32-bit numbers 🔢 The value zero is Reserved 🚫 Values from 0x80000000 to 0xFFFFFFFF (inclusive) are Private Use 🔒 IANA will assign numbers from 0x00000001 up to 0x7FFFFFFF (inclusive) 🦋

The registry is a list of assignments, and each one has five fields 📋

The name of the layout type. The name must have the prefix "LAYOUT4_" 🏷️ and it must be unique.
The value of the layout type. IANA assigns this number, so the registrant puts TBD1 in the request instead of a real value 🙈 The value assigned must be unique. A Designated Expert has to make sure that when the name and value get added to the NFSv4.1 layouttype4 enumerated data type in the NFSv4.1 XDR description [10], the result is still a valid XDR description 🧐
The Standards Track RFC(s) that describe the notification 📜 If they aren't published yet, the registrant uses RFCTBD2, RFCTBD3, and so on instead of an RFC number. Together, the RFC(s) must follow the guidelines in Section 22.5.3 ✨
How the RFC introduces the layout type. This is a single US-ASCII value 🔤 N means a minor revision to the NFSv4 protocol, and L means a brand new pNFS layout type 💖 Other values can be used with IESG Approval 👑
The minor versions of NFSv4 that are allowed to use the notification 🌈 These are numeric values, but IANA doesn't allocate or assign them. The author of the relevant RFCs assigns them with IESG Approval 🫶 Each time a new minor version of NFSv4 gets approved, a Designated Expert should review the registry and make recommended updates as needed 🔍

22.5.1. Initial Registry

okay here's the starting table 🍓 Table 27 is the initial registry:

    +=======================+=======+==========+=====+================+
    | Layout Type Name      | Value | RFC      | How | Minor Versions |
    +=======================+=======+==========+=====+================+
    | LAYOUT4_NFSV4_1_FILES | 0x1   | RFC 8881 | N   | 1              |
    +-----------------------+-------+----------+-----+----------------+
    | LAYOUT4_OSD2_OBJECTS  | 0x2   | RFC 5664 | L   | 1              |
    +-----------------------+-------+----------+-----+----------------+
    | LAYOUT4_BLOCK_VOLUME  | 0x3   | RFC 5663 | L   | 1              |
    +-----------------------+-------+----------+-----+----------------+

                 Table 27: Initial Layout Type Assignments

22.5.2. Updating Registrations

updating a registration needs IESG Approval, on the advice of a Designated Expert 🫧

22.5.3. Guidelines for Writing Layout Type Specifications

ok this is the recipe for getting a new pNFS layout type accepted as a Standards Track RFC 👩‍🍳💕 The author of a new pNFS layout spec follows these steps:

The author devises the new layout specification 💡
The new layout type specification MUST, at a minimum:
- Define the contents of the layout-type-specific fields of the following data types:
  - the da_addr_body field of the device_addr4 data type;
  - the loh_body field of the layouthint4 data type;
  - the loc_body field of layout_content4 data type (which in turn is the lo_content field of the layout4 data type);
  - the lou_body field of the layoutupdate4 data type;
- Describe or define the storage access protocol used to access the storage devices 🗄️
- Describe whether revocation of layouts is supported 🙋‍♀️
- At a minimum, describe the methods of recovery from:
  1. Failure and restart for client, server, storage device 🔄
  2. Lease expiration from perspective of the active client, server, storage device ⏰
  3. Loss of layout state resulting in fencing of client access to storage devices (for an example, see Section 12.7.3) 🚧
- Include an IANA considerations section, which will in turn include:
  - A request to IANA for a new layout type per Section 22.5.
  - A list of requests to IANA for any new recallable object types for CB_RECALL_ANY; each entry is to be presented in the form described in Section 22.4.
  - A list of requests to IANA for any new notification values for CB_NOTIFY_DEVICEID; each entry is to be presented in the form described in Section 22.3.
- Include a security considerations section. This section MUST explain how the NFSv4.1 authentication, authorization, and access-control models are preserved 🔐 That is, if a metadata server would restrict a READ or WRITE operation, how would pNFS via the layout similarly restrict a corresponding input or output operation? 🤔
The author documents the new layout specification as an Internet-Draft 📝
The author submits the Internet-Draft for review through the IETF standards process, as defined in "The Internet Standards Process-- Revision 3" (BCP 9 [35]) 🚀 The new layout specification will be submitted for eventual publication as a Standards Track RFC 🌟
The layout specification progresses through the IETF standards process 🎉

22.6. Path Variable Definitions 🌸

this section is all about the IANA stuff for the variable substitution feature for location names, the one described in Section 11.17.3 🎀 Variables that get substituted are made of a domain name and a specific name inside that domain, separated by a colon 🧩 There are two sets of IANA considerations here:

The list of variable names.
For each variable name, the list of possible values.

So there's going to be one registry for the list of variable names, and possibly one registry for listing the values of each variable name 💜

22.6.1. Path Variables Registry

IANA created a registry called the "NFSv4 Path Variables Registry" 🗃️

22.6.1.1. Path Variable Values

okay the naming rules are a whole thing, so bear with me 🥹 Variable names look like "${", then a domain name, then a colon (":"), then a domain-specific part of the name, then "}" 🌈 When the domain name is "ietf.org", every variable name has to be registered with IANA on a Standards Action basis, with Expert Review required 👑 Path variables with registered domain names that are neither part of nor equal to ietf.org get assigned on a Hierarchical Allocation basis, which means they get delegated to the domain owner, so IANA doesn't really care unless the domain owner decides to register a variable name from their domain 🫠 If they do, IANA handles it on a First Come First Serve basis 🙌 For people who don't have their own domain, IANA will take requests to register variables with the prefix "${FCFS.ietf.org:" on a First Come First Serve basis 💖 Assignments made on a First Come First Serve basis don't need Expert Review, unless the registrant also wants IANA to set up a registry for the values of the registered variable 🧁

The registry is a list of assignments, and each one has three fields 📋

The name of the variable. The name must start with "${" followed by a registered domain name, followed by ":", or it must start with "${FCFS.ietf.org". The name must be no more than 64 UTF-8 characters long, and it must be unique 🎯
For assignments made on a Standards Action basis, the Standards Track RFC(s) that describe the variable 📚 If the RFC(s) aren't published yet, the registrant uses RFCTBD1, RFCTBD2, and so on instead of an RFC number. Note that the RFCs don't have to be part of an NFS minor version 🤍 For assignments made on a First Come First Serve basis, there's an explanation of the variable's purpose (no more than 1024 bytes, or more if IANA permits) 📝 A reference to the explanation can be substituted instead 🔗
The point of contact, including an email address 📧 The point of contact can be up to 256 bytes (or more if IANA permits). For assignments made on a Standards Action basis, the point of contact is always IESG 👑

22.6.1.1.1. Initial Registry

here's the starting list, Table 28 🍬

         +========================+==========+==================+
         | Variable Name          | RFC      | Point of Contact |
         +========================+==========+==================+
         | ${ietf.org:CPU_ARCH}   | RFC 8881 | IESG             |
         +------------------------+----------+------------------+
         | ${ietf.org:OS_TYPE}    | RFC 8881 | IESG             |
         +------------------------+----------+------------------+
         | ${ietf.org:OS_VERSION} | RFC 8881 | IESG             |
         +------------------------+----------+------------------+

                 Table 28: Initial List of Path Variables

IANA has created registries for the values of the variable names ${ietf.org:CPU_ARCH} and ${ietf.org:OS_TYPE} ✨ See Sections 22.6.2 and 22.6.3 🧚

For the values of ${ietf.org:OS_VERSION}, no registry is needed, because the specifics of the values will vary with the value of ${ietf.org:OS_TYPE} 🤷‍♀️ So values for ${ietf.org:OS_VERSION} are on a Hierarchical Allocation basis and are of no concern to IANA 🫶

22.6.1.1.2. Updating Registrations

updating an assignment made on a Standards Action basis needs IESG Approval, on the advice of a Designated Expert 👑

The registrant can always update the point of contact of an assignment made on a First Come First Serve basis 💅 Any other update needs Expert Review 🔍

22.6.2. Values for the `${ietf.org:CPU_ARCH}` Variable

IANA created a registry called the "NFSv4 ${ietf.org:CPU_ARCH} Value Registry" 🖥️

assignments to this registry are made on a First Come First Serve basis 🙌 The zero-length value of ${ietf.org:CPU_ARCH} is Reserved 🚫 Values with a prefix of "PRIV" are designated for Private Use 🔒

The registry is a list of assignments, and each one has three fields 📋

A value of the ${ietf.org:CPU_ARCH} variable. The value must be 1 to 32 UTF-8 characters long, and it must be unique 🎀
An explanation (no more than 1024 bytes, or more if IANA permits) of what CPU architecture the value denotes 🧠 A reference to the explanation can be substituted instead 🔗
The point of contact, including an email address 📧 The point of contact can be up to 256 bytes (or more if IANA permits) 🌷

22.6.2.1. Initial Registry

there is no initial registry 🫧

22.6.2.2. Updating Registrations

the registrant is free to update the assignment, so they can change the explanation and/or point-of-contact fields 🦋

22.6.3. Values for the `${ietf.org:OS_TYPE}` Variable 💻✨

okay so IANA made a registry called the "NFSv4 ${ietf.org:OS_TYPE} Value Registry" 🗂️💖

assignments go in on a First Come First Serve basis 🏃‍♀️ and the zero-length value of ${ietf.org:OS_TYPE} is Reserved 🚫. anything starting with "PRIV" is designated for Private Use 🔒 (iykyk)

the registry is a list of assignments, and each one has three fields 📋✨

a value of the ${ietf.org:OS_TYPE} variable. the value must be 1 to 32 UTF-8 characters long, and it must be unique 🔢
an explanation (consuming no more than 1024 bytes, or more if IANA permits) of what CPU architecture the value denotes. a reference to the explanation can be substituted instead 📝
the point of contact, including an email address. the point of contact can consume up to 256 bytes (or more if IANA permits) 📮

22.6.3.1. Initial Registry

there is no initial registry 🫥 nothing to see here, bestie

22.6.3.2. Updating Registrations

the registrant is free to update the assignment, so they can change the explanation and/or point of contact fields 💅🌸

23. References 📚

23.1. Normative References

here's the whole reference list, every single entry, no cap 🎀

[1] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, https://www.rfc-editor.org/info/rfc2119.
[2] Eisler, M., Ed., "XDR: External Data Representation Standard", STD 67, RFC 4506, DOI 10.17487/RFC4506, May 2006, https://www.rfc-editor.org/info/rfc4506.
[3] Thurlow, R., "RPC: Remote Procedure Call Protocol Specification Version 2", RFC 5531, DOI 10.17487/RFC5531, May 2009, https://www.rfc-editor.org/info/rfc5531.
[4] Eisler, M., Chiu, A., and L. Ling, "RPCSEC_GSS Protocol Specification", RFC 2203, DOI 10.17487/RFC2203, September 1997, https://www.rfc-editor.org/info/rfc2203.
[5] Zhu, L., Jaganathan, K., and S. Hartman, "The Kerberos Version 5 Generic Security Service Application Program Interface (GSS-API) Mechanism: Version 2", RFC 4121, DOI 10.17487/RFC4121, July 2005, https://www.rfc-editor.org/info/rfc4121.
[6] The Open Group, "Section 3.191 of Chapter 3 of Base Definitions of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version", ISBN 1931624232, 2004, https://www.opengroup.org.
[7] Linn, J., "Generic Security Service Application Program Interface Version 2, Update 1", RFC 2743, DOI 10.17487/RFC2743, January 2000, https://www.rfc-editor.org/info/rfc2743.
[8] Recio, R., Metzler, B., Culley, P., Hilland, J., and D. Garcia, "A Remote Direct Memory Access Protocol Specification", RFC 5040, DOI 10.17487/RFC5040, October 2007, https://www.rfc-editor.org/info/rfc5040.
[9] Eisler, M., "RPCSEC_GSS Version 2", RFC 5403, DOI 10.17487/RFC5403, February 2009, https://www.rfc-editor.org/info/rfc5403.
[10] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "Network File System (NFS) Version 4 Minor Version 1 External Data Representation Standard (XDR) Description", RFC 5662, DOI 10.17487/RFC5662, January 2010, https://www.rfc-editor.org/info/rfc5662.
[11] The Open Group, "Section 3.372 of Chapter 3 of Base Definitions of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version", ISBN 1931624232, 2004, https://www.opengroup.org.
[12] Eisler, M., "IANA Considerations for Remote Procedure Call (RPC) Network Identifiers and Universal Address Formats", RFC 5665, DOI 10.17487/RFC5665, January 2010, https://www.rfc-editor.org/info/rfc5665.
[13] The Open Group, "Section 'read()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version", ISBN 1931624232, 2004, https://www.opengroup.org.
[14] The Open Group, "Section 'readdir()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version", ISBN 1931624232, 2004, https://www.opengroup.org.
[15] The Open Group, "Section 'write()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version", ISBN 1931624232, 2004, https://www.opengroup.org.
[16] Hoffman, P. and M. Blanchet, "Preparation of Internationalized Strings ("stringprep")", RFC 3454, DOI 10.17487/RFC3454, December 2002, https://www.rfc-editor.org/info/rfc3454.
[17] The Open Group, "Section 'chmod()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version", ISBN 1931624232, 2004, https://www.opengroup.org.
[18] International Organization for Standardization, "Information Technology - Universal Multiple-octet coded Character Set (UCS) - Part 1: Architecture and Basic Multilingual Plane", ISO Standard 10646-1, May 1993.
[19] Alvestrand, H., "IETF Policy on Character Sets and Languages", BCP 18, RFC 2277, DOI 10.17487/RFC2277, January 1998, https://www.rfc-editor.org/info/rfc2277.
[20] Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep Profile for Internationalized Domain Names (IDN)", RFC 3491, DOI 10.17487/RFC3491, March 2003, https://www.rfc-editor.org/info/rfc3491.
[21] The Open Group, "Section 'fcntl()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version", ISBN 1931624232, 2004, https://www.opengroup.org.
[22] The Open Group, "Section 'fsync()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version", ISBN 1931624232, 2004, https://www.opengroup.org.
[23] The Open Group, "Section 'getpwnam()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version", ISBN 1931624232, 2004, https://www.opengroup.org.
[24] The Open Group, "Section 'unlink()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version", ISBN 1931624232, 2004, https://www.opengroup.org.
[25] Schaad, J., Kaliski, B., and R. Housley, "Additional Algorithms and Identifiers for RSA Cryptography for use in the Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile", RFC 4055, DOI 10.17487/RFC4055, June 2005, https://www.rfc-editor.org/info/rfc4055.
[26] National Institute of Standards and Technology, "Computer Security Objects Register", May 2016, https://csrc.nist.gov/projects/computer-security-objects-register/algorithm-registration.
[27] Adamson, A. and N. Williams, "Remote Procedure Call (RPC) Security Version 3", RFC 7861, DOI 10.17487/RFC7861, November 2016, https://www.rfc-editor.org/info/rfc7861.
[28] Neuman, C., Yu, T., Hartman, S., and K. Raeburn, "The Kerberos Network Authentication Service (V5)", RFC 4120, DOI 10.17487/RFC4120, July 2005, https://www.rfc-editor.org/info/rfc4120.
[29] Arends, R., Austein, R., Larson, M., Massey, D., and S. Rose, "DNS Security Introduction and Requirements", RFC 4033, DOI 10.17487/RFC4033, March 2005, https://www.rfc-editor.org/info/rfc4033.
[30] Hu, Z., Zhu, L., Heidemann, J., Mankin, A., Wessels, D., and P. Hoffman, "Specification for DNS over Transport Layer Security (TLS)", RFC 7858, DOI 10.17487/RFC7858, May 2016, https://www.rfc-editor.org/info/rfc7858.
[31] Adamson, A. and N. Williams, "Requirements for NFSv4 Multi-Domain Namespace Deployment", RFC 8000, DOI 10.17487/RFC8000, November 2016, https://www.rfc-editor.org/info/rfc8000.
[32] Lever, C., Ed., Simpson, W., and T. Talpey, "Remote Direct Memory Access Transport for Remote Procedure Call Version 1", RFC 8166, DOI 10.17487/RFC8166, June 2017, https://www.rfc-editor.org/info/rfc8166.
[33] Lever, C., "Network File System (NFS) Upper-Layer Binding to RPC-over-RDMA Version 1", RFC 8267, DOI 10.17487/RFC8267, October 2017, https://www.rfc-editor.org/info/rfc8267.
[34] Hoffman, P. and P. McManus, "DNS Queries over HTTPS (DoH)", RFC 8484, DOI 10.17487/RFC8484, October 2018, https://www.rfc-editor.org/info/rfc8484.
[35] Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996.
Kolkman, O., Bradner, S., and S. Turner, "Characterization of Proposed Standards", BCP 9, RFC 7127, January 2014.
Dusseault, L. and R. Sparks, "Guidance on Interoperation and Implementation Reports for Advancement to Draft Standard", BCP 9, RFC 5657, September 2009.
Housley, R., Crocker, D., and E. Burger, "Reducing the Standards Track to Two Maturity Levels", BCP 9, RFC 6410, October 2011.
Resnick, P., "Retirement of the "Internet Official Protocol Standards" Summary Document", BCP 9, RFC 7100, December 2013.
Dawkins, S., "Increasing the Number of Area Directors in an IETF Area", BCP 9, RFC 7475, March 2015.
https://www.rfc-editor.org/info/bcp9

23.2. Informative References 📚✨

okay bestie, these are the extra reads 🧚 the spec points to for background, history, and deeper dives, so here they are in full 💅

[36] Roach, A., "Process for Handling Non-Major Revisions to Existing RFCs", Work in Progress, Internet-Draft, draft-roach-bis-documents-00, 7 May 2019, <https://tools.ietf.org/html/draft-roach-bis-documents-00>.
[37] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame, C., Eisler, M., and D. Noveck, "Network File System (NFS) version 4 Protocol", RFC 3530, DOI 10.17487/RFC3530, April 2003, <https://www.rfc-editor.org/info/rfc3530>.
[38] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS Version 3 Protocol Specification", RFC 1813, DOI 10.17487/RFC1813, June 1995, <https://www.rfc-editor.org/info/rfc1813>.
[39] Eisler, M., "LIPKEY - A Low Infrastructure Public Key Mechanism Using SPKM", RFC 2847, DOI 10.17487/RFC2847, June 2000, <https://www.rfc-editor.org/info/rfc2847>.
[40] Eisler, M., "NFS Version 2 and Version 3 Security Issues and the NFS Protocol's Use of RPCSEC_GSS and Kerberos V5", RFC 2623, DOI 10.17487/RFC2623, June 1999, <https://www.rfc-editor.org/info/rfc2623>.
[41] Juszczak, C., "Improving the Performance and Correctness of an NFS Server", USENIX Conference Proceedings, June 1990.
[42] Reynolds, J., Ed., "Assigned Numbers: RFC 1700 is Replaced by an On-line Database", RFC 3232, DOI 10.17487/RFC3232, January 2002, <https://www.rfc-editor.org/info/rfc3232>.
[43] Srinivasan, R., "Binding Protocols for ONC RPC Version 2", RFC 1833, DOI 10.17487/RFC1833, August 1995, <https://www.rfc-editor.org/info/rfc1833>.
[44] Werme, R., "RPC XID Issues", USENIX Conference Proceedings, February 1996.
[45] Nowicki, B., "NFS: Network File System Protocol specification", RFC 1094, DOI 10.17487/RFC1094, March 1989, <https://www.rfc-editor.org/info/rfc1094>.
[46] Bhide, A., Elnozahy, E. N., and S. P. Morgan, "A Highly Available Network Server", USENIX Conference Proceedings, January 1991.
[47] Halevy, B., Welch, B., and J. Zelenka, "Object-Based Parallel NFS (pNFS) Operations", RFC 5664, DOI 10.17487/RFC5664, January 2010, <https://www.rfc-editor.org/info/rfc5664>.
[48] Black, D., Fridella, S., and J. Glasgow, "Parallel NFS (pNFS) Block/Volume Layout", RFC 5663, DOI 10.17487/RFC5663, January 2010, <https://www.rfc-editor.org/info/rfc5663>.
[49] Callaghan, B., "WebNFS Client Specification", RFC 2054, DOI 10.17487/RFC2054, October 1996, <https://www.rfc-editor.org/info/rfc2054>.
[50] Callaghan, B., "WebNFS Server Specification", RFC 2055, DOI 10.17487/RFC2055, October 1996, <https://www.rfc-editor.org/info/rfc2055>.
[51] IESG, "IESG Processing of RFC Errata for the IETF Stream", July 2008, <https://www.ietf.org/about/groups/iesg/statements/processing-rfc-errata/>.
[52] Krawczyk, H., Bellare, M., and R. Canetti, "HMAC: Keyed-Hashing for Message Authentication", RFC 2104, DOI 10.17487/RFC2104, February 1997, <https://www.rfc-editor.org/info/rfc2104>.
[53] Shepler, S., "NFS Version 4 Design Considerations", RFC 2624, DOI 10.17487/RFC2624, June 1999, <https://www.rfc-editor.org/info/rfc2624>.
[54] The Open Group, "Protocols for Interworking: XNFS, Version 3W", ISBN 1-85912-184-5, February 1998.
[55] Floyd, S. and V. Jacobson, "The Synchronization of Periodic Routing Messages", IEEE/ACM Transactions on Networking, 2(2), pp. 122-136, April 1994.
[56] Chadalapaka, M., Satran, J., Meth, K., and D. Black, "Internet Small Computer System Interface (iSCSI) Protocol (Consolidated)", RFC 7143, DOI 10.17487/RFC7143, April 2014, <https://www.rfc-editor.org/info/rfc7143>.
[57] Snively, R., "Fibre Channel Protocol for SCSI, 2nd Version (FCP-2)", ANSI/INCITS, 350-2003, October 2003.
[58] Weber, R.O., "Object-Based Storage Device Commands (OSD)", ANSI/INCITS, 400-2004, July 2004, <https://www.t10.org/drafts.htm>.
[59] Carns, P. H., Ligon III, W. B., Ross, R. B., and R. Thakur, "PVFS: A Parallel File System for Linux Clusters.", Proceedings of the 4th Annual Linux Showcase and Conference, 2000.
[60] The Open Group, "The Open Group Base Specifications Issue 6, IEEE Std 1003.1, 2004 Edition", 2004, <https://www.opengroup.org>.
[61] Callaghan, B., "NFS URL Scheme", RFC 2224, DOI 10.17487/RFC2224, October 1997, <https://www.rfc-editor.org/info/rfc2224>.
[62] Chiu, A., Eisler, M., and B. Callaghan, "Security Negotiation for WebNFS", RFC 2755, DOI 10.17487/RFC2755, January 2000, <https://www.rfc-editor.org/info/rfc2755>.
[63] Cotton, M., Leiba, B., and T. Narten, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 8126, DOI 10.17487/RFC8126, June 2017, <https://www.rfc-editor.org/info/rfc8126>.
[64] RFC Errata, Erratum ID 2006, RFC 5661, <https://www.rfc-editor.org/errata/eid2006>.
[65] Spasojevic, M. and M. Satayanarayanan, "An Empirical Study of a Wide-Area Distributed File System", ACM Transactions on Computer Systems, Vol. 14, No. 2, pp. 200-222, DOI 10.1145/227695.227698, May 1996, <https://doi.org/10.1145/227695.227698>.
[66] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "Network File System (NFS) Version 4 Minor Version 1 Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010, <https://www.rfc-editor.org/info/rfc5661>.
[67] Noveck, D., "Rules for NFSv4 Extensions and Minor Versions", RFC 8178, DOI 10.17487/RFC8178, July 2017, <https://www.rfc-editor.org/info/rfc8178>.
[68] Haynes, T., Ed. and D. Noveck, Ed., "Network File System (NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530, March 2015, <https://www.rfc-editor.org/info/rfc7530>.
[69] Noveck, D., Ed., Shivam, P., Lever, C., and B. Baker, "NFSv4.0 Migration: Specification Update", RFC 7931, DOI 10.17487/RFC7931, July 2016, <https://www.rfc-editor.org/info/rfc7931>.
[70] Haynes, T., "Requirements for Parallel NFS (pNFS) Layout Types", RFC 8434, DOI 10.17487/RFC8434, August 2018, <https://www.rfc-editor.org/info/rfc8434>.
[71] Farrell, S. and H. Tschofenig, "Pervasive Monitoring Is an Attack", BCP 188, RFC 7258, DOI 10.17487/RFC7258, May 2014, <https://www.rfc-editor.org/info/rfc7258>.
[72] Rescorla, E. and B. Korver, "Guidelines for Writing RFC Text on Security Considerations", BCP 72, RFC 3552, DOI 10.17487/RFC3552, July 2003, <https://www.rfc-editor.org/info/rfc3552>.

Appendix A. The Need for This Update ✨

okay so this appendix is all about why this update even exists 🎀 It explains how clients and servers work out which network access paths to use to reach a file system 🦋 That covers what happens when the specific replica changes, or when the set of addresses used to reach it changes, and how to handle transfers of responsibility transparently 💖 It also covers cases where there's a shift from one replica to another, and cases where different network access paths get used to reach the same replica 🌸

Anyway, RFC 5661 [66] had some problems, so the specific updates in this document were needed 😭 Those updates are described in Appendix B.

RFC 5661 [66], while it dealt with situations in which various forms of clustering allowed coordination of the state assigned by cooperating servers to be used, made no provisions for Transparent State Migration. Within NFSv4.0, Transparent State Migration was first explained clearly in RFC 7530 [68] and corrected and clarified by RFC 7931 [69]. No corresponding explanation for NFSv4.1 had been provided. 💅
Although NFSv4.1 provided a clear definition of how trunking detection was to be done, there was no clear specification of how trunking discovery was to be done. That's wild, because the specification clearly indicated that this information could be made available via the file system location attributes. 🙈
Because the existence of multiple network access paths to the same file system was dealt with as if there were multiple replicas, issues relating to transitions between replicas could never be clearly distinguished from trunking-related transitions between the addresses used to access a particular file system instance. As a result, in situations in which both migration and trunking configuration changes were involved, neither of these could be clearly dealt with, and the relationship between these two features was not seriously addressed. 💀
Because use of two network access paths to the same file system instance (i.e., trunking) was often treated as if two replicas were involved, it was considered that two replicas were being used simultaneously. As a result, the treatment of replicas being used simultaneously in RFC 5661 [66] was not clear, as it covered the two distinct cases of a single file system instance being accessed by two different network access paths and two replicas being accessed simultaneously, with the limitations of the latter case not being clearly laid out. 🌈

The majority of the consequences of these issues are dealt with by presenting in Section 11 a replacement for Section 11 of RFC 5661 [66]. This replacement modifies existing subsections within that section and adds new ones as described in Appendix B.1. Also, some existing sections were deleted. These changes were made in order to do the following:

Reorganize the description so that the case of two network access paths to the same file system instance is distinguished clearly from the case of two different replicas since, in the former case, locking state is shared and there also can be sharing of session state. 🧚
Provide a clear statement regarding the desirability of transparent transfer of state between replicas together with a recommendation that either transparent transfer or a single-fs grace period be provided. ✨
Specifically delineate how a client is to handle such transfers, taking into account the differences from the treatment in [69] made necessary by the major protocol changes to NFSv4.1. 🫶
Discuss the relationship between transparent state transfer and Parallel NFS (pNFS). 🦋
Clarify the fs_locations_info attribute in order to specify which portions of the provided information apply to a specific network access path and which apply to the replica that the path is used to access. 👑

In addition, other sections of RFC 5661 [66] were updated to correct the consequences of the incorrect assumptions underlying the treatment of multi-server namespace issues. These are described in Appendices B.2 through B.4.

A revised introductory section regarding multi-server namespace facilities is provided. 🌸
A more realistic treatment of server scope is provided. This treatment reflects the more limited coordination of locking state adopted by servers actually sharing a common server scope. 🍓
Some confusing text regarding changes in server_owner has been clarified. 💜
The description of some existing errors has been modified to more clearly explain certain error situations to reflect the existence of trunking and the possible use of fs-specific grace periods. For details, see Appendix B.3. ⭐
New descriptions of certain existing operations are provided, either because the existing treatment did not account for situations that would arise in dealing with Transparent State Migration, or because some types of reclaim issues were not adequately dealt with in the context of fs-specific grace periods. For details, see Appendix B.2. 🎀

Appendix B. Changes in This Update 🌈

B.1. Revisions Made to Section 11 of RFC 5661

A number of areas have been revised or extended, in many cases replacing subsections within Section 11 of RFC 5661 [66]:

New introductory material, including a terminology section, replaces the material in RFC 5661 [66], ranging from the start of the original Section 11 up to and including Section 11.1. The new material starts at the beginning of Section 11 and continues through 11.2. 💖
A significant reorganization of the material in Sections 11.4 and 11.5 of RFC 5661 [66] was necessary. The reasons for the reorganization of these sections into a single section with multiple subsections are discussed in Appendix B.1.1 below. This replacement appears as Section 11.5.

New material relating to the handling of the file system location attributes is contained in Sections 11.5.1 and 11.5.7. 🦋

A new section describing requirements for user and group handling within a multi-server namespace has been added as Section 11.7. 🫶
A major replacement for Section 11.7 of RFC 5661 [66], entitled "Effecting File System Transitions", appears as Sections 11.9 through 11.14. The reasons for the reorganization of this section into multiple sections are discussed in Appendix B.1.2. 🔥
A replacement for Section 11.10 of RFC 5661 [66], entitled "The Attribute fs_locations_info", appears as Section 11.17, with Appendix B.1.3 describing the differences between the new section and the treatment within [66]. A revised treatment was necessary because the original treatment did not make clear how the added attribute information relates to the case of trunked paths to the same replica. These issues were not addressed in RFC 5661 [66] where the concepts of a replica and a network path used to access a replica were not clearly distinguished. 🌸

B.1.1. Reorganization of Sections 11.4 and 11.5 of RFC 5661

Previously, issues related to the fact that multiple location entries directed the client to the same file system instance were dealt with in Section 11.5 of RFC 5661 [66]. Because of the new treatment of trunking, these issues now belong within Section 11.5. 🧚

In this new section, trunking is covered in Section 11.5.2 together with the other uses of file system location information described in Sections 11.5.3 through 11.5.6.

As a result, Section 11.5, which replaces Section 11.4 of RFC 5661 [66], is substantially different than the section it replaces in that some original sections have been replaced by corresponding sections as described below, while new sections have been added:

The material in Section 11.5, exclusive of subsections, replaces the material in Section 11.4 of RFC 5661 [66] exclusive of subsections. 💅
Section 11.5.1 is the new first subsection of the overall section.
Section 11.5.2 is the new second subsection of the overall section.
Each of the Sections 11.5.4, 11.5.5, and 11.5.6 replaces (in order) one of the corresponding Sections 11.4.1, 11.4.2, and 11.4.3 of RFC 5661 [66].
Section 11.5.7 is the new final subsection of the overall section. 👑

B.1.2. Reorganization of Material Dealing with File System Transitions

okay so the file system transition stuff that used to live in Section 11.7 of RFC 5661 [66] got a total glow-up 💅✨ It was reorganized and beefed up like this:

Because the network access paths used to reach a file system instance can shift without any shift between replicas, a new Section 11.9 separates those cases: one where there's a shift between distinct replicas, and one where the access path changes but the replicas stay the same 🦋

So the new Section 11.10 handles network address transitions, while most of the old Section 11.7 of RFC 5661 [66] got extensively reworked into Section 11.11, which is now limited to cases where there's a shift between two different sets of replicas.

The extra Section 11.12 talks about moving to a different replica and transferring state so the client can keep access to its accumulated locking state on the new server 🔒
The extra Section 11.13 covers how the client responds to access transitions, how it figures out whether migration has occurred, and how it gets access to any transferred locking and session state 🧚
The extra Section 11.14 covers the responsibilities of the source and destination servers when transferring locking and session state 💜

This reorg renumbered the sections inside Section 11 of [66], like so:

The new Sections 11.9 and 11.10 bumped the existing sections with those numbers.
Section 11.7 of [66] got substantially modified and is now Section 11.11. The changes reflect that this section only deals with transitions between replicas, since transitions between network addresses are handled in other sections. There are more reorg details later in this section 🌸
Sections 11.12, 11.13, and 11.14 were added.
So Sections 11.8, 11.9, 11.10, and 11.11 in [66] are now Sections 11.15, 11.16, 11.17, and 11.18, respectively.

As part of the general reorg, Section 11.7 of RFC 5661 [66] was modified like this:

Sections 11.7 and 11.7.1 of RFC 5661 [66] were replaced by Sections 11.11 and 11.11.1, respectively.
Section 11.7.2 of RFC 5661 (and its included subsections) got deleted, bye bye 👋
Sections 11.7.3, 11.7.4, 11.7.5, 11.7.5.1, and 11.7.6 of RFC 5661 [66] were replaced by Sections 11.11.2, 11.11.3, 11.11.4, 11.11.4.1, and 11.11.5 respectively in this document.
Section 11.7.7 of RFC 5661 [66] was replaced by Section 11.11.9. This subsection got moved to the end of the section dealing with file system transitions.
Sections 11.7.8, 11.7.9, and 11.7.10 of RFC 5661 [66] were replaced by Sections 11.11.6, 11.11.7, and 11.11.8 respectively in this document.

B.1.3. Updates to the Treatment of fs_locations_info

ngl this one is so important 🥹 Various elements of the fs_locations_info attribute contain info that applies either to a specific file system replica or to a network path (or set of network paths) used to reach that replica. The original treatment of fs_locations_info (Section 11.10 of RFC 5661 [66]) didn't clearly separate these cases, partly because the document didn't clearly separate replicas from the paths used to access them 🙈

Some fields also got special clarification:

For handling FSLI4GF_GOING, it was clarified that this only applies to the unavailability of a replica, not to a path used to access a replica.
When describing the right value for a server to use for fli_valid_for, it was clarified that the client doesn't need to fetch the fs_locations_info value often to be ready for shifts in trunking patterns 🌈
The rules for extensions to the fls_info got clarified too. The original treatment reflected the extension model that was in effect when RFC 5661 [66] was written, and it's been updated to follow the extension model described in RFC 8178 [67] 💅

B.2. Revisions Made to Operations in RFC 5661

Descriptions got revised to address issues that popped up while making the changes needed for multi-server namespace features 🧚

The treatment of EXCHANGE_ID (Section 18.35 of RFC 5661 [66]) assumed that client IDs could only be created or confirmed by the EXCHANGE_ID and CREATE_SESSION operations. The necessary use of EXCHANGE_ID when recovering from migration and related situations also wasn't clearly addressed. A revised treatment was needed, and it lives in Section 18.35. The specific differences between it and the treatment in [66] are explained in Appendix B.2.1 below ✨
The treatment of RECLAIM_COMPLETE in Section 18.51 of RFC 5661 [66] wasn't clear enough about the purpose and use of the rca_one_fs, or how the server should deal with inappropriate values of this argument. The resulting confusion caused interoperability issues, so a new treatment of RECLAIM_COMPLETE was needed, and it appears in Section 18.51. The specific differences from the treatment in RFC 5661 [66] are discussed in Appendix B.2.2 below. The definitions of the reclaim-related errors also got an updated treatment in Section 15.1.9, to reflect that there are multiple contexts for lock reclaim operations 🔥

B.2.1. Revision of Treatment of EXCHANGE_ID

there were a number of issues in the original EXCHANGE_ID treatment in RFC 5661 [66] that caused problems for Transparent State Migration and for moving access between different network access paths to the same file system instance 😭

These issues came from the fact that the original treatment was written with two assumptions:

That a client ID can only become known to a server by being created through executing an EXCHANGE_ID, with confirmation of the ID only possible by executing a CREATE_SESSION.
That interactions between a client and a server only happen over a single network address.

Those assumptions stopped being valid once Transparent State Migration and active use of trunking came into the picture, so the treatment got modified in a few ways:

It had been assumed that an EXCHANGE_ID executed when the server already knew about a given client instance was either updating associated parameters (for example, with respect to callbacks) or retransmitting after a lost reply. As a result, any slot sequence returned by that operation would be of no use. The original treatment even said that it "MUST NOT" be used, although that usage wasn't in accord with [1]. This caused a problem when an EXCHANGE_ID is done after Transparent State Migration, since that slot sequence would need to be used in a later CREATE_SESSION 🤔

In the updated treatment, CREATE_SESSION is a way that client IDs are confirmed, but it's understood that other ways are possible. The slot sequence can be used as needed, and the cases where it would be of no use are noted appropriately.

It had been assumed that the only functions of EXCHANGE_ID were to inform the server of the client, to create the client ID, and to communicate it to the client. When multiple simultaneous connections are involved, as often happens with trunking, that treatment was inadequate, because it ignored the role of EXCHANGE_ID in associating the client ID with the connection it was done on. That association lets the ID be used by a later CREATE_SESSION whose parameters don't include an explicit client ID 🦋

The new treatment explicitly discusses the role of EXCHANGE_ID in associating the client ID with the connection, so it can be used by CREATE_SESSION, and in associating a connection with an existing session.

The new treatment can be found in Section 18.35 above. It supersedes the treatment in Section 18.35 of RFC 5661 [66].

B.2.2. Revision of Treatment of RECLAIM_COMPLETE

The following changes were made to the treatment of RECLAIM_COMPLETE in RFC 5661 [66] to arrive at the treatment in Section 18.51:

In a number of places, the text was made more explicit about the purpose of rca_one_fs and its connection to file system migration 💜
There's a discussion of situations in which particular forms of RECLAIM_COMPLETE would need to be done.
There's a discussion of interoperability issues between implementations that may have come up because the previous treatment of RECLAIM_COMPLETE wasn't clear enough.

B.3. Revisions Made to Error Definitions in RFC 5661

The new handling of various situations meant some existing error definitions needed revisions too 🎀

Because trunking-related issues needed to be addressed properly, some uses of the term "replica" in RFC 5661 [66] became problematic, since a shift in network access paths was considered to be a shift to a different replica. As a result, the original definition of NFS4ERR_MOVED (in Section 15.1.2.4 of RFC 5661 [66]) was updated to reflect the different handling of the unavailability of a particular fs via a specific network address.

That situation is no longer considered to be unavailability of a file system instance, so the description changed, even though the set of circumstances in which it's returned stays the same. The new paragraph explicitly recognizes that a different network address might be used, while the previous description misleadingly treated this as a shift between two replicas when only a single file system instance might be involved. The updated description appears in Section 15.1.2.4 🔁

Because fs-specific grace periods need to be accommodated, some of the definitions of reclaim-related errors in Section 15 of RFC 5661 [66] needed clarifying, so the text applies properly to reclaims for all types of grace periods. The updated descriptions appear within Section 15.1.9 ✨
Because the clarifications in errata report 2006 [64] needed to be provided, and adapted to properly explain how NFS4ERR_DELAY interacts with the reply cache, a revised description of NFS4ERR_DELAY appears in Section 15.1.1.3. This errata report, unlike many other RFC 5661 errata reports, is addressed in this document because NFS4ERR_DELAY is used so extensively in connection with state migration and session migration 🧚

B.4. Other Revisions Made to RFC 5661 🎀

Okay so, besides the big glow-up of Section 11 of RFC 5661 [66] and all the related tweaks to existing operations and errors, a bunch of other changes were also necessary 💅:

The summary in Section 1.7.3.3 of RFC 5661 [66] got revised to match the Section 11 changes. The updated version lives at Section 1.8.3.3 above ✨
The server scope discussion in Section 2.10.4 of RFC 5661 [66] got replaced 💀. It seemed to need a level of inter-server coordination that clashed with its whole job, which is avoiding the need for a globally uniform way to assign server_owner values 😭. The fresh take is in Section 2.10.4.
The trunking stuff in Section 2.10.5 of RFC 5661 [66] got a glow-up too 🦋. It now explains the multiple types of trunking support more clearly, plus how a client can be made aware of the existing trunking configuration. Also, the last paragraph (not counting subsections) of that section, the one about server_owner changes, was literally true but kept causing confusion 🙈. The original could be read as suggesting those changes be handled nondisruptively, so the revised Section 2.10.5 clears that up.

Appendix C. Security Issues That Need to Be Addressed 🚨✨

Okay, this is the serious bit, bestie 🥺. These are the security issues in the NFSv4.1 specification that still need to be addressed:

The Security Considerations Section of RFC 5661 [66] was not written in accordance with RFC 3552 (BCP 72) [72] 💀. The biggest red flag is that it had no threat analysis.
Initial analysis of the existing security issues with NFSv4.1 made it likely that a revised Security Considerations section for the existing protocol (one containing a threat analysis) would be likely to conclude that NFSv4.1 does not meet the goal of secure use on the Internet 😭.

The Security Considerations section of this document (Section 21) has not been thoroughly revised to fix the difficulties above 😬. Instead, it was modified to properly account for issues related to the multi-server namespace features discussed in Section 11, and the incomplete discussion and security weaknesses were left pretty much as they were 🙃.

These major security issues need a satisfactory fix before an updated Security Considerations section can be published as part of a bis document for NFSv4.1 🛠️:

The continued use of AUTH_SYS and the security exposures it creates still needs to be addressed 👀. Fixing this must not be limited to asking whether the designation of this as OPTIONAL was justified and whether it should be changed.

In any event, it may not be possible at this point to correct the security problems created by continued use of AUTH_SYS simply by revising this designation 🙅‍♀️.

The lack of attention within the protocol to the possibility of pervasive monitoring attacks, such as those described in RFC 7258 [71] (also BCP 188), is a big deal 👁️.

On that note, the use of CREATE_SESSION without privacy protection needs to be addressed, because it exposes the session ID to an attacker's view 😱. This is worrisome since it's exactly the type of protocol artifact alluded to in RFC 7258, and it can enable further mischief. It enables denial-of-service attacks that can be executed effectively with only a single, normally low-value, credential, even when RPCSEC_GSS authentication is in use 💅.

The lack of effective use of privacy and integrity, even where the infrastructure to support RPCSEC_GSS use is present, needs to be addressed 🔒.

Given the security exposures this situation creates, it is not enough to define a protocol that could address this problem with sufficient resources 🙄. What's needed is a way to provide the necessary security with very limited performance costs and without requiring security infrastructure, which experience has shown is difficult for many clients and servers to provide 🥹.

Trying to give a deployed protocol like NFSv4.1 a major security upgrade is a big ask 🧗‍♀️. The working group and the Internet community are likely to run into a bunch of considerations like these:

The need to accommodate existing deployments of protocols specified previously in existing Proposed Standards.
The difficulty of effecting changes to existing, interoperating implementations.
The difficulty of making changes to NFSv4 protocols other than those in the form of OPTIONAL extensions.
The tendency of those responsible for existing NFSv4 deployments to ignore security flaws in the context of local area networks, under the mistaken impression that network isolation provides, in and of itself, isolation from all potential attackers 🚫.

Given that the above-mentioned difficulties apply to minor version zero as well, it may make sense to deal with these security issues in a common document that applies to all NFSv4 minor versions ✨. If that approach is taken, the Security Considerations section of an eventual NFv4.1 bis document would reference that common document, and the defining RFCs for other minor versions might do so as well 🧚.

Acknowledgments 🫶

Acknowledgments for This Update

The authors want to give a shoutout to Andy Adamson of Netapp for clarifying why trunking discovery functionality is needed, and for exploring the role of the file system location attributes in providing the necessary support 🙌.

The authors also thank Tom Haynes of Hammerspace for pointing out that internationalization and security might be best handled in documents dealing with such protocol issues as they apply to all NFSv4 minor versions 🌈.

The authors also acknowledge the work of Xuan Qi of Oracle, who built NFSv4.1 client and server prototypes of Transparent State Migration functionality 👑.

The authors also want to thank others who brought attention to important issues 💜. Trond Myklebust of Primary Data's comments about trunking helped clarify the role of DNS in trunking discovery. Rick Macklem's comments brought attention to problems in the handling of the per-fs version of RECLAIM_COMPLETE 🔥. And thanks to Olga Kornievskaia of Netapp for her helpful review comments 🍓.

Acknowledgments for RFC 5661

The initial text for the SECINFO extensions was edited by Mike Eisler, with contributions from Peng Dai, Sergey Klyushin, and Carl Burnett 🎀.

The initial text for the SESSIONS extensions was edited by Tom Talpey, Spencer Shepler, and Jon Bauman, with contributions from Charles Antonelli, Brent Callaghan, Mike Eisler, John Howard, Chet Juszczak, Trond Myklebust, Dave Noveck, John Scott, Mike Stolarchuk, and Mark Wittle 💖.

The initial text relating to multi-server namespace features, including the concept of referrals, was contributed by Dave Noveck, Carl Burnett, and Charles Fan, with contributions from Ted Anderson, Neil Brown, and Jon Haswell 🦋.

The initial text for the Directory Delegations support was contributed by Saadia Khan, with input from Dave Noveck, Mike Eisler, Carl Burnett, Ted Anderson, and Tom Talpey 🌸.

The initial text for the ACL explanations was contributed by Sam Falkner and Lisa Week 💅.

The pNFS work was inspired by the NASD and OSD work done by Garth Gibson. Gary Grider has also been a champion of high-performance parallel I/O 🚀. Garth Gibson and Peter Corbett started the pNFS effort with a problem statement document for the IETF, which formed the basis for the pNFS work in NFSv4.1 ✨.

The initial text for the parallel NFS support was edited by Brent Welch and Garth Goodson. Additional authors for those documents were Benny Halevy, David Black, and Andy Adamson 🧚. Additional input came from the informal group that helped build the initial pNFS drafts. Specific shoutouts go to Gary Grider, Peter Corbett, Dave Noveck, Peter Honeyman, and Stephen Fridella 🫶.

Fredric Isaman found several errors in draft versions of the ONC RPC XDR description of the NFSv4.1 protocol 🔍.

Audrey Van Belleghem provided, in numerous ways, essential coordination and management of the process of editing the specification documents 👑.

Richard Jernigan gave feedback on the file layout's striping pattern design 🌈.

Several formal inspection teams were formed to review various areas of the protocol. All the inspections found significant errors and room for improvement 😭. NFSv4.1's inspection teams were:

ACLs, with the following inspectors: Sam Falkner, Bruce Fields, Rahul Iyer, Saadia Khan, Dave Noveck, Lisa Week, Mario Wurzl, and Alan Yoder.
Sessions, with the following inspectors: William Brown, Tom Doeppner, Robert Gordon, Benny Halevy, Fredric Isaman, Rick Macklem, Trond Myklebust, Dave Noveck, Karen Rochford, John Scott, and Peter Shah.
Initial pNFS inspection, with the following inspectors: Andy Adamson, David Black, Mike Eisler, Marc Eshel, Sam Falkner, Garth Goodson, Benny Halevy, Rahul Iyer, Trond Myklebust, Spencer Shepler, and Lisa Week.
Global namespace, with the following inspectors: Mike Eisler, Dan Ellard, Craig Everhart, Fredric Isaman, Trond Myklebust, Dave Noveck, Theresa Raj, Spencer Shepler, Renu Tewari, and Robert Thurlow.
NFSv4.1 file layout type, with the following inspectors: Andy Adamson, Marc Eshel, Sam Falkner, Garth Goodson, Rahul Iyer, Trond Myklebust, and Lisa Week.
NFSv4.1 locking and directory delegations, with the following inspectors: Mike Eisler, Pranoop Erasani, Robert Gordon, Saadia Khan, Eric Kustarz, Dave Noveck, Spencer Shepler, and Amy Weaver.
EXCHANGE_ID and DESTROY_CLIENTID, with the following inspectors: Mike Eisler, Pranoop Erasani, Robert Gordon, Benny Halevy, Fredric Isaman, Saadia Khan, Ricardo Labiaga, Rick Macklem, Trond Myklebust, Spencer Shepler, and Brent Welch.
Final pNFS inspection, with the following inspectors: Andy Adamson, Mike Eisler, Mark Eshel, Sam Falkner, Jason Glasgow, Garth Goodson, Robert Gordon, Benny Halevy, Dean Hildebrand, Rahul Iyer, Suchit Kaura, Trond Myklebust, Anatoly Pinchuk, Spencer Shepler, Renu Tewari, Lisa Week, and Brent Welch.

A review team worked together to generate the tables of assignments of error sets to operations, and made sure each assignment had two or more people validating it 📋. Participants included Andy Adamson, Mike Eisler, Sam Falkner, Garth Goodson, Robert Gordon, Trond Myklebust, Dave Noveck, Spencer Shepler, Tom Talpey, Amy Weaver, and Lisa Week 💅.

Jari Arkko, David Black, Scott Bradner, Lisa Dusseault, Lars Eggert, Chris Newman, and Tim Polk provided valuable review and guidance 🌟.

Olga Kornievskaia found several errors in the SSV specification 🕵️‍♀️.

Ricardo Labiaga found several places where the use of RPCSEC_GSS was underspecified 🧐.

Those who provided miscellaneous comments include Andy Adamson, Sunil Bhargo, Alex Burlyga, Pranoop Erasani, Bruce Fields, Vadim Finkelstein, Jason Goldschmidt, Vijay K. Gurbani, Sergey Klyushin, Ricardo Labiaga, James Lentini, Anshul Madan, Daniel Muntz, Daniel Picken, Archana Ramani, Jim Rees, Mahesh Siddheshwar, Tom Talpey, and Peter Varga 💖.

Authors' Addresses ✨

okay the very last bit, and it's literally the cutest part 🦋 these are the humans who wrote the whole thing, so here's where you can find them if you have questions (iykyk 💜)

David Noveck is the editor 👑 and works at NetApp:

David Noveck (editor)
NetApp
1601 Trapelo Road, Suite 16
Waltham, MA 02451
United States of America

Phone: +1-781-768-5347
Email: dnoveck@netapp.com

Charles Lever also helped write it and works at Oracle Corporation 🌸

Charles Lever
Oracle Corporation
1015 Granger Avenue
Ann Arbor, MI 48104
United States of America

Phone: +1-248-614-5091
Email: chuck.lever@oracle.com

so yeah, two real people, two real inboxes, and now you know exactly where to send your protocol questions 💌🧚

📦 where this page comes from

ietf: rfc8881 2020-08
💖 girlified by claude-haiku-5-5 on 2026-10-18: a language model's rewrite of the original, under its terms