RFC 5661

Network File System (NFS) Version 4 Minor Version 1 Protocol 📁✨

RFC 5661 💖 is the spec for NFS version 4 minor version 1, and it's from January 2010. The authors are S. Shepler, Ed. from Storspeed, Inc., M. Eisler, Ed. and D. Noveck, Ed. from NetApp 🎀. It's a Standards Track document (category), ISSN 2070-1721 🌸. It's an IETF thing, so the whole community signed off on it 🫶.

---

Abstract 🌈

okay so this doc is all about NFS version 4 minor version 1 💅. It keeps the stuff from the base protocol (NFS version 4 minor version 0, which is written up in RFC 3530) and adds new features that came later. The big new stuff is Sessions, Directory Delegations, and parallel NFS (pNFS) 🔥. Minor version 1 doesn't depend on minor version 0 at all, so it counts as its own separate protocol 🦋. That means this document neither updates nor obsoletes RFC 3530 💀. Minor version 1 is also considered superior to minor version 0 with no loss of functionality, so the use of version 1 is preferred over version 0 👑. The bonus is that both minor versions 0 and 1 can run at the same time on the same network, between the same client and server ✨. Literally so iconic 😭.

---

Status of This Memo 📝

This is an Internet Standards Track document 🔥.

It's a product of the Internet Engineering Task Force (IETF) 🧚. It represents the consensus of the IETF community, it got public review, and the Internet Engineering Steering Group (IESG) approved it for publication 🫶. If you want more on Internet Standards, check out Section 2 of RFC 5741 🌸.

For the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc5661 💜.

---

Copyright Notice 🎀

Copyright (c) 2010 IETF Trust and the persons identified as the document authors 🌷. All rights reserved ✨.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they're in effect on the date this document was published 📜. Read these carefully, bestie, because they explain your rights and restrictions for this document 🙈. Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions. They're provided without warranty, as described in the Simplified BSD License 💅.

---

Table of Contents 📚

Part 1 of 138 ends here, so more is coming! 🦋

okay so this part is basically the map of the whole spec, and the sections are the chapters 🗺️✨ no cap, the numbering matters bc other sections point back to these 💅

   1. Introduction ....................................................9
      1.1. The NFS Version 4 Minor Version 1 Protocol .................9
      1.2. Requirements Language ......................................9
      1.3. Scope of This Document .....................................9
      1.4. NFSv4 Goals ...............................................10
      1.5. NFSv4.1 Goals .............................................10
      1.6. General Definitions .......................................11
      1.7. Overview of NFSv4.1 Features ..............................13
      1.8. Differences from NFSv4.0 ..................................17
   2. Core Infrastructure ............................................18
      2.1. Introduction ..............................................18
      2.2. RPC and XDR ...............................................19
      2.3. COMPOUND and CB_COMPOUND ..................................22
      2.4. Client Identifiers and Client Owners ......................23
      2.5. Server Owners .............................................28
      2.6. Security Service Negotiation ..............................29
      2.7. Minor Versioning ..........................................34
      2.8. Non-RPC-Based Security Services ...........................37
      2.9. Transport Layers ..........................................37
      2.10. Session ..................................................40
   3. Protocol Constants and Data Types ..............................86
      3.1. Basic Constants ...........................................86
      3.2. Basic Data Types ..........................................87
      3.3. Structured Data Types .....................................89
   4. Filehandles ....................................................97
      4.1. Obtaining the First Filehandle ............................98
      4.2. Filehandle Types ..........................................99
      4.3. One Method of Constructing a Volatile Filehandle .........101
      4.4. Client Recovery from Filehandle Expiration ...............102
   5. File Attributes ...............................................103
      5.1. REQUIRED Attributes ......................................104
      5.2. RECOMMENDED Attributes ...................................104
      5.3. Named Attributes .........................................105
      5.4. Classification of Attributes .............................106
      5.5. Set-Only and Get-Only Attributes .........................107
      5.6. REQUIRED Attributes - List and Definition References .....107
      5.7. RECOMMENDED Attributes - List and Definition References ..108
      5.8. Attribute Definitions ....................................110
      5.9. Interpreting owner and owner_group .......................119
      5.10. Character Case Attributes ...............................121
      5.11. Directory Notification Attributes .......................121
      5.12. pNFS Attribute Definitions ..............................122
      5.13. Retention Attributes ....................................123
   6. Access Control Attributes .....................................126
      6.1. Goals ....................................................126
      6.2. File Attributes Discussion ...............................128
      6.3. Common Methods ...........................................144
      6.4. Requirements .............................................147
   7. Single-Server Namespace .......................................153
      7.1. Server Exports ...........................................153
      7.2. Browsing Exports .........................................153
      7.3. Server Pseudo File System ................................154
      7.4. Multiple Roots ...........................................155
      7.5. Filehandle Volatility ....................................155
      7.6. Exported Root ............................................155
      7.7. Mount Point Crossing .....................................156
      7.8. Security Policy and Namespace Presentation ...............156
   8. State Management ..............................................157
      8.1. Client and Session ID ....................................158
      8.2. Stateid Definition .......................................158
      8.3. Lease Renewal ............................................167
      8.4. Crash Recovery ...........................................170
      8.5. Server Revocation of Locks ...............................181
      8.6. Short and Long Leases ....................................182
      8.7. Clocks, Propagation Delay, and Calculating Lease
           Expiration ...............................................182
      8.8. Obsolete Locking Infrastructure from NFSv4.0 .............183
   9. File Locking and Share Reservations ...........................184
      9.1. Opens and Byte-Range Locks ...............................184
      9.2. Lock Ranges ..............................................188
      9.3. Upgrading and Downgrading Locks ..........................188
      9.4. Stateid Seqid Values and Byte-Range Locks ................189
      9.5. Issues with Multiple Open-Owners .........................189
      9.6. Blocking Locks ...........................................190
      9.7. Share Reservations .......................................191
      9.8. OPEN/CLOSE Operations ....................................192
      9.9. Open Upgrade and Downgrade ...............................192
      9.10. Parallel OPENs ..........................................193
      9.11. Reclaim of Open and Byte-Range Locks ....................194
   10. Client-Side Caching ..........................................194
      10.1. Performance Challenges for Client-Side Caching ..........195
      10.2. Delegation and Callbacks ................................196
      10.3. Data Caching ............................................200
      10.4. Open Delegation .........................................205
      10.5. Data Caching and Revocation .............................216
      10.6. Attribute Caching .......................................218
      10.7. Data and Metadata Caching and Memory Mapped Files .......220
      10.8. Name and Directory Caching without Directory
            Delegations .............................................222
      10.9. Directory Delegations ...................................225
   11. Multi-Server Namespace .......................................228
      11.1. Location Attributes .....................................228
      11.2. File System Presence or Absence .........................229
      11.3. Getting Attributes for an Absent File System ............230
      11.4. Uses of Location Information ............................232
      11.5. Location Entries and Server Identity ....................236
      11.6. Additional Client-Side Considerations ...................237
      11.7. Effecting File System Transitions .......................238
      11.8. Effecting File System Referrals .........................251
      11.9. The Attribute fs_locations ..............................258
      11.10. The Attribute fs_locations_info ........................261
      11.11. The Attribute fs_status ................................273
   12. Parallel NFS (pNFS) ..........................................277
      12.1. Introduction ............................................277
      12.2. pNFS Definitions ........................................278
      12.3. pNFS Operations .........................................284
      12.4. pNFS Attributes .........................................285
      12.5. Layout Semantics ........................................285
      12.6. pNFS Mechanics ..........................................300
      12.7. Recovery ................................................302
      12.8. Metadata and Storage Device Roles .......................307
      12.9. Security Considerations for pNFS ........................307
   13. NFSv4.1 as a Storage Protocol in pNFS: the File Layout Type ..309
      13.1. Client ID and Session Considerations ....................309
      13.2. File Layout Definitions .................................312
      13.3. File Layout Data Types ..................................312
      13.4. Interpreting the File Layout ............................317
      13.5. Data Server Multipathing ................................324
      13.6. Operations Sent to NFSv4.1 Data Servers .................325
      13.7. COMMIT through Metadata Server ..........................327
      13.8. The Layout Iomode .......................................328
      13.9. Metadata and Data Server State Coordination .............329
      13.10. Data Server Component File Size ........................332
      13.11. Layout Revocation and Fencing ..........................333
      13.12. Security Considerations for the File Layout Type .......334
   14. Internationalization .........................................334
     14.1.  Stringprep profile for the utf8str_cs type ..............336
     14.2.  Stringprep profile for the utf8str_cis type .............337
     14.3.  Stringprep profile for the utf8str_mixed type ...........338
     14.4.  UTF-8 Capabilities ......................................340
     14.5.  UTF-8 Related Errors ....................................340
   15. Error Values .................................................341
      15.1. Error Definitions .......................................341
      15.2. Operations and Their Valid Errors .......................361
      15.3. Callback Operations and Their Valid Errors ..............376
      15.4. Errors and the Operations That Use Them .................379
   16. NFSv4.1 Procedures ...........................................391
      16.1. Procedure 0: NULL - No Operation ........................392
      16.2. Procedure 1: COMPOUND - Compound Operations .............392
   17. Operations: REQUIRED, RECOMMENDED, or OPTIONAL ...............403
   18. NFSv4.1 Operations ...........................................407
      18.1. Operation 3: ACCESS - Check Access Rights ...............407
      18.2. Operation 4: CLOSE - Close File .........................413
      18.3. Operation 5: COMMIT - Commit Cached Data ................414
      18.4. Operation 6: CREATE - Create a Non-Regular File Object ..417
      18.5. Operation 7: DELEGPURGE - Purge Delegations
            Awaiting Recovery .......................................419
      18.6. Operation 8: DELEGRETURN - Return Delegation ............420
      18.7. Operation 9: GETATTR - Get Attributes ...................421
      18.8. Operation 10: GETFH - Get Current Filehandle ............423
      18.9. Operation 11: LINK - Create Link to a File ..............424
      18.10. Operation 12: LOCK - Create Lock .......................426
      18.11. Operation 13: LOCKT - Test for Lock ....................430
      18.12. Operation 14: LOCKU - Unlock File ......................432
      18.13. Operation 15: LOOKUP - Lookup Filename .................433
      18.14. Operation 16: LOOKUPP - Lookup Parent Directory ........435
      18.15. Operation 17: NVERIFY - Verify Difference in
             Attributes .............................................436
      18.16. Operation 18: OPEN - Open a Regular File ...............437
      18.17. Operation 19: OPENATTR - Open Named Attribute
             Directory ..............................................458
      18.18. Operation 21: OPEN_DOWNGRADE - Reduce Open File
             Access .................................................459
      18.19. Operation 22: PUTFH - Set Current Filehandle ...........461
      18.20. Operation 23: PUTPUBFH - Set Public Filehandle .........461

---

the section list is locked in ✅🧚 the good stuff starts with section 1 next 💖

1. Introduction ✨

1.1. The NFS Version 4 Minor Version 1 Protocol

okay so the NFS version 4 minor version 1 (NFSv4.1) protocol is the second minor version of the NFS version 4 (NFSv4) protocol 🧚 the first minor version, NFSv4.0, is described in [30]. it generally follows the minor versioning guidelines listed in Section 10 of RFC 3530, but it's not perfectly on track 🙈 it diverges from guideline 11 ("a client and server that support minor version X must support minor versions 0 through X-1") and guideline 12 ("no new features may be introduced as mandatory in a minor version").

why the drama? these divergences happen because of the sessions model for managing non-idempotent operations and the RECLAIM_COMPLETE operation. both of these are infrastructural, meaning they make it way easier to implement existing and new features. but making them anything but REQUIRED would add undue complexity to the protocol definition and implementation, and nobody wants that 😭 so NFSv4.1 updates the minor versioning guidelines (Section 2.7) 💅

as a minor version, NFSv4.1 stays consistent with the overall goals for NFSv4, but it extends the protocol to better meet those goals, based on experiences with NFSv4.0 🌸 it also adopts some extra goals of its own, which drive some of the major extensions in NFSv4.1. main character energy, honestly 👑

1.2. Requirements Language

the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [1]. these are the capital-letter words you'll see doing the heavy lifting 🎀 read them as the official vibe checks they are.

1.3. Scope of This Document

this document describes the NFSv4.1 protocol 🦋 when it comes to NFSv4.0, though, this document does not:

• describe the NFSv4.0 protocol, except where it's needed to contrast with NFSv4.1.
• modify the specification of the NFSv4.0 protocol.
• clarify the NFSv4.0 protocol.

the vibe is "we're here for NFSv4.1 only, not rewriting 4.0 for fun" 🙅‍♀️

1.4. NFSv4 Goals

the NFSv4 protocol is a further revision of the NFS protocol, which was already defined by NFSv3 [31] 🍓 it keeps the essentials from previous versions: easy recovery, independence from transport protocols, operating systems, and file systems, simplicity, and good performance. the NFSv4 goals are:

• Improved access and good performance on the Internet the protocol is designed to transit firewalls easily, perform well where latency is high and bandwidth is low, and scale to very large numbers of clients per server 🌈
• Strong security with negotiation built into the protocol the protocol builds on the work of the ONCRPC working group by supporting the RPCSEC_GSS protocol. on top of that, the NFSv4.1 protocol gives clients and servers a way to negotiate security, and it requires clients and servers to support a minimal set of security schemes 🔐 very responsible, very iconic.
• Good cross-platform interoperability the protocol features a file system model that provides a useful, common set of features. it doesn't unfairly favor one file system or operating system over another ✨
• Designed for protocol extensions the protocol is designed to accept standard extensions within a framework that enables and encourages backward compatibility 🫶

1.5. NFSv4.1 Goals

NFSv4.1 has the following goals, all within the framework set by the overall NFSv4 goals 💜

• To correct significant structural weaknesses and oversights discovered in the base protocol. the glow-up arc 💅
• To add clarity and specificity to areas that the base protocol left unaddressed or didn't cover in enough detail. however, as stated in Section 1.3, it is not a goal to clarify the NFSv4.0 protocol in the NFSv4.1 specification, so don't come for 4.0 here 🙈
• To add specific features based on experience with the existing protocol and recent industry developments. learning from the receipts, literally 🧾
• To provide protocol support for clustered server deployments, including the ability to provide scalable parallel access to files distributed among multiple servers. the big-brain energy is off the charts 🚀

1.6. General Definitions ✨

okay so before we get into the good stuff, we need to be on the same page 🫶 these definitions set the context for everything else, so bestie, read them like the group chat rules 💅

Byte: in this doc, a byte is an octet, i.e., a datum exactly 8 bits in length 🍓

Client: the client is the entity that accesses the NFS server's resources 🦋 it can be an application with the logic to reach the NFS server directly, or the traditional operating system client that provides remote file system services for a bunch of applications ✨

A client is uniquely identified by a client owner 🎀

With reference to byte-range locking, the client is also the entity that keeps a set of locks on behalf of one or more applications 🔒 this client is responsible for crash or failure recovery for those locks it manages 💀

Note that multiple clients may share the same transport and connection, and multiple clients may exist on the same network node 🌈

Client ID: the client ID is a 64-bit quantity used as a unique, short-hand reference to a client-supplied verifier and client owner 🧚 the server is responsible for supplying the client ID 👑

Client Owner: the client owner is a unique string, opaque to the server, that identifies a client 💜 multiple network connections, and the source network addresses coming from those connections, may share a client owner 🌸 the server is expected to treat requests from connections with the same client owner as coming from the same client 🫶

File System: the file system is the collection of objects on a server (as identified by the major identifier of a server owner, which is defined later in this section) that share the same fsid attribute (see Section 5.8.1.9) ✨

Lease: a lease is an interval of time defined by the server for which the client is irrevocably granted locks ⏳ at the end of a lease period, locks may be revoked if the lease has not been extended 😭 a lock must be revoked if a conflicting lock has been granted after the lease interval 💀

A server grants a client a single lease for all state 👑

Lock: the term "lock" refers to byte-range locks (in UNIX environments, also known as record locks), share reservations, delegations, or layouts, unless specifically stated otherwise 🔐

Secret State Verifier (SSV): the SSV is a unique secret key shared between a client and server 🗝️ the SSV serves as the secret key for an internal (that is, internal to NFSv4.1) Generic Security Services (GSS) mechanism (the SSV GSS mechanism; see Section 2.10.9) the SSV GSS mechanism uses the SSV to compute message integrity code (MIC) and Wrap tokens 🔥 see Section 2.10.8.3 for more details on how NFSv4.1 uses the SSV and the SSV GSS mechanism 📚

Server: the Server is the entity responsible for coordinating client access to a set of file systems, and it is identified by a server owner 🏰 a server can span multiple network addresses 🌐

Server Owner: the server owner identifies the server to the client 🪪 the server owner consists of a major identifier and a minor identifier 🧩 when the client has two connections, each to a peer with the same major identifier, the client assumes that both peers are the same server (the server namespace is the same via each connection) and that lock state is sharable across both connections 🤝 when each peer has both the same major and minor identifiers, the client assumes that each connection might be associable with the same session 🎀

Stable Storage: stable storage is storage from which data stored by an NFSv4.1 server can be recovered without data loss from multiple power failures (including cascading power failures, that is, several power failures in quick succession), operating system failures, and/or hardware failure of components other than the storage medium itself (such as disk, nonvolatile RAM, flash memory, etc.) 🛡️

Some examples of stable storage that are allowable for an NFS server include:

1. Media commit of data; that is, the modified data has been successfully written to the disk media, for example, the disk platter 💿
2. An immediate reply disk drive with battery-backed, on-drive intermediate storage or uninterruptible power system (UPS) 🔋
3. Server commit of data with battery-backed intermediate storage and recovery software 🛠️
4. Cache commit with uninterruptible power system (UPS) and recovery software 🧯

Stateid: a stateid is a 128-bit quantity returned by a server that uniquely defines the open and locking states provided by the server for a specific open-owner or lock-owner/open-owner pair for a specific file and type of lock 🪄

Verifier: a verifier is a 64-bit quantity generated by the client that the server can use to determine if the client has restarted and lost all previous lock state 🔄 main character restart arc, honestly 😭

---

1.7. Overview of NFSv4.1 Features 🌟

the major features of the NFSv4.1 protocol will be reviewed in brief 📝 this is to give context to both the reader who already knows the previous versions of the NFS protocol and the reader who is brand new to NFS 🥹 for the newbies, there's still a set of fundamental knowledge that's expected 🧠 the reader should be familiar with the External Data Representation (XDR) and Remote Procedure Call (RPC) protocols as described in [2] and [3] ✅ a basic knowledge of file systems and distributed file systems is expected as well 💅

In general, this specification of NFSv4.1 will not distinguish those features added in minor version 1 from those present in the base protocol, and will treat NFSv4.1 as a unified whole 🧩 see Section 1.8 for a summary of the differences between NFSv4.0 and NFSv4.1 🔍

1.7.1. RPC and Security 🔐

as with previous versions of NFS, the External Data Representation (XDR) and Remote Procedure Call (RPC) mechanisms used for the NFSv4.1 protocol are those defined in [2] and [3] 📜 to meet end-to-end security requirements, the RPCSEC_GSS framework [4] is used to extend the basic RPC security 🛡️ with RPCSEC_GSS, various mechanisms can be provided to offer authentication, integrity, and privacy to the NFSv4 protocol 🫧 Kerberos V5 is used as described in [5] to provide one security framework 🎫 with RPCSEC_GSS, other mechanisms may also be specified and used for NFSv4.1 security 🌈

to enable in-band security negotiation, the NFSv4.1 protocol has operations that give the client a way to ask the server about its policies on which security mechanisms must be used for access to the server's file system resources 🗣️ with this, the client can securely match the security mechanism that meets the policies specified at both the client and server 🤝

NFSv4.1 introduces parallel access (see Section 1.7.2.2), which is called pNFS 🚀 the security framework described in this section is significantly modified by the introduction of pNFS (see Section 12.9), because data access is sometimes not over RPC 😳 the level of significance varies with the storage protocol (see Section 12.2.5) and can be as low as zero impact (see Section 13.12) ✨

1.7.2. Protocol Structure 🧱

1.7.2.1. Core Protocol 🎯

Unlike NFSv3, which used a series of ancillary protocols (e.g., NLM, NSM (Network Status Monitor), MOUNT), within all minor versions of NFSv4 a single RPC protocol is used to make requests to the server 🙌 facilities that used to be separate protocols, such as locking, are now integrated into one unified protocol 💖

1.7.2.2. Parallel Access ⚡

Minor version 1 supports high-performance data access to a clustered server implementation by separating metadata access from data access, with the latter done to multiple servers in parallel 🏎️

Such parallel data access is controlled by recallable objects known as "layouts", which are integrated into the protocol locking model 🔁 clients direct requests for data access to a set of data servers specified by the layout, via a data storage protocol which may be NFSv4.1 or may be another protocol 📦

Because the protocols used for parallel data access are not necessarily RPC-based, the RPC-based security model (Section 1.7.1) is obviously impacted (see Section 12.9) 😬 the degree of impact varies with the storage protocol (see Section 12.2.5) used for data access, and can be as low as zero (see Section 13.12) 🫠

1.7.3. File System Model 🗂️

the general file system model used for the NFSv4.1 protocol is the same as in previous versions 🥰 the server file system is hierarchical, with the regular files contained within being treated as opaque byte streams 🫧 in a slight departure, file and directory names are encoded with UTF-8 to handle the basics of internationalization 🌍

The NFSv4.1 protocol does not require a separate protocol to provide the initial mapping between path name and filehandle 🗺️ all file systems exported by a server are presented as a tree, so that every file system is reachable from a special per-server global root filehandle 🌳 this allows LOOKUP operations to do the jobs the MOUNT protocol used to do 🪄 the server provides any necessary pseudo file systems to bridge any gaps that come from unexported space between exported file systems 🌉

1.7.3.1. Filehandles 🎟️

as in previous versions of the NFS protocol, opaque filehandles are used to identify individual files and directories 🔖 lookup-type and create operations translate file and directory names to filehandles, which are then used to identify objects in subsequent operations 🔁

The NFSv4.1 protocol provides support for persistent filehandles, which are guaranteed to be valid for the lifetime of the file system object they designate 💎 it also gives servers a way to provide filehandles with more limited validity guarantees, called volatile filehandles 🫧 bestie, those volatile ones are giving very "here today, gone tomorrow" energy 😭

1.7.3.2. File Attributes

okay so attributes are the little facts about every file object 🎀 NFSv4.1 has a rich, extensible attribute structure, split into REQUIRED, RECOMMENDED, and named attributes (see Section 5) ✨

Several (but not all) of the REQUIRED attributes come from NFSv3, so if you're curious, check the fattr3 data type in [31] 🧐 A good example of a REQUIRED attribute is the file object's type (Section 5.8.1.2). That's how you tell regular files apart from directories (aka folders in some operating environments) and other kinds of objects 📁 The full REQUIRED breakdown is in Section 5.1 💖

Three RECOMMENDED attributes are acl, sacl, and dacl 🦋 Together they define an Access Control List (ACL) on a file object (Section 6). ACLs give directory and file access control way beyond the NFSv3 model, with specific sets of permissions for individual users and groups 💅 ACL inheritance is the cherry on top: it propagates access permissions and restrictions down a directory tree as new file system objects get created 🌳 RECOMMENDED attributes are covered in Section 5.2.

A named attribute is an opaque byte stream tied to a directory or file and referred to by a string name 🧁 Client apps use them to stash their own application-specific data on a regular file or directory. NFSv4.1 changed named attributes compared to NFSv4.0 by tightening the allowed operations, so nobody ends up building non-interoperable implementations 🙅‍♀️ Named attributes are discussed in Section 5.3.

1.7.3.3. Multi-Server Namespace

NFSv4.1 has a bunch of features for building namespaces that cross server boundaries 🌍 They also make it easier to move support for individual file systems between servers without disrupting anyone 🦋 It's all based on attributes that let one file system point to alternate or new locations for itself.

These attributes can be paired with absent file systems, which give you the specs for extra locations but no actual file system content 👻 That unlocks a few important facilities:

• Location attributes can be used with absent file systems to implement referrals, where one server directs the client to a file system provided by another server. This is how extensive multi-server namespaces get built 🧚
• Location attributes can be provided for present file systems to point to alternate file system instances or replicas, for use if the current instance becomes unavailable 🛟
• Location attributes can be provided when a previously present file system becomes absent. This allows non-disruptive migration of file systems to alternate servers 🚚

1.7.4. Locking Facilities

as mentioned earlier, NFSv4.1 is a single protocol that includes locking facilities 🔐 And we're not talking one basic lock type, we're talking lots of them, including a number of sorts of recallable locks 👑

Recallable locks, like delegations, let the client be assured that certain events won't happen for as long as that lock is held 🫶 When circumstances change, the lock gets recalled through a callback request 📞 Those assurances mean clients can do more extensive caching safely when the situation allows it 🌸

The types of locks are:

• Share reservations, as established by OPEN operations.
• Byte-range locks.
• File delegations, which are recallable locks that assure the holder that inconsistent opens and file changes cannot occur so long as the delegation is held.
• Directory delegations, which are recallable locks that assure the holder that inconsistent directory modifications cannot occur so long as the delegation is held.
• Layouts, which are recallable objects that assure the holder that direct access to the file data may be performed directly by the client, and that no change to the data's location that is inconsistent with that access may be made so long as the layout is held.

All locks for a given client are tied together under a single client-wide lease 🎀 Every request made on sessions associated with the client renews that lease. If the client's lease isn't promptly renewed, the client's locks are subject to revocation 💀 And if the server restarts, clients get the chance to safely reclaim their locks during a special grace period 🙏

1.8. Differences from NFSv4.0

here's the tea ☕ The following summarizes the major differences between minor version 1 and the base protocol:

• Implementation of the sessions model (Section 2.10).
• Parallel access to data (Section 12).
• Addition of the RECLAIM_COMPLETE operation to better structure the lock reclamation process (Section 18.51).
• Enhanced delegation support, as follows:
  • Delegations on directories and other file types in addition to regular files (Section 18.39, Section 18.49).
  • Operations to optimize acquisition of recalled or denied delegations (Section 18.49, Section 20.5, Section 20.7).
  • Notifications of changes to files and directories (Section 18.39, Section 20.4).
  • A method to allow a server to indicate that it is recalling one or more delegations for resource management reasons, and thus a method to allow the client to pick which delegations to return (Section 20.6).
• Attributes can be set atomically during exclusive file create via the OPEN operation (see the new EXCLUSIVE4_1 creation method in Section 18.16) ⚡
• Open files can be preserved if removed and the hard link count ("hard link" is defined in an Open Group [6] standard) goes to zero. That means clients don't need to rename deleted files to partially hidden names, colloquially called "silly rename" 🙃 (see the new OPEN4_RESULT_PRESERVE_UNLINKED reply flag in Section 18.16).
• Improved compatibility with Microsoft Windows for Access Control Lists (Section 6.2.3, Section 6.2.2, Section 6.4.3.2) 🪟
• Data retention (Section 5.13) 🗄️
• Identification of the implementation of the NFS client and server (Section 18.35) 🪪
• Support for notification of the availability of byte-range locks (see the new OPEN4_RESULT_MAY_NOTIFY_LOCK reply flag in Section 18.16 and see Section 20.11) 🔔
• In NFSv4.1, LIPKEY and SPKM-3 are not required security mechanisms [32] 🙅

2. Core Infrastructure

2.1. Introduction

NFSv4.1 relies on core infrastructure that's common to nearly every operation 🧱 That's what the rest of this section is about 💕

2.2. RPC and XDR

The NFSv4.1 protocol is a Remote Procedure Call (RPC) application that uses RPC version 2 and the corresponding eXternal Data Representation (XDR) as defined in [3] and [2] 🤓

2.2.1. RPC-Based Security

okay this part is a plot twist 😭 Previous NFS versions were thought of as having a host-based authentication model, where the NFS server authenticates the NFS client and trusts the client to authenticate all users. But actually? NFS has always depended on RPC for authentication 🫢 One of the first forms of RPC authentication, AUTH_SYS, had no strong authentication and needed a host-based approach. NFSv4.1 also depends on RPC for basic security services, and it mandates RPC support for a user-based authentication model 👑 In that model, user principals are authenticated by a server, and the server is in turn authenticated by user principals. RPC provides some basic security services that NFSv4.1 uses 💖

2.2.1.1. RPC Security Flavors

As described in Section 7.2 ("Authentication") of [3], RPC security is encapsulated in the RPC header, via a security or authentication flavor, plus information specific to that flavor 🎀 Every RPC header conveys information used to identify and authenticate a client and server. As discussed in Section 2.2.1.1.1, some security flavors provide additional security services ✨

NFSv4.1 clients and servers MUST implement RPCSEC_GSS. (This requirement to implement is not a requirement to use.) Other flavors, such as AUTH_NONE and AUTH_SYS, MAY be implemented as well 🦋

2.2.1.1.1. RPCSEC_GSS and Security Services

RPCSEC_GSS [4] uses the functionality of GSS-API [7] 🔑 This lets the RPC layer use various security mechanisms without the extra implementation overhead of adding RPC security flavors for each one. Galaxy brain move 🧠

2.2.1.1.1.1. Identification, Authentication, Integrity, Privacy

Via the GSS-API, RPCSEC_GSS can be used to identify and authenticate users on clients to servers, and servers to users 🪪 It can also do integrity checking on the entire RPC message (including the RPC header) and on the arguments or results. And finally, privacy (usually via encryption) is a service available with RPCSEC_GSS 🔐 Privacy is performed on the arguments and results.

Here's how the services stack up 🥞 If privacy is selected, integrity, authentication, and identification are enabled. If privacy is not selected but integrity is, authentication and identification are enabled. If integrity and privacy are not selected but authentication is enabled, identification is enabled. RPCSEC_GSS does not provide identification as a separate service 🙈

Although GSS-API has an authentication service distinct from its privacy and integrity services, GSS-API's authentication service is not used for RPCSEC_GSS's authentication service. Instead, each RPC request and response header is integrity protected with the GSS-API integrity service, and this allows RPCSEC_GSS to offer per-RPC authentication and identity 🧩 See [4] for more information.

NFSv4.1 clients and servers MUST support RPCSEC_GSS's integrity and authentication service. NFSv4.1 servers MUST support RPCSEC_GSS's privacy service. NFSv4.1 clients SHOULD support RPCSEC_GSS's privacy service. 🫶

2.2.1.1.1.2. Security Mechanisms for NFSv4.1

RPCSEC_GSS, via GSS-API, normalizes access to mechanisms that provide security services 🌈 Therefore, NFSv4.1 clients and servers MUST support the Kerberos V5 security mechanism. Kerberos gets the spotlight 🔥

The use of RPCSEC_GSS requires selection of a mechanism, quality of protection (QOP), and service (authentication, integrity, privacy). For the mandated security mechanisms, NFSv4.1 specifies that a QOP of zero is used, leaving it up to the mechanism or the mechanism's configuration to map QOP zero to an appropriate level of protection 🛡️ Each mandated mechanism specifies a minimum set of cryptographic algorithms for implementing integrity and privacy. NFSv4.1 clients and servers MUST be implemented on operating environments that comply with the REQUIRED cryptographic algorithms of each REQUIRED mechanism 🔒 Iconic security energy, no cap 💅

2.2.1.1.1.2.1. Kerberos V5 ✨

okay so the Kerberos V5 GSS-API mechanism from [5] MUST be implemented with the RPCSEC_GSS services in this table 🎀

      column descriptions:
      1 == number of pseudo flavor
      2 == name of pseudo flavor
      3 == mechanism's OID
      4 == RPCSEC_GSS service
      5 == NFSv4.1 clients MUST support
      6 == NFSv4.1 servers MUST support

      1      2        3                    4                     5   6
      ------------------------------------------------------------------
      390003 krb5     1.2.840.113554.1.2.2 rpc_gss_svc_none      yes yes
      390004 krb5i    1.2.840.113554.1.2.2 rpc_gss_svc_integrity yes yes
      390005 krb5p    1.2.840.113554.1.2.2 rpc_gss_svc_privacy    no yes

the pseudo flavor number and name are only here as a mapping aid for the implementor 🗺️ NFSv4.1 already has a way to negotiate security and understands the GSS-API mechanism, so the pseudo flavor isn't needed there. it's only needed for NFSv3, since NFSv3 does its security negotiation through the MOUNT protocol, as described in [33] 🧚

back when NFSv4.1 was specified, AES with HMAC-SHA1 was a REQUIRED algorithm set for Kerberos V5 💅 NFSv4.0 was different. weaker algorithm sets were REQUIRED for Kerberos V5, and they were REQUIRED in the NFSv4.0 spec, because the Kerberos V5 spec at that time didn't specify stronger algorithms 🥲 the NFSv4.1 spec doesn't name any REQUIRED algorithms for Kerberos V5. instead, implementors are expected to keep up with the Kerberos V5 standard if and when stronger algorithms get specified 🌈

2.2.1.1.1.2.1.1. Security Considerations for Cryptographic Algorithms in Kerberos V5 🔐

bestie, this one's serious. 💀

when you deploy NFSv4.1, how strong your security is depends on your existing Kerberos V5 setup 🏰 the client and server can't see or pick the Kerberos V5 algorithms directly, so whoever uses NFSv4.1 needs to do some due diligence to make sure security is acceptable wherever it matters 🫶

2.2.1.1.1.3. GSS Server Principal 🎀

no matter which security mechanism under RPCSEC_GSS you're using, the NFS server MUST identify itself in GSS-API with a GSS_C_NT_HOSTBASED_SERVICE name type 💖 those names look like this:

     service@hostname

for NFS, the "service" part is nfs. implementations turn nfs@hostname into different forms depending on the security mechanism, so it's giving "many outfits, same person" 👗 for Kerberos V5, the RECOMMENDED form is:

     nfs/hostname

---

2.3. COMPOUND and CB_COMPOUND 🍓

okay this is the big one, bestie 😭 one of the biggest changes from NFS versions before NFSv4 is the COMPOUND procedure. in every NFSv4 minor version there are exactly two RPC procedures: NULL and COMPOUND. COMPOUND is a series of individual operations, and those operations do the same jobs the old traditional NFS procedures used to do 🔥

the server evaluates the operations inside a COMPOUND request in order, but there are no atomicity guarantees, so don't expect it to be all-or-nothing 🙈 there's a small set of facilities for passing results from one operation to the next. once an operation returns a failing result, evaluation stops, and the results of every operation evaluated so far get sent back to the client 💀

with COMPOUND, the client can build simple or complicated requests, and that means fewer RPCs for logical file system operations ✨ for example, a multi-component lookup can be built by stacking multiple LOOKUP operations. you can add things like GETATTR, READDIR, or OPEN plus READ on top, so you can do more complex stuff without adding latency 🚀 main character energy

NFSv4.1 also has a whole set of callback operations, where the server makes an RPC aimed at the client 📞 callback RPCs look a lot like normal server requests. in every NFSv4 minor version there are two callback RPC procedures: CB_NULL and CB_COMPOUND. CB_COMPOUND works the same way as COMPOUND, with its own set of callback operations 🦋

adding new server and callback operations inside the COMPOUND and CB_COMPOUND request framework gives the protocol a way to grow in later minor versions 🌸

except for a small number of operations needed for session creation, server requests and callback requests both happen inside the context of a session 🫧 sessions give every request a client context and support robust reply protection for non-idempotent requests, so retries don't cause chaos 🛡️ iconic, honestly.

2.4. Client Identifiers and Client Owners 🔐

Okay bestie, so every operation that grabs or relies on locking state needs the server to know exactly which client is asking. No mix-ups allowed 💅

Each distinct client instance gets a client ID. It's a 64-bit identifier for one specific client at one specific time ✨ The client ID changes whenever the client re-initializes, and it may change when the server re-initializes. Client IDs are what make lock identification and crash recovery possible 🦋

During steady state operation, the client ID for each operation comes from the session (see Section 2.10) that the operation is sent on. A session gets associated with a client ID when it's created, so the session carries the client ID with it 🎀

Unlike NFSv4.0, the only NFSv4.1 operations you can do before a client ID is established are the ones needed to establish the client ID. Main character rules 👑

Here's the setup: an EXCHANGE_ID operation followed by a CREATE_SESSION operation that uses that client ID (the eir_clientid returned from EXCHANGE_ID) is required to establish and confirm the client ID on the server. 🫶 When a new incarnation of the client identifies itself, it also immediately releases any locking state that an earlier incarnation of that same client might have had on the server. The released state includes all byte-range lock, share reservation, and layout state. It also includes all delegation state tied to the same client with the same identity, but only where the server supports neither the CLAIM_DELEGATE_PREV nor the CLAIM_DELEG_CUR_FH claim types. For the delegation state recovery story, check Section 10.2.1. For layout state recovery, check Section 12.7.1 📚

Releasing that state requires the server to be able to tell that one client instance is the successor of another. If it can't figure that out, for any of a number of reasons, the locking state hangs around for a while, subject to lease expiration (see Section 8.3). The new client then has to wait for that state to be removed if it makes conflicting lock requests. Ngl, that's a lot of waiting 😭

Client identification is wrapped up in this client owner data type:

struct client_owner4 {
        verifier4       co_verifier;
        opaque          co_ownerid<NFS4_OPAQUE_LIMIT>;
};

The first field, co_verifier, is a client incarnation verifier. If co_verifier is different from what the server has previously recorded for the identified client (matched up by the co_ownerid field), the server starts the process of canceling the client's leased state 🚨

The second field, co_ownerid, is a variable length string that uniquely defines the client. Later incarnations of the same client bear the same co_ownerid but with a different verifier 🌸

Now, the client needs to be thoughtful about how it generates this co_ownerid string. Here's the tea:

• The string should be unique, so that multiple clients don't present the same string. If two clients do, the fallout can range from one client getting an error all the way to one client having its leased state abruptly and unexpectedly cancelled. Yikes 💀
• The string should be picked so that later incarnations (e.g., restarts) of the same client present the same string. Implementors are cautioned against an approach that requires the string to be recorded in a local file. That approach rules out environments with no local disk, where all file access goes through an NFSv4.1 server 🙈
• The string should be the same for each server network address that the client accesses. That way, if a server has multiple interfaces, the client can trunk traffic over multiple network paths, as described in Section 2.10.5. (Note: the precise opposite was advised in the NFSv4.0 specification [30].) 🍓
• The algorithm shouldn't assume that the client's network address won't change, unless the client implementation knows it's using statically assigned network addresses. This includes changes between client incarnations and even changes while the client is still running in its current incarnation. So with dynamic address assignment, if the client puts just its network address in the co_ownerid string, there's a real risk that after the client gives up that address, another client using a similar algorithm would generate a conflicting co_ownerid string. Chaos 🌀

Given all that, a well-generated co_ownerid string is one that includes:

• If applicable, the client's statically assigned network address.
• Additional information that tends to be unique, such as one or more of:
  • The client machine's serial number. (For privacy reasons, it's best to run some one-way function on the serial number.)
  • A Media Access Control (MAC) address. (Again, a one-way function should be performed.)
  • The timestamp of when the NFSv4.1 software was first installed on the client. (This is subject to the earlier caution about using information stored in a file, because the file might only be accessible over NFSv4.1.)
  • A true random number. However, since this number ought to be the same between client incarnations, it shares the same problem as using the timestamp of the software installation. 🎲
• For a user-level NFSv4.1 client, additional information that distinguishes it from other user-level clients running on the same host, such as a process identifier or other unique sequence. 🧚

The client ID is assigned by the server (it's the eir_clientid result from EXCHANGE_ID), and it should be chosen so that it won't conflict with a client ID the server previously assigned. This applies across server restarts too 🔥

When the server restarts, a client may find out that its current client ID is no longer valid, because it receives an NFS4ERR_STALE_CLIENTID error. The exact circumstances depend on the characteristics of the sessions involved, specifically whether the session is persistent (see Section 2.10.6.5). In every case, though, the client gets this error when it tries to establish a new session with the existing client ID. The error tells it that a new client ID has to be obtained via EXCHANGE_ID, and the new session needs to be established with that client ID 🥲

When a session is not persistent, the client finds out it needs to create a new session because it gets an NFS4ERR_BADSESSION. The session was lost as part of the server restart. When the existing client ID is presented as part of creating a session and isn't recognized (which is what happens after a server restart), the server rejects the request with NFS4ERR_STALE_CLIENTID 💔

If the session is persistent, the client re-establishes communication using the existing session after the restart. That session is still associated with the existing client ID, but it may only be used to retransmit operations the client previously sent and didn't see replies to. Replies to operations the server already performed come from the reply cache. Otherwise, the server returns NFS4ERR_DEADSESSION. That's why such a session is called "dead" 💀 To do new operations, the client needs a new session. If it tries to establish that new session with the existing client ID, the server rejects the request with NFS4ERR_STALE_CLIENTID.

So when NFS4ERR_STALE_CLIENTID shows up in either of these situations, the client MUST obtain a new client ID by using the EXCHANGE_ID operation. Then it uses that client ID as the basis of a new session, and proceeds to any other necessary recovery for the server restart case (see Section 8.4.2). 🔄

See the descriptions of EXCHANGE_ID (Section 18.35) and CREATE_SESSION (Section 18.36) for the complete specification of these operations 📖

2.4.1. Upgrade from NFSv4.0 to NFSv4.1 🆙

To make upgrading from NFSv4.0 to NFSv4.1 smoother, a server may compare a value of data type client_owner4 in an EXCHANGE_ID with a value of data type nfs_client_id4 that was established using the SETCLIENTID operation of NFSv4.0. A server that does this lets an upgraded client skip waiting until the lease (the one established by the NFSv4.0 instance of the client) expires. ✨

This requires that the value of data type client_owner4 be constructed the same way as the value of data type nfs_client_id4. If the latter's contents included the server's network address (per the recommendations of the NFSv4.0 specification [30]), and the NFSv4.1 client doesn't want to use a client ID that prevents trunking, it should send two EXCHANGE_ID operations. The first EXCHANGE_ID has a client_owner4 equal to the nfs_client_id4. This clears the state created by the NFSv4.0 client. The second EXCHANGE_ID doesn't include the server's network address. The state created for the second EXCHANGE_ID won't have to wait for lease expiration, because there's no state left to expire 🧹

2.4.2. Server Release of Client ID 👋

NFSv4.1 introduces a new operation called DESTROY_CLIENTID (Section 18.50). The client SHOULD use it to destroy a client ID it no longer needs 🫧 This allows a graceful, bilateral release of a client ID. The operation can't be used if there are sessions associated with the client ID, or state with an unexpired lease.

If the server determines that the client holds no associated state for its client ID, the server MAY choose to unilaterally release the client ID to conserve resources. (Associated state includes unrevoked sessions, opens, locks, delegations, layouts, and wants.) If the client contacts the server after this release, the server MUST ensure that the client receives the appropriate error, so that it uses the EXCHANGE_ID/CREATE_SESSION sequence to establish a new client ID. 🫣

The server ought to be very hesitant to release a client ID, since the work the client has to do to recover from such an event is the same burden as if the server had failed and restarted. Typically, a server wouldn't release a client ID unless there had been no activity from that client for many minutes. As long as there are sessions, opens, locks, delegations, layouts, or wants, the server MUST NOT release the client ID. 🚫 See Section 2.10.13.1.4 for discussion on releasing inactive sessions.

2.4.3. Resolving Client Owner Conflicts 💅

Okay so this is where the server gets to be the referee 👀 When the server gets an EXCHANGE_ID for a client owner that has no state right now, or has state but the lease already expired, the server MUST allow the EXCHANGE_ID and confirm the new client ID, as long as it's followed by the appropriate CREATE_SESSION ✨

Now the drama: a new incarnation of a client owner shows up while the old one still has state and an unexpired lease 😭 In that case, the server is allowed to dispose of the old incarnation's state if one of the following is true:

• The principal that created the client ID for the client owner is the same as the principal sending the EXCHANGE_ID operation. Note that if the client ID was created with SP4_MACH_CRED state protection (Section 18.35), the principal MUST be based on RPCSEC_GSS authentication, the RPCSEC_GSS service used MUST be integrity or privacy, and the same GSS mechanism and principal MUST be used as that used when the client ID was created.
• The client ID was established with SP4_SSV protection (Section 18.35, Section 2.10.8.3) and the client sends the EXCHANGE_ID with the security flavor set to RPCSEC_GSS using the GSS SSV mechanism (Section 2.10.9).
• The client ID was established with SP4_SSV protection, and under the conditions described herein, the EXCHANGE_ID was sent with SP4_MACH_CRED state protection. Because the SSV might not persist across client and server restart, and because the first time a client sends EXCHANGE_ID to a server it does not have an SSV, the client MAY send the subsequent EXCHANGE_ID without an SSV RPCSEC_GSS handle. Instead, as with SP4_MACH_CRED protection, the principal MUST be based on RPCSEC_GSS authentication, the RPCSEC_GSS service used MUST be integrity or privacy, and the same GSS mechanism and principal MUST be used as that used when the client ID was created.

If none of the above situations apply, the server MUST return NFS4ERR_CLID_INUSE 🚫

Last bit: if the server accepts the principal and co_ownerid as matching the ones that created the client ID, but the co_verifier in the EXCHANGE_ID differs from the co_verifier used when the client ID was created, then once the server receives a CREATE_SESSION that confirms the client ID, the server deletes state. If the co_verifier values are the same (e.g., the client either is updating properties of the client ID (Section 18.35) or is attempting trunking (Section 2.10.5), the server MUST NOT delete state. Protecting the vibes ✨🫶

---

2.5. Server Owners 🏠

The server owner is basically a cousin of the client owner (Section 2.4), except there's no shorthand server ID 🙈 The server owner is defined in this data type:

struct server_owner4 {
    uint64_t       so_minor_id;
    opaque         so_major_id<NFS4_OPAQUE_LIMIT>;
};

The server owner gets returned from EXCHANGE_ID. When the so_major_id fields are the same in two EXCHANGE_ID results, the connections each EXCHANGE_ID was sent over can be assumed to address the same server (as defined in Section 1.6). If the so_minor_id fields are also the same, then not only do both connections reach the same server, but the session can be shared across both connections 🦋

Warning, bestie! ⚠️ Multiple servers may deliberately or accidentally claim to have the same so_major_id, or the same so_major_id/so_minor_id pair. So the reader should check out Sections 2.10.5 and 18.35 to avoid acting on falsely matching server owner values 👀

The rules for generating a so_major_id are similar to the ones for generating a co_ownerid string (see Section 2.4). Two servers generating conflicting so_major_id values is less dire than co_ownerid conflicts, because the client can use RPCSEC_GSS to compare the authenticity of each server (see Section 2.10.5) 🔐

---

2.6. Security Service Negotiation 🎀

With the NFSv4.1 server potentially offering multiple security mechanisms, the client needs a way to figure out (or negotiate) which mechanism to use for communication with the server 🤝 The NFS server may have multiple points within its file system namespace that NFS clients can use. These points count as security policy boundaries, and in some NFS implementations they're tied to NFS export points. The server may also be configured so that each of these boundaries has different or multiple security mechanisms in use 🌈

The security negotiation between client and server SHOULD be done over a secure channel, so a third party can't intercept the negotiation and force the client and server into a lower level of security than they need or want 🛡️ See Section 21 for further discussion.

2.6.1. NFSv4.1 Security Tuples 🍓

An NFS server can assign one or more "security tuples" to each security policy boundary in its namespace. Each security tuple is made of a security flavor (see Section 2.2.1.1) and, if the flavor is RPCSEC_GSS, a GSS-API mechanism Object Identifier (OID), a GSS-API quality of protection, and an RPCSEC_GSS service 💜

2.6.2. SECINFO and SECINFO_NO_NAME 🔍

The SECINFO and SECINFO_NO_NAME operations let the client figure out, per filehandle, which security tuple to use for server access. Usually the client won't need either operation except during initial communication with the server, or when it crosses security policy boundaries at the server 🧚

The server's policies can also change at any time and force the client to negotiate a new security tuple 😬

Here's the ordering tea: if using different security tuples would change the type of access allowed over the same connection used for the SECINFO or SECINFO_NO_NAME operation (e.g., read-only vs. read-write), then security tuples that allow greater access should be presented first. And when the general level of access is the same but different security flavors limit which principals' privileges are recognized (e.g., allowing or disallowing root access), flavors supporting the greatest range of principals should be listed first 👑

2.6.3. Security Error 🙅‍♀️

Based on the assumption that each NFSv4.1 client and server MUST support a minimum set of security (i.e., Kerberos V5 under RPCSEC_GSS), the NFS client starts file access to the server with one of the minimal security tuples. While they're talking, the client may get an NFS error of NFS4ERR_WRONGSEC. This error is the server saying "hey, the security tuple you're using goes against my security policy" 🚨 The client then has to figure out (see Section 2.6.3.1) which security tuples the server has available, and pick one that fits the client ✨

2.6.3.1. Using NFS4ERR_WRONGSEC, SECINFO, and SECINFO_NO_NAME 🧩

This section explains the mechanics of NFSv4.1 security negotiation. Let's dig in! 🤓

2.6.3.1.1. Put Filehandle Operations 📦

The term "put filehandle operation" refers to PUTROOTFH, PUTPUBFH, PUTFH, and RESTOREFH. Each subsection below describes how the server handles a subseries of operations that starts with a put filehandle operation 🎀

2.6.3.1.1.1. Put Filehandle Operation + SAVEFH 💾

The client is saving a filehandle for a future RESTOREFH, LINK, or RENAME. SAVEFH MUST NOT return NFS4ERR_WRONGSEC. To figure out whether the put filehandle operation returns NFS4ERR_WRONGSEC, the server implementation pretends SAVEFH isn't in the series of operations, then checks which of the situations in the other subsections of Section 2.6.3.1.1 apply 🙈

2.6.3.1.1.2. Two or More Put Filehandle Operations 👯

For a series of N put filehandle operations, the server MUST NOT return NFS4ERR_WRONGSEC to the first N-1 put filehandle operations. The Nth put filehandle operation is handled as if it were the first in a subseries of operations. For example, if the server gets a COMPOUND request with this series of operations -- PUTFH, PUTROOTFH, LOOKUP -- then the PUTFH operation is ignored for NFS4ERR_WRONGSEC purposes, and the PUTROOTFH, LOOKUP subseries is processed according to Section 2.6.3.1.1.3 🔁

2.6.3.1.1.3. Put Filehandle Operation + LOOKUP (or OPEN of an Existing Name) 🌸

This situation also applies to a put filehandle operation followed by a LOOKUP or an OPEN operation that specifies an existing component name 🏃‍♀️

In this situation, the client is potentially crossing a security policy boundary, and the set of security tuples the parent directory supports may differ from those of the child. The server implementation decides whether to impose any restrictions on security policy administration. There are at least three approaches (sec_policy_child is the tuple set of the child export, sec_policy_parent is that of the parent):

• (a) sec_policy_child <= sec_policy_parent (<= for subset). This means that the set of security tuples specified on the security policy of a child directory is always a subset of its parent directory.
• (b) sec_policy_child ^ sec_policy_parent != {} (^ for intersection, {} for the empty set). This means that the set of security tuples specified on the security policy of a child directory always has a non-empty intersection with that of the parent.
• (c) sec_policy_child ^ sec_policy_parent == {}. This means that the set of security tuples specified on the security policy of a child directory may not intersect with that of the parent. In other words, there are no restrictions on how the system administrator may set up these tuples.

To support approaches (b) (for the case when a client chooses a flavor that is not a member of sec_policy_parent) and (c), the put filehandle operation cannot return NFS4ERR_WRONGSEC when there's a security tuple mismatch. Instead, it should be returned from the LOOKUP (or OPEN by existing component name) that follows 🧐

Since the above guideline doesn't contradict approach (a), it should be followed in general. Even if approach (a) is implemented, it's possible for the security tuple used to be acceptable for the target of LOOKUP but not for the filehandles used in the put filehandle operation. The put filehandle operation could be a PUTROOTFH or PUTPUBFH, where the client can't know the security tuples for the root or public filehandle. Or the security policy for the filehandle used by the put filehandle operation could have changed since the filehandle was obtained 🔄

Therefore, an NFSv4.1 server MUST NOT return NFS4ERR_WRONGSEC in response to the put filehandle operation if the operation is immediately followed by a LOOKUP or an OPEN by component name. Security tuples, mismatches, and lookups: all sorted! 🌈✨

---

2.6.3.1.1.4. Put Filehandle Operation + LOOKUPP ✨

Okay so SECINFO only works its way down 🫠, which means LOOKUPP can't return NFS4ERR_WRONGSEC without SECINFO_NO_NAME. SECINFO_NO_NAME fixes this with style SECINFO_STYLE4_PARENT, which goes the opposite way from SECINFO. Just like in Section 2.6.3.1.1.3, a put filehandle operation that's followed by a LOOKUPP MUST NOT return NFS4ERR_WRONGSEC. If the server doesn't support SECINFO_NO_NAME, the client's only option is to send the put filehandle operation, LOOKUPP, GETFH sequence with every security tuple it supports. It's a lot, bestie 😭

Regardless of whether SECINFO_NO_NAME is supported, an NFSv4.1 server MUST NOT return NFS4ERR_WRONGSEC in response to a put filehandle operation if that operation is immediately followed by a LOOKUPP. 🫶

2.6.3.1.1.5. Put Filehandle Operation + SECINFO/SECINFO_NO_NAME 💅

A security-sensitive client is allowed to pick a strong security tuple when it asks a server which security tuples a file object permits. The tuple the client picks doesn't have to be in the security policy's tuple list for either the parent directory named in the put filehandle operation or the child file object named in SECINFO (or any parent directory named in SECINFO_NO_NAME). Of course, the server has to be configured for whatever tuple the client chooses, or the request fails at the RPC layer with an appropriate authentication error. 🔐

In theory, there's no link between the security flavor used by SECINFO or SECINFO_NO_NAME and the ones in the security policy. In practice though, the client may start by looking for strong flavors from the ones the security policy supports, and then move on to the ones in the REQUIRED set. Very strategic 🧠✨

The NFSv4.1 server MUST NOT return NFS4ERR_WRONGSEC to a put filehandle operation that's immediately followed by SECINFO or SECINFO_NO_NAME. The NFSv4.1 server MUST NOT return NFS4ERR_WRONGSEC from SECINFO or SECINFO_NO_NAME. 🙈

2.6.3.1.1.6. Put Filehandle Operation + Nothing 🍓

The NFSv4.1 server MUST NOT return NFS4ERR_WRONGSEC. Simple and iconic 👑

2.6.3.1.1.7. Put Filehandle Operation + Anything Else 💖

"Anything Else" includes OPEN by filehandle. Main character energy 🎀

The security policy enforcement applies to the filehandle named in the put filehandle operation. So the put filehandle operation MUST return NFS4ERR_WRONGSEC when there's a security tuple mismatch. This avoids the mess of adding NFS4ERR_WRONGSEC as an allowable error to every other operation. Honestly so smart 🧚

A COMPOUND with the put filehandle operation followed by SECINFO_NO_NAME (style SECINFO_STYLE4_CURRENT_FH) is an efficient way for the client to recover from NFS4ERR_WRONGSEC. ✨

The NFSv4.1 server MUST NOT return NFS4ERR_WRONGSEC to any operation other than a put filehandle operation, LOOKUP, LOOKUPP, and OPEN (by component name). 🌸

2.6.3.1.1.8. Operations after SECINFO and SECINFO_NO_NAME 🤯

Okay picture this 👀 A client sends a COMPOUND containing the series SEQUENCE, PUTFH, SECINFO_NONAME, READ, and the security tuple it used doesn't match what the target file needs. By the rule in Section 2.6.3.1.1.5, neither PUTFH nor SECINFO_NO_NAME can return NFS4ERR_WRONGSEC. By the rule in Section 2.6.3.1.1.7, READ can't return NFS4ERR_WRONGSEC either. It all works out because SECINFO and SECINFO_NO_NAME consume the current filehandle (and yes, this is a change from NFSv4.0 🙌). That leaves no current filehandle for READ to use, so READ returns NFS4ERR_NOFILEHANDLE. Plot twist, but it makes sense 💀

2.6.3.1.2. LINK and RENAME 🦋

The LINK and RENAME operations use both the current and saved filehandles. Technically, the server MAY return NFS4ERR_WRONGSEC from LINK or RENAME if the security policy of the saved filehandle rejects the security flavor used in the COMPOUND request's credentials. If the server does that, and there's no overlap between the security policies of the saved and current filehandles, then the client can never complete the LINK or RENAME it wanted. Sad but true 😭

For example, say the client sends this COMPOUND request, where bFH and aFH are filehandles for different directories:

SEQUENCE, PUTFH bFH, SAVEFH, PUTFH aFH, RENAME "c" "d"

Suppose no common security tuple exists between the security policies of aFH and bFH. If the client sends the request with credentials that bFH's security policy accepts but aFH's doesn't, the PUTFH aFH operation fails with NFS4ERR_WRONGSEC. Then, after a SECINFO_NO_NAME request, the client sends SEQUENCE, PUTFH bFH, SAVEFH, PUTFH aFH, RENAME "c" "d" using credentials that aFH's policy accepts but bFH's doesn't. The server returns NFS4ERR_WRONGSEC on the RENAME operation. Ugh, the loop 🌀

To stop a client from getting stuck in an endless cycle of a request with LINK or RENAME, followed by a request with SECINFO_NO_NAME or SECINFO, the server MUST detect when the security policies of the current and saved filehandles have no mutually acceptable security tuple. In that situation, the server MUST NOT return NFS4ERR_WRONGSEC from LINK or RENAME. Instead, the server MUST do one of two things:

• The server can return NFS4ERR_XDEV.
• The server can let the security policy of the current filehandle override that of the saved filehandle, and so return NFS4_OK.

---

2.7. Minor Versioning 🌈

To handle the need for an NFS protocol that can evolve as things change, the NFSv4.1 protocol includes the rules and framework for future minor changes or versioning. Growth arc unlocked 📈

The base assumption for minor versioning is that any future accepted minor version will be documented in one or more Standards Track RFCs. Minor version 0 of the NFSv4 protocol is represented by [30], and minor version 1 is represented by this RFC. The COMPOUND and CB_COMPOUND procedures support encoding the minor version that the client requests. 🧩

These are the basic rules for developing minor versions. Just note that a future minor version may modify or add to them as part of its own definition. 🫠

1. Procedures are not added or deleted.

To keep the general RPC model, NFSv4 minor versions won't add or delete procedures from the NFS program. 🔒

2. Minor versions may add operations to the COMPOUND and CB_COMPOUND procedures.

Adding operations to the COMPOUND and CB_COMPOUND procedures doesn't affect the RPC model. ✨

• Minor versions may append attributes to the bitmap4 that represents sets of attributes and to the fattr4 that represents sets of attribute values.

This lets the attribute model grow and adapt over time. 🌱

• Minor version X must append any new attributes after the last documented attribute.

Attribute results are an opaque array of per-attribute, XDR-encoded results, so adding new attributes in the middle of the current definitions would be way too burdensome 😩

3. Minor versions must not modify the structure of an existing operation's arguments or results.

Again, handling multiple structure definitions for one operation is too burdensome. So new operations should be added instead of changing existing structures for a minor version. 🙅‍♀️

This rule doesn't stop these adaptations in a minor version:

• adding bits to flag fields, such as new attributes to GETATTR's bitmap4 data type, and providing corresponding variants of opaque arrays, such as a notify4 used together with such bitmaps
• adding bits to existing attributes like ACLs that have flag words
• extending enumerated types (including NFS4ERR_*) with new values
• adding cases to a switched union

4. Minor versions must not modify the structure of existing attributes.
5. Minor versions must not delete operations.

This stops a particular operation "slot" from being reused in a future minor version. 🎰

6. Minor versions must not delete attributes.
7. Minor versions must not delete flag bits or enumeration values.
8. Minor versions may declare an operation MUST NOT be implemented.

Declaring that an operation MUST NOT be implemented is the same as obsoleting it. For the client, it means the operation MUST NOT be sent to the server. For the server, it can return an NFS error instead of "dropping" the request as an XDR decode error. This lets an operation go obsolete while keeping its structure, so a future minor version can bring it back. Iykyk 🔄

1. Minor versions may declare that an attribute MUST NOT be implemented.
2. Minor versions may declare that a flag bit or enumeration value MUST NOT be implemented.

9. Minor versions may downgrade features from REQUIRED to RECOMMENDED, or RECOMMENDED to OPTIONAL. 📉
10. Minor versions may upgrade features from OPTIONAL to RECOMMENDED, or RECOMMENDED to REQUIRED. 📈
11. A client and server that support minor version X SHOULD support minor versions zero through X-1 as well. 🤝
12. Except for infrastructural changes, a minor version must not introduce REQUIRED new features.

This lets new functionality in, but forces real implementation experience before anything becomes REQUIRED. Infrastructural features are different though, since they have broad effects. Making them RECOMMENDED or OPTIONAL would complicate implementing the minor version. 🧱

13. A client MUST NOT attempt to use a stateid, filehandle, or similar returned object from the COMPOUND procedure with minor version X for another COMPOUND procedure with minor version Y, where X != Y. 🚫

---

2.8. Non-RPC-Based Security Services 🔐

As described in Section 2.2.1.1.1.1, NFSv4.1 relies on RPC for identification, authentication, integrity, and privacy. NFSv4.1 itself provides or enables additional security services, which are covered in the next few subsections. Layers on layers 🧁

2.8.1. Authorization 👑

Authorization to access a file object through an NFSv4.1 operation is ultimately decided by the NFSv4.1 server. A client can check its access to a file object ahead of time with the OPEN (Section 18.16) and ACCESS (Section 18.1) operations.

Principals with the right access rights can change the authorization on a file object with the SETATTR (Section 18.30) operation. Attributes that affect access rights include mode, owner, owner_group, acl, dacl, and sacl. See Section 5. 🗝️

2.8.2. Auditing 📝

NFSv4.1 provides auditing on a per-file object basis, through the acl and sacl attributes described in Section 6. Specifying audit log formats or management policies is outside the scope of this specification. 🕵️‍♀️

2.8.3. Intrusion Detection 🚨

NFSv4.1 provides alarm control on a per-file object basis, through the acl and sacl attributes described in Section 6. Alarms may serve as the basis for intrusion detection. Specifying heuristics for detecting intrusion via alarms is outside the scope of this specification. 👀

---

2.9. Transport Layers 🚀

2.9.1. REQUIRED and RECOMMENDED Properties of Transports ✨

Okay bestie, so NFSv4.1 works over both RDMA and non-RDMA transports 🦋. It needs a few things from whatever transport it runs on:

• The transport supports reliable delivery of data, which NFSv4.1 requires but neither NFSv4.1 nor RPC has facilities for ensuring [34].
• The transport delivers data in the order it was sent. Ordered delivery simplifies detection of transmit errors, and simplifies the sending of arbitrary sized requests and responses via the record marking protocol [3].

If an NFSv4.1 implementation supports operation over the IP network protocol, any transport used between NFS and IP MUST be among the IETF-approved congestion control transport protocols. When this document was written, the only two transports with the above attributes were TCP and the Stream Control Transmission Protocol (SCTP) 💅. To help with interoperability, an NFSv4.1 implementation MUST support operation over the TCP transport protocol 🫶.

Even if NFSv4.1 is used over a non-IP network protocol, it is RECOMMENDED that the transport support congestion control 🌸.

A connectionless transport is allowed under NFSv4.1 😌, but reliable and in-order delivery of data combined with congestion control by the connectionless transport is REQUIRED. So, ngl, UDP by itself MUST NOT be used as an NFSv4.1 transport 🙈. NFSv4.1 assumes that a client transport address and a server transport address used to send data over a transport together constitute a connection, even if the underlying transport doesn't believe in the concept of a connection 💜.

2.9.2. Client and Server Transport Behavior

If a connection-oriented transport (e.g., TCP) is used, the client and server SHOULD use long-lived connections for at least three reasons 🧚:

1. This will prevent the weakening of the transport's congestion control mechanisms via short-lived connections.
2. This will improve performance for the WAN environment by eliminating the need for connection setup handshakes.
3. The NFSv4.1 callback model differs from NFSv4.0, and requires the client and server to maintain a client-created backchannel (see Section 2.10.3.1) for the server to use.

To reduce congestion, if a connection-oriented transport is used, and the request is not the NULL procedure:

• A requester MUST NOT retry a request unless the connection the request was sent over was lost before the reply was received.
• A replier MUST NOT silently drop a request, even if the request is a retry. (The silent drop behavior of RPCSEC_GSS [4] does not apply because this behavior happens at the RPCSEC_GSS layer, a lower layer in the request processing.) Instead, the replier SHOULD return an appropriate error (see Section 2.10.6.1), or it MAY disconnect the connection 🔥.

When sending a reply, the replier MUST send the reply to the same full network address (e.g., if using an IP-based transport, the source port of the requester is part of the full network address) from which the requester sent the request. If using a connection-oriented transport, replies MUST be sent on the same connection from which the request was received 👑.

If a connection is dropped after the replier receives the request but before the replier sends the reply, the replier might have a pending reply. If a connection is established with the same source and destination full network address as the dropped connection, then the replier MUST NOT send the reply until the requester retries the request. The reason for this prohibition is that the requester MAY retry a request over a different connection (provided that connection is associated with the original request's session) 🎀.

When using RDMA transports, there are other reasons for not tolerating retries over the same connection:

• RDMA transports use "credits" to enforce flow control, where a credit is a right to a peer to transmit a message. If one peer were to retransmit a request (or reply), it would consume an additional credit. If the replier retransmitted a reply, it would certainly result in an RDMA connection loss, since the requester would typically only post a single receive buffer for each request. If the requester retransmitted a request, the additional credit consumed on the server might lead to RDMA connection failure unless the client accounted for it and decreased its available credit, leading to wasted resources 😭.
• RDMA credits present a new issue to the reply cache in NFSv4.1. The reply cache may be used when a connection within a session is lost, such as after the client reconnects. Credit information is a dynamic property of the RDMA connection, and stale values must not be replayed from the cache. This implies that the reply cache contents must not be blindly used when replies are sent from it, and credit information appropriate to the channel must be refreshed by the RPC layer 🌈.

In addition, as described in Section 2.10.6.2, while a session is active, the NFSv4.1 requester MUST NOT stop waiting for a reply ⭐.

2.9.3. Ports

Historically, NFSv3 servers have listened over TCP port 2049. The registered port 2049 [35] for the NFS protocol should be the default configuration 🍓. NFSv4.1 clients SHOULD NOT use the RPC binding protocols as described in [36] 🦋.

2.10. Session ✨

NFSv4.1 clients and servers MUST support and MUST use the session feature as described in this section 💖.

2.10.1. Motivation and Overview

Previous versions and minor versions of NFS have suffered from the following:

• Lack of support for Exactly Once Semantics (EOS). This includes lack of support for EOS through server failure and recovery.
• Limited callback support, including no support for sending callbacks through firewalls, and races between replies to normal requests and callbacks.
• Limited trunking over multiple network paths.
• Requiring machine credentials for fully secure operation.

Through the introduction of a session, NFSv4.1 addresses the above shortfalls with practical solutions 💅:

• EOS is enabled by a reply cache with a bounded size, making it feasible to keep the cache in persistent storage and enable EOS through server failure and recovery. One reason that previous revisions of NFS did not support EOS was because some EOS approaches often limited parallelism. As will be explained in Section 2.10.6, NFSv4.1 supports both EOS and unlimited parallelism 🔥.
• The NFSv4.1 client (defined in Section 1.6, Paragraph 2) creates transport connections and provides them to the server to use for sending callback requests, thus solving the firewall issue (Section 18.34). Races between responses from client requests and callbacks caused by the requests are detected via the session's sequencing properties that are a consequence of EOS (Section 2.10.6.3) 🧚.
• The NFSv4.1 client can associate an arbitrary number of connections with the session, and thus provide trunking (Section 2.10.5) 🌸.
• The NFSv4.1 client and server produces a session key independent of client and server machine credentials which can be used to compute a digest for protecting critical session management operations (Section 2.10.8.3) 🔐.
• The NFSv4.1 client can also create secure RPCSEC_GSS contexts for use by the session's backchannel that do not require the server to authenticate to a client machine principal (Section 2.10.8.2) 🫶.

A session is a dynamically created, long-lived server object created by a client and used over time from one or more transport connections. Its function is to maintain the server's state relative to the connection(s) belonging to a client instance. This state is entirely independent of the connection itself, and indeed the state exists whether or not the connection exists. A client may have one or more sessions associated with it so that client-associated state may be accessed using any of the sessions associated with that client's client ID, when connections are associated with those sessions. When no connections are associated with any of a client ID's sessions for an extended time, such objects as locks, opens, delegations, layouts, etc. are subject to expiration. The session serves as an object representing a means of access by a client to the associated client state on the server, independent of the physical means of access to that state 🦋.

A single client may create multiple sessions. A single session MUST NOT serve multiple clients 🙅‍♀️.

2.10.2. NFSv4 Integration

Sessions are part of NFSv4.1 and not NFSv4.0. Normally, a major infrastructure change such as sessions would require a new major version number to an Open Network Computing (ONC) RPC program like NFS. However, because NFSv4 encapsulates its functionality in a single procedure, COMPOUND, and because COMPOUND can support an arbitrary number of operations, sessions have been added to NFSv4.1 with little difficulty 🧠. COMPOUND includes a minor version number field, and for NFSv4.1 this minor version is set to 1. When the NFSv4 server processes a COMPOUND with the minor version set to 1, it expects a different set of operations than it does for NFSv4.0 ✨.

NFSv4.1 defines the SEQUENCE operation, which is required for every COMPOUND that operates over an established session, with the exception of some session administration operations, such as DESTROY_SESSION (Section 18.37) 💖.

2.10.2.1. SEQUENCE and CB_SEQUENCE

In NFSv4.1, when the SEQUENCE operation is present, it MUST be the first operation in the COMPOUND procedure. The primary purpose of SEQUENCE is to carry the session identifier. The session identifier associates all other operations in the COMPOUND procedure with a particular session. SEQUENCE also contains required information for maintaining EOS (see Section 2.10.6). Session-enabled NFSv4.1 COMPOUND requests thus have the form:

    +-----+--------------+-----------+------------+-----------+----
    | tag | minorversion | numops    |SEQUENCE op | op + args | ...
    |     |   (== 1)     | (limited) |  + args    |           |
    +-----+--------------+-----------+------------+-----------+----

and the replies have the form:

    +------------+-----+--------+-------------------------------+--//
    |last status | tag | numres |status + SEQUENCE op + results |  //
    +------------+-----+--------+-------------------------------+--//
            //-----------------------+----
            // status + op + results | ...
            //-----------------------+----

A CB_COMPOUND procedure request and reply has a similar form to COMPOUND, but instead of a SEQUENCE operation, there is a CB_SEQUENCE operation. CB_COMPOUND also has an additional field called "callback_ident", which is superfluous in NFSv4.1 and MUST be ignored by the client. CB_SEQUENCE has the same information as SEQUENCE, and also includes other information needed to resolve callback races (Section 2.10.6.3) 🎉.

2.10.2.2. Client ID and Session Association 💖

Okay so every client ID (Section 2.4) can have zero or more active sessions 🧚. A client ID and its session are required to do file access in NFSv4.1, no cap 💅. And every time a session gets used (whether the client is sending a request to the server, or replying to a callback request from the server), the state leased to that client ID gets automatically renewed ✨.

State is tied to the client ID, not to any one session 🎀. That state can include share reservations, locks, delegations, and layouts (Section 1.7.4), and it lives on the client ID, so it doesn't belong to one specific session. Successive state changing operations from a given state owner MAY go over different sessions, as long as the session is associated with the same client ID 🦋. A callback MAY arrive over a different session than the one that originally acquired the state the callback is about 👑. For example, if session A is used to acquire a delegation, a request to recall that delegation MAY arrive over session B, but only if both sessions are associated with the same client ID 🫶. Sections 2.10.8.1 and 2.10.8.2 cover the security considerations around callbacks, so go read those too 🔥.

2.10.3. Channels 🌈

A channel is not a connection, bestie 💜. A channel is the direction ONC RPC requests get sent in 🌸.

Each session has one or two channels: the fore channel and the backchannel 🍓. There are at most two channels per session, and each one has its own job, so channels don't get identifiers 🧚.

The fore channel is for the ordinary requests going from the client to the server, and it carries COMPOUND requests and responses 💅. A session always has a fore channel ⭐.

The backchannel is for callback requests going from the server to the client, and it carries CB_COMPOUND requests and responses 🦋. Whether or not there is a backchannel is up to the client 🙈. Still, lots of NFSv4.1 features need a backchannel, so it's kind of a must-have. NFSv4.1 servers MUST support backchannels 🔥.

Each session has resources for each channel, including separate reply caches (see Section 2.10.6.1) 💖. Even the backchannel needs a reply cache (or at least a slot table, so retries can be detected), because some callback operations are nonidempotent 😭.

2.10.3.1. Association of Connections, Channels, and Sessions 🫶

Each channel is associated with zero or more transport connections, and those can be the same transport protocol or different ones 🦋. A connection can be associated with one channel or with both channels of a session 💅. The client and server negotiate whether a connection carries traffic for one channel or both, using the CREATE_SESSION (Section 18.36) and BIND_CONN_TO_SESSION (Section 18.34) operations 👑. When a session is created via CREATE_SESSION, the connection that carried the CREATE_SESSION request is automatically associated with the fore channel, and optionally the backchannel too ✨. If the client specifies no state protection (Section 18.35) when the session is created, then when SEQUENCE is sent on a different connection, that connection gets automatically associated with the fore channel of the session named in the SEQUENCE operation 🎀.

A connection's association with a session is not exclusive 🌈. A connection associated with the channel(s) of one session can be associated with the channel(s) of other sessions at the same time, including sessions tied to other client IDs 🧚.

It's totally permissible for connections of multiple transport types to be associated with the same channel 🌸. For example, both TCP and RDMA connections can be associated with the fore channel 💜. If an RDMA connection and a non-RDMA connection are associated with the same channel, the maximum number of slots SHOULD be at least one more than the total number of RDMA credits (Section 2.10.6.1) 🍓. That way, if every RDMA credit is used up, the non-RDMA connection still gets at least one outstanding request 😭. If a server supports multiple transport types, it MUST allow a client to associate connections from each transport to a channel 🔥.

It's also permissible for a connection of one transport type to be associated with the fore channel, while a connection of a different type is associated with the backchannel 🦋.

2.10.4. Server Scope 🌸

Each server specifies a server scope value, which is an opaque string called eir_server_scope, returned as part of the results of an EXCHANGE_ID operation 💅. The point of the server scope is to let a group of servers tell clients that a set of servers sharing the same server scope value has agreed to use compatible values for identifiers that would otherwise be opaque 💖. So an identifier generated by one server in that set can be presented to another server in the same scope 👑.

Using compatible values doesn't mean a value generated by one server will always be accepted by another 🙈. In most cases, it won't be. But a server won't accept a value generated somewhere else by accident 😭. When it does accept one, it's because that value is recognized as valid and means the same thing as it does on another server in the same scope ✨.

When servers share the same server scope, this compatibility of values applies to the following identifiers follow 🧚:

• Filehandle values. A filehandle value accepted by two servers of the same server scope denotes the same object. A WRITE operation sent to one server is reflected immediately in a READ sent to the other, and locks obtained on one server conflict with those requested on the other 🔥.
• Session ID values. A session ID value accepted by two servers of the same server scope denotes the same session 🎀.
• Client ID values. A client ID value accepted as valid by two servers of the same server scope is associated with two clients with the same client owner and verifier 🦋.
• State ID values. A state ID value is recognized as valid when the corresponding client ID is recognized as valid. If the same stateid value is accepted as valid on two servers of the same scope, and the client IDs on those two servers represent the same client owner and verifier, then the two stateid values designate the same set of locks and are for the same file 🌈.
• Server owner values. When the server scope values are the same, server owner values may be validly compared. When the server scope values are different, server owner values are treated as different, even if they contain all identical bytes 💜.

Getting servers to this level of compatibility can be pretty minimal 🍓. It can be limited to a simple partition of the ID space 💅. Recognizing common values takes some extra implementation work, but it can be tailored to the specific situations where that recognition is wanted ✨.

Clients will have reasons to compare the server scope values of multiple servers in a few situations, and each one gets covered in its own functional section 👑:

• When server owner values received in response to EXCHANGE_ID operations sent to multiple network addresses get compared, to decide whether various forms of trunking are valid (as described in Section 2.10.5) 🫶.
• When network or server reconfiguration causes the same network address to possibly point to different servers, and the client has to decide when lock reclaim should be attempted (as described in Section 8.4.2.1) 😭.
• When file system migration moves responsibility for a file system between servers, and the client needs to figure out whether state was transferred along with the file system (as described in Section 11.7.7), or whether it needs to reclaim state the same way it would after a server restart (as described in Section 8.4.2) 🧚.

When two EXCHANGE_ID replies, each from a different server network address, have the same server scope, there are a few ways a client can check that the shared scope is because two servers are working together as a group 🌸.

• If both EXCHANGE_ID requests were sent with RPCSEC_GSS authentication, and the server principal is the same for both targets, then the equality of server scope is validated. It is RECOMMENDED that two servers intending to share the same server scope also share the same principal name 🔥.
• The client may accept the appearance of the second server in the fs_locations or fs_locations_info attribute for a relevant file system 💖. For example, if there's a migration event for a particular file system, or there are locks to be reclaimed on it, the attributes for that file system may be used 🦋. The client sends the GETATTR request to the first server for the fs_locations or fs_locations_info attribute, using RPCSEC_GSS authentication 🎀. It may need to do this before it actually needs to verify the common server scope 🙈. If the client successfully authenticates the reply to GETATTR, and the GETATTR request and reply containing the fs_locations or fs_locations_info attribute refer to the second server, then the equality of server scope is supported 👑. A client may choose to limit the use of this form of support to information relevant to the specific file system involved (e.g. a file system being migrated) 💅.

2.10.5. Trunking 🦋

Okay bestie, trunking is when you use multiple connections between a client and server so the data zooms faster ✨ NFSv4.1 has two flavors: session trunking and client ID trunking 💖

NFSv4.1 servers MUST support both forms of trunking within the context of a single server network address, and they MUST support both forms within the context of the set of network addresses used to reach a single server 💅 Clustered NFSv4.1 servers MAY let network addresses for different servers use client ID trunking too 🌈

Clients can use either form, periodt, as long as they don't break the servers' rules about which kinds of trunking are allowed when they trunk between different server network addresses (more on that in a sec) 🫶 About callback channels: the client MUST allow the server to choose among all callback channels valid for a given client ID, and the client MUST support trunking when the connections backing the backchannel allow session or client ID trunking to be used for callbacks 📞

Session trunking is basically hooking up multiple connections, each with potentially different target and/or source network addresses, to the same session 🧚 When the target network addresses (server addresses) of the two connections are the same, the server MUST support that session trunking 💅 When the target network addresses are different, the server MAY indicate that support using the data returned by the EXCHANGE_ID operation (see below) 👀

Client ID trunking is linking multiple sessions to the same client ID 🌸 Servers MUST support client ID trunking for two target network addresses whenever they allow session trunking for those same two network addresses 💖 Also, a server MAY, by presenting the same major server owner ID (Section 2.5) and server scope (Section 2.10.4), allow an extra case of client ID trunking ✨ When two servers return the same major server owner and server scope, it means they're cooperating on locking state management, and that's a prerequisite for client ID trunking 🔥

Figuring out when the client is allowed to use session or client ID trunking means understanding how the results of the EXCHANGE_ID operation (Section 18.35) identify a server 🕵️‍♀️ Say a client sends EXCHANGE_IDs over two different connections, each with a possibly different target network address, but each EXCHANGE_ID operation has the same value in the eia_clientowner field. If the same NFSv4.1 server is listening over each connection, then each EXCHANGE_ID result MUST return the same values of eir_clientid, eir_server_owner.so_major_id, and eir_server_scope. The client can then treat each connection as referring to the same server (subject to verification; see Section 2.10.5.1 later in this section), and it can use each connection to trunk requests and replies. The client's choice is whether session trunking or client ID trunking applies 👑

Session Trunking. If the eia_clientowner argument is the same in two different EXCHANGE_ID requests, and the eir_clientid, eir_server_owner.so_major_id, eir_server_owner.so_minor_id, and eir_server_scope results match in both EXCHANGE_ID results, then the client is permitted to perform session trunking. If the client has no session mapping to the tuple of eir_clientid, eir_server_owner.so_major_id, eir_server_scope, and eir_server_owner.so_minor_id, then it creates the session via a CREATE_SESSION operation over one of the connections, which associates the connection to the session. If there is already a session for the tuple, the client can send BIND_CONN_TO_SESSION to associate the connection to that session 🦋

Of course, if the client doesn't want to use session trunking, it isn't required to 💅 It can invoke CREATE_SESSION on the connection, which will result in client ID trunking (described below). It can also decide to drop the connection if it doesn't choose to use trunking 🙈

Client ID Trunking. If the eia_clientowner argument is the same in two different EXCHANGE_ID requests, and the eir_clientid, eir_server_owner.so_major_id, and eir_server_scope results match in both EXCHANGE_ID results, then the client is permitted to perform client ID trunking (regardless of whether the eir_server_owner.so_minor_id results match). The client can associate each connection with different sessions, where each session is associated with the same server 🧚

The client completes client ID trunking by invoking CREATE_SESSION on each connection, using the same client ID that was returned in eir_clientid. Those invocations create two sessions and also associate each connection with its respective session. The client is free to decline client ID trunking by simply dropping the connection at this point 🌈

When doing client ID trunking, locking state is shared across sessions associated with that same client ID, so the server has to coordinate state across those sessions 🫶

The client should be prepared for the possibility that eir_server_owner values differ on later EXCHANGE_ID requests to the same network address, because of various reconfiguration events. When that happens and the changes invalidate previously valid forms of trunking, the client should stop using those forms, either by dropping connections or by adding sessions 🥺 For a discussion of lock reclaim as it relates to such reconfiguration events, see Section 8.4.2.1.

---

2.10.5.1. Verifying Claims of Matching Server Identity 🔍

When two servers over two connections claim matching or partially matching eir_server_owner, eir_server_scope, and eir_clientid values, the client does not have to trust those claims. The client may verify them before trunking traffic in the following ways:

• For session trunking, clients SHOULD reliably verify whether connections between different network paths are in fact associated with the same NFSv4.1 server and usable on the same session, and servers MUST allow clients to perform reliable verification. When a client ID is created, the client SHOULD specify that BIND_CONN_TO_SESSION is to be verified according to the SP4_SSV or SP4_MACH_CRED (Section 18.35) state protection options. For SP4_SSV, reliable verification depends on a shared secret (the SSV) that is established via the SET_SSV (Section 18.47) operation.

When a new connection is associated with the session (via the BIND_CONN_TO_SESSION operation, see Section 18.34), if the client specified SP4_SSV state protection for the BIND_CONN_TO_SESSION operation, the client MUST send the BIND_CONN_TO_SESSION with RPCSEC_GSS protection, using integrity or privacy, and an RPCSEC_GSS handle created with the GSS SSV mechanism (Section 2.10.9) 🔐

If the client mistakenly tries to associate a connection to a session of a wrong server, the server will either reject the attempt because it is not aware of the session identifier of the BIND_CONN_TO_SESSION arguments, or it will reject the attempt because the RPCSEC_GSS authentication fails. Even if the server mistakenly or maliciously accepts the connection association attempt, the RPCSEC_GSS verifier it computes in the response will not be verified by the client, so the client will know it cannot use the connection for trunking the specified session 🚨

If the client specified SP4_MACH_CRED state protection, the BIND_CONN_TO_SESSION operation will use RPCSEC_GSS integrity or privacy, using the same credential that was used when the client ID was created. Mutual authentication via RPCSEC_GSS assures the client that the connection is associated with the correct session of the correct server 🎀

• For client ID trunking, the client has at least two options for verifying that the same client ID obtained from two different EXCHANGE_ID operations came from the same server. The first option is to use RPCSEC_GSS authentication when sending each EXCHANGE_ID operation. Each time an EXCHANGE_ID is sent with RPCSEC_GSS authentication, the client notes the principal name of the GSS target. If the EXCHANGE_ID results indicate that client ID trunking is possible, and the GSS targets' principal names are the same, the servers are the same and client ID trunking is allowed 🧁

The second option for verification is to use SP4_SSV protection. When the client sends EXCHANGE_ID, it specifies SP4_SSV protection. The first EXCHANGE_ID the client sends always has to be confirmed by a CREATE_SESSION call. The client then sends SET_SSV. Later, the client sends EXCHANGE_ID to a second destination network address, different from the one the first EXCHANGE_ID was sent to. The client checks that each EXCHANGE_ID reply has the same eir_clientid, eir_server_owner.so_major_id, and eir_server_scope. If so, the client verifies the claim by sending a CREATE_SESSION operation to the second destination address, protected with RPCSEC_GSS integrity using an RPCSEC_GSS handle returned by the second EXCHANGE_ID. If the server accepts the CREATE_SESSION request, and if the client verifies the RPCSEC_GSS verifier and integrity codes, then the client has proof the second server knows the SSV, and thus the two servers are cooperating for the purposes of specifying server scope and client ID trunking 🌈✨ that's the receipts, bestie! 💅

2.10.6. Exactly Once Semantics 💖

Okay bestie, so through the session, NFSv4.1 gives you exactly once semantics (EOS) for requests sent over a channel. ✨ EOS is supported on both the fore channel and the backchannel 🎀.

Each COMPOUND or CB_COMPOUND request that is sent with a leading SEQUENCE or CB_SEQUENCE operation MUST be executed by the receiver exactly once 💅. This requirement holds whether or not the request is sent with reply caching specified (see Section 2.10.6.1.3). It even holds if the requester is sending the request over a session created between a pNFS data client and a pNFS data server 🦋. To understand the rationale, divide the requests into three classifications:

• Non-idempotent requests.
• Idempotent modifying requests.
• Idempotent non-modifying requests.

Okay, first up is the non-idempotent crew. RENAME is the classic example 🙈. If a replier executes the same RENAME twice and the first execution succeeds, the re-execution will fail. If the replier hands back the result of that re-execution, the result is wrong. That's why EOS is required for non-idempotent requests.

Next, idempotent modifying requests. A COMPOUND containing a WRITE is the example here. Repeating the same WRITE has the same effect as doing it once. Still, enforcing EOS for WRITEs and other idempotent modifying requests is necessary to avoid data corruption 😭.

Picture this: a client sends WRITE A to a noncompliant server that does not enforce EOS, and gets no response, maybe because of a network partition. The client reconnects and re-sends WRITE A. Now the server has two outstanding instances of A 💀. The server can execute and reply to the retry of A while the first A is still waiting in the server's internal I/O system for some resource. When the client gets the reply to that second attempt, it believes its WRITE is done, so it's free to send WRITE B, which overlaps the byte-range of A. Then the original A gets dispatched from the server's I/O system and executed. The second time A is written, it can overwrite what B wrote, and that data gets corrupted. Yikes.

Last one: idempotent non-modifying requests. A COMPOUND containing SEQUENCE, PUTFH, and READLINK, and nothing else, is the example. Re-executing it won't cause data corruption or produce an incorrect result. Nonetheless, to keep the implementation simple, the replier MUST enforce EOS for all requests, whether or not they are idempotent and non-modifying 🫶.

Note that true and complete EOS is not possible unless the server persists the reply cache in stable storage, and unless the server is somehow implemented to never require a restart. Honestly, if a server like that exists, the difference between a reply cache kept in stable storage and one that isn't has no meaning 🌈. See Section 2.10.6.5 for a discussion of persistence in the reply cache. Regardless, even if the server does not persist the reply cache, EOS improves robustness and correctness over previous versions of NFS, because the legacy duplicate request/reply caches were based on the ONC RPC transaction identifier (XID). Section 2.10.6.1 explains the shortcomings of the XID as a basis for a reply cache and describes how NFSv4.1 sessions improve upon the XID ✨.

2.10.6.1. Slot Identifiers and Reply Cache 🎀

Let's talk about the XID. The RPC layer provides a transaction ID (XID) that is required to be unique, but it isn't super convenient for tracking requests, for two reasons 🧐. First, the XID is only meaningful to the requester. The replier can't interpret it, except to test it for equality with previously sent requests. When you look something up in an RPC-based duplicate request cache, the XID's opaqueness forces a computationally expensive lookup (often a hash that includes the XID and source address). NFSv4.1 requests use a non-opaque slot ID instead, which is an index into a slot table. That's way more efficient 🔥.

Second, because RPC requests can be executed by the replier in any order, there's no bound on how many requests can be outstanding at any time. To get perfect EOS, ONC RPC would need to store every reply in the reply cache. XIDs are 32 bits, and storing over four billion (2^32) replies in the reply cache is not practical 😵. So previous NFS versions chose to store a fixed number of replies and use a least recently used (LRU) approach to replace cache entries when the cache is full. In NFSv4.1, the number of outstanding requests is bounded by the size of the slot table, and a sequence ID per slot tells the replier when it's safe to delete a cached reply 👑.

In the NFSv4.1 reply cache, when the requester sends a new request, it selects a slot ID in the range 0..N, where N is the replier's current maximum slot ID granted to the requester on the session the request is going over. N starts out equal to ca_maxrequests - 1 (Section 18.36), but the response to SEQUENCE or CB_SEQUENCE can adjust it, as described later in this section. The slot ID must be unused by any of the requests the requester already has active on the session. "Unused" here means the requester has no outstanding request for that slot ID 🍓.

A slot contains a sequence ID and the cached reply for the request sent with that sequence ID. The sequence ID is a 32-bit unsigned value, so it's in the range 0..0xFFFFFFFF (2^32 - 1). The first time a slot is used, the requester MUST specify a sequence ID of one (Section 18.36). Each time a slot is reused, the request MUST specify a sequence ID that is one greater than the previous request on that slot. If the previous sequence ID was 0xFFFFFFFF, then the next request for the slot MUST have the sequence ID set to zero (i.e., (2^32 - 1) + 1 mod 2^32) 🌸.

The sequence ID rides along with the slot ID in each request. It's the critical check at the replier: it efficiently determines whether a request using a certain slot ID is a retransmit or a brand-new, never-before-seen request 🧚. The requester can't just assert that it's retransmitting to make this work, because for any given request, the requester can't know whether the replier has seen it unless the replier actually replies. Of course, if the requester has seen the reply, it wouldn't retransmit anyway.

The replier compares each received request's sequence ID with the last one previously received for that slot ID, to see if the new request is:

• A new request, where the sequence ID is one greater than the one previously seen in the slot (accounting for sequence wraparound). The replier proceeds to execute the new request, and the replier MUST increase the slot's sequence ID by one.
• A retransmitted request, where the sequence ID is equal to the one currently recorded in the slot. If the original request has executed to completion, the replier returns the cached reply. See Section 2.10.6.2 for direction on how the replier deals with retries of requests that are still in progress 🦋.
• A misordered retry, where the sequence ID is less than the one previously seen in the slot (accounting for sequence wraparound). The replier MUST return NFS4ERR_SEQ_MISORDERED (as the result from SEQUENCE or CB_SEQUENCE).
• A misordered new request, where the sequence ID is two or more than the one previously seen in the slot (accounting for sequence wraparound). Note that because the sequence ID MUST wrap around to zero once it reaches 0xFFFFFFFF, a misordered new request and a misordered retry cannot be distinguished. Thus, the replier MUST return NFS4ERR_SEQ_MISORDERED (as the result from SEQUENCE or CB_SEQUENCE) 🙈.

Unlike the XID, the slot ID is always within a specific range, and that has two implications. The first is that, for a given session, the replier only needs to cache the results of a limited number of COMPOUND requests. The second follows from the first: unlike XID-indexed reply caches (also known as duplicate request caches - DRCs), a slot ID-based reply cache cannot be overflowed 🙌. Using the sequence ID to identify retransmitted requests means the replier doesn't even have to cache the request itself, which cuts the reply cache's storage needs even further. These facilities make it practical to maintain all the required entries for an effective reply cache ✨.

The slot ID, sequence ID, and session ID therefore take over the traditional role of the XID and source network address in the replier's reply cache implementation. This approach is considerably more portable and completely robust. It isn't affected by ports getting reassigned when clients reconnect over IP networks 💜. Also, the RPC XID isn't used in the reply cache, which makes the cache sturdier against any rapid reuse of XIDs by the requester. The replier doesn't care about the XID for reply cache management (but the replier MUST return the same XID that was in the request). Even so, there are still XID considerations in NFSv4.1 that are the same as in all previous versions of NFS. The RPC XID stays in each message and needs to be formulated in NFSv4.1 requests just like in any other ONC RPC request. The reasons include:

• The RPC layer retains its existing semantics and implementation.
• The requester and replier must be able to interoperate at the RPC layer, before the NFSv4.1 decoding of the SEQUENCE or CB_SEQUENCE operation.
• If an operation is used that does not start with SEQUENCE or CB_SEQUENCE (e.g., BIND_CONN_TO_SESSION), then the RPC XID is needed for correct operation, to match the reply to the request.
• The SEQUENCE or CB_SEQUENCE operation may generate an error. If so, the embedded slot ID, sequence ID, and session ID (if present) in the request will not be in the reply, and the requester only has the XID to match the reply to the request 🥺.

Given that well-formulated XIDs continue to be required, you might be wondering: why do SEQUENCE and CB_SEQUENCE replies have a session ID, slot ID, and sequence ID? 🤔 Great question! Having the session ID in the reply means the requester doesn't have to use the XID to look up the session ID, which it would need to do if the connection were associated with multiple sessions. Having the slot ID and sequence ID in the reply means the requester doesn't have to use the XID to look those up either. And since the XID is only 32 bits, it's too small to guarantee the re-association of a reply with its request [37]. Having the session ID, slot ID, and sequence ID in the reply lets the client validate that the reply really belongs to the matched request 💅🩷

The SEQUENCE (and CB_SEQUENCE) op also carries a highest_slotid value 💅, which tells the replier how many request slots the requester is actually using 🧚. The requester MUST always put in the slot ID for its outstanding request with the highest-numbered slot value. Always, no exceptions 🫶. The requester should in all cases send the most conservative value possible, though it can bump that number a bit above what it's really using to keep some minimum or optimal level. This is how the requester gives unused request slots back to the replier, and the replier can use that info to reshuffle resources 🦋✨

The replier answers with both a new target highest_slotid and an enforced highest_slotid. Here's how they work 👑

• The target highest_slotid tells the requester which highest_slotid the replier wants it to be using. This lets the replier take back (or hand out) resources from a requester that isn't using them, so demand from other requesters gets shared more fairly 💜. The requester MUST always comply with the replier's value updates, since they set newly established hard limits on what the requester can touch in the session 😭. But because of request pipelining, the requester may still have active requests in flight that reflect older values, so the replier must not immediately require the requester to comply 🌸
• The enforced highest_slotid is the highest slot ID the requester is permitted to use on a subsequent SEQUENCE or CB_SEQUENCE operation 🎀. The replier's enforced highest_slotid SHOULD be no less than the highest_slotid the requester sent in the SEQUENCE or CB_SEQUENCE arguments.

A requester can be intransigent about lowering its highest_slotid argument to a Sequence operation. That means it keeps ignoring the target highest_slotid in the response to a Sequence operation and keeps setting its highest_slotid argument higher than the target 🙈. This can be seen as super egregious behavior when the replier knows there are no outstanding requests with slot IDs higher than its target highest_slotid 💀. When faced with that kind of stubbornness, the replier is free to take more forceful action, and MAY reply with a new enforced highest_slotid that is less than its previous enforced highest_slotid. After that, if the requester keeps sending requests with a highest_slotid greater than the replier's new enforced highest_slotid, the server MAY return NFS4ERR_BAD_HIGH_SLOT, unless the slot ID in the request is greater than the new enforced highest_slotid and the request is a retry 🧚

The replier SHOULD retain the slots it wants to retire until the requester sends a request with a highest_slotid less than or equal to the replier's new enforced highest_slotid 🍓

The requester can also be intransigent about sending non-retry requests that have a slot ID exceeding the replier's highest_slotid. Once the replier has forcibly lowered the enforced highest_slotid, the requester is only allowed to send retries on slots that exceed the replier's highest_slotid 🙅‍♀️ If a request comes in with a slot ID higher than the new enforced highest_slotid, and the sequence ID is one higher than what's in the slot's reply cache, then the server can both retire the slot and return NFS4ERR_BADSLOT. However, the server MUST NOT do one and not the other! The reason it's safe to retire the slot is that by using the next sequence ID, the requester is showing it has received the previous reply for that slot 🫶

• The requester SHOULD use the lowest available slot when sending a new request. That way, the replier may be able to retire slot entries faster 🦋 However, where the replier is actively adjusting its granted highest_slotid, it won't be able to rely only on the slot ID and highest_slotid in the request. Neither the slot ID nor the highest_slotid used in a request may reflect the replier's current idea of the requester's session limit, because the request may have been sent before the requester received the update 💅 So in the downward adjustment case, the replier may have to keep a number of reply cache entries at least as large as the old value of maximum requests outstanding, until it can infer that the requester has seen a reply containing the new granted highest_slotid. The replier can infer that the requester has seen such a reply when it receives a new request with the same slot ID as the request replied to and the next higher sequence ID ✨

2.10.6.1.1. Caching of SEQUENCE and CB_SEQUENCE Replies 🎀

When a SEQUENCE or CB_SEQUENCE operation is successfully executed, its reply MUST always be cached. Specifically, session ID, sequence ID, and slot ID MUST be cached in the reply cache. The reply from SEQUENCE also includes the highest slot ID, target highest slot ID, and status flags. Instead of caching these values, the server MAY re-compute them from the current state of the fore channel, session, and/or client ID as appropriate. Similarly, the reply from CB_SEQUENCE includes a highest slot ID and target highest slot ID. The client MAY re-compute those values from the current state of the session as appropriate 🧚

Whether or not the replier is re-computing the highest slot ID, target slot ID, and status on replies to retries, the requester MUST NOT assume those values are being re-computed whenever it gets a reply after a retry is sent. It has no way of knowing whether the reply it got was sent by the replier in response to the retry or is a delayed response to the original request 😭

So the highest slot ID, target slot ID, or status bits may reflect how things looked when the request was first executed. Acting on that delayed info is valid, but it may make the receiver do unneeded work 🙈 Requesters MAY choose to send additional requests to get the current state of affairs, or use the state reported by subsequent requests, rather than acting right away on data that might be out of date 🫶

2.10.6.1.2. Errors from SEQUENCE and CB_SEQUENCE 💀

Any time SEQUENCE or CB_SEQUENCE returns an error, the sequence ID of the slot MUST NOT change. The replier MUST NOT modify the reply cache entry for the slot whenever an error is returned from SEQUENCE or CB_SEQUENCE 🔥

2.10.6.1.3. Optional Reply Caching 🌈

On a per-request basis, the requester can choose to direct the replier to cache the reply to all operations after the first operation (SEQUENCE or CB_SEQUENCE) using the sa_cachethis or csa_cachethis fields of the SEQUENCE or CB_SEQUENCE arguments. The reason it wouldn't direct the replier to cache the entire reply is that the request is made up of all idempotent operations [34]. Caching the reply may offer little benefit. If the reply is too large (see Section 2.10.6.4), it may not be cacheable anyway. Even if the reply to an idempotent request is small enough to cache, caching it unnecessarily slows the server down and increases RPC latency 🐢

Whether or not the requester asks for the reply to be cached has no effect on slot processing. If the results of SEQUENCE or CB_SEQUENCE are NFS4_OK, then the slot's sequence ID MUST be incremented by one ✨ If a requester does not direct the replier to cache the reply, the replier MUST do one of the following:

• The replier can cache the entire original reply. Even though sa_cachethis or csa_cachethis is FALSE, the replier is always free to cache. It may choose this approach to simplify its implementation 🍓
• The replier enters into its reply cache a reply consisting of the original results of the SEQUENCE or CB_SEQUENCE operation, with the next operation in COMPOUND or CB_COMPOUND having the error NFS4ERR_RETRY_UNCACHED_REP. So if the requester later retries the request, it gets NFS4ERR_RETRY_UNCACHED_REP. If a replier gets a retried Sequence operation where the reply to the COMPOUND or CB_COMPOUND wasn't cached, then the replier:
  • MAY return NFS4ERR_RETRY_UNCACHED_REP in reply to a Sequence operation if the Sequence operation is not the first operation (granted, a requester that does so is in violation of the NFSv4.1 protocol) 🙅‍♀️
  • MUST NOT return NFS4ERR_RETRY_UNCACHED_REP in reply to a Sequence operation if the Sequence operation is the first operation 💅
• If the second operation is an illegal operation, or an operation that was legal in a previous minor version of NFSv4 and MUST NOT be supported in the current minor version (e.g., SETCLIENTID), the replier MUST NOT ever return NFS4ERR_RETRY_UNCACHED_REP. Instead, the replier MUST return NFS4ERR_OP_ILLEGAL, NFS4ERR_BADXDR, or NFS4ERR_NOTSUPP as appropriate 🧚
• If the second operation can result in another error status, the replier MAY return a status other than NFS4ERR_RETRY_UNCACHED_REP, provided the operation isn't executed in a way that changes the replier's state. Examples of such error statuses include: NFS4ERR_NOTSUPP returned for an operation that is legal but not REQUIRED in the current minor versions, and thus not supported by the replier; NFS4ERR_SEQUENCE_POS; and NFS4ERR_REQ_TOO_BIG 🦋

The discussion above assumes the retried request matches the original one. Section 2.10.6.1.3.1 covers what the replier might do, and MUST do, when the original and retried requests don't match. Since the replier may only cache a small amount of the information it would need to tell whether this is a false retry, the replier may send the client any of the following responses:

• The cached reply to the original request (if the replier cached it in its entirety and the users of the original request and retry match).
• A reply that consists only of the Sequence operation with the error NFS4ERR_FALSE_RETRY 🙈
• A reply consisting of the response to Sequence with the status NFS4_OK, together with the second operation as it appeared in the retried request, with an error of NFS4ERR_RETRY_UNCACHED_REP or another error as described above.
• A reply that consists of the response to Sequence with the status NFS4_OK, together with the second operation as it appeared in the original request, with an error of NFS4ERR_RETRY_UNCACHED_REP or another error as described above 🌸

2.10.6.1.3.1. False Retry ✨

okay bestie, picture this 🎀 a requester sends a Sequence operation with a slot ID and sequence ID that are already sitting in the reply cache, BUT the replier spots that the retry isn't the same as the original. maybe it has different operations, or different arguments in those operations, or it comes from a different principal in the RPC request's credential field that translates to a different user. that's a false retry 💅 and when the replier catches one, it's permitted (but not always obligated) to send back NFS4ERR_FALSE_RETRY in response to the Sequence operation.

Translations of particularly privileged user values to other users, because the credentials aren't appropriately secure as configured on the replier, should be applied before deciding whether the users are the same or different 🔐. If the replier decides the users are different between the original request and the retry, then the replier MUST return NFS4ERR_FALSE_RETRY 🚨.

If an operation in the retry is an illegal operation, or an operation that was legal in a previous minor version of NFSv4 and MUST NOT be supported in the current minor version (for example, SETCLIENTID), the replier MAY return NFS4ERR_FALSE_RETRY (and MUST do so if the users of the original request and the retry differ) 🙈. Otherwise, the replier MAY return NFS4ERR_OP_ILLEGAL or NFS4ERR_BADXDR or NFS4ERR_NOTSUPP, whichever fits best. Note that this is different from how the replier handles retries with no cached reply, and that's on purpose 🧚. NFS4ERR_FALSE_RETRY is only a valid error for Sequence operations, whereas NFS4ERR_RETRY_UNCACHED_REP is valid for every operation EXCEPT illegal operations and operations that MUST NOT be supported in the current minor version of NFSv4. iconic distinction, honestly 👑

---

2.10.6.2. Retry and Replay of Reply 🔁

A requester MUST NOT retry a request unless the connection it used to send the request disconnects. Then the requester can reconnect and re-send the request, or it can re-send the request over a different connection that is associated with the same session 🦋.

If the requester is a server that wants to re-send a callback operation over the backchannel of a session, the server obviously can't reconnect, because only the client can associate connections with the backchannel 🙃. The server can re-send the request over another connection that is bound to the same session's backchannel. If there's no such connection, the server MUST indicate that the session has no backchannel by setting the SEQ4_STATUS_CB_PATH_DOWN_SESSION flag bit in the response to the next SEQUENCE operation from the client. The client MUST then associate a connection with the session (or destroy the session) 💀.

It's not fatal for a requester to retry without a disconnect between the request and the retry, ngl. BUT the retry still eats resources, especially with RDMA, where each request (retry or not) consumes a credit 🍓. Retries for no reason, especially ones sent shortly after the previous attempt, are a poor use of network bandwidth and totally defeat the purpose of a transport's inherent congestion control system 😭.

A requester MUST wait for a reply to a request before using the slot for another request. If it doesn't wait, the requester doesn't know what sequence ID to use for the slot on its next request 🤷‍♀️. For example, say a requester sends a request with sequence ID 1 and doesn't wait for the response. The next time it uses the slot, it sends the new request with sequence ID 2. If the replier hasn't seen the request with sequence ID 1, then the replier isn't expecting sequence ID 2, and it rejects the new request with NFS4ERR_SEQ_MISORDERED (as the result from SEQUENCE or CB_SEQUENCE) 🚫.

RDMA fabrics don't guarantee that the memory handles (Steering Tags) within each RPC/RDMA "chunk" [8] are valid on any scope outside a single connection 🌈. So handles used by the direct operations become invalid after a connection loss. The server must ensure that any RDMA operations that must be replayed from the reply cache use the newly provided handle(s) from the most recent request. main character energy, the newest handle always wins ✨

A retry might be sent while the original request is still in progress on the replier. The replier SHOULD deal with this by returning NFS4ERR_DELAY as the reply to the SEQUENCE or CB_SEQUENCE operation, but implementations MAY return NFS4ERR_MISORDERED. Since errors from SEQUENCE and CB_SEQUENCE are never recorded in the reply cache, this approach lets the results of the original request's execution get properly recorded in the reply cache (assuming the requester specified that the reply be cached) 🫶.

---

2.10.6.3. Resolving Server Callback Races 🥊

It's possible for server callbacks to arrive at the client BEFORE the reply from related fore channel operations 😱. For example, a client may have been granted a delegation to a file it has opened, but the reply to the OPEN (the thing that tells the client about the delegation) gets delayed in the network. If a conflicting operation arrives at the server, the server will recall the delegation using the backchannel, which might be on a different transport connection, maybe even a different network, or even a different session associated with the same client ID. chaos, but make it structured 🌸

Good news though: the presence of a session between the client and server alleviates this issue ✨. When a session is in place, each client request is uniquely identified by its { session ID, slot ID, sequence ID } triple. By the rules under which slot entries (reply cache entries) are retired, the server knows whether the client has "seen" each of the server's replies. So the server can provide enough information for the client to tell an erroneous or conflicting callback race condition apart from the rest 🧠.

For each client operation that might result in some sort of server callback, the server SHOULD "remember" the { session ID, slot ID, sequence ID } triple of the client request until the slot ID retirement rules let the server determine that the client has, in fact, seen the server's reply 👀. Until that triple can be retired, any recalls of the associated object MUST carry an array of these referring identifiers (in the CB_SEQUENCE operation's arguments), for the client's benefit 💖. After that point, the server doesn't need to include this info in related callbacks, since a race condition can no longer happen. bye bye race 👋

The CB_SEQUENCE operation that begins each server callback carries a list of "referring" { session ID, slot ID, sequence ID } triples. If the client finds the request matching the referring session ID, slot ID, and sequence ID to be currently outstanding (meaning the server's reply hasn't been seen by the client yet), it can tell that the callback raced the reply, and it can act accordingly 🏃‍♀️. If the client does NOT find the matching request outstanding (including when the session ID refers to a destroyed session), then there's no race with respect to that triple 🎉. The server SHOULD limit the referring triples to just the requests that apply to the objects referred to in the CB_COMPOUND procedure, so no unnecessary drama 💅.

The client must not simply wait forever for the expected server reply to arrive before responding to the CB_COMPOUND that won the race, because that reply could be delayed indefinitely 🕰️. The client should assume the likely case, that the reply will arrive within the average round-trip time for COMPOUND requests to the server, and wait that long. If that period expires, the client can respond to the CB_COMPOUND with NFS4ERR_DELAY 🫠. There are other scenarios where callbacks may race replies, too. Among them are pNFS layout recalls, as described in Section 12.5.5.2 🌈

2.10.6.4. COMPOUND and CB_COMPOUND Construction Issues

Okay bestie, really big requests and really big replies are a whole mood 😭 They can cause buffer drama (especially with RDMA 🫠) and reply cache drama too. When the session gets created (see Section 18.36 🎀), each channel, fore and back ✨, gets its size limits negotiated by client and server: the biggest request they'll send or process (ca_maxrequestsize), the biggest reply they'll return or process (ca_maxresponsesize), and the biggest reply they'll store in the reply cache (ca_maxresponsesize_cached) 💅

If a request goes over ca_maxrequestsize, the reply comes back with the NFS4ERR_REQ_TOO_BIG status 💀 A replier MAY return NFS4ERR_REQ_TOO_BIG as the status for the first operation (SEQUENCE or CB_SEQUENCE) in the request. That means nothing in the request ran and the slot in the reply cache is unchanged 🦋 Or it MAY opt to return it on a later operation in that same COMPOUND or CB_COMPOUND request. That means at least one operation did execute, and the state of the slot in the reply cache does change. The replier SHOULD set NFS4ERR_REQ_TOO_BIG on the operation that exceeds ca_maxrequestsize.

If a reply exceeds ca_maxresponsesize, the reply will have the status NFS4ERR_REP_TOO_BIG 🙈 A replier MAY return NFS4ERR_REP_TOO_BIG as the status for the first operation (SEQUENCE or CB_SEQUENCE) in the request, or it MAY opt to return it on a subsequent operation (in the same COMPOUND or CB_COMPOUND reply). A replier MAY return NFS4ERR_REP_TOO_BIG in the reply to SEQUENCE or CB_SEQUENCE, even if the response would still exceed ca_maxresponsesize.

If sa_cachethis or csa_cachethis is TRUE, then the replier MUST cache a reply, except if an error is returned by the SEQUENCE or CB_SEQUENCE operation (see Section 2.10.6.1.2) 🫶 If the reply exceeds ca_maxresponsesize_cached (and sa_cachethis or csa_cachethis is TRUE), then the server MUST return NFS4ERR_REP_TOO_BIG_TO_CACHE. Even if NFS4ERR_REP_TOO_BIG_TO_CACHE (or any other error for that matter) is returned on an operation other than the first operation (SEQUENCE or CB_SEQUENCE), then the reply MUST be cached if sa_cachethis or csa_cachethis is TRUE. For example, say a COMPOUND has eleven operations, including SEQUENCE. The fifth operation is a RENAME, and the tenth operation is a READ for one million bytes. The server may return NFS4ERR_REP_TOO_BIG_TO_CACHE on the tenth operation. Since the server already executed several operations, especially the non-idempotent RENAME, the client's request to cache the reply needs to be honored for exactly once semantics to keep working correctly 🔥 If the client retries the request, the server will have cached a reply containing results for ten of the eleven requested operations, with the tenth operation having a status of NFS4ERR_REP_TOO_BIG_TO_CACHE 😭

A client needs to take care that when it sends operations that change the current filehandle (except for PUTFH, PUTPUBFH, PUTROOTFH, and RESTOREFH), it does not exceed the maximum reply buffer before the GETFH operation 🧚 Otherwise, the client will have to retry the operation that changed the current filehandle, in order to get the filehandle it wanted. For the OPEN operation (see Section 18.16), retry is not always available as an option. These guidelines for handling filehandle-changing operations are advised:

• Within the same COMPOUND procedure, a client SHOULD send GETFH immediately after a current filehandle-changing operation. A client MUST send GETFH after a current filehandle-changing operation that is also non-idempotent (e.g., the OPEN operation), unless the operation is RESTOREFH. RESTOREFH is an exception, because even though it is non-idempotent, the filehandle RESTOREFH produced originated from an operation that is either idempotent (e.g., PUTFH, LOOKUP) or non-idempotent (e.g., OPEN, CREATE). If the origin is non-idempotent, then because the client MUST send GETFH after the origin operation, the client can recover if RESTOREFH returns an error ✨
• A server MAY return NFS4ERR_REP_TOO_BIG or NFS4ERR_REP_TOO_BIG_TO_CACHE (if sa_cachethis is TRUE) on a filehandle-changing operation if the reply would be too large on the next operation 🌸
• A server SHOULD return NFS4ERR_REP_TOO_BIG or NFS4ERR_REP_TOO_BIG_TO_CACHE (if sa_cachethis is TRUE) on a filehandle-changing, non-idempotent operation if the reply would be too large on the next operation, especially if the operation is OPEN 👑
• A server MAY return NFS4ERR_UNSAFE_COMPOUND to a non-idempotent, current filehandle-changing operation if it looks at the next operation (in the same COMPOUND procedure) and finds that it is not GETFH. The server SHOULD do this if it is unable to determine in advance whether the total response size would exceed ca_maxresponsesize_cached or ca_maxresponsesize 🧠

---

2.10.6.5. Persistence

Since the reply cache is bounded, it is practical for the reply cache to persist across server restarts 🌈 The replier MUST persist the following information if it agreed to persist the session (when the session was created; see Section 18.36):

• The session ID.
• The slot table, including the sequence ID and cached reply for each slot.

The above are enough for a replier to provide EOS semantics for any requests that were sent and executed before the server restarted 💜 If the replier is a client, there's no need to persist anything more, unless the client persists all other state across client restart. In that case, the server will never see any NFSv4.1-level protocol manifestation of a client restart. If the replier is a server, and only the slot table and session ID persist, then any requests the client retries after the server restart will return the results cached in the reply cache. Any new requests (i.e., the sequence ID is one greater than the slot's sequence ID) MUST be rejected with NFS4ERR_DEADSESSION (returned by SEQUENCE) 💀 That session is considered dead. A server MAY re-animate a session after a server restart, so that the session accepts new requests as well as retries. To re-animate a session, the server needs to persist additional information through the restart:

• The client ID. This is a prerequisite to let the client create more sessions associated with the same client ID as the re-animated session.
• The client ID's sequence ID that is used for creating sessions (see Sections 18.35 and 18.36). This is a prerequisite to let the client create more sessions.
• The principal that created the client ID. This allows the server to authenticate the client when it sends EXCHANGE_ID.
• The SSV, if SP4_SSV state protection was specified when the client ID was created (see Section 18.35). This lets the client create new sessions and associate connections with the new and existing sessions.
• The properties of the client ID as defined in Section 18.35.

A persistent reply cache places certain demands on the server 🎀 The execution of the sequence of operations (starting with SEQUENCE) and the placement of its results in the persistent cache MUST be atomic. If a client retries a sequence of operations that was previously executed on the server, the only acceptable outcomes are either the original cached reply or an indication that the client ID or session has been lost. That second outcome indicates a catastrophic loss of the reply cache, or a session that was deleted because the client failed to use it for an extended period of time 😭

A server could fail and restart in the middle of a COMPOUND procedure that contains one or more non-idempotent or idempotent-but-modifying operations. That creates an even bigger challenge for atomic execution and placement of results in the reply cache 🫠 One way to view the problem is as a single transaction: each operation in the COMPOUND, followed by storing the result in persistent storage, and then finally a transaction commit. If there's a failure before the transaction is committed, the server rolls back the transaction. If the server itself fails, then when it restarts, its recovery logic could roll back the transaction before starting the NFSv4.1 server.

The implementation details for atomic execution of the request and caching of the reply are beyond the scope of this document, but an example implementation for NFSv2 [38] is described in [39] 🦋

---

2.10.7. RDMA Considerations

A complete discussion of how RPC-based protocols operate over RDMA transports is in [8] 🌟 A discussion of how NFSv4, including NFSv4.1, operates over RDMA is in [9]. Where RDMA is considered, this specification assumes the use of such a layering. It addresses only the upper-layer issues relevant to making the best use of RPC/RDMA 💖

2.10.7.1. RDMA Connection Resources

RDMA requires its consumers to register memory and post buffers of a specific size and number for receive operations 🧚

Registering memory can be a relatively high-overhead operation, since it requires pinning buffers, assigning attributes (e.g., readable/writable), and initializing hardware translation. Preregistration is desirable to reduce overhead 🌸 These registrations are specific to hardware interfaces and even to RDMA connection endpoints, so negotiating their limits is desirable to manage resources effectively.

After basic registration, the RPC layer must post these buffers to handle receives. These buffers remain in use by the RPC/NFSv4.1 implementation. The size and number of them must be known to the remote peer, in order to avoid RDMA errors that would cause a fatal error on the RDMA connection 😱

NFSv4.1 manages slots as resources on a per-session basis (see Section 2.10), while RDMA connections manage credits on a per-connection basis. This means that for a peer to send data over RDMA to a remote buffer, it has to have both an NFSv4.1 slot and an RDMA credit 🦋 If multiple RDMA connections are associated with a session, then if the total number of credits across all RDMA connections associated with the session is X, and the number of slots in the session is Y, the maximum number of outstanding requests is the lesser of X and Y 👑

2.10.7.2. Flow Control ✨

Okay so, older NFS versions don't have flow control 💅 They just lean on the windowing that transports like TCP provide to throttle requests 🌸 That does NOT work with RDMA, bestie, because RDMA has no operation flow control and will terminate a connection in error once the limits get exceeded 💀 So limits like the maximum number of requests outstanding get negotiated when a session is created (see the ca_maxrequests field in Section 18.36 🎀). Those limits are the maxima that every connection tied to the session's channel(s) has to stay within 🦋 RDMA connections are managed inside those limits as Section 3.3 of [8] describes, and if there are multiple RDMA connections, the channel's maximum number of requests gets divided among them ✨ Put differently, the replier is the one on the hook to make sure the total number of RDMA credits across all connections on the replier's channel does exceed the channel's maximum number of outstanding requests. Main character responsibility, fr 👑

The limits can also be changed dynamically at the replier's choosing by tweaking certain parameters in each NFSv4.1 reply 🔥 And the CB_RECALL_SLOT callback operation (see Section 20.8) can be sent by a server to a client to hand RDMA credits back to the server 🫶 That lowers the maximum number of requests the client can have outstanding to the server, so it's giving "scale it down" energy 🍓

2.10.7.3. Padding ✨

Header padding gets requested by each peer at session initiation (see the ca_headerpadsize argument to CREATE_SESSION in Section 18.36) and then used by the RPC RDMA layer, as described in [8] 🌈 Zero padding is permitted, so you can go bare if you want 🙈

Okay this is my favorite part 😭 Padding leans on the super useful property that RDMA preserves alignment of data, even when it gets placed into anonymous (untagged) buffers 💖 If it's requested, client inline writes will insert the right pad bytes inside the request header to line the data payload up on the specified boundary 🧚 The client is encouraged to add enough padding (up to the negotiated size) so that the "data" field of the WRITE operation is aligned 🎀 Most servers can make really good use of that, since it lets them chain receive buffers so any data carried by client requests lands in the right buffers on the server, ready for file system processing 🦋 The receiver's RPC layer pays no overhead for skipping over pad bytes, and RDMA's high performance makes inserting and sending padding on the sender a huge optimization 🔥 So servers don't need to do RDMA Read to satisfy all but the largest client writes, and that's iconic 💅 An extra bonus is fewer message round trips on the network, which is a potentially good trade when latency is in the picture ⭐

Picking the padding value is its own whole thing 🙈 A big source of variable-length data in the RPC header is the authentication information, whose form is client-determined, possibly in response to what the server specifies 🌸 The contents of COMPOUNDs, the sizes of strings like the ones passed to RENAME, and so on all feed into the maximal NFSv4.1 request size, and therefore the minimal buffer size 💜 So the client has to pick its offered value carefully, so it doesn't overburden the server, and vice versa 🫶 A well-chosen padding value means higher performance, so it's worth the thought ✨

                 Sender gather:
     |RPC Request|Pad  bytes|Length| -> |User data...|
     \------+----------------------/      \
             \                             \
              \    Receiver scatter:        \-----------+- ...
         /-----+----------------\            \           \
         |RPC Request|Pad|Length|   ->  |FS buffer|->|FS buffer|->...

In the case above, the server can recycle unused buffers to the next posted receive if the actual received request didn't use them, or it can pass the now-complete buffers by reference for normal write processing 🧚 For a server that can use this, there's no need to copy incoming data, and you don't have to go through complicated end-to-end buffer advertisement and management either 🔥 That includes most kernel-based and integrated server designs, among many others 👑 The client can do similar optimizations too, if it wants 💅

2.10.7.4. Dual RDMA and Non-RDMA Transports ✨

Some RDMA transports (e.g., RFC 5040 [10]) allow a "streaming" (non-RDMA) phase, where ordinary traffic might flow before "stepping up" into RDMA mode and kicking off RDMA traffic 🦋 Some RDMA transports always start connections in RDMA mode 🌈 NFSv4.1 allows a streaming phase before RDMA mode, but it doesn't assume one exists 🫶 When a connection gets associated with a session, the client and server negotiate whether that connection gets used in RDMA or non-RDMA mode (see Sections 18.36 and 18.34) 🎀

2.10.8. Session Security 🔐

2.10.8.1. Session Callback Security ✨

Through session/connection association, NFSv4.1 improves security over what NFSv4.0 provided for the backchannel 🫶 The connection is client-initiated (see Section 18.34) and subject to the same firewall and routing checks as the fore channel 🧱 If the client chooses to (see Section 18.35), connection association gets fully authenticated before it's activated (see Section 18.34) 🔒 And traffic from the server over the backchannel is authenticated exactly as the client specifies (see Section 2.10.8.2) 💖

2.10.8.2. Backchannel RPC Security ✨

When the NFSv4.1 client sets up the backchannel, it tells the server which security flavors and principals to use when sending requests 🎀 If the security flavor is RPCSEC_GSS, the client expresses the principal as an established RPCSEC_GSS context 🦋 The server is free to use any of the flavor/principal combinations the client offers, but it MUST NOT use combinations that weren't offered 🚫 This means the client doesn't have to provide a target GSS principal for the backchannel like it did with NFSv4.0, and the server doesn't have to implement an RPCSEC_GSS initiator like it did with NFSv4.0 [30] 🧚

The CREATE_SESSION (Section 18.36) and BACKCHANNEL_CTL (Section 18.33) operations let the client specify those flavor/principal combinations 🌸

Also, the SP4_SSV state protection mode (see Sections 18.35 and 2.10.8.3) has a side benefit: it provides SSV-derived RPCSEC_GSS contexts (Section 2.10.9) 💜

2.10.8.3. Protection from Unauthorized State Changes ✨

Okay so, as the spec stands up to this point, the NFSv4.1 state model is vulnerable to an attacker 😱 The attacker sends a SEQUENCE operation with a forged session ID, and with a slot ID that they expect the legitimate client to use next 💀 When the legit client uses that slot ID with the same sequence number, the server returns the attacker's result from the reply cache, which messes with the legit client and denies it service 🙈 It gets worse: an attacker could send a CREATE_SESSION with a forged client ID to create a new session associated with that client ID 🔥 Then the attacker could send requests on the new session that change locking state, like LOCKU operations that release locks the legitimate client has acquired 😭 Putting a security policy on the file that requires RPCSEC_GSS credentials when manipulating the file's state is one possible workaround 🫶 But it has the downside of stopping a legit client from releasing state when RPCSEC_GSS is required to do so and a GSS context can't be obtained (maybe because the user has logged off the client) 🥺

NFSv4.1 gives a client three options for state protection, which get specified when the client creates a client ID via EXCHANGE_ID (Section 18.35) 🌈

The first one, SP4_NONE, is to simply waive state protection 🙅‍♀️

The other two options, SP4_MACH_CRED and SP4_SSV, share several traits 💜

• An RPCSEC_GSS-based credential is used to authenticate client ID and session maintenance operations, including creating and destroying a session, associating a connection with the session, and destroying the client ID 🔐
• Because RPCSEC_GSS authenticates client ID and session maintenance, the attacker can't associate a rogue connection with a legit session, or a rogue session with a legit client ID to mess with the client ID's lock state via CLOSE, LOCKU, DELEGRETURN, LAYOUTRETURN, and so on 🛡️
• If the server's security policies on part of its namespace require RPCSEC_GSS authentication, a client might have to use an RPCSEC_GSS credential to remove per-file state (e.g., LOCKU, CLOSE, etc.) 🥹 The server might require that the principal removing the state matches certain criteria (e.g., the principal might have to be the same one that acquired the state) 🧐 But the client might not have an RPCSEC_GSS context for that principal, and might not be able to create one (perhaps because the user has logged off) 😵 When the client establishes SP4_MACH_CRED or SP4_SSV protection, it can specify a list of operations that the server MUST allow using the machine credential (if SP4_MACH_CRED is used) or the SSV credential (if SP4_SSV is used) 🎀

SP4_MACH_CRED uses a machine credential, and the principal that creates the client ID MUST also be the principal that performs client ID and session maintenance operations 💅 The security of this machine credential approach depends entirely on safeguarding the per-machine credential 🔒 If that's properly safeguarded when used for operations like CREATE_SESSION, BIND_CONN_TO_SESSION, DESTROY_SESSION, and DESTROY_CLIENTID, the attacker can't associate a rogue connection with a session or a rogue session with a client ID 🙌

There are at least three scenarios for SP4_MACH_CRED 🧩

1. The system administrator configures a unique, permanent per-machine credential for one of the mandated GSS mechanisms (e.g., if Kerberos V5 is used, a "keytab" containing a principal derived from a client host name could be used) 🗝️
2. The client is used by a single user, so the client ID and its sessions are used by just that user. If the user's credential expires, session and client ID maintenance can't happen, but since there's only one user, only that user is inconvenienced 🙃
3. The physical client has multiple users, but the client implementation has a unique client ID for each user. This is basically the same as the second scenario, but the downside is that each user needs at least one session allocated, so the approach lacks economy 💸

SP4_SSV uses the SSV (Section 1.6), via RPCSEC_GSS and the SSV GSS mechanism (Section 2.10.9), to protect state from attack 🛡️ It's meant for a situation where the client has multiple active users and a system administrator who wants to avoid installing a permanent machine credential on every client 🙏 The SSV is established and updated on the server via SET_SSV (see Section 18.47) 🌸 To prevent eavesdropping, a client SHOULD send SET_SSV via RPCSEC_GSS with the privacy service 🔐 Several aspects of the SSV make it intractable for an attacker to guess the SSV, and therefore to associate rogue connections with a session, and rogue sessions with a client ID 🧠:

• The arguments to and results of SET_SSV carry digests of the old and new SSV, respectively. 🔐
• Okay so the initial SSV value is zero, which means it's known to everyone 👀. So the client that picks SP4_SSV protection and applies it to BIND_CONN_TO_SESSION and CREATE_SESSION MUST send at least one SET_SSV before the first BIND_CONN_TO_SESSION, or before the second CREATE_SESSION, on a client ID. If it doesn't, the SSV mechanism won't generate tokens (Section 2.10.9). A client SHOULD send SET_SSV as soon as a session is created. 💖
• A SET_SSV request does NOT swap the SSV for its argument 🙅‍♀️. Instead, the server's current SSV gets logically XORed with the argument. Every time a new principal uses a client ID for the first time, the client SHOULD send a SET_SSV using that principal's RPCSEC_GSS credentials, with the RPCSEC_GSS service set to RPC_GSS_SVC_PRIVACY. 🎀

Okay, so here are the sneaky ways an attacker named Eve could go after a victim named Bob, and how SP4_SSV protection shuts each one down 💅

• Say Eve is the first person to log into a legit client. Her use of an NFSv4.1 file system makes the client create a client ID with SP4_SSV protection, and it specifies that BIND_CONN_TO_SESSION MUST use the SSV credential. Her file system use also creates an SSV. The SET_SSV that creates the SSV gets protected by the RPCSEC_GSS context the legit client made, which uses Eve's GSS principal and credentials. Eve can eavesdrop while her RPCSEC_GSS context is being created and her SET_SSV is sent 👀. Even if the legit client sends that SET_SSV with RPC_GSS_SVC_PRIVACY, Eve knows her own credentials, so she can decrypt the SSV. 😭 She can then compute an RPCSEC_GSS credential that BIND_CONN_TO_SESSION will accept, and attach a new connection to the legit session. She can also change the slot ID and sequence state of a legit session, and/or the SSV state, so that when Bob uses the server through the same legit client, the client can't use the session anymore. 💀

The client's only way out is to create a new client ID for Bob and set up a new SSV for it. It can't delete the old client ID, so it just lets that lease expire. 🥀

Once the legit client establishes an SSV over the new session using Bob's RPCSEC_GSS context, Eve can still use the new session through the legit client, but she can't mess with Bob anymore. 🙌 Also, because the client SHOULD have changed the SSV when Eve used the new session, Bob can't get revenge on Eve by attaching a rogue connection to it. 😤

So how did the legit client even notice Eve hijacked the old session? When it sees a new principal, Bob, wanting to use the session, it SHOULD have sent a SET_SSV, and that leads to these sub-scenarios:

• Let's say Eve sent a SET_SSV from the rogue connection, using the same slot ID and sequence ID the legit client later uses. The server treats the SET_SSV sent with Bob's credentials as a retry, and hands the legit client the reply it gave Eve. But unless Eve can correctly guess the SSV the legit client will use, the digest checks in that SET_SSV response will fail. That tells the client the session has apparently been hijacked. 🚨
• Or, Eve sent a SET_SSV with a different slot ID than the one the legit client uses for its SET_SSV. Then the digest verification of the SET_SSV sent with Bob's credentials fails on the server, and the error sent back makes it obvious the session got hijacked. 🕵️‍♀️
• Or, Eve sent some operation other than SET_SSV, but with the same slot ID and sequence the legit client uses for its SET_SSV. The server returns the response it gave Eve to the legit client. The client sees it's not at all what it expected 🙃. It assumes either a session hijack or a server bug, and either way it destroys the old session.

• Eve attaches a rogue connection to the session like before, and then destroys the session. Bob goes to use the server from the legit client, which sends a SET_SSV with Bob's credentials. The client gets an error saying the session doesn't exist. When the client tries to create a new session, that fails too, because the SSV it has doesn't match the server's, and now it knows the session was hijacked. The legit client sets up a new client ID. 🫶
• If Eve creates a connection before the legit client establishes an SSV, she can send a SET_SSV that passes the digest verification check, since the initial SSV value is zero and therefore known. However, because the new connection hasn't been associated with the session, that SET_SSV gets rejected for that reason. 🙅‍♀️

Lowkey the big takeaway: when SP4_SSV protection is in use, an attacker can only mess with state during the formative period of a client ID, its first session, and the setup of the SSV ⏳. Once a non-malicious user is using the client ID, the client quickly catches any hijack and fixes it 🔧. And once a non-malicious user successfully changes the SSV, the attacker can't use NFSv4.1 operations to disrupt them anymore. 👑

Heads up, bestie: neither the SP4_MACH_CRED nor the SP4_SSV protection approaches stop hijacking of a transport connection that's already been associated with a session. If your counter-threat strategy is to prevent connection hijacking, using IPsec is RECOMMENDED. 🔐

If a connection hijack does happen, the hijacker could in theory change locking state and mess with service for legit clients. 😰 But if the server is configured to require RPCSEC_GSS with integrity or privacy on the affected file objects, AND the EXCHGID4_FLAG_BIND_PRINC_STATEID capability (Section 18.35) is in force, that blocks unauthorized attempts to change locking state. Dramatic, I know, but it's the difference between a lock and a free-for-all 🔒✨

2.10.9. The Secret State Verifier (SSV) GSS Mechanism ✨

Okay bestie, the SSV is giving secret key energy 🔐 It provides the secret key for a GSS mechanism that lives inside NFSv4.1, and NFSv4.1 uses it for state protection 🛡️ The contexts for this mechanism are NOT set up through RPCSEC_GSS 🙅‍♀️ Instead, they get created automatically when EXCHANGE_ID specifies SP4_SSV protection 💅 The only tokens defined are the PerMsgToken (emitted by GSS_GetMIC) and the SealedMessage token (emitted by GSS_Wrap). Two tokens, zero drama 💖

The mechanism OID for the SSV mechanism is iso.org.dod.internet.private.enterprise.Michael Eisler.nfs.ssv_mech (1.3.6.1.4.1.28882.1.1) 🦋 The SSV mechanism doesn't define any initial context tokens, but the OID can be used so servers can say the SSV mechanism is acceptable whenever the client sends a SECINFO or SECINFO_NO_NAME operation (see Section 2.6). So the server just gets to flag it as an option, which is iconic 👑

The SSV mechanism defines four subkeys derived from the SSV value 🔑✨ Every time SET_SSV is invoked, the client and server recalculate the subkeys 🔄 Each of the four subkeys depends on its own one of the four ssv_subkey4 enumerated values. The math uses HMAC [11], with the current SSV as the key, the one-way hash algorithm negotiated by EXCHANGE_ID, and the input text being the XDR encoded enumeration value for that subkey (of data type ssv_subkey4). Now here's the important part 🙈 If the HMAC output is longer than the key length of the encryption algorithm (also negotiated by EXCHANGE_ID), the subkey MUST be truncated from the HMAC output. That means if the subkey is N bytes long, the first N bytes of the HMAC output MUST be used for the subkey. Also, EXCHANGE_ID says the length of the HMAC output MUST NOT be less than the length of subkey needed for the encryption algorithm (see Section 18.35). No cutting corners 🚫

/* Input for computing subkeys */
enum ssv_subkey4 {
        SSV4_SUBKEY_MIC_I2T     = 1,
        SSV4_SUBKEY_MIC_T2I     = 2,
        SSV4_SUBKEY_SEAL_I2T    = 3,
        SSV4_SUBKEY_SEAL_T2I    = 4
};

Okay so the subkey from SSV4_SUBKEY_MIC_I2T is used for message integrity codes (MICs) that come from the NFSv4.1 client. That covers a request on the fore channel or a response on the backchannel 🧚 The subkey from SSV4_SUBKEY_MIC_T2I is for MICs coming from the NFSv4.1 server. The subkey from SSV4_SUBKEY_SEAL_I2T is for encryption text from the client, and SSV4_SUBKEY_SEAL_T2I is for encryption text from the server 🌸

The PerMsgToken is based on an XDR definition 📜

/* Input for computing smt_hmac */
struct ssv_mic_plain_tkn4 {
  uint32_t        smpt_ssv_seq;
  opaque          smpt_orig_plain<>;
};


/* SSV GSS PerMsgToken token */
struct ssv_mic_tkn4 {
  uint32_t        smt_ssv_seq;
  opaque          smt_hmac<>;
};

The field smt_hmac is an HMAC 🧮 It's calculated using the subkey from SSV4_SUBKEY_MIC_I2T or SSV4_SUBKEY_MIC_T2I as the key, the one-way hash algorithm negotiated by EXCHANGE_ID, and the input text, which is the data of type ssv_mic_plain_tkn4. The field smpt_ssv_seq is the same as smt_ssv_seq. The field smpt_orig_plain is the "message" input passed to GSS_GetMIC() (see Section 2.3.1 of [7]). The caller of GSS_GetMIC() gives a pointer to a buffer with the plain text. The SSV mechanism's entry point for GSS_GetMIC() encodes that into an opaque array. The encoding includes an initial four-byte length, plus any necessary padding. Then the XDR encoded value of smpt_ssv_seq gets prepended to it, which makes up an XDR encoding of a ssv_mic_plain_tkn4 value. That whole thing is the input into the HMAC 💅

The token that GSS_GetMIC() emits is XDR encoded and is of XDR data type ssv_mic_tkn4. The field smt_ssv_seq comes from the SSV sequence number. That number equals one after SET_SSV (Section 18.47) is called the first time on a client ID, and then it goes up by one on each SET_SSV after that 📈 So smt_ssv_seq tells you which version of the SSV was in use when GSS_GetMIC() was called. As noted in Section 18.35, the client and server can keep multiple concurrent versions of the SSV. This is the trick that lets the SSV change without having to serialize every RPC call that uses the SSV mechanism with SET_SSV operations ✨ Once the HMAC is calculated, it's XDR encoded into smt_hmac. That includes an initial four-byte length and any necessary padding, and the XDR encoded value of smt_ssv_seq gets prepended to it 🎀

The SealedMessage is based on an XDR definition too 🫶

/* Input for computing ssct_encr_data and ssct_hmac */
struct ssv_seal_plain_tkn4 {
  opaque          sspt_confounder<>;
  uint32_t        sspt_ssv_seq;
  opaque          sspt_orig_plain<>;
  opaque          sspt_pad<>;
};


/* SSV GSS SealedMessage token */
struct ssv_seal_cipher_tkn4 {
  uint32_t      ssct_ssv_seq;
  opaque        ssct_iv<>;
  opaque        ssct_encr_data<>;
  opaque        ssct_hmac<>;
};

The token that GSS_Wrap() emits is XDR encoded and is of XDR data type ssv_seal_cipher_tkn4. The ssct_ssv_seq field means the same thing as smt_ssv_seq 🦋

The ssct_encr_data field is the result of encrypting a value of the XDR encoded data type ssv_seal_plain_tkn4. The encryption key is the subkey from SSV4_SUBKEY_SEAL_I2T or SSV4_SUBKEY_SEAL_T2I, and the encryption algorithm is the one negotiated by EXCHANGE_ID 🔐

The ssct_iv field is the initialization vector (IV) for the encryption algorithm, if one applies, and it gets sent in clear text 👀 The content and size of the IV MUST comply with the specification of the encryption algorithm. For example, the id-aes256-CBC algorithm MUST use a 16-byte initialization vector (IV), and that IV MUST be unpredictable for each instance of a ssv_seal_plain_tkn4 value that gets encrypted with a particular SSV key. Unpredictable IVs are a must-have, periodt 💖

The ssct_hmac field is the result of computing an HMAC using the value of the XDR encoded data type ssv_seal_plain_tkn4 as the input text. The key is the subkey from SSV4_SUBKEY_MIC_I2T or SSV4_SUBKEY_MIC_T2I, and the one-way hash algorithm is the one negotiated by EXCHANGE_ID 🧮

The sspt_confounder field is a random value 🎲 The sspt_ssv_seq field is the same as ssvt_ssv_seq. The sspt_orig_plain field is the original plaintext, and it's the "input_message" input passed to GSS_Wrap() (see Section 2.3.3 of [7]). Just like the GSS_GetMIC() entry point handles plaintext, the GSS_Wrap() entry point expects a pointer to the plaintext. It XDR encodes an opaque array into sspt_orig_plain to represent the plain text, along with the other fields of a ssv_seal_plain_tkn4 instance 🍓

The sspt_pad field is there to support encryption algorithms that need inputs in fixed-size blocks 🧱 Its content is zero filled, except for the length. Beware, bestie 👀 The XDR encoding of ssv_seal_plain_tkn4 has three variable-length arrays, so each array uses four bytes for its length, and each array that follows its length is always padded to a multiple of four bytes, per the XDR standard.

Let's do the example together 🧠 Say the encryption algorithm uses 16-byte blocks, the sspt_confounder is three bytes long, and the sspt_orig_plain field is 15 bytes long. The XDR encoding of sspt_confounder uses eight bytes (4 + 3 + 1 byte pad). The XDR encoding of sspt_ssv_seq uses four bytes. The XDR encoding of sspt_orig_plain uses 20 bytes (4 + 15 + 1 byte pad). The smallest XDR encoding of sspt_pad is four bytes. That's 36 bytes total. The next multiple of 16 is 48, so the length field of sspt_pad gets set to 12 bytes, for a total encoding of 16 bytes. The grand total is 8 + 4 + 20 + 16 = 48 XDR encoded bytes. Math is slay 🧮💅

GSS_Wrap() emits a token that's an XDR encoding of a ssv_seal_cipher_tkn4 value. Here's the lowkey wild part 😭 No matter whether the caller of GSS_Wrap() asks for confidentiality, the token ALWAYS has confidentiality. That's because the SSV mechanism is for RPCSEC_GSS, and RPCSEC_GSS never produces GSS_wrap() tokens without confidentiality 🔒🌈

There is one SSV per client ID. There's a single GSS context for each client ID / SSV pair, and all the SSV mechanism RPCSEC_GSS handles for a client ID / SSV pair share that same GSS context 🫂 SSV GSS contexts don't expire, except when the SSV is destroyed. Causes for that include the client ID being destroyed or a server restart ⚠️ Now, one purpose of context expiration is to swap out keys that have been in use for "too long", since those are more vulnerable to compromise by brute force or accident. So the client can replace the SSV key by sending periodic SET_SSV operations, and it does that by cycling through different users' RPCSEC_GSS credentials. That way the SSV gets replaced without destroying the SSV's GSS contexts. Galaxy brain move 🧠👑

SSV RPCSEC_GSS handles can be expired or deleted by the server at any time. The EXCHANGE_ID operation can be used to create more SSV RPCSEC_GSS handles 🧚 Expiration of an SSV RPCSEC_GSS handle does NOT mean the SSV or its GSS context has expired. Different things, bestie 🙏

The client MUST establish an SSV via SET_SSV before the SSV GSS context can be used to emit tokens from GSS_Wrap() and GSS_GetMIC(). If SET_SSV has not been successfully called, attempts to emit tokens MUST fail. No SET_SSV, no tokens, full stop 🛑

The SSV mechanism does not support replay detection and sequencing in its tokens, because RPCSEC_GSS does not use those features (See Section 5.2.2, "Context Creation Requests", in [4]). However, Section 2.10.10 discusses special considerations for the SSV mechanism when it's used with RPCSEC_GSS 🎀💜

2.10.10. Security Considerations for RPCSEC_GSS When Using the SSV Mechanism 🔐

Okay bestie, so when a client ID gets created with SP4_SSV state protection (see Section 18.35), the client is allowed to hook multiple RPCSEC_GSS handles up to one single SSV GSS context (see Section 2.10.9). 🦋 But here's the tea: RPCSEC_GSS works out the reply verifier in a specific way (both version 1 and version 2, see [4] and [12]), so the NFSv4.1 client has to be extra careful to stop man-in-the-middle attacks 😭. The verifier on an RPCSEC_GSS reply is the output of GSS_GetMIC() applied to the seq_num field of the RPCSEC_GSS credential (data type rpc_gss_cred_ver_1_t), see Section 5.3.3.2 of [4]. So if a few RPCSEC_GSS handles share the same GSS context, and one handle sends a request with the same seq_num as another handle, an attacker could block the reply and swap in the verifier from the other handle. Not it 💀.

There are a few ways to stop this attack on the SSV RPCSEC_GSS verifier in the reply. The simplest one is believed to be this 💅:

• Each time one or more new SSV RPCSEC_GSS handles get created via EXCHANGE_ID, the client SHOULD send a SET_SSV operation to change the SSV. Changing the SSV means the new handles can't end up reusing an SSV RPCSEC_GSS verifier in a reply. ✨
• When a requester decides to use N SSV RPCSEC_GSS handles, it SHOULD give each handle its own unique, non-overlapping range of seq_nums. The size of each range SHOULD be equal to MAXSEQ / N (see Section 5 of [4] for the definition of MAXSEQ). When an SSV RPCSEC_GSS handle hits its maximum, it SHOULD force the replier to destroy the handle by sending a NULL RPC request with seq_num set to MAXSEQ + 1 (see Section 5.3.3.3 of [4]).
• When the requester wants to increase or decrease N, it SHOULD force the replier to destroy all N handles by sending a NULL RPC request on each handle with seq_num set to MAXSEQ + 1. If the requester is the client, it SHOULD send a SET_SSV operation before using new handles. If the requester is the server, then the client SHOULD send a SET_SSV operation when it detects that the server has forced it to destroy a backchannel's SSV RPCSEC_GSS handle. Sending a SET_SSV changes the SSV, so the attacker can't successfully replay an older verifier in a reply to the requester. 🙈

Note that if the replier carefully creates the SSV RPCSEC_GSS handles, the related risk of a man-in-the-middle splicing a forged SSV RPCSEC_GSS credential together with a verifier for another handle does not exist. 🧚 This is because the verifier in an RPCSEC_GSS request is computed from input that includes both the RPCSEC_GSS handle and seq_num (see Section 5.3.1 of [4]). As long as the replier avoids reusing the value of an RPCSEC_GSS handle it creates, such as by including a generation number in the handle, the man-in-the-middle won't be able to successfully replay a previous verifier in the request to a replier. 👑

---

2.10.11. Session Mechanics - Steady State

2.10.11.1. Obligations of the Server

The server has the main job of keeping an eye on the backchannel resources the client has created for it (RPCSEC_GSS contexts and backchannel connections). 👀 If those resources vanish, the server takes action as specified in Section 2.10.13.2.

2.10.11.2. Obligations of the Client

The client SHOULD keep up with the following obligations so it can actually use the session 💖:

• Keep the session from going idle on the server. A client that needs a session but isn't sending any operations risks having the server destroy it. Sessions eat up resources, and when resources get tight, the server may cull an inactive session. A server MAY consider a session inactive if the client hasn't used it before the session inactivity timer (Section 2.10.12) has expired. 😭
• Destroy the session when you don't need it. If a client has several sessions, and one of them has no requests waiting for replies and has been idle for a while, it SHOULD destroy that session. 🧹
• Maintain GSS contexts and RPCSEC_GSS handles for the backchannel. If the client needs the server to use the RPCSEC_GSS security flavor for callbacks, it has to make sure the RPCSEC_GSS handles and/or their GSS contexts handed to the server via BACKCHANNEL_CTL or CREATE_SESSION are still unexpired. ⏰
• Preserve a connection for a backchannel. The server needs a backchannel to gracefully recall recallable state or to notify the client of certain events. If the connection isn't being used for the fore channel, the client has no way to tell whether it's still alive (for example, the server restarted without sending a disconnect). The onus is on the server, not the client, to figure out whether the backchannel's connection is alive, and to indicate in the response to a SEQUENCE operation when the last connection associated with a session's backchannel has disconnected. 🌈

2.10.11.3. Steps the Client Takes to Establish a Session

If the client doesn't have a client ID yet, it sends EXCHANGE_ID to establish one. If it opts for SP4_MACH_CRED or SP4_SSV protection, it SHOULD at minimum specify CREATE_SESSION, DESTROY_SESSION, BIND_CONN_TO_SESSION, BACKCHANNEL_CTL, and DESTROY_CLIENTID in the spo_must_enforce list of operations. 📝 If it opts for SP4_SSV protection, the client needs to ask for SSV-based RPCSEC_GSS handles.

The client uses the client ID to send a CREATE_SESSION on a connection to the server. The results of CREATE_SESSION tell the client whether the server will persist the session reply cache through a server restart, and the client should note that for later. 🧠

If the client specified SP4_SSV state protection when the client ID was created, then it SHOULD send SET_SSV in the first COMPOUND after the session is created. Each time a new principal goes to use the client ID, it SHOULD send a SET_SSV again. 🎀

If the client wants delegations, layouts, directory notifications, or any other state that needs a backchannel, it has to add a connection to the backchannel if CREATE_SESSION didn't already do that. The client creates a connection and calls BIND_CONN_TO_SESSION to link the connection to the session and the session's backchannel. If CREATE_SESSION didn't already do this, the client MUST tell the server what security is required for the client to accept callbacks, and it does that via BACKCHANNEL_CTL. If the client picked SP4_MACH_CRED or SP4_SSV protection when it called EXCHANGE_ID, then the client SHOULD specify that the backchannel use RPCSEC_GSS contexts for security. 🦋

If the client wants more connections for the backchannel, it calls BIND_CONN_TO_SESSION on each connection it wants to use with the session. If it wants extra connections for the fore channel, it needs to call BIND_CONN_TO_SESSION if it specified SP4_SSV or SP4_MACH_CRED state protection when the client ID was created. 💅

At this point, the session has reached steady state. 🎉

---

2.10.12. Session Inactivity Timer

The server MAY maintain a session inactivity timer for each session. If that timer expires, the server MAY destroy the session. To avoid losing a session to inactivity, the client MUST renew the session inactivity timer. The length of the session inactivity timer MUST NOT be less than the lease_time attribute (Section 5.8.1.11). Just like lease renewal (Section 8.3), when the server gets a SEQUENCE operation, it resets the session inactivity timer, and it MUST NOT let the timer expire while the rest of the operations in the COMPOUND procedure's request are still running. ⏳ Once the last operation finishes, the server MUST set the session inactivity timer to expire no sooner than the sum of the current time and the value of the lease_time attribute. 💜

---

2.10.13. Session Mechanics - Recovery

2.10.13.1. Events Requiring Client Action

The following events require the client to take action to recover. 🚨

2.10.13.1.1. RPCSEC_GSS Context Loss by Callback Path

If all RPCSEC_GSS handles the client granted to the server for callback use have expired, the client MUST establish a new handle via BACKCHANNEL_CTL. The sr_status_flags field of the SEQUENCE results tells the client when callback handles are nearly expired or fully expired (see Section 18.46.3). 😭

2.10.13.1.2. Connection Loss

If the client loses the last connection of the session and wants to keep the session, it needs to create a new connection. If BIND_CONN_TO_SESSION was in the spo_must_enforce list when the client ID was created, the client MUST use BIND_CONN_TO_SESSION to link the connection to the session. 🔗

If a request was outstanding when the connection was lost, and the client wants to keep using the session, it MUST retry the request, as described in Section 2.10.6.2. It's not necessary to retry over a connection with the same source network address or the same destination network address as the lost one. As long as the session ID, slot ID, and sequence ID in the retry match the original request, the server will recognize it as a retry if it already executed the request before the disconnect. 🙌

If the connection that was lost was the last one associated with the backchannel, and the client wants to keep the backchannel and/or prevent revocation of recallable state, the client needs to reconnect. If it does, it MUST associate the connection with the session and backchannel via BIND_CONN_TO_SESSION. The server SHOULD indicate when it has no callback connection via the sr_status_flags result from SEQUENCE. 🌸

2.10.13.1.3. Backchannel GSS Context Loss

Through the sr_status_flags result of the SEQUENCE operation or some other means, the client will find out whether some or all of the RPCSEC_GSS contexts it assigned to the backchannel have been lost. If the client wants to keep the backchannel and/or avoid putting recallable state at risk of revocation, it needs to use BACKCHANNEL_CTL to assign new contexts. 🦋

2.10.13.1.4. Loss of Session 🫥

Okay so the replier can literally lose its record of the session 😭 Here's what can cause it:

• Replier failure and restart 💀
• A catastrophe that causes the reply cache to be corrupted or lost on the media on which it was stored. This applies even if the replier indicated in the CREATE_SESSION results that it would persist the cache.
• The server purges the session of a client that has been inactive for a very extended period of time 🕰️
• As a result of configuration changes among a set of clustered servers, a network address previously connected to one server becomes connected to a different server that has no knowledge of the session in question. Such a configuration change will generally only happen when the original server ceases to function for a time 🙈

Losing the reply cache is the same thing as losing the session, bestie 💅 The replier tells the requester about it by returning NFS4ERR_BADSESSION on the next operation that uses the session ID for the lost session.

After something like a server restart, the client might have lost its connections. It assumes, for the moment, that the session is still fine 🥹 It reconnects, and if it asked for connection association enforcement when the session was created, it invokes BIND_CONN_TO_SESSION using the session ID. Otherwise, it invokes SEQUENCE. If BIND_CONN_TO_SESSION or SEQUENCE returns NFS4ERR_BADSESSION, the client knows the session isn't available to it when it's talking to that network address. If the connection survives the session loss, the next SEQUENCE operation the client sends over that connection gets back NFS4ERR_BADSESSION, and the client again knows the session was lost. Main character realization moment 👑

Here is one suggested algorithm for the client when it gets NFS4ERR_BADSESSION. It is not obligatory: if a client doesn't want to take advantage of features like trunking, it can skip parts of it. But it's a super useful example that shines a light on the different recovery issues that can pop up ✨

1. If the client has other connections to other server network addresses associated with the same session, attempt a COMPOUND with a single operation, SEQUENCE, on each of the other connections.
2. If the attempts succeed, the session is still alive, and that's a strong sign the server's network address has moved 🦋 The client might send an EXCHANGE_ID on the connection that returned NFS4ERR_BADSESSION to see if there are chances for client ID trunking (i.e., the same client ID and so_major are returned). The client might also use DNS to see if the moved network address was replaced with another one, so the performance and availability benefits of session trunking can keep going 🚀
3. If the SEQUENCE requests fail with NFS4ERR_BADSESSION, then the session no longer exists on any of the server network addresses the client has connections associated with that session ID. It's possible the session is still alive and available on other network addresses 🤞 The client sends an EXCHANGE_ID on all the connections to see if the server owner is still listening on those network addresses. If the same server owner is returned but a new client ID is returned, that's a strong indicator of a server restart. If both the same server owner and the same client ID are returned, that's a strong indication that the server did delete the session, and the client will need to send a CREATE_SESSION if it has no other sessions for that client ID. If a different server owner is returned, the client can use DNS to find other network addresses. If it doesn't, or if DNS doesn't find any other addresses for the server, then the client won't be able to provide NFSv4.1 service, and fatal errors should be returned to processes that were using the server. If the client is using a "mount" paradigm, unmounting the server is advised 😩
4. If the client knows of no other connections associated with the session ID and the server network addresses that are, or have been, associated with the session ID, then the client can use DNS to find other network addresses. If it doesn't, or if DNS doesn't find any other addresses for the server, then the client won't be able to provide NFSv4.1 service, and fatal errors should be returned to processes that were using the server. If the client is using a "mount" paradigm, unmounting the server is advised.

If there's a reconfiguration event that results in the same network address being assigned to servers where the eir_server_scope value is different, it can't be guaranteed that a session ID generated by the first server will be recognized as invalid by the first server 🫠 So when managing server reconfigurations among servers with different server scope values, it's necessary to make sure all clients have disconnected from the first server before the reconfiguration happens. Still, clients should not assume that servers will always follow this requirement, so clients MUST be prepared to deal with unexpected effects of server reconfigurations 🙃 Even where a session ID is inappropriately recognized as valid, it's likely that either the connection won't be recognized as valid or a sequence value for a slot won't be correct. So when a client receives results indicating such unexpected errors, using EXCHANGE_ID to determine the current server configuration is RECOMMENDED 💜

A variation on the above: after a server's network address moves, there may be no NFSv4.1 server listening, e.g., no listener on port 2049. In this case, one of the following happens: the NFSv4 server returns NFS4ERR_MINOR_VERS_MISMATCH, the NFS server returns a PROG_MISMATCH error, the RPC listener on 2049 returns PROG_UNVAIL, or attempts to reconnect to the network address time out. These SHOULD be treated as equivalent to SEQUENCE returning NFS4ERR_BADSESSION for these purposes 🎀

When the client detects session loss, it needs to call CREATE_SESSION to recover. Any non-idempotent operations that were in progress might have been performed on the server at the time of session loss, and the client has no general way to recover from that 😬

Note that loss of session does not imply loss of byte-range lock, open, delegation, or layout state, because locks, opens, delegations, and layouts are tied to the client ID and depend on the client ID, not the session. Nor does loss of byte-range lock, open, delegation, or layout state imply loss of session state, because the session depends on the client ID. Loss of client ID, however, does imply loss of session, byte-range lock, open, delegation, and layout state. See Section 8.4.2. A session can survive a server restart, but lock recovery may still be needed 🔐

It's possible that CREATE_SESSION will fail with NFS4ERR_STALE_CLIENTID (e.g., the server restarts and doesn't preserve client ID state). If so, the client needs to call EXCHANGE_ID, followed by CREATE_SESSION 🔁

---

2.10.13.2. Events Requiring Server Action 🛠️

The following events require server action to recover.

2.10.13.2.1. Client Crash and Restart

As described in Section 18.35, a restarted client sends EXCHANGE_ID in such a way that it causes the server to delete any sessions it had. Iconic clean slate energy ✨

2.10.13.2.2. Client Crash with No Restart

If a client crashes and never comes back, it will never send EXCHANGE_ID with its old client owner. So the server has session state that will never be used again 👻 After an extended period of time, and if the server has resource constraints, it MAY destroy the old session as well as the locking state.

2.10.13.2.3. Extended Network Partition

To the server, an extended network partition may be no different from a client crash with no restart (see Section 2.10.13.2.2). Unless the server can discern that there is a network partition, it is free to treat the situation as if the client has crashed permanently 🫣

2.10.13.2.4. Backchannel Connection Loss

If there were callback requests outstanding at the time of a connection loss, then the server MUST retry the requests, as described in Section 2.10.6.2. It isn't necessary to retry requests over a connection with the same source network address or the same destination network address as the lost connection. As long as the session ID, slot ID, and sequence ID in the retry match those of the original request, the callback target will recognize the request as a retry, even if it did see the request before the disconnect 🔄

If the lost connection is the last one associated with the backchannel, then the server MUST indicate that in the sr_status_flags field of every SEQUENCE reply until the backchannel is re-established. There are two situations, each of which uses different status flags: no connectivity for the session's backchannel, and no connectivity for any session backchannel of the client. See Section 18.46 for a description of the appropriate flags in sr_status_flags 🚩

2.10.13.2.5. GSS Context Loss

The server SHOULD monitor when the number of RPCSEC_GSS handles assigned to the backchannel reaches one, and when that one handle is near expiry (i.e., between one and two periods of lease time), and indicate so in the sr_status_flags field of all SEQUENCE replies. The server MUST indicate when all of the backchannel's assigned RPCSEC_GSS handles have expired via the sr_status_flags field of all SEQUENCE replies. Watch those handles, babe 👀

---

2.10.14. Parallel NFS and Sessions 🧚

A client and server can potentially be a non-pNFS implementation, a metadata server implementation, a data server implementation, or two or three types of implementations at once. The EXCHGID4_FLAG_USE_NON_PNFS, EXCHGID4_FLAG_USE_PNFS_MDS, and EXCHGID4_FLAG_USE_PNFS_DS flags (not mutually exclusive) are passed in the EXCHANGE_ID arguments and results. They let the client say how it wants to use sessions created under the client ID, and let the server say how it will allow the sessions to be used. See Section 13.1 for pNFS sessions considerations 💖

3. Protocol Constants and Data Types 🎀

The syntax and semantics used to describe the data types of the NFSv4.1 protocol are defined in the XDR RFC 4506 [2] and RPC RFC 5531 [3] documents. The next sections build upon the XDR data types to define constants, types, and structures specific to this protocol. The full list of XDR data types is in [13] 🌸

3.1. Basic Constants ✨

okay so first up are the basic constants 🧚 these are the fixed numbers everything else in NFSv4.1 leans on, so they're kind of the foundation of the whole vibe 💅 they're all listed in a code block like this:

   const NFS4_FHSIZE               = 128;
   const NFS4_VERIFIER_SIZE        = 8;
   const NFS4_OPAQUE_LIMIT         = 1024;
   const NFS4_SESSIONID_SIZE       = 16;

   const NFS4_INT64_MAX            = 0x7fffffffffffffff;
   const NFS4_UINT64_MAX           = 0xffffffffffffffff;
   const NFS4_INT32_MAX            = 0x7fffffff;
   const NFS4_UINT32_MAX           = 0xffffffff;

   const NFS4_MAXFILELEN           = 0xffffffffffffffff;
   const NFS4_MAXFILEOFF           = 0xfffffffffffffffe;

unless it says otherwise, every one of these is measured in bytes 🎀

• NFS4_FHSIZE is the maximum size of a filehandle. 📁
• NFS4_VERIFIER_SIZE is the fixed size of a verifier. 🔐
• NFS4_OPAQUE_LIMIT is the maximum size of certain opaque information. 🫧
• NFS4_SESSIONID_SIZE is the fixed size of a session identifier. 🪪
• NFS4_INT64_MAX is the maximum value of a signed 64-bit integer. 🔢
• NFS4_UINT64_MAX is the maximum value of an unsigned 64-bit integer. 🔢
• NFS4_INT32_MAX is the maximum value of a signed 32-bit integer. 🔢
• NFS4_UINT32_MAX is the maximum value of an unsigned 32-bit integer. 🔢
• NFS4_MAXFILELEN is the maximum length of a regular file. 📄
• NFS4_MAXFILEOFF is the maximum offset into a regular file. 📍

---

3.2. Basic Data Types 🍓

these are the base NFSv4.1 data types, the building blocks that every other structure is made from. it's giving LEGO bricks honestly 🧱 the table is kept exactly as it is:

+---------------+---------------------------------------------------+
| Data Type     | Definition                                        |
+---------------+---------------------------------------------------+
| int32_t       | typedef int int32_t;                              |
| uint32_t      | typedef unsigned int uint32_t;                    |
| int64_t       | typedef hyper int64_t;                            |
| uint64_t      | typedef unsigned hyper uint64_t;                  |
| attrlist4     | typedef opaque attrlist4<>;                       |
|               | Used for file/directory attributes.               |
| bitmap4       | typedef uint32_t bitmap4<>;                       |
|               | Used in attribute array encoding.                 |
| changeid4     | typedef uint64_t changeid4;                       |
|               | Used in the definition of change_info4.           |
| clientid4     | typedef uint64_t clientid4;                       |
|               | Shorthand reference to client identification.     |
| count4        | typedef uint32_t count4;                          |
|               | Various count parameters (READ, WRITE, COMMIT).   |
| length4       | typedef uint64_t length4;                         |
|               | The length of a byte-range within a file.         |
| mode4         | typedef uint32_t mode4;                           |
|               | Mode attribute data type.                         |
| nfs_cookie4   | typedef uint64_t nfs_cookie4;                     |
|               | Opaque cookie value for READDIR.                  |
| nfs_fh4       | typedef opaque nfs_fh4<NFS4_FHSIZE>;              |
|               | Filehandle definition.                            |
| nfs_ftype4    | enum nfs_ftype4;                                  |
|               | Various defined file types.                       |
| nfsstat4      | enum nfsstat4;                                    |
|               | Return value for operations.                      |
| offset4       | typedef uint64_t offset4;                         |
|               | Various offset designations (READ, WRITE, LOCK,   |
|               | COMMIT).                                          |
| qop4          | typedef uint32_t qop4;                            |
|               | Quality of protection designation in SECINFO.     |
| sec_oid4      | typedef opaque sec_oid4<>;                        |
|               | Security Object Identifier.  The sec_oid4 data    |
|               | type is not really opaque.  Instead, it contains  |
|               | an ASN.1 OBJECT IDENTIFIER as used by GSS-API in  |
|               | the mech_type argument to GSS_Init_sec_context.   |
|               | See [7] for details.                              |
| sequenceid4   | typedef uint32_t sequenceid4;                     |
|               | Sequence number used for various session          |
|               | operations (EXCHANGE_ID, CREATE_SESSION,          |
|               | SEQUENCE, CB_SEQUENCE).                           |
| seqid4        | typedef uint32_t seqid4;                          |
|               | Sequence identifier used for locking.             |
| sessionid4    | typedef opaque sessionid4[NFS4_SESSIONID_SIZE];   |
|               | Session identifier.                               |
| slotid4       | typedef uint32_t slotid4;                         |
|               | Sequencing artifact for various session           |
|               | operations (SEQUENCE, CB_SEQUENCE).               |
| utf8string    | typedef opaque utf8string<>;                      |
|               | UTF-8 encoding for strings.                       |
| utf8str_cis   | typedef utf8string utf8str_cis;                   |
|               | Case-insensitive UTF-8 string.                    |
| utf8str_cs    | typedef utf8string utf8str_cs;                    |
|               | Case-sensitive UTF-8 string.                      |
| utf8str_mixed | typedef utf8string utf8str_mixed;                 |
|               | UTF-8 strings with a case-sensitive prefix and a  |
|               | case-insensitive suffix.                          |
| component4    | typedef utf8str_cs component4;                    |
|               | Represents pathname components.                   |
| linktext4     | typedef utf8str_cs linktext4;                     |
|               | Symbolic link contents ("symbolic link" is        |
|               | defined in an Open Group [14] standard).          |
| pathname4     | typedef component4 pathname4<>;                   |
|               | Represents pathname for fs_locations.             |
| verifier4     | typedef opaque verifier4[NFS4_VERIFIER_SIZE];     |
|               | Verifier used for various operations (COMMIT,     |
|               | CREATE, EXCHANGE_ID, OPEN, READDIR, WRITE)        |
|               | NFS4_VERIFIER_SIZE is defined as 8.               |
+---------------+---------------------------------------------------+

                          End of Base Data Types

                                  Table 1

---

3.3. Structured Data Types 🎀

3.3.1. nfstime4 🕰️

   struct nfstime4 {
           int64_t         seconds;
           uint32_t        nseconds;
   };

okay so nfstime4 is how we count time, bestie ⏰ it's seconds plus nanoseconds, counted from midnight (or zero hour) on January 1, 1970 in UTC. positive seconds mean dates after that moment, negative seconds mean dates before it. the nseconds part just gets added on top of the seconds for the final time. so if you want half a second before 1970, seconds is -1 and nseconds is 500000000 🌙 anything over 999,999,999 for nseconds is invalid, no cap 🚫

this type is what carries time and date info around. the server converts to and from its own local time representation, keeping as much accuracy as it can. if a file system stores timestamps with less precision than this, some precision gets lost, which is lowkey unavoidable 🥲 an adjunct time maintenance protocol is RECOMMENDED to reduce client and server time skew, because clocks drifting apart is a whole mess 🫠

3.3.2. time_how4 ⏳

   enum time_how4 {
           SET_TO_SERVER_TIME4 = 0,
           SET_TO_CLIENT_TIME4 = 1
   };

3.3.3. settime4 ⌚

   union settime4 switch (time_how4 set_it) {
    case SET_TO_CLIENT_TIME4:
            nfstime4       time;
    default:
            void;
   };

time_how4 and settime4 are the types you use to set timestamps on file object attributes. if set_it is SET_TO_SERVER_TIME4, the server just grabs its own local time for the value 🖥️ otherwise the client supplies the time itself.

3.3.4. specdata4 🔌

   struct specdata4 {
    uint32_t specdata1; /* major device number */
    uint32_t specdata2; /* minor device number */
   };

this one holds the device numbers for the device file types NF4CHR and NF4BLK 💫

3.3.5. fsid4 🆔

   struct fsid4 {
           uint64_t        major;
           uint64_t        minor;
   };

3.3.6. change_policy4 🔄

   struct change_policy4 {
           uint64_t        cp_major;
           uint64_t        cp_minor;
   };

the change_policy4 type is used for the change_policy RECOMMENDED attribute. it does the same job as the change attribute, telling you when something has changed 👀 here's the clever bit: it has to keep working even after the server restarts, without needing any persistent storage. so it uses two 64-bit numbers. one can be a server instance ID, and the other gets incremented without being saved, but only within that one server instance. iconic engineering, honestly 👑

3.3.7. fattr4 📂

   struct fattr4 {
           bitmap4         attrmask;
           attrlist4       attr_vals;
   };

fattr4 is what represents file and directory attributes 🗂️

the bitmap is a counted array of 32-bit integers that holds bit values. to find which integer in the array holds bit n, you compute (n / 32), and its position inside that integer is (n mod 32) 🧮 picture it like this:

   0            1
   +-----------+-----------+-----------+--
   |  count    | 31  ..  0 | 63  .. 32 |
   +-----------+-----------+-----------+--

3.3.8. change_info4 📝

   struct change_info4 {
           bool            atomic;
           changeid4       before;
           changeid4       after;
   };

this data type is used with the CREATE, LINK, OPEN, REMOVE, and RENAME operations 💅 it's how the client finds out the change attribute value for the directory where the target object lives. main character energy for directories, no cap ✨

3.3.9. netaddr4 🌐

   struct netaddr4 {
           /* see struct rpcb in RFC 1833 */
           string na_r_netid<>; /* network id */
           string na_r_addr<>;  /* universal address */
   };

netaddr4 is used to identify network transport endpoints 🛰️ its r_netid and r_addr fields hold a netid and a uaddr respectively. both concepts are defined in [15]. the netid and uaddr formats for TCP over IPv4 and TCP over IPv6 live in [15] too, specifically Tables 2 and 3 and Sections 5.2.3.3 and 5.2.3.4 📚

3.3.10. state_owner4 🫶

   struct state_owner4 {
           clientid4       clientid;
           opaque          owner<NFS4_OPAQUE_LIMIT>;
   };

   typedef state_owner4 open_owner4;
   typedef state_owner4 lock_owner4;

state_owner4 is the base type for open_owner4 (Section 3.3.10.1) and lock_owner4 (Section 3.3.10.2). it's the shared foundation, like a group chat that both sub-chats branch off of 💜

3.3.10.1. open_owner4 🔓

this data type is used to identify the owner of OPEN state 🗝️

3.3.10.2. lock_owner4 🔐

this structure is used to identify the owner of byte-range locking state 🧷

3.3.11. open_to_lock_owner4 🔐

struct open_to_lock_owner4 {
        seqid4          open_seqid;
        stateid4        open_stateid;
        seqid4          lock_seqid;
        lock_owner4     lock_owner;
};

okay so this one is for the first LOCK an open_owner4 does 💖 It bundles the open_stateid and the lock_owner together so the transition goes from a valid open_stateid sequence over to the new lock_stateid sequence. The best part is that it skips confirming the lock_owner/lock_seqid pair, because that's already tied to established state in the form of open_stateid/open_seqid. no cap, that's so efficient 👑

3.3.12. stateid4 🦋

struct stateid4 {
        uint32_t        seqid;
        opaque          other[12];
};

this type is used for the different state sharing mechanisms between client and server ✨ The client never modifies a value of stateid, so hands off bestie 🙈 The starting value of the "seqid" field is undefined. The server is required to increment the "seqid" field by one at each transition of the stateid. This matters a LOT because the client looks at the seqid in OPEN stateids to figure out the order the server processed OPENs in. main character energy for sequence numbers 💅

3.3.13. layouttype4 🌸

enum layouttype4 {
        LAYOUT4_NFSV4_1_FILES   = 0x1,
        LAYOUT4_OSD2_OBJECTS    = 0x2,
        LAYOUT4_BLOCK_VOLUME    = 0x3
};

this one says what kind of layout is being used 🎀 The file server advertises which layout types it supports through the fs_layout_type file system attribute (Section 5.12.1). A client asks for layouts of a particular type in LAYOUTGET, then handles those layouts with its layout-type-specific logic.

The layouttype4 data type is 32 bits long, and the range is split into three parts. Type 0x0 is reserved. Types in the range 0x00000001-0x7FFFFFFF are globally unique and get assigned according to the description in Section 22.4; they're maintained by IANA. Types in the range 0x80000000-0xFFFFFFFF are site specific and for private use only 🔒

The LAYOUT4_NFSV4_1_FILES enumeration says the NFSv4.1 file layout type, as defined in Section 13, is the one to use. LAYOUT4_OSD2_OBJECTS says the object layout, as defined in [40], is the one. And LAYOUT4_BLOCK_VOLUME says the block/volume layout, as defined in [41], is the one 🧚

3.3.14. deviceid4 🧚

const NFS4_DEVICEID4_SIZE = 16;

typedef opaque  deviceid4[NFS4_DEVICEID4_SIZE];

layout info includes device IDs, which point at a storage device through a compact handle ✨ You get the addressing and type info with the GETDEVICEINFO operation. Device IDs are not guaranteed to stay valid across metadata server restarts, so don't get too attached 🥲 A device ID is unique per client ID and layout type. See Section 12.2.10 for more details.

3.3.15. device_addr4 🌈

struct device_addr4 {
        layouttype4             da_layout_type;
        opaque                  da_addr_body<>;
};

the device address is what sets up a communication channel with the storage device 🦋 Different layout types need different data types to define how they talk to storage devices. The opaque da_addr_body field gets interpreted based on the specified da_layout_type field.

This document defines the device address for the NFSv4.1 file layout (see Section 13.3), which identifies a storage device by network IP address and port number. That's enough for clients to talk to NFSv4.1 storage devices, and it may be enough for other layout types too. Device types for object-based storage devices and block storage devices (e.g., Small Computer System Interface (SCSI) volume labels) are defined by their own respective layout specifications 💜

3.3.16. layout_content4 🍓

struct layout_content4 {
        layouttype4 loc_type;
        opaque      loc_body<>;
};

the loc_body field gets interpreted based on the layout type (loc_type) ✨ This document defines the loc_body for the NFSv4.1 file layout type, so check Section 13.3 for the definition.

3.3.17. layout4 🫶

struct layout4 {
        offset4                 lo_offset;
        length4                 lo_length;
        layoutiomode4           lo_iomode;
        layout_content4         lo_content;
};

the layout4 data type defines a layout for a file 🎀 The layout type specific data is opaque within lo_content. Since layouts can be sub-divided, the offset and length together with the file's filehandle, the client ID, iomode, and layout type are what identify the layout. it's giving puzzle pieces 🧩

3.3.18. layoutupdate4 💌

struct layoutupdate4 {
        layouttype4             lou_type;
        opaque                  lou_body<>;
};

the client uses layoutupdate4 to send updated layout info back to the metadata server through the LAYOUTCOMMIT (Section 18.42) operation ✨ This type is a channel for passing layout type specific info (in lou_body) back to the metadata server. For example, with the block/volume layout type, this could include the list of reserved blocks that were written. The contents of the opaque lou_body argument are decided by the layout type. The NFSv4.1 file-based layout does not use this data type, so if lou_type is LAYOUT4_NFSV4_1_FILES, the lou_body field MUST have a zero length. strict rule, no exceptions 💀

3.3.19. layouthint4 💜

struct layouthint4 {
        layouttype4             loh_type;
        opaque                  loh_body<>;
};

the client uses layouthint4 to pass in a hint about what type of layout it'd like created for a particular file 🌸 It's the data type specified by the layout_hint attribute described in Section 5.12.4. The metadata server may ignore the hint, or it may selectively ignore fields within it. This hint should be provided at create time as part of the initial attributes within OPEN. The loh_body field is specific to the type of layout (loh_type). The NFSv4.1 file-based layout uses the nfsv4_1_file_layouthint4 data type, as defined in Section 13.3.

3.3.20. layoutiomode4 🦋

enum layoutiomode4 {
        LAYOUTIOMODE4_READ      = 1,
        LAYOUTIOMODE4_RW        = 2,
        LAYOUTIOMODE4_ANY       = 3
};

the iomode says whether the client just wants to read, or wants to both read and write the data the layout covers 👑 LAYOUTIOMODE4_ANY is a no-go in the arguments to LAYOUTGET, since it MUST NOT be used there 🙅 But it MAY be used in the arguments to LAYOUTRETURN and CB_LAYOUTRECALL. In those spots, LAYOUTIOMODE4_ANY means layouts for both LAYOUTIOMODE4_READ and LAYOUTIOMODE4_RW iomodes are being returned or recalled, respectively. How the metadata server uses the iomode can depend on the layout type. The storage devices MAY validate I/O accesses against the iomode and reject invalid accesses 🔥

3.3.21. nfs_impl_id4 🎀

struct nfs_impl_id4 {
        utf8str_cis   nii_domain;
        utf8str_cs    nii_name;
        nfstime4      nii_date;
};

this one identifies client and server implementation details ✨ The nii_domain field is the DNS domain name the implementor is associated with. The nii_name field is the product name of the implementation, and it's totally free form. It is RECOMMENDED that nii_name be used to distinguish machine architecture, machine platforms, revisions, versions, and patch levels 🌈 The nii_date field is the timestamp for when the software instance was published or built.

3.3.22. threshold_item4 🍓

struct threshold_item4 {
        layouttype4     thi_layout_type;
        bitmap4         thi_hintset;
        opaque          thi_hintlist<>;
};

this data type holds a list of hints for a specific layout type, helping the client decide when to send I/O directly through the metadata server versus the storage devices 🧚 It's made up of the layout type (thi_layout_type), a bitmap (thi_hintset) describing the set of hints the server supports (these can differ by layout type), and a list of hints (thi_hintlist) whose content is determined by the hintset bitmap. See the mdsthreshold attribute for more details.

The thi_hintset field is a bitmap of the following values:

+-------------------------+---+---------+---------------------------+
| name                    | # | Data    | Description               |
|                         |   | Type    |                           |
+-------------------------+---+---------+---------------------------+
| threshold4_read_size    | 0 | length4 | If a file's length is     |
|                         |   |         | less than the value of    |
|                         |   |         | threshold4_read_size,     |
|                         |   |         | then it is RECOMMENDED    |
|                         |   |         | that the client read from |
|                         |   |         | the file via the MDS and  |
|                         |   |         | not a storage device.     |
| threshold4_write_size   | 1 | length4 | If a file's length is     |
|                         |   |         | less than the value of    |
|                         |   |         | threshold4_write_size,    |
|                         |   |         | then it is RECOMMENDED    |
|                         |   |         | that the client write to  |
|                         |   |         | the file via the MDS and  |
|                         |   |         | not a storage device.     |
| threshold4_read_iosize  | 2 | length4 | For read I/O sizes below  |
|                         |   |         | this threshold, it is     |
|                         |   |         | RECOMMENDED to read data  |
|                         |   |         | through the MDS.          |
| threshold4_write_iosize | 3 | length4 | For write I/O sizes below |
|                         |   |         | this threshold, it is     |
|                         |   |         | RECOMMENDED to write data |
|                         |   |         | through the MDS.          |
+-------------------------+---+---------+---------------------------+

3.3.23. mdsthreshold4 💅

struct mdsthreshold4 {
        threshold_item4 mth_hints<>;
};

this data type holds an array of threshold_item4 elements, each valid for a particular layout type 🌸 An array is needed because a server can support multiple layout types for a single file.

4. Filehandles ✨

the filehandle in the NFS protocol is a per-server unique identifier for a file system object 🔑 The contents of the filehandle are opaque to the client. So the server is responsible for translating the filehandle into an internal representation of the file system object. the server does all the heavy lifting here 🫶

4.1. Obtaining the First Filehandle

okay so every NFS operation is defined in terms of one or more filehandles 📁, which means the client needs a filehandle before it can even start talking to the server 💅. With NFSv3 (RFC 1813 [31]), there's a sidekick protocol that gets this first filehandle for you. The MOUNT protocol, RPC program number 100005, turns a string-based file system pathname into a filehandle that the NFS protocols can then use 🎀.

The MOUNT protocol has security problems and doesn't play nicely with firewalls 🙈. This is one reason the public filehandle was introduced in RFC 2054 [42] and RFC 2055 [43]. Using the public filehandle together with the LOOKUP operation in NFSv3, it was shown that MOUNT isn't needed for a workable interaction between NFS client and server ✨.

So NFSv4.1 is not going to use a sidekick protocol to turn string-based pathnames into filehandles 🙅‍♀️. Instead, there are two special filehandles that act as starting points for the NFS client 🌸.

4.1.1. Root Filehandle

The first special filehandle is the ROOT filehandle 🌳. It's the "conceptual" root of the file system namespace at the NFS server. The client uses or starts with the ROOT filehandle by employing the PUTROOTFH operation 🧚. PUTROOTFH tells the server to set the "current" filehandle to the ROOT of the server's file tree. Once PUTROOTFH has been used, the client can traverse the entire server file tree with the LOOKUP operation 👑. A full discussion of the server namespace is in Section 7.

4.1.2. Public Filehandle

The second special filehandle is the PUBLIC filehandle 💜. Unlike ROOT, the PUBLIC filehandle may be bound to, or represent, an arbitrary file system object at the server. The server is responsible for this binding 🫶. It's possible that the PUBLIC and ROOT filehandles refer to the same file system object. However, it's up to the administrative software at the server and the policies of the server administrator to define how the PUBLIC filehandle is bound to a server file system object. The client may not make any assumptions about this binding 🙈. The client uses the PUBLIC filehandle via the PUTPUBFH operation.

4.2. Filehandle Types

In NFSv3 there was one type of filehandle with a single set of semantics 🍓. That type is called "persistent" in NFSv4.1, and its semantics stay exactly the same as before ✨. NFSv4.1 introduces a new type called the "volatile" filehandle, which is there to accommodate certain server environments 🦋.

The volatile filehandle type was added to deal with server functionality or implementation issues that make a correct persistent filehandle implementation infeasible 💀. Some server environments don't provide a file-system-level invariant that can be used to build a persistent filehandle. The underlying file system might not provide the invariant, or the server's file system programming interfaces might not give access to it 😭. Volatile filehandles can make things like hierarchical storage management or file system reorganization or migration easier to implement for the server 🔥. The catch is that they make life harder for the client 💅.

Since the client has to handle persistent and volatile filehandles differently, there's a file attribute that the client can use to figure out which filehandle types the server is returning 🌈.

4.2.1. General Properties of a Filehandle

The filehandle contains all the information the server needs to tell one individual file apart from another 🔍. To the client, the filehandle is opaque 🎀. The client stores filehandles for use in later requests, and it can compare two filehandles from the same server for equality by doing a byte-by-byte comparison ✨. However, the client MUST NOT otherwise interpret the contents of filehandles 🙅‍♀️. If two filehandles from the same server are equal, they MUST refer to the same file 💖. Servers SHOULD try to maintain a one-to-one correspondence between filehandles and files, but this is not required 🦋. Clients MUST use filehandle comparisons only to improve performance, not for correct behavior 🫶. All clients need to be prepared for situations in which it can't be determined whether two filehandles denote the same object, and in those cases they should avoid making invalid assumptions that might cause incorrect behavior 😭. More on filehandle and attribute comparison in the context of data caching is in Section 10.3.4.

For example, if two different pathnames end up at the same file system object when traversed at the server, the server SHOULD return the same filehandle for each path 🧚. This can happen if a hard link (see [6]) is used to create two file names that refer to the same underlying file object and associated data 🌸. So if paths /a/b/c and /a/d/c refer to the same file, the server SHOULD return the same filehandle for both pathnames' traversals 💅.

4.2.2. Persistent Filehandle

A persistent filehandle has a fixed value for the lifetime of the file system object it refers to 🔒. Once the server creates the filehandle for a file system object, the server MUST accept the same filehandle for that object for the lifetime of the object 👑. If the server restarts, the NFS server MUST honor the same filehandle value it used in its previous instantiation 🔥. Likewise, if the file system is migrated, the new NFS server MUST honor the same filehandle as the old NFS server 🌈.

A persistent filehandle becomes stale or invalid when its file system object is removed 💀. When the server gets a persistent filehandle that refers to a deleted object, it MUST return an error of NFS4ERR_STALE 🙈. A filehandle may also become stale when the file system containing the object is no longer available. That can happen if the file system lives on removable media and the media is no longer available at the server, if the file system as a whole has been destroyed, or if the file system has simply been removed from the server's namespace (in a UNIX environment, that's unmounting) 😭.

4.2.3. Volatile Filehandle

A volatile filehandle does not have the same longevity as a persistent filehandle 🦋. The server may decide that a volatile filehandle is no longer valid at many different points in time 🫠. If the server can definitively determine that a volatile filehandle refers to a removed object, the server should return NFS4ERR_STALE to the client, just like it does for persistent filehandles 💅. In every other case where the server decides a volatile filehandle can't be used anymore, it should return an error of NFS4ERR_FHEXPIRED ✨.

The REQUIRED attribute "fh_expire_type" is how the client finds out what type of filehandle the server is providing for a particular file system 🧚. This attribute is a bitmask with the following values 🎀:

• FH4_PERSISTENT: this value indicates a persistent filehandle, which is valid until the object is removed from the file system 💖. The server will not return NFS4ERR_FHEXPIRED for this filehandle. FH4_PERSISTENT is defined as a value in which none of the bits specified below are set 🌸.
• FH4_VOLATILE_ANY: the filehandle may expire at any time, except as specifically excluded (for example, FH4_NO_EXPIRE_WITH_OPEN) 🦋.
• FH4_NOEXPIRE_WITH_OPEN: this may only be set when FH4_VOLATILE_ANY is set 🔒. When it's set, the meaning of FH4_VOLATILE_ANY is qualified to exclude any expiration of the filehandle while it's open 🫶.
• FH4_VOL_MIGRATION: the filehandle will expire as a result of a file system transition (migration or replication), in cases where the continuity of filehandle use isn't specified by handle class information within the fs_locations_info attribute 🌈. When this bit is set, clients without access to fs_locations_info information should assume that filehandles will expire on file system transitions 🙈.
• FH4_VOL_RENAME: the filehandle will expire during a rename 🔥. That includes a rename by the requesting client or a rename by any other client. If FH4_VOL_ANY is set, FH4_VOL_RENAME is redundant 💅.

Servers that provide volatile filehandles that can expire while open need special care when it comes to RENAMEs and REMOVEs 😭. This situation can come up if FH4_VOL_MIGRATION or FH4_VOL_RENAME is set, if FH4_VOLATILE_ANY is set and FH4_NOEXPIRE_WITH_OPEN is not set, or if a non-read-only file system has a transition target in a different handle class 💀. In these cases, the server should deny a RENAME or REMOVE that would affect an OPEN file of any of the components leading to the OPEN file 🙅‍♀️. The server should also deny all RENAME or REMOVE requests during the grace period, so that reclaims of files whose filehandles may have expired don't reclaim the wrong file 🫠.

Volatile filehandles are especially well suited to implementing the pseudo file systems used to bridge exports 🌉. See Section 7.5 for a discussion of this ✨.

4.3. One Method of Constructing a Volatile Filehandle

A volatile filehandle is opaque to the client, but it could contain something like this 🧩:

[volatile bit = 1 | server boot time | slot | generation number]

• slot is an index in the server volatile filehandle table 🗂️
• generation number is the generation number for the table entry/slot 🔢

When the client presents a volatile filehandle, the server makes the following checks, which assume the check for the volatile bit has passed 🕵️‍♀️. If the server boot time is less than the current server boot time, the server returns NFS4ERR_FHEXPIRED. If slot is out of range, the server returns NFS4ERR_BADHANDLE. If the generation number doesn't match, the server returns NFS4ERR_FHEXPIRED 💖.

When the server restarts, the table is gone, because it's volatile 💨.

If the volatile bit is 0, then it's a persistent filehandle with a different structure following it 🌈.

4.4. Client Recovery from Filehandle Expiration

okay so if it's possible, the client SHOULD recover when it gets an NFS4ERR_FHEXPIRED error 💅 that's the move. The client must take on extra responsibility so it can get ready to recover from a volatile filehandle expiring 😭 If the server returns persistent filehandles, the client doesn't need these extra steps ✨

for volatile filehandles, the big thing is that the client will usually need to store the component names leading up to and including the file system object in question 🎀 With those names, the client should be able to recover by finding a filehandle in the namespace that's still available, or by starting at the root of the server's file system namespace 🌸

if the expired filehandle points to an object that's been removed from the file system, obviously the client won't be able to recover from it 🥲

it's also possible that the expired filehandle refers to a file that got renamed 🦋 if another client did the renaming, the original client might not be able to recover, sadly. BUT if the client itself is renaming the file and the file is open, it may be able to recover! the client can figure out the new pathname from how it processed the rename request, and then regenerate the new filehandle from that pathname 🧚 the client could also use the COMPOUND procedure to line up a series of operations like this:

             RENAME A B
             LOOKUP B
             GETFH

ngl, COMPOUND does NOT give you atomicity 🚨 this example just cuts down on the overhead of recovering from an expired filehandle, that's it.

---

5. File Attributes ✨

okay so attributes need to be handled in a flexible way, because the whole point is extensibility and playing nicely with non-UNIX platforms 🌈 the NFSv3 fattr3 structure has a fixed list of attributes that not every client and server can support or even cares about 💀 it can't be extended as new needs pop up, and it has no way to say "not supported." with NFSv4.1, the client can ask what attributes the server supports and build requests with only those (or a subset of them) 👑

so attributes are split into three groups: REQUIRED, RECOMMENDED, and named 🍓 both REQUIRED and RECOMMENDED attributes have a specific, well-defined encoding in NFSv4.1 and are identified by number. you request them by setting a bit in the bit vector sent in the GETATTR request, and the server's response has a bit vector listing which attributes actually came back 💜 new REQUIRED or RECOMMENDED attributes can be added as part of a new minor version, by publishing a Standards Track RFC that allocates a new attribute number and defines its encoding. see Section 2.7 for more on that 🔥

named attributes are accessed with the new OPENATTR operation, which opens a hidden directory of attributes attached to a file system object 🫶 OPENATTR takes a filehandle for the object and returns a filehandle for the attribute hierarchy. that filehandle is a directory you can LOOKUP or READDIR, and it contains files whose names are the named attributes and whose data bytes are the attribute's value ✨ for example:

        +----------+-----------+---------------------------------+
        | LOOKUP   | "foo"     | ; look up file                  |
        | GETATTR  | attrbits  |                                 |
        | OPENATTR |           | ; access foo's named attributes |
        | LOOKUP   | "x11icon" | ; look up specific attribute    |
        | READ     | 0,4096    | ; read stream of bytes          |
        +----------+-----------+---------------------------------+

named attributes are meant for data that applications need, rather than data an NFS client implementation needs 🧁 NFS implementors are strongly encouraged to define their new attributes as RECOMMENDED attributes by taking them through the IETF Standards Track process 🌟

the REQUIRED set is deliberately small, because servers have to do whatever it takes to support those 🫠 a server should support as many RECOMMENDED attributes as it possibly can, but by definition it isn't required to support all of them. an attribute only gets REQUIRED status if the data is needed by a large number of clients AND the client can't reasonably compute it on its own when the server doesn't provide it 🍒

quick note: the hidden directory from OPENATTR is just a convenience for protocol processing 🙈 clients shouldn't assume anything about how the server implements named attributes, or whether the underlying file system has a named attribute directory at all. so operations like SETATTR and GETATTR on the named attribute directory are undefined 🤷‍♀️

5.1. REQUIRED Attributes

these MUST be supported by every NFSv4.1 client and server, so there's a minimum level of interoperability 💖 the server MUST store and return these attributes, and the client MUST be able to function with an attribute set limited to just these. yes, some client functionality may be impaired or limited with only the REQUIRED attributes, but it works 🎀 a client can ask for any of these by setting a bit in the GETATTR request, and the server MUST return their value 🫶

5.2. RECOMMENDED Attributes

these are understood well enough to deserve support in NFSv4.1 🌸 but they might not be supported on every client and server, so a client can ask for any of them (by setting a bit in GETATTR) but MUST handle the case where the server doesn't return them. wait, sorry, let me be precise here: a client that asks for them must handle the case where the server doesn't return them 💅 a client MAY ask for the set of attributes the server supports, and it SHOULD NOT request attributes the server doesn't support 🙅‍♀️

a server should be tolerant of requests for unsupported attributes and just not return them, rather than treating the request as an error 🥰 servers are expected to support every attribute they comfortably can, and only skip the ones that are really hard to support in their operating environment 🧚 a server should provide attributes whenever it doesn't have to "tell lies" to the client. for example, a file modification time should either be an accurate time or not be supported at all ⏰ this can be tough for clients, but a client is better positioned to decide whether and how to fabricate or construct an attribute, or whether to just do without it 🦋

5.3. Named Attributes

these attributes aren't directly encoded in the NFSv4 protocol. instead they're accessed by string names rather than numbers, and each one is an uninterpreted stream of bytes stored with the file system object 🌈 the namespace can be accessed with the OPENATTR operation, which returns a filehandle for a virtual "named attribute directory." after that, you can poke around and change the namespace with operations that work on normal directories. specifically, READDIR can list the named attributes, and LOOKUP and OPEN can select a particular attribute. creating a new named attribute can happen as the result of an OPEN that specifies file creation ✨

once you've done an OPEN, named attributes can be examined and changed with normal READ and WRITE operations, using the filehandles and stateids that OPEN returned 💜

named attributes and the named attribute directory can have their own (non-named) attributes too 🍓 each of these objects MUST have all of the REQUIRED attributes and may have additional RECOMMENDED attributes. the set of attributes for named attributes and the named attribute directory doesn't need to be as big as for other objects in that file system, and typically won't be 🫠

named attributes and the named attribute directory might be the target of delegations (for the named attribute directory, these will be directory delegations) 🔥 but since granting delegations is up to the server, a server doesn't need to support delegations on named attributes or the named attribute directory 🤷‍♀️

It is RECOMMENDED that servers support arbitrary named attributes 💖 a client should not depend on being able to store any named attributes in the server's file system. if a server does support named attributes, a client that can also handle them should be able to copy a file's data and metadata from one location to another with complete transparency. that would imply names allowed for regular directory entries are valid for named attribute names too 👑

in NFSv4.1, the structure of named attribute directories is restricted in several ways 🛡️ this is so we don't end up with non-interoperable implementations, where some servers support a fully general hierarchical directory structure for named attributes and others support a limited but adequate one. in that kind of environment, clients or applications might start depending on non-portable extensions 😬 the restrictions are:

• CREATE is not allowed in a named attribute directory. so objects like symbolic links and special files aren't allowed to be named attributes. also, directories can't be created in a named attribute directory, so no hierarchical structure of named attributes for a single object is allowed 🙅‍♀️
• If OPENATTR is done on a named attribute directory or on a named attribute, the server MUST return NFS4ERR_WRONG_TYPE 🚫
• Doing a RENAME of a named attribute to a different named attribute directory, or to an ordinary (i.e., non-named-attribute) directory, is not allowed 🙈
• Creating hard links between named attribute directories, or between named attribute directories and ordinary directories, is not allowed 🧚

names of attributes won't be controlled by this document or other IETF Standards Track documents. see Section 22.1 for further discussion 🎀

5.4. Classification of Attributes ✨

Okay bestie, so every REQUIRED and RECOMMENDED attribute falls into one of three buckets 🪣💅. The per-server bucket holds values that stay the same for every file object sharing the same server owner (check Section 2.5 for what a server owner is 🫶). The per file system bucket holds values that stay the same for some or all file objects sharing the same fsid attribute (Section 5.8.1.9) and server owner 🌈. The last bucket is per file system object, which is the most individual one 🦋.

Here's the tea ☕: some per file system attributes can still vary inside a single file system, depending on the homogeneous attribute (Section 5.8.2.16) 👀. Also, time_access_set and time_modify_set aren't listed here at all 🙈 because they're write-only versions of time_access and time_modify, and they only show up in a special instance of SETATTR.

• The per-server attribute is:
  • lease_time
• The per-file system attributes are:
  • supported_attrs, suppattr_exclcreat, fh_expire_type, link_support, symlink_support, unique_handles, aclsupport, cansettime, case_insensitive, case_preserving, chown_restricted, files_avail, files_free, files_total, fs_locations, homogeneous, maxfilesize, maxname, maxread, maxwrite, no_trunc, space_avail, space_free, space_total, time_delta, change_policy, fs_status, fs_layout_type, fs_locations_info, fs_charset_cap
• The per-file system object attributes are:
  • type, change, size, named_attr, fsid, rdattr_error, filehandle, acl, archive, fileid, hidden, maxlink, mimetype, mode, numlinks, owner, owner_group, rawdev, space_used, system, time_access, time_backup, time_create, time_metadata, time_modify, mounted_on_fileid, dir_notif_delay, dirent_notif_delay, dacl, sacl, layout_type, layout_hint, layout_blksize, layout_alignment, mdsthreshold, retention_get, retention_set, retentevt_get, retentevt_set, retention_hold, mode_set_masked

For quota_avail_hard, quota_avail_soft, and quota_used, check out their own definitions below for the right classification 📚🎀.

5.5. Set-Only and Get-Only Attributes 🙌

Some REQUIRED and RECOMMENDED attributes are set-only 🔧, which means you can set them with SETATTR but you can't read them back with GETATTR. On the flip side, some REQUIRED and RECOMMENDED attributes are get-only 👀, so GETATTR can retrieve them but SETATTR can't change them. If a client tries to set a get-only attribute or get a set-only attribute, the server MUST return NFS4ERR_INVAL 🚫. Nope, not allowed!

5.6. REQUIRED Attributes - List and Definition References 📋

The list of REQUIRED attributes lives in Table 2 🧾. Here's what each column means:

• Name: the name of the attribute 🏷️.
• Id: the number assigned to the attribute. If the assigned number ever clashes with [13], the latter is likely the one that counts 🤓, but any mismatch should get sorted out with Errata to this document and/or [13]. See [44] for the Errata process.
• Data Type: the XDR data type of the attribute 🧩.
• Acc: who's allowed to touch the attribute 🔐. R means read-only (GETATTR may retrieve it, SETATTR may not set it). W means write-only (SETATTR may set it, GETATTR may not retrieve it). R W means read/write (GETATTR may retrieve it, SETATTR may set it).
• Defined in: the section of this specification that describes the attribute 📖.

+--------------------+----+------------+-----+------------------+
| Name               | Id | Data Type  | Acc | Defined in:      |
+--------------------+----+------------+-----+------------------+
| supported_attrs    | 0  | bitmap4    | R   | Section 5.8.1.1  |
| type               | 1  | nfs_ftype4 | R   | Section 5.8.1.2  |
| fh_expire_type     | 2  | uint32_t   | R   | Section 5.8.1.3  |
| change             | 3  | uint64_t   | R   | Section 5.8.1.4  |
| size               | 4  | uint64_t   | R W | Section 5.8.1.5  |
| link_support       | 5  | bool       | R   | Section 5.8.1.6  |
| symlink_support    | 6  | bool       | R   | Section 5.8.1.7  |
| named_attr         | 7  | bool       | R   | Section 5.8.1.8  |
| fsid               | 8  | fsid4      | R   | Section 5.8.1.9  |
| unique_handles     | 9  | bool       | R   | Section 5.8.1.10 |
| lease_time         | 10 | nfs_lease4 | R   | Section 5.8.1.11 |
| rdattr_error       | 11 | enum       | R   | Section 5.8.1.12 |
| filehandle         | 19 | nfs_fh4    | R   | Section 5.8.1.13 |
| suppattr_exclcreat | 75 | bitmap4    | R   | Section 5.8.1.14 |
+--------------------+----+------------+-----+------------------+

                              Table 2

5.7. RECOMMENDED Attributes - List and Definition References 🌸

The RECOMMENDED attributes are all defined in Table 3 🍓. The column headers mean the same thing as in Table 2, so peek at Section 5.6 for the details 💜.

+--------------------+----+----------------+-----+------------------+
| Name               | Id | Data Type      | Acc | Defined in:      |
+--------------------+----+----------------+-----+------------------+
| acl                | 12 | nfsace4<>      | R W | Section 6.2.1    |
| aclsupport         | 13 | uint32_t       | R   | Section 6.2.1.2  |
| archive            | 14 | bool           | R W | Section 5.8.2.1  |
| cansettime         | 15 | bool           | R   | Section 5.8.2.2  |
| case_insensitive   | 16 | bool           | R   | Section 5.8.2.3  |
| case_preserving    | 17 | bool           | R   | Section 5.8.2.4  |
| change_policy      | 60 | chg_policy4    | R   | Section 5.8.2.5  |
| chown_restricted   | 18 | bool           | R   | Section 5.8.2.6  |
| dacl               | 58 | nfsacl41       | R W | Section 6.2.2    |
| dir_notif_delay    | 56 | nfstime4       | R   | Section 5.11.1   |
| dirent_notif_delay | 57 | nfstime4       | R   | Section 5.11.2   |
| fileid             | 20 | uint64_t       | R   | Section 5.8.2.7  |
| files_avail        | 21 | uint64_t       | R   | Section 5.8.2.8  |
| files_free         | 22 | uint64_t       | R   | Section 5.8.2.9  |
| files_total        | 23 | uint64_t       | R   | Section 5.8.2.10 |
| fs_charset_cap     | 76 | uint32_t       | R   | Section 5.8.2.11 |
| fs_layout_type     | 62 | layouttype4<>  | R   | Section 5.12.1   |
| fs_locations       | 24 | fs_locations   | R   | Section 5.8.2.12 |
| fs_locations_info  | 67 | *              | R   | Section 5.8.2.13 |
| fs_status          | 61 | fs4_status     | R   | Section 5.8.2.14 |
| hidden             | 25 | bool           | R W | Section 5.8.2.15 |
| homogeneous        | 26 | bool           | R   | Section 5.8.2.16 |
| layout_alignment   | 66 | uint32_t       | R   | Section 5.12.2   |
| layout_blksize     | 65 | uint32_t       | R   | Section 5.12.3   |
| layout_hint        | 63 | layouthint4    |   W | Section 5.12.4   |
| layout_type        | 64 | layouttype4<>  | R   | Section 5.12.5   |
| maxfilesize        | 27 | uint64_t       | R   | Section 5.8.2.17 |
| maxlink            | 28 | uint32_t       | R   | Section 5.8.2.18 |
| maxname            | 29 | uint32_t       | R   | Section 5.8.2.19 |
| maxread            | 30 | uint64_t       | R   | Section 5.8.2.20 |
| maxwrite           | 31 | uint64_t       | R   | Section 5.8.2.21 |
| mdsthreshold       | 68 | mdsthreshold4  | R   | Section 5.12.6   |
| mimetype           | 32 | utf8str_cs     | R W | Section 5.8.2.22 |
| mode               | 33 | mode4          | R W | Section 6.2.4    |
| mode_set_masked    | 74 | mode_masked4   |   W | Section 6.2.5    |
| mounted_on_fileid  | 55 | uint64_t       | R   | Section 5.8.2.23 |
| no_trunc           | 34 | bool           | R   | Section 5.8.2.24 |
| numlinks           | 35 | uint32_t       | R   | Section 5.8.2.25 |
| owner              | 36 | utf8str_mixed  | R W | Section 5.8.2.26 |
| owner_group        | 37 | utf8str_mixed  | R W | Section 5.8.2.27 |
| quota_avail_hard   | 38 | uint64_t       | R   | Section 5.8.2.28 |
| quota_avail_soft   | 39 | uint64_t       | R   | Section 5.8.2.29 |
| quota_used         | 40 | uint64_t       | R   | Section 5.8.2.30 |
| rawdev             | 41 | specdata4      | R   | Section 5.8.2.31 |
| retentevt_get      | 71 | retention_get4 | R   | Section 5.13.3   |
| retentevt_set      | 72 | retention_set4 |   W | Section 5.13.4   |
| retention_get      | 69 | retention_get4 | R   | Section 5.13.1   |
| retention_hold     | 73 | uint64_t       | R W | Section 5.13.5   |
| retention_set      | 70 | retention_set4 |   W | Section 5.13.2   |
| sacl               | 59 | nfsacl41       | R W | Section 6.2.3    |
| space_avail        | 42 | uint64_t       | R   | Section 5.8.2.32 |
| space_free         | 43 | uint64_t       | R   | Section 5.8.2.33 |
| space_total        | 44 | uint64_t       | R   | Section 5.8.2.34 |
| space_used         | 45 | uint64_t       | R   | Section 5.8.2.35 |
| system             | 46 | bool           | R W | Section 5.8.2.36 |
| time_access        | 47 | nfstime4       | R   | Section 5.8.2.37 |
| time_access_set    | 48 | settime4       |   W | Section 5.8.2.38 |
| time_backup        | 49 | nfstime4       | R W | Section 5.8.2.39 |
| time_create        | 50 | nfstime4       | R W | Section 5.8.2.40 |
| time_delta         | 51 | nfstime4       | R   | Section 5.8.2.41 |
| time_metadata      | 52 | nfstime4       | R   | Section 5.8.2.42 |
| time_modify        | 53 | nfstime4       | R   | Section 5.8.2.43 |
| time_modify_set    | 54 | settime4       |   W | Section 5.8.2.44 |
+--------------------+----+----------------+-----+------------------+

                              Table 3

• fs_locations_info4

5.8. Attribute Definitions 🧁

5.8.1. Definitions of REQUIRED Attributes 💎

5.8.1.1. Attribute 0: supported_attrs

This is the bit vector that would fetch every REQUIRED and RECOMMENDED attribute supported for this object 🪄. Its scope covers all objects with a matching fsid 🌈.

5.8.1.2. Attribute 1: type

This one tells you what kind of object you're looking at, using one of a bunch of special constants 🧸:

• NF4REG designates a regular file 📄.
• NF4DIR designates a directory 📁.
• NF4BLK designates a block device special file 🧱.
• NF4CHR designates a character device special file ⌨️.
• NF4LNK designates a symbolic link 🔗.
• NF4SOCK designates a named socket special file 🧦.
• NF4FIFO designates a fifo special file 🚰.
• NF4ATTRDIR designates a named attribute directory 🗂️.
• NF4NAMEDATTR designates a named attribute 🏷️.

Inside the explanatory text and operation descriptions, these phrases have the following meanings 💬:

• The phrase "is a directory" means that the object's type attribute is NF4DIR or NF4ATTRDIR.
• The phrase "is a special file" means that the object's type attribute is NF4BLK, NF4CHR, NF4SOCK, or NF4FIFO.
• The phrases "is an ordinary file" and "is a regular file" mean that the object's type attribute is NF4REG or NF4NAMEDATTR.

5.8.1.3. Attribute 2: fh_expire_type

The server uses this to tell the client how filehandle expiration behaves ⏳. Check Section 4 for more details 🌿.

5.8.1.4. Attribute 3: change

This is a value the server makes up so the client can tell whether file data, directory contents, or attributes of the object have changed 🔍. The server may return the object's time_metadata attribute as this attribute's value, but only if the file system object can't be updated more often than the resolution of time_metadata ⏰.

5.8.1.5. Attribute 4: size ✨

okay so this one is giving simple 💅 The size of the object in bytes. that's it bestie, one number, in bytes 🧚

5.8.1.6. Attribute 5: link_support 🔗

TRUE, if the object's file system supports hard links. so if the file system can do hard links, this is TRUE. iykyk 🦋

5.8.1.7. Attribute 6: symlink_support 🌸

TRUE, if the object's file system supports symbolic links. same energy as the one before, but for symlinks instead of hard links 💖

5.8.1.8. Attribute 7: named_attr 🎀

TRUE, if this object has named attributes. In other words, object has a non-empty named attribute directory. so basically TRUE means the object's named attribute directory isn't empty. it's giving "has stuff inside" vibes ✨

5.8.1.9. Attribute 8: fsid 👑

Unique file system identifier for the file system holding this object. The fsid attribute has major and minor components, each of which are of data type uint64_t. ngl this one is iconic: it's a unique ID for the file system, and it's got two parts (major and minor), each one a uint64_t 💜

5.8.1.10. Attribute 9: unique_handles 🔐

TRUE, if two distinct filehandles are guaranteed to refer to two different file system objects. so if this is TRUE, the server is promising that two different filehandles never point at the same object. no cap, that's a big promise 🫶

5.8.1.11. Attribute 10: lease_time 🕰️

Duration of the lease at server in seconds. so it's how long the lease lasts, measured in seconds, as the server sees it 🌈

5.8.1.12. Attribute 11: rdattr_error ⚠️

Error returned from an attempt to retrieve attributes during a READDIR operation. so if grabbing attributes goes wrong during a READDIR, this is where that error shows up 😭

5.8.1.13. Attribute 19: filehandle 🧚

The filehandle of this object (primarily for READDIR requests). mostly this one matters for READDIR requests, but it's the object's filehandle in general 💅

5.8.1.14. Attribute 75: suppattr_exclcreat 🍓

The bit vector that would set all REQUIRED and RECOMMENDED attributes that are supported by the EXCLUSIVE4_1 method of file creation via the OPEN operation. The scope of this attribute applies to all objects with a matching fsid. so it's a bit vector that would set every REQUIRED and RECOMMENDED attribute that EXCLUSIVE4_1 can handle during OPEN file creation, and it covers every object sharing the same fsid. periodt 💖

---

5.8.2. Definitions of Uncategorized RECOMMENDED Attributes 🦋

The definitions of most of the RECOMMENDED attributes follow. Collections that share a common category are defined in other sections. so the rest of the RECOMMENDED attributes get their definitions right here, and the ones in a shared category live elsewhere 🌸

5.8.2.1. Attribute 14: archive 📦

TRUE, if this file has been archived since the time of last modification (deprecated in favor of time_backup). so TRUE means it got archived after its last change. heads up, this one's deprecated in favor of time_backup 🙈

5.8.2.2. Attribute 15: cansettime 🕰️

TRUE, if the server is able to change the times for a file system object as specified in a SETATTR operation. so this tells you whether the server can actually change the times through SETATTR 💜

5.8.2.3. Attribute 16: case_insensitive 🔤

TRUE, if file name comparisons on this file system are case insensitive. bestie, TRUE means "Foo" and "foo" count as the same name here 🎀

5.8.2.4. Attribute 17: case_preserving 🔠

TRUE, if file name case on this file system is preserved. so TRUE means the case of file names gets kept as-is, no sneaky changes 🫶

5.8.2.5. Attribute 60: change_policy ⚖️

A value created by the server that the client can use to determine if some server policy related to the current file system has been subject to change. If the value remains the same, then the client can be sure that the values of the attributes related to fs location and the fss_type field of the fs_status attribute have not changed. On the other hand, a change in this value does necessarily imply a change in policy. It is up to the client to interrogate the server to determine if some policy relevant to it has changed. See Section 3.3.6 for details.

so the server makes up a value, and the client checks it to see if policy stuff changed. if the value stays the same, the client can be sure the fs location attributes and the fss_type field of fs_status haven't changed. BUT if the value does change, that doesn't automatically mean policy changed. the client has to go ask the server to be sure 🔍 check Section 3.3.6 for the full tea ☕

This attribute MUST change when the value returned by the fs_locations or fs_locations_info attribute changes, when a file system goes from read-only to writable or vice versa, or when the allowable set of security flavors for the file system or any part thereof is changed.

this one is a MUST, so it's non-negotiable. change_policy MUST change when fs_locations or fs_locations_info changes, when a file system goes read-only to writable (or back), or when the allowed security flavors for the file system or any part of it change 🚨

5.8.2.6. Attribute 18: chown_restricted 🔒

If TRUE, the server will reject any request to change either the owner or the group associated with a file if the caller is not a privileged user (for example, "root" in UNIX operating environments or, in Windows 2000, the "Take Ownership" privilege). so if this is TRUE, only privileged users can change the owner or group, and the server will say no to everyone else. "root" on UNIX, or the "Take Ownership" privilege on Windows 2000, for example 👑

5.8.2.7. Attribute 20: fileid 🆔

A number uniquely identifying the file within the file system. just one number, unique inside its file system 💖

5.8.2.8. Attribute 21: files_avail 📂

File slots available to this user on the file system containing this object -- this should be the smallest relevant limit. so it's how many file slots this user can still use, and you should go with the smallest relevant limit 🧮

5.8.2.9. Attribute 22: files_free 🗂️

Free file slots on the file system containing this object -- this should be the smallest relevant limit. free slots, smallest relevant limit, same as before 🌷

5.8.2.10. Attribute 23: files_total 🧾

Total file slots on the file system containing this object. the total count of file slots on that file system 🍓

5.8.2.11. Attribute 76: fs_charset_cap 🔡

Character set capabilities for this file system. See Section 14.4. the character set capabilities of the file system, so go check Section 14.4 for the details 🦋

5.8.2.12. Attribute 24: fs_locations 📍

Locations where this file system may be found. If the server returns NFS4ERR_MOVED as an error, this attribute MUST be supported. See Section 11.9 for more details. so this is where the file system can be found, and if the server ever returns NFS4ERR_MOVED, this attribute MUST be supported. check Section 11.9 for the deets 🗺️

5.8.2.13. Attribute 67: fs_locations_info 🧭

Full function file system location. See Section 11.10 for more details. the full-featured version of the location info, and Section 11.10 has the rest 🌈

5.8.2.14. Attribute 61: fs_status 📊

Generic file system type information. See Section 11.11 for more details. basically the generic type info for the file system, with more in Section 11.11 💜

5.8.2.15. Attribute 25: hidden 🙈

TRUE, if the file is considered hidden with respect to the Windows API. so TRUE means Windows treats the file as hidden 👀

5.8.2.16. Attribute 26: homogeneous 🧁

TRUE, if this object's file system is homogeneous; i.e., all objects in the file system (all objects on the server with the same fsid) have common values for all per-file-system attributes. so TRUE means every object sharing this fsid has the same values for all the per-file-system attributes. super consistent, very slay ✨

5.8.2.17. Attribute 27: maxfilesize 📏

Maximum supported file size for the file system of this object. the biggest file size the file system will support 🐘

5.8.2.18. Attribute 28: maxlink 🔗

Maximum number of links for this object. the most links this object can have 💅

5.8.2.19. Attribute 29: maxname 🏷️

Maximum file name size supported for this object. the longest file name this object can have 🎀

5.8.2.20. Attribute 30: maxread 📖

Maximum amount of data the READ operation will return for this object. the most data a READ will hand back for this object 🌸

5.8.2.21. Attribute 31: maxwrite ✍️

Maximum amount of data the WRITE operation will accept for this object. This attribute SHOULD be supported if the file is writable. Lack of this attribute can lead to the client either wasting bandwidth or not receiving the best performance.

so this is the most data a WRITE will accept. this attribute SHOULD be supported if the file is writable, and if it's missing the client might waste bandwidth or miss out on the best performance. not ideal bestie 😭

5.8.2.22. Attribute 32: mimetype 📄

MIME body type/subtype of this object. the MIME body type and subtype of the object 🦋

5.8.2.23. Attribute 55: mounted_on_fileid 🗻

Like fileid, but if the target filehandle is the root of a file system, this attribute represents the fileid of the underlying directory.

so it's like fileid, except if the filehandle is the root of a file system, this gives you the fileid of the directory underneath 🌳

UNIX-based operating environments connect a file system into the namespace by connecting (mounting) the file system onto the existing file object (the mount point, usually a directory) of an existing file system. When the mount point's parent directory is read via an API like readdir(), the return results are directory entries, each with a component name and a fileid. The fileid of the mount point's directory entry will be different from the fileid that the stat() system call returns. The stat() system call is returning the fileid of the root of the mounted file system, whereas readdir() is returning the fileid that stat() would have returned before any file systems were mounted on the mount point.

okay so UNIX mounting is a whole thing. you hook one file system onto a mount point (usually a directory) in another one. when you readdir() the parent, the mount point's fileid is different from what stat() gives you, because stat() reports the root of the mounted file system, while readdir() reports the fileid stat() would've given back before anything got mounted there 🧠

Unlike NFSv3, NFSv4.1 allows a client's LOOKUP request to cross other file systems. The client detects the file system crossing whenever the filehandle argument of LOOKUP has an fsid attribute different from that of the filehandle returned by LOOKUP. A UNIX-based client will consider this a "mount point crossing". UNIX has a legacy scheme for allowing a process to determine its current working directory. This relies on readdir() of a mount point's parent and stat() of the mount point returning fileids as previously described. The mounted_on_fileid attribute corresponds to the fileid that readdir() would have returned as described previously.

ngl this part is so good 🙌 unlike NFSv3, NFSv4.1 lets a LOOKUP cross into other file systems. the client spots a crossing when the LOOKUP's filehandle argument has an fsid different from the filehandle LOOKUP returns, and a UNIX client calls that a "mount point crossing". UNIX also has an old trick for figuring out the current working directory, which depends on readdir() and stat() giving back fileids like we just talked about. mounted_on_fileid lines up with what readdir() would have returned 🦋

While the NFSv4.1 client could simply fabricate a fileid corresponding to what mounted_on_fileid provides (and if the server does not support mounted_on_fileid, the client has no choice), there is a risk that the client will generate a fileid that conflicts with one that is already assigned to another object in the file system. Instead, if the server can provide the mounted_on_fileid, the potential for client operational problems in this area is eliminated.

so the NFSv4.1 client could just make up a fileid matching mounted_on_fileid (and if the server doesn't support it, it kinda has to). but that risks picking a fileid that's already taken by another object in the file system 😬 if the server CAN provide mounted_on_fileid, though, those client problems just vanish ✨

If the server detects that there is no mounted point at the target file object, then the value for mounted_on_fileid that it returns is the same as that of the fileid attribute.

no mount point at the target? then mounted_on_fileid just matches fileid 🫶

The mounted_on_fileid attribute is RECOMMENDED, so the server SHOULD provide it if possible, and for a UNIX-based server, this is straightforward. Usually, mounted_on_fileid will be requested during a READDIR operation, in which case it is trivial (at least for UNIX-based servers) to return mounted_on_fileid since it is equal to the fileid of a directory entry returned by readdir(). If mounted_on_fileid is requested in a GETATTR operation, the server should obey an invariant that has it returning a value that is equal to the file object's entry in the object's parent directory, i.e., what readdir() would have returned. Some operating environments allow a series of two or more file systems to be mounted onto a single mount point. In this case, for the server to obey the aforementioned invariant, it will need to find the base mount point, and not the intermediate mount points.

mounted_on_fileid is RECOMMENDED, so the server SHOULD provide it when it can, and on a UNIX-based server that's easy 💅 during READDIR it's trivial, since it just matches the fileid of a directory entry. if it's asked for in GETATTR, the server should keep the invariant that it matches what readdir() would've returned for the object's entry in its parent directory. and if you've got a bunch of file systems stacked on one mount point, the server needs to find the base mount point, not the intermediate ones, to keep that invariant 🌈

5.8.2.24. Attribute 34: no_trunc ✂️

If this attribute is TRUE, then if the client uses a file name longer than name_max, an error will be returned instead of the name being truncated. so with TRUE, a too-long name gets an error instead of getting chopped 🙅‍♀️

5.8.2.25. Attribute 35: numlinks 🔗

Number of hard links to this object. the count of hard links pointing at this object 💜

5.8.2.26. Attribute 36: owner 👤

The string name of the owner of this object. the owner's name, as a string 🧚

5.8.2.27. Attribute 37: owner_group 👥

The string name of the group ownership of this object. the group that owns it, also as a string 🌸

5.8.2.28. Attribute 38: quota_avail_hard 💾

The value in bytes that represents the amount of additional disk space beyond the current allocation that can be allocated to this file or directory before further allocations will be refused. It is understood that this space may be consumed by allocations to other files or directories.

so it's the extra bytes you can still allocate to this file or directory before new allocations get refused. that space might get eaten up by other files or directories too, and that's just how it is 😭

5.8.2.29. Attribute 39: quota_avail_soft 🫧

The value in bytes that represents the amount of additional disk space that can be allocated to this file or directory before the user may reasonably be warned. It is understood that this space may be consumed by allocations to other files or directories though there is a rule as to which other files or directories.

this is the extra bytes before the user might get a warning. same deal, other files or directories can eat that space too, though there's a rule about which ones 🎀

5.8.2.30. Attribute 40: quota_used 📊

The value in bytes that represents the amount of disk space used by this file or directory and possibly a number of other similar files or directories, where the set of "similar" meets at least the criterion that allocating space to any file or directory in the set will reduce the "quota_avail_hard" of every other file or directory in the set.

Note that there may be a number of distinct but overlapping sets of files or directories for which a quota_used value is maintained, e.g., "all files with a given owner", "all files with a given group owner", etc. The server is at liberty to choose any of those sets when providing the content of the quota_used attribute, but should do so in a repeatable way. The rule may be configured per file system or may be "choose the set with the smallest quota".

so quota_used is the bytes used by this file or directory, plus maybe some other similar files or directories. "similar" means that allocating space to any one of them lowers the quota_avail_hard of all the others 🧩 heads up, there can be lots of overlapping sets, like "all files with a given owner" or "all files with a given group owner". the server picks whichever set it wants, but it should be repeatable about it. the rule can be set per file system, or it can be "choose the set with the smallest quota" 🫧

5.8.2.31. Attribute 41: rawdev 🧱

Raw device number of file of type NF4BLK or NF4CHR. The device number is split into major and minor numbers. If the file's type attribute is not NF4BLK or NF4CHR, the value returned SHOULD NOT be considered useful.

so it's the raw device number for NF4BLK or NF4CHR files, split into major and minor numbers. if the file's type isn't NF4BLK or NF4CHR, the value SHOULD NOT be considered useful, so don't lean on it 🙅‍♀️

5.8.2.32. Attribute 42: space_avail 💽

Disk space in bytes available to this user on the file system containing this object -- this should be the smallest relevant limit. the disk space in bytes this user can use on the file system, and yes, smallest relevant limit again 🌷

5.8.2.33. Attribute 43: space_free 💿

Free disk space in bytes on the file system containing this object -- this should be the smallest relevant limit. free disk space in bytes, smallest relevant limit, you know the drill 🍓

5.8.2.34. Attribute 44: space_total 🗄️

Total disk space in bytes on the file system containing this object. the total disk space in bytes for that file system 🦋

5.8.2.35. Attribute 45: space_used 📈

Number of file system bytes allocated to this object. how many file system bytes are allocated to this object 💖

5.8.2.36. Attribute 46: system 🪟

This attribute is TRUE if this file is a "system" file with respect to the Windows operating environment. so TRUE means Windows considers this a "system" file 👑

5.8.2.37. Attribute 47: time_access ✨

Okay bestie, time_access is the time of last access to the object by a READ operation sent to the server 👀. What counts as an "access" depends on the server's operating environment and/or its file system semantics 🧚. Like, servers following POSIX semantics only update time_access for READ and READDIR, not for any of the operations that modify the content of the object [16], [17], [18] 💅. And obviously, setting the matching time_access_set attribute is another way to change time_access 🦋.

Whenever the file object lives on a writable file system, the server should make its best efforts to record time_access into stable storage 💖. But to keep performance from tanking, especially when the server is serving the object's content straight from its cache, the server MAY cache access time updates and lazily write them to stable storage 🫶. Admins are also allowed to turn time_access updates off if they want 🙈.

5.8.2.38. Attribute 48: time_access_set 💜

Sets the time of last access to the object. SETATTR use only 🎀.

5.8.2.39. Attribute 49: time_backup 🌸

The time of last backup of the object 💾.

5.8.2.40. Attribute 50: time_create 🍓

The time of creation of the object. This one has zero relation to the classic UNIX file attribute "ctime" or "change time", so don't mix them up 🙅‍♀️.

5.8.2.41. Attribute 51: time_delta ⏱️

Smallest useful server time granularity ✨.

5.8.2.42. Attribute 52: time_metadata 📝

The time of last metadata modification of the object 💅.

5.8.2.43. Attribute 53: time_modify 🌷

The time of last modification to the object 🥹.

5.8.2.44. Attribute 54: time_modify_set 💫

Sets the time of last modification to the object. SETATTR use only 🔒.

---

5.9. Interpreting owner and owner_group 🎀

Okay so the RECOMMENDED attributes owner and owner_group (and also the users and groups inside the acl attribute) are represented as a UTF-8 string 💖. The reason is that it avoids tying the representation to whatever implementation the client or server happens to be running 🧚. Section 6.1 of RFC 2624 [45] gives extra rationale if you're curious 🤓.

Here's the vibe: the client and server each keep their own local representation of owner and owner_group for storage or for showing the end user 💅. So when these attributes get transferred between them, the local representation gets translated into the form user@dns_domain ✨. That way, a client and server that don't use the same local representation can still translate to a common syntax they both understand 🫶.

Security principals can also be represented in different ways by different security mechanisms 🔐. Servers usually translate those into a common format, typically the one used for local storage, so they can identify the users behind each principal 👑. When those local identifiers get translated into the owner form for files created by such principals, they identify, in a common format, the users tied to each set of principals 🦋.

The translation used to interpret owner and group strings isn't specified as part of the protocol, so lots of solutions can work 🌈. For example, a local translation table can map a numeric identifier to the user@dns_domain syntax, or a name service can do the job 💜. A server can also offer a more general service that isn't limited by any particular translation (which would only translate a limited set of strings) by storing the owner and owner_group attributes in local storage without any translation 🍓. Or it can augment a translation method by storing the entire string for attributes that have no translation available, while using the local representation for the cases where a translation does exist 🌸.

Servers that don't support all possible values of the owner and owner_group attributes SHOULD return an error (NFS4ERR_BADOWNER) when a string with no translation is presented as the value to set with a SETATTR of the owner, owner_group, or acl attributes 🚫. When a server does accept an owner or owner_group value as valid on a SETATTR (and similarly for the owner and group strings in an acl), it's making a promise to return that same string when a corresponding GETATTR is done 🤞. Configuration changes (including changes to how the string maps to the local representation) and ill-constructed name translations (the ones that contain aliasing) can make that promise impossible to keep 😬. Servers should make appropriate efforts so these attributes don't change value when no real change to ownership has happened 🫶.

The "dns_domain" part of the owner string is meant to be a DNS domain name, like user@example.org 🌐. Servers should accept as valid a set of users for at least one domain 💅. A server may treat other domains as having no valid translations 🙈. A more general service shows up when a server can accept users for multiple domains, or for all domains, subject to security constraints 🔒.

If there's no translation available to the client or server, the attribute value gets constructed without the "@" 🧐. So if the owner or owner_group attribute has no "@", it means no translation was available at the sender, and the receiver should not use that string as a basis for translating into its own internal format 🚧. Even though it can't be translated, it may still be useful ✨. For a client, the attribute string can be used for local display of ownership 👀.

For extra compatibility with NFSv3, which identified users and groups by 32-bit unsigned user identifiers and group identifiers, owner and group strings made of decimal numeric values with no leading zeros can get a special interpretation 🧚. Clients and servers can choose to support this if they want 🎀. The receiver may treat such a user or group string as representing the same user that an NFSv3 uid or gid with that numeric value would represent 🌸. A server isn't obligated to accept such a string, though, and may return NFS4ERR_BADOWNER instead 🙅‍♀️. To stop this from being used to sneak past user and group translation (so a client could send every owner and group in numeric form), a server SHOULD return an NFS4ERR_BADOWNER error when there's a valid translation for the user or owner designated this way 🛑. In that case, the client must use the appropriate name@domain string and not the special form for compatibility 💅.

The owner string "nobody" may be used to designate an anonymous user 🥸. It gets associated with a file created by a security principal that can't be mapped through normal means to the owner attribute 👻. Users and implementations of NFSv4.1 SHOULD NOT use "nobody" to designate a real user whose access is not anonymous 🚫💜.

5.10. Character Case Attributes 🔤

With respect to the case_insensitive and case_preserving attributes, each UCS-4 character (which UTF-8 encodes) can be mapped according to Appendix B.2 of RFC 3454 [19] ✨. For general character handling and internationalization issues, see Section 14 🌍.

5.11. Directory Notification Attributes 🔔

As described in Section 18.39, the client can request a minimum delay for notifications of changes to attributes, but the server is free to ignore what the client asks for 🙃. The client can figure out in advance which notification delays the server will accept by sending a GETATTR operation for either or both of two directory notification attributes 🔍. When the client calls the GET_DIR_DELEGATION operation and asks for attribute change notifications, it should request notification delays that are no less than the values in the server-provided attributes 💖.

5.11.1. Attribute 56: dir_notif_delay ⏳

The dir_notif_delay attribute is the minimum number of seconds the server will delay before notifying the client of a change to the directory's attributes 🕰️.

5.11.2. Attribute 57: dirent_notif_delay 🕰️

The dirent_notif_delay attribute is the minimum number of seconds the server will delay before notifying the client of a change to a file object that has an entry in the directory 📂.

5.12. pNFS Attribute Definitions 🧚

5.12.1. Attribute 62: fs_layout_type 🗂️

The fs_layout_type attribute (see Section 3.3.13) applies to a file system and indicates which layout types the file system supports 💅. When the client encounters a new fsid, the client SHOULD obtain the value of the fs_layout_type attribute associated with that new file system 🙌. The client uses this attribute to figure out whether the layout types the server supports match any of the layout types the client supports 🫶.

5.12.2. Attribute 66: layout_alignment 📐

When a client holds layouts on files of a file system, the layout_alignment attribute indicates the preferred alignment for I/O to files on that file system 🎯. Where possible, the client should send READ and WRITE operations with offsets that are whole multiples of the layout_alignment attribute ✨.

5.12.3. Attribute 65: layout_blksize 🧱

When a client holds layouts on files of a file system, the layout_blksize attribute indicates the preferred block size for I/O to files on that file system 💖. Where possible, the client should send READ operations with a count argument that's a whole multiple of layout_blksize, and WRITE operations with a data argument whose size is a whole multiple of layout_blksize 🌸.

5.12.4. Attribute 63: layout_hint 💡

The layout_hint attribute (see Section 3.3.19) may be set on newly created files to influence the metadata server's choice for the file's layout 🧭. If possible, this attribute is one of the ones set in the initial attributes within the OPEN operation 🚪. The metadata server may choose to ignore this attribute 🙈. The layout_hint attribute is a subset of the layout structure returned by LAYOUTGET 🍰. For example, instead of specifying particular devices, this would be used to suggest the stripe width of a file 🧵. The server implementation decides which fields within the layout get used 👑.

5.12.5. Attribute 64: layout_type 🎈

This attribute lists the layout type(s) available for a file 📜. The value the server returns is for informational purposes only 🫧. The client uses the LAYOUTGET operation to get the information it needs to perform I/O, for example, the specific device information for the file and its layout 🔥.

5.12.6. Attribute 68: mdsthreshold ✨

okay so this attribute is the server whispering a hint to the client 🫶 It tells the client when it's actually more efficient to send READ and WRITE operations to the metadata server vs. the data server 💅 There are two kinds of thresholds: a file size threshold and an I/O size threshold 🧚 If a file is smaller than the file size threshold, data accesses SHOULD be sent to the metadata server. If an I/O request is shorter than the I/O size threshold, that I/O SHOULD go to the metadata server too. And each threshold gets specified separately for read and for write 🌸

The server MAY give both types of thresholds for a file. If it gives both, the client SHOULD reach or exceed both thresholds before it sends its read or write requests to the data server 👑 But if only one of the specified thresholds is reached or exceeded, the I/O requests go to the metadata server instead. Lowkey strict, but make it make sense 💖

For each threshold type, a value of zero means no READ or WRITE should go to the metadata server, while a value of all ones means all READs or WRITEs should go to the metadata server 🌈

This attribute is available on a per-filehandle basis. If the current filehandle refers to a non-pNFS file or directory, the metadata server should return an attribute that's representative of the filehandle's file system 🦋 It's suggested that you query this attribute as part of the OPEN operation ✨ And here's the tea: because of dynamic system changes, the client should not assume the attribute stays constant for any specific time period, so it should be periodically refreshed 😭

---

5.13. Retention Attributes 🔒

Retention is a concept where a file object can be placed in an immutable, undeletable, unrenamable state for a fixed or infinite duration of time 🎀 Once something is in this "retained" state, it can't be moved out of it until the retention duration has been reached. Very protective, very iconic 💅

When retention is enabled, retention MUST extend to the data of the file and the name of the file 🫶 The server MAY extend retention to any other property of the file, including any subset of REQUIRED, RECOMMENDED, and named attributes, with the exceptions noted in this section 🦋

Servers MAY support or not support retention on any file object type 🧚

The five retention attributes are explained in the next subsections 💜

5.13.1. Attribute 69: retention_get 👀

If retention is enabled for the associated file, this attribute's value represents the retention begin time of the file object 💖 This attribute is only readable with the GETATTR operation and MUST NOT be modified by the SETATTR operation (Section 5.5). Its value consists of:

const RET4_DURATION_INFINITE    = 0xffffffffffffffff;
struct retention_get4 {
        uint64_t        rg_duration;
        nfstime4        rg_begin_time<1>;
};

The field rg_duration is the duration in seconds indicating how long the file will be retained once retention is enabled ⏳ The field rg_begin_time is an array of up to one absolute time value. If the array is zero length, no beginning retention time has been established, and retention is not enabled 🙈 If rg_duration is equal to RET4_DURATION_INFINITE, the file will be retained for an infinite duration once retention is enabled. Forever bestie 💀

If (as soon as) rg_duration is zero, then rg_begin_time will be of zero length, and again, retention is not (no longer) enabled 🍓

5.13.2. Attribute 70: retention_set 🔐

This attribute is used to set the retention duration and optionally enable retention for the associated file object 🌸 It's only modifiable via the SETATTR operation and MUST NOT be retrieved by the GETATTR operation (Section 5.5). It corresponds to retention_get. Its value consists of:

struct retention_set4 {
        bool            rs_enable;
        uint64_t        rs_duration<1>;
};

If the client sets rs_enable to TRUE, then it's enabling retention on the file object, with the begin time of retention starting from the server's current time and date ⭐ The duration of the retention can also be provided if the rs_duration array is of length one. The duration is the time in seconds from the begin time of retention, and if it's set to RET4_DURATION_INFINITE, the file is to be retained forever 💜 If retention is enabled with no duration specified in either this SETATTR or a previous SETATTR, the duration defaults to zero seconds. The server MAY restrict enabling retention, or the duration of retention, on the basis of the ACE4_WRITE_RETENTION ACL permission 🔥

The enabling of retention MUST NOT prevent the enabling of event-based retention or the modification of the retention_hold attribute 🫶

The following rules apply to both the retention_set and retentevt_set attributes 📜

• As long as retention is not enabled, the client is permitted to decrease the duration. 🌷
• The duration can always be set to an equal or higher value, even if retention is enabled. Note that once retention is enabled, the actual duration (as returned by the retention_get or retentevt_get attributes; see Section 5.13.1 or Section 5.13.3) is constantly counting down to zero (one unit per second), unless the duration was set to RET4_DURATION_INFINITE. Thus, it will not be possible for the client to precisely extend the duration on a file that has retention enabled. 🥲
• While retention is enabled, attempts to disable retention or decrease the retention's duration MUST fail with the error NFS4ERR_INVAL. 🚫
• If the principal attempting to change retention_set or retentevt_set does not have ACE4_WRITE_RETENTION permissions, the attempt MUST fail with NFS4ERR_ACCESS. 🙅‍♀️

5.13.3. Attribute 71: retentevt_get 🗓️

This gets the event-based retention duration and, if enabled, the event-based retention begin time of the file object 💅 It's like retention_get, but it refers to event-based retention. The event that triggers event-based retention is not defined by the NFSv4.1 specification 🤷‍♀️

5.13.4. Attribute 72: retentevt_set ⏰

This sets the event-based retention duration and optionally enables event-based retention on the file object 🎀 It corresponds to retentevt_get and is like retention_set, but it refers to event-based retention. When event-based retention is set, the file MUST be retained even if non-event-based retention has been set and the duration of non-event-based retention has been reached. Conversely, when non-event-based retention has been set, the file MUST be retained even if event-based retention has been set and the duration of event-based retention has been reached 🧠 The server MAY restrict enabling event-based retention, or its duration, on the basis of the ACE4_WRITE_RETENTION ACL permission. The enabling of event-based retention MUST NOT prevent the enabling of non-event-based retention or the modification of the retention_hold attribute 🫶

5.13.5. Attribute 73: retention_hold 🧊

This gets or sets administrative retention holds, one hold per bit position 🦋

This attribute allows one to 64 administrative holds, one hold per bit on the attribute ✨ If retention_hold is not zero, then the file MUST NOT be deleted, renamed, or modified, even if the duration on enabled event or non-event-based retention has been reached 🚧 The server MAY restrict modification of retention_hold on the basis of the ACE4_WRITE_RETENTION_HOLD ACL permission. Enabling administrative retention holds does not prevent the enabling of event-based or non-event-based retention 🌈

If the principal attempting to change retention_hold does not have ACE4_WRITE_RETENTION_HOLD permissions, the attempt MUST fail with NFS4ERR_ACCESS 🙅‍♀️

---

6. Access Control Attributes 🛡️

Access Control Lists (ACLs) are file attributes that specify fine-grained access control 👑 This section covers the "acl", "dacl", "sacl", "aclsupport", "mode", and "mode_set_masked" file attributes and how they interact. Note that file attributes may apply to any file system object 💜

6.1. Goals 🎯

ACLs and modes are two well-established models for specifying permissions 💅 This section specifies requirements that try to meet the following goals:

• If a server supports the mode attribute, it should provide reasonable semantics to clients that only set and retrieve the mode attribute. 🌸
• If a server supports ACL attributes, it should provide reasonable semantics to clients that only set and retrieve those attributes. 🫶
• On servers that support the mode attribute, if ACL attributes have never been set on an object, via inheritance or explicitly, the behavior should be traditional UNIX-like behavior. 🐧
• On servers that support the mode attribute, if the ACL attributes have been previously set on an object, either explicitly or via inheritance:
  • Setting only the mode attribute should effectively control the traditional UNIX-like permissions of read, write, and execute on owner, owner_group, and other. 🔐
  • Setting only the mode attribute should provide reasonable security. For example, setting a mode of 000 should be enough to ensure that future OPEN operations for OPEN4_SHARE_ACCESS_READ or OPEN4_SHARE_ACCESS_WRITE by any principal fail, regardless of a previously existing or inherited ACL. 🚫
• NFSv4.1 may introduce different semantics relating to the mode and ACL attributes, but it does not render invalid any previously existing implementations. Additionally, this section provides clarifications based on previous implementations and discussions around them. 📚
• On servers that support both the mode and the acl or dacl attributes, the server must keep the two consistent with each other. The value of the mode attribute (with the exception of the three high-order bits described in Section 6.2.4) must be determined entirely by the value of the ACL, so that use of the mode is never required for anything other than setting the three high-order bits. See Section 6.4.1 for exact requirements. 🧩
• When a mode attribute is set on an object, the ACL attributes may need to be modified in order to not conflict with the new mode. In such cases, it is desirable that the ACL keep as much information as possible. This includes information about inheritance, AUDIT and ALARM ACEs, and permissions granted and denied that do not conflict with the new mode. main character energy 🦋

6.2. File Attributes Discussion 🌟

6.2.1. Attribute 12: acl ✨

Okay bestie, the acl attribute is basically the NFSv4.1 Access Control List 💅 It holds an array of Access Control Entries (ACEs) that are attached to the file system object 🦋 Clients can both set and get it, BUT the server is the one actually using it to do access control 👑 Want to check access without touching any data or metadata? Use the OPEN or ACCESS operations 🫶

The NFS ACE structure looks like this 🌸

typedef uint32_t        acetype4;

typedef uint32_t aceflag4;


typedef uint32_t        acemask4;


struct nfsace4 {
        acetype4        type;
        aceflag4        flag;
        acemask4        access_mask;
        utf8str_mixed   who;
};

So here's how a request gets its yes or no 🔥 The server goes through each nfsace4 entry in order. Only ACEs whose "who" matches the requester get considered. Each ACE gets processed until all of the requester's access bits have been ALLOWED. Once a bit (see below) has been ALLOWED by an ACCESS_ALLOWED_ACE, it's no longer considered in the processing of later ACEs. If an ACCESS_DENIED_ACE shows up while the requester's access still has unALLOWED bits in common with that ACE's "access_mask", the request is denied 💀 And when the whole ACL has been processed, if there are bits in the requester's mask that were never ALLOWED or DENIED, access is denied too. Harsh, but fair 😭

Now the ALARM and AUDIT ACE types are a different vibe. They don't change a requester's access at all. Instead, they're there to trigger events when someone tries to access something 👀 That's why AUDIT and ALARM ACEs only get processed after the ALLOW and DENY ACEs are done 🙈

The NFSv4.1 ACL model is seriously rich ✨ Some server platforms have access control that goes beyond the UNIX-style mode attribute, but it's still not as rich as the NFS ACL model. So that users can actually use that more limited functionality, a server may support the acl attributes by mapping between its own ACL model and the NFSv4.1 ACL model 🎀 Servers must ensure that the ACL they actually store or enforce is at least as strict as the NFSv4 ACL that was set. It's tempting to make this easy by rejecting any ACL that falls outside the small set that can be represented accurately, but that can leave ACLs totally unusable without special client-side knowledge of the server's mapping, which defeats the whole point of having a common NFSv4 ACL protocol 😤 So servers should accept every ACL that they can without compromising security. To help with that, servers may make a special exception, for unsupported permission bits, to the rule that bits not ALLOWED or DENIED by an ACL must be denied. For example, a UNIX-style server might choose to silently allow read attribute permissions even though an ACL doesn't explicitly allow them. (An ACL that explicitly denies permission to read attributes should still be rejected, no exceptions 🙅‍♀️)

It gets more complicated because a server may have multiple modules that enforce ACLs 🧚 For example, enforcement for NFSv4.1 access may be different from, but not weaker than, enforcement for local access, and both may be different from enforcement for access through other protocols such as SMB (Server Message Block). So it may be useful for a server to accept an ACL even if not all of its modules can support it.

The guiding principle for NFSv4 access is simple: the server must not accept ACLs that appear to make access to the file more restrictive than it really is 🔐

6.2.1.1. ACE Type

The constants used for the type field (acetype4) are as follows 💜

const ACE4_ACCESS_ALLOWED_ACE_TYPE      = 0x00000000;
const ACE4_ACCESS_DENIED_ACE_TYPE       = 0x00000001;
const ACE4_SYSTEM_AUDIT_ACE_TYPE        = 0x00000002;
const ACE4_SYSTEM_ALARM_ACE_TYPE        = 0x00000003;

Only the ALLOWED and DENIED bits may be used in the dacl attribute, and only the AUDIT and ALARM bits may be used in the sacl attribute. All four are permitted in the acl attribute 🍓

+------------------------------+--------------+---------------------+
| Value                        | Abbreviation | Description         |
+------------------------------+--------------+---------------------+
| ACE4_ACCESS_ALLOWED_ACE_TYPE | ALLOW        | Explicitly grants   |
|                              |              | the access defined  |
|                              |              | in acemask4 to the  |
|                              |              | file or directory.  |
| ACE4_ACCESS_DENIED_ACE_TYPE  | DENY         | Explicitly denies   |
|                              |              | the access defined  |
|                              |              | in acemask4 to the  |
|                              |              | file or directory.  |
| ACE4_SYSTEM_AUDIT_ACE_TYPE   | AUDIT        | Log (in a           |
|                              |              | system-dependent    |
|                              |              | way) any access     |
|                              |              | attempt to a file   |
|                              |              | or directory that   |
|                              |              | uses any of the     |
|                              |              | access methods      |
|                              |              | specified in        |
|                              |              | acemask4.           |
| ACE4_SYSTEM_ALARM_ACE_TYPE   | ALARM        | Generate an alarm   |
|                              |              | (in a               |
|                              |              | system-dependent    |
|                              |              | way) when any       |
|                              |              | access attempt is   |
|                              |              | made to a file or   |
|                              |              | directory for the   |
|                              |              | access methods      |
|                              |              | specified in        |
|                              |              | acemask4.           |
+------------------------------+--------------+---------------------+

The "Abbreviation" column is the shorthand we'll use for these types through the rest of this section, so keep it in mind 📝

6.2.1.2. Attribute 13: aclsupport

A server need not support all of the above ACE types 🦋 This attribute tells you which ACE types are supported for the current file system. The bitmask constants used to represent the above definitions within the aclsupport attribute are as follows ✨

const ACL4_SUPPORT_ALLOW_ACL    = 0x00000001;
const ACL4_SUPPORT_DENY_ACL     = 0x00000002;
const ACL4_SUPPORT_AUDIT_ACL    = 0x00000004;
const ACL4_SUPPORT_ALARM_ACL    = 0x00000008;

Servers that support either the ALLOW or DENY ACE type SHOULD support both ALLOW and DENY ACE types 🫶

Clients should not attempt to set an ACE unless the server claims support for that ACE type, so check first, bestie 🙏 If the server gets a request to set an ACE that it cannot store, it MUST reject the request with NFS4ERR_ATTRNOTSUPP. If the server gets a request to set an ACE that it can store but cannot enforce, the server SHOULD reject the request with NFS4ERR_ATTRNOTSUPP 🥺

Support for any of the ACL attributes is optional (albeit RECOMMENDED). However, a server that supports either of the new ACL attributes (dacl or sacl) MUST allow use of the new ACL attributes to access all of the ACE types that it supports. In other words, if such a server supports ALLOW or DENY ACEs, then it MUST support the dacl attribute, and if it supports AUDIT or ALARM ACEs, then it MUST support the sacl attribute 👑

6.2.1.3. ACE Access Mask

The bitmask constants used for the access mask field are as follows, and yes there are a lot 😭🌈

const ACE4_READ_DATA            = 0x00000001;
const ACE4_LIST_DIRECTORY       = 0x00000001;
const ACE4_WRITE_DATA           = 0x00000002;
const ACE4_ADD_FILE             = 0x00000002;
const ACE4_APPEND_DATA          = 0x00000004;
const ACE4_ADD_SUBDIRECTORY     = 0x00000004;
const ACE4_READ_NAMED_ATTRS     = 0x00000008;
const ACE4_WRITE_NAMED_ATTRS    = 0x00000010;
const ACE4_EXECUTE              = 0x00000020;
const ACE4_DELETE_CHILD         = 0x00000040;
const ACE4_READ_ATTRIBUTES      = 0x00000080;
const ACE4_WRITE_ATTRIBUTES     = 0x00000100;
const ACE4_WRITE_RETENTION      = 0x00000200;
const ACE4_WRITE_RETENTION_HOLD = 0x00000400;

const ACE4_DELETE               = 0x00010000;
const ACE4_READ_ACL             = 0x00020000;
const ACE4_WRITE_ACL            = 0x00040000;
const ACE4_WRITE_OWNER          = 0x00080000;
const ACE4_SYNCHRONIZE          = 0x00100000;

Heads up, some masks have coincident values, like ACE4_READ_DATA and ACE4_LIST_DIRECTORY 👀 The ACE4_LIST_DIRECTORY, ACE4_ADD_FILE, and ACE4_ADD_SUBDIRECTORY entries are meant for directory objects, while ACE4_READ_DATA, ACE4_WRITE_DATA, and ACE4_APPEND_DATA are meant for non-directory objects. Same number, different job, iconic 💅

6.2.1.3.1. Discussion of Mask Attributes ✨

okay bestie, this is where we go mask by mask 💅 each one gets its list of operations it touches, then the tea on what it actually means 🫶

ACE4_READ_DATA 📖

Operation(s) affected:

• READ
• OPEN

Discussion:

This one is the permission to read a file's data 🌸

Servers SHOULD let a user read the data of the file even when only the ACE4_EXECUTE access mask bit is allowed 🙈

ACE4_LIST_DIRECTORY 🍓

Operation(s) affected:

• READDIR

Discussion:

Permission to list what's inside a directory 📂

ACE4_WRITE_DATA ✏️

Operation(s) affected:

• WRITE
• OPEN
• SETATTR of size

Discussion:

Permission to modify a file's data 💖

ACE4_ADD_FILE 🧚

Operation(s) affected:

• CREATE
• LINK
• OPEN
• RENAME

Discussion:

Permission to add a brand new file to a directory ✨ The CREATE operation is affected when nfs_ftype4 is NF4LNK, NF4BLK, NF4CHR, NF4SOCK, or NF4FIFO. (NF4DIR isn't on the list because it's covered by ACE4_ADD_SUBDIRECTORY, so no double dipping 💅) OPEN is affected when it's used to create a regular file. LINK and RENAME are always affected 🦋

ACE4_APPEND_DATA 🌈

Operation(s) affected:

• WRITE
• OPEN
• SETATTR of size

Discussion:

This is the ability to modify a file's data, but only starting at EOF 🎀 It's how you get append-only files: allow ACE4_APPEND_DATA and deny ACE4_WRITE_DATA to the same user or group. Like, iconic. If a file has an ACL like that and a WRITE request comes in for somewhere other than EOF, the server SHOULD return NFS4ERR_ACCESS 😭

ACE4_ADD_SUBDIRECTORY 🌷

Operation(s) affected:

• CREATE
• RENAME

Discussion:

Permission to create a subdirectory inside a directory 📁 The CREATE operation is affected when nfs_ftype4 is NF4DIR. The RENAME operation is always affected 🫶

ACE4_READ_NAMED_ATTRS 👀

Operation(s) affected:

• OPENATTR

Discussion:

Permission to read a file's named attributes, or to look up the named attribute directory 🔍 OPENATTR is affected when it's not being used to create a named attribute directory. That happens when 1) createdir is TRUE but a named attribute directory already exists, or 2) createdir is FALSE 🙃

ACE4_WRITE_NAMED_ATTRS 🖋️

Operation(s) affected:

• OPENATTR

Discussion:

Permission to write a file's named attributes, or to create a named attribute directory 🌸 OPENATTR is affected when it's used to create a named attribute directory, which is when createdir is TRUE and no named attribute directory exists yet. Checking whether one exists depends on being able to look it up, so users also need ACE4_READ_NAMED_ATTRS to create one. Main character energy requires the full setup 👑

ACE4_EXECUTE 🔥

Operation(s) affected:

• READ
• OPEN
• REMOVE
• RENAME
• LINK
• CREATE

Discussion:

Permission to execute a file 🏃‍♀️

Servers SHOULD let a user read the data of the file when only the ACE4_EXECUTE access mask bit is allowed. The reason is that there's literally no way to execute a file without reading its contents 🤓 Though a server may treat ACE4_EXECUTE and ACE4_READ_DATA identically when deciding whether to permit a READ operation, it SHOULD still let the two bits be set independently in ACLs, and it MUST distinguish between them when replying to ACCESS operations. In particular, servers SHOULD NOT silently turn on one of the two bits when the other is set, because then the client couldn't correctly enforce the difference between read and execute permissions. That's a whole mess, bestie 💀

As an example, following a SETATTR of this ACL:

nfsuser:ACE4_EXECUTE:ALLOW

a subsequent GETATTR of the ACL for that file SHOULD return:

nfsuser:ACE4_EXECUTE:ALLOW

and NOT this:

nfsuser:ACE4_EXECUTE/ACE4_READ_DATA:ALLOW

Wait, correction. The original says "Rather than", so I'm keeping that exact vibe:

Rather than:

nfsuser:ACE4_EXECUTE/ACE4_READ_DATA:ALLOW

ACE4_EXECUTE 🧭

Operation(s) affected:

• LOOKUP

Discussion:

Permission to traverse or search a directory 🗺️ (Yes, it's the same name as the one above, the RFC really did that 😂)

ACE4_DELETE_CHILD 🗑️

Operation(s) affected:

• REMOVE
• RENAME

Discussion:

Permission to delete a file or directory within a directory. See Section 6.2.1.3.2 for how ACE4_DELETE and ACE4_DELETE_CHILD interact 💅

ACE4_READ_ATTRIBUTES 👁️

Operation(s) affected:

• GETATTR of file system object attributes
• VERIFY
• NVERIFY
• READDIR

Discussion:

The ability to read basic attributes (not ACLs) of a file ✨ On a UNIX system, basic attributes are the stat-level stuff. Having this bit means you can run "ls -l" and stat 🧁 If a READDIR operation asks for attributes, this mask MUST be allowed for the READDIR to succeed.

ACE4_WRITE_ATTRIBUTES 🕰️

Operation(s) affected:

• SETATTR of time_access_set, time_backup,
• time_create, time_modify_set, mimetype, hidden, system

Discussion:

Permission to change the times on a file or directory to an arbitrary value 🕐 It also covers changing the mimetype, hidden, and system attributes. A user with ACE4_WRITE_DATA or ACE4_WRITE_ATTRIBUTES can set a file's times to the current server time, which is giving "good enough" energy ⏰

ACE4_WRITE_RETENTION 🔒

Operation(s) affected:

• SETATTR of retention_set, retentevt_set.

Discussion:

Permission to modify the durations of event and non-event-based retention 🗓️ It's also permission to enable event and non-event-based retention. A server MAY behave such that setting ACE4_WRITE_ATTRIBUTES allows ACE4_WRITE_RETENTION 🫣

ACE4_WRITE_RETENTION_HOLD 🛑

Operation(s) affected:

• SETATTR of retention_hold.

Discussion:

Permission to modify the administrative retention holds 🧷 A server MAY map ACE4_WRITE_ATTRIBUTES to ACE_WRITE_RETENTION_HOLD 🪄

ACE4_DELETE 🗑️

Operation(s) affected:

• REMOVE

Discussion:

Permission to delete the file or directory itself 💫 See Section 6.2.1.3.2 for how ACE4_DELETE and ACE4_DELETE_CHILD interact 🦋

ACE4_READ_ACL 📜

Operation(s) affected:

• GETATTR of acl, dacl, or sacl
• NVERIFY
• VERIFY

Discussion:

Permission to read the ACL 👀

ACE4_WRITE_ACL ✍️

Operation(s) affected:

• SETATTR of acl and mode

Discussion:

Permission to write the acl and mode attributes 💖

ACE4_WRITE_OWNER 👑

Operation(s) affected:

• SETATTR of owner and owner_group

Discussion:

Permission to write the owner and owner_group attributes 🎀 On UNIX systems this is the ability to run chown() and chgrp() 🐧

ACE4_SYNCHRONIZE 🔄

Operation(s) affected:

• NONE

Discussion:

Permission to use the file object as a synchronization primitive for interprocess communication 🤝 This permission is not enforced or interpreted by the NFSv4.1 server on the client's behalf.

Typically, ACE4_SYNCHRONIZE only means something on local file systems, meaning file systems that aren't accessed via NFSv4.1 🏠 The bit exists because some operating environments, like Windows, use ACE4_SYNCHRONIZE 🪟

For example, say a client copies a file that has ACE4_SYNCHRONIZE set from a local file system to an NFSv4.1 server, and then later copies it from the NFSv4.1 server back to a local file system. If ACE4_SYNCHRONIZE was set in the original, the client will probably want it set in the second copy too 🥹 The first copy won't have the permission set unless the NFSv4.1 server has a way to set the ACE4_SYNCHRONIZE bit. The second copy won't have it set unless the NFSv4.1 server has a way to retrieve the ACE4_SYNCHRONIZE bit. It's a whole chain, bestie 🔗

Server implementations don't have to offer the granularity this list of masks implies 🧠 For example, POSIX-based systems might not tell ACE4_APPEND_DATA (the ability to append to a file) apart from ACE4_WRITE_DATA (the ability to change existing contents). Both masks would be tied to one single "write" permission [20]. When a server like that returns attributes to the client, it shows both ACE4_APPEND_DATA and ACE4_WRITE_DATA if and only if the write permission is enabled 🫶

If a server gets a SETATTR request it can't implement accurately, it should err in the direction of more restricted access, except in the cases of execute and read we talked about earlier 🙅‍♀️ Say a server can't tell overwriting data apart from appending new data, like in the paragraph above. If a client submits an ALLOW ACE where ACE4_APPEND_DATA is set but ACE4_WRITE_DATA isn't (or the other way around), the server should either turn off ACE4_APPEND_DATA or reject the request with NFS4ERR_ATTRNOTSUPP 😤

6.2.1.3.2. ACE4_DELETE vs. ACE4_DELETE_CHILD 🗝️

Two access mask bits control the ability to delete a directory entry 🧩 ACE4_DELETE is on the object itself (the "target"), and ACE4_DELETE_CHILD is on the containing directory (the "parent") 🌳

Lots of systems also use the "sticky bit" (MODE4_SVTX) on a directory, which only lets a user unlink if they own either the target or the parent. On some of those systems, the decision also depends on whether the target is writable 🤔

Servers SHOULD allow unlink if either ACE4_DELETE is permitted on the target, or ACE4_DELETE_CHILD is permitted on the parent. (Note that this is true even if the parent or target explicitly denies one of these permissions. Plot twist 😳)

If the ACLs in question neither explicitly ALLOW nor DENY either of the above, and MODE4_SVTX is not set on the parent, then the server SHOULD allow the removal if and only if ACE4_ADD_FILE is permitted 🎯 When MODE4_SVTX is set, the server may also require the remover to own either the parent or the target, or may require the target to be writable 🔐

This lets servers support something close to traditional UNIX-like semantics, with ACE4_ADD_FILE taking the place of the write bit 🐧✨

6.2.1.4. ACE flag ✨

Okay bestie, the flag field gets its own bitmask constants 🎀:

const ACE4_FILE_INHERIT_ACE             = 0x00000001;
const ACE4_DIRECTORY_INHERIT_ACE        = 0x00000002;
const ACE4_NO_PROPAGATE_INHERIT_ACE     = 0x00000004;
const ACE4_INHERIT_ONLY_ACE             = 0x00000008;
const ACE4_SUCCESSFUL_ACCESS_ACE_FLAG   = 0x00000010;
const ACE4_FAILED_ACCESS_ACE_FLAG       = 0x00000020;
const ACE4_IDENTIFIER_GROUP             = 0x00000040;
const ACE4_INHERITED_ACE                = 0x00000080;

A server need not support any of these flags 💅. If the server has flags that are kinda like these but not exactly the same, the implementation may define a mapping between the protocol-defined flags and the implementation-defined flags. That's a whole vibe 🧚.

For example, say a client tries to set an ACE with ACE4_FILE_INHERIT_ACE on but not ACE4_DIRECTORY_INHERIT_ACE. If the server doesn't support any form of ACL inheritance, the server should reject the request with NFS4ERR_ATTRNOTSUPP 🙈. If the server supports a single "inherit ACE" flag that applies to both files and directories, the server may reject the request (i.e., the client has to set both the file and directory inheritance flags). The server may also accept the request and quietly turn on the ACE4_DIRECTORY_INHERIT_ACE flag 🦋.

6.2.1.4.1. Discussion of Flag Bits 🌸

ACE4_FILE_INHERIT_ACE

Any non-directory file in any sub-directory will get this ACE inherited 📁.

ACE4_DIRECTORY_INHERIT_ACE

You can put this on a directory, and it says "hey, add this ACE to every new directory that gets created" 🍓. If this flag is set in an ACE in an ACL attribute that's going on a non-directory file system object, the operation trying to set the ACL SHOULD fail with NFS4ERR_ATTRNOTSUPP 💀.

ACE4_NO_PROPAGATE_INHERIT_ACE

Also goes on a directory. It tells the server that inheritance of this ACE should stop at newly created child directories 🛑.

ACE4_INHERIT_ONLY_ACE

This one can go on a directory, but it doesn't apply to that directory itself 👑. ALLOW and DENY ACEs with this bit set don't affect access to the directory, and AUDIT and ALARM ACEs with this bit set don't trigger log or alarm events. These ACEs only take effect once they're applied (with this bit cleared) to newly created files and directories, as specified by the ACE4_FILE_INHERIT_ACE and ACE4_DIRECTORY_INHERIT_ACE flags 🔥.

If this flag is present on an ACE, but neither ACE4_DIRECTORY_INHERIT_ACE nor ACE4_FILE_INHERIT_ACE is present, then an operation trying to set that attribute SHOULD fail with NFS4ERR_ATTRNOTSUPP 😭.

ACE4_SUCCESSFUL_ACCESS_ACE_FLAG and ACE4_FAILED_ACCESS_ACE_FLAG

The ACE4_SUCCESSFUL_ACCESS_ACE_FLAG (SUCCESS) and ACE4_FAILED_ACCESS_ACE_FLAG (FAILED) flag bits can only be set on ACE4_SYSTEM_AUDIT_ACE_TYPE (AUDIT) and ACE4_SYSTEM_ALARM_ACE_TYPE (ALARM) ACE types 🕵️‍♀️. If, while the server is processing the file's ACL, it runs into an AUDIT or ALARM ACE that matches the principal attempting the OPEN, the server notes that fact, plus whether the SUCCESS and FAILED flags were present on that ACE. Once the server finishes processing the ACL, it then notes whether the operation succeeded or failed. If the operation succeeded, and the SUCCESS flag was set on a matching AUDIT or ALARM ACE, then the appropriate AUDIT or ALARM event occurs. If the operation failed, and the FAILED flag was set on the matching AUDIT or ALARM ACE, then the appropriate AUDIT or ALARM event occurs. Either or both of SUCCESS or FAILED can be set, but if neither is set, the AUDIT or ALARM ACE isn't useful 🙃.

The processing I just described applies to ACCESS operations even when they return NFS4_OK. For the purposes of AUDIT and ALARM, we consider an ACCESS operation to be a "failure" if it fails to return a bit that was requested and supported ✨.

ACE4_IDENTIFIER_GROUP

This indicates that the "who" refers to a GROUP as defined under UNIX, or a GROUP ACCOUNT as defined under Windows 🧑‍🤝‍🧑. Clients and servers MUST ignore the ACE4_IDENTIFIER_GROUP flag on ACEs with a who value equal to one of the special identifiers outlined in Section 6.2.1.5 🚫.

ACE4_INHERITED_ACE

This says the ACE was inherited from a parent directory. A server that supports automatic inheritance will put this flag on any ACEs inherited from the parent directory when it creates a new object. Client applications use this to do automatic inheritance 🌈. Clients and servers MUST clear this bit in the acl attribute. It may only be used in the dacl and sacl attributes 🔒.

6.2.1.5. ACE Who 💖

The "who" field of an ACE is an identifier that names the principal or principals the ACE applies to. It can refer to a user or a group, and the flag bit ACE4_IDENTIFIER_GROUP is what tells you which one 🫶.

There are a few special identifiers that everyone needs to understand universally, not just in the context of one particular DNS domain 🌍. Some of them can't be understood when an NFS client accesses the server, but they do mean something when a local process accesses the file. Being able to display and modify these permissions is allowed over NFS, even if none of the server's access methods understand the identifiers 💅.

   +---------------+--------------------------------------------------+
   | Who           | Description                                      |
   +---------------+--------------------------------------------------+
   | OWNER         | The owner of the file.                           |
   | GROUP         | The group associated with the file.              |
   | EVERYONE      | The world, including the owner and owning group. |
   | INTERACTIVE   | Accessed from an interactive terminal.           |
   | NETWORK       | Accessed via the network.                        |
   | DIALUP        | Accessed as a dialup user to the server.         |
   | BATCH         | Accessed from a batch job.                       |
   | ANONYMOUS     | Accessed without any authentication.             |
   | AUTHENTICATED | Any authenticated user (opposite of ANONYMOUS).  |
   | SERVICE       | Access from a system service.                    |
   +---------------+--------------------------------------------------+

                                  Table 4

To avoid conflicts, these special identifiers get an appended "@" and should look like "xxxx@" (with no domain name after the "@"), for example, ANONYMOUS@ 🎀.

The ACE4_IDENTIFIER_GROUP flag MUST be ignored on entries with these special identifiers 🙅‍♀️. When encoding entries with these special identifiers, the ACE4_IDENTIFIER_GROUP flag SHOULD be set to zero 🧹.

6.2.1.5.1. Discussion of EVERYONE@ 🌟

Super important to note: "EVERYONE@" is not the same as the UNIX "other" entity 😤. That's because, by definition, UNIX "other" doesn't include the owner or owning group of a file. "EVERYONE@" means literally everyone, including the owner and owning group 🫣.

6.2.2. Attribute 58: dacl 💅

The dacl attribute is like the acl attribute, except dacl only allows ALLOW and DENY ACEs. The dacl attribute supports automatic inheritance (see Section 6.4.3.2) 🦋.

6.2.3. Attribute 59: sacl 👀

The sacl attribute is like the acl attribute, except sacl only allows AUDIT and ALARM ACEs. The sacl attribute supports automatic inheritance (see Section 6.4.3.2) 🔍.

6.2.4. Attribute 33: mode 🌷

The NFSv4.1 mode attribute is based on the UNIX mode bits. These bits are defined:

   const MODE4_SUID = 0x800;  /* set user id on execution */
   const MODE4_SGID = 0x400;  /* set group id on execution */
   const MODE4_SVTX = 0x200;  /* save text even after use */
   const MODE4_RUSR = 0x100;  /* read permission: owner */
   const MODE4_WUSR = 0x080;  /* write permission: owner */
   const MODE4_XUSR = 0x040;  /* execute permission: owner */
   const MODE4_RGRP = 0x020;  /* read permission: group */
   const MODE4_WGRP = 0x010;  /* write permission: group */
   const MODE4_XGRP = 0x008;  /* execute permission: group */
   const MODE4_ROTH = 0x004;  /* read permission: other */
   const MODE4_WOTH = 0x002;  /* write permission: other */
   const MODE4_XOTH = 0x001;  /* execute permission: other */

Bits MODE4_RUSR, MODE4_WUSR, and MODE4_XUSR apply to the principal identified in the owner attribute 👑. Bits MODE4_RGRP, MODE4_WGRP, and MODE4_XGRP apply to principals identified in the owner_group attribute who are not identified in the owner attribute 👥. Bits MODE4_ROTH, MODE4_WOTH, and MODE4_XOTH apply to any principal that doesn't match the owner attribute and doesn't have a group matching the owner_group attribute 🌍.

Bits within a mode other than those specified above aren't defined by this protocol. A server MUST NOT return bits other than those defined above in a GETATTR or READDIR operation, and it MUST return NFS4ERR_INVAL if bits other than those defined above are set in a SETATTR, CREATE, OPEN, VERIFY, or NVERIFY operation 🚨.

6.2.5. Attribute 74: mode_set_masked 🧚‍♀️

The mode_set_masked attribute is write-only, and it lets you set or reset individual bits in the mode attribute without changing the others 💫. For example, it lets you modify MODE4_SUID, MODE4_SGID, and MODE4_SVTX while leaving any of the nine low-order mode bits for permissions untouched 🫧.

When the nine low-order bits are left unmodified, neither the acl nor the dacl attribute should be automatically modified, as discussed in Section 6.4.1 🙌.

The mode_set_masked attribute is made of two words, each in the form of a mode4. The first is the value to be applied to the current mode value, and the second is a mask. Only bits set to one in the mask word get changed (set or reset) in the file's mode. Every other bit in the mode stays the same. Bits in the first word that line up with zero bits in the mask get ignored, except that undefined bits are still checked for validity and can result in NFS4ERR_INVAL, as described below 🧐.

The mode_set_masked attribute is only valid in a SETATTR operation. If it's used in a CREATE or OPEN operation, the server MUST return NFS4ERR_INVAL 🚫.

Bits that aren't defined as valid in the mode attribute aren't valid in either word of the mode_set_masked attribute. The server MUST return NFS4ERR_INVAL if any such bits are set to one in a SETATTR. If the mode and mode_set_masked attributes are both specified in the same SETATTR, the server MUST also return NFS4ERR_INVAL 💥.

6.3. Common Methods 🎀

The requirements in this section will be referred to in future sections, especially Section 6.4 📌.

6.3.1. Interpreting an ACL 🦋

6.3.1.1. Server Considerations 💖

Okay bestie, so the server uses the algorithm from Section 6.2.1 to figure out whether an ACL lets you into an object. BUT the ACL is not always the only thing deciding access 👀 There are a bunch of other things that can get involved:

• In the case of a file system exported as read-only, the server may deny write access even though an object's ACL grants it. 📖
• Server implementations MAY grant ACE4_WRITE_ACL and ACE4_READ_ACL permissions so nobody ends up in a situation where there's no valid way to ever modify the ACL. Literally a safety net 🫶
• Every server will let a user read the data of a file when only the execute permission is granted. So if the ACL denies ACE4_READ_DATA but allows ACE4_EXECUTE, the server will still let that user read the file's data. Iconic behavior 🦋
• Lots of servers have owner-override, where the owner of the object can override accesses the ACL denies. This can be super helpful for letting users keep access to files they already have open after the permissions changed 🌸
• Lots of servers also have a "superuser" with powers beyond a regular user. The superuser may be able to read or write data or metadata in ways the ACL would never allow. Main character energy, but make it a little scary 💀
• A retention attribute might also block access that ACLs would otherwise allow (see Section 5.13). 🔒

6.3.1.2. Client Considerations 🎀

Clients SHOULD NOT do their own access checks based on their own reading of the ACL. Instead, they should use the OPEN and ACCESS operations to do the checks. That way the client can just act on what the server decided after looking at the ACL, which is so much less drama 💅

Clients must be aware that there are situations where an object's ACL defines a certain access but the server won't actually enforce it. In general, and especially in these situations, the client needs to do its part in enforcing access the way the ACL defines it. To do that, the client MAY send the appropriate ACCESS operation before servicing the request of the user or application, so it can check whether that user or app should get the access they're asking for. For examples of ACLs defining accesses the server doesn't enforce, see Section 6.3.1.1. 🧚

6.3.2. Computing a Mode Attribute from an ACL 🌈

The following method can be used to calculate the MODE4_R*, MODE4_W*, and MODE4_X* bits of a mode attribute, based on an ACL. Let's gooo 🔥

First, for each of the special identifiers OWNER@, GROUP@, and EVERYONE@, evaluate the ACL in order. Only consider the ALLOW and DENY ACEs for EVERYONE@ and for the identifier you're currently looking at. The result of that evaluation is an NFSv4 ACL mask showing exactly which bits are permitted to that identifier. ✨

Then translate the calculated mask for OWNER@, GROUP@, and EVERYONE@ into mode bits for the user, group, and other, respectively, like this:

1. Set the read bit (MODE4_RUSR, MODE4_RGRP, or MODE4_ROTH) if and only if ACE4_READ_DATA is set in the corresponding mask. 📖
2. Set the write bit (MODE4_WUSR, MODE4_WGRP, or MODE4_WOTH) if and only if both ACE4_WRITE_DATA and ACE4_APPEND_DATA are set in the corresponding mask. ✍️
3. Set the execute bit (MODE4_XUSR, MODE4_XGRP, or MODE4_XOTH) if and only if ACE4_EXECUTE is set in the corresponding mask. 🏃‍♀️

6.3.2.1. Discussion 🙈

Some server implementations also add bits permitted to named users and groups to the group bits (MODE4_RGRP, MODE4_WGRP, and MODE4_XGRP). Tea. ☕

Implementations are discouraged from doing this, because it's been found to confuse users. They see members of a file's group getting denied access even though the mode bits look like they should allow it 😭 (DENY ACEs can cause this kind of thing too, but DENY ACEs are expected to be used less often.)

The same user confusion shows up when setting the mode doesn't actually control permissions for the owner, group, and other users. That's what motivates some of the requirements coming up next 🫶

6.4. Requirements 👑

The server that supports both mode and ACL must take care to synchronize the MODE4_*USR, MODE4_*GRP, and MODE4_*OTH bits with the ACEs whose who fields are OWNER@, GROUP@, and EVERYONE@. This way, the client can see whether semantically equivalent access permissions exist whether it asks for the owner, owner_group, and mode attributes or just for the ACL. Consistency is everything 💜

In this section, a lot of stuff refers back to the methods in Section 6.3.2. But note that those methods have behaviors specified with "SHOULD". That's intentional, so existing implementations that compute the mode according to the withdrawn POSIX ACL draft (1003.1e draft 17) don't get invalidated, instead of computing it from the actual permissions on owner, group, and other. Respect the legacy era 🧚

6.4.1. Setting the Mode and/or ACL Attributes 💅

In the case where a server supports the sacl or dacl attribute in addition to the acl attribute, the server MUST fail a request to set the acl attribute at the same time as a dacl or sacl attribute. The error to be given is NFS4ERR_ATTRNOTSUPP. No mixing those up, periodt 🚫

6.4.1.1. Setting Mode and not ACL 🌸

When any of the nine low-order mode bits are subject to change, that happens either because the mode attribute was set, or because the mode_set_masked attribute was set and its mask included one or more of the nine low-order mode bits. This applies when no ACL attribute is explicitly set. In that case, the acl and dacl attributes must be modified to match the updated value of those bits. This must happen even if the value of the low-order bits is the same after the mode is set as it was before. Yes, even if nothing changed 🙃

Note that any AUDIT or ALARM ACEs (so any ACEs in the sacl attribute) are unaffected by changes to the mode. Those stay put 🔒

When the permission bits are subject to change, the acl and dacl attributes MUST be modified so the mode computed via the method in Section 6.3.2 yields the low-order nine bits (MODE4_R*, MODE4_W*, MODE4_X*) of the mode attribute as modified by the attribute change. The ACL attributes SHOULD also be modified so that:

1. If MODE4_RGRP is not set, entities explicitly listed in the ACL other than OWNER@ and EVERYONE@ SHOULD NOT be granted ACE4_READ_DATA. 📖
2. If MODE4_WGRP is not set, entities explicitly listed in the ACL other than OWNER@ and EVERYONE@ SHOULD NOT be granted ACE4_WRITE_DATA or ACE4_APPEND_DATA. ✍️
3. If MODE4_XGRP is not set, entities explicitly listed in the ACL other than OWNER@ and EVERYONE@ SHOULD NOT be granted ACE4_EXECUTE. 🏃‍♀️

Access mask bits other than those listed above, appearing in ALLOW ACEs, MAY also be disabled. 🦋

Note that ACEs with the flag ACE4_INHERIT_ONLY_ACE set do not affect the permissions of the ACL itself, and neither do ACEs of type AUDIT and ALARM. So it's desirable to leave these ACEs unmodified when modifying the ACL attributes. Leave them alone, bestie ✨

Also note that the requirement can be met by discarding the acl and dacl in favor of an ACL that represents the mode and only the mode. That's permitted, but it's preferable for a server to preserve as much of the ACL as possible without violating the above requirements. Discarding the ACL makes it effectively impossible for a file created with a mode attribute to inherit an ACL (see Section 6.4.3). Hot take: don't just nuke it 💀

6.4.1.2. Setting ACL and Not Mode 🍓

When setting the acl or dacl and not setting the mode or mode_set_masked attributes, the permission bits of the mode need to be derived from the ACL. In this case, the ACL attribute SHOULD be set as given. The nine low-order bits of the mode attribute (MODE4_R*, MODE4_W*, MODE4_X*) MUST be modified to match the result of the method in Section 6.3.2. The three high-order bits of the mode (MODE4_SUID, MODE4_SGID, MODE4_SVTX) SHOULD remain unchanged. Clean and simple 💖

6.4.1.3. Setting Both ACL and Mode 💜

When setting both the mode (which includes using either the mode attribute or the mode_set_masked attribute) and the acl or dacl attributes in the same operation, the attributes MUST be applied in this order: mode (or mode_set_masked), then ACL. The mode-related attribute is set as given, then the ACL attribute is set as given, which might change the final mode as described above in Section 6.4.1.2. Order matters, bestie 🎀

6.4.2. Retrieving the Mode and/or ACL Attributes 🌈

This section only applies to servers that support both the mode and ACL attributes. ✨

Some server implementations may have a concept of "objects without ACLs." That means all permissions are granted and denied according to the mode attribute, and no ACL attribute is stored for that object. If an ACL attribute is requested from such a server, the server SHOULD return an ACL that doesn't conflict with the mode. In other words, the ACL returned SHOULD represent the nine low-order bits of the mode attribute (MODE4_R*, MODE4_W*, MODE4_X*) as described in Section 6.3.2. 🔥

For other server implementations, the ACL attribute is always present for every object. Such servers SHOULD store at least the three high-order bits of the mode attribute (MODE4_SUID, MODE4_SGID, MODE4_SVTX). The server SHOULD return a mode attribute if one is requested, and the low-order nine bits of the mode (MODE4_R*, MODE4_W*, MODE4_X*) MUST match the result of applying the method in Section 6.3.2 to the ACL attribute. The vibes have to match 🫶

6.4.3. Creating New Objects ✨

okay bestie, so if a server supports any ACL attributes, it may look at the ACL attributes on the parent directory and use them to build a starting ACL for a brand new object 🌸 that starting ACL is called the inherited ACL in this section. adding ACEs to it that come from the parent directory's ACL is called inheriting an ACE here 🦋

implementors should standardize what CREATE and OPEN do depending on whether the mode and ACL attributes are there or not 💅 the rules go like this:

1. If just the mode is given in the call: inheritance SHOULD take place, but the mode MUST be applied to the inherited ACL as described in Section 6.4.1.1, which modifies the ACL 🎀
2. If just the ACL is given in the call: inheritance SHOULD NOT take place. the ACL from the CREATE or OPEN gets set exactly as it is, no changes, and the mode gets changed like in Section 6.4.1.2 🧚
3. If both mode and ACL are given in the call: inheritance SHOULD NOT take place, and both attributes get set the way Section 6.4.1.3 describes 👑
4. If neither mode nor ACL is given in the call: okay this one is a whole thing 🙈 if an object gets created with no initial attributes at all, like an OPEN with an opentype4 of OPEN4_CREATE and a createmode4 of EXCLUSIVE4, inheritance SHOULD NOT take place. (fun fact: EXCLUSIVE4_1 is the better pick for createmode4 because it does allow initial attributes ✨) instead, the server SHOULD set permissions to deny all access to the new object. the client is expected to set the attributes it actually wants in a later SETATTR, and the server SHOULD allow that SETATTR to succeed no matter what permissions the object was created with. like, an empty ACL denies everything, but the server should still let the owner's SETATTR go through even though WRITE_ACL is implicitly denied 😭

in every other case, inheritance SHOULD take place and no changes get made to the ACL. the mode attribute, if supported, MUST be whatever Section 6.3.2 computes, with the MODE4_SUID, MODE4_SGID, and MODE4_SVTX bits cleared 💜 if there are no inheritable ACEs on the parent directory, the rules for creating acl, dacl, or sacl attributes are implementation defined. and if either the dacl or sacl attribute is supported, the ACL4_DEFAULTED flag SHOULD be set on the newly created attributes 🍓

---

6.4.3.1. The Inherited ACL 🦋

if the new object is not a directory, the inherited ACL SHOULD NOT inherit ACEs from the parent directory ACL unless the ACE4_FILE_INHERIT_FLAG is set 🌈

if the new object is a directory, the inherited ACL should inherit every inheritable ACE from the parent directory. those are the ones with ACE4_FILE_INHERIT_ACE or ACE4_DIRECTORY_INHERIT_ACE set 🫶 if an inheritable ACE has ACE4_FILE_INHERIT_ACE set but ACE4_DIRECTORY_INHERIT_ACE is clear, then the inherited ACE on the new directory MUST have the ACE4_INHERIT_ONLY_ACE flag set. that way the directory doesn't get affected by ACEs that were meant for non-directories 💅

when a new directory is created, the server MAY split any inherited ACE that is both inheritable and effective (meaning it has neither ACE4_INHERIT_ONLY_ACE nor ACE4_NO_PROPAGATE_INHERIT_ACE set) into two ACEs. one has no inheritance flags, and the other has ACE4_INHERIT_ONLY_ACE set. (for a dacl or sacl attribute, both of those ACEs SHOULD also have the ACE4_INHERITED_ACE flag set 🔥) this makes it way easier to change the effective permissions on the directory without touching the ACE that gets passed down to the new directory's children ✨

---

6.4.3.2. Automatic Inheritance 🌸

the acl attribute is just an array of ACEs, but the sacl (Section 6.2.3) and dacl (Section 6.2.2) attributes also come with an extra flag field 💖

struct nfsacl41 {
        aclflag4        na41_flag;
        nfsace4         na41_aces<>;
};

the flag field applies to the whole sacl or dacl, and three flag values are defined 🧚

const ACL4_AUTO_INHERIT         = 0x00000001;
const ACL4_PROTECTED            = 0x00000002;
const ACL4_DEFAULTED            = 0x00000004;

all other bits must be cleared. the ACE4_INHERITED_ACE flag may be set in the ACEs of the sacl or dacl, but it must always be cleared in the acl 🎀

all of this together lets a server support automatic inheritance, and here's how that works 👑

normally, inheritable ACEs only get inherited by child objects at the moment those children are created 🙈 changing an inheritable ACE later does not change the inherited ACEs on descendants.

BUT the dacl and sacl give you an OPTIONAL mechanism that lets a client application push changes to inheritable ACEs down a whole directory hierarchy 🌈

a server that supports this does inheritance at creation time the normal way, and it SHOULD set the ACE4_INHERITED_ACE flag on any inherited ACEs as they get added to the new object 💅

then a client application, like an ACL editor, can propagate changes to inheritable ACEs. it walks recursively through the directory's descendants and changes each ACL it finds. it removes any ACEs with the ACE4_INHERITED_ACE flag and replaces them with the new inheritable ACEs (which also have ACE4_INHERITED_ACE set). it uses the existing ACE inheritance flags in the obvious way to decide which ACEs to pass along. (heads up: while it's walking down the hierarchy it might run into more inheritable ACEs, and those have to be taken into account too when propagating to further descendants 🫶)

the reach of this propagation can be limited in two ways 🦋 first, automatic inheritance is not performed from any directory ACL that has the ACL4_AUTO_INHERIT flag cleared. second, automatic inheritance stops wherever an ACL has the ACL4_PROTECTED flag set. that stops that ACL from being modified, and also (if the ACL is on a directory) the ACL on any of the object's descendants 😭

this propagation runs independently for the sacl and the dacl attributes. so ACL4_AUTO_INHERIT and ACL4_PROTECTED can be set separately for the sacl and the dacl, and propagation of one type can keep going down a hierarchy even after propagation of the other has stopped 🍓

new objects should be created with a dacl and a sacl that both have ACL4_PROTECTED cleared, and with ACL4_AUTO_INHERIT set to the same value as on the parent object's sacl or dacl, respectively 💜

both the dacl and sacl attributes are RECOMMENDED, and a server may support one without supporting the other 🌸

a server that supports both the old acl attribute and one or both of the new dacl or sacl attributes must do it in a way that keeps all three attributes consistent with each other. so the ACEs reported in the acl attribute should be the union of the ACEs reported in the dacl and sacl attributes, except that ACE4_INHERITED_ACE must be cleared from the ACEs in the acl. and of course, a client that only queries the acl won't be able to figure out the values of the sacl or dacl flag fields 🙈

when a client does a SETATTR for the acl attribute, the server SHOULD set ACL4_PROTECTED to true on both the sacl and the dacl. using the acl attribute instead of dacl or sacl tells the server the client might not understand automatic inheritance, so it can't be trusted to set an ACL where automatic inheritance would make sense 💅

when a client application queries an ACL, changes it, and sets it again, it should leave any ACEs marked with ACE4_INHERITED_ACE alone, in their original order, at the end of the ACL. if the application can't do that, it should set the ACL4_PROTECTED flag. servers don't enforce this behavior, but breaking this rule may lead to unexpected results when applications do automatic inheritance ⭐

if a server also supports the mode attribute, it SHOULD set the mode in a way that leaves inherited ACEs unchanged, in their original order, at the end of the ACL. if it can't, it SHOULD set the ACL4_PROTECTED flag on the file's dacl 🌈

last one: if the request that creates a new file or directory doesn't also set permissions, and there are no ACEs to inherit from the parent directory, then the server's choice of ACL for the new object is implementation-dependent. in that case the server SHOULD set the ACL4_DEFAULTED flag on the ACL it picks. an application doing automatic inheritance reads ACL4_DEFAULTED as a sign that the ACL should be completely replaced by one generated with the automatic inheritance rules 👑

---

7. Single-Server Namespace ✨

this section is about the NFSv4 single-server namespace 🦋 single-server namespaces can be shown straight to clients, or they can be used as building blocks for bigger multi-server namespaces (like site-wide or organization-wide ones) that get shown to clients, as described in Section 11 🌸

7.1. Server Exports 🧚

on a UNIX server, the namespace covers every file you can reach by a pathname under the root directory, or "/" 💖 on a Windows server, the namespace is all the files on the disks with mapped disk letters 🎀 NFS server admins almost never make the entire server's file system namespace available to NFS clients. usually they only share parts of it through an "export" feature 💅 in earlier NFS versions, the root filehandle for each export was obtained through the MOUNT protocol. the client sent a string naming the export within the namespace, and the server sent back the root filehandle for that export 🍓 the MOUNT protocol also had an EXPORTS procedure that listed the server's exports 🫶

7.2. Browsing Exports 🗺️✨

Okay bestie, so the NFSv4.1 protocol gives clients a root filehandle 🌸, and clients can get filehandles for a server's exports by doing a series of LOOKUP operations in a COMPOUND to walk down a path. The vibe people expect is a file "Open" dialog where you click through folders one by one 📂. So the client must be able to move from one export to another via single-component, progressive LOOKUP operations.

This style of browsing is not well supported by NFSv3 🙈. In NFSv3, the client expects all LOOKUP operations to stay within a single server file system. For example, the device attribute will not change. That means a client can't take namespace paths that span exports, which is kind of a bummer 😭.

With NFSv3, an automounter on the client can grab a snapshot of the server's namespace using the EXPORTS procedure of the MOUNT protocol 📸. If it understands the server's pathname syntax, it can build an image of the server's namespace on the client. The parts that aren't exported get filled in with directories that might be built similarly to an NFSv4.1 "pseudo file system" (see Section 7.3), which lets the user browse from one mounted file system to another 🦋. The catch is that this copy is static. If the server administrator adds a new export, the client has no idea it exists 💀.

7.3. Server Pseudo File System 🧚

NFSv4.1 servers dodge that namespace inconsistency by presenting all the exports for a given server inside a single namespace for that server 💖. An NFSv4.1 client uses LOOKUP and READDIR operations to browse seamlessly from one export to another ✨.

Where parts of the server namespace aren't exported, clients need some way to traverse them to reach the actual exported file systems. One technique servers may use is to bridge the unexported portion of the namespace with a "pseudo file system" that shows only exported directories 🌈. A pseudo file system has a unique fsid and behaves like a normal, read-only file system.

Depending on how the server's namespace is built, there could be multiple pseudo file systems. For example:

           /a              pseudo file system
           /a/b            real file system
           /a/b/c          pseudo file system
           /a/b/c/d        real file system

Each of the pseudo file systems is considered a separate entity, so it MUST have its own fsid, unique among all the fsids for that server 👑.

7.4. Multiple Roots 🎀

Some operating environments are sometimes described as having "multiple roots" 💅. In those environments, individual file systems are usually represented by disk or volume names 💾. NFSv4 servers for these platforms can build a pseudo file system above those root names, so disk letters or volume names are just directory names in the pseudo root 🫶.

7.5. Filehandle Volatility 💧

The server's pseudo file system is a logical representation of the file system(s) available from the server, so it's most likely constructed dynamically when the server first starts up 🌸. It's expected that the pseudo file system may not have an on-disk counterpart to build persistent filehandles from. Even though it's preferable for the server to provide persistent filehandles for the pseudo file system, the NFS client should expect that pseudo file system filehandles are volatile 🙃. You can confirm this by checking the associated fh_expire_type attribute for those filehandles in question 🔍. If the filehandles are volatile, the NFS client must be prepared to recover a filehandle value (e.g., with a series of LOOKUP operations) when it receives an error of NFS4ERR_FHEXPIRED 🔁.

Since servers are quite likely to implement pseudo file systems with volatile filehandles, clients need to be ready for that, rather than assuming every filehandle is persistent 🍓.

7.6. Exported Root 🌈

If the server's root file system is exported, you might think a pseudo file system isn't needed. Not necessarily, bestie 🙅‍♀️. Say the server has these file systems:

           /       fs1  (exported)
           /a      fs2  (not exported)
           /a/b    fs3  (exported)

Because fs2 isn't exported, fs3 can't be reached with simple LOOKUPs 🚧. The server must bridge the gap with a pseudo file system 🌉.

7.7. Mount Point Crossing 🦋

A server's file system environment can be set up so that one file system has a directory that is 'covered' or mounted upon by a second file system 🧩. For example:

           /a/b            (file system 1)
           /a/b/c/d        (file system 2)

The pseudo file system for this server might look like this:

           /               (place holder/not exported)
           /a/b            (file system 1)
           /a/b/c/d        (file system 2)

It's the server's responsibility to present a complete pseudo file system to the client 👑. If the client sends a LOOKUP request for /a/b/c/d, the server responds with the filehandle of the root of the file system /a/b/c/d. In previous versions of the NFS protocol, the server would respond with the filehandle of directory /a/b/c/d within the file system /a/b 🕰️.

The NFS client can tell it has crossed a server mount point by a change in the value of the fsid attribute ✨.

7.8. Security Policy and Namespace Presentation 🔐

Because NFSv4 clients can change the security mechanisms they use (after figuring out what's allowed, using SECINFO and SECINFO_NONAME) 🕵️‍♀️, the server SHOULD NOT present a different view of the namespace based on the security mechanism a client is using. Instead, it should present a consistent view and return NFS4ERR_WRONGSEC if someone tries to access data with an inappropriate security mechanism 🚫.

If security concerns make it necessary to hide the existence of a particular file system (as opposed to all the data inside it), the server can apply the security policy of a shared resource in the server's namespace to the components of that resource's ancestors 🙈. For example:

           /                           (place holder/not exported)
           /a/b                        (file system 1)
           /a/b/MySecretProject        (file system 2)

The /a/b/MySecretProject directory is a real file system and it's the shared resource 🗝️. Say the security policy for /a/b/MySecretProject is Kerberos with integrity, and you want to limit knowledge of this file system's existence. In that case, the server should apply the same security policy to /a/b. This lets you keep the existence of a file system secured when that's what you want 🫣.

For the case of multiple, disjoint security mechanisms in the server's resources, applying that kind of policy would mean the higher-level file system can't be accessed using any security flavor 😵. So that kind of configuration isn't compatible with hiding the existence (as opposed to the contents) from clients that use multiple disjoint sets of security flavors 💔.

In other situations, a good policy is for the security of a particular object in the server's namespace to include the union of all security mechanisms of all its direct descendants 🧠. A common and convenient practice, unless strong security requirements say otherwise, is to make the entire pseudo file system accessible by all of the valid security mechanisms 🌸.

If there's concern about the security of data on the network, clients should use strong security mechanisms to access the pseudo file system, so they can prevent man-in-the-middle attacks 🚨. Seriously, don't skip this one 😤.

---

8. State Management 🎭

Integrating locking into the NFS protocol necessarily makes it stateful 🧠. With features like share reservations, file and directory delegations, recallable layouts, and support for mandatory byte-range locking, the protocol becomes way more dependent on proper state management than the traditional combo of NFS and NLM (Network Lock Manager) [46] 📚. These features include expanded locking facilities, which provide some measure of inter-client exclusion, but state also offers things that just aren't easy to provide with a stateless model ✨. There are three components to making this state manageable:

• clear division between client and server
• ability to reliably detect inconsistency in state between client and server
• simple and robust recovery mechanisms

In this model, the server owns the state information 👑. The client requests changes in locks, and the server responds with the changes it made. Changes to locking state that didn't come from the client are rare 🌙. The client gets prompt notification of those changes and can adjust its view of the locking state to match the server's changes 💜.

Individual pieces of state that the server creates and hands to the client on request are represented by 128-bit stateids 🔢. These stateids may represent a particular open file, a set of byte-range locks held by a particular owner, or a recallable delegation of privileges to access a file in particular ways or at a particular location 🦋.

In every case, there's a transition from the most general information that represents a client as a whole to the eventual lightweight stateid used for most client and server locking interactions 🪜. The details of this transition vary with the type of object, but it always starts with a client ID 🆔.

8.1. Client and Session ID 🪪

A client must establish a client ID (see Section 2.4) and then one or more sessionids (see Section 2.10) before performing any operations to open, byte-range lock, delegate, or obtain a layout for a file object 🔐. Each session ID is associated with a specific client ID, so it serves as a shorthand reference to an NFSv4.1 client 💅.

For some types of locking interactions, the client represents some number of internal locking entities called "owners", which normally correspond to processes inside the client 🧃. For other locking-related objects, like delegations and layouts, no such intermediate entities are provided, and those objects are considered to be transferred directly between the server and a unitary client 💖.

8.2. Stateid Definition ✨

okay so whenever the server grants a lock of any type (opens, byte-range locks, delegations, layouts, the whole squad 💅), it hands back a unique stateid. That stateid stands for a set of locks (often just one 🦋) on the same file, of the same type, with the same ownership vibes. So opens of the same file by different open-owners each get their own stateid, and every set of byte-range locks on a file owned by a specific lock-owner gets its own identifying stateid too. Delegations and layouts have stateids you can reference as well. Basically the stateid is your shorthand for a lock or a group of locks. Given a stateid, the server can figure out the associated state-owner or state-owners (for an open-owner/lock-owner pair, that's two) and the associated filehandle. And when you use a stateid, the current filehandle has to be the one that goes with that stateid. no exceptions bestie 🫶

All stateids tied to a given client ID are connected to one shared lease. That lease is basically the server's receipt saying "these stateids and the objects they point to are still claimed, keep them alive." See Section 8.3 for the lease tea ☕

The server can assign stateids independently for different clients. So the same bit pattern could mean totally different locks for a different client. 🙈 The stateid always gets read in terms of the client ID tied to the current session. Stateids apply to every session connected to that client ID, so a client can grab a stateid from one session and use it on another session with the same client ID. iconic flex 💖

8.2.1. Stateid Types

Apart from special stateids (see Section 8.3, no wait, see Section 8.2.3 🙃), each stateid stands for locking objects of one of the set of types defined by the NFSv4.1 protocol. Quick heads up: whenever we say "guarantee" here, just know there are situations like a client restart or a lock revocation that can void that guarantee. Plot twists happen 💀

• Stateids may represent opens of files.

Each stateid in this case represents the OPEN state for a given client ID/open-owner/filehandle triple. These stateids can change (with the seqid going up) when OPENs result in upgrades and OPEN_DOWNGRADE operations. glow-up era 🌸

• Stateids may represent sets of byte-range locks.

All locks held on a particular file by a particular owner, gotten under the aegis of a particular open file, hang out under one stateid. The seqid goes up whenever LOCK and LOCKU operations affect that set of locks. 🔥

• Stateids may represent file delegations, which are recallable guarantees by the server to the client that other clients will not reference or modify a particular file, until the delegation is returned. In NFSv4.1, file delegations may be obtained on both regular and non-regular files. 👑

A stateid represents a single delegation held by a client for a particular filehandle.

• Stateids may represent directory delegations, which are recallable guarantees by the server to the client that other clients will not modify the directory, until the delegation is returned. 🧚

A stateid represents a single delegation held by a client for a particular directory filehandle.

• Stateids may represent layouts, which are recallable guarantees by the server to the client that particular files may be accessed via an alternate data access protocol at specific locations. That access is limited to particular sets of byte-ranges and may keep going until those byte-ranges are reduced or the layout is returned. 🦋

A stateid represents the set of all layouts held by a particular client for a particular filehandle with a given layout type. The seqid gets updated as the layouts of that set of byte-ranges change, through layout stateid changing operations like LAYOUTGET and LAYOUTRETURN. so much movement 😭

8.2.2. Stateid Structure

Stateids are split into two fields: a 96-bit "other" field that identifies the specific set of locks, and a 32-bit "seqid" sequence value. Except for special stateids (see Section 8.2.3), a particular value of the "other" field denotes a set of locks of the same type (byte-range locks, opens, delegations, or layouts), for a specific file or directory, sharing the same ownership characteristics. The seqid picks out one specific instance of that set, and it goes up to show changes in the set, like adding or deleting locks, changing the byte-range they cover, or upgrading or downgrading the type of one or more locks. It's the version number, basically 🎀

When a set of locks is first created, the server hands back a stateid with a seqid value of one. Then, on later operations that modify the set, the server is required to increment the "seqid" field by one whenever it returns a stateid for the same state-owner/file/type combination AND there's some change in the set of locks actually designated. In that case the server returns a stateid with the same "other" field that was used before for that state-owner/file/type combination, but with an incremented "seqid" field. This keeps going until the seqid gets incremented past NFS4_UINT32_MAX, and then one (not zero) is the next seqid value. the loop continues, not me reset-ing to zero 💅

Why bother incrementing the seqid? It lets the server tell the client what order the operations that changed the locking state were processed in, and it lets the client send requests that only go through if the set of locks hasn't changed since the stateid was returned. It's giving order and receipts 🧾

Except for layout stateids (Section 12.5.3), when a client sends a stateid to the server, it has two choices for the seqid it sends. It can set the seqid to zero, which tells the server "just give me the most up-to-date seqid for that stateid's 'other' field." This is the usual move for a stateid sent with a READ or WRITE operation. Or it can set a non-zero value, and then the server checks whether that seqid is the correct one. In that case the server is required to return NFS4ERR_OLD_STATEID if the seqid is lower than the most current value, and NFS4ERR_BAD_STATEID if the seqid is greater than the most current value. This is the usual move for stateids sent with a CLOSE or OPEN_DOWNGRADE.

Here's the drama: because OPENs can be sent in parallel for the same owner, a client might close a file without knowing the server had already done an OPEN upgrade that changed the lock in question. If the CLOSE went out with a zero seqid, the OPEN upgrade would get cancelled before the client even heard that an upgrade happened. 😭 that's why the non-zero seqid is the move for CLOSE

When the server sends a stateid to the client as part of a callback operation, it doesn't get checked for a current seqid, and NFS4ERR_OLD_STATEID doesn't get returned for it. That's because the client isn't in a position to know the most up-to-date seqid, so it can't verify it. Unless specially noted, the seqid value for a stateid the server sends as part of a callback is required to be zero, and NFS4ERR_BAD_STATEID gets returned if it isn't. 🙃

When comparing seqids, both the client (figuring out the order of operations) and the server (deciding whether to return NFS4ERR_OLD_STATEID) have to account for the seqid wrapping around past NFS4_UINT32_MAX. Here's how: when two seqid values are being compared, the total count of slots for all sessions associated with the current client is used. If one seqid value is less than this total slot count, and another seqid value is greater than NFS4_UINT32_MAX minus the total slot count, the former gets treated as lower than the latter, even though it's numerically greater. Wild, right? it's literally a time-warp 🌈

8.2.3. Special Stateids

Stateid values whose "other" field is all zeros or all ones are reserved. The server may not assign them, but they have special meanings defined by the protocol. The meaning depends on whether the "other" field is all zeros or all ones, and on the specific value of the "seqid" field. ✨

The following combinations of "other" and "seqid" are defined in NFSv4.1:

• When "other" and "seqid" are both zero, the stateid is treated as a special anonymous stateid. It can be used in READ, WRITE, and SETATTR requests to indicate the absence of any OPEN state associated with the request. When an anonymous stateid value is used and an existing open denies the form of access requested, access is denied to the request. This stateid MUST NOT be used on operations to data servers (Section 13.6). 🙅‍♀️
• When "other" and "seqid" are both all ones, the stateid is a special READ bypass stateid. When this value is used in WRITE or SETATTR, it gets treated like the anonymous value. When used in READ, the server MAY grant access, even if access would normally be denied to READ operations. This stateid MUST NOT be used on operations to data servers. 🦋
• When "other" is zero and "seqid" is one, the stateid represents the current stateid, which is whatever value was the last stateid returned by an operation within the COMPOUND. In the case of an OPEN, the stateid returned for the open file is used, not the one for the delegation. The stateid passed to the operation in place of the special value has its "seqid" set to zero, except when the current stateid is used by the operation CLOSE or OPEN_DOWNGRADE. If no operation in the COMPOUND has returned a stateid value, the server MUST return the error NFS4ERR_BAD_STATEID. As illustrated in Figure 6, if the value of a current stateid is a special stateid, and the stateid of an operation's arguments has "other" set to zero and "seqid" set to one, the server MUST return the error NFS4ERR_BAD_STATEID. 💀 this one's a whole plot
• When "other" is zero and "seqid" is NFS4_UINT32_MAX, the stateid represents a reserved stateid value that is defined to be invalid. When this stateid is used, the server MUST return the error NFS4ERR_BAD_STATEID. 🚫

If a stateid value is used that has all zeros or all ones in the "other" field but doesn't match any of the cases above, the server MUST return the error NFS4ERR_BAD_STATEID. no vibes allowed 🙅‍♀️

Special stateids, unlike other stateids, aren't tied to individual client IDs or filehandles. They can be used with all valid client IDs and filehandles. For a special stateid that designates the current stateid, though, the current stateid value that gets substituted is associated with a particular client ID and filehandle. So if it's used where the current filehandle doesn't match the one tied to the current stateid, the operation the stateid is passed to will return NFS4ERR_BAD_STATEID. the filehandle has to match the vibe 🎀

8.2.4. Stateid Lifetime and Validation ✨

okay bestie, let's talk about how long a stateid actually lives 💅 A stateid must stay valid until either a client restart or a server restart happens, or until the client gives back all of the locks tied to it with an operation like CLOSE or DELEGRETURN. 🎀

If the locks get yanked because of revocation, the stateid is still a valid designation of that revoked state as long as the client ID is valid. It stays that way until the client frees it with FREE_STATEID 🦋 Byte-range lock stateids are the exception 👀 They stay valid even if a LOCKU frees all the remaining locks, as long as the open file they belong to stays open, unless the client frees them with FREE_STATEID.

It's worth noting that the client's locks can go invalid without the client asking for them back 😭 That includes lease expiration and a number of revocation types that happen within the lease period. In those cases the stateid is still valid, and the client can use it to figure out what happened to the lost locks 🫶

An "other" value must never be reused for a different purpose (different filehandle, owner, or type of locks) within the same client ID 🚫 A server may keep an "other" value for the same purpose even after it could have freed it, but if it does, it must maintain "seqid" continuity with the previous values 🌸

One mechanism for making sure the server can spot invalid and out-of-date stateids is to split the "other" field into two parts:

• an index into a table of locking-state structures.
• a generation number that is incremented on each allocation of a table entry for a particular use.

Then each table entry stores:

• the client ID with which the stateid is associated.
• the current generation number for the (at most one) valid stateid sharing this index value.
• the filehandle of the file on which the locks are taken.
• an indication of the type of stateid (open, byte-range lock, file delegation, directory delegation, layout).
• the last "seqid" value returned corresponding to the current "other" value.
• an indication of the current status of the locks associated with this stateid, in particular, whether these have been revoked and if so, for what reason.

With all of that info, an incoming stateid can be validated and the right error returned when needed 🔥 Special and non-special stateids get handled separately (see Section 8.2.3 for the special stateid tea ☕).

Stateids are implicitly qualified by the current client ID, which comes from the client ID tied to the current session. The semantics of the session prevent stateids from a previous client or server instance from being analyzed by this procedure 🧚

If a server restart leaves you with an invalid client ID or an invalid session ID, SEQUENCE returns an error, and the operation that takes a stateid as an argument never gets processed 💀

If there's been a server restart with a persistent session and all leased state is gone, the session is still valid but gets marked as dead. Any operation not satisfied from the reply cache gets NFS4ERR_DEADSESSION and doesn't get processed the way described below 🙈

When a stateid is being tested and the "other" field is all zeros or all ones, the server checks that the "other" and "seqid" fields match a defined combination for a special stateid, and the results go like this:

• If the "other" and "seqid" fields don't match a defined combination for a special stateid, NFS4ERR_BAD_STATEID gets returned.
• If the special stateid designates the current stateid and there is a current stateid, the current stateid gets swapped in for the special stateid, and the checks for non-special stateids get done.
• If the combination is valid in general but doesn't fit the context where the stateid is used (for example, an all-zero stateid is used when an OPEN stateid is required in a LOCK operation), NFS4ERR_BAD_STATEID is also returned.
• Otherwise, the check is complete and the special stateid is accepted as valid ✅

When a stateid is being tested and the "other" field is neither all zeros nor all ones, the following procedure could be used to validate an incoming stateid and return the right error when needed. This assumes the "other" field is split into a table index and an entry generation:

• If the table index field is outside the range of the table, return NFS4ERR_BAD_STATEID.
• If the selected table entry has a different generation than the one in the incoming stateid, return NFS4ERR_BAD_STATEID.
• If the selected table entry doesn't match the current filehandle, return NFS4ERR_BAD_STATEID.
• If the client ID in the table entry doesn't match the client ID associated with the current session, return NFS4ERR_BAD_STATEID.
• If the stateid represents revoked state, return NFS4ERR_EXPIRED, NFS4ERR_ADMIN_REVOKED, or NFS4ERR_DELEG_REVOKED, whichever fits.
• If the stateid type isn't valid for the context it appears in, return NFS4ERR_BAD_STATEID. A stateid can be valid in general (the kind of thing TEST_STATEID would report) and still be invalid for a particular operation. For example, a stateid that doesn't represent byte-range locks gets passed to the non-from_open case of LOCK or to LOCKU, or a stateid that doesn't represent an open gets passed to CLOSE or OPEN_DOWNGRADE. In cases like these, the server MUST return NFS4ERR_BAD_STATEID 🚨
• If the "seqid" field is not zero and it is greater than the current sequence value for the current "other" field, return NFS4ERR_BAD_STATEID.
• If the "seqid" field is not zero and it is less than the current sequence value for the current "other" field, return NFS4ERR_OLD_STATEID.
• Otherwise, the stateid is valid, and the table entry should hold any extra info about the stateid type and anything tied to that type. That includes the associated set of locks (like open-owner and lock-owner info) and the specifics of those locks (like open modes and byte-ranges) 🎀

---

8.2.5. Stateid Use for I/O Operations 🌈

clients doing I/O need to pick the right stateid based on the locks they hold (including opens and delegations) and the different kinds of state-owners sending the I/O requests 💅 SETATTR operations that change the file size get treated like I/O operations here too.

These rules, applied in order from highest to lowest priority, control which stateid gets picked. While following them, the client only considers locks it has actually been told about through an appropriate operation response or callback 👀 Heads up: the rules are a bit different for I/O to data servers when file layouts are in use (see Section 13.9.1).

• If the client holds a delegation for the file in question, the delegation stateid SHOULD be used.
• Otherwise, if the entity behind the lock-owner (like a process) sending the I/O has a byte-range lock stateid for the open file, then the byte-range lock stateid for that lock-owner and open file SHOULD be used.
• If there's no byte-range lock stateid, then the OPEN stateid for the open file in question SHOULD be used.
• Finally, if none of the above apply, then a special stateid SHOULD be used.

Ignoring these rules can leave the server without the info it needs to process the request properly 😬 For example, when mandatory byte-range locks are in effect, if the stateid doesn't show the proper lock-owner through a lock stateid, a request might get rejected when it didn't have to be.

The server, however, should not try to enforce these ordering rules. It should use whatever info is available to process I/O requests properly 🧚 In particular, when a client has a delegation for a given file, it SHOULD take note of that fact while processing a request, even if the request is sent with a special stateid 🫶

---

8.2.6. Stateid Use for SETATTR Operations 💖

every operation is tied to a session ID, and from that the clientid can be figured out. So operations don't need to include a stateid for the server to decide whether they should cause a delegation to be recalled or be treated as done within the scope of the delegation 🦋

SETATTR operations do include a stateid. For cases other than those that set the file size, the client may send either a special stateid or, when it holds a delegation for the file in question, the delegation stateid. The server SHOULD validate the stateid, and it may use the stateid to speed up figuring out whether a delegation is held. Even when a special stateid is sent, it SHOULD note that a delegation is present, and it MUST accept a valid delegation stateid when one is sent 🔥

8.3. Lease Renewal 🦋

okay so every client/server pair (aka one client ID) gets exactly one lease 🎀 the whole point is giving the client a super low-overhead way to say "hey, still alive!" so the server knows to keep its locks ✨ if a client crashes or gets unreachable, the server can clean up its locking stuff once the lease expires 💀 that way other clients aren't stuck waiting forever on locks some ghost is hoarding 👻 and just so we're clear, a lease is not a cache consistency thing 🙅‍♀️ also, lease renewals may not be denied if the lease interval hasn't expired yet.

since every session belongs to one specific client (the client ID), any operation sent on that session proves the client is reachable 💅 when a request goes out on a session, a successful SEQUENCE (or a successful grab of the SEQUENCE result from the reply cache) on an unexpired lease implicitly renews it, for the standard renewal period. that period is equal to the lease_time attribute 🫶

here's the renewal rule, bestie 📜

• if the client ID's lease has not expired when the server gets a SEQUENCE operation, the server MUST renew the lease 💖
• if the lease has expired when the server gets a SEQUENCE, the server MAY renew it. it depends on whether any state got revoked because the client failed to renew before expiration 🤷‍♀️

absent other activity that would renew the lease, a COMPOUND with just one SEQUENCE operation will do the trick ✨ the client should also factor in communication delays and make sure renewal messages actually reach the server in good time 🕰️ for example:

• when trunking is in effect, the client should consider sending multiple requests on different connections. that way renewal still happens even if the path for one of those connections gets blocked 🚧
• transport retransmission delays can get so big they approach or exceed the lease period 😭 this is especially likely when the server is unresponsive because it restarted (see Section 8.4.2.1). if the client implementation isn't careful, retransmission delays can make it miss a server restart before the grace period ends 💀 the scenario goes like this: the client uses a transport with exponential backoff, so the maximum retransmission timeout is bigger than both the grace period and lease_time. a network partition makes the connection's retransmission interval back off, and even after the partition heals, the next retransmission goes out after the server has restarted and its grace period has ended 🥲

the client MUST either recover from the ensuing NFS4ERR_NO_GRACE errors, or it MUST make sure that, even with retransmission intervals longer than lease_time, a SEQUENCE operation gets sent to renew the lease before expiration 🎯 one way to do that is associating a new connection with the session and sending a SEQUENCE on it. but if setting up that new connection gets delayed (say, by exponential backoff on the connection establishment packets), the client will have to abort the connection establishment attempt before the lease expires, and then try to reconnect 🔄

if the server renews the lease upon receiving a SEQUENCE operation, the server MUST NOT let the lease expire while the rest of the operations in the COMPOUND request are still running 🙅‍♀️

once the last operation finishes and the response to COMPOUND has been sent, the server MUST set the lease to expire no sooner than the sum of the current time and the value of the lease_time attribute ⏳

a client ID's lease can expire once it has been at least the lease interval (lease_time) since the last lease-renewing SEQUENCE was sent on any of the client ID's sessions, and there are no active COMPOUND operations on any of those sessions 🍂

SEQUENCE is the basic way to renew a lease, and it has to happen at least once per lease period, so it's also the natural way for the server to tell the client about lease status changes 📣 the client should inspect the status flags (sr_status_flags) returned by SEQUENCE and take the appropriate action (see Section 18.46.3 for details) 💅

• the bits SEQ4_STATUS_CB_PATH_DOWN and SEQ4_STATUS_CB_PATH_DOWN_SESSION flag problems with the backchannel that the client may need to fix so it can receive callback requests 📞
• the bits SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING and SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED flag problems with GSS contexts or RPCSEC_GSS handles for the backchannel. the client might have to address those so callback requests can be sent 🔐
• the bits SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED, SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, SEQ4_STATUS_ADMIN_STATE_REVOKED, and SEQ4_STATUS_RECALLABLE_STATE_REVOKED tell the client about lock revocation events 🚨 when they're set, the client should use TEST_STATEID to find out which stateids were revoked, and FREE_STATEID to acknowledge the loss of that state 🧹
• the bit SEQ4_STATUS_LEASE_MOVE means responsibility for lease renewal has moved to one or more new servers 🧚
• the bit SEQ4_STATUS_RESTART_RECLAIM_NEEDED means the server restarted, so the client has to reclaim its locking state 🔁
• the bit SEQ4_STATUS_BACKCHANNEL_FAULT means the server ran into an unrecoverable backchannel fault (for example, it lost track of a sequence ID for a backchannel slot) 🙈

---

8.4. Crash Recovery 💥

the big critical requirement here is that the client and the server both know when the other one has failed 👀 on top of that, a client needs to see a consistent view of data across server restarts. every READ and WRITE that might be queued up in the client or network buffers has to wait until the client has successfully recovered the locks protecting those READ and WRITE operations 🔒 anything that reaches the server before the server can safely tell that the client has recovered enough locking state must be rejected 🚫 that happens for one of two reasons:

• the state presented is no longer valid, because it's tied to a client ID that's now invalid. in that case the client gets NFS4ERR_BADSESSION or NFS4ERR_DEADSESSION, and trying to attach a new session to that invalid client ID gets NFS4ERR_STALE_CLIENTID 💔
• recovering the locks might make the operation inappropriate, so it gets NFS4ERR_GRACE 🥀

8.4.1. Client Failure and Recovery 🫠

if a client fails, the server may release the client's locks once the associated lease has expired. conflicting locks from another client can only be granted after that lease expiration ⏰ like we covered in Section 8.3, if a client hasn't failed and re-establishes its lease before expiration, requests for conflicting locks won't be granted 🙅‍♀️

to keep client delay low on restart, lock requests get associated with an instance of the client through a client-supplied verifier 🧾 this verifier is part of the client_owner4 sent in the first EXCHANGE_ID call the client makes. the server returns a client ID as the result of EXCHANGE_ID 🪪 then the client confirms the client ID by establishing a session tied to it (see Section 18.36.3 for how that's done) ✅ all locks (opens, byte-range locks, delegations, and layouts) obtained by sessions using that client ID are associated with that client ID 💜

since the client changes the verifier on every initialization, the server can compare a new verifier to the one attached to currently held locks and see that they don't match 🕵️‍♀️ that means the client has started a new instance and lost its locking state (once the new client ID is confirmed). so the server is free to release all locks held under the old client ID, which came from the old verifier 🗑️ at that point, conflicting locks from other clients that were kept waiting while the lease hadn't expired yet can finally be granted 🎉 also, all stateids associated with the old client ID can be freed, since they can no longer be referenced 🧹

note that the verifier has to have the same uniqueness properties as the verifier for the COMMIT operation 🔍

8.4.2. Server Failure and Recovery 🌪️

if the server loses locking state (usually because it restarted), it needs to give clients time to figure that out and re-establish the lost locking state ⏳ clients have to be able to re-establish that state without the server denying valid requests just because it granted conflicting access to another client 🙏 same goes if there's a chance that clients haven't re-established their locking state for a file yet and that state could make READ or WRITE operations invalid. for example, if mandatory locks are a possibility, the server must disallow READ and WRITE operations for that file 🚫

a client can figure out that locking state was lost in a few different ways 🕵️‍♀️

1. when a SEQUENCE (the most common) or another operation returns NFS4ERR_BADSESSION, it could mean the session was destroyed but the client ID is still valid. the client sends a CREATE_SESSION request with the client ID to re-establish the session. if CREATE_SESSION fails with NFS4ERR_STALE_CLIENTID, the client must establish a new client ID (see Section 8.1) and re-establish its lock state with the new client ID once the CREATE_SESSION operation succeeds (see Section 8.4.2.1) 💫
2. when a SEQUENCE (the most common) or another operation on a persistent session returns NFS4ERR_DEADSESSION, it means that session isn't usable for new operations anymore (ones not satisfied from the reply cache) 🪦 once all pending operations are sorted out as either performed before the retry or not performed at all, the client sends a CREATE_SESSION request with the client ID to re-establish the session. if CREATE_SESSION fails with NFS4ERR_STALE_CLIENTID, the client must establish a new client ID (see Section 8.1) and re-establish its lock state after CREATE_SESSION succeeds with the new client ID (Section 8.4.2.1) 🌷
3. when an operation that's neither SEQUENCE nor preceded by SEQUENCE (for example, CREATE_SESSION or DESTROY_SESSION) returns NFS4ERR_STALE_CLIENTID, the client MUST establish a new client ID (Section 8.1) and re-establish its lock state (Section 8.4.2.1) 🚨

8.4.2.1. State Reclaim ✨

okay so, when a server restarts and the state info and locks go poof 💨, the protocol must provide a way to get that state back. The move is to define, for most types of locking state (layouts are the exception 🙅‍♀️), a request whose whole job is letting a client re-establish a lock it first got from a previous instance. These are usually variants of the requests you'd normally use to create locks of that type, and they're called "reclaim-type" requests. The process of re-establishing those locks is called "reclaiming" them 💅

Every client needs a chance to reclaim all of its locks without some other client grabbing a conflicting one in the meantime, so there's a "grace period" devoted to the reclaim process 🫧 During it, requests that create client IDs and sessions get handled normally, but locking requests have special restrictions. Only reclaim-type locking requests are allowed, unless the server can reliably figure out (through state kept persistently across restart instances) that granting any such lock cannot possibly conflict with a later reclaim. If a new lock request (not reclaim-type) shows up during the grace period and that can't be determined, the server must return the error NFS4ERR_GRACE 🚫

Once a session is up with the new client ID, the client uses reclaim-type locking requests (like LOCK operations with reclaim set to TRUE, and OPEN operations with a claim type of CLAIM_PREVIOUS; see Section 9.11) to rebuild its locking state. When that's done, or if there's no locking state to reclaim at all, the client sends a global RECLAIM_COMPLETE operation, meaning one with the rca_one_fs argument set to FALSE, to say it has reclaimed everything it's going to reclaim ✨ After sending that, the client can try non-reclaim locking operations, though it might still get NFS4ERR_GRACE from each one until the special handling is over. For the analogous lock reclamation when file systems move from server to server, check out Section 11.7.7 🧚

During the grace period, the server must reject READ and WRITE operations and non-reclaim locking requests (that's the other LOCK and OPEN operations) with NFS4ERR_GRACE, unless it can guarantee they can be done safely, as described below 🙅‍♀️

The grace period can last until every client that might have had locks has sent a global RECLAIM_COMPLETE, saying they're done reclaiming what they held before the restart. So a client that already sent RECLAIM_COMPLETE still has to be ready to get NFS4ERR_GRACE when it tries to acquire new locks ⏳ To know that every client with possible prior lock state has done that, the server must keep a list of the clients that may have such locks in stable storage 📋 The server may also end the grace period before every client has sent a global RECLAIM_COMPLETE. The server SHOULD NOT end the grace period before a time equal to the lease period, so clients get an opportunity to find out about the restart, since they send requests on their sessions at a frequency governed by the lease time. Heads up though 😬: if a client doesn't send those requests (or they're sent but never received by the server), the grace period could expire before that client finds out the server restarted.

Some additional time may be added to the lease time so a client can establish a new client ID and session and do its lock reclaims 🫶 Analogous rules apply to file system-specific grace periods, which are discussed in Section 11.7.7.

If the server can reliably determine that granting a non-reclaim request won't conflict with other clients reclaiming locks, NFS4ERR_GRACE doesn't have to come back, even inside the grace period. But NFS4ERR_GRACE must always be returned to clients attempting a non-reclaim lock request before they've done their own global RECLAIM_COMPLETE. For the server to handle READ and WRITE operations during the grace period, it must again be able to guarantee that no possible conflict could arise between a potential reclaim locking request and the READ or WRITE. If the server can't offer that guarantee, NFS4ERR_GRACE must be returned to the client 🥺

The easiest route to simple, valid grace period handling is to reject all non-reclaim locking requests and all READ and WRITE operations by returning NFS4ERR_GRACE. However, a server can keep information about granted locks in stable storage. With that info, the server could work out whether a locking, READ or WRITE operation can be safely processed 🧠

For example, if the server kept summary info in stable storage about whether mandatory locks exist (either mandatory byte-range locks, or share reservations specifying deny modes), lots of requests could be allowed during the grace period. If it's known that no such share reservations exist, OPEN requests that don't specify deny modes can be safely granted. And if it's also known that no mandatory byte-range locks exist (either through info stored on stable storage, or simply because the server doesn't support such locks), READ and WRITE operations can be safely processed during the grace period. Another important case: when it's known that no mandatory byte-range locks exist (either because the server doesn't provide support for them, or because their absence is known from persistently recorded data), READ and WRITE operations specifying stateids derived from reclaim-type operations can be validly processed during the grace period. That's because the valid reclaim ensures that no lock granted later can prevent the I/O 💖

To reiterate, for a server that allows non-reclaim lock and I/O requests to be processed during the grace period, it MUST determine that no lock subsequently reclaimed will be rejected, and that no lock subsequently reclaimed would have prevented any I/O operation processed during the grace period. 🚨

Clients should be prepared for the return of NFS4ERR_GRACE errors for non-reclaim lock and I/O requests. In that case, the client should employ a retry mechanism for the request. A delay (on the order of several seconds) between retries should be used, to avoid overwhelming the server. Further discussion of the general issue is included in [47]. The client must account for servers that can perform I/O and non-reclaim locking requests within the grace period, as well as those that cannot do so 🫶

A reclaim-type locking request outside the server's grace period can only succeed if the server can guarantee that no conflicting lock or I/O request has been granted since restart.

A server may, upon restart, establish a new value for the lease period. So once a new client ID is established, clients should refetch the lease_time attribute and use it as the basis for lease renewal for the lease associated with that server. However, the server must establish, for this restart event, a grace period at least as long as the lease period for the previous server instantiation. This allows the client state obtained during the previous server instance to be reliably re-established 🔄

The client might end up communicating with a server different from the one where the locks were obtained, because of server configuration events. That's shown by the combination of eir_server_scope and eir_server_owner. This raises the question of if and when the client should attempt to reclaim locks that were previously obtained on what's now being reported as a different server. The rules to resolve this are as follows:

• If the server scope is different, the client should not attempt to reclaim locks. In this situation, no lock reclaim is possible. Any attempt to re-obtain the locks with non-reclaim operations is problematic, since there is no guarantee that the existing filehandles will be recognized by the new server, or that if recognized, they denote the same objects. It is best to treat the locks as having been revoked by the reconfiguration event.
• If the server scope is the same, the client should attempt to reclaim locks, even if the eir_server_owner value is different. In this situation, it is the responsibility of the server to return NFS4ERR_NO_GRACE if it cannot provide correct support for lock reclaim operations, including the prevention of edge conditions.

The eir_server_owner field is not used in making this determination. Its function is to specify trunking possibilities for the client (see Section 2.10.5), and not to control lock reclaim 🦋

8.4.2.1.1. Security Considerations for State Reclaim 🔐

During the grace period, a client can reclaim state that it believes or asserts it had before the server restarted. Unless the server maintained a complete record of all the state the client had, the server has little choice but to trust the client. (Of course, if the server maintained a complete record, it wouldn't have to force the client to reclaim state after a server restart 🙄) While the server has to trust the client to tell the truth, that trust doesn't have any negative consequences for security. The fundamental rule for the server when processing reclaim requests is that it MUST NOT grant the reclaim if an equivalent non-reclaim request would not be granted during steady state due to access control or access conflict issues. For example, an OPEN request during a reclaim will be refused with NFS4ERR_ACCESS if the principal making the request does not have access to open the file according to the discretionary ACL (Section 6.2.2) on the file.

Nonetheless, it is possible that a client operating in error or maliciously could, during reclaim, prevent another client from reclaiming access to state. For example, an attacker could send an OPEN reclaim operation with a deny mode that prevents another client from reclaiming the OPEN state it had before the server restarted. The attacker could perform the same denial of service during steady state prior to server restart, as long as the attacker had permissions. Given that the attack vectors are equivalent, the grace period does not offer any additional opportunity for denial of service, and any concerns about this attack vector, whether during grace or steady state, are addressed the same way: use RPCSEC_GSS for authentication and limit access to the file only to principals that the owner of the file trusts. 🚨

Note that if, prior to restart, the server had client IDs with the EXCHGID4_FLAG_BIND_PRINC_STATEID (Section 18.35) capability set, then the server SHOULD record in stable storage the client owner and the principal that established the client ID via EXCHANGE_ID. If the server does not, there is a risk that a client will be unable to reclaim state if it does not have a credential for a principal that was originally authorized to establish the state 🙏

8.4.3. Network Partitions and Recovery 🌨️

Okay so bestie, picture this: a network partition lasts longer than the lease period the server gave you 😭. The server never gets a lease renewal from the client, and that's the moment things get messy. The server can either free all the locks held for that client, or let the lock state hang around for a while 🕰️. There's a catch, though. If someone asks for a conflicting lock, locks tied to an expired lease don't block that new lock from being granted. They MUST be revoked as necessary so they don't get in the way of the conflicting requests. Lowkey ruthless, but fair 💅✨.

If the server waits to free lock state until there's a conflict, it has two options. It can free all of the client's locks once the conflict shows up, or it can revoke only the minimum set of locks needed to let the conflicting requests through 🎀. If it goes with the finer-grained approach, it must revoke all locks associated with a given stateid, even when the conflict only involves a subset of them. Main character energy for the stateid, no exceptions 👑.

When the server frees all of a client's lock state, whether that happens right when the lease expires or because of the first attempt to get a conflicting lock, it can tell the client about the loss in a few different ways 🦋.

The server may choose to invalidate the session and the associated client ID. In that case, once the client can talk to the server again, it gets NFS4ERR_BADSESSION 💀. If it tries to create a new session, it gets NFS4ERR_STALE_CLIENTID. After the client creates a new client ID and new session, it'll try to reclaim its locks. Normally the server won't let it, because the server won't be in its recovery grace period 🙈.

Another option is for the server to keep the session and client ID, but make all the stateids held by the client invalid or stale. Once the client can reach the server after the partition, the status returned by the SEQUENCE operation will show that locking state was lost. The flag SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED will be set in sr_status_flags. On top of that, any I/O the client sends with those now-invalid stateids will fail with NFS4ERR_EXPIRED 😤. Once the client finds out about the lost locks, it should let the apps that held them know. Then it should clean up the invalidated stateids, either by setting up a new client ID with a new verifier or by sending a FREE_STATEID operation for each invalidated stateid 🧹.

When the server takes the finer-grained route to revoking locks after a lease expires, usually only some stateids become invalid during the partition. Once the client can reach the server again, the SEQUENCE status will show a partial loss of locking state with SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED 🌗. Operations with invalid stateids, including I/O, will fail with NFS4ERR_EXPIRED. After the client learns about the loss, it uses the TEST_STATEID operation on all of its stateids to figure out which locks were lost, then tells the applications that held those locks. The client can then release the invalidated locking state and confirm the revocation of those locks by sending a FREE_STATEID operation for each invalidated stateid 🫶.

When a network partition happens at the same time as a server restart, there are edge conditions that the server has to handle, or it could end up with silent data corruption after the restart 🥶. Two of these edge conditions are known, and we'll go through them below.

The first edge condition comes from a sequence of events like this:

1. Client A acquires a lock.
2. Client A and server experience mutual network partition, such that client A is unable to renew its lease.
3. Client A's lease expires, and the server releases the lock.
4. Client B acquires a lock that would have conflicted with that of client A.
5. Client B releases its lock.
6. Server restarts.
7. Network partition between client A and server heals.
8. Client A connects to a new server instance and finds out about server restart.
9. Client A reclaims its lock within the server's grace period.

So at that last step, the server has wrongly granted client A's lock reclaim 😱. If client B changed the object that the lock was protecting, client A ends up with corrupted data. Yikes.

The second known edge condition comes from situations like this one:

1. Client A acquires one or more locks.
2. Server restarts.
3. Client A and server experience mutual network partition, such that client A is unable to reclaim all of its locks within the grace period.
4. Server's reclaim grace period ends. Client A has either no locks or an incomplete set of locks known to the server.
5. Client B acquires a lock that would have conflicted with a lock of client A that was not reclaimed.
6. Client B releases the lock.
7. Server restarts a second time.
8. Network partition between client A and server heals.
9. Client A connects to new server instance and finds out about server restart.
10. Client A reclaims its lock within the server's grace period.

Same as with the first edge condition, the final step of this scenario has the server wrongly granting client A's lock reclaim 🙃.

To solve the first and second edge conditions, the server has to do one of two things. Either it always assumes after a restart that some edge condition has happened, so it returns NFS4ERR_NO_GRACE for every reclaim attempt, or it records some information in stable storage 🗂️. How much the server records in stable storage is inversely proportional to how harsh it plans to be whenever edge conditions come up. A server that tolerates every edge condition would record in stable storage every lock that's acquired, and only remove that lock record once the lock is released. For the two edge conditions above, the harshest a server can be, while still supporting a grace period for reclaims, means recording some minimal information in stable storage. For example, a server implementation could save the following record in stable storage for each client:

• the co_ownerid field from the client_owner4 presented in the EXCHANGE_ID operation.
• a boolean that indicates if the client's lease expired or if there was administrative intervention (see Section 8.5) to revoke a byte-range lock, share reservation, or delegation and there has been no acknowledgment, via FREE_STATEID, of such revocation.
• a boolean that indicates whether the client may have locks that it believes to be reclaimable in situations in which the grace period was terminated, making the server's view of lock reclaimability suspect. The server will set this for any client record in stable storage where the client has not done a suitable RECLAIM_COMPLETE (global or file system-specific depending on the target of the lock request) before it grants any new (i.e., not reclaimed) lock to any client.

With that record keeping in place, here's how the first edge condition plays out. After the server restarts, the record showing that client A's lease expired means another client could have acquired a conflicting byte-range lock, share reservation, or delegation. So the server must reject a reclaim from client A with the error NFS4ERR_NO_GRACE 🚫.

For the second edge condition, after the server restarts a second time, the record showing that the client hadn't finished its reclaims when the grace period ended means the server must reject a reclaim from client A with the error NFS4ERR_NO_GRACE 🚫.

When either edge condition happens, the client's attempt to reclaim locks results in NFS4ERR_NO_GRACE. Once that comes back, or after the client restarts with no lock state, the client sends a global RECLAIM_COMPLETE. When the server receives that RECLAIM_COMPLETE, the server and client agree again on which locks are reclaimable, and both booleans in persistent storage can be reset. They only get set again when a later event makes lock reclaim operations questionable 🌈.

No matter what level or approach the server uses for record keeping, the server MUST implement one of the following strategies, which apply to reclaims of share reservations, byte-range locks, and delegations:

1. Reject all reclaims with NFS4ERR_NO_GRACE. This is extremely unforgiving, but it's necessary if the server doesn't record lock state in stable storage 🥀.
2. Record enough state in stable storage that all known edge conditions involving server restart, including the two noted in this section, are detected. It's acceptable to wrongly recognize an edge condition and not allow a reclaim, when, with sufficient knowledge, the reclaim would have been allowed. The error the server would return in this case is NFS4ERR_NO_GRACE. Note that it is not known if there are other edge conditions 🤷‍♀️.

In the event that, after a server restart, the server determines there is unrecoverable damage or corruption to the information in stable storage, then for all clients and/or locks that may be affected, the server MUST return NFS4ERR_NO_GRACE 💔.

How the client handles NFS4ERR_NO_GRACE is outside the scope of this specification 📋. The right strategy really depends on the client's operating environment. Still, here's one potential approach.

When the client gets NFS4ERR_NO_GRACE, it could look at the change attribute of the objects it's trying to reclaim state for, then use that to decide whether to re-establish the state with normal OPEN or LOCK operations. This works as long as the client's operating environment allows it. In other words, the client implementor is advised to document this behavior for their users 📝. The client could also tell the application that its byte-range locks or share reservations (whether or not they were delegated) have been lost, using something like a UNIX signal or a Graphical User Interface (GUI) pop-up window 🪟. For a discussion of what the client should do with unreclaimed delegations on client state, see Section 10.5 🧐.

For further discussion of revocation of locks, see Section 8.5 ✨.

8.5. Server Revocation of Locks 🔒

okay so at any point the server can revoke locks a client is holding 😭 and the client has to be prepared for that. when the client detects that its locks have been, or may have been, revoked, it's responsible for validating the state info between itself and the server. validating locking state means the client must verify or reclaim the state for each lock it currently holds. bestie, that's the whole job 💅

the first time this happens is a server restart. this also covers the case where sessions are persistent but locking state gets lost anyway. in that situation the client gets an error (NFS4ERR_STALE_CLIENTID) on an operation that takes a client ID, usually during recovery from some problem with the current session. then the client just goes through normal crash recovery, as described in Section 8.4.2.1. 🧚

the second time is when the client can't renew the lease before it expires, which is discussed in Section 8.4.3. ngl this is considered rare or unusual, but the client still has to be ready to recover 🫶 the server decides exactly what the lease expiration means and tells the client how far the lock revocation reaches. the client then reads the status info the server sends back in the SEQUENCE results (field sr_status_flags, see Section 18.46.3) and syncs its locking state with the server so it can recover 🌸

the third time is revocation within the lease period. this can happen because of administrative intervention, or because a recallable lock (a delegation or layout) wasn't returned within the lease period after being recalled. again, rare, but very possible, so the client has to be ready for these too 👀 when either one happens, the client finds out through the status returned by the SEQUENCE operation. any use of stateids tied to locks revoked during the lease period gets the error NFS4ERR_ADMIN_REVOKED or NFS4ERR_DELEG_REVOKED, whichever one applies.

in every situation where a subset of locking state might have been revoked (that includes every case where locking state is revoked within the lease period), the client has to figure out which locks were revoked and which weren't. it does that with the TEST_STATEID operation on the right set of stateids 🔍 once the revoked set is known, the applications can be notified, and the invalidated stateids can be freed and the lock revocation acknowledged with FREE_STATEID. iconic cleanup arc ✨

8.6. Short and Long Leases ⏳

when picking the time period for the server lease, the usual lease tradeoffs apply 💖 a short lease means fast server recovery, but it costs more operations to renew the lease (that's only when there are no other operations during the period that would renew it as a side effect). a long lease is way kinder to servers handling a huge number of clients 🫶 the number of extra requests to renew the lock drops in inverse proportion to the lease time. the downside of a long lease is that recovery after certain failures can be slower. after a server failure, a longer grace period may be needed when some clients don't reclaim their locks quickly and send a global RECLAIM_COMPLETE. and when a client fails, the longer wait for the lease to expire means conflicting requests have to wait longer 😩

a long lease is practical if the server can store lease state in stable storage. then, on recovery, the server can rebuild the lease state from that storage and keep going with its clients 🌈

8.7. Clocks, Propagation Delay, and Calculating Lease Expiration 🕰️

so to avoid needing synchronized clocks, the server grants lease times as a time delta 🧠 but the client and server clocks still can't drift too much over the length of the lease. then there's propagation delay across the network, which could easily be several hundred milliseconds, plus the chance that requests get lost and need to be retransmitted. it's a lot, bestie 😭

to account for propagation delay, the client should subtract it from lease times. for example, if the client estimates the one-way propagation delay as 200 milliseconds, it can assume the lease is already 200 milliseconds old when it gets it. on top of that, it'll take another 200 milliseconds to get a response back to the server. so the client must send a lease renewal or write data back to the server at least 400 milliseconds before the lease would expire 🏃‍♀️ if the propagation delay changes over the life of the lease (like when the client is on a mobile host), the client needs to keep subtracting the increase in propagation delay from the lease times.

the server's lease period config should take into account the network distance of the clients that will be hitting its resources 🗺️ the lease period is expected to account for network propagation delays and other network delay factors for the client population. the protocol has no automatic way to figure out a good lease period, so the server's administrator may have to tune it by hand 🎀

8.8. Obsolete Locking Infrastructure from NFSv4.0 🗑️

there are a bunch of operations and fields in existing operations that don't do anything in NFSv4.1 anymore 💀 one way or another, all of these changes come from sessions. sessions give the protocol client context and exactly-once semantics as a base feature, separate from locking itself. it's giving clean architecture 👑

the following NFSv4.0 operations MUST NOT be implemented in NFSv4.1. if any of them show up in an NFSv4.1 COMPOUND, the server MUST return NFS4ERR_NOTSUPP.

• SETCLIENTID, since its function has been replaced by EXCHANGE_ID.
• SETCLIENTID_CONFIRM, since client ID confirmation now happens through CREATE_SESSION.
• OPEN_CONFIRM, because state-owner-based seqids have been replaced by the sequence ID in the SEQUENCE operation.
• RELEASE_LOCKOWNER, because lock-owners with no associated locks don't have any sequence-related state, so the server can delete them whenever it wants.
• RENEW, because every SEQUENCE operation for a session causes lease renewal, which makes a separate operation superfluous.

there are also a few fields in existing operations, related to locking, that have no use in minor version 1 🥀 they were used in minor version 0 to do jobs that are now handled differently.

• Sequence IDs used to sequence requests for a given state-owner and to provide retry protection. Those are now handled via sessions.
• Client IDs used to identify the client for a given request. Client identification now uses the client ID tied to the current session, so there's no explicit client ID field needed.

these vestigial fields have no function in NFSv4.1 and the server ignores them 🙈 just note that client IDs in operations new to NFSv4.1 (like CREATE_SESSION and DESTROY_CLIENTID) are not ignored! main character energy is knowing which ones still count 💅

9. File Locking and Share Reservations ✨

to support Win32 share reservations, you need operations that atomically open or create files 🎀 a separate share/unshare operation would not allow correct implementation of the Win32 OpenFile API. to get share semantics right, the old NFS mechanisms used when a file is opened or created (LOOKUP, CREATE, ACCESS) have to be replaced. so NFSv4.1 defines an OPEN operation that can atomically look up, create, and lock a file on the server. iykyk, it's the whole point 🔥

9.1. Opens and Byte-Range Locks 🦋

manipulating a byte-range lock is assumed to be rare compared to READ and WRITE operations 📊 server restarts and network partitions are also assumed to be relatively rare. so it matters that READ and WRITE have a lightweight way to indicate whether they hold a lock. a LOCK operation carries the heavyweight info needed to set up a byte-range lock and uniquely define its owner 🧚

9.1.1. State-Owner Definition 🧾

when opening a file or requesting a byte-range lock, the client must specify an identifier that represents the owner of the requested lock. this identifier is a state-owner, represented in the protocol by a state_owner4, which is a variable-length opaque array. when it's concatenated with the current client ID, it uniquely defines the owner of a lock the client manages. it can be a thread ID, a process ID, or some other unique value 💜

owners of opens and owners of byte-range locks are separate entities, and they stay separate even if the same opaque arrays are used to designate owners of each 🌸 the protocol tells them apart with open-owners (represented by open_owner4 structures) and lock-owners (represented by lock_owner4 structures).

each open is tied to a specific open-owner. each byte-range lock is tied to a lock-owner and an open-owner, and the open-owner is the one associated with the open file under which the LOCK operation was done 🫶 delegations and layouts, on the other hand, aren't tied to a specific owner. they're associated with the client as a whole (identified by a client ID) 👑

9.1.2. Use of the Stateid and Locking ✨

Okay bestie, so every READ, WRITE, and SETATTR op carries a stateid 🦋. For this section, a SETATTR that changes the size attribute gets treated like it's writing the area between the old and new sizes 🎀 (the byte range that got chopped off or tacked on), even if the text never explicitly mentions SETATTR. The stateid you pass to one of these ops must be one that represents an open, a set of byte-range locks, or a delegation, or it can be a special stateid for anonymous access or the special bypass stateid 💅.

If the state-owner does a READ or WRITE while it has a byte-range lock or share reservation on the server (any OPEN counts as a share reservation 👑), the stateid the server handed back earlier must be used to say which locks, both byte-range locks and share reservations, the state-owner is holding 🔐. If the client hasn't set up any state, meaning no byte-range lock and no share reservation, then the special anonymous stateid gets used. That one is zero for "other" and "seqid" 🧚. (Check out Section 8.2.3 for the full scoop on "special" stateids 📖.) Whether it's the anonymous one or a server-returned one, if there's a conflicting share reservation or mandatory byte-range lock on the file, the server MUST refuse to service the READ or WRITE operation 🙅.

Share reservations come from OPEN operations, and they're mandatory by nature. When an OPEN denies READ or WRITE, those ops get rejected with the error NFS4ERR_LOCKED 🚫. Byte-range locks can be mandatory or advisory, or the server can decide based on the file being accessed. For example, some UNIX-based servers have a "mandatory lock bit" on the mode attribute, and if it's set, byte-range locks are required before I/O is possible 🗝️. With advisory locks, they only stop conflicting lock requests from being granted and have zero effect on READs or WRITEs. Mandatory byte-range locks, though, block conflicting I/O, and those attempts get rejected with NFS4ERR_LOCKED. When the client gets NFS4ERR_LOCKED on a file where it knows it has the proper share reservation, it needs to send a LOCK op covering the byte range the I/O was aimed at, with an appropriate locktype (READ*_LT for a READ, WRITE*_LT for a WRITE) 🔁.

Heads up: for UNIX environments that support mandatory byte-range locking, advisory vs. mandatory is kinda subtle 🫣. In fact, advisory and mandatory byte-range locks are exactly the same as far as the APIs and implementation requirements go. If the mandatory lock attribute is set on the file, the server checks whether the lock-owner has an appropriate shared (READ_LT) or exclusive (WRITE_LT) byte-range lock covering the byte range it wants to READ from or WRITE to. If there's no appropriate lock, the server checks for a conflicting lock. It can do that by trying to acquire the conflicting lock on behalf of the lock-owner and, if that works, releasing it once the READ or WRITE is done. If there is a conflict, the server returns NFS4ERR_LOCKED.

For Windows environments, byte-range locks are always mandatory, so the server always checks for byte-range locks during I/O requests 🪟.

So the LOCK operation doesn't need to distinguish between advisory and mandatory byte-range locks. The server's processing of READ and WRITE operations is what creates the difference 💖.

Every stateid that's validly passed to READ, WRITE, or SETATTR, except for special stateid values, defines an access mode for the file. That's OPEN4_SHARE_ACCESS_READ, OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH 🎯.

• For stateids associated with opens, the access mode is the mode set by the original OPEN that allocated the OPEN stateid, as modified by any later OPENs and OPEN_DOWNGRADEs for the same open-owner/file pair.
• For stateids returned by byte-range LOCK operations, the appropriate mode is the access mode of the OPEN stateid associated with the lock set that the stateid represents.
• For delegation stateids, the access mode depends on the type of delegation.

When a READ, WRITE, or SETATTR (one that specifies the size attribute) gets done, the op gets checked against the access mode, to make sure it's okay given the stateid it's tied to 🔍.

For WRITE-type operations (that's WRITEs and SETATTRs that set size), the server MUST verify that the access mode allows writing, and it MUST return an NFS4ERR_OPENMODE error if it doesn't 🙅‍♀️. For READ, the server may do the matching check on the access mode. Or it may choose to allow READ on OPENs for OPEN4_SHARE_ACCESS_WRITE, to accommodate clients whose WRITE implementation might unavoidably do reads (for example, because of buffer cache constraints) 🧠. Even if READs are allowed in those circumstances, though, the server MUST still check for locks that conflict with the READ. For example, another OPEN might have specified OPEN4_SHARE_DENY_READ or OPEN4_SHARE_DENY_BOTH 🚧. Also, a server that enforces the access mode check on READs doesn't need to explicitly check for conflicting share reservations, because the existence of an OPEN for OPEN4_SHARE_ACCESS_READ already guarantees that no conflicting share reservation can exist 🌸.

The READ bypass special stateid (all bits of "other" and "seqid" set to one) says "I want to skip the locking checks" 🙈. The server MAY allow READ operations to bypass locking checks when this special stateid is used. WRITE operations with this special stateid value, though, MUST NOT bypass locking checks. They get treated exactly the same as if the special stateid for anonymous state had been used 🚫.

A lock may not be granted while a READ or WRITE using one of the special stateids is in progress, if the scope of the lock to be granted would conflict with that READ or WRITE. This can happen when:

• A mandatory byte-range lock is requested with a byte range that conflicts with the byte range of the READ or WRITE. For this purpose, there's a conflict when a shared lock is requested while a WRITE is in progress, or when an exclusive lock is requested while either a READ or a WRITE is in progress.
• A share reservation is requested that denies reading and/or writing, and the corresponding operation is in progress.
• A delegation is about to be granted, and the delegation type would stop the I/O operation. READ and WRITE conflict with an OPEN_DELEGATE_WRITE delegation, and WRITE conflicts with an OPEN_DELEGATE_READ delegation.

When a client holds a delegation, it needs to make sure the stateid it sends shows the operation is tied to the delegation, so the delegation doesn't get recalled when it doesn't have to be 🦋. When the delegation stateid is used (or a stateid open associated with that delegation, or a stateid for byte-range locks derived from such an open), the server knows the READ, WRITE, or SETATTR doesn't conflict with the delegation but is sent under its aegis ✨. Even though the server could figure out from the client ID (via the session ID) that the client really does have a delegation, the server isn't obliged to check that. So using a special stateid can lead to an avoidable recall of the delegation 😭.

---

9.2. Lock Ranges

The protocol lets a lock-owner request a lock with a byte range and then upgrade, downgrade, or unlock a sub-range of that initial lock. It can also do the same to a byte range that overlaps, fully or partially, with that initial lock or with a combination of existing locks for the same lock-owner 🧩. This is expected to be an uncommon type of request. Either way, servers or server file systems might not be able to support sub-range lock semantics 🤷‍♀️. If a server gets a locking request that represents a sub-range of the current locking state for the lock-owner, the server is allowed to return the error NFS4ERR_LOCK_RANGE to signal that it doesn't support sub-range lock operations. So the client should be prepared to receive this error and, if appropriate, report it to the requesting application 📣.

The client is discouraged from combining multiple independent locking ranges that happen to be adjacent into a single request. The reason is that the server may not support sub-range requests for reasons related to recovering byte-range locking state after a server failure 🛠️. As discussed in Section 8.4.2, the server may use certain optimizations during recovery that only work well when the client's behavior during lock recovery looks like its locking behavior before the server failure 💅.

---

9.3. Upgrading and Downgrading Locks

If a client has a WRITE_LT lock on a byte range, it can ask for an atomic downgrade of that lock to a READ_LT lock via the LOCK operation, by setting the type to READ_LT 🔽. If the server supports atomic downgrade, the request will succeed. If not, it'll return NFS4ERR_LOCK_NOTSUPP. The client should be prepared to receive this error and, if appropriate, report it to the requesting application 🫶.

If a client has a READ_LT lock on a byte range, it can ask for an atomic upgrade of that lock to a WRITE_LT lock via the LOCK operation, by setting the type to WRITE_LT or WRITEW_LT 🔼. If the server doesn't support atomic upgrade, it'll return NFS4ERR_LOCK_NOTSUPP. If the upgrade can happen without an existing conflict, the request will succeed. Otherwise, the server returns either NFS4ERR_DENIED or NFS4ERR_DEADLOCK. The error NFS4ERR_DEADLOCK comes back if the client sent the LOCK operation with the type set to WRITEW_LT and the server has detected a deadlock 💀. The client should be prepared to receive these errors and, if appropriate, report them to the requesting application 🌈.

---

9.4. Stateid Seqid Values and Byte-Range Locks

When a LOCK or LOCKU operation is performed, the stateid returned has the same "other" value as the argument's stateid. Its "seqid" value is incremented (relative to the argument's stateid) to reflect that the LOCK or LOCKU happened 🔢. The server MUST increment the value of the "seqid" field whenever there's any change to the locking status of any byte offset covered by any of the locks in the stateid. A change in locking status includes going from locked to unlocked (or the reverse), or going from being locked for READ_LT to being locked for WRITE_LT (or the reverse) ⭐.

When there's no such change, for example when a range already locked for WRITE_LT gets locked again for WRITE_LT, the server MAY increment the "seqid" value 🍓.

9.5. Issues with Multiple Open-Owners

Okay so picture this 💅 the same file gets opened by multiple open-owners, and now the client is holding a bunch of OPEN stateids for that one file, each tied to a different open-owner ✨. That means you can get multiple LOCK and LOCKU requests for the same lock-owner going out through different OPEN stateids. So you can end up with several stateids all representing byte-range locks on the same file, all held by the same lock-owner, just each linked to a different open-owner 🧚 chaotic, but it makes sense once you see the rules.

In that situation, the locking status of each byte (whether it's locked, whether it's a READ_LT or WRITE_LT type of lock, and which lock-owner is holding it) MUST reflect the last LOCK or LOCKU operation done for the lock-owner in question, no matter which stateid that request was sent through 🔥

When a byte is locked by that lock-owner, the open-owner that byte-range lock is assigned to SHOULD be the open-owner tied to the stateid used for the last LOCK on that byte 🎀 And here's the fun part: if the open-owner associated with the locks for a stateid changes because of a LOCK or LOCKU, the seqid field of that stateid MUST be incremented, even if the lock-owner stuff hasn't changed at all 💜 But if the set of locked bytes moves over to a different stateid for the same lock-owner (so, a different open-owner), the seqid value for that stateid MUST NOT be incremented 🙈

---

9.6. Blocking Locks

Some clients really need blocking locks 🫶 NFSv4.1 does have a callback for when a lock that wasn't available becomes available, but that's an OPTIONAL feature, so clients can't count on it being there ngl. Clients need to be ready to keep polling for the lock, which creates a fairness problem 😭 Two of the lock types, READW_LT and WRITEW_LT, tell the server that the client wants a blocking lock. When the callback isn't being used, the server should keep an ordered list of pending blocking locks. When the conflicting lock gets released, the server may wait for a period equal to lease_time for the first waiting client to re-request the lock. Once that lease period runs out, the next waiting client's request is allowed to get the lock. Clients are required to poll at an interval small enough that they're likely to get the lock in a timely way ⏰ The server isn't required to keep a list of pending blocked locks, because that list is there to improve fairness, not correct operation. Since crash recovery doesn't happen in any particular order, you'd need to store lock state in stable storage to guarantee that blocking locks get granted in order 🧚

Servers can also note the lock types and hold off on sending back a denial, so there's extra time for a conflicting lock to get released and the request can succeed instead 🌸 That way, clients avoid the hassle of constantly polling for blocking locks. The server should be careful about how long that delay is in case the client retransmits the request 👑

If a server gets a blocking LOCK operation, denies it, and then later gets a nonblocking request for the same lock that also gets denied, the server should remove that lock from its list of pending blocking locks 💀 Clients should send that nonblocking request to say "this is the last time I'm polling for this lock," which can happen when the process asking for the lock gets interrupted. It's a courtesy to the server so it doesn't wait a whole lease period for nothing before granting other LOCK operations. But clients aren't required to do this courtesy, and servers must not depend on them doing it. Also, clients have to be ready for the possibility that this final locking request gets accepted anyway ✨

When a server indicates, using the flag OPEN4_RESULT_MAY_NOTIFY_LOCK, that CB_NOTIFY_LOCK callbacks might happen for the current open file, the client should take note 🔥 But this is just a hint, so the client can't rely on a CB_NOTIFY_LOCK always showing up. A client can reasonably poll less often for a denied lock, since the extra latency is likely to be cancelled out by a prompt callback, but it still needs to poll 🦋 When it does get a CB_NOTIFY_LOCK, it should quickly try to get the lock. It should keep in mind that other clients may be polling too, and the server has no obligation to hold the lock for that particular client 💅

---

9.7. Share Reservations

A share reservation is a way to control access to a file 🎀 It's a separate and independent mechanism from byte-range locking, so don't mix them up bestie. When a client opens a file, it sends an OPEN operation to the server saying what kind of access it needs (READ, WRITE, or BOTH) and what access it wants to deny to everyone else (OPEN4_SHARE_DENY_NONE, OPEN4_SHARE_DENY_READ, OPEN4_SHARE_DENY_WRITE, or OPEN4_SHARE_DENY_BOTH). If the OPEN fails, the client fails the application's open request 😭

Here's the pseudo-code for the semantics:

        if (request.access == 0) {
          return (NFS4ERR_INVAL)
        } else {
          if ((request.access & file_state.deny)) ||
             (request.deny & file_state.access)) {
            return (NFS4ERR_SHARE_DENIED)
        }
        return (NFS4ERR_OK);

When the server checks share reservations on OPEN, the file_state used in the algorithm includes bits for every current open, including the ones belonging to the open-owner making the new OPEN request 🧚

The constants used for the access and deny fields in the OPEN and OPEN_DOWNGRADE operations are these:

const OPEN4_SHARE_ACCESS_READ   = 0x00000001;
const OPEN4_SHARE_ACCESS_WRITE  = 0x00000002;
const OPEN4_SHARE_ACCESS_BOTH   = 0x00000003;

const OPEN4_SHARE_DENY_NONE     = 0x00000000;
const OPEN4_SHARE_DENY_READ     = 0x00000001;
const OPEN4_SHARE_DENY_WRITE    = 0x00000002;
const OPEN4_SHARE_DENY_BOTH     = 0x00000003;

---

9.8. OPEN/CLOSE Operations

To get share semantics right, a client MUST use the OPEN operation to get the initial filehandle and say what access it wants and what access, if any, to deny 🫶 Even if the client plans to use a special stateid for anonymous state or READ bypass, it still has to get the filehandle for the regular file with OPEN so the right share semantics can be applied. Clients that don't have a deny mode built into their programming interfaces for opening a file should request a deny mode of OPEN4_SHARE_DENY_NONE 🌈

The OPEN operation with the CREATE flag also takes over the job of the CREATE operation for regular files from earlier versions of the NFS protocol 🍓 That means a create with a share can be done atomically, which is so clean ✨

The CLOSE operation removes all share reservations that the open-owner holds on that file. If byte-range locks are held, the client SHOULD release all of them before sending a CLOSE operation 🙈 The server MAY free all outstanding locks on CLOSE, but some servers might not support closing a file that still has byte-range locks held. The server MUST return failure, NFS4ERR_LOCKS_HELD, if any locks would exist after the CLOSE 💀

The LOOKUP operation hands back a filehandle without setting up any lock state on the server 🦋 Without a valid stateid, the server assumes the client has the least access. For example, say one client opened a file with OPEN4_SHARE_DENY_BOTH, and another client reaches that file through a filehandle it got from LOOKUP. That second client could only read the file, and only using the special read bypass stateid 👀 It couldn't WRITE the file at all, because it wouldn't have a valid stateid from OPEN, and the special anonymous stateid wouldn't be allowed access. Totally iconic restriction 💅

---

9.9. Open Upgrade and Downgrade

When an OPEN is done for a file and the open-owner it's being done for already has that file open, the result is an upgrade 🌸 The open file status kept on the server gets expanded to include the access and deny bits from the new OPEN as well as the ones from the existing OPEN. So as far as the protocol is concerned, there's one open file, and it has the union of the access and deny bits from all the OPEN requests that completed ✨ That one open is represented by a single stateid whose other value matches the original open's, and whose seqid value gets incremented to show the upgrade happened. The increment is required even when the upgrade doesn't change the open mode at all (for example, an OPEN for read when the existing open file is already opened for OPEN4_SHARE_ACCESS_BOTH) 🧚 Only one CLOSE will be done to undo the effects of both OPENs. The client can use the stateid returned by the OPEN that did the upgrade, or a stateid with the same other field and a seqid of zero, but it needs to be careful with upgrades that happen while a CLOSE is pending 🫶 Also keep in mind that when the client sends the OPEN, it may not know the same file is actually being opened. This all only applies if both OPENs end up with the opened object designated by the same filehandle.

When the server exports multiple filehandles for the same file object and hands back different filehandles on two different OPENs of that file object, the server MUST NOT "OR" the access and deny bits together and merge the two open files 🙅‍♀️ Instead, the server has to keep separate OPENs with separate stateids, and it'll need separate CLOSEs to free them.

When multiple open files on the client get merged into a single OPEN file object on the server, closing one of those open files on the client might mean the server's access and deny status for that open file has to change 🔥 That's because the union of the access and deny bits for the remaining opens might be smaller (a proper subset) than before. The OPEN_DOWNGRADE operation is how to make that change, and the client should use it to update the server so other clients' share reservation requests get handled properly 🦋 The returned stateid has the same other field as the one passed to the server. The seqid value in the returned stateid MUST be incremented, even when the access and deny bits for the file haven't changed at all ✨

9.10. Parallel OPENs ✨

Okay so this is the big difference from NFSv4.0 🫶. In 4.0, OPENs for the same open-owner were lowkey stuck in a line, because the owner-based seqid forced them to go one at a time. In 4.1, you can send multiple OPENs for the same open-owner in parallel 🔥. Bestie, the catch is that because of hard links, two OPENs might actually open the same file, and a later OPEN can end up being an upgrade of the first one. The client only finds out this happened once the operations complete 💀.

So how do you figure out what order things happened in? Look at the stateids that came back 🧚. If they share the same other value, they're opening the same file. Then you can read the order from the seqid fields, keeping in mind that the 32-bit field can wrap around. Main character detective energy ✨.

Now, if there's a chance you'll send parallel OPENs for one open-owner, an open upgrade could sneak in without you ever knowing it was coming 👀. That's why CLOSEs and OPEN_DOWNGRADEs should generally go out with a non-zero seqid in the stateid. That way the status change from the upgrade doesn't get accidentally lost 🎀. Protect your upgrades, periodt 💅.

9.11. Reclaim of Open and Byte-Range Locks 🔄

So sometimes the server crashes and you need to put your stuff back together 😭. That's when the special forms of LOCK and OPEN come in clutch. They exist so you can re-establish byte-range locks or opens after a server failure 🌸.

• To reclaim existing opens, you do an OPEN using CLAIM_PREVIOUS. You already opened the file in this situation and you already have its filehandle, so the current filehandle has to be the target file itself, not a directory, and you don't specify a file name 🍓.
• To reclaim byte-range locks, you use a LOCK operation with the reclaim parameter set to true 🔐.

Reclaims of opens tied to delegations get their own explanation in Section 10.2.1 🦋.

10. Client-Side Caching 💜

Caching is basically essential for good performance with NFS ✨. Client-side caching of data, file attributes, and file names is the move. The problem is that distributed cache coherence is genuinely hard, and earlier NFS versions just didn't try it 😩. Instead, clients used a bunch of implementation tricks to soften the blow of not having coherence. Those tricks were never clearly defined in earlier specs, so it was often unclear what counted as valid or invalid client behavior 🙈.

NFSv4.1 uses a lot of the same techniques as before. It still doesn't provide distributed cache coherence, but it does promise a more limited set of caching guarantees 💖. Those guarantees let locks and share reservations work without destructive interference from client-side caching. Protection energy 🛡️.

Then 4.1 brings in delegations 🎀. A delegation lets clients make a bunch of decisions locally that the server would normally make. It's super efficient for the common cases where sharing is rare or read-only 🌈. Iconic.

10.1. Performance Challenges for Client-Side Caching 🐢

Old-school caching has been really successful for performance, ngl 🌸. But when you have a huge number of clients, scalability challenges start popping up 😬. It gets even messier when clients are spread out geographically, since that usually makes revalidation requests take longer ✈️.

Earlier NFS versions revalidate their file data cache every time a file gets opened. That can cause real performance problems 💀. The most common case is a file that only one client ever touches, which means sharing is basically nonexistent 🙃.

In that situation, asking the server over and over just to hear "no conflicts" is a waste 🙄. The better move is letting a client that keeps opening the same file do it without checking with the server at all. That continues until another client actually does something that could conflict 🦋.

Byte-range locking has the same problem 😤. Sending LOCK and LOCKU operations, plus the READ and WRITE operations needed to keep data caching consistent with the locking semantics (see Section 10.3.2), can seriously tank performance 📉. If you're only using locks to guard against infrequent conflicts, the penalty is huge, and that can scare apps away from byte-range locking entirely 😭.

NFSv4.1 goes harder on caching with these design goals:

• Compatibility with a large range of server semantics 💅.
• Giving you the same caching benefits as previous NFS versions when the server can't support the more aggressive model 🫶.
• Organizing the requirements for aggressive caching so you still get a big chunk of the benefit even when not every requirement can be met ✨.

The server requirements for each specific form of caching show up in later sections, see Section 10.4 🍓.

10.2. Delegation and Callbacks 🤝

Okay, delegations are giving performance 🚀. A server can hand a file's responsibilities over to a client, which avoids a ton of repeated requests as long as nobody else is poking at the file. If a second client starts sharing the delegated file, the server uses a "callback" RPC to recall the delegation 📞.

When a delegation is handed over, it names the object and the type of delegation. Every type has a stateid for the delegation, which you use for any operation that depends on it 🔑. It's kind of like the stateids for locks and share reservations, except a delegation stateid is tied to a client ID and can be used on behalf of every open-owner for that client 🧚. The delegation belongs to the client as a whole, not to any particular process or thread inside it 💜.

The backchannel gets set up by CREATE_SESSION and BIND_CONN_TO_SESSION, and the client is required to keep it running 📡. The backchannel can be down, even briefly, so correct protocol operation doesn't depend on it 🙈. You can do a preliminary check on the backchannel with a CB_COMPOUND that has just one operation, CB_SEQUENCE, to see whether it's still connected 🔍. The server won't hand out delegations until it has confirmed the backchannel exists 🎀.

Here's the important bit 👑. Granting a delegation always depends on there being no conflicting access, so clients MUST NOT assume a delegation will be granted. They MUST always be prepared for OPENs, WANT_DELEGATIONs, and GET_DIR_DELEGATIONs to get processed without any delegation being granted 😭. Expect the unexpected, no cap.

Locks and delegations are different here. A second client touching a delegated file makes the server recall the delegation through a callback 📞. The IMPLEMENTATION section lays out, for each individual operation, when that kind of operation needs to trigger a recall. A few points are worth noting first 💅.

• The server is free to recall a delegation whenever it feels like it 🌈. It can do that even when no operations requiring a recall are happening.
• Operations that happen outside NFSv4.1, like access through other protocols or local access, also need to trigger a delegation recall when they make analogous changes to file system data 👀. What really matters is whether the change would invalidate the guarantees the delegation gives. If that's possible, the delegation needs to be recalled and MUST be returned or revoked before the operation can go ahead 🛑.
• File system semantics decide when a recall is required 🧠. If a change inside a specific implementation changes a file attribute, a recall is required even when that operation wasn't specifically listed as needing one. Again, the question is whether the delegation's guarantees are being invalidated 🫶.

Even with all those caveats, the implementation sections for a bunch of operations spell out situations where a recall would be needed in common circumstances 📋:

• For GETATTR, see Section 18.7.4.
• For OPEN, see Section 18.16.4.
• For READ, see Section 18.22.4.
• For REMOVE, see Section 18.25.4.
• For RENAME, see Section 18.26.4.
• For SETATTR, see Section 18.30.4.
• For WRITE, see Section 18.32.4.

When a recall happens, the client holding the delegation has to flush any modified state (like modified data) back to the server and return the delegation 📦. The conflicting request doesn't get acted on until the recall is done ⏳. The recall counts as complete once the client returns the delegation, or once the server runs out of time waiting and revokes the delegation because of the timeout ⏰. In the meantime, the server either delays responding to the conflicting requests or responds to them with NFS4ERR_DELAY 🙈. Once the recall resolves, the server has what it needs to grant or deny the second client's request ✨.

When a client gets a recall, it might have a ton of state to flush to the server 😵. So the server should give enough time for the delegation to come back, since that might take a lot of RPCs 🐌. If the server can tell the client is diligently flushing state because of the recall, it may stretch the usual recall time 💖. Still, the time allowed for recall completion should not be unbounded ⏱️. There's a limit, bestie.

One example is when a client is responsible for mediating opens on a given file (see Section 10.4) 🎀. The server won't know which opens are active on that client. Without that info, it can't tell whether the access and deny states for the file allow any particular open until the delegation is returned 🤷‍♀️.

A client failure or a network partition can mean a recall callback never gets a response 📡💀. In that case the server revokes the delegation, which makes any modified state still sitting on the client totally useless 😭. Rip to that data.

10.2.1. Delegation Recovery ✨

okay bestie, delegation recovery has to handle three situations 💅:

• client restart
• server restart
• network partition (full or backchannel-only)

If the client restarts and fails to renew its lease, byte-range locks and share reservations get revoked 💀. Delegations are treated a bit differently though 🦋.

Sometimes delegations need to be re-established after a client restarts. The reason is that the client may have file data stored locally, and that data is tied to the delegations it held before. So the client has to re-establish the right file state on the server 🫶.

To make this kind of recovery possible, the server MAY extend the delegation recovery period beyond the usual lease expiration period. That means requests from other clients that conflict with these delegations have to wait ⏳. Recalling a delegation can take a while, since the client has to flush its changed state to the server, so other clients need to be ready for delays caused by a conflicting delegation. The longer interval gives clients more time to restart and check stable storage so their delegations can be reclaimed 🌸. For OPEN delegations, reclaiming happens with OPEN using the claim type CLAIM_DELEGATE_PREV or CLAIM_DELEG_PREV_FH (see Sections 10.5 and 18.16 for the OPEN delegation discussion and the details of OPEN, respectively) 💖.

A server MAY support the claim types CLAIM_DELEGATE_PREV and CLAIM_DELEG_PREV_FH. If it does, it MUST NOT remove delegations upon a CREATE_SESSION that confirms a client ID created by EXCHANGE_ID. Instead, the server MUST keep the client's delegations for a period of time no less than the value of the lease_time attribute, so the client has time to send CLAIM_DELEGATE_PREV and/or CLAIM_DELEG_PREV_FH requests 🧚. A server that supports CLAIM_DELEGATE_PREV and/or CLAIM_DELEG_PREV_FH MUST support the DELEGPURGE operation too 👑.

When the server restarts, delegations get reclaimed (using the OPEN operation with CLAIM_PREVIOUS) in a similar way to byte-range locks and share reservations. But there's a slight semantic difference 👀. Normally, if the server decides a delegation shouldn't be granted, it just performs the requested action (e.g., OPEN) and grants no delegation. For reclaim, the server grants the delegation but applies a special designation, so the client treats it as granted but recalled by the server. That means the client has the duty to write all modified state to the server and then return the delegation 🎀. This whole process reconciles three principles of the NFSv4.1 protocol:

• Upon reclaim, a client reporting resources assigned to it by an earlier server instance must be granted those resources.
• The server has unquestionable authority to determine whether delegations are to be granted and, once granted, whether they are to be continued.
• The use of callbacks should not be depended upon until the client has proven its ability to receive them.

When a client needs to reclaim a delegation and there's no associated open, it may use the CLAIM_PREVIOUS variant of the WANT_DELEGATION operation. But the server isn't required to support that operation, so an alternative is to reclaim through a dummy OPEN together with the delegation, using an OPEN of type CLAIM_PREVIOUS. Then the dummy open file can be released with CLOSE, which re-establishes the original state being reclaimed: a delegation without an associated open 🍓.

When a client has more than one open associated with a delegation, the state for those additional opens can be established using OPEN operations of type CLAIM_DELEGATE_CUR. When these are used to establish opens associated with reclaimed delegations, the server MUST allow them when they're made within the grace period 🌈.

When a network partition occurs, delegations are subject to being freed by the server once the lease renewal period expires. This is similar to how locks and share reservations behave. For delegations, though, the server may extend the period in which conflicting requests are held off ⭐. Eventually, a conflicting request from another client will cause the delegation to be revoked. A loss of the backchannel (e.g., from a later network configuration change) has the same effect. A recall request will fail, and revocation of the delegation results 💀.

A client normally finds out about a revoked delegation when it uses a stateid associated with that delegation and gets one of the errors NFS4ERR_EXPIRED, NFS4ERR_ADMIN_REVOKED, or NFS4ERR_DELEG_REVOKED. It may also find out after a client restart, when it tries to reclaim a delegation and gets that same error 🙈. Note that a revoked OPEN_DELEGATE_WRITE delegation comes with issues, because data may have been modified by the client whose delegation was revoked and separately by other clients. See Section 10.5.1 for a discussion of those issues 😭. Also note that when delegations are revoked, information about the revoked delegation is written by the server to stable storage (as described in Section 8.4.3). This covers the case where a server restarts after revoking a delegation but before the client holding it has been notified about the revocation 🫶.

---

10.3. Data Caching 🌸

When applications share access to a set of files, they need to be implemented so they take into account the possibility of conflicting access by another application. This is true whether the applications run on different clients or live on the same client 💜.

Share reservations and byte-range locks are the facilities the NFSv4.1 protocol gives applications to coordinate access through mutual exclusion. The NFSv4.1 protocol's data caching must be implemented so that it doesn't invalidate the assumptions that those using these facilities depend on 🔥.

10.3.1. Data Caching and OPENs 🦋

To avoid invalidating the sharing assumptions applications rely on, NFSv4.1 clients should not provide cached data to applications, or modify it on an application's behalf, when it would not be valid to obtain or modify that same data via a READ or WRITE operation 💅.

Furthermore, in the absence of an OPEN delegation (see Section 10.4), two additional rules apply. Note that many NFSv3 clients obey these rules in practice 🌈.

• First, cached data present on a client must be revalidated after doing an OPEN. Revalidating means the client fetches the change attribute from the server, compares it with the cached change attribute, and, if they differ, declares the cached data (as well as the cached attributes) invalid. This makes sure the data for the OPENed file is still correctly reflected in the client's cache ✨. This validation must be done at least when the client's OPEN operation includes a deny of OPEN4_SHARE_DENY_WRITE or OPEN4_SHARE_DENY_BOTH, which ends a period in which other clients may have had the chance to open the file with OPEN4_SHARE_ACCESS_WRITE/OPEN4_SHARE_ACCESS_BOTH access 🙈. Clients may choose to revalidate more often (i.e., at OPENs specifying a deny mode of OPEN4_SHARE_DENY_NONE) to parallel the NFSv3 protocol's practice, for the benefit of users who expect this degree of cache revalidation 🎀.

Since the change attribute is updated for both data and metadata modifications, some client implementors may be tempted to use the time_modify attribute instead of the change attribute to validate cached data, so that metadata changes don't spuriously invalidate clean data. I have to caution implementors against this approach 🚫. The change attribute is guaranteed to change for each update to the file, whereas time_modify is guaranteed to change only at the granularity of the time_delta attribute. If the client's data cache validation logic uses time_modify and not change, it runs the risk of incorrectly marking stale data as valid 💀. So any cache validation approach by the client MUST include the use of the change attribute 👑.

• Second, modified data must be flushed to the server before closing a file OPENed for OPEN4_SHARE_ACCESS_WRITE. This is complementary to the first rule. If the data isn't flushed at CLOSE, the revalidation done after the client OPENs a file can't achieve its purpose 😭. The other part of flushing before close is that the data must be committed to stable storage at the server before the client requests the CLOSE operation 🫶. If the server restarts after a file has been CLOSEd, it may not be possible to retransmit the data to be written to the file, which is why this requirement exists 🍓.

10.3.2. Data Caching and File Locking 🔒✨

Okay bestie, so some apps skip share reservations and use byte-range locks to keep file access from getting messy 🦋 There's a whole matching set of caching rules for that setup too. These rules only work if your byte-range locks line up exactly with the READ and WRITE operations you actually run 🎯 That's different from locking that's just vibes-based, where everyone agrees on a convention but nothing enforces it 💀

Here's an example. Say you have a two-megabyte file 📂 You split it into two one-megabyte ranges and guard each range with a byte-range lock on byte zero and byte one. A WRITE_LT lock on byte zero is your ticket to do READ and WRITE operations on the first range. A WRITE_LT lock on byte one is your ticket for the second range. If every app touching the file follows this convention, it'll work fine on a local file system ✨ But it may not work with the NFSv4.1 protocol unless clients refrain from data caching. Plot twist 😭

The rules for data caching in the byte-range locking environment are:

• First, when a client obtains a byte-range lock for a particular byte-range, the data cache for that byte-range (if any cached data exists) must be revalidated. If the change attribute shows that the file may have been updated since the cached data was obtained, the client must flush or invalidate the cached data for the newly locked byte-range. A client might choose to invalidate all of the non-modified cached data it has for the file, but the only requirement for correct operation is to invalidate all of the data in the newly locked byte-range. Honestly, the newly locked range is the only part that's a must 💅
• Second, before releasing a WRITE_LT lock for a byte-range, all modified data for that byte-range must be flushed to the server. The modified data must also be written to stable storage. Lock it in, then save it 🔐

Flushing data to the server and invalidating cached data must reflect the actual byte-ranges locked or unlocked. Rounding these up or down to match client cache block boundaries will cause problems if it isn't done really carefully 🙈 For example, writing a modified block when only half of that block is inside the area being unlocked may cause an invalid modification to the byte-range outside the unlocked area. That outside range could be part of a byte-range locked by another client, and that is a full-on disaster 😱 Clients can avoid this by synchronously performing the portions of WRITE operations that overlap the initial or final part that isn't a full block. Similarly, invalidating a locked area that isn't an integral number of full buffer blocks would require the client to read one or two partial blocks from the server if the revalidation procedure shows that the client's data may not be valid. So yes, it's extra work, but it's worth it 🧚

The data that gets written to the server as a prerequisite to unlocking a byte-range must be written to stable storage on the server. The client can do this either with synchronous writes or by following asynchronous writes with a COMMIT operation. This is required because retransmitting the modified data after a server restart might conflict with a lock held by another client. Nobody wants a ghost write showing up after the lock is gone 👻

A client implementation may choose to accommodate applications that use byte-range locking in non-standard ways (for example, using a byte-range lock as a global semaphore 🚦) by flushing more data to the server upon a LOCKU than is covered by the locked range. That extra data may include modified data within files other than the one whose unlocks are being done. In these cases, the client must not interfere with applications whose READs and WRITEs are happening only within the bounds of byte-range locks that the application holds. Respect the boundaries, babe 🫶 For example, an application locks a single byte of a file and then writes that single byte. A client that handles a LOCKU by flushing all modified data to the server could validly write that single byte in response to an unrelated LOCKU operation. However, it would not be valid to write the entire block containing that byte, because the block includes an area that is not locked and might be locked by another client. Clients can avoid this problem by dividing files with modified data into two groups: those where every modification is made to areas covered by an appropriate byte-range lock, and those where some modifications aren't covered by a byte-range lock. Any writes for the first group of files must not include areas that aren't locked and therefore aren't modified on the client. Keep your writes inside your lanes 🛣️💖

---

10.3.3. Data Caching and Mandatory File Locking 🛡️

Client-side data caching needs to respect mandatory byte-range locking when it's in effect, no exceptions 💅 The presence of mandatory byte-range locking for a given file is indicated when the client gets back NFS4ERR_LOCKED from a READ or WRITE operation on a file for which it has an appropriate share reservation. When mandatory locking is in effect for a file, the client must check for an appropriate byte-range lock on the data being read or written. If a byte-range lock exists for the range being read or written, the client may satisfy the request from its validated cache. If an appropriate byte-range lock is not held for the range of the read or write, the request must not be satisfied from the client's cache, and it must be sent to the server for processing. When a read or write request partially overlaps a locked byte-range, the request should be split into multiple pieces, with each byte-range (locked or not) treated appropriately. Divide and conquer, iykyk 🍓

---

10.3.4. Data Caching and File Identity 🪪

When clients cache data, the file data needs to be organized according to the file system object it belongs to 📁 For NFSv3 clients, the usual practice has been to assume, for caching purposes, that distinct filehandles represent distinct file system objects. The client can then choose to organize and maintain its data cache on that basis. Simple and clean, it's giving classic NFSv3 ✨

In the NFSv4.1 protocol, things are less straightforward. There's now a real possibility of significant deviations from the "one filehandle per object" model, because a filehandle may be constructed from the object's pathname 🌀 So clients need a reliable way to determine whether two filehandles designate the same file system object. If clients just assumed that all distinct filehandles denote distinct objects and cached data on that basis, caching inconsistencies would pop up between the distinct client-side objects that map to the same server-side object. Not ideal, not ideal at all 😩

By providing a way to tell filehandles apart, the NFSv4.1 protocol avoids a potential functional regression compared with NFSv3 💜 Without that method, caching inconsistencies could happen within the same client, and that hasn't been an issue in previous versions of the NFS protocol. It's possible to get inconsistencies with applications running on multiple clients too, but that's not what's being addressed here, so don't mix up the two 🙃

For data caching purposes, the following steps let an NFSv4.1 client determine whether two distinct filehandles denote the same server-side object:

• If GETATTR directed to two filehandles returns different values of the fsid attribute, then the filehandles represent distinct objects.
• If GETATTR for any file with an fsid that matches the fsid of the two filehandles in question returns a unique_handles attribute with a value of TRUE, then the two objects are distinct.
• If GETATTR directed to the two filehandles does not return the fileid attribute for both of the handles, then it cannot be determined whether the two objects are the same. Therefore, operations that depend on that knowledge (for example, client-side data caching) cannot be done reliably. Note that if GETATTR does not return the fileid attribute for both filehandles, it will return it for neither of them, since the fsid for both filehandles is the same. Sneaky, right? 🕵️‍♀️
• If GETATTR directed to the two filehandles returns different values for the fileid attribute, then they are distinct objects.
• Otherwise, they are the same object. Ta-da, identity crisis solved 👑

10.4. Open Delegation ✨

okay so when a file is getting OPENed, the server is allowed to hand the opening client the keys to handle further opens and closes for that file 🔑 (that's the delegation!). but here's the tea: any delegation can be recalled 🫶, because the stuff that made it possible can change. especially if the server gets a conflicting OPEN from another client, the server must recall the delegation before it decides whether that other OPEN can be granted. making a delegation is totally up to the server, so clients should not assume any particular OPEN will or won't get one 💅. here's a typical checklist servers might use when deciding whether an OPEN should be delegated 📝:

• The client must be able to respond to the server's callback requests. If a backchannel has been established, the server will send a CB_COMPOUND request, containing a single operation, CB_SEQUENCE, for a test of backchannel availability.
• The client must have responded properly to previous recalls.
• There must be no current OPEN conflicting with the requested delegation.
• There should be no current delegation that conflicts with the delegation being requested.
• The probability of future conflicting open requests should be low based on the recent history of the file.
• The existence of any server-specific semantics of OPEN/CLOSE that would make the required handling incompatible with the prescribed handling that the delegated client would apply (see below).

there are two types of OPEN delegations 👯: OPEN_DELEGATE_READ and OPEN_DELEGATE_WRITE. an OPEN_DELEGATE_READ delegation lets a client handle, all on its own, requests to open a file for reading that don't deny OPEN4_SHARE_ACCESS_READ access to others. multiple OPEN_DELEGATE_READ delegations can be outstanding at the same time and they don't conflict, so no drama 🌸. an OPEN_DELEGATE_WRITE delegation lets the client handle all opens on its own 👑. only one OPEN_DELEGATE_WRITE delegation can exist for a given file at a given time, and it's inconsistent with any OPEN_DELEGATE_READ delegations, so they can't be in the same room 🙈.

when a client has an OPEN_DELEGATE_READ delegation, it's assured that neither the contents, the attributes (with the exception of time_access), nor the names of any links to the file will change without its knowledge, as long as the delegation is held 🧚. when a client has an OPEN_DELEGATE_WRITE delegation, it may modify the file data locally, since no other client will be accessing the file's data ✍️. the client holding an OPEN_DELEGATE_WRITE delegation may only locally affect file attributes that are intimately connected with the file data: size, change, time_access, time_metadata, and time_modify. all other attributes must be reflected on the server 📡.

when a client has an OPEN delegation, it does not need to send OPENs or CLOSEs to the server. instead, it can update the appropriate status internally 🧠. for an OPEN_DELEGATE_READ delegation, opens that can't be handled locally (opens for OPEN4_SHARE_ACCESS_WRITE/OPEN4_SHARE_ACCESS_BOTH, or opens that deny OPEN4_SHARE_ACCESS_READ access) must be sent to the server 📬.

when an OPEN delegation is made, the reply to the OPEN contains an OPEN delegation structure that specifies the following 🎁:

• the type of delegation (OPEN_DELEGATE_READ or OPEN_DELEGATE_WRITE).
• space limitation information to control flushing of data on close (OPEN_DELEGATE_WRITE delegation only; see Section 10.4.1)
• an nfsace4 specifying read and write permissions
• a stateid to represent the delegation

the delegation stateid is separate and distinct from the stateid for the OPEN proper, so they are not the same thing 🙅‍♀️. the standard stateid, unlike the delegation stateid, is associated with a particular lock-owner, and it will keep being valid after the delegation is recalled and the file remains open 🔓.

when a request internal to the client is made to open a file and an OPEN delegation is in effect, it gets accepted or rejected solely on the basis of the following conditions. if something needs other checks to be made by the delegate, the OPEN delegation should be denied so the server can make those checks itself 🛑.

• The access and deny bits for the request and the file as described in Section 9.7.
• The read and write permissions as determined below.

the nfsace4 passed with the delegation can be used to avoid frequent ACCESS calls 🏃‍♀️. the permission check goes like this:

• If the nfsace4 indicates that the open may be done, then it should be granted without reference to the server.
• If the nfsace4 indicates that the open may not be done, then an ACCESS request must be sent to the server to obtain the definitive answer.

⚠️ the server may return an nfsace4 that is more restrictive than the actual ACL of the file. that includes an nfsace4 that specifies denial of all access. also, some common practices, like mapping the traditional user "root" to the user "nobody" (see Section 5.9), may make it incorrect to return the actual ACL of the file in the delegation response. yikes 😬

using a delegation together with other forms of caching creates the possibility that no server authentication and authorization will ever happen for a given user, since all of the user's requests might be satisfied locally 👀. so if the client is depending on the server for authentication and authorization, it should make sure that authentication and authorization happens for each user by using the ACCESS operation 🔐. this should be the case even if an ACCESS operation wouldn't be required otherwise. like I said before, the server may enforce frequent authentication by returning an nfsace4 denying all access with every OPEN delegation 💖.

10.4.1. Open Delegation and Data Caching

an OPEN delegation cuts out a lot of the message overhead that comes with opening and closing files 🍓. an open while an OPEN delegation is in effect does not require a validation message to be sent to the server ✨. the continued endurance of the "OPEN_DELEGATE_READ delegation" guarantees that no OPEN for OPEN4_SHARE_ACCESS_WRITE/OPEN4_SHARE_ACCESS_BOTH has happened, and so no write has happened either 📖. similarly, when closing a file that was opened for OPEN4_SHARE_ACCESS_WRITE/OPEN4_SHARE_ACCESS_BOTH, and an OPEN_DELEGATE_WRITE delegation is in effect, the data written doesn't have to go to the server until the OPEN delegation is recalled 📦. the continued endurance of the OPEN delegation guarantees that no open, and so no READ or WRITE, has been done by another client 🫧.

for the purposes of OPEN delegation, READs and WRITEs done without an OPEN are treated as the functional equivalents of a corresponding type of OPEN 🔄. even though a client SHOULD NOT use special stateids when an open exists, delegation handling on the server can use the client ID associated with the current session to figure out whether the operation was done by the holder of the delegation (in which case no recall is necessary) or by another client (in which case the delegation must be recalled, and I/O must not proceed until the delegation is recalled or revoked) 🕵️‍♀️.

with delegations, a client can avoid writing data to the server when the CLOSE of a file is serviced 🙌. the file close system call is the usual point where the client gets told about a lack of stable storage for the modified file data the application generated 😰. at the close, file data is written to the server, and through normal accounting the server can tell if the available file system space for the data has been exceeded (i.e., the server returns NFS4ERR_NOSPC or NFS4ERR_DQUOT). that accounting includes quotas 📊. the introduction of delegations means there has to be a different method in place for that same kind of communication between client and server 🛠️.

in the delegation response, the server provides either the limit of the size of the file, or the number of modified blocks and the associated block size 📏. the server must make sure the client will be able to write modified data to the server of a size equal to what was given in the original delegation 💯. the server must make this assurance for all outstanding delegations. so the server has to be careful about how it manages available space for new or modified data, taking into account available file system space and any applicable quotas 🧮. the server can recall delegations as a result of managing the available file system space 🔁. the client should abide by the server's state space limits for delegations. if the client exceeds the stated limits for the delegation, the server's behavior is undefined 💀.

based on server conditions, quotas, or available file system space, the server may grant OPEN_DELEGATE_WRITE delegations with very restrictive space limitations 🥲. those limitations may be defined in a way that will always force modified data to be flushed to the server on close 🚿.

with respect to authentication, flushing modified data to the server after a CLOSE has happened can be kind of messy 😵‍💫. for example, the user of the application may have logged off the client, and unexpired authentication credentials might not be present. in that case, the client may need to take special care to make sure local unexpired credentials will actually be available. this can be done by tracking the expiration time of credentials and flushing data well before they expire, or by making private copies of credentials so they're available when needed 🔑💜.

10.4.2. Open Delegation and File Locks

when a client holds an OPEN_DELEGATE_WRITE delegation, lock operations are performed locally 🏠. that includes the ones needed for mandatory byte-range locking. this works because the delegation implies there can be no conflicting locks 🙅‍♀️. similarly, all of the revalidations that would normally go along with obtaining locks, and the flushing of data that goes with releasing them, don't need to be done 🌈.

when a client holds an OPEN_DELEGATE_READ delegation, lock operations are not performed locally 🚫. all lock operations, including those requesting non-exclusive locks, are sent to the server for resolution 📨.

10.4.3. Handling of CB_GETATTR 🎀

Okay bestie, the server needs to do something special for a GETATTR when the target file has an OPEN_DELEGATE_WRITE delegation in effect 💅. Why? The client holding that delegation may have changed the data, and the server has to show those changes to the second client that sent the GETATTR. So that holder needs to be interrogated, and the server does it with the CB_GETATTR operation. The only attributes the server can reliably query via CB_GETATTR are size and change. 🦋

Since CB_GETATTR is only there to answer someone else's GETATTR, the server really just needs to know whether the delegation holder has a modified version of the file 🧚. If the client's copy isn't modified (data or size), the server can answer the second client from the attributes it already stores locally. If the file is modified, the server only needs to know that part. And if the server decides the file is currently modified, it answers the second client's GETATTR as if the file had been modified locally at the server 😭.

The server picks the form of the change attribute, and it's opaque to the client, so the two of them need to agree on how to say "hey, I've got modified stuff over here." For size, the client just reports its current view of the file size. For change, it gets a little more involved 🌈.

For the client, these are the steps when it receives an OPEN_DELEGATE_WRITE delegation:

• The value of the change attribute is obtained from the server and cached. Let's call this value c.
• The client creates a value greater than c that gets used for communicating that modified data is held at the client. Let's call this value d.
• When the client is queried via CB_GETATTR for the change attribute, it checks to see if it holds modified data. If the file is modified, the value d is returned for the change attribute. If the file is not currently modified, the client returns the value c for change.

For simplicity of implementation, the client MAY return the same value d for each CB_GETATTR. This stays true even if, between successive CB_GETATTR operations, the client modifies the file's data or metadata in its cache again 🫶. The client can do this because the only requirement is that it can tell the server it holds modified data. So the value of d may always be c + 1 ✨.

Here's the fun bit: change is opaque to the client in the sense that it has no idea what units of time, if any, the server is counting in. But it is not opaque in that the client has to treat it as an unsigned integer, and the server has to be able to see the results of the client's changes to that integer 🔥. So the server MUST encode the change attribute in network order when sending it to the client. The client MUST decode it from network order to its native order when receiving it, and the client MUST encode it in network order when sending it to the server. This is exactly why change is defined as an unsigned integer rather than an opaque array of bytes 🍓.

For the server, these are the steps when it provides an OPEN_DELEGATE_WRITE delegation:

• When the server provides an OPEN_DELEGATE_WRITE delegation, it caches a copy of the change attribute in the data structure it uses to record the delegation. Let's call this value sc.
• When a second client sends a GETATTR operation on the same file to the server, the server obtains the change attribute from the first client. Let's call this value cc.
• If the value cc is equal to sc, the file is not modified, and the server returns the current values for change, time_metadata, and time_modify (for example) to the second client 💖.
• If the value cc is not equal to sc, the file is currently modified at the first client and most likely will be modified at the server at a future time. The server then uses its current time to construct attribute values for time_metadata and time_modify. A new value of sc, which we'll call nsc, is computed by the server, such that nsc >= sc + 1. The server then returns the constructed time_metadata, time_modify, and nsc values to the requester. The server replaces sc in the delegation record with nsc. To stop time_modify, time_metadata, and change from appearing to go backward (which would happen if the client holding the delegation fails to write its modified data to the server before the delegation is revoked or returned), the server SHOULD update the file's metadata record with the constructed attribute values 😭. For reasons of reasonable performance, committing the constructed attribute values to stable storage is OPTIONAL. 🧚

As we talked about earlier in this section, the client MAY return the same cc value on subsequent CB_GETATTR calls, even if the file was modified in the client's cache yet again between successive CB_GETATTR calls. So the server must assume the file has been modified yet again, and it MUST take care to make sure the new nsc it constructs and returns is greater than the previous nsc it returned 👑. An example implementation's delegation record would do this by including a boolean field (let's call it "modified") that is set to FALSE when the delegation is granted, and an sc value set at grant time to the change attribute value. The modified field would be set to TRUE the first time cc != sc, and it would stay TRUE until the delegation is returned or revoked. The processing for constructing nsc, time_modify, and time_metadata would use this pseudo code:

    if (!modified) {
        do CB_GETATTR for change and size;

        if (cc != sc)
            modified = TRUE;
    } else {
        do CB_GETATTR for size;
    }

    if (modified) {
        sc = sc + 1;
        time_modify = time_metadata = current_time;
        update sc, time_modify, time_metadata into file's metadata;
    }

This would return to the client (the one that sent GETATTR) the attributes it requested, but make sure size comes from what CB_GETATTR returned. The server would not update the file's metadata with the client's modified size 🙈.

If the file attribute size is different from the server's current value, the server treats this as a modification no matter what the change attribute retrieved via CB_GETATTR says, and it responds to the second client just like in the last step above 💅.

This methodology resolves issues of clock differences between client and server, plus other scenarios where the use of CB_GETATTR would break down 🌸.

It's worth noting that the server is under no obligation to use CB_GETATTR, so the server MAY simply recall the delegation to avoid using it at all. Totally valid, no judgment 💜.

10.4.4. Recall of Open Delegation ✨

okay bestie, these are the events that call for a recall of an OPEN delegation 💅:

• potentially conflicting OPEN request (or a READ or WRITE operation done with a special stateid)
• SETATTR sent by another client
• REMOVE request for the file
• RENAME request for the file as either the source or target of the RENAME

whether a RENAME of a directory in the path leading to the file sets off a recall depends on how the server's file system works 🦋 if that file system denies such RENAMEs when a file is open, the recall must be performed to find out whether the file is, in fact, open.

outside of those situations, the server may choose to recall OPEN delegations at any time if resource constraints make it advisable 🌸 clients should always be prepared for the possibility of a recall 🫶

when a client gets a recall for an OPEN delegation, it needs to update state on the server before giving the delegation back. these same updates must be done whenever a client chooses to return a delegation voluntarily. here's the state that needs handling:

• If the file associated with the delegation is no longer open and no previous CLOSE operation has been sent to the server, a CLOSE operation must be sent to the server.
• If a file has other open references at the client, then OPEN operations must be sent to the server. The appropriate stateids will be provided by the server for subsequent use by the client, since the delegation stateid will no longer be valid. These OPEN requests are done with the claim type of CLAIM_DELEGATE_CUR. This lets the client present the delegation stateid so it can establish the right to perform the OPEN. (see Section 18.16, which describes the OPEN operation, for details.)
• If there are granted byte-range locks, the corresponding LOCK operations need to be performed. This applies to the OPEN_DELEGATE_WRITE delegation case only.
• For an OPEN_DELEGATE_WRITE delegation, if at the time of recall the file is not open for OPEN4_SHARE_ACCESS_WRITE/OPEN4_SHARE_ACCESS_BOTH, all modified data for the file must be flushed to the server. If the delegation had not existed, the client would have done this data flush before the CLOSE operation.
• For an OPEN_DELEGATE_WRITE delegation when a file is still open at the time of recall, any modified data for the file needs to be flushed to the server.
• With the OPEN_DELEGATE_WRITE delegation in place, the file may have been truncated during the life of the delegation 😭 for example, the truncation could have happened because of an OPEN UNCHECKED with a size attribute value of zero. So if a truncation has occurred and hasn't been propagated to the server, the truncation must occur before any modified data is written to the server.

byte-range locking adds some extra requirements for the OPEN_DELEGATE_WRITE case 🔐 to precisely keep the associated invariant, it is required to flush any modified data in any byte-range for which a WRITE_LT lock was released while the OPEN_DELEGATE_WRITE delegation was in effect. BUT, because that delegation implies no other locking by other clients, a simpler approach is to flush all modified data for the file (as described just above) if any WRITE_LT lock has been released while the delegation was in effect. easy mode 💖

an implementation doesn't have to wait until a recall (or the decision to return a delegation voluntarily) to do any of the above, if implementation considerations (e.g., resource availability constraints) make that a better move ✨ generally though, the actual OPEN state of the file may keep changing, so it isn't worth sending info about opens and closes to the server except as part of returning the delegation. the exception is when the client has no more internal opens of the file. then sending a CLOSE is useful because it reduces resource utilization on the client and server 🧚 regardless of when the client chooses to schedule these actions, all of them must be performed before the delegation is returned, including (when applicable) the close that corresponds to the OPEN that resulted in the delegation. these actions can be done either in previous requests or in previous operations in the same COMPOUND request.

---

10.4.5. Clients That Fail to Honor Delegation Recalls 💀

a client may fail to respond to a recall for various reasons, like a failure of the backchannel from the server to the client. the client might not even know the backchannel is broken 🙈 that lack of awareness could mean the client only finds out long after the failure that its delegation was revoked, and another client has changed the data the client had a delegation for. this is an especially big problem for a client that held an OPEN_DELEGATE_WRITE delegation.

status bits returned by SEQUENCE operations give an alternate way to tell the client about backchannel problems and recalled delegations 🦋 when the backchannel isn't available, the server returns the status bit SEQ4_STATUS_CB_PATH_DOWN on SEQUENCE operations. the client can react by trying to re-establish the backchannel, and by returning recallable objects if the backchannel can't be successfully re-established.

whether the backchannel is working or not, the recalled delegation may still not get returned. the client's lease might still be renewed even though the recalled delegation isn't returned 👀 in this situation, servers SHOULD revoke delegations that aren't returned within a period of time equal to the lease period. that period should give the client time to notice the backchannel-down status and re-establish the backchannel 🌈

when delegations are revoked, the server will return the SEQ4_STATUS_RECALLABLE_STATE_REVOKED status bit set on subsequent SEQUENCE operations. the client should notice this and then use TEST_STATEID to find out which delegations have been revoked 🔥

---

10.4.6. Delegation Revocation 👑

at the point a delegation is revoked, if there are associated opens on the client, those opens may or may not be revoked. if no byte-range lock or open is granted that conflicts with the existing open, the stateid for the open may stay valid and be disconnected from the revoked delegation, just like what would happen if the delegation were returned ✨

for example, if an OPEN for OPEN4_SHARE_ACCESS_BOTH with a deny of OPEN4_SHARE_DENY_NONE is associated with the delegation, granting another such OPEN to a different client will revoke the delegation, but it doesn't need to revoke the OPEN, since the two OPENs are consistent with each other 💅 on the other hand, if an OPEN denying write access is granted, then the existing OPEN must be revoked 💀

when opens and/or locks are revoked, the applications holding those opens or locks need to be notified 🚨 this notification usually happens by returning errors for READ/WRITE operations, or when a close is attempted on the open file.

if no opens exist for the file at the point the delegation is revoked, then notifying anyone about the revocation is unnecessary. however, if there's modified data present at the client for the file, the user of the application should be notified 😭 unfortunately, it may not be possible to notify the user, since active applications may not be present at the client. see Section 10.5.1 for additional details.

---

10.4.7. Delegations via WANT_DELEGATION 🧚

in addition to giving out delegations as part of the reply to OPEN operations, servers MAY provide delegations separately from open, via the OPTIONAL WANT_DELEGATION operation 🦋 this lets a client get delegations in advance of an OPEN that might benefit from them, for objects that aren't a valid target of OPEN, or to handle cases where a delegation was recalled and the client wants to try to re-establish it, if the absence of use by other clients allows that.

the WANT_DELEGATION operation may be performed on any type of file object other than a directory.

when a delegation is obtained using WANT_DELEGATION, any open files for the same filehandle held by that client are treated as subordinate to the delegation, just as if they had been created using an OPEN of type CLAIM_DELEGATE_CUR. otherwise they stay the same as far as seqid, access and deny modes, and their relationship with byte-range locks go. similarly, because existing byte-range locks are subordinate to an open, those byte-range locks also become indirectly subordinate to that new delegation 🌸

the WANT_DELEGATION operation provides for delivery of delegations via callbacks, for when delegations aren't immediately available 📬 when a requested delegation becomes available, it's delivered to the client via a CB_PUSH_DELEG operation. when that happens, open files for the same filehandle become subordinate to the new delegation at the point the delegation is delivered, just as if they had been created using an OPEN of type CLAIM_DELEGATE_CUR. this also happens for existing byte-range locks that are subordinate to an open.

---

10.5. Data Caching and Revocation 🎀

when locks and delegations are revoked, the assumptions that successful caching depends on are no longer guaranteed 😬 for any locks or share reservations that have been revoked, the corresponding state-owner needs to be notified. this notification includes applications with a file open that has a corresponding delegation that's been revoked. cached data associated with the revocation must be removed from the client. if there's modified data sitting in the client's cache, that data must be removed from the client without being written to the server 💀 as mentioned, the assumptions the client made stop being valid at the point a lock or delegation is revoked. for example, another client may have been granted a conflicting byte-range lock after the first client's byte-range lock was revoked. so the data within the lock range may have been modified by that other client. obviously, the first client can't guarantee to the application what happened to the file after the revocation 🙃

notifying a state-owner will, in many cases, just mean returning an error on the next and all subsequent READs/WRITEs to the open file, or on the close. where the methods available to a client make that kind of notification impossible, because errors for certain operations may not be returned, more drastic action like signals or process termination may be appropriate 🚨 the reasoning is that an invariant the application depends on may have been violated. depending on how errors are normally handled in the client's operating environment, further levels of notification, including logging, console messages, and GUI pop-ups, may be appropriate 🌈

10.5.1. Revocation Recovery for Write Open Delegation 🎀

okay so revocation recovery for an OPEN_DELEGATE_WRITE delegation is lowkey the trickiest case 😭 because there can be modified data sitting in the client cache while the file is not open. 🙈 So any client that does not flush modified data to the server on each close must make sure the user gets proper notification that something failed because of the revocation. 🚨 Some of these situations need a human to fix things, so notification schemes that alert the right user or administrator may be necessary. Logging and console messages are the classic examples ✨

If there is modified data on the client, it must not be flushed normally to the server. 💅 A client may try to save a copy of the file data as modified during the delegation under a different name in the file system namespace, which makes recovery way easier 🫶 Note that when the client can tell the file has not been modified by any other client, or when the client has a complete cached copy of the file, that saved copy of the client's view can be super valuable for recovery 👑 In another case, recovering with a copy that is partly the client's cached data and partly the server's copy as modified by other clients is anything but straightforward 💀 So clients may skip saving file contents in these situations, or specially mark the results to warn users of possible problems 🌸

Saving modified data in delegation revocation situations may be limited to files of a certain size, or might only happen when there's sufficient disk space in the target file system 🍓 Saving may also be restricted to situations where the client has enough buffering resources to keep the cached copy available until it is properly stored to the target file system 🧚

---

10.6. Attribute Caching 🦋

this section is all about caching a file's attributes on a client that does not hold a delegation on the file 💖

The attributes discussed in this section do not include named attributes. 🌈 Individual named attributes are analogous to files, so caching their data needs to be handled just like data caching is for ordinary files. Similarly, LOOKUP results from an OPENATTR directory (as well as the directory's contents) are to be cached on the same basis as any other pathnames 🎀

Clients may cache file attributes obtained from the server and use them to avoid subsequent GETATTR requests ✨ This caching is write through, which means modifications to file attributes are always done by means of requests to the server. They should not be done locally and should not be cached 💅 The exception is modifications to attributes that are intimately connected with data caching 🫶 So extending a file by writing data to the local data cache is reflected immediately in the size as seen on the client, without this change being immediately reflected on the server 😭 Normally, such changes are not propagated directly to the server. But when the modified data is flushed to the server, analogous attribute changes are made on the server too 🔥 When OPEN delegation is in effect, the modified attributes may be returned to the server in reaction to a CB_RECALL call 👑

The result of local caching of attributes is that the attribute caches on individual clients will not be coherent 🙃 Changes made in one order on the server may be seen in a different order on one client and in a third order on another client 💀

Typical file system application programming interfaces do not provide ways to atomically modify or interrogate attributes for multiple files at the same time 😩 The following rules create an environment where those potential incoherencies can be reasonably managed. These rules come from the practice of previous NFS protocols 🌸

• All attributes for a given file (per-fsid attributes excepted) are cached as a unit at the client, so no non-serializability can arise within the context of a single file 💖
• An upper time boundary is maintained on how long a client cache entry can be kept without being refreshed from the server ⏰
• When operations are performed that change attributes at the server, the updated attribute set is requested as part of the containing RPC. This includes directory operations that update attributes indirectly. This is accomplished by following the modifying operation with a GETATTR operation, then using the results of the GETATTR to update the client's cached attributes 🧚

Note that if the full set of attributes to be cached is requested by READDIR, the results can be cached by the client on the same basis as attributes obtained via GETATTR 🍓

A client may validate its cached version of attributes for a file by fetching both the change and time_access attributes. It then assumes that if the change attribute has the same value it had when the attributes were cached, no attributes other than time_access have changed 🦋 The reason time_access is also fetched is that many servers operate in environments where the operation that updates change does not update time_access. For example, POSIX file semantics do not update access time when a file is modified by the write system call [18] 🤷‍♀️ So a client that wants a current time_access value should fetch it with change during attribute cache validation processing, and update its cached time_access 💅

The client may maintain a cache of modified attributes for those attributes intimately connected with data of modified regular files (size, time_modify, and change). ✨ Other than those three attributes, the client MUST NOT maintain a cache of modified attributes. Instead, attribute changes are immediately sent to the server 🚀

In some operating environments, the equivalent of time_access is expected to be implicitly updated by each read of the content of the file object 👀 If an NFS client is caching the content of a file object, whether it is a regular file, directory, or symbolic link, the client SHOULD NOT update the time_access attribute (via SETATTR or a small READ or READDIR request) on the server with each read that is satisfied from cache. 🙅‍♀️ The reason is that this can defeat the performance benefits of caching content, especially since an explicit SETATTR of time_access may alter the change attribute on the server 😤 If the change attribute changes, clients caching the content will think the content has changed, and will re-read unmodified data from the server 😭 The client isn't encouraged to keep a modified version of time_access in its cache either 🙈 It would either eventually have to write the access time to the server (bad performance effects) or never update the server's time_access. That would create a situation where an application that caches access time between a close and open of the same file sees the access time oscillating between the past and present 🌀 The time_access attribute always means the time of last access to a file by a read that was satisfied by the server. 🎯 This way clients will tend to only see time_access changes that go forward in time ⏩

10.7. Data and Metadata Caching and Memory Mapped Files ✨

Okay so some operating environments let an app map a file's content straight into its address space 🗺️ Every time the app touches a memory location that matches a block that hasn't been loaded yet, a page fault happens and the file gets read. (If the block doesn't exist in the file, it gets allocated and then instantiated in the app's address space instead 🧚)

As long as every memory-mapped access triggers a page fault, the attributes used to detect access and modification (time_access, time_metadata, time_modify, and change) will get updated 💅 But in lots of operating environments, when page faults aren't needed, these attributes won't get updated on reads or updates through memory access, whether the file is local or accessed remotely 😭 A client or server MAY fail to update attributes of a file that is being accessed via memory-mapped I/O. This has a few implications:

• If there's an application on the server that has memory mapped a file that a client is also accessing, the client might not be able to get a consistent change value to tell whether its cache is stale 🙈 A server that knows the file is memory-mapped could always pessimistically return updated values for change, so the app is forced to always get the most up-to-date data and metadata. But because of the negative performance implications, that behavior is OPTIONAL.
• If the memory-mapped file isn't being modified on the server, and is just being read by an application through the memory-mapped interface, the client won't see an updated time_access attribute. But in lots of operating environments, neither will any process running on the server 🤷‍♀️ So NFS clients are at no disadvantage compared to local processes 💜
• If there's another client memory mapping the file, and that client is holding an OPEN_DELEGATE_WRITE delegation, the same set of issues from the previous two bullet points applies 🌸 So when a server does a CB_GETATTR to a file that the client has modified in its cache, the reply from CB_GETATTR won't necessarily be accurate. As discussed earlier, the client's obligation is to report that the file has been modified since the delegation was granted, not whether it has been modified again between successive CB_GETATTR calls, and the server MUST assume that any file the client has modified in cache has been modified again between successive CB_GETATTR calls. Depending on the client's memory management system, this weak obligation may not be possible 🥺 A client MAY return stale information in CB_GETATTR whenever the file is memory-mapped.
• Mixing memory mapping and byte-range locking on the same file is problematic 🔥 Check out this scenario, where the page size on each client is 8192 bytes:
  • Client A memory maps the first page (8192 bytes) of file X.
  • Client B memory maps the first page (8192 bytes) of file X.
  • Client A WRITE_LT locks the first 4096 bytes.
  • Client B WRITE_LT locks the second 4096 bytes.
  • Client A, via a STORE instruction, modifies part of its locked byte-range.
  • Simultaneous to client A, client B executes a STORE on part of its locked byte-range.

Here the challenge is for each client to resynchronize so it gets a correct view of the first page 😵‍💫 In lots of operating environments, the virtual memory management systems on each client only know that a page is modified, not that a subset of the page (the part corresponding to the respective lock byte-ranges) has been modified. So it's not possible for each client to do the right thing, which is to write to the server only the portion of the page that's locked. For example, if client A simply writes out the page, and then client B writes out the page, client A's data is lost 💀

Moreover, if mandatory locking is enabled on the file, we've got a totally different problem 🙈 When clients A and B execute the STORE instructions, the resulting page faults require a byte-range lock on the entire page. Each client then tries to extend its locked range to the entire page, which results in a deadlock. Communicating the NFS4ERR_DEADLOCK error to a STORE instruction is difficult at best 😩

If a client is locking the entire memory-mapped file, there's no problem with advisory or mandatory byte-range locking, at least until the client unlocks a byte-range in the middle of the file 🫧

Given the above issues, the following are permitted:

• Clients and servers MAY deny memory mapping a file for which they know there are byte-range locks.
• Clients and servers MAY deny a byte-range lock on a file they know is memory-mapped.
• A client MAY deny memory mapping a file that it knows requires mandatory locking for I/O. If mandatory locking is enabled after the file is opened and mapped, the client MAY deny the application further access to its mapped file.

---

10.8. Name and Directory Caching without Directory Delegations 🗂️

The NFSv4.1 directory delegation facility (described in Section 10.9 below) is OPTIONAL for servers to implement. Even where it is implemented, it may not always be functional because of resource availability issues or other constraints 🥲 So it's important to understand how name and directory caching are done when there are no directory delegations. We'll cover these topics in the next two subsections 💖

10.8.1. Name Caching 🏷️

The results of LOOKUP and READDIR operations may be cached to avoid the cost of subsequent LOOKUP operations. Just like with attribute caching, inconsistencies can pop up among the various client caches 😬 To mitigate the effects of these inconsistencies, and given the context of typical file system APIs, an upper time boundary is maintained for how long a client name cache entry can be kept without verifying that the entry hasn't been made invalid by a directory change operation performed by another client ⏰

When a client isn't making changes to a directory that has name cache entries, the client needs to periodically fetch attributes for that directory to make sure it isn't being modified 👀 After determining that no modification has occurred, the expiration time for the associated name cache entries may be updated to be the current time plus the name cache staleness bound.

When a client is making changes to a given directory, it needs to determine whether other clients have made changes to the directory too 🕵️‍♀️ It does this by using the change attribute as reported before and after the directory operation, in the associated change_info4 value returned for the operation. The server is able to communicate to the client whether the change_info4 data is provided atomically with respect to the directory operation. If the change values are provided atomically, the client has a basis for determining, given proper care, whether other clients are modifying the directory in question ✨

The simplest way for the client to make this determination is to serialize all changes made to a specific directory 🎀 When this is done, and the server provides before and after values of the change attribute atomically, the client can simply compare the after value of the change attribute from one operation on a directory with the before value on the subsequent operation modifying that directory. When these are equal, the client is assured that no other client is modifying the directory in question 🦋

When serialization isn't used, and there may be multiple simultaneous outstanding operations modifying a single directory sent from a single client, figuring this out gets more complicated 🌀 If two such operations complete in a different order than they were actually performed, that might look like modification by another client. Where this appears to happen, the client needs to await the completion of all such modifications that were started previously, to see if the outstanding before and after change numbers can be sorted into a chain. In that chain, the before value of one change number matches the after value of a previous one, consistent with this client being the only one modifying the directory 🔗

In either of these cases, the client is able to determine whether the directory is being modified by another client. If the comparison indicates that the directory was updated by another client, the name cache associated with the modified directory is purged from the client 🧹 If the comparison indicates no modification, the name cache can be updated on the client to reflect the directory operation, and the associated timeout can be extended. The post-operation change value needs to be saved as the basis for future change_info4 comparisons 💾

As the scenario above shows, name caching requires that the client revalidate name cache data by inspecting the change attribute of a directory at the point when the name cache item was cached. This requires that the server update the change attribute for directories when the contents of the corresponding directory are modified 🔄 For a client to use the change_info4 information appropriately and correctly, the server must report the pre- and post-operation change attribute values atomically. When the server is unable to report the before and after values atomically with respect to the directory operation, the server must indicate that fact in the change_info4 return value. When the information isn't reported atomically, the client should not assume that other clients haven't changed the directory 🙅‍♀️

10.8.2. Directory Caching ✨

okay so READDIR results can be reused to skip later READDIR calls, bestie 📂. But just like with attribute and name caching, the different client caches can end up disagreeing with each other 😭. To keep that drama under control, and given how typical file system APIs work, these rules should be followed:

• Cached READDIR information for a directory that wasn't gathered in a single READDIR operation must always be a consistent snapshot of the directory contents. To check this, use a GETATTR before the first READDIR and after the last READDIR that contributes to the cache.
• An upper time boundary is kept to show how long a directory cache entry counts as valid before the client has to revalidate it. 🕰️

Revalidation works just like it did for name caching 💅. When the client isn't changing the directory, checking the directory's change attribute with GETATTR is enough, and the cache entry's lifetime can be extended at those checkpoints. When a client is modifying the directory, it needs to use the change_info4 data to figure out whether other clients are modifying it too. If it turns out that no other client is making changes, the client may update its directory cache to reflect its own changes ✨.

As we saw earlier, directory caching means the client has to revalidate its directory cache by inspecting the directory's change attribute at the point when the directory was cached 🔍. That only works if the server updates the change attribute whenever the contents of the directory change 🙌. For a client to use change_info4 correctly and appropriately, the server MUST report the pre- and post-operation change attribute values atomically with respect to the directory operation. When the server can't report the before and after values atomically, it MUST indicate that in the change_info4 return value. And when the information isn't reported atomically, the client should not assume that other clients haven't changed the directory 🚫.

---

10.9. Directory Delegations 🎀

10.9.1. Introduction to Directory Delegations

okay so directory caching in NFSv4.1 works pretty much like file caching did in earlier versions 📁. Clients usually cache directory info for however long the client decides. When that timeout runs out, the client asks the server whether the directory has changed. By caching attributes, clients make fewer GETATTR calls to the server just to validate things 💖. Frequently used files and directories, like the current working directory, have their attributes cached on the client, so some NFS operations can happen without making an RPC call at all ✨. Clients also keep a Directory Name Lookup Cache (DNLC) of name and inode info for recently looked-up entries, so they don't have to send LOOKUP calls to the server every single time those files get accessed 🙈.

This caching setup does a pretty good job of cutting network traffic in lots of environments 🌸. But it doesn't help much when there are tons of lookups for files that don't exist. These are the "misses," and in those cases the client has to keep asking the server so it keeps the application semantics reasonable and notices new directory entries quickly 🥲. Software development compilation is a classic example of high-miss activity 💻. Because of this, NFS's current behavior limits how well it scales and how well it works for sharing across wide areas 📉. Other distributed stateful file systems, like AFS and DFS, have shown that adding state around directory contents can seriously cut network traffic in high-miss environments 👑.

Delegation of directory contents is an OPTIONAL feature of NFSv4.1 🫶. Directory delegations bring the same kind of traffic savings that file delegations do. By letting clients cache directory contents read-only while still being told about changes, clients skip the constant checking of slowly changing directories, which means less network traffic and better client performance 🚀. Delegations also make it easier to tell whether other clients are changing the directory, which helps a lot when this client is making lots of changes and those changes aren't serialized 🧠.

Directory delegations get better namespace cache consistency through delegations and synchronous recalls, even without notifications 🔒. Plus, if time-based consistency is good enough, asynchronous notifications can give the client (and maybe the server too) real performance wins in common situations, like directories that change slowly or are super large 🌈.

10.9.2. Directory Delegation Design

NFSv4.1 adds the GET_DIR_DELEGATION (Section 18.39) operation so a client can ask for a directory delegation 🙋‍♀️. The delegation covers the directory's attributes and all of the entries in the directory. If either of those changes, the delegation gets recalled synchronously, and the operation that caused the change has to wait until the recall finishes ⏳. Changes to the attributes of directory entries won't cause the delegation to be recalled 💅.

Clients can also ask for notifications about certain events, on top of asking for delegations 🔔. Those events include changes to the directory's attributes, its contents, or both. If a client asks to be notified about an event, the server will tell it when that event happens. Getting a notification does not cause the delegation to be recalled for that client 🙌. Notifications are asynchronous, and they're a way to dodge recalls when a directory is changing so much that the pure recall model stops working well, while the client still gets a lot of benefit 🦋. Without notifications, once a delegation is recalled, the client has to refresh its directory cache, and that might not be very efficient for really large directories 😩.

The delegation is read-only. The client may not change the directory except by sending NFSv4.1 operations that modify the directory or its associated file attributes, so that the server knows about the changes 🧾. To keep the client's namespace in sync with the server, the server will notify the delegation-holding client (as long as it asked for notifications) about changes that result from that client's own directory-modifying operations. This way the client doesn't need to send GETATTR or READDIR operations to the server afterward 🎯. If a single client holds the delegation and makes changes to the directory itself (meaning those changes go through operations sent on a session tied to the client ID that holds the delegation), the delegation will not be recalled 💖. Multiple clients can hold a delegation on the same directory, but if any one of them modifies the directory, the server MUST recall the delegation from the other clients, unless those clients have arranged to be notified about that kind of modification 🚨.

The server can recall delegations at any time 🕰️. Usually it'll recall the delegation when the directory changes in a way the notifications don't cover, or when the directory changes and nobody asked for notifications. And if another client removes the directory a delegation was granted for, the server will recall that delegation 💀.

10.9.3. Attributes in Support of Directory Notifications

check Section 5.11 for the attributes that go with directory notifications 📝.

10.9.4. Directory Delegation Recall

The server recalls a directory delegation by sending a callback to the client 📞. It uses the same callback procedure it uses to recall file delegations 🔁. The server will recall the delegation when the directory changes in a way the notification doesn't cover. But the server doesn't need to recall the delegation just because an entry's attributes inside the directory changed 🙅‍♀️.

If the server notices that handing out a directory delegation is causing too many notifications to go out, it can stop handing out delegations for that directory, recall the ones it already granted, or both 📣. If a client tries to remove the directory a delegation was granted for, the server will recall all of the associated delegations 🧨.

A bunch of operations' implementation sections describe when notification or delegation recall would be needed in common situations 📚. A similar set of caveats to the ones listed in Section 10.2 apply here too 🤓.

• For CREATE, see Section 18.4.4.
• For LINK, see Section 18.9.4.
• For OPEN, see Section 18.16.4.
• For REMOVE, see Section 18.25.4.
• For RENAME, see Section 18.26.4.
• For SETATTR, see Section 18.30.4.

10.9.5. Directory Delegation Recovery

Recovery after a client or server restart for state on regular files has two main jobs: not breaking application guarantees around locked files, and getting cached updates delivered from the client 🔐. Neither of those jobs applies to directories protected by OPEN_DELEGATE_READ delegations and notifications 🙃. So there's no special process for reclaiming directory delegations after a client or server restart. The client can just set up a directory delegation the same way it did the first time 🌸.

---

11. Multi-Server Namespace 🌍

NFSv4.1 supports attributes that let a namespace stretch beyond the boundaries of a single server 🧚. It is RECOMMENDED that clients and servers support building these multi-server namespaces 💜. Using them is OPTIONAL, though, and for a lot of purposes a single-server namespace is perfectly fine 🫶. Multi-server namespaces are still really useful, because they separate a file system's logical position in the namespace from the (possibly changing) logistical and administrative stuff that decides which server a file system actually lives on 👑✨

11.1. Location Attributes ✨

okay so NFSv4.1 has some RECOMMENDED attributes 💖 that let a file system on one server be linked up with one or more copies of that same file system on other servers 🦋. Each location is described by a server address target (either a DNS name that stands for one or more IP addresses, or a literal IP address) plus the path to that file system inside its single-server namespace 🌸.

The fs_locations_info RECOMMENDED attribute lets you list one or more file system instance locations where the data for a given file system can be found 🔥. It's giving extra info too: stuff like priority for use, writability, currency, etc. So the client can actually pick the best copy 💅. It also helps the client switch between instances as smoothly as possible if it ever has to 🫶.

The fs_locations RECOMMENDED attribute is inherited from NFSv4.0 🎀. It only tells you where the data can be found, nothing more. Servers SHOULD make this attribute available whenever fs_locations_info is supported, but client use of fs_locations_info is to be preferred 👑.

11.2. File System Presence or Absence 🦋

A given location in an NFSv4.1 namespace (usually a multi-server namespace, but not always) can have a bunch of file system instance locations attached to it, via the fs_locations or fs_locations_info attribute 🌈. There might also be an actual current file system right there that you can reach with normal namespace operations, like LOOKUP 💖. When that's the case, the file system is said to be "present" at that spot. Clients will typically use it and only save the other location-related attributes for when the principal location is no longer available 😭.

When there is no actual file system at the namespace location in question, it's "absent" 🙈. An absent file system has nothing in it except the root. Any reference to it gets an error, NFS4ERR_MOVED, except for a small set of attributes that help figure out where to go next 💀. Note that if the server ever returns NFS4ERR_MOVED, it MUST support the fs_locations attribute and SHOULD support the fs_locations_info and fs_status attributes ✨.

The name is kind of misleading 🥺. It sounds like the file system was once present and then got moved away. That's only one possibility, though. A position in the namespace can be permanently absent, with the set of file system(s) from the location attributes being the only "real" version of it. The name NFS4ERR_MOVED reflects an earlier, more limited idea of what it does, but this error gets returned whenever the referenced file system is absent, whether it moved or not 🌸.

Except for GETATTR-type operations (more on those later 👀), when the current filehandle at the start of an operation is within an absent file system, that operation doesn't get performed. You get NFS4ERR_MOVED back to tell you the file system is absent on the current server 🚫.

GETFH can't succeed if the current filehandle is within an absent file system, so filehandles in an absent file system can't be transferred to the client 💅. If a client does have filehandles inside an absent file system, it's because it grabbed them while the file system was present and then it became absent later 🍓.

Here's the lowkey tricky part 🧚. The check for the current filehandle being in an absent file system happens at the start of every operation. So operations that change the current filehandle so it lands inside an absent file system won't error out 🙌. That's what makes combos like PUTFH-GETATTR and LOOKUP-GETATTR work for grabbing attribute info, especially location attribute info, which we'll get to below ✨.

The RECOMMENDED file system attribute fs_status can be used to check whether a given file system is present or absent 💜.

11.3. Getting Attributes for an Absent File System 🌷

When a file system is absent, most attributes aren't available. But the client still needs access to the small set of attributes that are available, especially the ones that tell it where the correct current locations for this file system are: fs_locations and fs_locations_info 🗺️.

11.3.1. GETATTR within an Absent File System 🎀

As mentioned above, GETATTR gets a special exception 💖. Attributes can be obtained for a filehandle within an absent file system. This exception only applies if the attribute mask has at least one attribute bit saying the client wants a result about an absent file system: fs_locations, fs_locations_info, or fs_status. If none of those are requested, GETATTR results in an NFS4ERR_MOVED error 🙈.

When GETATTR runs on an absent file system, the set of supported attributes is super limited 😭. Lots of attributes, including the ones that are normally REQUIRED, won't be available on an absent file system. Besides the attributes already mentioned (fs_locations, fs_locations_info, fs_status), the following attributes SHOULD be available on absent file systems. For RECOMMENDED attributes, they should be available at least to the same degree they're available on present file systems ✨.

• change_policy: This attribute is useful for absent file systems and can help summarize to the client when any of the location-related attributes change 🦋.
• fsid: This attribute should be provided so the client can figure out file system boundaries, including, in particular, the boundary between present and absent file systems 🌈. Its value MUST be different from any other fsid on the current server, and it doesn't need any particular relationship to fsids on whatever destination the client might get sent to 🔥.
• mounted_on_fileid: For objects at the top of an absent file system, this attribute needs to be available 💅. Since the fileid lives in the present parent file system, there shouldn't be any need to reference the absent file system to provide it 🧚.

Other attributes SHOULD NOT be made available for absent file systems, even when it's possible to provide them 🚫. The server shouldn't assume more info is always better, and it should avoid gratuitously giving extra info 🙅‍♀️.

When a GETATTR includes a bit mask for one of fs_locations, fs_locations_info, or fs_status, but the mask also includes attributes that aren't supported, GETATTR won't return an error. Instead, it returns the mask of the actual supported attributes along with the results 💖.

VERIFY/NVERIFY is kind of like GETATTR here, with one difference 👀. If the attribute mask doesn't include fs_locations, fs_locations_info, or fs_status, you get NFS4ERR_MOVED. The other difference: if the mask includes any attribute that isn't supported for an absent file system (and yes, that includes some normally REQUIRED attributes), that also causes an NFS4ERR_MOVED result 😭.

11.3.2. READDIR and Absent File Systems 📂

A READDIR run while the current filehandle is within an absent file system results in NFS4ERR_MOVED. No exception gets made for READDIR, unlike GETATTR 🙅‍♀️.

You can still fetch attributes for an absent file system through a READDIR on a directory in a present file system, as long as that directory contains the root directories of one or more absent file systems 🌸. Here's how that goes:

• If the attribute set requested includes one of fs_locations, fs_locations_info, or fs_status, then fetching attributes proceeds normally, and no NFS4ERR_MOVED indication is returned, even when rdattr_error is requested 🎉.
• If the attribute set requested does NOT include one of fs_locations, fs_locations_info, or fs_status, then if rdattr_error is requested, each directory entry for the root of an absent file system reports NFS4ERR_MOVED as the value of rdattr_error 🫠.
• If the attribute set requested doesn't include any of fs_locations, fs_locations_info, fs_status, or rdattr_error, then the root of an absent file system showing up in the directory makes the READDIR fail with an NFS4ERR_MOVED error 💀.
• When an attribute is unavailable because the file system is absent, even one that's ordinarily REQUIRED, it doesn't cause any error indication 🌈. The set of attributes returned for the root directory of the absent file system just gets trimmed down to the ones that are actually available 💅.

11.4. Uses of Location Information 🌍

The location-bearing attributes (fs_locations and fs_locations_info), together with the possibility of absent file systems, give you a bunch of important features for reliable, manageable, and scalable data access 💜.

When a file system is present, these attributes can provide alternative locations to access the same data 🔥. That helps when there are server failures, communication problems, or other stuff that makes continued access to the current file system impossible or just impractical 😭. Sometimes multiple alternate locations can be used at the same time for higher-performance access 🚀. Providing these alternate locations is called "replication" 🎀. Heads up though: sometimes the replicated sets of data aren't actually there, and the "replicas" are just different paths to the same data 🙈.

When a file system is present and then becomes absent, clients can be given a chance to keep accessing their data at an alternate location ✨. In that case, trying to use the data in the now-absent file system results in NFS4ERR_MOVED. At that point, the successor locations (usually just one, though multiple choices are possible) can be fetched and used to keep going 🦋. Moving the file system contents to the new location is called "migration" 🧳. But keep in mind that, like "replication," this term can also be used in cases where no actual data migration happens 💅.

Where a file system wasn't previously present, specifying its location is how file systems on one server get linked into a namespace defined by another server 🌸. That's what makes a general multi-server namespace facility possible 👑. A designation like this, standing in for an absent file system, is called a "referral" 🫶.

Because client support for location-related attributes is OPTIONAL, a server may (but doesn't have to) hide migration and referral events from clients that don't support them, for example by acting as a proxy 🧚. The server can tell whether a client supports these from the arguments of the EXCHANGE_ID operation (see Section 18.35.3) 💖.

11.4.1. File System Replication ✨

okay so the fs_locations and fs_locations_info attributes are basically a list of backup spots 🧚 they tell you where else to get your data, either instead of the current file system or alongside it 💖. the first time you touch a file system, the client should grab the alternate locations by asking for fs_locations or fs_locations_info, and bestie, fs_locations_info is the one to go for because it's preferred 👑

if server failures, communication problems, or other drama make it impossible (or just not practical) to keep using the current file system, the client can switch to those alternate locations and keep its data flowing 🦋 depending on the details listed in fs_locations_info, multiple locations can even be used at the same time to get better performance by using several paths between client and target file system. the way that works is so iconic 🔥

the alternate locations might be physical replicas of the file system data (usually read-only), or they might just be different paths to the same server, or they could be server clusters where multiple servers offer different ways into the same physical file system 🌸 how these different modes of file system transition get represented in fs_locations and fs_locations_info, and how the client handles the transition, will be explained in detail below 💅

multiple server addresses can point to the same actual server, whether they come from one entry with a DNS name that stands for a set of IP addresses, or from several entries that each have their own server address 🫶 you can tell two addresses belong to the same server when they share a so_major_id field inside the eir_server_owner field that EXCHANGE_ID returns (see Section 18.35.3) ✨

for a deeper look at how server address targets work with the server identity set by the server owner field, check out Section 11.5 🎀

11.4.2. File System Migration 🌈

when a file system is present and then becomes absent, clients can get the chance to keep accessing their data at an alternate location, as specified by fs_locations or fs_locations_info 🦋 usually what happens is a client is busy using the file system, gets an NFS4ERR_MOVED error, and then checks fs_locations or fs_locations_info to find where the data went now 💖 when fs_locations_info is used, you get extra info that describes how the client should handle moving over to the new server 👑

migration is super helpful for load balancing and general resource shuffling 💅 the protocol doesn't say how the file system gets moved between servers, though. a bunch of different server-to-server transfer mechanisms are expected to be used, and the server implementor gets to pick ✨ what NFSv4.1 does specify is how the migration event gets communicated between client and server 🧚

the new location might be another communication path to the same server, or, with certain server clusters, a different server that gives access to the same physical file system 🌸 what the client needs to do during this transition depends on the exact nature of the new access path, plus how and whether the data was actually migrated 🔥 we'll get into those issues in detail below 💀

when multiple server addresses point to the same actual server (shown by a shared so_major_id value in the eir_server_owner field from EXCHANGE_ID), the location or locations can name alternate server addresses as specific server network addresses 🫶 those addresses can be used to reach the file system, and they're useful when it's no longer accessible at the original address 🎀

one successor location is the norm, but you can get multiple locations, along with info in fs_locations_info that shows which choice has priority 🌈 where it makes sense, clustering mechanisms can offer several identical file systems or paths to them, giving the client room to deal with any resource or communication problems that could limit data availability 💜

when an alternate location is picked as the migration target, it must designate the same data, with the metadata matching to the degree that fs_locations_info indicates. where file systems are writable, any change made on the original file system must be visible on all migration targets. where a file system isn't writable but is a read-only copy (possibly updated now and then) of a writable one, similar rules apply to how updates get propagated. any change already visible in the original file system must be fully in effect on all migration targets, so a client switching over never sees the file system state go backwards 😭

11.4.3. Referrals 🧚

referrals are a way to put a file system in a spot in the namespace pretty much without caring where it physically lives on a given server 💖 that lets one server, or a group of servers, present a multi-server namespace that covers file systems sitting on lots of different servers 🌸 likely uses include site-wide or organization-wide namespaces, or even stitching them together into a truly global namespace 🌍✨

referrals happen when a client figures out, the first time it references a spot in the current namespace, that this is part of a new file system and that the file system is absent 🦋 when that happens, usually because of an NFS4ERR_MOVED error, the client can find out where the file system actually is by fetching fs_locations or fs_locations_info 👑

the locations-related attribute can point to one file system location or several, picked based on what the client needs 🎀 in fs_locations_info, the server can attach priorities to the different location choices. it can assign different priorities to different locations for different clients, to match a client's physical location or to balance load 💅 when there are both read-only and read-write file systems around, some read-only locations might not be totally up-to-date (which they'd have to be for replication and migration) 🫶 servers can also use client-substituted variables in file system locations, so different clients get sent to different file systems (with different data) based on things like CPU architecture 🔥

when fs_locations_info lists multiple possible targets, the relationships between them can matter a lot to the client when it picks one 🌈 the same rules from Section 11.4.1 about data propagation apply to these replicas too 💜 for example, the client might prefer a writable target on a server that has more writable replicas it could later switch to 🦋 and unlike replication, there's no need to worry about propagating updates made by the current client here, because the current client hasn't accessed that file system yet 💖

multi-server namespaces are enabled by NFSv4.1, but using them isn't required 👑 how much they get used, and their scope, depends on the applications and the system administrator's preferences 🧚

a multi-server namespace can be set up by a single server that provides a big set of referrals to all the included file systems 🌸 or it can be administratively split up, with separate referral file systems (on separate servers) for each separately administered part of the namespace 🎀 the top-level referral file system, or any segment, can use replicated referral file systems for higher availability ✨

usually, multi-server namespaces are pretty uniform: the same data a client sees at a given location in the namespace is what every other client sees at that location too 💅 that said, there are facilities that let different clients be directed to different sets of data, so the namespace can adapt to things like CPU architecture 🌈

11.5. Location Entries and Server Identity 🦋

like we said above, one location entry can have a server address target in the form of a DNS name that stands for multiple IP addresses, and multiple location entries can each have their own server address targets that point at the same server 💖 two IP addresses are the same server when they share a common so_major_id field inside the eir_server_owner field returned by EXCHANGE_ID (see Section 18.35.3), but that's still subject to further verification (for the details, see Section 2.10.5) 🔥

when multiple addresses for the same server exist, the client can assume that for each file system in the namespace of a given server network address, there are file systems at matching namespace locations for each of the other server network addresses 🫶 it can do this even if fs_locations and fs_locations_info don't list them explicitly 🧚 those corresponding file system locations can be used as alternate locations, just like the ones explicitly specified in fs_locations and fs_locations_info ✨ if these specific addresses are explicitly designated in fs_locations_info, the conditions of use in that attribute (like priorities or simultaneous use) may limit how the client uses those alternate locations 💅

if a single location entry lists multiple server IP addresses, the client can't assume they're multiple paths to the same server 🙈 in most cases they will be, but the client MUST verify that before acting on that assumption 👑 when two server addresses in one location entry correspond to different servers, that usually points to some sort of misconfiguration, so the client should avoid those location entries when there are alternatives 😭 if there aren't any alternatives, clients should pick one of the IP addresses and use it, without using the others that don't go to the same server 💀🎀

11.6. Additional Client-Side Considerations ✨

okay so when clients use servers that do referrals, replication, and migration, they need to keep things coherent 🧩 A user who mounts a file system with a referral or a relocated file system should still see one clean picture of their user-side file system, even though it's secretly made of a bunch of server-side file systems that might live on different servers 🫶

One big thing to watch is upward navigation 🆙 That's going from the root of a server-side file system up to its parent (the .. in UNIX). It gets tricky when the client landed in that file system because of a referral, migration, or a replication transition. When the client is at that spot and wants to ascend, it must go back to the parent as seen in the multi-server namespace. It should NOT send a LOOKUPP to the server, because that would give back the parent inside that server's single-server namespace, which is not the same parent 🙃 So the client needs to remember the filehandles for those file system roots and use them instead of sending LOOKUPP to the current server. This keeps the namespace consistent for apps, so going up and going down always match up 💅

Another thing is refreshing referral locations 🔄 When referrals get used a lot, they can change as server configurations change. Clients are expected to cache the info they need for traversing referrals, so future requests can be resolved locally without bugging the server. This is usually built on client-side name lookup caching. Clients should periodically purge that data for referral points so they can pick up location changes. And when the change_policy attribute changes for directories holding referral entries, or for the referral entries themselves, clients should treat any cached referral info tied to them as out of date 🗑️✨

11.7. Effecting File System Transitions 🦋

Transitions between file system instances are the kind of thing we deal with all together, whether they happen because a replica goes down and we switch, or because the server starts a migration 🚚 For the server, practical stuff usually means planned and unplanned transitions get built differently. Still, even though replication and migration look pretty different in their typical use cases, when you think about every possibility, they're secretly the same thing from the client's point of view. That's why it makes sense to treat them as one topic 🫶

Servers have a bunch of options for copying data and tracking client state, so clients can switch between file system instances with minimal disruption 🌈 On one end, you've got inter-server clustering that keeps the changes the client sees super small. On the other end, you've got less aggressive setups that use more standard data replication, which puts more of the adapting work on the client 😅

NFSv4.1 doesn't force clients and servers to pick a spot on that spectrum. There are lots of valid choices, depending on what the client and app need and how those line up with what the server implements 💜 What NFSv4.1 does do is define the specific choices that are possible, how those choices get communicated to the client, and how the client should handle any discontinuities.

Below, we're going to bring up different possible server implementation choices, just to show the range of transition scenarios clients might run into 🧚 This isn't meant to define or limit how servers are built. It's just a way to show all the issues a client could face.

In the discussion, we'll talk about a file system having a particular property, or about two file systems (usually the source and the destination) that belong to a shared class of some type. Two file systems in the same class share important behaviors that clients can rely on when they're present, which makes a seamless transition way easier 🌸 If the file systems don't share a class, the client has to handle various implementation discontinuities, and those can cause performance or other problems during the transition 😬

When the fs_locations_info attribute is available, this classification data is handed straight to the client (see Section 11.10 for details). When only fs_locations is available, the client has to infer default assumptions about these classifications (see Section 11.9 for details) 🔍

Also, if one server is going to accept opaque values from the client that came from a different server, the servers SHOULD encode those "opaque" values in big-endian byte order. Doing that means servers acting as replicas or immigrating file systems can parse values like stateids, directory cookies, filehandles, and so on, even if their native byte order is different from the other servers working on the replication or migration 🧠✨

11.7.1. File System Transitions and Simultaneous Access 💫

So when one file system can be accessed at multiple locations, that's the situation we're in 👀 This happens either because the fs_locations or fs_locations_info attributes say the file system identity is the same, or because two file system instances have corresponding locations on server addresses that go to the same server (shown by a common so_major_id field in the eir_server_owner field returned by EXCHANGE_ID). Depending on the circumstances we'll get into below, the client will either:

• Access multiple instances at the same time, each one being an alternate path to the same data and metadata.
• Access one instance (or set of instances) and then transition to another instance (or set of instances) because of network issues, server unresponsiveness, or a server-directed migration. The transition might involve changes to filehandles, fileids, the change attribute, and/or locking state, depending on the attributes of the source and destination file system instances, as given in the fs_locations_info attribute.

Which of these is possible, and how a transition actually happens, is controlled by equivalence classes of file system instances reported in the fs_locations_info attribute 🗂️ It's also controlled, for file system instances in the same location within a multi-homed single-server namespace, by the value of the so_major_id field in the eir_server_owner field returned by EXCHANGE_ID 💖

11.7.2. Simultaneous Use and Transparent Transitions 🎀

When two file system instances have the same location in their own single-server namespaces, and those two server network addresses point to the same server (same so_major_id value in the eir_server_owner field returned from EXCHANGE_ID), those instances can be treated as the same. They can be used together at the same time or one after another, with no transition activity needed from the client 🙌 We call this kind of transition "transparent". The client moving access from one to the other is doing the same thing it would do if communication got interrupted, which means a new connection and maybe a new session gets set up to keep accessing the same file system 🦋

Whether you can use the two instances at the same time depends on whether fs_locations_info shows them in the same simultaneous-use class. Check Section 11.10.1 for the definitions of the different use classes, including the simultaneous-use class 📚

Here's the thing though. For two file systems like this, if fs_locations_info says the two instances need special transition activity (like showing up with different handle, fileid, write-verifier, change, or readdir classes), that's a serious problem 🚨 The client must not treat it as a transparent transition, and it should only allow the transition at all if it's okay with that. The server SHOULD NOT say these instances belong to different handle, fileid, write-verifier, change, or readdir classes. That's true whether or not the two instances are shown in the same simultaneous-use class 😤

If those conditions don't apply, then it's a non-transparent file system instance transition 🔀 The details depend on the handle, fileid, write-verifier, change, and readdir classes of the two instances, plus whether the two servers' addresses have the same eir_server_scope value reported by EXCHANGE_ID 🫠

11.7.2.1. Simultaneous Use of File System Instances 🌟

When the conditions in Section 11.7.2 hold, the client may use the two file system instances at the same time in either of these two cases:

• The fs_locations_info attribute doesn't have separate per-network-address entries for file system instances at the distinct network addresses. This also covers the case where fs_locations_info isn't available at all. Here, the two server addresses connecting to the same server (shown by them sharing the same so_major_id value, and then confirmed as described in Section 2.10.5) is enough to justify simultaneous use, as long as nothing in fs_locations_info contradicts that 🧐
• The fs_locations_info attribute says the two file system instances belong to the same simultaneous-use class.

In this case, the client may use both instances at once, as representations of the same file system 🪞 That could be because the two network addresses go to the same physical server, or because different servers connect to clustered file systems and export their data in common. When simultaneous use is in effect, any change made to one file system instance must be immediately reflected in the other file system instance(s) ⚡ Locks are treated as part of a common lease, tied to a common client ID 🔐 And depending on the details of the eir_server_owner returned by EXCHANGE_ID, the two server instances might be accessed by different sessions or by one session in common 💅

11.7.2.2. Transparent File System Transitions ✨

Okay bestie, this is the dreamy part 🦋. When the conditions from Section 11.7.2.1 hold, AND fs_locations_info explicitly says these different network addresses belong to different simultaneous-use classes, the file system instances should not be used by the client at the same time 🙅‍♀️. Instead, they get used one at a time in a serial, taking-turns kind of way. You keep using one instance unless and until communication difficulties, lack of responsiveness, or an explicit migration event makes the client switch to another instance (or to a set of instances sharing a common simultaneous-use class) 💜.

When a switch is happening, the client keeps using the same client ID that's already in effect 🎀. If the client already has connections to the new server address, those connections get used 🙌. If not, new connections to existing sessions, or new sessions tied to the existing client ID, get set up. Which one is used is decided by the eir_server_owner that comes back from EXCHANGE_ID 🧚.

In every one of these transparent transition cases, the following apply 💅:

• If filehandles are persistent, they stay the same. If filehandles are volatile, they either stay the same or expire, but the expiration can't be blamed on the file system transition 🌸.
• Fileid values do not change across the transition 🍓.
• The file system will have the same fsid in both the old and new locations 👑.
• Change attribute values stay consistent across the transition, so they don't have to be refetched. When change attributes say a cached object is still valid, it can stay cached 💖.
• Client and state identifiers keep their validity across the transition, EXCEPT where their staleness gets recognized and reported by the new server. Unless that staleness forces it, no lock reclamation is needed 🔒. Any such staleness is a sign that the server should be treated as having restarted, and it gets reported the way Section 8.4.2 explains 🚨.
• Write verifiers are presumed to stay valid, so they can be compared with the verifiers returned by COMMIT on the new server. If COMMIT on the new server returns an identical verifier, then the new server is expected to have all the data that was written unstably to the original server, and to have committed that data to stable storage as requested 💾.
• Readdir cookies are presumed to stay valid too, so they can be handed to later READDIR requests together with the readdir verifier they're associated with. When the verifier is accepted as valid, the cookie continues the READDIR operation, so the client can get the entire directory 📂✨.

---

11.7.3. Filehandles and File System Transitions 🎟️

There are a bunch of ways filehandles can be handled across a file system transition. They fall into two big buckets, depending on whether the two file systems share enough state to keep some kind of continuity going 🤝.

When there's no cooperation in filehandle assignment, the two file systems are reported as being in different handle classes. In that case, all filehandles are assumed to expire as part of the transition 💀. Heads up: this behavior doesn't depend on the fh_expire_type attribute, and it overrides the FH4_VOL_MIGRATION bit spec, which only matters when fs_locations_info isn't available 🙈.

When there IS cooperation in filehandle assignment, the two file systems are reported as being in the same handle classes. Then persistent filehandles stay valid after the transition 🥰. Volatile filehandles are still subject to expiring on the target server, EXCEPT the ones that are only volatile because of the FH4_VOL_MIGRATION bit 🌈.

---

11.7.4. Fileids and File System Transitions 🦄

Okay real talk, in NFSv4.0 nobody really dealt with fileid continuity during a file system transition 😬. The general vibe was: if a single vendor made both instances using some kind of file system image copy, fileids stay consistent. But in multi-vendor transitions, they wouldn't. That's rough, especially for a client that doesn't know what transition mechanisms the server is using 😩. Note that even though fileid isn't a REQUIRED attribute, lots of servers support it and lots of clients have APIs that depend on it 🧠.

Here's the tea 🍵: clients themselves might be totally fine with a fileid changing during a transition, but applications can see the fileid (for example, through stat). So an application can work perfectly fine when there's no transition, or when any transition happens between instances from one vendor. But if a multi-vendor transition lands at the wrong moment, it might not cope 😭.

Providing the same fileids across multiple server vendors has generally been held to be pretty difficult. There's work left to do, but it's also kinda self-inflicted 🤦‍♀️. Servers have usually identified fileid with the inode number, which is basically a value used to find the file. That makes migrating a file system between vendors hard, because giving the same index to a given file might not be possible 🙃. Here's the key bit: a fileid doesn't have to be useful for finding the file. It only has to be unique within the given file system ✅. Servers that can accept a fileid as one piece of metadata, stored separately from the index used to locate the file, can pretty easily keep the fileid value across a migration event. That allows a truly transparent migration event 🦋✨.

In any case, when servers CAN provide continuity of fileids, they should, and the client should be able to find out that continuity is available and act on it 👀. Information about whether fileids stay continuous (or don't) across a transition gets represented by saying whether the file systems are in the same fileid class 🏷️.

Now, the spicy part 🌶️. When consistent fileids don't exist across a transition (either because there's no continuity, or because fileid isn't a supported attribute on one of the instances), AND there are no reliable filehandles across the transition (either because there's no filehandle continuity, or because the filehandles are volatile), the client can't verify that the files it was using before the transition are still the same objects 😱. It has to assume no object has been renamed. Unless something guarantees that (for example, the file system is read-only), applications may run into problems 🙀. So these configurations should only be used where the problems they might cause can be tolerated 🫶.

---

11.7.5. Fsids and File System Transitions 🧩

Fsids are generally only unique on a per-server basis, so they're likely to change during a file system transition 🔄. The exception is transparent transitions. But in that case, there are multiple network addresses defined as the same server, which is shown by a common value of the so_major_id field of eir_server_owner 💞. Clients should not show the fsids they get from the server to applications, since they might not be globally unique and they might change during a transition 🙅‍♀️. Apps are happiest when they're shielded from those transitions as much as possible 🛡️.

Normally a single source file system transitions to a single target file system. But there's a provision for splitting one source file system into multiple target file systems, using the FSLI4F_MULTI_FS flag 🪓✨.

---

11.7.5.1. File System Splitting ✂️

When a file system transition is made and fs_locations_info says the file system might be split into multiple file systems (via the FSLI4F_MULTI_FS flag), the client SHOULD do GETATTRs to find the fsid attribute on all known objects within the file system going through the transition. That way it can figure out the new file system boundaries 🗺️.

Clients may keep the fsids that were passed to existing applications by mapping all of the fsids for the descendant file systems to the common fsid used for the original file system 🎭.

Splitting a file system may happen on a transition between file systems of the same fileid class 💜. The reason it works: fileids are unique within the source file system, so they'll stay unique in each of the target file systems 🧁.

---

11.7.6. The Change Attribute and File System Transitions 🕰️

The change attribute is defined as server-specific, so change attributes fetched from one server are normally presumed invalid on another 🙅. That's a problem, because it would invalidate every cached change attribute and force everything to be refetched 😵. Even worse, with no assured continuity for the change attribute, even if the same value comes back on a refetch, you can't conclude whether the object changed. That identical change attribute could just be a side effect of a modified file using a different change attribute construction algorithm, where the new algorithm happened to produce the same value 🤯.

When the two file systems have consistent change attribute formats, AND that's communicated to the client by reporting them in the same change class, the client may assume continuity of change attribute construction. Then it can handle the situation just like it would without any file system transition at all 🌟💅.

11.7.7. Lock State and File System Transitions ✨

Okay so when a file system transitions, the client has to be ready for two different situations 💅: one where the two servers cooperated on state management, and one where they didn't. Cooperating means the servers have to coordinate their client IDs. So before the client tries to use a client ID from one server in a request to the other file system's server, it has to rule out the chance that two servers that didn't cooperate accidentally handed out the same client ID 😭. To check, the client compares the eir_server_scope values returned by each server. If the scopes don't match, the servers didn't cooperate. If they do match, then the servers have cooperated enough to reject client IDs that refer to state they don't know about 🫶. See Section 2.10.4 for more on server scope.

In the case of migration, the servers involved SHOULD transfer all server state from the original server to the new one 🦋. And when that happens, it must be done in a way that's totally transparent to the client. With replication, that level of shared state is usually not the case 🙈. Clients should still look at the eir_server_scope returned by EXCHANGE_ID (as tweaked by the validation procedures in Section 2.10.4) to figure out whether sharing might be going on. Don't make assumptions based on the reason for the transition, bestie 🧠.

This state transfer is a huge deal because it cuts down on disruption when a transition happens 💖. If the servers successfully transfer all the state, the client can try to set up sessions tied to the client ID it used on the source file system instance. If the server accepts that as a valid client ID, then the client may keep using the existing stateids from the old file system instance with that same client ID on the transitioned file system instance 👑. And if the client already had a client ID on the target system, it may check the stateid values from the source system under that new client ID. The assurance is that if they're accepted as valid, they represent properly transferred lock state for the source file system, which has moved over to the target server ✨.

When the two servers belong to the same server scope, that does not mean the client gets to skip reclaiming state during the transition 😅. What it does mean is that the client may keep using its current client ID when it talks to the new server. The new server will either recognize that client ID as valid or reject it, and if it rejects it, the locks have to be reclaimed by the client 🔁.

File systems that cooperate on state management might actually share state, or they might just split up the identifier space so each one recognizes (and rejects as stale) the other's stateids and client IDs 🧩. Servers that do share state might not do it under every condition or at all times 🌙. So if a server can't be sure, when it accepts a client ID, that the ID reflects the locks the client was given, it must treat all associated state as stale and report it that way to the client 💀.

When the two file system instances live on servers that don't share a server scope value, the client must establish a new client ID on the destination if it doesn't already have one, and reclaim locks if the server allows it 🌸. In this situation, old stateids and client IDs should not be presented to the new server, since there's no guarantee they won't clash with IDs that are valid on that server 🙅‍♀️. Also, lock reclaim may be attempted even when the servers involved have different server scope values (see Section 8.4.2.1 for the opposite case, reclaim after a server reboot) 🔄. Servers with different server scope values can still cooperate to allow reclaim for locks tied to a file system transfer, even if they don't cooperate enough to share a server scope 🤝.

Either way, when it isn't known whether actual locks are being maintained, the destination server may set up a grace period specific to that file system. During that grace period, non-reclaim locks get rejected for that file system, even though normal locks are being granted for other file systems ⏳. Clients should not infer that a file system being transitioned to a server has no grace period based on how the server responded to requests for other file systems 🚫.

Lock reclamation after a file system transition can run into edge cases similar to those after a server restart 😬. (With the planned state transfer that comes with migration, though, these can be avoided by securely recording lock state as part of the state migration 🔐.) Unless the destination server can guarantee that locks won't be incorrectly granted, it should not allow lock reclaims and should avoid establishing a grace period 🛑.

Once all locks have been reclaimed, or if there were no locks to reclaim, the client says "that's it, no more reclaims for this file system" by sending a RECLAIM_COMPLETE operation with the rca_one_fs parameter set to true 🎀. After that, non-reclaim locking operations can happen, and any later attempt to do reclaims gets rejected with the error NFS4ERR_NO_GRACE 🚫.

Client identity info can be passed between servers in the form of client_owner4 and its associated verifiers, under the assumption that the client presents the same values to all the servers it deals with 🪪.

Servers are encouraged to offer ways to reclaim locks on the new server after a file system transition 🌈. But often, when the two servers don't share a server scope value, those facilities aren't available. So the client should be ready to re-obtain its locks, even though its LOCK or OPEN request might get denied because of a conflicting lock 😤.

What happens when there's no way to reclaim locks on the new server depends on the environment 🌍. In some places, like a transition between read-only file systems, denied locks shouldn't cause huge problems in practice 📚. When an attempt to re-establish a lock on the new server gets denied, the client should treat it as if its original lock had been revoked 💔. And even when the lock is granted, the client can't assume that no conflicting lock was granted in the meantime 👀. If change attribute continuity is present, the client may check the change attribute to look for unwanted file modifications 🔍. If even that isn't available and the file system isn't read-only, a client may reasonably treat all pending locks as revoked 🫠.

11.7.7.1. Leases and File System Transitions 🦋

In the case of lease renewal, the client might not be sending requests for a file system that has been transferred to another server 🤔. That can happen because of how the lease renewal mechanism works: the client renews the lease for all file systems whenever it submits a request on an associated session, no matter which file system the request is actually for 🔁.

So the client can schedule renewal of its lease when there's locking state that may have moved to the new server, it has to find out about the lease relocation before that lease expires ⏰. To make that happen, the SEQUENCE operation returns the status bit SEQ4_STATUS_LEASE_MOVED if responsibility for any of the renewed locking state has been transferred to a new server 📬. That bit keeps showing up until the client receives an NFS4ERR_MOVED error for each file system where locking state got relocated 🏠.

When a client gets a SEQ4_STATUS_LEASE_MOVED indication from a server, it should perform an operation for each file system on that server where the client has locking state 🧾. For simplicity, the client may choose to reference all file systems. But what really matters is that it must reference every file system that had locking state which has since moved 📦. Once the client receives an NFS4ERR_MOVED error for each of those file systems, the server clears the SEQ4_STATUS_LEASE_MOVED indication ✨. The client can stop checking file systems once that indication is cleared (but only if it has received a reply for all outstanding SEQUENCE requests on all of its sessions with the server), since there are no others whose locking state moved 🏁.

A client can use GETATTR on the fs_status (or fs_locations_info) attribute across all of the file systems to get absence indications in a single request (or a few) 🧺. Absent file systems won't cause an error in this context, which is neat 😌. Even so, the client still has to do an operation that gets NFS4ERR_MOVED on each file system in order to clear the SEQ4_STATUS_LEASE_MOVED indication 🧹.

Once the client knows which file systems had their locking state transferred, it can follow the normal process to get the new server information (through the fs_locations and fs_locations_info attributes) and renew that lease on the new server 🗺️. The exception is when information in fs_locations_info shows that no state could have been transferred 🙅. If the server didn't have state transferred to it transparently, the client will get NFS4ERR_STALE_CLIENTID from the new server, as described above. At that point, the client can reclaim its locks the same way it would after a server failure 🔄💖.

11.7.7.2. Transitions and the Lease_time Attribute 🌸

To manage its lease properly during a file system transition, the client needs the destination server to set up correct values for the lease_time attribute ⏳.

When state gets transferred transparently, that state should include the correct value of the lease_time attribute 🎀. The lease_time attribute on the destination server must never be less than the value on the source, because a smaller value would cause a lease granted by the source server to expire too early 💀. When state is transferred transparently, the client doesn't have to re-fetch the lease_time attribute, and it may keep using the value it got earlier from the source server 👑.

If state hasn't been transferred transparently, either because the servers show different eir_server_scope strings or because the new server rejects the client ID, the client should fetch the value of lease_time from the new (that is, destination) server and use it for later locking requests 🔍. However, the server must respect a grace period at least as long as the lease_time on the source server 🕰️. That way, clients have plenty of time to reclaim their locks before any potentially conflicting non-reclaimed locks get granted 🫶🦋.

11.7.8. Write Verifiers and File System Transitions 🧦

okay so, when a file system transition happens, the two file systems might be clustered together in how they handle unstably written data 🫶 when that's true, and they both belong to the same write-verifier class, the verifiers one system hands back can be compared with the ones from the other, and you can skip the superfluous writes ✨ iconic efficiency.

but if the two file systems belong to different write-verifier classes, any verifier one of them generated must not be compared to one the other gave back 🙅‍♀️ instead, it should be treated as not equal, even when the values look identical 😭 same numbers, different classes, not the same vibe!! 💅

11.7.9. Readdir Cookies and Verifiers and File System Transitions 🍓

in a file system transition, the two file systems might be consistent in how they handle READDIR cookies and verifiers 🌸 if that's the case and they belong to the same readdir class, cookies and verifiers from one system can be recognized by the other, and READDIR operations started on one server can be validly continued on the other. you just present the cookie and verifier that a READDIR done on the first file system returned, straight to the second one, and that's it 🦋 so easy, so clean.

now if the two file systems belong to different readdir classes, any READDIR cookie and verifier generated by one is not valid on the second, and the client must not present it to that server 🚫 the client should act as if the verifier was rejected 💀 no sneaking it through, bestie!

11.7.10. File System Data and File System Transitions 🌈

when multiple replicas exist and get used at the same time or one after another by a client, the apps using them will usually expect the replicas to hold either the same data or data that's consistent with the normal kinds of changes other clients make while updating the file system's data (with the metadata matching to the degree that the fs_locations_info attribute says) ✨ but here's the tea: when multiple file systems are presented as replicas of each other, the exact relationship between one one's data and another's isn't, in general, specified by the NFSv4.1 protocol 🙈 it's totally possible to present file systems as replicas where the data is different enough that some apps struggle with switching between them 😬

the namespace is usually built so apps can pick the level of support they want 🎀 so in one spot you might see a whole mixed bag of replicas listed, while somewhere else only the up-to-date ones count as replicas. 💖 the protocol does define four special cases of the replica relationship that the server has to specify and clients can rely on 👑

• When multiple server addresses correspond to the same actual server, as indicated by a common so_major_id field within the eir_server_owner field returned by EXCHANGE_ID, the client may depend on the fact that changes to data, metadata, or locks made on one file system are immediately reflected on others. 🦋
• When multiple replicas exist and are used simultaneously by a client (see the FSLIB4_CLSIMUL definition within fs_locations_info), they must designate the same data. Where file systems are writable, a change made on one instance must be visible on all instances, immediately upon the earlier of the return of the modifying requester or the visibility of that change on any of the associated replicas. This allows a client to use these replicas simultaneously without any special adaptation to the fact that there are multiple replicas. In this case, locks (whether share reservations or byte-range locks) and delegations obtained on one replica are immediately reflected on all replicas, even though these locks will be managed under a set of client IDs. 🔥 this one is giving total synchronization energy!!
• When one replica is designated as the successor instance to another existing instance after return NFS4ERR_MOVED (i.e., the case of migration), the client may depend on the fact that all changes written to stable storage on the original instance are written to stable storage of the successor (uncommitted writes are dealt with in Section 11.7.8). 🧚 the migration glow-up, where nothing written to stable storage gets left behind!
• Where a file system is not writable but represents a read-only copy (possibly periodically updated) of a writable file system, clients have similar requirements with regard to the propagation of updates. They may need a guarantee that any change visible on the original file system instance must be immediately visible on any replica before the client transitions access to that replica, in order to avoid any possibility that a client, in effecting a transition to a replica, will see any reversion in file system state. The specific means of this guarantee varies based on the value of the fss_type field that is reported as part of the fs_status attribute (see Section 11.11). Since these file systems are presumed to be unsuitable for simultaneous use, there is no specification of how locking is handled; in general, locks obtained on one file system will be separate from those on others. Since these are going to be read-only file systems, this is not expected to pose an issue for clients or applications. 🌸 read-only copies don't get to rewind time, period!

11.8. Effecting File System Referrals 🧚‍♀️

referrals get effected when an absent file system is encountered and one or more alternate locations are made available by the fs_locations or fs_locations_info attributes ✨ the client will usually get an NFS4ERR_MOVED error, fetch the right location info, and then go access the file system on a different server, even though it keeps its logical position within the original namespace 💅 referrals are different from migration events, because they only happen when the client has never referenced the file system in question (so there's nothing to transition). 🙈 and referrals can only come into effect when an absent file system is encountered at its root.

the examples in the sections below are kinda artificial, ngl 🤭 an actual client usually won't do a multi-component lookup, and it'll have cached info about the upper levels of the name hierarchy. but these examples are picked to make the required behavior clear and easy to fit into a small number of requests, without getting too deep into how specific clients might choose to cache things 🫶 simple, clean, no cap.

11.8.1. Referral Example (LOOKUP) ✨

Okay, picture this: the client sends a COMPOUND in an environment where /this/is/the/path is absent from the target server 🧚. There could be a few reasons for that. Maybe the file system has moved, or maybe the target server is mainly there to refer clients to the servers where the various file systems live 💅.

PUTROOTFH
LOOKUP "this"
LOOKUP "is"
LOOKUP "the"
LOOKUP "path"
GETFH
GETATTR (fsid, fileid, size, time_modify)

Here's the tea on what happens 🫶:

PUTROOTFH --> NFS_OK.  The current fh is now the root of the pseudo-fs.
LOOKUP "this" --> NFS_OK.  The current fh is for /this and is within the pseudo-fs.
LOOKUP "is" --> NFS_OK.  The current fh is for /this/is and is within the pseudo-fs.
LOOKUP "the" --> NFS_OK.  The current fh is for /this/is/the and is within the pseudo-fs.
LOOKUP "path" --> NFS_OK.  The current fh is for /this/is/the/path and is within a new, absent file system, but ... the client will never see the value of that fh.
GETFH --> NFS4ERR_MOVED.  Fails because current fh is in an absent file system at the start of the operation, and the specification makes no exception for GETFH.
GETATTR (fsid, fileid, size, time_modify).  Not executed because the failure of the GETFH stops processing of the COMPOUND.

So the GETFH fails, and now the client has the job 😭 of figuring out where the root of the absent file system is and where to find it, meaning the server and the path relative to that server's root fh. Lowkey tricky, because the client didn't grab filehandles and attributes like fsid for the intermediate directories, so it can't be sure where the absent file system actually starts. It could be that /this/is/the is the root of the moved file system, and the LOOKUP "path" worked only because the file system wasn't absent yet on that operation and got moved between the last LOOKUP and the GETFH. Remember, COMPOUND is not atomic 🙈. Even with fsids for all the intermediate directories, there'd be no way to know /this/is/the/path was the root of a new file system, since we don't have its fsid yet.

To fix that, the client re-sends the chain of LOOKUPs with GETFHs and GETATTRs, at least to get the fsids, so it knows exactly where the file system boundaries are 🔥. It could also grab fs_locations_info at the same time, but most of the time the client will already have a good guess about the boundaries (from where NFS4ERR_MOVED showed up and where it didn't), so fetching fs_locations_info isn't needed.

• OP01: PUTROOTFH --> NFS_OK: the current fh is the root of the pseudo-fs.
• OP02: GETATTR(fsid) --> NFS_OK: just for completeness. Normally, clients will know the fsid of the pseudo-fs as soon as they establish communication with a server.
• OP03: LOOKUP "this" --> NFS_OK
• OP04: GETATTR(fsid) --> NFS_OK: get the current fsid to see where the file system boundaries are. The fsid will be the one for the pseudo-fs in this example, so there's no boundary.
• OP05: GETFH --> NFS_OK: the current fh is for /this and is within the pseudo-fs.
• OP06: LOOKUP "is" --> NFS_OK: the current fh is for /this/is and is within the pseudo-fs.
• OP07: GETATTR(fsid) --> NFS_OK: get the current fsid to see where the file system boundaries are. The fsid will be the one for the pseudo-fs in this example, so there's no boundary.
• OP08: GETFH --> NFS_OK: the current fh is for /this/is and is within the pseudo-fs.
• OP09: LOOKUP "the" --> NFS_OK: the current fh is for /this/is/the and is within the pseudo-fs.
• OP10: GETATTR(fsid) --> NFS_OK: get the current fsid to see where the file system boundaries are. The fsid will be the one for the pseudo-fs in this example, so there's no boundary.
• OP11: GETFH --> NFS_OK: the current fh is for /this/is/the and is within the pseudo-fs.
• OP12: LOOKUP "path" --> NFS_OK: the current fh is for /this/is/the/path and is within a new, absent file system, but ...
  • The client will never see the value of that fh.
• OP13: GETATTR(fsid, fs_locations_info) --> NFS_OK:
  • We're getting the fsid to find out where the file system boundaries are. In this operation, the fsid is different from that of the parent directory (which we got back in OP10). Note that the fsid we're given won't necessarily be kept at the new location. That fsid might be different, and in fact the fsid we have for this file system might be a valid fsid of a different file system on that new server.
  • In this particular case, we're pretty sure that what moved is /this/is/the/path rather than /this/is/the, since we have the fsid of the latter and it matches the pseudo-fs, which presumably can't move. However, in other examples we might not have that kind of info to lean on (for instance, /this/is/the might be a non-pseudo file system separate from /this/is/the/path). So we need some other reliable source of info about the boundary of the file system that moved. If, say, the file system /this/is had moved, we'd be looking at a migration rather than a referral, and once the boundaries of the migrated file system were clear, we could fetch fs_locations_info.
  • We're fetching fs_locations_info because getting NFS4ERR_MOVED at this point means it's most likely a referral and we need the destination. Even if /this/is/the turned out to be a file system that migrated, we'd still need the location info for that file system.
• OP14: GETFH --> NFS4ERR_MOVED: fails because the current fh is in an absent file system at the start of the operation, and the specification makes no exception for GETFH. Note that this means the server will never send the client a filehandle from within an absent file system.

So, bestie, the client now knows the root of the absent file system is /this/is/the/path, because that's where the fsid changed (between "the" and "path") 🌈. The fs_locations_info attribute also gives the client the actual location of the absent file system, so the referral can keep going 🦋. The server only hands over the bare minimum about the absent file system, so there's very little room for conflicts between what the referring server says and what the file system's home says 💜. No filehandles and very few attributes come from the referring server, and the client should treat whatever it does get as temporary info whose only job is to make the referral happen ✨.

---

11.8.2. Referral Example (READDIR) 🍓

Another place a client can run into referrals is when it does a READDIR on a directory where some of the subdirectories are roots of absent file systems 👑.

Say that directory gets read like this:

PUTROOTFH
LOOKUP "this"
LOOKUP "is"
LOOKUP "the"
READDIR (fsid, size, time_modify, mounted_on_fileid)

In this case, rdattr_error isn't requested, fs_locations_info isn't requested, and some of the attributes can't be provided, so the READDIR comes back with an NFS4ERR_MOVED error. The detailed results look like this 🙈:

PUTROOTFH --> NFS_OK.  The current fh is at the root of the pseudo-fs.
LOOKUP "this" --> NFS_OK.  The current fh is for /this and is within the pseudo-fs.
LOOKUP "is" --> NFS_OK.  The current fh is for /this/is and is within the pseudo-fs.
LOOKUP "the" --> NFS_OK.  The current fh is for /this/is/the and is within the pseudo-fs.
READDIR (fsid, size, time_modify, mounted_on_fileid) --> NFS4ERR_MOVED.  Note that the same error would have been returned if /this/is/the had migrated, but it is returned because the directory contains the root of an absent file system.

Now let's re-send with rdattr_error 💖:

PUTROOTFH
LOOKUP "this"
LOOKUP "is"
LOOKUP "the"
READDIR (rdattr_error, fsid, size, time_modify, mounted_on_fileid)

The results are:

PUTROOTFH --> NFS_OK.  The current fh is at the root of the pseudo-fs.
LOOKUP "this" --> NFS_OK.  The current fh is for /this and is within the pseudo-fs.
LOOKUP "is" --> NFS_OK.  The current fh is for /this/is and is within the pseudo-fs.
LOOKUP "the" --> NFS_OK.  The current fh is for /this/is/the and is within the pseudo-fs.
READDIR (rdattr_error, fsid, size, time_modify, mounted_on_fileid) --> NFS_OK.  The attributes for directory entry with the component named "path" will only contain rdattr_error with the value NFS4ERR_MOVED, together with an fsid value and a value for mounted_on_fileid.

Next, we do another READDIR to get fs_locations_info (although we could've used a GETATTR directly, like in Section 11.8.1) 🧚:

PUTROOTFH
LOOKUP "this"
LOOKUP "is"
LOOKUP "the"
READDIR (rdattr_error, fs_locations_info, mounted_on_fileid, fsid, size, time_modify)

The results would be:

PUTROOTFH --> NFS_OK.  The current fh is at the root of the pseudo-fs.
LOOKUP "this" --> NFS_OK.  The current fh is for /this and is within the pseudo-fs.
LOOKUP "is" --> NFS_OK.  The current fh is for /this/is and is within the pseudo-fs.
LOOKUP "the" --> NFS_OK.  The current fh is for /this/is/the and is within the pseudo-fs.
READDIR (rdattr_error, fs_locations_info, mounted_on_fileid, fsid, size, time_modify) --> NFS_OK.  The attributes will be as shown below.

For the directory entry with the component named "path", you'll only get:

• rdattr_error (value: NFS_OK)
• fs_locations_info
• mounted_on_fileid (value: unique fileid within referring file system)
• fsid (value: unique value within referring server)

Entry "path" won't include size or time_modify, because those attributes aren't available within an absent file system 🦋💅.

11.9. The Attribute fs_locations 🦋

okay so the fs_locations attribute has a very specific shape, bestie, and it's giving structure 💅

struct fs_location4 {
        utf8str_cis     server<>;
        pathname4       rootpath;
};


struct fs_locations4 {
        pathname4       fs_root;
        fs_location4    locations<>;
};

the fs_location4 data type is how we represent where a file system lives 📍 you give it a server name plus the path to the root of that file system inside that server's namespace. when a bunch of servers have matching file systems at the same path in their namespaces, you can pass an array of server names 🌸 each entry is a UTF-8 string and it's one of three things: a traditional DNS host name, an IPv4 address, an IPv6 address, or a zero-length string 🧚 an IPv4 or IPv6 address shows up as a universal address (see Section 3.3.9 and [15]), minus the netid, and it can have the trailing ".p1.p2" suffix that carries the port number or leave it off entirely. if the suffix is missing, the default port, 2049, SHOULD be assumed ✨ and a zero-length string SHOULD be used to mean "the current address being used for the RPC call" 🫶 it's not a requirement that every server sharing the same rootpath gets listed in one fs_location4 instance. the server array is just there for convenience 💖 servers sharing a rootpath can also get their own separate fs_location4 entries in the fs_locations attribute.

the fs_locations4 data type and the fs_locations attribute hold an array of those locations 🎀 and because every server can build its namespace differently, there's a fs_root field too 👑 the path in fs_root is where the file system sits in the current server's namespace, meaning the server the fs_locations attribute came from. the fs_root path exists to help the client clearly spot the root of the file system whose locations are being reported, no matter which object inside that file system the current filehandle points at 🦋 basically, fs_root is just the pathname the client used to reach the object on the current server, which is the object the fs_locations attribute applies to. iconic, honestly.

when someone asks for fs_locations and there are no alternate file system locations, the server SHOULD return a zero-length array of fs_location4 structures, together with a valid fs_root 🍓

okay, example time 🧚 say there's a replicated file system sitting on two servers, servA and servB. on servA, the file system lives at /a/b/c. on servB, it lives at /x/y/z. if the client asks for the fs_locations value for the directory at /a/b/c/d, it might not know that the file system's root is at /a/b/c in servA's namespace 😭 when the client switches over to servB, it needs to figure out that the directory it first referenced on servA is now at /x/y/z/d on servB. to make that work, the fs_locations attribute from servA would have an fs_root value of /a/b/c and two entries in fs_locations. one entry is for itself (servA) and the other is for servB with a path of /x/y/z. with that info, the client can swap /x/y/z in for the /a/b/c at the front of its access path and build /x/y/z/d to use on the new server 💅

note: there's no rule that the number of components in each rootpath has to match 🙈 there's no relationship between the number of components in rootpath or fs_root, and none of the components in a rootpath or fs_root have to match each other. in that same example, we could add a third element to the locations array with server set to "servC" and rootpath set to "/I/II", and a fourth element with server set to "servD" and rootpath set to "/aleph/beth/gimel/daleth/he" 🌈

the relationship between fs_root and a rootpath is simple: the client swaps out the pathname given in fs_root (for the current server) with the substitute given in rootpath (for the new server) 🦋

now for a referred or migrated file system 💀 suppose there's a file system at serv1, living at /az/buky/vedi/glagoli. the client finds that the object at glagoli has migrated (or is a referral). the client grabs the fs_locations attribute, which has an fs_root of /az/buky/vedi/glagoli and one element in the locations array with server set to serv2 and rootpath set to /izhitsa/fita. the client replaces /az/buky/vedi/glagoli with /izhitsa/fita and uses that new pathname on serv2 ✨

so, the server MUST return an fs_root that equals the path the client used to reach the object the fs_locations attribute applies to 🔥 otherwise, the client can't figure out the new path to use on the new server, and that's a whole mess 😭

since the fs_locations attribute doesn't include info defining the various attributes of each file system choice, it SHOULD only be interrogated and used when fs_locations_info is not available 🌸 when fs_locations does get used, info about the specific locations should be assumed based on the following rules.

the following rules are general and apply no matter the context 💜

• all listed file system instances should be considered as of the same handle class, if and only if, the current fh_expire_type attribute does not include the FH4_VOL_MIGRATION bit. note that for referrals, filehandle issues don't apply, since there can't be any filehandles known within the current file system, and there's no access to the fh_expire_type attribute on the referring (absent) file system either.
• all listed file system instances should be considered as of the same fileid class if and only if the fh_expire_type attribute indicates persistent filehandles and does not include the FH4_VOL_MIGRATION bit. note that for referrals, fileid issues don't apply, since there can't be any fileids known within the referring (absent) file system, and there's no access to the fh_expire_type attribute.
• all file system instances on different servers should be considered as of different change classes.

for all the other class assignments, how you handle a file system transition depends on why the transition happened 🫶

• when the transition is due to migration, meaning the client was pointed to a new file system after getting an NFS4ERR_MOVED error, the target should be treated as being of the same write-verifier class as the source 🎀
• when the transition is due to failover to another replica, meaning the client picked another replica without getting an NFS4ERR_MOVED error, the target should be treated as being of a different write-verifier class from the source 🌈

those specific choices line up with typical implementation patterns for failover and controlled migration, respectively 🧚 since other choices are possible and useful too, this info is better obtained by using fs_locations_info. when a server implementation needs to communicate other choices, it MUST support the fs_locations_info attribute 👑

and for the security flavor that should be used by any GETATTR operation requesting the "fs_locations" attribute, check out Section 21 🔐✨

---

11.10. The Attribute fs_locations_info 🦋

Okay bestie, so fs_locations_info is basically the glow-up of fs_locations 💅 It's meant to be a more functional replacement, but fs_locations is NOT going anywhere. It will keep existing and being supported 🫶 Clients use this one to get a much more complete picture of the alternative file system locations 🗺️ If the server doesn't support fs_locations_info, the client can fall back on fs_locations to get a subset of the info. And a server that supports fs_locations_info MUST support fs_locations as well. No exceptions, periodt 🔥

There's extra stuff in fs_locations_info that fs_locations just doesn't have 👀

• Attribute continuity information. This is the tea ☕ It lets a client pick a location that meets the transparency requirements of the apps accessing the data, and it unlocks optimizations because the server guarantees attribute continuity. For example, if the change attribute of a file in the file system is continuous across multiple server locations, the client doesn't have to invalidate the file's cache when the change attribute is the same among all locations. Iconic efficiency 😭
• File system identity information. This shows when multiple replicas, from the client's point of view, correspond to the same target file system. That means they can be used interchangeably, without disruption, as multiple paths to the same thing 🦋
• Suitability information. This bears on how good each replica is, depending on what the client is planning to do. Lots of apps need an absolutely up-to-date copy (the ones that write, for example), while others only need access to the most up-to-date copy reasonably available 💜
• Server-derived preference information. The server can rank replicas so load-balancing can happen, while still handing the client the entire file system list to use in case the primary fails 🧚

The fs_locations_info attribute is structured similarly to fs_locations. Picture it like nesting dolls 🪆 The top-level structure, fs_locations_info4, holds the entire attribute, including the root pathname of the file system and an array of lower-level structures. Those lower-level structures define replicas that share a common rootpath on their respective servers. Next down, fs_locations_item4 holds a specific pathname and info on one or more individual server replicas. At that lowest level, fs_locations_info has an fs_locations_server4 structure. It has per-server-replica info plus the server name. That per-replica info includes a nominally opaque array called fls_info, where specific pieces of info live at specific indices (listed below) 🎀

The attribute will always contain at least a single fs_locations_server entry. Usually it's an entry with the FS4LIGF_CUR_REQ flag set. The exception is a referral, where there won't be any entry with that flag set 🌸

Heads up, though 👀 fs_locations_info attributes returned by servers for various replicas may be different, for a bunch of reasons. One server might know about replicas that other servers don't know about. Compatibility attributes might differ too. Filehandles might be of the same class going from replica A to replica B but not the other way around. That can happen because the filehandles are the same, but replica B's server implementation might not have a way to note and report that equivalence 🙈

The fs_locations_info attribute is made of a root pathname (fli_fs_root, just like fs_root in fs_locations), plus an array of fs_location_item4 structures. Those fs_location_item4 structures each contain a root pathname (fli_rootpath) and an array (fli_entries) of fs_locations_server4 elements. Here's the full setup 🧁

/*
 * Defines an individual server replica
 */
struct  fs_locations_server4 {
        int32_t         fls_currency;
        opaque          fls_info<>;
        utf8str_cis     fls_server;
};

/*
 * Byte indices of items within
 * fls_info: flag fields, class numbers,
 * bytes indicating ranks and orders.
 */
const FSLI4BX_GFLAGS            = 0;
const FSLI4BX_TFLAGS            = 1;
const FSLI4BX_CLSIMUL           = 2;
const FSLI4BX_CLHANDLE          = 3;
const FSLI4BX_CLFILEID          = 4;
const FSLI4BX_CLWRITEVER        = 5;
const FSLI4BX_CLCHANGE          = 6;
const FSLI4BX_CLREADDIR         = 7;

const FSLI4BX_READRANK          = 8;
const FSLI4BX_WRITERANK         = 9;
const FSLI4BX_READORDER         = 10;
const FSLI4BX_WRITEORDER        = 11;

/*
 * Bits defined within the general flag byte.
 */
const FSLI4GF_WRITABLE          = 0x01;
const FSLI4GF_CUR_REQ           = 0x02;
const FSLI4GF_ABSENT            = 0x04;
const FSLI4GF_GOING             = 0x08;
const FSLI4GF_SPLIT             = 0x10;
/*
 * Bits defined within the transport flag byte.
 */
const FSLI4TF_RDMA              = 0x01;

/*
 * Defines a set of replicas sharing
 * a common value of the rootpath
 * with in the corresponding
 * single-server namespaces.
 */
struct  fs_locations_item4 {
        fs_locations_server4    fli_entries<>;
        pathname4               fli_rootpath;
};

/*
 * Defines the overall structure of
 * the fs_locations_info attribute.
 */
struct  fs_locations_info4 {
        uint32_t                fli_flags;
        int32_t                 fli_valid_for;
        pathname4               fli_fs_root;
        fs_locations_item4      fli_items<>;
};

/*
 * Flag bits in fli_flags.
 */
const FSLI4IF_VAR_SUB           = 0x00000001;

typedef fs_locations_info4 fattr4_fs_locations_info;

As mentioned above, fs_locations_info (when it's supported) can be requested for absent file systems without causing NFS4ERR_MOVED to come back ✨ It's generally expected to be available for both present and absent file systems. That's true even if there's only one fs_locations_server4 entry (the current, present file system), or two entries (the previous location of an absent file system, the one just referenced, and its successor location). Servers are strongly urged to support this attribute on all file systems if they support it on any file system 💖

The data in fs_locations_info can come from the server in a bunch of ways. The administrator can specify it, or it can come through current protocols for transferring data among replicas, or through protocols not yet developed 🌈 NFSv4.1 only defines how this information gets presented by the server to the client.

11.10.1. The fs_locations_server4 Structure 🧩

The fs_locations_server4 structure has these items:

• fls_currency: how up-to-date the file system is, in seconds, relative to the master copy. A negative value means the server can't give any reasonably useful value here. A value of zero means the file system is the actual writable data or a reliably coherent and fully up-to-date copy. Positive values show how out-of-date this copy can normally be before it's considered for update. That isn't a guarantee those updates will always happen on the required schedule. It's more of a hint about how far behind the most up-to-date copy this data would be expected to run ⏰
• fls_info: a counted array of one-byte values with info about this particular file system instance. That includes general flags, transport capability flags, file system equivalence class info, and selection priority info. The encoding gets discussed below 🔢
• fls_server: the server string. For the replica currently being accessed (via GETATTR), a zero-length string MAY be used to indicate the current address being used for the RPC call 📡 The fls_server field can also be an IPv4 or IPv6 address, formatted the same way as an IPv4 or IPv6 address in the "server" field of the fs_location4 data type (see Section 11.9).

Data in the fls_info array is made of 8-bit data items. Constants give the offsets within the array for the various values describing this file system instance. This style got chosen over explicit XDR structure definitions for a few reasons 🎀

• The kinds of data in fls_info (flags, file system classes, and priorities among sets of file systems representing the same data) work just fine with 8 bits. Even where there might be more than 256 file system instances, having more than 256 distinct classes or priorities is unlikely.
• Explicit XDR definitions would limit expandability. Any extension within a later minor version would need yet another attribute, which leads to clumsy specs and clumsy implementations 🙃
• Explicit definitions would also make it impossible to propose Standards Track extensions apart from a full minor version.

This encoding scheme can be adapted to multi-byte numeric values, even though none are defined yet. If extensions are made via Standards Track RFCs, multi-byte quantities will be encoded as a range of bytes with a range of indices. The byte is interpreted in big-endian byte order. Any such index assignments are also constrained so the relevant quantities won't cross XDR word boundaries 🧠

The set of fls_info data can expand in a future minor version, or in a Standards Track RFC within the context of a single minor version. The server SHOULD NOT send, and the client MUST NOT use, indices within the fls_info array that aren't defined in Standards Track RFCs 🚫

The fls_info array contains:

• Two 8-bit flag fields. One is for general file-system characteristics, and the second is reserved for transport-related capabilities.
• Six 8-bit class values that define various file system equivalence classes (more on that below).
• Four 8-bit priority values that govern file system selection (also more below) 💅

The general file system characteristics flag (at byte index FSLI4BX_GFLAGS) has these bits defined within it:

• FSLI4GF_WRITABLE indicates that this file system target is writable, so it can be selected by clients that may need to write on this file system. When the current file system instance is writable and it's defined as being in the same simultaneous use class (as specified by the value at index FSLI4BX_CLSIMUL) that the client was previously writing to, then it must incorporate within its data any committed write made on the source file system instance. See Section 11.7.8, which discusses the write-verifier class. There's no harm in not setting this flag for a file system that turns out to be writable. But turning the flag on for a read-only file system can cause problems for clients that select a migration or replication target based on the flag and then find themselves unable to write 😭
• FSLI4GF_CUR_REQ 🎯 means this replica is the one the request is being made on. Only a single server entry may have this flag set, and in the case of a referral, no entry will have it.
• FSLI4GF_ABSENT 👻 means this entry is for an absent file system replica. It can only be set if FSLI4GF_CUR_REQ is set too. When both bits are set, the file system instance isn't usable, but the info in the entry can still tell you what kinds of continuity you get when switching from this replica to other possible replicas. Since this bit can only be true if FSLI4GF_CUR_REQ is true, you could work it out from the fs_status attribute, but it's included here so clients don't have to go hunting. An entry with this bit, since it represents a true file system (albeit absent), doesn't show up in a referral. It only appears when a file system has been accessed at this location and then migrated. 🧳
• FSLI4GF_GOING 🚪 means a replica is still available but shouldn't be used anymore. A client that's using it should make an orderly transfer to another file system instance as expeditiously as possible. File systems going out of service are expected to be announced as FSLI4GF_GOING some time before the actual loss of service. The fli_valid_for value is also expected to be small enough to let clients detect and act on scheduled events, while still large enough that fetching the fs_locations_info values doesn't get expensive. Values on the order of ten minutes seem reasonable. ⏰

When this flag is seen as part of a transition into a new file system, a client might choose to transfer immediately to another replica, or it may reference the current file system and only transition when a migration event occurs. Similarly, when this flag appears on a replica in the referral, clients would likely avoid being referred to this instance whenever there is another choice. 🙈

• FSLI4GF_SPLIT ✂️ means that when a transition occurs from the current file system instance to this one, the replacement may consist of multiple file systems. In this case, the client has to be prepared for objects that were on the same file system before migration to end up on different ones afterward. Note that FSLI4GF_SPLIT is not incompatible with the file systems belonging to the same fileid class, since if one has a set of fileids that are unique within a file system, each subset assigned to a smaller file system after migration won't have any conflicts inside that file system. 🧩

In the case of a split file system, a client will interrogate existing files it still has a continuing connection to (it's free to simply forget cached filehandles). If the client remembers the directory filehandle associated with each open file, it may proceed upward using LOOKUPP to find the new file system boundaries. Note that in the event of a referral, there won't be any such files, so these actions won't be performed. Instead, a reference to a portion of the original file system that has now split off into other file systems will encounter an fsid change and possibly a further referral. 🔍

Once the client recognizes that one file system has been split into two, it can prevent the disruption of running applications by presenting the two file systems as a single one until a convenient point to recognize the transition, such as a restart. This would require a mapping from the server's fsids to fsids as seen by the client, but that's already necessary for other reasons. As noted above, existing fileids within the two descendant file systems will not conflict. Providing non-conflicting fileids for newly created files on the split file systems is the responsibility of the server (or servers working in concert). The server can encode filehandles such that filehandles generated before the split event can be told apart from those generated after, which allows the server to determine when the need for emulating two file systems as one is over. 💅

Although it is possible for this flag to be present in the event of a referral, it would generally be of little interest to the client, since the client is not expected to have information regarding the current contents of the absent file system. 🤷‍♀️

The transport-flag field (at byte index FSLI4BX_TFLAGS) contains the following bits related to the transport capabilities of the specific file system. 🚚

• FSLI4TF_RDMA 🚀 indicates that this file system provides NFSv4.1 file system access using an RDMA-capable transport.

Attribute continuity and file system identity information are expressed by defining equivalence relations on the sets of file systems presented to the client. Each such relation is expressed as a set of file system equivalence classes. For each relation, a file system has an 8-bit class number. Two file systems belong to the same class if both have identical non-zero class numbers. Zero is treated as non-matching. Most often, the relevant question for the client will be whether a given replica is identical to or continuous with the current one in a given respect, but the information should also be available as to whether two other replicas match in that respect as well. 🪞

The following fields specify the file system's class numbers for the equivalence relations used in determining the nature of file system transitions. See Section 11.7 and its various subsections for details about how this information is to be used. Servers may assign these values as they wish, so long as file system instances that share the same value have the specified relationship to one another. Conversely, file systems that have the specified relationship to one another share a common class value. As each instance entry is added, the relationships of this instance to previously entered instances can be consulted, and if one is found that bears the specified relationship, that entry's class value can be copied to the new entry. When no such previous entry exists, a new value for that byte index (not previously used) can be selected, most likely by incrementing the value of the last class value assigned for that index. 🧁

• The field with byte index FSLI4BX_CLSIMUL defines the simultaneous-use class for the file system. 👯
• The field with byte index FSLI4BX_CLHANDLE defines the handle class for the file system. 🏷️
• The field with byte index FSLI4BX_CLFILEID defines the fileid class for the file system. 🆔
• The field with byte index FSLI4BX_CLWRITEVER defines the write-verifier class for the file system. ✍️
• The field with byte index FSLI4BX_CLCHANGE defines the change class for the file system. 🔄
• The field with byte index FSLI4BX_CLREADDIR defines the readdir class for the file system. 📖

Server-specified preference information is also provided via 8-bit values within the fls_info array. The values provide a rank and an order (see below) to be used with separate values specifiable for the cases of read-only and writable file systems. These values are compared for different file systems to establish the server-specified preference, with lower values indicating "more preferred". 👑

Rank is used to express a strict server-imposed ordering on clients, with lower values indicating "more preferred". Clients should attempt to use all replicas with a given rank before they use one with a higher rank. Only if all of those file systems are unavailable should the client proceed to those of a higher rank. Because specifying a rank will override client preferences, servers should be conservative about using this mechanism, particularly when the environment is one in which client communication characteristics are neither tightly controlled nor visible to the server. 🚦

Within a rank, the order value is used to specify the server's preference to guide the client's selection when the client's own preferences are not controlling, with lower values of order indicating "more preferred". If replicas are approximately equal in all respects, clients should defer to the order specified by the server. When clients look at server latency as part of their selection, they are free to use this criterion, but it is suggested that when latency differences are not significant, the server-specified order should guide selection. 📶

• The field at byte index FSLI4BX_READRANK gives the rank value to be used for read-only access. 📕
• The field at byte index FSLI4BX_READORDER gives the order value to be used for read-only access. 📗
• The field at byte index FSLI4BX_WRITERANK gives the rank value to be used for writable access. 📝
• The field at byte index FSLI4BX_WRITEORDER gives the order value to be used for writable access. 🖊️

Depending on the potential need for write access by a given client, one of the pairs of rank and order values is used. The read rank and order should only be used if the client knows that only reading will ever be done, or if it is prepared to switch to a different replica in the event that any write access capability is required in the future. 🌷

11.10.2. The fs_locations_info4 Structure ✨

okay so the fs_locations_info4 structure is what encodes the fs_locations_info attribute, and it's got a few fields that are lowkey the whole vibe 💅

• The fli_flags field holds general flags that change how you read this fs_locations_info4 structure and every fs_locations_item4 structure inside it. Right now the only flag that's defined is FSLI4IF_VAR_SUB. Any bits in fli_flags that aren't defined should always be returned as zero 🫶
• The fli_fs_root field holds the pathname of the root of the current file system on the current server, same as it does in the fs_locations4 structure 🌸
• There's an array called fli_items made of fs_locations4_item structures, and they hold info about replicas of the current file system. If the current file system is actually there (or used to be), so it's not a referral situation, one of the fs_locations_item4 structures will include an fs_locations_server4 for the current server. That structure will have FSLI4GF_ABSENT set if the current file system is absent, which means normal access to it gets you NFS4ERR_MOVED 😭
• The fli_valid_for field is a number of seconds that tells you how long it's reasonable to keep using the fs_locations_info attribute without refetching it 🦋. But it is NOT a promise that it's still valid, because servers can go down or become unreachable for all kinds of random reasons. Clients are well-advised to refetch this info for an actively used file system every fli_valid_for seconds. This matters a lot when replicas are being taken down on purpose, since the FSLI4GF_GOING flag is how a server says "hey, something's changing." The server should set fli_valid_for to a value that gives well-behaved clients time to spot FSLI4GF_GOING and switch over calmly before the service actually drops 🌈. If the value is zero, there's no refetch interval that makes sense, and the client doesn't need to refetch on any schedule at all 🧚. When you transition to a new file system instance, a fresh fs_locations_info value gets fetched from the destination. That one might have a different fli_valid_for value, and the client should just use it the same way it used the previous one ✨

The FSLI4IF_VAR_SUB flag inside fli_flags decides whether variable substitution is turned on. Check out Section 11.10.3 for the explanation of variable substitution 🎀

11.10.3. The fs_locations_item4 Structure ✨

The fs_locations_item4 structure has a pathname in the field fli_rootpath, and that pathname tells you where the target file system replicas live on the group of servers listed by the included fs_locations_server4 entries 🍓. How that target location gets written depends on whether the FSLI4IF_VAR_SUB flag is set in the associated fs_locations_info4 structure 👑.

If that flag isn't set, fli_rootpath just points to where the target file system sits inside each server's single-server namespace, same as the rootpath in the fs_location4 structure does 💅. But when the bit is set, certain components get client-specific variable substitution. That lets the namespace be a little non-uniform, so different clients can be sent to different file system targets that fit their architectures or other traits 🦋.

When substitution is happening, any variable that starts with "${" and ends with "}" and contains a colon gets swapped out for the client-specific value tied to that variable. If the client doesn't have a value for a variable, it should use the string "unknown" 🙈. The resulting pathname is what designates the target file system, so different clients can end up with different file systems that all match that same spot in the multi-server namespace 🌸.

As we just said, substituted pathname variables have a colon in them. The part before the colon is supposed to be a DNS domain name, and the part after it is a case-insensitive alphanumeric string 🫶.

If the domain is ietf.org, only variable names defined in this document or in later Standards Track RFCs can be substituted. Organizations are free to use their own domain names to make their own sets of client-specific variables that can be substituted 🌈. When those variables are meant to be used by more than one organization, publishing an Informational RFC that defines them is RECOMMENDED ⭐.

The variable ${ietf.org:CPU_ARCH} says what CPU architecture the object files get compiled for. This spec doesn't limit the allowed values (they just have to be valid UTF-8 strings), but things like "x86", "x86_64", and "sparc" are what you'd expect, matching industry practice 🧚.

The variable ${ietf.org:OS_TYPE} says which operating system, and therefore which kernel and library APIs, the code might get compiled against. Again, no limit on the values (valid UTF-8 strings only), but "linux" and "freebsd" are the kind of thing you'd expect, in line with industry practice 💜.

The variable ${ietf.org:OS_VERSION} says which OS version, so the specific details of versioned interfaces the code might use. This spec doesn't limit the values either (valid UTF-8 strings only). Still, combos of numbers and letters with dots mixed in are what you'd expect, following industry practice. The exact version format depends on which ${ietf.org:OS_TYPE} value it's paired with 🎀.

Using these variables can send different clients to different file systems on the same server, depending on which clients they are 🦋. If the target file systems live on different servers, one server could act as a referral point. Each valid combo of variable values would then point to a referral hosted on that one server, and the targets of those referrals would be spread across a bunch of other servers 🌸.

Since namespace admin gets affected by which values get picked for each variable, clients should give people convenient ways to find out which variable substitutions the client will actually do. Where it makes sense, clients should also let people control which substitutions get used. How exactly that's done is outside the scope of this spec 🙈.

Variable substitution fits best with referrals, but it can also be used with replication and migration 🍓. If you do use it in those contexts, the server must make sure that no matter what values the client sends for the substituted variables, the result is always a valid successor file system instance to the one the transition is coming from. In other words, the data has to be identical or be a later image of a writable file system ✨.

One more thing: when fli_rootpath is a null pathname (one with zero components), the file system being designated is at the root of the specified server, and that's true whether or not the FSLI4IF_VAR_SUB flag is set in the associated fs_locations_info4 structure 💖.

11.11. The Attribute fs_status ✨

okay so picture this bestie 🌸: you've got a bunch of copies of the same basic set of data floating around, and knowing where each copy came from and how the copies are related is literally SO helpful for giving apps consistent data 💅

enum fs4_status_type {
        STATUS4_FIXED = 1,
        STATUS4_UPDATED = 2,
        STATUS4_VERSIONED = 3,
        STATUS4_WRITABLE = 4,
        STATUS4_REFERRAL = 5
};
struct fs4_status {
        bool            fss_absent;
        fs4_status_type fss_type;
        utf8str_cs      fss_source;
        utf8str_cs      fss_current;
        int32_t         fss_age;
        nfstime4        fss_version;
};

The boolean fss_absent tells you whether the file system is currently absent 🫣. This gets set if the file system was previously present and then became absent, or if it has never been present and the type is STATUS4_REFERRAL. When this boolean is set and the type is not STATUS4_REFERRAL, the rest of the fs4_status reflects the last valid state from when the file system was present.

The fss_type field says what kind of file system image you're looking at, and it's giving main character energy when you're working out the right succession of images using the version values ✨. When fss_absent is set and the file system was previously present, the fss_type value is whatever it was when the file was last present. There are five values:

• STATUS4_FIXED, which is a read-only image in the sense that it will never change 🧊. It's possible that, after a migration or a switch to a different image, changed data can be accessed, but inside this instance no change is allowed. The client can use this fact to cache aggressively 🙌.
• STATUS4_VERSIONED, which is updated externally, just like the STATUS4_UPDATED case 🔄. The difference is that it comes with a guarantee that the server will carefully update an associated version value. That way the client can protect itself from reading data from one version of the file system and then later reading data from an earlier version of the same file system 🛡️. We'll talk about how this works down below.
• STATUS4_UPDATED, which is an image the user can't write to, but it can still change externally 🕰️. This is usually because it's a periodically updated copy of some other writable file system somewhere else. In this case there's no version information, and the client isn't responsible for making sure the version only moves forward on a file system instance transition. Instead, it's the server's job to make sure the data presented after a transition is a proper successor image, and that it includes all changes the client has already seen plus any change made before all of those.
• STATUS4_WRITABLE, which means the file system is an actual writable one ✍️. The client doesn't have to write to it, of course, but once it does, it should not accept a transition to anything other than a writable instance of that same file system.
• STATUS4_REFERRAL, which means the file system in question is absent and has never been present on this server 👻.

⚠️ bestie, this part is important: in the STATUS4_UPDATED and STATUS4_VERSIONED cases, the server is responsible for handling locks that clash with external changes to delegations. If a server hands out delegations, they SHOULD be recalled before an inconsistent change is made to the data, and they MUST be revoked if that isn't possible 🚨. Same vibe for OPENs: if an OPEN is inconsistent with data that's changing (the OPEN has OPEN4_SHARE_DENY_WRITE/OPEN4_SHARE_DENY_BOTH and the data is changed), that OPEN SHOULD be considered administratively revoked 💀.

The opaque strings fss_source and fss_current are how you present info about where the current file system image came from 🗂️. The client isn't supposed to do anything with this info except pass it along to administrative tools 🛠️. It's meant to help when you're researching possible problems with a file system image, especially when it's unclear whether the correct image is being accessed and, if it isn't, how that image got made 🔍. This diagnostic info is super helpful because, as seems likely, copies of file systems get made in lots of different ways (simple user-level copies, file-system-level point-in-time copies, clones of the underlying storage) under all kinds of administrative setups 🌈. In those environments, knowing how a data set was constructed can really help solve problems.

The opaque string fss_source is used to show where a file system came from, with the hope that tools which create a file system image will pass this info along when they can 🎀. It's understood that this won't always be possible, because a user-level copy can be seen as creating a brand new data set, and the tools used might have no way to carry the info over 🤷‍♀️. When a file system is first created, it's really good to attach info about how it was created, where, who made it, and so on 📝. Putting this in a human-readable string helps applications and sysadmins, and it also makes the info available whenever the original file system is used to make later copies 🫶.

The opaque string fss_current should give whatever info is available about where the current copy came from 🧾. That includes the tool that made it, any relevant parameters for that tool, the time the copy was done, the user who made the change, the server where the change was made, and so on. All of this info should be in a human-readable string 🍓.

The field fss_age shows how out-of-date the file system currently is compared with its ultimate data source (in case of cascading data updates) ⏳. It works together with the fls_currency field of fs_locations_server4 (see Section 11.10) like this: fls_currency gives a bound on how out of date the data in a file system might typically get, while fss_age gives a bound on how out of date that data actually is 📊. A negative value means no info is available. Zero means the data is known to be current 💯. A positive value means the data is known to be no older than that many seconds compared with the ultimate data source. Using this value, the client may be able to decide a copy is too old and go look for a newer version to use 🏃‍♀️.

The field fss_version gives a version identification in the form of a time value, so successive versions always have later time values ⏰. When the fs_type is anything other than STATUS4_VERSIONED, the server may provide such a value, but there's no guarantee it's valid, and clients won't use it except to add extra info to fss_source and fss_current 🤏.

When fss_type is STATUS4_VERSIONED, servers SHOULD provide a value of fss_version that progresses monotonically whenever any new version of the data is established 📈. This lets the client, if reliable image progression matters to it, fetch this attribute as part of each COMPOUND where data or metadata from the file system is used 💖.

When it matters to the client that only valid successor images get accepted, it must make sure it doesn't read data or metadata from the file system without first updating its sense of the current state of the image 🧠. This avoids the risk that the fs_status the client holds belongs to an earlier image. That would make the client accept a new file system instance that's later than that but still earlier than the updated data the client read 😬.

To accept valid images reliably, the client must do a GETATTR of the fs_status attribute that follows any reading of data or metadata within the file system in question 🎯. It's often easiest to tack that GETATTR onto the end of all the other operations that reference a given file system 🧁. If errors happen between reading file system data and doing that GETATTR, be extra careful that the data in question isn't used before the proper fs_status value is obtained ⚠️. Related to this: if an OPEN is done within such a versioned file system and the associated GETATTR of fs_status doesn't complete successfully, the open file must not be accessed until that fs_status is fetched 🔒.

The procedure above makes sure that, before using any data from the file system, the client has a freshly fetched, current version of the file system image ✅. When there are multiple values for multiple requests in flight, you can sort them into the required partial order (the elements should form a total order within that partial order) and use the last one 🧩. Then, when switching among file system instances, the client may decline to use an instance that doesn't have an fss_type of STATUS4_VERSIONED or whose fss_version field is earlier than the last one obtained from the predecessor file system instance 🫡.

---

12. Parallel NFS (pNFS) 🦋

12.1. Introduction ✨

okay so pNFS is an OPTIONAL feature within NFSv4.1 💖 it lets clients go straight to the storage devices that hold the file data 🦋 when one NFSv4 server's file data is spread across several storage devices (especially faster ones than the server can handle alone), file access can get way faster ✨ The whole vibe with multiple clients, one server, and several storage devices (where the server and clients can all reach every storage device) looks like this:

     +-----------+
     |+-----------+                                 +-----------+
     ||+-----------+                                |           |
     |||           |        NFSv4.1 + pNFS          |           |
     +||  Clients  |<------------------------------>|   Server  |
      +|           |                                |           |
       +-----------+                                |           |
            |||                                     +-----------+
            |||                                           |
            |||                                           |
            ||| Storage        +-----------+              |
            ||| Protocol       |+-----------+             |
            ||+----------------||+-----------+  Control   |
            |+-----------------|||           |    Protocol|
            +------------------+||  Storage  |------------+
                                +|  Devices  |
                                 +-----------+

                               Figure 1

In this model the clients, the server, and the storage devices all share the job of managing file access 🌸 Compare that to NFSv4 without pNFS, where it's mostly the server's job, and only some of it can be handed to the client under strictly specified conditions 👑 Check out Section 12.2.5 for the Storage Protocol tea, and Section 12.2.6 for the Control Protocol 💅

pNFS shows up as OPTIONAL operations that manage protocol objects called 'layouts' (Section 12.2.7) 🧚 Each layout holds a byte-range and storage location info. Layouts get handled kind of like NFSv4.1 data delegations, so they're leased, recallable, and revocable 🔥 BUT they're totally their own thing and get manipulated with brand-new operations 🙈 When a client holds a layout, it gets the power to directly access that byte-range at the storage location the layout names ✨

Layouts also interact with other NFSv4.1 stuff, like data delegations and byte-range locking 🫶 Delegation drama is covered in Section 12.5.5, and byte-range locking drama is in Sections 12.2.9 and 12.5.1 💜

12.2. pNFS Definitions

NFSv4.1's pNFS feature gives parallel data access to a file system that stripes its content across multiple storage servers 🌈 The first version of pNFS, part of NFSv4.1, splits file system protocol processing into two parts: metadata processing and data processing ✨ Data is the contents of regular files that get striped across storage servers. Striping can happen at least two ways: file by file, and (inside big enough files) block by block 🍓 Striped access to metadata isn't offered to pNFS clients in NFSv4.1, even though the server's file system back end might stripe metadata 🙃 Metadata is everything else, including the contents of non-regular files (like directories); see Section 12.2.1 🎀 The metadata side is handled by an NFSv4.1 server that supports pNFS and the operations in Section 18, and that server is called a metadata server (Section 12.2.2) 👑

The data side is handled by one or more storage devices, and the client talks to each one through a storage protocol 💖 A subset of NFSv4.1 (defined in Section 13.6) is one such storage protocol. The pNFS feature also introduces new terms to the NFSv4.1 vocabulary and clarifies some existing ones so everything can be described properly 🦋

12.2.1. Metadata

Information about a file system object, like its name, location within the namespace, owner, ACL, and other attributes 💅 Metadata can also include storage location info, and that varies based on whatever underlying storage mechanism is being used 🌸

12.2.2. Metadata Server

An NFSv4.1 server that supports the pNFS feature ✨ There are lots of architectural choices for the metadata server and how it uses the file system info it holds. Some servers only hold metadata for file objects living on the metadata server itself, while the file data sits on associated storage devices 🧚 Other metadata servers hold both metadata and some amount of file data, varying from server to server 💜

12.2.3. pNFS Client

An NFSv4.1 client that supports pNFS operations and at least one storage protocol for doing I/O to storage devices 🔥

12.2.4. Storage Device

A storage device stores a regular file's data, but leaves the metadata management to the metadata server 🌈 A storage device could be another NFSv4.1 server, an object-based storage device (OSD), a block device accessed over a System Area Network (SAN, e.g., either FiberChannel or iSCSI SAN), or some other entity 🎀

12.2.5. Storage Protocol

As shown in Figure 1, the storage protocol is how the client stores and retrieves data directly from the storage devices 🦋

The NFSv4.1 pNFS feature was built so a whole variety of storage protocols can be defined and used 💖 One example is NFSv4.1 itself (documented in Section 13) 🫶 Other options get described elsewhere, and they include:

• Block/volume protocols such as Internet SCSI (iSCSI) [48] and FCP [49]. The block/volume protocol support can be independent of the addressing structure of the block/volume protocol used, which means more than one protocol can access the same file data, and it leaves room for extensibility to other block/volume protocols. See [41] for a layout specification that lets pNFS use block/volume storage protocols 💜
• Object protocols such as OSD over iSCSI or Fibre Channel [50]. See [40] for a layout specification that lets pNFS use object storage protocols ✨

It's possible that different storage protocols are available to both the client and the server, but it's also possible that a client and server don't have a matching storage protocol available to them 🙈 Because of that, the pNFS server MUST support normal NFSv4.1 access to any file accessible by the pNFS feature 🫶 This keeps the interoperability between an NFSv4.1 client and server working, no matter what 💅

12.2.6. Control Protocol

As shown in Figure 1, the control protocol is used by the exported file system between the metadata server and the storage devices 🧚 The specification of these protocols is outside the scope of the NFSv4.1 protocol 🌸 Control protocols would be used for things like allocating and deallocating storage, managing the state the storage devices need to do client access control, and (depending on the storage protocol) enforcing authentication and authorization, so that restrictions the metadata server would enforce also get enforced by the storage device 👑

A particular control protocol is not REQUIRED by NFSv4.1, but there are requirements placed on the control protocol for maintaining attributes like modify time, the change attribute, and the end-of-file (EOF) position 🔥 Also, if pNFS is layered over a clustered, parallel file system (e.g., PVFS [51]), the mechanisms that enable clustering and parallelism in that file system can be considered the control protocol 🌈

12.2.7. Layout Types

A layout describes how a file's data maps onto the storage devices that hold it 🎀 A layout belongs to a specific layout type (data type layouttype4, see Section 3.3.13) 🦋 The layout type lets variants handle different storage protocols, like the ones tied to block/volume [41], object [40], and file (Section 13) layout types 💖 A metadata server, along with its control protocol, MUST support at least one layout type 💜 A private sub-range of the layout type namespace is also defined, and values from that private range MAY be used for internal testing or experimentation (see Section 3.3.13) 🧚

As an example, the organization of the file layout type could be an array of tuples (e.g., device ID, filehandle), along with a definition of how the data gets stored across the devices (e.g., striping) ✨ A block/volume layout might be an array of tuples that store <device ID, block number, block count>, along with info about the block size and the file offset associated with the block number 🌸 An object layout might be an array of tuples <device ID, object ID> plus an extra structure (the aggregation map) that defines how the logical byte sequence of the file data gets serialized into the different objects 🍓 Just so you know, actual layouts are usually way more complex than these simple examples 🙈

Requests for pNFS-related operations will often specify a layout type 🫶 Examples of these operations are GETDEVICEINFO and LAYOUTGET 💅 The responses to these operations include structures like a device_addr4 or a layout4, and each of those includes a layout type inside it 👑 The layout type the server sends MUST always be the same one the client requested 🔥 If a server sends a response with a different layout type, the client SHOULD ignore that response and behave as if the server had returned an error response 💀

12.2.8. Layout

A layout defines how a file's data is organized on one or more storage devices 🌈 There are many potential layout types, and each one is differentiated by the storage protocol used to access data and by the aggregation scheme that lays out the file data on the underlying storage devices 🎀 A layout is precisely identified by the tuple <client ID, filehandle, layout type, iomode, range>, where filehandle refers to the filehandle of the file on the metadata server 🦋

It's important to define when layouts overlap and/or conflict with each other ✨ For two layouts with overlapping byte-ranges to actually overlap each other, both layouts must be of the same layout type, correspond to the same filehandle, and have the same iomode 💖 Layouts conflict when they overlap and differ in the content of the layout (i.e., the storage device/file mapping parameters differ) 🙈 Note that differing iomodes do not lead to conflicting layouts 🫶 It's permissible for layouts with different iomodes, pertaining to the same byte-range, to be held by the same client 💜 An example would be copy-on-write functionality for a block/volume layout type 🧚

12.2.9. Layout Iomode 🎀

okay so the layout iomode (data type layoutiomode4, see Section 3.3.20) is how the client tells the metadata server its plan: just READ stuff, or a mix of READ and WRITE 💅. For certain layout types it's super useful to say this up front when sending LAYOUTGET (Section 18.43). Like, for block/volume-based protocols, block allocation could happen when LAYOUTIOMODE4_RW is requested 🧚. There's also a special LAYOUTIOMODE4_ANY iomode, but it's only for LAYOUTRETURN and CB_LAYOUTRECALL, not LAYOUTGET 🙈. It basically means layouts for both LAYOUTIOMODE4_READ and LAYOUTIOMODE4_RW are being returned or recalled.

A storage device might check I/O against the iomode, and that really depends on the device implementation and layout type 🔥. So if the client's iomode doesn't match what it's actually doing, the storage device can reject the I/O with an error saying the client should get a new layout with the right iomode through LAYOUTGET. For example, if a client gets a layout with a LAYOUTIOMODE4_READ iomode and then tries a WRITE to a storage device, the device is allowed to reject that WRITE 💀.

Layout iomode doesn't clash with OPEN share modes or byte-range LOCK operations ✨. Open share mode and byte-range lock conflicts still get enforced the same way they would without pNFS, and they're logically separate from the pNFS layout level. Open share modes and byte-range locks are the preferred way to restrict user access to data files 👑. For instance, an OPEN with OPEN4_SHARE_ACCESS_WRITE doesn't conflict with a LAYOUTGET using LAYOUTIOMODE4_RW from another client. And if apps need to write into the same file at the same time, they can use byte-range locking to line up their accesses 🫶.

12.2.10. Device IDs 🦋

the device ID (data type deviceid4, see Section 3.3.14) identifies a group of storage devices 🌸. Its scope is the pair <client ID, layout type>. Storage devices can need a lot of info to be fully addressed, so instead of stuffing all of that into a layout, layouts just embed device IDs. To get the full address info (including all the device addresses for the device ID) for a storage device, you use the NFSv4.1 operation GETDEVICEINFO (Section 18.40), based on its layout type and device ID. For example, an NFSv4.1 data server or an object-based storage device could be addressed by an IP address and port, while a block storage device could just be a volume label 🍓.

Clients cannot expect the mapping between a device ID and its storage device address(es) to survive a metadata server restart 😭. Section 12.7.4 explains how recovery works in that situation.

A device ID lives as long as there's a layout referring to it. If nothing refers to it anymore, the server is free to delete it whenever it wants 🌈. But once the server deletes a device ID, it MUST NOT reuse that device ID for the same layout type and client ID again 🚨. This requirement is totally doable, since the device ID is 16 bytes long, which leaves plenty of room for a generation number if the server's implementation needs most of the rest of the device ID's content to get reused. It's needed because otherwise the race conditions between asynchronous notifications of device ID addition and deletion would be way too hard to untangle 🙈.

Device ID to device address mappings are not leased, and they can change at any time 💜. (Note that even though those mappings are likely to change after the metadata server restarts, the server is not required to change them.) A server has two choices for changing mappings: it can recall all layouts referring to the device ID, or it can use a notification mechanism 🧚.

NFSv4.1 has no optimal way to recall all layouts that referred to a particular device ID, unless the server ties a single device ID to a single fsid or a single client ID. In that case, CB_LAYOUTRECALL has options for recalling all layouts associated with the fsid, client ID pair, or just the client ID 🎀.

With a notification mechanism (see Section 20.12), device ID to device address mappings can change while the server runs, without recalling or revoking the layouts that refer to the device ID ✨.

The notification mechanism can also delete a device ID, but only if the client has no layouts referring to it. A notification of a change to a device ID to device address mapping will immediately or eventually invalidate some or all of that device ID's mappings 💅. The server MUST support notifications, and the client must request them before they can be used 🔥. For more on the notification types, see Section 20.12.

12.3. pNFS Operations 🌸

NFSv4.1 has several operations that pNFS servers need, no matter the layout type or storage protocol 🧚. All of these get sent to a metadata server, and they're summarized here. pNFS is an OPTIONAL feature, but if you implement it, some operations are REQUIRED to comply with pNFS 👑. See Section 17.

These are the fore channel pNFS operations:

• GETDEVICEINFO (Section 18.40), as noted previously (Section 12.2.10), returns the mapping of device ID to storage device address 🦋.
• GETDEVICELIST (Section 18.41) lets clients fetch all device IDs for a specific file system 🍓.
• LAYOUTGET (Section 18.43) is how a client gets a layout for a file 💜.
• LAYOUTCOMMIT (Section 18.42) tells the metadata server the client's intent to commit data that has been written to the storage device (the storage device as originally indicated in the return value of LAYOUTGET) 🫶.
• LAYOUTRETURN (Section 18.44) returns layouts for a file, a file system ID (FSID), or a client ID 🌈.

These are the backchannel pNFS operations:

• CB_LAYOUTRECALL (Section 20.3) recalls a layout, all layouts belonging to a file system, or all layouts belonging to a client ID 😭.
• CB_RECALL_ANY (Section 20.6) tells a client it needs to return some number of recallable objects, including layouts, to the metadata server 🚨.
• CB_RECALLABLE_OBJ_AVAIL (Section 20.7) tells a client that a recallable object it was denied (for pNFS, a layout denied by LAYOUTGET) due to resource exhaustion is now available 🎀.
• CB_NOTIFY_DEVICEID (Section 20.12) notifies the client of changes to device IDs ✨.

12.4. pNFS Attributes 💅

A bunch of attributes specific to pNFS are listed and described in Section 5.12 🫶.

12.5. Layout Semantics 🦋

12.5.1. Guarantees Provided by Layouts 👑

Layouts give the client the ability to access data located at a storage device using the appropriate storage protocol 🧚. The client is guaranteed that the layout will be recalled when one of two things happens: either a conflicting layout is requested, or the state encapsulated by the layout becomes invalid (which can happen when an event directly or indirectly modifies the layout) 💀. When a layout gets recalled and returned by the client, the client keeps the ability to access file data with normal NFSv4.1 operations through the metadata server. Only the ability to access the storage devices is affected 🌸.

The NFSv4.1 rule that all user access rights MUST be obtained through the appropriate OPEN, LOCK, and ACCESS operations doesn't change just because layouts exist 🔥. Layouts are handed to NFSv4.1 clients, and user access still follows the protocol's rules as if layouts weren't a thing 💅. For a client to access a storage device, it's a requirement that the client holds a layout. If a storage device gets an I/O request for a byte-range the client doesn't hold a layout for, the storage device SHOULD reject that I/O request 🙈. Note that modifying a file that has a layout held for it doesn't necessarily conflict with holding that layout describing the file being modified. So it's the storage protocol or layout type's requirement that decides the necessary behavior. For example, block/volume layout types require that the layout's iomode agree with the type of I/O being performed 🍓.

Depending on the layout type and storage protocol in use, storage device access permissions may be granted by LAYOUTGET, and they may be encoded within the type-specific layout 🧚. For an example of storage device access permissions, see an object-based protocol such as [50] 🦋. If access permissions are encoded within the layout, the metadata server SHOULD recall the layout when those permissions become invalid for any reason, like when a file becomes unwritable or inaccessible to a client ⭐. Heads up, clients are still required to perform the appropriate OPEN, LOCK, and ACCESS operations as described above 😭. How far a client could get around those operations, and what happens if it does, must be clearly specified by the individual layout type specifications. Those specs also have to be clear about the requirements and non-requirements for the checking the server performs 👑.

With pNFS in the mix, mandatory byte-range locks MUST behave as they would without pNFS 💜. So if mandatory file locks and layouts are provided at the same time, the storage device MUST be able to enforce the mandatory byte-range locks 🔥. For example, if one client gets a mandatory byte-range lock and a second client accesses the storage device, the storage device MUST appropriately restrict I/O for the range of that mandatory byte-range lock 🫶. If the storage device can't provide this check when mandatory byte-range locks are present, then the metadata server MUST NOT grant layouts and mandatory byte-range locks simultaneously 🚨.

12.5.2. Getting a Layout ✨

okay so getting a layout is done with the LAYOUTGET operation 💅 the metadata server will grant layouts of a particular type (e.g., block/volume, object, or file), and the client picks one that the server supports and that the client is actually prepared to use 🫶 the layout you get back might not exactly match the byte-range you asked for, as described in Section 18.43.3. if you need to, a client can send multiple LAYOUTGET operations, and these might end up as multiple overlapping, non-conflicting layouts (see Section 12.2.8) 🦋

before you can get a layout, you have to open the file first with the OPEN operation 🎀 when a client doesn't have a layout on a file yet, it MUST present an open stateid, a delegation stateid, or a byte-range lock stateid in the loga_stateid argument. a successful LAYOUTGET gives you a layout stateid back 💖 the first successful LAYOUTGET that the server processes using a non-layout stateid as an argument MUST have the "seqid" field of the layout stateid in the response set to one. after that, the client MUST use a layout stateid (see Section 12.5.3) on future LAYOUTGET calls on the file, and the "seqid" MUST NOT be set to zero 🌸 once you've got the layout, it can be held across multiple OPEN and CLOSE sequences, so a client can hold a layout for a file that isn't currently open by any user on the client. this is what allows layouts to be cached beyond CLOSE ✨ it's giving efficiency, honestly 😭

the storage protocol the client uses to reach the data on the storage device is decided by the layout's type 🔥 the client is responsible for matching that layout type with an available method to interpret and use the layout. how that layout type selection works is outside the scope of the pNFS functionality, so it's not our problem here 🙈

even though the metadata server is in charge of the layout for a file, the pNFS client can give the server hints when a file is opened or created about the preferred layout type and aggregation schemes 🧚 pNFS introduces a layout_hint attribute (Section 5.12.4) that the client can set at file creation time to give the server a hint for new files. setting this attribute separately, after the file has already been created, might make it difficult, or even impossible, for the server implementation to comply 💅

here's the tea: the EXCLUSIVE4 createmode4 doesn't allow attributes to be set at file creation time, so NFSv4.1 brings in the EXCLUSIVE4_1 createmode4, which does allow attributes to be set then 🌈 but if the session is created with persistent reply caches, EXCLUSIVE4_1 is neither necessary nor allowed. instead, GUARDED4 both works better and is prescribed 👑 Table 10 in Section 18.16.3 summarizes how a client is allowed to send an exclusive create, so go check that out 🍓

---

12.5.3. Layout Stateid 🦋

okay this one is a whole vibe 🌸 just like all the other stateids, the layout stateid has a "seqid" and an "other" field. once a layout stateid is established, the "other" field stays constant unless the stateid is revoked or the client returns all layouts on the file and the server gets rid of the stateid 💜 the "seqid" starts out set to one, and it is never zero on any NFSv4.1 operation that uses layout stateids, whether that's a fore channel or a backchannel operation ✨ after the layout stateid is established, the server increments the "seqid" by one in each later LAYOUTGET and LAYOUTRETURN response, and in each CB_LAYOUTRECALL request 🫶

here's why this one is different: pNFS is all about parallelism 🔥 so the client is expected to send LAYOUTGET and LAYOUTRETURN operations in parallel. the "seqid" is how the client sorts out responses to LAYOUTGET and LAYOUTRETURN 💅 it's also used to prevent race conditions between LAYOUTGET and CB_LAYOUTRECALL. since the processing rules for layout stateids are different from other stateid types, only the pNFS sections of this document should be considered when figuring out how to handle layout stateids 🎀

once the client gets a layout stateid, it MUST use the correct "seqid" for later LAYOUTGET or LAYOUTRETURN operations 💖 the correct "seqid" is defined as the highest "seqid" value from responses of fully processed LAYOUTGET or LAYOUTRETURN operations, or from arguments of a fully processed CB_LAYOUTRECALL operation. since the server bumps the "seqid" on every layout operation, the client can work out the order things were processed just by looking at the "seqid" 🧚 with overlapping layout ranges, that ordering tells the client which layout ranges it's holding. overlapping layout ranges can happen because of the client's specific requests, or because the server is allowed to expand the range of a requested layout and tell the client about it in the LAYOUTRETURN results ✨ extra layout stateid sequencing requirements are in Section 12.5.5.2 🌈

getting a "seqid" isn't enough to use it, though 🙈 the client has to fully process the operations first. for LAYOUTGET results, if the client is not using the forgetful model (Section 12.5.5.1), it MUST first update its record of which ranges of the file's layout it has, before using the "seqid". for LAYOUTRETURN results, the client MUST delete the range from its record of which ranges of the file's layout it had, before using the "seqid". for CB_LAYOUTRECALL arguments, the client MUST send a response to the recall before using the "seqid" 😭 the big rule here is that the "seqid" is what gives you the order of processing. LAYOUTGET results may be processed in parallel, and LAYOUTRETURN results may be processed in parallel too. LAYOUTGET and LAYOUTRETURN responses may be processed in parallel as long as their ranges don't overlap. CB_LAYOUTRECALL request processing MUST be processed in "seqid" order at all times 👑

once a client has no more layouts on a file, the layout stateid is no longer valid and MUST NOT be used 💀 any attempt to use a layout stateid like that will result in NFS4ERR_BAD_STATEID 🥺

---

12.5.4. Committing a Layout 🍓

so, allowing for the fact that storage protocols have different capabilities, pNFS does not require the metadata server and the storage devices to have a consistent view of file attributes and data location mappings 🌸 data location mapping is about things like which offsets store data as opposed to storing holes (see Section 13.4.4 for a discussion). related issues come up for storage protocols where a layout may hold provisionally allocated blocks, where that allocation doesn't survive a complete restart of both the client and the server 💀

because of this inconsistency, it's necessary to resynchronize the client with the metadata server and its storage devices, and to make any potential changes available to other clients ✨ that's done with the LAYOUTCOMMIT operation 💖

the LAYOUTCOMMIT operation is responsible for committing a modified layout to the metadata server 🫶 the data should be written and committed to the appropriate storage devices before the LAYOUTCOMMIT happens. the scope of LAYOUTCOMMIT depends on the storage protocol in use. it's important to note that the level of synchronization is from the point of view of the client that sent the LAYOUTCOMMIT 🔥 the updated state on the metadata server only needs to reflect the state as of the client's last operation before the LAYOUTCOMMIT. the metadata server is not REQUIRED to maintain a global view that accounts for other clients' I/O that may have happened in the same time frame 🙅‍♀️

for block/volume-based layouts, LAYOUTCOMMIT may need to update the block list that makes up the file, and commit this layout to stable storage. for file-based layouts, synchronizing attributes between the metadata and storage devices, mainly the size attribute, is required 🎀

the control protocol is free to synchronize the attributes before it receives a LAYOUTCOMMIT; however, upon successful completion of a LAYOUTCOMMIT, the state on the metadata server that describes the file MUST be synchronized with the state on the storage devices that make up that file, as of the client's last sent operation 😭 so, a client that queries the size of a file between a WRITE to a storage device and the LAYOUTCOMMIT might see a size that doesn't reflect the actual data written 🙈

the client MUST have a layout in order to send a LAYOUTCOMMIT operation 🔒

---

12.5.4.1. LAYOUTCOMMIT and change/time_modify 🧚

the change and time_modify attributes may be updated by the server when the LAYOUTCOMMIT operation is processed 💜 the reason is that some layout types don't support updating these attributes when the storage devices process I/O operations. if a client has a layout with the LAYOUTIOMODE4_RW iomode on the file, the client MAY provide a suggested value to the server for time_modify within the arguments to LAYOUTCOMMIT 🌈 based on the layout type, that provided value may or may not be used. the server should sanity-check the client-provided values before they're used. for example, the server should make sure that time does not flow backwards ⏳ the client always has the option to set time_modify through an explicit SETATTR operation 👑

for some layout protocols, the storage device can tell the metadata server that an I/O happened, so the change and time_modify attributes may be updated at the metadata server ✨ for a metadata server that can monitor updates to change and time_modify, LAYOUTCOMMIT processing is not required to update the change attribute. in this case, the metadata server must make sure that no further update to the data has happened since the last update of the attributes. file-based protocols may have enough information to figure that out, or they may update the change attribute on each file modification 🦋 this also applies to the time_modify attribute. if the server implementation can determine that the file hasn't been modified since the last time_modify update, the server doesn't need to update time_modify at LAYOUTCOMMIT 🫶 at LAYOUTCOMMIT completion, the updated attributes should be visible if the file was modified since the latest previous LAYOUTCOMMIT or LAYOUTGET 💖

12.5.4.2. LAYOUTCOMMIT and size ✨

okay so the size of a file can get updated when the client sends LAYOUTCOMMIT 📁💖. One of the fields in the argument is loca_last_write_offset, and it tells the server the highest byte offset that got written but hasn't been committed with LAYOUTCOMMIT yet 💅. Its data type is newoffset4, and it's switched on a boolean called no_newoffset, which basically says whether a previous write even happened 🫶. If no_newoffset is FALSE, no offset is given. But if the client has a layout with the LAYOUTIOMODE4_RW iomode on the file, and that layout's byte-range (the values of lo_offset and lo_length) overlaps loca_last_write_offset, then the client MAY set no_newoffset to TRUE and provide an offset that will update the file size 🎀. Keep in mind that offset is NOT the same thing as length, even though they're related 🙈. For example, a loca_last_write_offset value of zero means one byte was written at offset zero, so the length of the file is at least one byte 🧚.

The metadata server may do one of the following:

1. Update the file's size using the last write offset the client provided, either as the true file size or as a hint of the file size 🌸. If the metadata server has a method available, any new value for file size should be sanity-checked 🧐. For example, the file must not be truncated if the client presents a last write offset less than the file's current size 🙅‍♀️.
2. Ignore the client-provided last write offset. In that case the metadata server must have sufficient knowledge from other sources to determine the file's size 🔍. For example, the metadata server queries the storage devices with the control protocol 🛰️.

The method chosen to update the file's size depends on the storage device's and/or the control protocol's capabilities 🍓. For example, if the storage devices are block devices with no knowledge of file size, the metadata server must rely on the client to set the last write offset appropriately 🫠.

The results of LAYOUTCOMMIT contain a new size value in the form of a newsize4 union data type 👑. If the file's size is set as a result of LAYOUTCOMMIT, the metadata server must reply with the new size. Otherwise, the new size is not provided. If the file size is updated, the metadata server SHOULD update the storage devices so that the new file size is reflected when LAYOUTCOMMIT processing is complete ✨. For example, the client should be able to read up to the new file size 📖.

The client can extend the length of a file or truncate a file by sending a SETATTR operation to the metadata server with the size attribute specified 💜. If the size specified is larger than the current size of the file, the file is "zero extended", i.e., zeros are implicitly added between the file's previous EOF and the new EOF 🦋. (In many implementations, the zero-extended byte-range of the file consists of unallocated holes in the file 🕳️.) When the client writes past EOF via WRITE, the SETATTR operation does not need to be used 😌.

---

12.5.4.3. LAYOUTCOMMIT and layoutupdate 🧩

The LAYOUTCOMMIT argument contains a loca_layoutupdate field (Section 18.42.1) of data type layoutupdate4 (Section 3.3.18) 🎀. This argument is a layout-type-specific structure, which means it can carry arbitrary layout-type-specific information from the client to the metadata server at LAYOUTCOMMIT time 📦. For example, if you're using a block/volume layout, the client can tell the metadata server which reserved or allocated blocks it used or didn't use 🧱. The content of loca_layoutupdate (field lou_body) doesn't need to be the same layout-type-specific content that LAYOUTGET returned in the loc_body field of the lo_content field of the logr_layout field (Section 18.43.2) 🙃. The content of loca_layoutupdate is defined by the layout type specification and is opaque to LAYOUTCOMMIT 🤐.

---

12.5.5. Recalling a Layout 📞

okay this part is so important 🥹. Since a layout protects a client's access to a file through a direct client-storage-device path, a layout only needs to be recalled when it's no longer able to do that job semantically 🛡️. Usually that happens when the layout no longer captures where the file really lives over the byte-range it covers 📍. Any operation or action that changes the layout, like server-driven restriping or load balancing, will result in a recall of the layout ⚖️. A layout gets recalled with the CB_LAYOUTRECALL callback operation (see Section 20.3) and gets returned with LAYOUTRETURN (see Section 18.44) 📬. CB_LAYOUTRECALL can recall a layout identified by a byte-range, all layouts associated with a file system ID (FSID), or all layouts associated with a client ID. Section 12.5.5.2 talks about the sequencing issues around getting, returning, and recalling layouts 🧠.

An iomode also gets specified when recalling a layout 🎯. Generally, the iomode in the recall request has to match the layout being returned. For example, a recall with an iomode of LAYOUTIOMODE4_RW should make the client return only LAYOUTIOMODE4_RW layouts, not LAYOUTIOMODE4_READ layouts 🙅. However, there's a special LAYOUTIOMODE4_ANY enumeration that lets you recall a layout of any iomode 🌈. In other words, the client must return both LAYOUTIOMODE4_READ and LAYOUTIOMODE4_RW layouts 🫶.

A REMOVE operation SHOULD cause the metadata server to recall the layout 🗑️. This stops the client from accessing a file that doesn't exist anymore, and it lets the server reclaim state stored on the client 🧹. Since a REMOVE can get delayed until the last close of the file has happened, the recall can get delayed until that point too ⏳. After the last reference on the file has been released and the file has been removed, the client should no longer be able to perform I/O using the layout 🚫. For a file-based layout, the data server SHOULD return NFS4ERR_STALE in response to any operation on the removed file 💀.

Once a layout has been returned, the client MUST NOT send I/Os to the storage devices for the file, byte-range, and iomode that the returned layout represented 🚨. If a client does send an I/O to a storage device for which it doesn't hold a layout, the storage device SHOULD reject the I/O 🛑.

Okay, the next bit is a little spicy 🌶️. pNFS doesn't change how clients cache file data, or what those caching semantics mean 🤝. But it does recognize that some clients may do more aggressive write-behind caching to get the most out of pNFS 🏃‍♀️. The catch is that write-behind caching can hurt how long it takes to return a layout in response to a CB_LAYOUTRECALL 😬. This is similar to file delegations and the way file data caching affects DELEGRETURN 🔁. Client implementations SHOULD limit how much unwritten data they have outstanding at any one time, so they don't cause excessively long responses to CB_LAYOUTRECALL 🙏. Once a layout is recalled, a server MUST wait one lease period before taking further action 🕰️. As soon as a lease period has passed, the server may choose to fence the client's access to the storage devices if it thinks the client has taken too long to return the layout 🚧. But, just like with data delegations and DELEGRETURN, the server may also choose to wait, as long as the client is showing forward progress on its way to returning the layout 🏁. That forward progress can look like successful interaction with the storage devices, or like sub-portions of the layout being returned by the client 🧩. The server can also limit exposure to these problems by limiting the byte-ranges it initially provides in layouts, which in turn limits the amount of outstanding modified data 🫧.

12.5.5.1. Layout Recall Callback Robustness ✨

Okay so, up until now we've been assuming that the client's pNFS state (layout ranges and iomode) for a file matches the server's state for that file EXACTLY 💅. That means any callback gets answered with a LAYOUTRETURN or a batch of them that lines up perfectly with the range in the callback, because both sides are literally on the same page 📄. BUT it can be super useful if that assumption doesn't always hold, so let's talk about why 👀

• If conflicts that need callbacks are super rare, the server can use a multi-file callback to clean up per-client resources (like an FSID recall or a multi-file recall inside one CB_COMPOUND). That can mean WAY less pNFS traffic between client and server 🙌
• Servers can also keep track of what a client holds on a coarse-grained basis, so the server's layout ranges end up bigger than what the client actually has. At the extreme, a server could handle conflicts per file and only send whole-file callbacks, even though clients might request and get sub-file ranges 🧚
• Clients can "forget" the details of what layouts and ranges they have, so the server's ranges end up bigger than what the client "thinks" it has. As long as the client doesn't assume it has layouts beyond what the server actually granted, this is a totally safe move 🫶 But when a client forgets its stuff and gets a CB_LAYOUTRECALL, the client MUST follow up with a LAYOUTRETURN for what the server recalled, or else return the NFS4ERR_NOMATCHING_LAYOUT error if it has no layout to return in that recalled range.
• Here's the big one, bestie 🚨 To avoid errors, a client must not give itself layout permissions beyond what the server granted, and the server must not forget layout permissions it has granted. On the flip side, if the server thinks a client holds a layout the client has no idea about, it's helpful for the client to cleanly wrap up the requested recall. It can do that by sending a LAYOUTRETURN for the entire requested range, or by returning NFS4ERR_NOMATCHING_LAYOUT to the CB_LAYOUTRECALL.

So, with all that in mind, it makes sense for a server to be able to send callbacks for layout ranges it never granted to that client, and for a client to return ranges it doesn't actually hold 🦋 Still, a pNFS client MUST always return layouts that cover the full range specified by the recall. Note: the full recalled range doesn't have to come back in one single operation, it can be returned in pieces 🍓 That lets the client stage its flushing of dirty data, its commits, and its layout returns, and it also tells the metadata server that the client is making progress 💜

When a layout is returned, the client MUST NOT have any outstanding I/O requests to the storage devices involved in that layout. Said another way, the client MUST NOT return the layout while it still has outstanding I/O requests to the storage device 🙅‍♀️

Even with that rule, I/O requests can still show up at a storage device that's no longer allowed to handle them 😭 The server has zero strict control over when the client decides to return the layout, so the server may later decide to unilaterally revoke the client's access to the storage devices that the layout gave it. When revoking, the server has to deal with the possibility of lingering I/O requests, meaning requests still in flight to the storage devices named in the revoked layout. Every layout type specification MUST define whether unilateral layout revocation by the metadata server is supported. If it is, the spec also has to describe how lingering writes get processed. For example, the storage devices named in the revoked layout could be fenced off from the client that held that layout 🔒

To keep client and server converged on layout state, the final LAYOUTRETURN in a sequence of LAYOUTRETURN operations for a particular recall MUST specify the entire range being recalled. It has to echo the recalled layout type, iomode, recall/return type (FILE, FSID, or ALL), and byte-range, even if layouts for partial ranges were already returned earlier 🎀 Also, if the client holds no layouts that overlap the range being recalled, the client should return the NFS4ERR_NOMATCHING_LAYOUT error code to CB_LAYOUTRECALL. That way the server can update its view of the client's layout state 📊

12.5.5.2. Sequencing of Layout Operations 🧩

Just like with other stateful operations, pNFS needs layout operations to happen in the correct order 🙏 pNFS uses the seqid in the layout stateid to keep things in order between regular operations and callbacks. The server is responsible for avoiding inconsistencies in the layouts it hands out, and the client is responsible for properly serializing its layout requests and layout returns 💅

12.5.5.2.1. Layout Recall and Return Sequencing 🌸

This is a really critical one, not gonna lie 😤 The protocol has to protect against races between the reply to a LAYOUTGET or LAYOUTRETURN operation and a later CB_LAYOUTRECALL. A client MUST NOT process a CB_LAYOUTRECALL that implies one or more outstanding LAYOUTGET or LAYOUTRETURN operations that the client hasn't received a reply to yet. The client can spot this kind of CB_LAYOUTRECALL by checking the seqid field of the recall's layout stateid. If that seqid isn't exactly one higher than what the client currently has recorded, and the client has at least one LAYOUTGET and/or LAYOUTRETURN still outstanding, then the client knows the server sent the CB_LAYOUTRECALL after replying to an outstanding LAYOUTGET or LAYOUTRETURN. The client MUST wait before processing such a CB_LAYOUTRECALL until it has processed all the replies for outstanding LAYOUTGET and LAYOUTRETURN operations on the corresponding file whose seqid is less than the seqid given by CB_LAYOUTRECALL (lor_stateid; see Section 20.3.) 🧐

On top of the seqid mechanism, Section 2.10.6.3 describes the sessions mechanism, which also lets the client detect callback race conditions and delay processing that CB_LAYOUTRECALL. The server MAY reference conflicting operations in the CB_SEQUENCE that comes right before the CB_LAYOUTRECALL. Since the server already sent replies for those operations before the callback went out, those replies might race with the CB_LAYOUTRECALL. So the client MUST wait for all the referenced calls to complete and update its view of the layout state before it processes the CB_LAYOUTRECALL 🫧

12.5.5.2.1.1. Get/Return Sequencing 🔁

The protocol lets the client send concurrent LAYOUTGET and LAYOUTRETURN operations to the server. It doesn't give the server any way to process those requests in the same order they were created in 🙃 BUT the seqid field in the layout stateid lets the client work out the order the server handled parallel outstanding operations in. So when a layout retrieved by an outstanding LAYOUTGET overlaps with a layout returned by an outstanding LAYOUTRETURN on the same file, the order those two conflicting operations were processed in decides the final state of the overlapping layout. That order comes from the seqid returned in each operation: the operation with the higher seqid ran later 🏁

It's totally fine for the client to send multiple parallel LAYOUTGET operations for the same file, multiple parallel LAYOUTRETURN operations for the same file, or a mix of both 💖

It's also permissible for the client to use the current stateid (see Section 16.2.3.1.2) for LAYOUTGET operations, for example when compounding LAYOUTGETs or compounding OPEN and LAYOUTGETs. Using the current stateid when compounding LAYOUTRETURNs is fine too 🌈

Using the current stateid when combining LAYOUTRETURN and LAYOUTGET operations for the same file in the same COMPOUND request is permissible, since the server MUST process these in order. HOWEVER, if a client does send compound requests like that, it MUST NOT have more than one of them outstanding for the same file at the same time, and it MUST NOT have other LAYOUTGET or LAYOUTRETURN operations outstanding at that same time for that same file ⚠️

12.5.5.2.1.2. Client Considerations 🤔

Picture this: a pNFS client has sent a LAYOUTGET, and before the reply comes back, it receives a CB_LAYOUTRECALL for the same file with an overlapping range. There are two possibilities, and the client can tell them apart by looking at the layout stateid in the recall 👀

1. The server processed the LAYOUTGET before it sent the recall. The client has to wait for that LAYOUTGET, because it may be carrying layout information that will need to be returned to deal with the CB_LAYOUTRECALL 📦
2. The server sent the callback before it received the LAYOUTGET. The server won't respond to the LAYOUTGET until the CB_LAYOUTRECALL has been processed 🔄

If the client couldn't tell these possibilities apart, it could end up in a deadlock 💀 In the first case the client must wait for the LAYOUTGET response before it processes the recall, but in the second case that response won't arrive until the recall has been processed. Here's the trick: in the first case the seqid in the recall's layout stateid is two greater than what the client has recorded, while in the second case it's only one greater. That gap lets the client disambiguate the two cases, so it always knows exactly which possibility applies ✨

In case 1, the client knows it has to wait for the LAYOUTGET response before processing the recall (or the client can return NFS4ERR_DELAY). In case 2, the client will NOT wait for the LAYOUTGET response before processing the recall, because waiting would cause a deadlock 🙅‍♀️ So the client only has to wait when it hasn't yet seen the server's earlier responses to the LAYOUTGET operation(s).

The recall process is considered complete once the final LAYOUTRETURN for the recalled range is completed 🏆 That LAYOUTRETURN uses the layout stateid (with its seqid) specified in CB_LAYOUTRECALL. If the client uses multiple LAYOUTRETURNs to process the recall, the first LAYOUTRETURN uses the layout stateid as specified in CB_LAYOUTRECALL. The later LAYOUTRETURNs use the highest seqid, which is the usual case 🦄

12.5.5.2.1.3. Server Considerations ✨

okay so let's look at this race from the metadata server's side 👀 The server has sent a CB_LAYOUTRECALL and then gets an overlapping LAYOUTGET for the same file before the LAYOUTRETURN(s) that answer the CB_LAYOUTRECALL come in. There are three possible cases, and they're kind of iconic 💅

1. The client sent the LAYOUTGET before it processed the CB_LAYOUTRECALL. The "seqid" in the layout stateid of the LAYOUTGET arguments is one less than the "seqid" in CB_LAYOUTRECALL. The server returns NFS4ERR_RECALLCONFLICT, which tells the client there's a pending recall 🫶
2. The client sent the LAYOUTGET after it processed the CB_LAYOUTRECALL, but the LAYOUTGET got there before the LAYOUTRETURN and the response to CB_LAYOUTRECALL that finished that processing. The "seqid" in the layout stateid of LAYOUTGET is equal to or greater than the "seqid" in CB_LAYOUTRECALL. The server hasn't gotten a response to the CB_LAYOUTRECALL yet, so it returns NFS4ERR_RECALLCONFLICT 🔥
3. The client sent the LAYOUTGET after it processed the CB_LAYOUTRECALL, and the server did receive the CB_LAYOUTRECALL response, but the LAYOUTGET arrived before the LAYOUTRETURN that finished that processing. The "seqid" in the layout stateid of LAYOUTGET is equal to the "seqid" in CB_LAYOUTRECALL. Since the server has already gotten a response to the CB_LAYOUTRECALL, it returns NFS4ERR_RETURNCONFLICT 💀

12.5.5.2.1.4. Wraparound and Validation of Seqid 🎀

the rules for layout stateid processing are different from the other stateids in the protocol, and that's lowkey the whole point 🦋 The "seqid" value can't be zero, and the stateid's "seqid" value changes in a CB_LAYOUTRECALL operation. Because layout operations run in parallel, a set of LAYOUTGET and LAYOUTRETURN operations can end up with the same "seqid" value. The server uses a slightly modified version of the modulo arithmetic from Section 2.10.6.1 when it increments the layout stateid's "seqid". The twist is that zero is not a valid "seqid": when a "seqid" hits 0xFFFFFFFF, the next valid value is 0x00000001. ✨ The modulo arithmetic is also used to compare "seqid" values when processing CB_LAYOUTRECALL events, as described above in Section 12.5.5.2.1.3.

Just like the server validates the "seqid" for CB_LAYOUTRECALL usage (see Section 12.5.5.2.1.3), it also checks that the "seqid" value is in an appropriate range 👑 That range is the degree of parallelism the server supports for layout stateids. If the client is sending several layout operations in parallel, then by definition the "seqid" in the supplied stateid won't be the current "seqid" the server has. The range of parallelism goes from the highest, or current, "seqid" back to a "seqid" value in the past. To keep things clear, the server's current "seqid" value for a layout stateid is called SERVER_CURRENT_SEQID. The lowest "seqid" value the server will accept is called PAST_SEQID. And the range of valid "seqid"s, or the range of parallelism, is called VALID_SEQID_RANGE. So the following holds:

VALID_SEQID_RANGE = SERVER_CURRENT_SEQID - PAST_SEQID

All arithmetic in the following is the modulo arithmetic described above 🧚

The server MUST support a minimum VALID_SEQID_RANGE. That minimum is defined as:

VALID_SEQID_RANGE = summation over 1..N of (ca_maxoperations(i) - 1)

Here N is the number of session fore channels, and ca_maxoperations(i) is the value of the ca_maxoperations returned from CREATE_SESSION for the i'th session. The "- 1" is there to leave room for the required SEQUENCE operation 🙈 The server MAY support a VALID_SEQID_RANGE value larger than the minimum. The maximum VALID_SEQID_RANGE is (2 ^ 32 - 2), which accounts for zero not being a valid "seqid" value.

If the server finds that the "seqid" is zero, it returns the NFS4ERR_BAD_STATEID error to the client. The server then checks the "seqid" to make sure it's inside the range of parallelism, VALID_SEQID_RANGE. If the "seqid" value is outside that range, the server returns the error NFS4ERR_OLD_STATEID to the client. When the client gets NFS4ERR_OLD_STATEID, it updates the stateid in the layout request based on what it learned from processing other layout requests, and then re-sends the operation to the server 🔁

12.5.5.2.1.5. Bulk Recall and Return 🌈

pNFS can recall and return all layouts for files that belong to a particular fsid (LAYOUTRECALL4_FSID, LAYOUTRETURN4_FSID) or to a particular client ID (LAYOUTRECALL4_ALL, LAYOUTRETURN4_ALL). There are no "bulk" stateids, so you can't detect races through the seqid. The server MUST NOT initiate a bulk recall while another recall is in progress, or while the corresponding LAYOUTRETURN is in progress or pending. If the server sends a bulk recall while the client has a pending or in-progress LAYOUTRETURN, CB_LAYOUTRECALL, or LAYOUTGET, the client returns NFS4ERR_DELAY. If the client sends a LAYOUTGET or LAYOUTRETURN while a bulk recall is in progress, the server returns NFS4ERR_RECALLCONFLICT. If the client sends a LAYOUTGET or LAYOUTRETURN after the server gets NFS4ERR_DELAY from a bulk recall, the server MAY return NFS4ERR_RECALLCONFLICT, so that things keep moving forward 🚀

Once a CB_LAYOUTRECALL of LAYOUTRECALL4_ALL is sent, the server MUST NOT allow the client to use any layout stateid except for LAYOUTCOMMIT operations. Once the client receives a CB_LAYOUTRECALL of LAYOUTRECALL4_ALL, it MUST NOT use any layout stateid except for LAYOUTCOMMIT operations. Once a LAYOUTRETURN of LAYOUTRETURN4_ALL is sent, all layout stateids granted to the client ID are freed. The client MUST NOT use the layout stateids again. It MUST use LAYOUTGET to get new layout stateids 🫠

Once a CB_LAYOUTRECALL of LAYOUTRECALL4_FSID is sent, the server MUST NOT allow the client to use any layout stateid that refers to a file with the specified fsid except for LAYOUTCOMMIT operations. Once the client receives a CB_LAYOUTRECALL of LAYOUTRECALL4_ALL, it MUST NOT use any layout stateid that refers to a file with the specified fsid except for LAYOUTCOMMIT operations. Once a LAYOUTRETURN of LAYOUTRETURN4_FSID is sent, all layout stateids granted to the referenced fsid are freed. The client MUST NOT use those freed layout stateids for files with the referenced fsid again. After that, for any file with the referenced fsid, the client MUST first send a LAYOUTGET operation to get a new layout stateid for that file before it can use a layout 💖

If the server has sent a bulk CB_LAYOUTRECALL and then receives a LAYOUTGET, or a LAYOUTRETURN with a stateid, the server MUST return NFS4ERR_RECALLCONFLICT. If the server has sent a bulk CB_LAYOUTRECALL and receives a LAYOUTRETURN with an lr_returntype that is not equal to the lor_recalltype of that CB_LAYOUTRECALL, the server MUST return NFS4ERR_RECALLCONFLICT too 😭

12.5.6. Revoking Layouts ⭐

Parallel NFS lets servers revoke layouts from clients that don't answer recalls and/or don't renew their lease in time. Depending on the layout type, the server might revoke the layout and might also take certain actions on the client's I/O to data servers 🍓

12.5.7. Metadata Server Write Propagation 🦋

asynchronous writes that go through the metadata server might get propagated lazily to the storage devices 🥺 For data written asynchronously through the metadata server, a client reading from the appropriate storage device isn't guaranteed to see the new data until a COMMIT happens at the metadata server. While the write is pending, reads to the storage device might return the old data, the new data, or a mix of new and old, so be careful bestie ⚠️

Once a synchronous WRITE or a COMMIT (for asynchronously written data) finishes, the metadata server MUST ensure that the storage devices give out the new data and that the data has been written to stable storage. If the server stores its data in any way where it can't follow these constraints, then it MUST recall the layouts so that reads that can't be handled correctly don't happen. Also, the layouts MUST be recalled before the server responds to the associated WRITE operations. 🚨

12.6. pNFS Mechanics ✨

okay so this section walks through the whole operations flow a pNFS client follows when it's talking to a metadata server and a storage device 🧚💅

when a pNFS client runs into a brand new FSID, it sends a GETATTR to the NFSv4.1 server asking for the fs_layout_type (Section 5.12.1) attribute 👀 if that comes back with at least one layout type, and the types returned are ones the client actually supports, then the client knows pNFS is a possibility for that file system 🎀 now, if the client doesn't already have a client ID from an EXCHANGE_ID result that came back with EXCHGID4_FLAG_USE_PNFS_MDS from that same server, it MUST send an EXCHANGE_ID with the EXCHGID4_FLAG_USE_PNFS_MDS bit set 💖 and if the server's response doesn't have EXCHGID4_FLAG_USE_PNFS_MDS, then (even though fs_layout_type said otherwise 💀) the server doesn't support pNFS, so the client can't use pNFS with that server. in that case the server MUST return NFS4ERR_NOTSUPP for any pNFS operation 🙈

next, the client creates a session and asks for a persistent one, so exclusive creates can happen in a single round trip using the createmode4 of GUARDED4 🦋 if the session turns out not to be persistent, the client falls back to EXCLUSIVE4_1 for exclusive creates ✨

if a file is getting created on a pNFS-enabled file system, the client uses the OPEN operation 🌸 among the usual attributes that can go along with OPEN for creation, there's an OPTIONAL layout_hint attribute 💜 this is how the client says what layout type it'd prefer and what layout details it likes. the client can pass layout_hint at create time with a createmode4 of UNCHECKED4, GUARDED4, or EXCLUSIVE4_1 🧚 the client MUST NOT use EXCLUSIVE4 (see Table 10) 🙅‍♀️ the client is RECOMMENDED to put a GETATTR right after the OPEN in the same COMPOUND 🫶 that GETATTR can grab the layout_type for the newly created file, so the client finds out which layout type the server picked and therefore which storage protocol it has to speak 🔥

if the client is opening a file that already exists, it also includes a GETATTR to figure out which layout type the file supports 👑

in either the create case or the plain open case, that GETATTR can also ask for layout_blksize and layout_alignment, so the client can work out the best offsets and lengths for I/O on the file 🍓

assuming the client supports the layout type that GETATTR returned and decides to use pNFS for data access, it sends LAYOUTGET with the filehandle and stateid from OPEN, plus the range it wants to do I/O on 🌈 the response is a layout, which might cover only part of the range that was asked for. it also has device IDs and a description of how the data is organized (or, when writing, how it's supposed to be organized) across those devices ✨ the device IDs and data description are encoded in a format specific to the layout type, but the client is expected to understand it 💅

when the client wants to send an I/O, it looks at the data description in the layout to figure out which device ID the I/O command goes to 🦋 then it sends a GETDEVICEINFO to find the device address(es) for that device ID 💖 the client then sends the I/O request to one of those device addresses, using the storage protocol defined for the layout type 🎀 and fyi, if a client has multiple I/Os to send, those requests may be done in parallel ⭐

if the I/O was a WRITE, the client may at some point want to use LAYOUTCOMMIT to commit the modification time and the new file size (if it thinks it extended the file) to the metadata server, and the modified data to the file system 🫶😭

---

12.7. Recovery 🌸

recovery is kinda complicated bc pNFS is distributed 😵‍💫 in general, crash recovery for layouts works a lot like crash recovery for delegations in base NFSv4.1 🦋 BUT the client being able to do I/O without checking in with the metadata server adds some subtleties 💀 those have to be handled correctly or the file system could get corrupted, so pay attention bestie 👑

12.7.1. Recovery from Client Restart 🧚

client recovery for layouts is similar to client recovery for other lock and delegation state ✨ when a pNFS client restarts, it loses all info about the layouts it used to own 🥺 there are two ways the server can reclaim those resources and let conflicting layouts go to other clients 💅

the first way is lease expiry. if the client's recovery takes longer than the lease period, the lease expires and the server knows that state can be released 🎀 for layouts, the server can release the state right away when the lease expires, or it can let the layout hang around in case the lease gets revived, as long as no other layout conflicts with it 🦋

the second way is when the client restarts faster than the lease period runs out 🏃‍♀️ then the client contacts the server through the normal EXCHANGE_ID protocol. the server sees that the client's co_ownerid matches the previous invocation's co_ownerid, but the verifier is different 👀 that tells the server to release all layout state tied to the client's previous invocation 🔥 in this case, data the client wrote but that isn't covered by a successful LAYOUTCOMMIT is in an undefined state 😭 it may have been written, or it may be lost 💀 this is acceptable behavior, and it's the client's job to use LAYOUTCOMMIT to get the stability it wants 🫶

12.7.2. Dealing with Lease Expiration on the Client 💜

if a client thinks its lease has expired, it MUST NOT send I/O to the storage device until it has validated the lease 🙅‍♀️ the client can send a SEQUENCE operation to the metadata server. if SEQUENCE succeeds but sr_status_flag has SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED, SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, or SEQ4_STATUS_ADMIN_STATE_REVOKED set, the client MUST NOT use its currently held layouts 🥹 the client has two choices to recover. first, for all modified but uncommitted data, it writes that data to the metadata server using the FILE_SYNC4 flag on the WRITEs, or with WRITE and COMMIT 🌈 second, it re-establishes a client ID and session with the server, gets new layouts and device-ID-to-device-address mappings for the modified data ranges, and then writes the data to the storage devices using those new layouts ✨

if sr_status_flags from the metadata server has SEQ4_STATUS_RESTART_RECLAIM_NEEDED set (or SEQUENCE returns NFS4ERR_BAD_SESSION and CREATE_SESSION returns NFS4ERR_STALE_CLIENTID), then the metadata server has restarted, and the client SHOULD recover using the methods in Section 12.7.4 💖

if sr_status_flags from the metadata server has SEQ4_STATUS_LEASE_MOVED set, the client recovers by following the procedure in Section 11.7.7.1 🦋 after that, the client may get an indication that the layout state wasn't moved along with the file system. the client then recovers the same way it would in the other applicable situations from the first two paragraphs of this section 🎀

if sr_status_flags reports no loss of state, then the lease for the client's layouts is valid and renewed, and the client can go back to sending I/O requests to the storage devices 🥳

the client SHOULD NOT send I/Os to storage devices that could run past the lease expiration time period 🙈 but that isn't always possible. for example, a long network partition could start after the I/O is sent and not heal until the storage device gets the request 😱 so the metadata server and/or storage devices are responsible for protecting themselves from I/Os that were sent before the lease expired but arrive after it 🛡️ see Section 12.7.3 💅

12.7.3. Dealing with Loss of Layout State on the Metadata Server 🚨

this section covers the case where all of the following are true:

• the metadata server has not restarted
• a pNFS client's layouts have been discarded (usually because the client's lease expired) and are invalid
• an I/O from the pNFS client arrives at the storage device

the metadata server and its storage devices MUST solve this by fencing the client 🚫 in other words, they MUST solve this by preventing I/O operations from the client to the storage devices after layout state loss 💀 the details of how fencing is done are specific to the layout type. the solution for NFSv4.1 file-based layouts is described in (Section 13.11), and solutions for other layout types are in their respective external specification documents 📚

12.7.4. Recovery from Metadata Server Restart ✨

okay bestie, the pNFS client finds out the metadata server restarted through the methods in Section 8.4.2 🔍 and there's a pNFS-specific version of that in Paragraph 2, of Section 12.7.2 🦋. The client MUST stop using layouts and delete the device ID to device address mappings it got from the metadata server 🧹. Once that's done, if it wrote data to the storage device without committing the layouts via LAYOUTCOMMIT, it has extra work to do so the client, metadata server, and storage device(s) are all on the same page about the data 🫶

• If the client has data still modified and unwritten in its memory, it only has two choices 💀
  1. The client can obtain a layout via LAYOUTGET after the server's grace period and write the data to the storage devices.
  2. The client can WRITE that data through the metadata server using the WRITE (Section 18.32) operation, and then obtain layouts as desired.
• If the client asynchronously wrote data to the storage device but still has a copy in its memory, it has the recovery options from the bullet above available. If the metadata server is also in its grace period, it has the options in the next bullet too 🌸
• The client doesn't have a copy of the data in memory and the metadata server is still in its grace period. This is the tricky one 😭 The client cannot use LAYOUTGET (within or outside the grace period) to reclaim a layout, because the response may not match what it had before. The range might be different, or the client might get the same range with different layout content. Even if the content looks the same, the device IDs may map to different device addresses, and even if the device addresses match, they could have been assigned to a different storage device. Retrieving the data from the storage device and writing it to the metadata server isn't an option either, because the mappings of range to device ID, device ID to device address, and device address to physical device are all stale. New mappings from a new LAYOUTGET don't fix that.

The only recovery option here is sending a LAYOUTCOMMIT in reclaim mode, which the metadata server will accept as long as it's still in its grace period 🧚 Using reclaim mode tells the metadata server that the layout has changed. It is critical that the metadata server gets this information before its grace period ends, and therefore before it starts allowing updates to the file system ⭐

To send LAYOUTCOMMIT in reclaim mode, the client sets the loca_reclaim field of the operation's arguments (Section 18.42.1) to TRUE. During the metadata server's recovery grace period (and only during it), the metadata server is prepared to accept LAYOUTCOMMIT requests with the loca_reclaim field set to TRUE.

When loca_reclaim is TRUE, the client is trying to commit changes to the layout that happened before the metadata server restarted. The metadata server runs some consistency checks on the loca_layoutupdate field of the arguments to decide whether the client can commit the data written to the storage device to the file system. loca_layoutupdate is of data type layoutupdate4 and holds layout-type-specific content (in the lou_body field of loca_layoutupdate). The layout-type-specific info that loca_layoutupdate might have is discussed in Section 12.5.4.3. If the metadata server's checks on loca_layoutupdate succeed, the metadata server MUST commit the data (as described by the loca_offset, loca_length, and loca_layoutupdate fields of the arguments) that was written to the storage device 🎉 If the checks fail, the metadata server rejects the LAYOUTCOMMIT operation and makes no changes to the file system.

🚨 However, any time LAYOUTCOMMIT with loca_reclaim TRUE fails, the pNFS client has lost all the data in the range defined by <loca_offset, loca_length>. A client can defend against this risk by caching all data, whether it was written synchronously or asynchronously, in its memory, and by not releasing that cached data until a successful LAYOUTCOMMIT. This doesn't hold true for every layout type. For example, file-based storage devices need not suffer from this limitation.

• The client doesn't have a copy of the data in memory and the metadata server is no longer in its grace period. In other words, the metadata server returns NFS4ERR_NO_GRACE. Just like the bullet above, if LAYOUTCOMMIT fails, the data in the range <loca_offset, loca_length> is lost 💔 The defense is the same: cache all written data on the client until a successful LAYOUTCOMMIT.

12.7.5. Operations during Metadata Server Grace Period 🕰️

Some of the recovery scenarios above noted that some operations (namely, WRITE and LAYOUTGET) might be permitted during the metadata server's grace period. The metadata server may allow these operations during its grace period. For LAYOUTGET, the metadata server must reliably determine that servicing such a request will not conflict with an impending LAYOUTCOMMIT reclaim request. For WRITE, the metadata server must reliably determine that servicing the request will not conflict with an impending OPEN or with a LOCK where the file has mandatory byte-range locking enabled 🔒

As mentioned earlier, for expediency the metadata server might reject some operations (namely, WRITE and LAYOUTGET) during its grace period, because the simplest correct approach is to reject all non-reclaim pNFS requests and WRITE operations by returning the NFS4ERR_GRACE error. However, depending on the storage protocol (which is specific to the layout type) and the metadata server implementation, the metadata server may be able to determine that a particular request is safe. For example, a metadata server may save provisional allocation mappings for each file to stable storage, along with information about potentially conflicting OPEN share modes and mandatory byte-range locks that might have been in effect at the time of restart. The metadata server may then use this information during the recovery grace period to determine that a WRITE request is safe 🧠

12.7.6. Storage Device Recovery 💾

Recovery from storage device restart mostly depends on the layout type in use. However, there are a few general techniques a client can use if it discovers a storage device has crashed while holding modified, uncommitted data that was asynchronously written. First and foremost, it's important to realize that the client is the only one with the information needed to recover non-committed data, since it holds the modified data and probably nothing else does 👀 Second, the best solution is for the client to err on the side of caution and attempt to rewrite the modified data through another path 🛤️

The client SHOULD immediately WRITE the data to the metadata server, with the stable field in the WRITE4args set to FILE_SYNC4. Once it does this, there's no need to wait for the original storage device 🙌

12.8. Metadata and Storage Device Roles 🎭

If the same physical hardware is used to implement both a metadata server and a storage device, then that hardware entity is understood to be playing two distinct roles, and it's important to be clear about which role the hardware is acting on behalf of at any given time 💅

Two sub-cases can be distinguished.

1. The storage device uses NFSv4.1 as the storage protocol, i.e., the same physical hardware is used to implement both a metadata and data server. See Section 13.1 for a description of how multiple roles are handled.
2. The storage device does not use NFSv4.1 as the storage protocol, and the same physical hardware is used to implement both a metadata and storage device. Whether distinct network addresses are used to access the metadata server and storage device is immaterial. This is because it's always clear to the pNFS client and server, from the upper-layer protocol being used (NFSv4.1 or non-NFSv4.1), which role a request to the common server network address is aimed at 🦋

12.9. Security Considerations for pNFS 🔐

okay bestie, so pNFS splits file system metadata and data apart and gives you access to both 🦋 The pNFS-specific operations (listed in Section 12.3) get you to the metadata, and every existing NFSv4.1 conventional (non-pNFS) security mechanism and feature applies to accessing that metadata 💅 The whole combo of pieces in a pNFS system (check out Figure 1) has to keep NFSv4.1's security properties intact for anyone reaching a storage device from a client 🫶 That includes the countermeasures against threats NFSv4.1 defends against, in environments where those threats are seriously concerning ✨

In some cases, the countermeasures for storage device connections look more like physical isolation, or just a "maybe don't use pNFS here" 🙈 For example, it can be impractical to give some storage protocols confidentiality protection against eavesdropping. If eavesdropping on those protocols is worrying enough to need countermeasures, then physical isolation of the channel (like a direct connection from client(s) to storage device(s)) and/or choosing not to use pNFS (and falling back to conventional NFSv4.1) may be appropriate courses of action 💀

Where communication with storage devices faces the same threats as client-to-metadata server communication, the protocols used for it need security mechanisms as strong as, or no weaker than, the ones RPCSEC_GSS offers for NFSv4.1 🔥 The exception is the storage protocol for the LAYOUT4_NFSV4_1_FILES layout (see Section 13). In other words, except for NFSv4.1 itself, specifying security mechanisms for storage access protocols is beyond the scope of this document 🌈

pNFS implementations MUST NOT remove NFSv4.1's access controls. 🚫 The clients, storage devices, and metadata server all work together to make sure every client-to-storage-device file data access respects NFSv4.1's ACLs and file open modes 👑 That means doing both of these checks on every access, either in the client, the storage device, or both (depending on what applies; when the storage device is an NFSv4.1 server, the storage device is ultimately responsible for controlling access, as described in Section 13.9.2) ✨ If a pNFS configuration only does these checks in the client, then the risk of a misbehaving client getting unauthorized access is a big deal when deciding if that setup is appropriate 😭 Such layout types SHOULD NOT be used when client-only access checks don't give sufficient assurance that NFSv4.1 access control is being applied correctly. 🙅‍♀️ (This isn't a problem for the file layout type in Section 13, because the storage access protocol for LAYOUT4_NFSv4_1_FILES is NFSv4.1, so the security model for storage device access via LAYOUT4_NFSv4_1_FILES is the same as the metadata server's 💖) For access control specific to a layout, the reader should look at the layout specification, such as the NFSv4.1/file-based layout (Section 13) of this document, the blocks layout [41], and the objects layout [40] 🧚

---

13. NFSv4.1 as a Storage Protocol in pNFS: the File Layout Type 🎀

This section is all about the semantics and format of NFSv4.1 file-based layouts for pNFS 🦋 NFSv4.1 file-based layouts use the LAYOUT4_NFSV4_1_FILES layout type, and that type defines striping data across multiple NFSv4.1 data servers 💜

13.1. Client ID and Session Considerations 🌸

Sessions are a REQUIRED feature of NFSv4.1, and that extends to both the metadata server and the file-based (NFSv4.1-based) data servers ✨

The role a server plays in pNFS is determined by the result it returns from EXCHANGE_ID. The roles are:

• Metadata server (EXCHGID4_FLAG_USE_PNFS_MDS is set in the result eir_flags).
• Data server (EXCHGID4_FLAG_USE_PNFS_DS).
• Non-metadata server (EXCHGID4_FLAG_USE_NON_PNFS). This is an NFSv4.1 server that doesn't support operations (e.g., LAYOUTGET) or attributes that pertain to pNFS 🙃

The client MAY request zero or more of EXCHGID4_FLAG_USE_NON_PNFS, EXCHGID4_FLAG_USE_PNFS_DS, or EXCHGID4_FLAG_USE_PNFS_MDS, even though some combinations (e.g., EXCHGID4_FLAG_USE_NON_PNFS | EXCHGID4_FLAG_USE_PNFS_MDS) are contradictory 🤪 However, the server MUST only return the following acceptable combinations:

+--------------------------------------------------------+
| Acceptable Results from EXCHANGE_ID                    |
+--------------------------------------------------------+
| EXCHGID4_FLAG_USE_PNFS_MDS                             |
| EXCHGID4_FLAG_USE_PNFS_MDS | EXCHGID4_FLAG_USE_PNFS_DS |
| EXCHGID4_FLAG_USE_PNFS_DS                              |
| EXCHGID4_FLAG_USE_NON_PNFS                             |
| EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_NON_PNFS |
+--------------------------------------------------------+

As the table implies, a server can have one or two roles. A server can be both a metadata server and a data server, or both a data server and a non-metadata server 🧠 On top of returning two roles in the EXCHANGE_ID's results (so it serves both roles through a common client ID), a server can also serve two roles by returning a unique client ID and server owner for each role, across two EXCHANGE_ID results, with each result indicating one role each 💅

In the case of a server with concurrent pNFS roles served by a common client ID, if the EXCHANGE_ID request has zero or a combination of the bits set in eia_flags, the server result should set the bits representing the higher of the acceptable combination of server roles, with a preference for matching the roles the client requested 🌈 So if a client request has (EXCHGID4_FLAG_USE_NON_PNFS | EXCHGID4_FLAG_USE_PNFS_MDS | EXCHGID4_FLAG_USE_PNFS_DS) flags set, and the server is both a metadata server and a data server serving both roles through a common client ID, the server SHOULD return with (EXCHGID4_FLAG_USE_PNFS_MDS | EXCHGID4_FLAG_USE_PNFS_DS) set ⭐

In the case of a server with multiple concurrent pNFS roles, each served by a unique client ID, if the client specifies zero or a combination of roles in the request, the server results SHOULD return only one of the roles from the combination the client requested 🍓 If the role in the server result doesn't match what the client intended to use, the client should send the EXCHANGE_ID specifying just the pNFS role it's interested in 🫶

If a pNFS metadata client gets a layout that refers it to an NFSv4.1 data server, it needs a client ID on that data server 🦋 If it doesn't yet have one from the server that had EXCHGID4_FLAG_USE_PNFS_DS set in its EXCHANGE_ID results, then the client needs to send an EXCHANGE_ID to the data server, using the same co_ownerid it sent to the metadata server, with EXCHGID4_FLAG_USE_PNFS_DS set in the arguments ✨ If the server's EXCHANGE_ID results have EXCHGID4_FLAG_USE_PNFS_DS set, then the client may use that client ID to create sessions that exchange pNFS data operations 💜 The client ID returned by the data server has no relationship with the client ID returned by a metadata server, unless the client IDs are equal and the server owners and server scopes of the data server and metadata server are equal 🤔

In NFSv4.1, the session ID in the SEQUENCE operation implies the client ID, which in turn might be used by the server to map the stateid to the right client/server pair. But here's the tea 🍵: when a data server gets a READ or WRITE operation with a stateid, that stateid is associated with a client ID on a metadata server. The session ID in the preceding SEQUENCE operation is tied to the data server's client ID, so the data server has no obvious way to figure out the metadata server from the COMPOUND procedure, and therefore no way to validate the stateid 😭 One RECOMMENDED approach is for pNFS servers to encode metadata server routing and/or identity information in the data server filehandles returned in the layout 💖

If metadata server routing and/or identity information is encoded in data server filehandles, then when the metadata server identity or location changes, the data server filehandles it handed out become invalid (stale), so the metadata server MUST first recall the layouts 🚨 Invalidating a data server filehandle does not make the NFS client's data cache invalid 🙌 The client's cache should map a data server filehandle to a metadata server filehandle, and a metadata server filehandle to cached data 🧚

If a server is both a metadata server and a data server, it might need to tell apart operations on files directed to the metadata server from those directed to the data server 🎀 It is RECOMMENDED that the values of the filehandles returned by the LAYOUTGET operation differ from the value of the filehandle returned by OPEN for the same file 🌷

Another scenario is when the metadata server and the storage device are distinct from one client's point of view, but the roles are reversed from another client's point of view 🔄 For example, in the cluster file system model, a metadata server to one client might be a data server to another client 👯 If NFSv4.1 is being used as the storage protocol, then pNFS servers need to encode the values of filehandles according to their specific roles 💅

13.1.1. Sessions Considerations for Data Servers 🌙

Section 2.10.11.2 says a client has to keep its lease renewed so the server doesn't delete its session ⏰ If the reply to EXCHANGE_ID has just the EXCHGID4_FLAG_USE_PNFS_DS role set, then (as noted in Section 13.6) the client won't be able to determine the data server's lease_time attribute, because GETATTR won't be permitted 😵 Instead, the rule is that any time a client receives a layout referring it to a data server that returns just the EXCHGID4_FLAG_USE_PNFS_DS role, the client MAY assume that the lease_time attribute from the metadata server that returned the layout applies to the data server 🧃 So the data server MUST be aware of the values of all lease_time attributes of all metadata servers it's providing I/O for, and it MUST use the maximum of all those lease_time values as the lease interval for all client IDs and sessions established on it 👑

For example, say one metadata server has a lease_time attribute of 20 seconds, and a second metadata server has a lease_time attribute of 10 seconds. If both servers return layouts referring to an EXCHGID4_FLAG_USE_PNFS_DS-only data server, then the data server MUST renew a client's lease if the interval between two SEQUENCE operations on different COMPOUND requests is less than 20 seconds ⏳💖

13.2. File Layout Definitions 🎀

Okay so these are the words we use for the LAYOUT4_NFSV4_1_FILES layout type, and they might be useful for other layout types too 👀 Let's get the vocab down, bestie 💅

• Unit. A unit is a fixed-size quantity of data written to a data server 📦
• Pattern. A pattern is a method of distributing one or more equal sized units across a set of data servers. A pattern is iterated one or more times 🔁
• Stripe. A stripe is a set of data distributed across a set of data servers in a pattern before that pattern repeats ✨
• Stripe Count. A stripe count is the number of units in a pattern 🔢
• Stripe Width. A stripe width is the size of a stripe in bytes. The stripe width = the stripe count * the size of the stripe unit 📏

Hereafter, this document will refer to a unit that is written in a pattern as a "stripe unit" 💖

A pattern may have more stripe units than data servers. If so, some data servers will have more than one stripe unit per stripe. A data server that has multiple stripe units per stripe MAY store each unit in a different data file (and depending on the implementation, will possibly assign a unique data filehandle to each data file). It's giving flexibility, no cap 🦋

13.3. File Layout Data Types 🧚

Okay, so the high level NFSv4.1 layout types are nfsv4_1_file_layouthint4, nfsv4_1_file_layout_ds_addr4, and nfsv4_1_file_layout4. Main character energy 👑

The SETATTR operation supports a layout hint attribute (Section 5.12.4). When the client sets a layout hint (data type layouthint4) with a layout type of LAYOUT4_NFSV4_1_FILES (the loh_type field), the loh_body field contains a value of data type nfsv4_1_file_layouthint4.

   const NFL4_UFLG_MASK            = 0x0000003F;
   const NFL4_UFLG_DENSE           = 0x00000001;
   const NFL4_UFLG_COMMIT_THRU_MDS = 0x00000002;
   const NFL4_UFLG_STRIPE_UNIT_SIZE_MASK
                                   = 0xFFFFFFC0;

   typedef uint32_t nfl_util4;

   enum filelayout_hint_care4 {
           NFLH4_CARE_DENSE        = NFL4_UFLG_DENSE,

           NFLH4_CARE_COMMIT_THRU_MDS
                                   = NFL4_UFLG_COMMIT_THRU_MDS,

           NFLH4_CARE_STRIPE_UNIT_SIZE
                                   = 0x00000040,

           NFLH4_CARE_STRIPE_COUNT = 0x00000080
   };

   /* Encoded in the loh_body field of data type layouthint4: */

   struct nfsv4_1_file_layouthint4 {
           uint32_t        nflh_care;
           nfl_util4       nflh_util;
           count4          nflh_stripe_count;
   };

The generic layout hint structure is described in Section 3.3.19. The client uses the layout hint in the layout_hint (Section 5.12.4) attribute to indicate the preferred type of layout to be used for a newly created file. The LAYOUT4_NFSV4_1_FILES layout-type-specific content for the layout hint is composed of three fields.

The first field, nflh_care, is a set of flags indicating which values of the hint the client cares about. If the NFLH4_CARE_DENSE flag is set, then the client indicates in the second field, nflh_util, a preference for how the data file is packed (Section 13.4.4), which is controlled by the value of the expression nflh_util & NFL4_UFLG_DENSE ("&" represents the bitwise AND operator). If the NFLH4_CARE_COMMIT_THRU_MDS flag is set, then the client indicates a preference for whether the client should send COMMIT operations to the metadata server or data server (Section 13.7), which is controlled by the value of nflh_util & NFL4_UFLG_COMMIT_THRU_MDS. If the NFLH4_CARE_STRIPE_UNIT_SIZE flag is set, the client indicates its preferred stripe unit size, which is indicated in nflh_util & NFL4_UFLG_STRIPE_UNIT_SIZE_MASK (thus, the stripe unit size MUST be a multiple of 64 bytes). The minimum stripe unit size is 64 bytes. If the NFLH4_CARE_STRIPE_COUNT flag is set, the client indicates in the third field, nflh_stripe_count, the stripe count. The stripe count multiplied by the stripe unit size is the stripe width.

When LAYOUTGET returns a LAYOUT4_NFSV4_1_FILES layout (indicated in the loc_type field of the lo_content field), the loc_body field of the lo_content field contains a value of data type nfsv4_1_file_layout4. Among other content, nfsv4_1_file_layout4 has a storage device ID (field nfl_deviceid) of data type deviceid4. The GETDEVICEINFO operation maps a device ID to a storage device address (type device_addr4). When GETDEVICEINFO returns a device address with a layout type of LAYOUT4_NFSV4_1_FILES (the da_layout_type field), the da_addr_body field contains a value of data type nfsv4_1_file_layout_ds_addr4.

   typedef netaddr4 multipath_list4<>;

   /*
    * Encoded in the da_addr_body field of
    * data type device_addr4:
    */
   struct nfsv4_1_file_layout_ds_addr4 {
           uint32_t        nflda_stripe_indices<>;
           multipath_list4 nflda_multipath_ds_list<>;
   };

The nfsv4_1_file_layout_ds_addr4 data type represents the device address. It is composed of two fields:

1. nflda_multipath_ds_list: An array of lists of data servers, where each list can be one or more elements, and each element represents a data server address that may serve equally as the target of I/O operations (see Section 13.5). The length of this array might be different than the stripe count.
2. nflda_stripe_indices: An array of indices used to index into nflda_multipath_ds_list. The value of each element of nflda_stripe_indices MUST be less than the number of elements in nflda_multipath_ds_list. Each element of nflda_multipath_ds_list SHOULD be referred to by one or more elements of nflda_stripe_indices. The number of elements in nflda_stripe_indices is always equal to the stripe count.

   /*
    * Encoded in the loc_body field of
    * data type layout_content4:
    */
   struct nfsv4_1_file_layout4 {
            deviceid4      nfl_deviceid;
            nfl_util4      nfl_util;
            uint32_t       nfl_first_stripe_index;
            offset4        nfl_pattern_offset;
            nfs_fh4        nfl_fh_list<>;
   };

The nfsv4_1_file_layout4 data type represents the layout. It is composed of the following fields:

1. nfl_deviceid: The device ID that maps to a value of type nfsv4_1_file_layout_ds_addr4.
2. nfl_util: Like the nflh_util field of data type nfsv4_1_file_layouthint4, a compact representation of how the data on a file on each data server is packed, whether the client should send COMMIT operations to the metadata server or data server, and the stripe unit size. If a server returns two or more overlapping layouts, each stripe unit size in each overlapping layout MUST be the same.
3. nfl_first_stripe_index: The index into the first element of the nflda_stripe_indices array to use.
4. nfl_pattern_offset: This field is the logical offset into the file where the striping pattern starts. It is required for converting the client's logical I/O offset (e.g., the current offset in a POSIX file descriptor before the read() or write() system call is sent) into the stripe unit number (see Section 13.4.1).

If dense packing is used, then nfl_pattern_offset is also needed to convert the client's logical I/O offset to an offset on the file on the data server corresponding to the stripe unit number (see Section 13.4.4).

Note that nfl_pattern_offset is not always the same as lo_offset. For example, via the LAYOUTGET operation, a client might request a layout starting at offset 1000 of a file that has its striping pattern start at offset zero. 🤓

5. nfl_fh_list: An array of data server filehandles for each list of data servers in each element of the nflda_multipath_ds_list array. The number of elements in nfl_fh_list depends on whether sparse or dense packing is being used.
   • If sparse packing is being used, the number of elements in nfl_fh_list MUST be one of three values:
     • Zero. This means that filehandles used for each data server are the same as the filehandle returned by the OPEN operation from the metadata server.
     • One. This means that every data server uses the same filehandle: what is specified in nfl_fh_list[0].
     • The same number of elements in nflda_multipath_ds_list. Thus, in this case, when sending an I/O operation to any data server in nflda_multipath_ds_list[X], the filehandle in nfl_fh_list[X] MUST be used.

See the discussion on sparse packing in Section 13.4.4.

• If dense packing is being used, the number of elements in nfl_fh_list MUST be the same as the number of elements in nflda_stripe_indices. Thus, when sending an I/O operation to any data server in nflda_multipath_ds_list[nflda_stripe_indices[Y]], the filehandle in nfl_fh_list[Y] MUST be used. In addition, any time there exists i and j, (i != j), such that the intersection of nflda_multipath_ds_list[nflda_stripe_indices[i]] and nflda_multipath_ds_list[nflda_stripe_indices[j]] is not empty, then nfl_fh_list[i] MUST NOT equal nfl_fh_list[j]. In other words, when dense packing is being used, if a data server appears in two or more units of a striping pattern, each reference to the data server MUST use a different filehandle.

Indeed, if there are multiple striping patterns, as indicated by the presence of multiple objects of data type layout4 (either returned in one or multiple LAYOUTGET operations), and a data server is the target of a unit of one pattern and another unit of another pattern, then each reference to each data server MUST use a different filehandle.

See the discussion on dense packing in Section 13.4.4. ✨

The details on the interpretation of the layout are in Section 13.4.

13.4. Interpreting the File Layout 🦋

13.4.1. Determining the Stripe Unit Number 💅

To find the stripe unit number that corresponds to the client's logical file offset, the pattern offset will also be used. The i'th stripe unit (SUi) is:

    relative_offset = file_offset - nfl_pattern_offset;
    SUi = floor(relative_offset / stripe_unit_size);

Bestie, that's the whole math, and it's giving clean 🌈✨

13.4.2. Interpreting the File Layout Using Sparse Packing ✨

okay bestie, so when sparse packing is in use, here's the exact algorithm for figuring out which filehandle and which set of data-server addresses to write stripe unit i (SUi) to 💅

stripe_count = number of elements in nflda_stripe_indices;

j = (SUi + nfl_first_stripe_index) % stripe_count;

idx = nflda_stripe_indices[j];

fh_count = number of elements in nfl_fh_list;
ds_count = number of elements in nflda_multipath_ds_list;

switch (fh_count) {
  case ds_count:
    fh = nfl_fh_list[idx];
    break;

  case 1:
    fh = nfl_fh_list[0];
    break;

  case 0:
    fh = filehandle returned by OPEN;
    break;

  default:
    throw a fatal exception;
    break;
}

address_list = nflda_multipath_ds_list[idx];

the client then picks a data server from address_list and sends a READ or WRITE operation using the filehandle in fh. easy peasy 🎀

Let's look at an example, because it's giving clarity 🦋

Say we have a device address with seven data servers, sorted into three equivalence (Section 13.5) classes. where A through G are network addresses:

{ A, B, C, D }, { E }, { F, G }

So the list looks like this:

nflda_multipath_ds_list<> = { A, B, C, D }, { E }, { F, G }

i.e.,

nflda_multipath_ds_list[0] = { A, B, C, D }

nflda_multipath_ds_list[1] = { E }

nflda_multipath_ds_list[2] = { F, G }

The striping index array is:

nflda_stripe_indices<> = { 2, 0, 1, 0 }

Now the client gets a layout with a device ID that maps to that device address. The initial index is:

nfl_first_stripe_index = 2,

and the filehandle list is:

nfl_fh_list = { 0x36, 0x87, 0x67 }.

If the client wants to write SU0, the valid { network address, filehandle } combos for SUi come from the first stripe index. Starting with nfl_first_stripe_index = 2:

idx = nflda_stripe_indices[(0 + 2) % 4]

    = nflda_stripe_indices[2]

    = 1

So:

nflda_multipath_ds_list[1] = { E }

nfl_fh_list[1] = { 0x87 }

which means the client can write SU0 to { 0x87, { E } } 🫶

Here are the destinations for the first 13 storage units:

                  +-----+------------+--------------+
                  | SUi | filehandle | data servers |
                  +-----+------------+--------------+
                  | 0   | 87         | E            |
                  | 1   | 36         | A,B,C,D      |
                  | 2   | 67         | F,G          |
                  | 3   | 36         | A,B,C,D      |
                  | 4   | 87         | E            |
                  | 5   | 36         | A,B,C,D      |
                  | 6   | 67         | F,G          |
                  | 7   | 36         | A,B,C,D      |
                  | 8   | 87         | E            |
                  | 9   | 36         | A,B,C,D      |
                  | 10  | 67         | F,G          |
                  | 11  | 36         | A,B,C,D      |
                  | 12  | 87         | E            |
                  +-----+------------+--------------+

13.4.3. Interpreting the File Layout Using Dense Packing 🌈

now for dense packing! same vibe, different filehandle drama 💅 When dense packing is used, the algorithm for finding the filehandle and set of data server network addresses to write stripe unit i (SUi) to is:

stripe_count = number of elements in nflda_stripe_indices;

j = (SUi + nfl_first_stripe_index) % stripe_count;

idx = nflda_stripe_indices[j];

fh_count = number of elements in nfl_fh_list;
ds_count = number of elements in nflda_multipath_ds_list;

switch (fh_count) {
  case stripe_count:
    fh = nfl_fh_list[j];
    break;

  default:
    throw a fatal exception;
    break;
}

address_list = nflda_multipath_ds_list[idx];

Again, the client picks a data server from address_list and sends a READ or WRITE operation using the filehandle in fh.

Let's do an example! It's the same as the sparse packing one, except for the filehandle list 🍓

Suppose we have a device address with seven data servers, sorted into three equivalence (Section 13.5) classes. where A through G are network addresses:

{ A, B, C, D }, { E }, { F, G }

So:

nflda_multipath_ds_list<> = { A, B, C, D }, { E }, { F, G }

i.e.,

nflda_multipath_ds_list[0] = { A, B, C, D }

nflda_multipath_ds_list[1] = { E }

nflda_multipath_ds_list[2] = { F, G }

The striping index array is:

nflda_stripe_indices<> = { 2, 0, 1, 0 }

The client gets a layout with a device ID that maps to that device address. The initial index is:

nfl_first_stripe_index = 2,

and:

nfl_fh_list = { 0x67, 0x37, 0x87, 0x36 }.

The interesting cases for dense packing are SU1 and SU3, because each stripe unit points at the same data server list, yet each stripe unit MUST use a different filehandle 👀 If the client wants to write SU1, the valid { network address, filehandle } combos for SUi are determined by the first stripe index. Starting with nfl_first_stripe_index = 2:

j = (1 + 2) % 4 = 3

   idx = nflda_stripe_indices[j]

   = nflda_stripe_indices[3]

   = 0

So:

nflda_multipath_ds_list[0] = { A, B, C, D }

and:

nfl_fh_list[3] = { 0x36 }

which means the client can write SU1 to { 0x36, { A, B, C, D } } ✨

For SU3, j = (3 + 2) % 4 = 1, and nflda_stripe_indices[1] = 0. Then nflda_multipath_ds_list[0] = { A, B, C, D }, and nfl_fh_list[1] = 0x37. The client can therefore write SU3 to { 0x37, { A, B, C, D } } 🦋

Here are the destinations for the first 13 storage units:

                  +-----+------------+--------------+
                  | SUi | filehandle | data servers |
                  +-----+------------+--------------+
                  | 0   | 87         | E            |
                  | 1   | 36         | A,B,C,D      |
                  | 2   | 67         | F,G          |
                  | 3   | 37         | A,B,C,D      |
                  | 4   | 87         | E            |
                  | 5   | 36         | A,B,C,D      |
                  | 6   | 67         | F,G          |
                  | 7   | 37         | A,B,C,D      |
                  | 8   | 87         | E            |
                  | 9   | 36         | A,B,C,D      |
                  | 10  | 67         | F,G          |
                  | 11  | 37         | A,B,C,D      |
                  | 12  | 87         | E            |
                  +-----+------------+--------------+

13.4.4. Sparse and Dense Stripe Unit Packing 🧚

the flag NFL4_UFLG_DENSE of the nfl_util4 data type (the field nflh_util of nfsv4_1_file_layouthint4 and the field nfl_util of nfsv4_1_file_layout_ds_addr4) decides how data gets packed inside the data file on a data server 🎀 There are two packing types, sparse and dense, and the packing type sets the math used to map the client-visible file offset to the offset inside the data file on the data server.

If nfl_util & NFL4_UFLG_DENSE is zero, sparse packing is in use. That means the logical offsets of the file, as seen by a client sending READs and WRITEs straight to the metadata server, are the same offsets each data server uses when storing a stripe unit. For striping patterns with at least two stripe units, each data server file ends up sparse, or "holey" 🙈 For example, say there's a pattern with three stripe units, the stripe unit size is 4096 bytes, and there are three data servers in the pattern. Then the file on data server 1 has stripe units 0, 3, 6, 9, ... filled. Data server 2's file has stripe units 1, 4, 7, 10, ... filled, and data server 3's file has stripe units 2, 5, 8, 11, ... filled. The unfilled stripe units in each file are holes, so the files on each data server are sparse.

If sparse packing is in use and a client tries I/O on one of the holes, then an error MUST be returned by the data server. Using the example above, if data server 3 gets a READ or WRITE for block 4, it returns NFS4ERR_PNFS_IO_HOLE. Basically, data servers need to understand the striping pattern to support sparse packing 💀

If nfl_util & NFL4_UFLG_DENSE is one, dense packing is in use and the data server files have no holes ✨ Dense packing might get picked because the data server doesn't (efficiently) support holey files, or because the data server can't recognize read-ahead unless there are no holes. When dense packing is indicated in the layout, the data files get packed. Using the same striping pattern and stripe unit size as the sparse example, the matching dense packing example has all stripe units of all data files filled, like this:

• Logical stripe units 0, 3, 6, ... of the file live on stripe units 0, 1, 2, ... of the file on data server 1.
• Logical stripe units 1, 4, 7, ... of the file live on stripe units 0, 1, 2, ... of the file on data server 2.
• Logical stripe units 2, 5, 8, ... of the file live on stripe units 0, 1, 2, ... of the file on data server 3.

Since dense packing leaves no holes on the data servers, the pNFS client is allowed to write to any offset of any data file on any data server in the stripe. So the data servers don't need to know the file's striping pattern 😭

The calculation for the byte offset within the data file for dense data server layouts is:

stripe_width = stripe_unit_size * N;
   where N = number of elements in nflda_stripe_indices.

relative_offset = file_offset - nfl_pattern_offset;

data_file_offset = floor(relative_offset / stripe_width)
   * stripe_unit_size
   + relative_offset % stripe_unit_size

If dense packing is in use and a data server shows up more than once in a striping pattern, then the data server MUST use a different filehandle to tell one stripe unit from another 🔥 Let's say there are two data servers. Logical stripe units 0, 3, 6 are served by data server 1. Logical stripe units 1, 4, 7 are served by data server 2, and logical stripe units 2, 5, 8 are also served by data server 2. Unless data server 2 has two filehandles (each one pointing at a different data file), a write to logical stripe unit 1 overwrites the write to logical stripe unit 2, because both logical stripe units land in the same stripe unit (0) of data server 2 💖

13.5. Data Server Multipathing ✨

okay so the NFSv4.1 file layout can talk to multiple data server addresses, and bestie, this is so iconic 💅 It gives you bandwidth scaling through trunking (see Section 2.10.5) and extra high availability if a data server goes down. The best part is that the client can switch to another address without going back to the metadata server for a new layout. That might be a totally different data server that's exporting the same data stripe unit 🦋

So each element of nflda_multipath_ds_list holds an array of one or more data server network addresses. That array is a multipath_list4, and it's basically a list of data servers (each one identified by a network address), where a data server is allowed to show up more than once 🌸

The client is free to pick any of those network addresses to send its data server requests to 🫶 If some addresses are less optimal paths to the data than others, the MDS SHOULD NOT put those addresses in an element of nflda_multipath_ds_list. If you still want less optimal addresses around for failover, the RECOMMENDED way to offer them is in a replacement device-ID-to-device-address mapping, or a replacement device ID 💜

Now the "everything's on fire" scenario 💀 When a client finds that none of the data servers in an element of nflda_multipath_ds_list respond, it SHOULD send a GETDEVICEINFO to try to replace the existing device-ID-to-device-address mappings. If the MDS notices that all the data servers in that element are unavailable, the MDS SHOULD send a CB_NOTIFY_DEVICEID (but only if the client has said it wants device ID notifications for changed device IDs) so the mappings move over to the available data servers ✨ And if the device ID itself is getting replaced, the MDS SHOULD recall all layouts with that device ID. That forces the client to go get new layouts and device ID mappings through LAYOUTGET and GETDEVICEINFO 🔥

Generally, if two network addresses show up in the same element of nflda_multipath_ds_list, they're designating the same data server. Those two addresses will support client ID or session trunking (session trunking is the RECOMMENDED one) as defined in Section 2.10.5. They'll share the same server owner, or the same major ID of the server owner. Bestie, it isn't always required that trunking is in use for the two addresses to point at the same server. For example, the data could be read-only, with the data made up of exact replicas 🙈

13.6. Operations Sent to NFSv4.1 Data Servers 🎀

Clients that access data on an NFSv4.1 data server MUST send only the NULL procedure and COMPOUND procedures. Those COMPOUNDs can only use operations from two restricted subsets of the valid NFSv4.1 operations, so it's a strict guest list 💅 Clients MUST also use the filehandle specified by the layout when they access data on NFSv4.1 data servers 🧚

The first subset is the management operations. It's made up of BACKCHANNEL_CTL, BIND_CONN_TO_SESSION, CREATE_SESSION, DESTROY_CLIENTID, DESTROY_SESSION, EXCHANGE_ID, SECINFO_NO_NAME, SET_SSV, and SEQUENCE. The client can use these to set up and keep up the client IDs, sessions, and security contexts it needs to talk to the data server. From here on, these are called data-server housekeeping operations 🫶

The second subset is COMMIT, READ, WRITE, and PUTFH. These operations MUST be used with a current filehandle specified by the layout. With PUTFH, the new current filehandle MUST be one taken from the layout. These are called data-server I/O operations from now on 💖 As described in Section 12.5.1, a client MUST NOT send an I/O to a data server it doesn't hold a valid layout for, and the data server MUST reject that I/O. Big rule, no exceptions 😭

Now here's the twist. Unless the server has a concurrent non-data-server personality (aka it returns EXCHANGE_ID results with (EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_PNFS_MDS) or (EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_NON_PNFS), see Section 13.1), any attempt to use an operation against a data server that isn't in those two subsets MUST return NFS4ERR_NOTSUPP to the client. Not me saying no to the ops outside the guest list 🙈

When the server has both data-server and non-data-server personalities at once, each COMPOUND the client sends MUST be built so it fits exactly one of the two personalities. It MUST NOT contain operations aimed at a mix of them, and the server MUST enforce this. To get the constraints, operations inside a COMPOUND are sorted into these three classes 🍓

1. An operation that's ambiguous about which personality it belongs to. This includes all the data-server housekeeping operations. Also, if the server has assigned filehandles so the ones defined by the layout are the same as the ones the metadata server uses, every operation using those filehandles is in this class. There's one exception: if the operation uses a stateid that's incompatible with a data-server personality (for example a special stateid, or a stateid with a non-zero seqid field, see Section 13.9.1), the operation is in class 3, which we'll get to below. A COMPOUND holding multiple class 1 operations (and nothing from any other class) MAY be sent to a server with multiple concurrent data-server and non-data-server personalities 🦋
2. An operation that clearly belongs to the data-server personality. This includes data-server I/O operations where the filehandle can only be validly directed to the data-server personality 💜
3. An operation that clearly belongs to the non-data-server personality. This includes every COMPOUND operation that's neither data-server housekeeping nor data-server I/O, plus data-server I/O operations where the current fh (or the one about to become the current fh for PUTFH) is only valid on the metadata server, or where a stateid is used that's incompatible with the data server (a special stateid, or one with a non-zero seqid value) ✨

When a COMPOUND first runs an operation from class 3, it acts like a normal COMPOUND on any other server, and the data-server personality stops mattering. There are no special restrictions on the operations in that COMPOUND to limit them to ones meant for a data server. When a PUTFH happens, filehandles derived from the layout are not valid. If their format isn't normally acceptable, then NFS4ERR_BADHANDLE MUST result. In the same way, current filehandles for other operations don't accept filehandles derived from layouts, and they're not normally usable on the metadata server. Using them gets you NFS4ERR_STALE 💀

When a COMPOUND first runs an operation from class 2 (which means a PUTFH where the filehandle comes from a layout), the COMPOUND from then on gets interpreted with respect to the data-server personality. Operations outside the two classes we talked about MUST result in NFS4ERR_NOTSUPP. Filehandles get validated using the data server's rules, which can give you NFS4ERR_BADHANDLE and/or NFS4ERR_STALE even when the non-data-server personality wouldn't. Stateids have to obey the data server's rules too, so any use of a special stateid, or a stateid with a non-zero seqid value, must result in NFS4ERR_BAD_STATEID 🫶

Until the server first runs an operation from class 2 or class 3, the client MUST NOT depend on whether the operation gets executed by the data-server or the non-data-server personality. The server MUST pick one personality consistently for a given COMPOUND. The only transition allowed is a single one, when the first operation from class 2 or class 3 runs 🌈

Assigning filehandles so they work on both a data server and a metadata server adds a lot of complexity 😭 So where the same server can have both personalities, it is RECOMMENDED that the server gives both personalities separate, unique filehandles. That makes it totally unambiguous which server a given request is meant for 👑

GETATTR and SETATTR MUST be directed to the metadata server. For a SETATTR of the size attribute, the control protocol is responsible for passing size updates and truncations on to the data servers. For extending WRITEs to the data servers, the new size must be visible on the metadata server once a LAYOUTCOMMIT has completed (see Section 12.5.4.2). Section 13.10 describes how the client should handle data-server files that don't reflect the metadata server's size. Read that one, it's the plot twist 🍓🧚

13.7. COMMIT through Metadata Server 🎀

Okay so the file layout gives you TWO ways to handle COMMIT for data that got written through the data servers. The flag NFL4_UFLG_COMMIT_THRU_MDS lives in the field nfl_util of the file layout (data type nfsv4_1_file_layout4), and it's how the metadata server tells the client which method is the REQUIRED way to do COMMIT: either send it to the data server or send it to the metadata server 💅. These two approaches line up with two broad styles of implementation for a pNFS server that supports the file layout type.

• When the flag is FALSE, COMMIT operations MUST to be sent to the data server that the matching WRITE operations went to 🌸. This is kinda useful when file striping happens inside the pNFS server (instead of the file system), with each data server running its own file system.
• When the flag is TRUE, COMMIT operations MUST be sent to the metadata server instead of the individual data servers 🦋. This is handy when striping happens inside the clustered file system that sits behind the pNFS server. If you COMMIT to each data server separately there, you could end up with repeated writes of metadata blocks, and that's a hit to write performance 😭. One COMMIT to the metadata server is way more efficient, as long as there's a clustered file system that can actually pull off a coordinated COMMIT.

If nfl_util & NFL4_UFLG_COMMIT_THRU_MDS is TRUE, then to keep the current NFSv4.1 commit and recovery model intact, the data servers MUST return a common writeverf verifier in all WRITE responses for a given file layout, and the metadata server's COMMIT implementation must return that same writeverf. The value of the writeverf verifier MUST be changed at the metadata server or at any data server referenced in the layout, whenever there's a server event that could possibly lead to loss of uncommitted data 💀. The verifier can be scoped to a single file or to the entire pNFS server. Keeping it at the file level is harder for the server, but the payoff is that only events affecting a given file trigger recovery action 🧚.

Note that if the layout specified dense packing, then the offset used in a COMMIT to the MDS may differ from the offset used in a COMMIT to the data server 👑.

The single COMMIT to the metadata server returns a verifier, and the client should compare it to all the verifiers from the WRITEs and fail the COMMIT if there are any mismatched verifiers 🙈. If the COMMIT to the metadata server fails, the client should re-send WRITEs for all the modified data in the file. The client should treat modified data with a mismatched verifier as a WRITE failure and try to recover by resending the WRITEs to the original data server, or by using another path to that data if the layout has not been recalled. Alternatively, the client can obtain a new layout or rewrite the data directly to the metadata server ✨. If nfl_util & NFL4_UFLG_COMMIT_THRU_MDS is FALSE, sending a COMMIT to the metadata server might have no effect. And if nfl_util & NFL4_UFLG_COMMIT_THRU_MDS is FALSE, a COMMIT sent to the metadata server should only be used to commit data that was written to the metadata server. See Section 12.7.6 for recovery options 🫶.

---

13.8. The Layout Iomode 🌈

The metadata server doesn't need to use the layout iomode when it's servicing NFSv4.1 file-based layouts, although in some situations it can be useful 🍓. For example, if the server implementation supports reading from read-only replicas or mirrors, it'd be useful for the server to hand back a layout that lets the client read from them 👀. So the client SHOULD set the iomode based on whether it intends to read or write the data. The client may default to an iomode of LAYOUTIOMODE4_RW 💜.

The data servers don't need to check the iomode when clients do I/O. However, the data servers SHOULD still validate that the client holds a valid layout, and return an error if the client doesn't 🔥.

---

13.9. Metadata and Data Server State Coordination 💖

13.9.1. Global Stateid Requirements ⭐

When the client sends I/O to a data server, the stateid it uses MUST NOT be a layout stateid, the kind returned by LAYOUTGET or sent by CB_LAYOUTRECALL 🙅‍♀️. The permitted stateids are based on one of these: an OPEN stateid (the stateid field of data type OPEN4resok, as returned by OPEN), a delegation stateid (the stateid field of data types open_read_delegation4 and open_write_delegation4, as returned by OPEN or WANT_DELEGATION, or as sent by CB_PUSH_DELEG), or a stateid returned by the LOCK or LOCKU operations 🦋. The stateid sent to the data server MUST be sent with the seqid set to zero, which says "use the most current version of that stateid" rather than pointing at a specific non-zero seqid value 🎀. In no case is the use of special stateid values allowed 🚫.

The stateid used for I/O MUST have the same effect, and be subject to the same validation on a data server, as it would if the I/O were being done on the metadata server itself without pNFS 💅. What that means is stateids are globally valid on both the metadata and data servers. To make that work, the metadata server has to propagate changes in LOCK and OPEN state to the data servers, so the data servers can validate I/O accesses 🫶. That's covered more in Section 13.9.2. Depending on when stateids get propagated, having a valid stateid on the data server can act as proof of a valid layout 🌸.

Clients doing I/O need to pick an appropriate stateid based on the locks they hold (including opens and delegations) and on the various types of state-owners sending the I/O requests 💫. The rules for doing that when talking to data servers are somewhat different from the ones in Section 8.2.5, which apply when accessing metadata servers 🌷.

The following rules, applied in order of decreasing priority, govern picking the right stateid 👑:

• If the client holds a delegation for the file in question, the delegation stateid should be used.
• Otherwise, there must be an OPEN stateid for the current open-owner, and that OPEN stateid for the open file in question is used, unless mandatory locking prevents that. See below.
• If the data server had previously responded with NFS4ERR_LOCKED to use of the OPEN stateid, then the client should use the byte-range lock stateid whenever one exists for that open file with the current lock-owner.
• Special stateids should never be used. If they are used, the data server MUST reject the I/O with an NFS4ERR_BAD_STATEID error 🙅‍♀️.

13.9.2. Data Server State Propagation 🧚

The metadata server handles byte-range lock and open-mode state changes, as well as ACLs, and it might not be co-located with the data servers where I/O accesses get validated. So the server implementation MUST take care of propagating changes to this state to the data servers 💖. Once propagation to the data servers is complete, the full effect of those changes MUST be in effect at the data servers. However, some state changes don't need to be propagated immediately, although all changes SHOULD be propagated promptly 🌈. These state propagations affect how the control protocol gets designed, even though the control protocol is outside the scope of this specification 🔥. Immediate propagation means synchronous propagation of state from the metadata server to the data server(s). The propagation has to be complete before returning to the client ⏳.

13.9.2.1. Lock State Propagation 🔒

If the pNFS server supports mandatory byte-range locking, any mandatory byte-range locks on a file MUST be made effective at the data servers before the request that establishes them returns to the caller. The effect MUST be the same as if the mandatory byte-range lock state were synchronously propagated to the data servers, even though the details of the control protocol may skip the actual transfer of the state in some circumstances ✨.

On the other hand, advisory byte-range lock state isn't used for checking I/O accesses at the data servers, so there's no semantic reason to propagate it there 🦋. Updates to advisory locks neither grant nor remove privileges, so these changes don't need to be propagated immediately, and may not even need to be propagated promptly 🍓. Advisory lock updates only need to be propagated when a data server has to resolve a question about a stateid. Honestly, if byte-range locking is not mandatory (that is, it's advisory), clients are advised to avoid using byte-range lock-based stateids for I/O 💅. The stateids returned by OPEN are enough, and they skip the overhead of this kind of state propagation 🌸.

If a client gets an NFS4ERR_LOCKED error back from a data server, that's a sign that mandatory byte-range locking is in force 🚨. To recover, the client gets a byte-range lock covering the affected range and re-sends the I/O with the stateid of that byte-range lock 🫶.

13.9.2.2. Open and Deny Mode Validation 🔐

Open and deny mode validation MUST be performed against the open and deny mode(s) held by the data servers. When access is reduced or a deny mode gets more restrictive (because of CLOSE or OPEN_DOWNGRADE), the data server MUST prevent any I/Os that would be denied if they were performed on the metadata server 🙅‍♀️. When access is expanded, the data server MUST make sure that no requests get rejected because of open or deny issues that no longer apply, given the earlier relaxation 🥰.

13.9.2.3. File Attributes ✨

okay bestie, this one is about file attributes 💅 SETATTR can change state that shows up on both the metadata server AND the data servers (like the size 📏), so you gotta make sure everything lines up across all the data servers, especially when you truncate or grow a file 🫶.

As we already covered, LAYOUTCOMMIT is how the metadata stays in sync with what happened on the data servers. For the NFSv4.1-based data storage protocol, stuff like the size attribute and the mtime/change/atime settings needs to be re-synced 🔄. Check out Section 12.5.4 for the full tea on LAYOUTCOMMIT and attribute synchronization. It's also worth knowing that with an NFSv4.1-based layout type, you can sync this state before LAYOUTCOMMIT even happens 🧚. For example, the control protocol can be used to query the attributes sitting on the data servers.

Any changes to file attributes that control authorization or access, the kind reflected by ACCESS calls or READs and WRITEs on the metadata server, MUST be propagated to the data servers so they can enforce them on READ and WRITE I/O calls 🔐. And if the metadata server changes make access more restrictive for any user, those changes MUST be propagated to the data servers synchronously ⚡. No waiting around!

The OPEN operation (Section 18.16.4) doesn't require that I/O on an open file use the same credentials as the OPEN itself (unless EXCHGID4_FLAG_BIND_PRINC_STATEID is set when EXCHANGE_ID creates the client ID) 🙈. So the server's READ and WRITE operations have to do their own appropriate access checking. ACL changes need new access checking by READ and WRITE on the server too 💀. Propagating access-right changes caused by ACL changes may be asynchronous, but only if the server can tell that the updated ACL isn't more restrictive for any user in the old ACL. ACL updates don't happen super often, though, so it's suggested that all changes just get propagated synchronously 🍓.

---

13.10. Data Server Component File Size 📏

okay, this is a potential problem 😭. A component data file on one data server can grow past EOF, and this happens with both dense and sparse layouts. Picture this: a client creates a new file (size == 0), writes to byte 131072, then seeks back to the beginning and reads byte 100. The client should get zeroes back from that READ 🤍. But if the striping pattern sends the READ to a different data server than the one that got the original WRITE, that server might think the file is still 0 bytes long 🙃. In that case its READ response has zero bytes and an EOF indication. A data server can only return zeroes if it knows the file has been extended, and telling every data server about the new size right away would be super costly 💸.

So the client that started extending the file's size MUST be ready to handle these EOF conditions 💪. When the offset in the READ arguments is less than the client's view of the file size, and the READ response says EOF and/or has fewer bytes than requested, the client treats that as a hole in the file and the NFS client substitutes zeroes for the data 🌸.

The NFSv4.1 protocol only gives you close-to-open file data cache semantics 🎀. That means when a file is closed, all the modified data gets written to the server. When a later OPEN of the file happens, the change attribute gets checked against a cached value. For the case above, that means a LAYOUTCOMMIT happens at close (along with the data WRITEs) and updates the file's size and change attribute ✨. Any access from another client after that point will return the right size 👑.

---

13.11. Layout Revocation and Fencing 🚧

as Section 12.7 explains, the layout-type-specific storage protocol is in charge of dealing with I/Os that started before lease expiration and kept going through it 🕰️. The LAYOUT4_NFSV4_1_FILES layout type can stop all I/Os to data servers from running after lease expiration (this is called "fencing"), and it doesn't need a precise client lease timer or data servers keeping lease timers either 🧠. The LAYOUT4_NFSV4_1_FILES pNFS server can revoke individual layouts, so it can fence I/O on a per-file basis 🎯.

Besides lease expiration, a layout can also get revoked when the client fails to respond to a CB_LAYOUTRECALL, when the metadata server restarts, or because of administrative intervention 🛠️. No matter the reason, once a client's layout is revoked, the pNFS server MUST stop the client from sending I/O for the affected file from and to all data servers. In other words, it MUST fence the client from the affected file on the data servers 🚫.

Here's how fencing works 🔍. As Section 13.1 describes, in COMPOUND procedure requests to the data server, the data filehandle given by the PUTFH operation and the stateid in the READ or WRITE operation make sure the client has a valid layout for the I/O. If it doesn't, the I/O gets rejected with NFS4ERR_PNFS_NO_LAYOUT 🙅. The server can just check the stateid and, on top of that, make the data filehandle stale if the layout specified a data filehandle that's different from the metadata server's filehandle for the file (see the nfl_fh_list description in Section 13.3) 🫠.

Before the metadata server does anything to revoke layout state from a previous instance, it has to make sure all of that layout state is invalidated at the data servers 🧹. This has a few implications:

• The metadata server must not restripe a file until it has contacted all of the data servers to invalidate the layouts from the previous instance.
• The metadata server must not give out mandatory locks that conflict with layouts from the previous instance without either doing a specific layout invalidation (as it would have to do anyway) or doing a global data server invalidation.

---

13.12. Security Considerations for the File Layout Type 🔐

this section is SO important, bestie 😤. The NFSv4.1 file layout type MUST follow the security considerations in Section 12.9. NFSv4.1 data servers MUST make all of the required access checks on each READ or WRITE I/O, as the NFSv4.1 protocol says 🛡️. If the metadata server would deny a READ or WRITE on a file because of its ACL, mode attribute, open access mode, open deny mode, mandatory byte-range lock state, or any other attributes and state, the data server MUST also deny that READ or WRITE 🚷. This affects the control protocol and how state gets passed from the metadata server to the data servers. See Section 13.9.2 for more details 🎀.

The authentication, integrity, and privacy methods for data servers based on the LAYOUT4_NFSV4_1_FILES layout type are the same ones metadata servers use 🫶. Metadata and data servers use ONC RPC security flavors to authenticate, and they use SECINFO and SECINFO_NO_NAME to negotiate the security mechanism and services. So with the LAYOUT4_NFSV4_1_FILES layout type, the impact on the RPC-based security model from pNFS (as hinted at in Sections 1.7.1 and 1.7.2.2) is zero 🌈.

For a given file object, a metadata server MAY require different security parameters (secinfo4 value) than the data server. For a given file object with multiple data servers, the secinfo4 value SHOULD be the same across all data servers 🤝. If the secinfo4 values across a metadata server and its data servers differ for a specific file, the mapping of the principal to the server's internal user identifier MUST be the same. That's how the access-control checks based on ACL, mode, open and deny mode, and mandatory locking stay consistent across the pNFS server 🔥.

If an NFSv4.1 implementation supports pNFS and NFSv4.1 file layouts, then the implementation MUST support the SECINFO_NO_NAME operation on both the metadata and data servers 💖.

---

14. Internationalization 🌍

okay so the big I18N issue for NFSv4.1 is file names and other strings used in the protocol 🗂️. The string representation has to give clients who speak all kinds of languages reasonable name and string access. The UTF-8 encoding of the UCS (Universal Multiple-Octet Coded Character Set), as defined by ISO10646 [21], does this and follows the policy in "IETF Policy on Character Sets and Languages", RFC 2277 [22] 🦋.

RFC 3454 [19], otherwise know as "stringprep", is a framework for using Unicode/UTF-8 in networking protocols so that "to increase the likelihood that string input and string comparison work in ways that make sense for typical users throughout the world" 🌏. A protocol has to define a profile of stringprep "in order to fully specify the processing options". The rest of this section defines the NFSv4.1 stringprep profiles, and a lot of the terminology comes from stringprep 📚.

NFSv4.1 has three UTF-8 string types: utf8str_cs, utf8str_cis, and utf8str_mixed 🍭. Each one gets its own profile. Each profile defines the following, as stringprep requires:

• The intended applicability of the profile.
• The character repertoire that is the input and output to stringprep (which is Unicode 3.2 for the referenced version of stringprep). However, NFSv4.1 implementations are not limited to 3.2.
• The mapping tables from stringprep used (as described in Section 3 of stringprep).
• Any additional mapping tables specific to the profile.
• The Unicode normalization used, if any (as described in Section 4 of stringprep).
• The tables from the stringprep listing of characters that are prohibited as output (as described in Section 5 of stringprep).
• The bidirectional string testing used, if any (as described in Section 6 of stringprep).
• Any additional characters that are prohibited as output specific to the profile.

Stringprep talks about Unicode characters, but NFSv4.1 uses UTF-8 characters 🤓. UTF-8 maps one-to-one to Unicode, so whenever this document says Unicode, you should read UTF-8 💜.

Much of the text for the profiles comes from RFC 3491 [23] 🎀.

---

14.1. Stringprep Profile for the utf8str_cs Type 💅

every use of the utf8str_cs type definition in the NFSv4 protocol specification follows the profile named nfs4_cs_prep ✨.

14.1.1. Intended Applicability of the nfs4_cs_prep Profile

okay so utf8str_cs is a case-sensitive string of UTF-8 characters 💖 Its main gig in NFSv4.1 is naming components and pathnames, and those end up stored on the server's file system 🗂️ Here's the tea: two different UTF-8 strings can end up looking the same after going through the utf8str_cs profile 😭 So if they're two names sitting inside one directory, the NFSv4.1 server has to pick one of these options:

• disallow creating the second name if its post-processed form collides with that of an existing name, or
• allow creating the second name, but arrange things so that after post-processing, it's different from the post-processed form of the first name.

14.1.2. Character Repertoire of nfs4_cs_prep

the nfs4_cs_prep profile uses Unicode 3.2, as defined in stringprep's Appendix A.1 📚 BUT, and this is important, NFSv4.1 implementations are not limited to 3.2 🌈

14.1.3. Mapping Used by nfs4_cs_prep

the nfs4_cs_prep profile specifies mapping using these tables from stringprep:

Table B.1

okay, Table B.2 is normally NOT part of nfs4_cs_prep, because it's mainly for case-insensitive comparisons 🙈 however, if the NFSv4.1 file server supports the case_insensitive file system attribute, and case_insensitive is TRUE, then the NFSv4.1 server MUST use Table B.2 (in addition to Table B1) when processing utf8str_cs strings, and the NFSv4.1 client MUST assume Table B.2 (in addition to Table B.1) is being used 🔥

and if the case_preserving attribute is present and set to FALSE, the NFSv4.1 server MUST use Table B.2 to map case when processing utf8str_cs strings 💅 whether the server maps lower to upper or upper to lower is an implementation dependency, so it's up to the implementation 🦋

14.1.4. Normalization used by nfs4_cs_prep

the nfs4_cs_prep profile does not specify a normalization form ✨ a later revision of this spec might pick one, so the server and client can expect to get unnormalized characters inside protocol requests and responses 🫶 if the operating environment needs normalization, then the implementation must normalize utf8str_cs strings within the protocol before presenting the info to an application (at the client) or the local file system (at the server) 👑

14.1.5. Prohibited Output for nfs4_cs_prep

the nfs4_cs_prep profile RECOMMENDS prohibiting the use of these tables from stringprep:

Table C.5

Table C.6

14.1.6. Bidirectional Output for nfs4_cs_prep

the nfs4_cs_prep profile does not specify any checking of bidirectional strings 🧚

14.2. Stringprep Profile for the utf8str_cis Type

every use of the utf8str_cis type definition in the NFSv4.1 protocol spec follows the profile named nfs4_cis_prep 💜

14.2.1. Intended Applicability of the nfs4_cis_prep Profile

the utf8str_cis type is a case-insensitive string of UTF-8 characters 🍓 its main use in NFSv4.1 is naming NFS servers 🌸

14.2.2. Character Repertoire of nfs4_cis_prep

the nfs4_cis_prep profile uses Unicode 3.2, as defined in stringprep's Appendix A.1 📖 however, NFSv4.1 implementations are not limited to 3.2 ✨

14.2.3. Mapping Used by nfs4_cis_prep

the nfs4_cis_prep profile specifies mapping using these tables from stringprep:

Table B.1

Table B.2

14.2.4. Normalization Used by nfs4_cis_prep

the nfs4_cis_prep profile specifies using Unicode normalization form KC, as described in stringprep 🌈

14.2.5. Prohibited Output for nfs4_cis_prep

the nfs4_cis_prep profile specifies prohibiting the use of these tables from stringprep:

Table C.1.2

Table C.2.2

Table C.3

Table C.4

Table C.5

Table C.6

Table C.7

Table C.8

Table C.9

14.2.6. Bidirectional Output for nfs4_cis_prep

the nfs4_cis_prep profile specifies checking bidirectional strings as described in stringprep's Section 6 🦋

14.3. Stringprep Profile for the utf8str_mixed Type

every use of the utf8str_mixed type definition in the NFSv4.1 protocol spec follows the profile named nfs4_mixed_prep 🎀

14.3.1. Intended Applicability of the nfs4_mixed_prep Profile

the utf8str_mixed type is a string of UTF-8 characters with a prefix that is case sensitive, a separator equal to '@', and a suffix that is a fully qualified domain name 🫶 its main use in NFSv4.1 is naming principals identified in an Access Control Entry 🔐

14.3.2. Character Repertoire of nfs4_mixed_prep

the nfs4_mixed_prep profile uses Unicode 3.2, as defined in stringprep's Appendix A.1 📚 however, NFSv4.1 implementations are not limited to 3.2 🌸

14.3.3. Mapping Used by nfs4_cis_prep

for the prefix and the separator of a utf8str_mixed string, the nfs4_mixed_prep profile specifies mapping using this table from stringprep:

Table B.1

for the suffix of a utf8str_mixed string, the nfs4_mixed_prep profile specifies mapping using these tables from stringprep:

Table B.1

Table B.2

14.3.4. Normalization Used by nfs4_mixed_prep

the nfs4_mixed_prep profile specifies using Unicode normalization form KC, as described in stringprep 💅

14.3.5. Prohibited Output for nfs4_mixed_prep

the nfs4_mixed_prep profile specifies prohibiting the use of these tables from stringprep:

Table C.1.2

Table C.2.2

Table C.3

Table C.4

Table C.5

Table C.6

Table C.7

Table C.8

Table C.9

14.3.6. Bidirectional Output for nfs4_mixed_prep

the nfs4_mixed_prep profile specifies checking bidirectional strings as described in stringprep's Section 6 🌈

14.4. UTF-8 Capabilities

const FSCHARSET_CAP4_CONTAINS_NON_UTF8  = 0x1;
const FSCHARSET_CAP4_ALLOWS_ONLY_UTF8   = 0x2;

typedef uint32_t        fs_charset_cap4;

okay, because some operating environments and file systems don't enforce character set encodings, NFSv4.1 supports the fs_charset_cap attribute (Section 5.8.2.11) to tell the client about a file system's UTF-8 capabilities 🧁 the attribute is an integer holding a pair of flags 🎀

the first flag is FSCHARSET_CAP4_CONTAINS_NON_UTF8. if it's set to one, the client is told the file system contains non-UTF-8 characters, and the server will not convert non-UTF characters to UTF-8 if the client reads a symlink or directory. operations with component names or pathnames in the arguments won't convert the strings to UTF-8 either 🙈

the second flag is FSCHARSET_CAP4_ALLOWS_ONLY_UTF8. if it's set to one, the server will accept (and generate) only UTF-8 characters on the file system 💜 if FSCHARSET_CAP4_ALLOWS_ONLY_UTF8 is set to one, then FSCHARSET_CAP4_CONTAINS_NON_UTF8 MUST be set to zero. FSCHARSET_CAP4_ALLOWS_ONLY_UTF8 SHOULD always be set to one 👑

14.5. UTF-8 Related Errors

where the client sends an invalid UTF-8 string, the server should return NFS4ERR_INVAL (see Table 5) 🚫 this includes cases where inappropriate prefixes are detected, and where the count includes trailing bytes that don't make up a full UCS character 😭

where the client-supplied string is valid UTF-8 but contains characters the server doesn't support as a value for that string (for example, names containing characters outside of Unicode plane 0 on file systems that fail to support them even though they're in the Unicode standard), the server should return NFS4ERR_BADCHAR 💀

where a UTF-8 string is used as a file name, and the file system (even while supporting all of the characters in the name) doesn't allow that particular name to be used, the server should return the error NFS4ERR_BADNAME (Table 5) 🙅‍♀️ this covers situations where the server file system imposes a normalization constraint on name strings, and also things like file system prohibitions of "." and ".." as file names for certain operations, plus other constraints like that ✨

15. Error Values ✨

NFS error numbers are assigned to failed operations within a Compound (COMPOUND or CB_COMPOUND) request 🎀 a Compound request holds a bunch of NFS operations, and their results get encoded in sequence in a Compound reply 🧚 if an operation succeeds, its result is an NFS4_OK status followed by the encoded results of that operation 💖 but if an NFS operation fails, an error status goes into the reply and the Compound request gets terminated 😭

15.1. Error Definitions 💖

okay bestie, here's the full list of every NFSv4.1 error, with where to find each one explained 🧚 The table is in a code block so it stays exactly as written:

                        Protocol Error Definitions

    +-----------------------------------+--------+-------------------+
    | Error                             | Number | Description       |
    +-----------------------------------+--------+-------------------+
    | NFS4_OK                           | 0      | Section 15.1.3.1  |
    | NFS4ERR_ACCESS                    | 13     | Section 15.1.6.1  |
    | NFS4ERR_ATTRNOTSUPP               | 10032  | Section 15.1.15.1 |
    | NFS4ERR_ADMIN_REVOKED             | 10047  | Section 15.1.5.1  |
    | NFS4ERR_BACK_CHAN_BUSY            | 10057  | Section 15.1.12.1 |
    | NFS4ERR_BADCHAR                   | 10040  | Section 15.1.7.1  |
    | NFS4ERR_BADHANDLE                 | 10001  | Section 15.1.2.1  |
    | NFS4ERR_BADIOMODE                 | 10049  | Section 15.1.10.1 |
    | NFS4ERR_BADLAYOUT                 | 10050  | Section 15.1.10.2 |
    | NFS4ERR_BADNAME                   | 10041  | Section 15.1.7.2  |
    | NFS4ERR_BADOWNER                  | 10039  | Section 15.1.15.2 |
    | NFS4ERR_BADSESSION                | 10052  | Section 15.1.11.1 |
    | NFS4ERR_BADSLOT                   | 10053  | Section 15.1.11.2 |
    | NFS4ERR_BADTYPE                   | 10007  | Section 15.1.4.1  |
    | NFS4ERR_BADXDR                    | 10036  | Section 15.1.1.1  |
    | NFS4ERR_BAD_COOKIE                | 10003  | Section 15.1.1.2  |
    | NFS4ERR_BAD_HIGH_SLOT             | 10077  | Section 15.1.11.3 |
    | NFS4ERR_BAD_RANGE                 | 10042  | Section 15.1.8.1  |
    | NFS4ERR_BAD_SEQID                 | 10026  | Section 15.1.16.1 |
    | NFS4ERR_BAD_SESSION_DIGEST        | 10051  | Section 15.1.12.2 |
    | NFS4ERR_BAD_STATEID               | 10025  | Section 15.1.5.2  |
    | NFS4ERR_CB_PATH_DOWN              | 10048  | Section 15.1.11.4 |
    | NFS4ERR_CLID_INUSE                | 10017  | Section 15.1.13.2 |
    | NFS4ERR_CLIENTID_BUSY             | 10074  | Section 15.1.13.1 |
    | NFS4ERR_COMPLETE_ALREADY          | 10054  | Section 15.1.9.1  |
    | NFS4ERR_CONN_NOT_BOUND_TO_SESSION | 10055  | Section 15.1.11.6 |
    | NFS4ERR_DEADLOCK                  | 10045  | Section 15.1.8.2  |
    | NFS4ERR_DEADSESSION               | 10078  | Section 15.1.11.5 |
    | NFS4ERR_DELAY                     | 10008  | Section 15.1.1.3  |
    | NFS4ERR_DELEG_ALREADY_WANTED      | 10056  | Section 15.1.14.1 |
    | NFS4ERR_DELEG_REVOKED             | 10087  | Section 15.1.5.3  |
    | NFS4ERR_DENIED                    | 10010  | Section 15.1.8.3  |
    | NFS4ERR_DIRDELEG_UNAVAIL          | 10084  | Section 15.1.14.2 |
    | NFS4ERR_DQUOT                     | 69     | Section 15.1.4.2  |
    | NFS4ERR_ENCR_ALG_UNSUPP           | 10079  | Section 15.1.13.3 |
    | NFS4ERR_EXIST                     | 17     | Section 15.1.4.3  |
    | NFS4ERR_EXPIRED                   | 10011  | Section 15.1.5.4  |
    | NFS4ERR_FBIG                      | 27     | Section 15.1.4.4  |
    | NFS4ERR_FHEXPIRED                 | 10014  | Section 15.1.2.2  |
    | NFS4ERR_FILE_OPEN                 | 10046  | Section 15.1.4.5  |
    | NFS4ERR_GRACE                     | 10013  | Section 15.1.9.2  |
    | NFS4ERR_HASH_ALG_UNSUPP           | 10072  | Section 15.1.13.4 |
    | NFS4ERR_INVAL                     | 22     | Section 15.1.1.4  |
    | NFS4ERR_IO                        | 5      | Section 15.1.4.6  |
    | NFS4ERR_ISDIR                     | 21     | Section 15.1.2.3  |
    | NFS4ERR_LAYOUTTRYLATER            | 10058  | Section 15.1.10.3 |
    | NFS4ERR_LAYOUTUNAVAILABLE         | 10059  | Section 15.1.10.4 |
    | NFS4ERR_LEASE_MOVED               | 10031  | Section 15.1.16.2 |
    | NFS4ERR_LOCKED                    | 10012  | Section 15.1.8.4  |
    | NFS4ERR_LOCKS_HELD                | 10037  | Section 15.1.8.5  |
    | NFS4ERR_LOCK_NOTSUPP              | 10043  | Section 15.1.8.6  |
    | NFS4ERR_LOCK_RANGE                | 10028  | Section 15.1.8.7  |
    | NFS4ERR_MINOR_VERS_MISMATCH       | 10021  | Section 15.1.3.2  |
    | NFS4ERR_MLINK                     | 31     | Section 15.1.4.7  |
    | NFS4ERR_MOVED                     | 10019  | Section 15.1.2.4  |
    | NFS4ERR_NAMETOOLONG               | 63     | Section 15.1.7.3  |
    | NFS4ERR_NOENT                     | 2      | Section 15.1.4.8  |
    | NFS4ERR_NOFILEHANDLE              | 10020  | Section 15.1.2.5  |
    | NFS4ERR_NOMATCHING_LAYOUT         | 10060  | Section 15.1.10.5 |
    | NFS4ERR_NOSPC                     | 28     | Section 15.1.4.9  |
    | NFS4ERR_NOTDIR                    | 20     | Section 15.1.2.6  |
    | NFS4ERR_NOTEMPTY                  | 66     | Section 15.1.4.10 |
    | NFS4ERR_NOTSUPP                   | 10004  | Section 15.1.1.5  |
    | NFS4ERR_NOT_ONLY_OP               | 10081  | Section 15.1.3.3  |
    | NFS4ERR_NOT_SAME                  | 10027  | Section 15.1.15.3 |
    | NFS4ERR_NO_GRACE                  | 10033  | Section 15.1.9.3  |
    | NFS4ERR_NXIO                      | 6      | Section 15.1.16.3 |
    | NFS4ERR_OLD_STATEID               | 10024  | Section 15.1.5.5  |
    | NFS4ERR_OPENMODE                  | 10038  | Section 15.1.8.8  |
    | NFS4ERR_OP_ILLEGAL                | 10044  | Section 15.1.3.4  |
    | NFS4ERR_OP_NOT_IN_SESSION         | 10071  | Section 15.1.3.5  |
    | NFS4ERR_PERM                      | 1      | Section 15.1.6.2  |
    | NFS4ERR_PNFS_IO_HOLE              | 10075  | Section 15.1.10.6 |
    | NFS4ERR_PNFS_NO_LAYOUT            | 10080  | Section 15.1.10.7 |
    | NFS4ERR_RECALLCONFLICT            | 10061  | Section 15.1.14.3 |
    | NFS4ERR_RECLAIM_BAD               | 10034  | Section 15.1.9.4  |
    | NFS4ERR_RECLAIM_CONFLICT          | 10035  | Section 15.1.9.5  |
    | NFS4ERR_REJECT_DELEG              | 10085  | Section 15.1.14.4 |
    | NFS4ERR_REP_TOO_BIG               | 10066  | Section 15.1.3.6  |
    | NFS4ERR_REP_TOO_BIG_TO_CACHE      | 10067  | Section 15.1.3.7  |
    | NFS4ERR_REQ_TOO_BIG               | 10065  | Section 15.1.3.8  |
    | NFS4ERR_RESTOREFH                 | 10030  | Section 15.1.16.4 |
    | NFS4ERR_RETRY_UNCACHED_REP        | 10068  | Section 15.1.3.9  |
    | NFS4ERR_RETURNCONFLICT            | 10086  | Section 15.1.10.8 |
    | NFS4ERR_ROFS                      | 30     | Section 15.1.4.11 |
    | NFS4ERR_SAME                      | 10009  | Section 15.1.15.4 |
    | NFS4ERR_SHARE_DENIED              | 10015  | Section 15.1.8.9  |
    | NFS4ERR_SEQUENCE_POS              | 10064  | Section 15.1.3.10 |
    | NFS4ERR_SEQ_FALSE_RETRY           | 10076  | Section 15.1.11.7 |
    | NFS4ERR_SEQ_MISORDERED            | 10063  | Section 15.1.11.8 |
    | NFS4ERR_SERVERFAULT               | 10006  | Section 15.1.1.6  |
    | NFS4ERR_STALE                     | 70     | Section 15.1.2.7  |
    | NFS4ERR_STALE_CLIENTID            | 10022  | Section 15.1.13.5 |
    | NFS4ERR_STALE_STATEID             | 10023  | Section 15.1.16.5 |
    | NFS4ERR_SYMLINK                   | 10029  | Section 15.1.2.8  |
    | NFS4ERR_TOOSMALL                  | 10005  | Section 15.1.1.7  |
    | NFS4ERR_TOO_MANY_OPS              | 10070  | Section 15.1.3.11 |
    | NFS4ERR_UNKNOWN_LAYOUTTYPE        | 10062  | Section 15.1.10.9 |
    | NFS4ERR_UNSAFE_COMPOUND           | 10069  | Section 15.1.3.12 |
    | NFS4ERR_WRONGSEC                  | 10016  | Section 15.1.6.3  |
    | NFS4ERR_WRONG_CRED                | 10082  | Section 15.1.6.4  |
    | NFS4ERR_WRONG_TYPE                | 10083  | Section 15.1.2.9  |
    | NFS4ERR_XDEV                      | 18     | Section 15.1.4.12 |
    +-----------------------------------+--------+-------------------+

Table 5 🎀

15.1.1. General Errors 🌸

these errors are the all-purpose ones, the ones that can show up across a whole bunch of different situations 💅

15.1.1.1. NFS4ERR_BADXDR (Error Code 10036) 😭

the arguments for this operation don't match what the XDR definition says they should be 🙈 this also covers the case where the request ends before all the arguments have been seen. note that this error applies when fixed enumerations (these include booleans) have a value in the input stream that isn't valid for the enum. a replier may pre-parse all operations for a Compound procedure before doing any operation execution and return RPC-level XDR errors in that case 🧚

15.1.1.2. NFS4ERR_BAD_COOKIE (Error Code 10003) 🍓

this one is used for operations that give back a set of information indexed by some quantity. that quantity can be something the client provided, or a cookie the server sent back on an earlier call. if the value can't be used for what it was meant for, this is the error you get ✨

15.1.1.3. NFS4ERR_DELAY (Error Code 10008) 🦋

okay so, for a bunch of possible reasons the replier just couldn't get this operation done in what it considered a reasonable amount of time. the client should wait and then try the request again with a new slot and sequence value 🫶

some scenarios that might lead to this situation:

• a server that supports hierarchical storage gets a request to process a file that had been migrated.
• an operation needs a delegation recall to go through first, and waiting on that recall makes handling this request in a timely way impossible.

so the whole point of NFS4ERR_DELAY is that it lets those preparatory operations go ahead without tying up client resources like a session slot 💜 after waiting a while, the client can re-send the operation in question, but not with the same slot ID and sequence ID. one or both MUST be different on the re-send 🔥

ngl this is the part to remember: without the ability to return NFS4ERR_DELAY, and without the client being willing to re-send when it gets one, deadlock might happen 💀 for example, if a recall is done, and the delegation return (or the operations prepping for it) gets held up by other operations that need that delegation returned, session slots might not be available. and then everyone is stuck waiting on everyone else. the result could be deadlock 😭

15.1.1.4. NFS4ERR_INVAL (Error Code 22) 🌈

the arguments for this operation aren't valid for some reason, even though they match the XDR definition for the request. the shape is right, the content is wrong 🙃

15.1.1.5. NFS4ERR_NOTSUPP (Error Code 10004) 🚫

operation not supported, either because the operation is OPTIONAL and this server doesn't support it, or because the operation MUST NOT be implemented in the current minor version 💅

15.1.1.6. NFS4ERR_SERVERFAULT (Error Code 10006) 💀

an error happened on the server that doesn't map to any of the specific legal NFSv4.1 protocol error values. the client should translate this into an appropriate error. UNIX clients may choose to translate this to EIO 🧚

15.1.1.7. NFS4ERR_TOOSMALL (Error Code 10005) 🎀

this is used where an operation returns a variable amount of data, with a limit that the client set. if the data returned can't fit inside the limit the client specified, this error shows up 🍓

15.1.2. Filehandle Errors 🦋

these errors are all about when the current or saved filehandle, or the filehandle passed to PUTFH that's supposed to become the current filehandle, is invalid in some way 🫶 this includes situations where the filehandle is valid in general but isn't the right object type for the current operation.

when the error description points at a problem with the current or saved filehandle, just know that filehandles only get checked for the condition if they're implicit arguments of the operation in question 🔍

15.1.2.1. NFS4ERR_BADHANDLE (Error Code 10001) 💜

illegal NFS filehandle for the current server. the current file handle failed internal consistency checks. once it's been accepted as valid (by PUTFH), no later status change can make the filehandle generate this error. slay, it's stable once it's in 👑

15.1.2.2. NFS4ERR_FHEXPIRED (Error Code 10014) 🧊

okay so this one is about a filehandle that's gone stale on us 🥀 The current or saved filehandle you passed as an argument to the current operation is volatile, and the server says it has expired. bestie, that's giving "this link was valid yesterday" energy 💀

15.1.2.3. NFS4ERR_ISDIR (Error Code 21) 📁

this error is for when the current or saved filehandle points at a directory, but the operation you're trying to run doesn't accept a directory as its target. so you tried to do the thing on a folder when it only works on files 🙈 not it, sis 💅

15.1.2.4. NFS4ERR_MOVED (Error Code 10019) 🚚

the file system that holds the current filehandle object isn't available at this server. maybe it got relocated or migrated to another server, or maybe it was never here at all 🫠 the good news is the client can grab the new location by asking for the fs_locations or fs_locations_info attribute for the current filehandle. if you want the tea on that, check out Section 11.2 🍵

15.1.2.5. NFS4ERR_NOFILEHANDLE (Error Code 10020) 🫥

this one is for when the logical current or saved filehandle value is needed by the current operation but nobody set it. usually that means the COMPOUND was put together wrong, like there's no PUTFH or PUTROOTFH before an operation that needs the current filehandle to already be set 🧩 bestie, you forgot the setup step!

15.1.2.6. NFS4ERR_NOTDIR (Error Code 20) 📄

the current (or saved) filehandle points at something that isn't a directory, but the operation needs a directory. so it's like trying to open the closet when you're holding a sock 🧦

15.1.2.7. NFS4ERR_STALE (Error Code 70) 🪦

the current or saved filehandle you passed in is invalid. the file it refers to doesn't exist anymore, or access to it has been revoked. this one is pure "it's over" energy 😭

15.1.2.8. NFS4ERR_SYMLINK (Error Code 10029) 🔗

the current filehandle is a symbolic link, but the current operation doesn't allow a symlink as its target. so the server is basically like "I see the link, but this operation isn't having it" 🙅‍♀️

15.1.2.9. NFS4ERR_WRONG_TYPE (Error Code 10083) 🎭

the current (or saved) filehandle is an object of the wrong type for the current operation, and there isn't a more specific error (like NFS4ERR_ISDIR or NFS4ERR_SYMLINK) that fits. ngl this is the catch-all of the filehandle errors 🧺 fun fact from the original: in NFSv4.0, these situations usually came back as the less-specific NFS4ERR_INVAL instead 🌸

15.1.3. Compound Structure Errors 🧱

these errors are about the overall shape of a Compound request (and when we say Compound, we mean both COMPOUND and CB_COMPOUND), not about any one particular operation inside it 💜

there are a bunch of basic rules for which operations can show up in a Compound request. sessions add even more rules, since they require a Sequence operation (either SEQUENCE or CB_SEQUENCE) at the start of the Compound 👑

15.1.3.1. NFS_OK (Error code 0) ✅

this just means the operation finished successfully, because every single one of the constituent operations completed without error. main character energy 🌟

15.1.3.2. NFS4ERR_MINOR_VERS_MISMATCH (Error code 10021) 🔢

the minor version you asked for isn't one the current listener supports. this comes back in the overall status for the Compound, not tied to any specific operation, since the results report a result count of zero 🫧

15.1.3.3. NFS4ERR_NOT_ONLY_OP (Error Code 10081) ☝️

some operations are allowed outside of a session, but when a Compound doesn't start with a Sequence operation, those operations MUST be the only operation in that Compound. if they're not, you get this error 🚫 one-op-only, bestie!

15.1.3.4. NFS4ERR_OP_ILLEGAL (Error Code 10044) 👾

the operation code isn't valid for the current Compound procedure. in the result stream, this error gets matched with the ILLEGAL value, even though the value in the request stream might look different 🤖 and if the replier pre-parses all the operations in a Compound before running any of them, an RPC-level XDR error might come back instead.

15.1.3.5. NFS4ERR_OP_NOT_IN_SESSION (Error Code 10071) 🔐

most forward operations and all callback operations only work inside the context of a session. so the Compound request in question MUST begin with a Sequence operation. if someone tries to run these outside a session, this error shows up 🙃

15.1.3.6. NFS4ERR_REP_TOO_BIG (Error Code 10066) 📦

the reply to a Compound would be bigger than the channel's negotiated maximum response size. way too much tea to fit in one cup ☕

15.1.3.7. NFS4ERR_REP_TOO_BIG_TO_CACHE (Error Code 10067) 🗄️

the reply to a Compound would be bigger than the channel's negotiated maximum size for replies cached in the reply cache. this only matters when the Sequence for the current request says this request should be cached 🧠

15.1.3.8. NFS4ERR_REQ_TOO_BIG (Error Code 10065) 🏋️

the Compound request itself is over the channel's negotiated maximum size for requests. bestie, you packed a suitcase for a weekend trip and it won't zip 🧳

15.1.3.9. NFS4ERR_RETRY_UNCACHED_REP (Error Code 10068) 🔁

the requester tried to retry a Compound that it had earlier asked not to be placed in the reply cache. it's a "you said don't save it, so I can't replay it" situation 🫠

15.1.3.10. NFS4ERR_SEQUENCE_POS (Error Code 10064) 📍

a Sequence operation showed up somewhere other than the first operation of a Compound request. position matters, babe 💖

15.1.3.11. NFS4ERR_TOO_MANY_OPS (Error Code 10070) 🔢

the Compound request has too many operations. it's over the count that got negotiated when the session was created 🧾

15.1.3.12. NFS4ERR_UNSAFE_COMPOUND (Error Code 10068) ⚠️

the client sent a COMPOUND request with an unsafe mix of operations. specifically, there's a non-idempotent operation that changes the current filehandle, and it isn't followed by a GETFH 🚨 this one is kind of a red flag, not gonna lie.

15.1.4. File System Errors 💾

these errors describe things that happened inside the underlying file system implementation, not in the protocol itself or in any NFSv4.x feature 🗂️

15.1.4.1. NFS4ERR_BADTYPE (Error Code 10007) 🧬

someone tried to create an object with an inappropriate type specified to CREATE. this could be because the type is undefined, because the server doesn't support it, or because the type isn't meant to be created by CREATE. for example, a regular file or named attribute is made with OPEN instead 🛠️

15.1.4.2. NFS4ERR_DQUOT (Error Code 19) 🪣

the resource (quota) hard limit has been exceeded. the user's resource limit on the server is maxed out 😩

15.1.4.3. NFS4ERR_EXIST (Error Code 17) 👯

a file with the specified target name already exists. this comes up when you're creating, renaming, or linking 🫂

15.1.4.4. NFS4ERR_FBIG (Error Code 27) 🐘

the file is too big. the operation would have made the file grow past the server's limit 🦕

15.1.4.5. NFS4ERR_FILE_OPEN (Error Code 10046) 📂

the operation isn't allowed because a file involved in it is currently open. servers may, but are not required to, block linking to, removing, or renaming open files. so it depends on the server, babe 🤷‍♀️

15.1.4.6. NFS4ERR_IO (Error Code 5) ⚡

an I/O error happened, and the file system couldn't recover from it. bestie, this one is a full-on emergency 🚑

15.1.4.7. NFS4ERR_MLINK (Error Code 31) 🔗

the request would have pushed a file past the server's limit on the number of hard links it can have 🪢

15.1.4.8. NFS4ERR_NOENT (Error Code 2) 🔍

no such file or directory. the file or directory name you specified just doesn't exist 🫥

15.1.4.9. NFS4ERR_NOSPC (Error Code 28) 🕳️

there's no space left on the device. the operation would have pushed the server's file system over its limit 📉

15.1.4.10. NFS4ERR_NOTEMPTY (Error Code 66) 🗑️

someone tried to remove a directory that still has stuff in it. you can't delete a full closet, sis 👗

15.1.4.11. NFS4ERR_ROFS (Error Code 30) 🔒

this means a read-only file system. a modifying operation was attempted on one, which is a no-go 🚫📖

15.1.4.12. NFS4ERR_XDEV (Error Code 18) 🌉

this is for an operation, like linking, that crosses a boundary it isn't allowed to cross. the boundaries can be things like:

• that between file systems (where the fsids are different).
• that between different named attribute directories or between a named attribute directory and an ordinary directory.
• that between byte-ranges of a file system that the file system implementation treats as separate (for example, for space accounting purposes), and where cross-connection between the byte-ranges are not allowed.

so basically, you tried to hop a border without a passport 🛂✨

15.1.5. State Management Errors 🎟️

these errors are about problems with the stateid (or one of the stateids) passed to an operation. that includes stateids that are invalid, and also stateids that are valid but point at locking state that has been revoked. depending on the operation, a valid stateid can point at opens, byte-range locks, file or directory delegations, layouts, or device maps 🦋

15.1.5.1. NFS4ERR_ADMIN_REVOKED (Error Code 10047) 🛑

a stateid points at locking state of any type that got revoked because of an administrative action. this can happen even while the lease is still valid 👮‍♀️

15.1.5.2. NFS4ERR_BAD_STATEID (Error Code 10026) ❌

a stateid doesn't properly point at any valid state. if you want to see how stateids get validated, check out Sections 8.2.4 and 8.2.3 📚

15.1.5.3. NFS4ERR_DELEG_REVOKED (Error Code 10087) 🫗

a stateid points at recallable locking state of any type (a delegation or a layout) that got revoked because the client didn't return the lock when it was recalled 😤

15.1.5.4. NFS4ERR_EXPIRED (Error Code 10011) ⌛

a stateid points at locking state that got revoked because the client's lease expired. this can happen right when the lease expires, or later, after a conflicting lock request comes in 🕰️

15.1.5.5. NFS4ERR_OLD_STATEID (Error Code 10024) 👴

a stateid with a non-zero seqid value doesn't match the current seqid for the state the user designated. it's the outdated version of the stateid, bestie 🍂

15.1.6. Security Errors 🛡️

these are the various permission-related errors in NFSv4.1 🔑

15.1.6.1. NFS4ERR_ACCESS (Error Code 13) 🙅

this means permission denied. the caller doesn't have the right permission for the requested operation. contrast this with NFS4ERR_PERM (Section 15.1.6.2), which only covers owner or privileged-user permission failures, and NFS4ERR_WRONG_CRED (Section 15.1.6.4), which is about having the right permission to delete or modify transient objects based on the credentials of whoever created them 🎀

15.1.6.2. NFS4ERR_PERM (Error Code 1) 👑

this means the requester is not the owner. the operation wasn't allowed because the caller is neither a privileged user (root) nor the owner of the target 🙋‍♀️

15.1.6.3. NFS4ERR_WRONGSEC (Error Code 10016) 🔄

the security mechanism the client is using for this operation doesn't match the server's security policy. the client should switch to a different security mechanism and send the operation again, but not with the same slot ID and sequence ID. one or both MUST be different on the re-send 🔀 SECINFO and SECINFO_NO_NAME can help figure out the right mechanism 🧭

15.1.6.4. NFS4ERR_WRONG_CRED (Error Code 10082) 🚫

an operation that manipulates state was attempted by a principal who wasn't allowed to modify that piece of state. not your state to touch, sweetie 🫣

15.1.7. Name Errors 🏷️

names in NFSv4 are UTF-8 strings. if the strings aren't valid UTF-8 or have length zero, the error NFS4ERR_INVAL comes back. beyond that, there are a number of other errors for specific name problems 💅

15.1.7.1. NFS4ERR_BADCHAR (Error Code 10040) 🔤

a UTF-8 string contains a character that the server doesn't support in the context where it's being used. some letters just aren't invited to this party 🥲

15.1.7.2. NFS4ERR_BADNAME (Error Code 10041) 📛

a name string in a request is made of valid UTF-8 characters the server supports, but the server doesn't accept that name as valid for the current operation. the example given is creating a file or directory named ".." on a server whose file system uses that name for links to parent directories. bestie, that name is already taken by the parent folder 🫶

15.1.7.3. NFS4ERR_NAMETOOLONG (Error Code 63) 🧵

okay so this one is giving "way too long, bestie" 💀 It shows up when the filename in an operation goes over the server's implementation limit 🙈

15.1.8. Locking Errors 🔒

this section is all about locking errors, both share reservations and byte-range locking 💖 it doesn't cover errors that happen while reclaiming locks. those live in Section 15.1.9 ✨

15.1.8.1. NFS4ERR_BAD_RANGE (Error Code 10042) 🎀

the byte-range in a LOCK, LOCKT, or LOCKU operation isn't allowed by the server 🙅‍♀️ for example, this pops up when a server that only handles 32-bit ranges gets a range it just can't handle 🌸 (See Section 18.10.3.)

15.1.8.2. NFS4ERR_DEADLOCK (Error Code 10045) 💀

the server figured out there's a byte-range locking deadlock for a READW_LT or WRITEW_LT LOCK operation 😭

15.1.8.3. NFS4ERR_DENIED (Error Code 10010) 🦋

an attempt to lock a file got denied 🥺 since this might be temporary, the client is encouraged to re-send the lock request (but not with the same slot ID and sequence ID; one or both MUST be different on the re-send) until it gets accepted ✨ See Section 9.6 for the deets on re-sending 💅

15.1.8.4. NFS4ERR_LOCKED (Error Code 10012) 🔐

this one happens when a READ or WRITE runs into a conflict with an existing lock 😤 specifically:

• there's a share reservation that doesn't match the I/O being done.
• the range being read or written overlaps an existing mandatory byte-range lock.

15.1.8.5. NFS4ERR_LOCKS_HELD (Error Code 10037) 🙈

an operation got blocked because locks were showing up when they weren't expected 😬

15.1.8.6. NFS4ERR_LOCK_NOTSUPP (Error Code 10043) 🚫

a LOCK operation tried to upgrade or downgrade a byte-range lock range the owner already holds, and the server doesn't support atomic upgrade or downgrade of locks 💔

15.1.8.7. NFS4ERR_LOCK_RANGE (Error Code 10028) 🌀

a LOCK operation is working on a range that partly overlaps a byte-range lock the current lock-owner already holds, and doesn't exactly match a single one of those locks 🫠 this happens when the server doesn't support this kind of request, so it doesn't implement POSIX locking semantics [24] ✨ See Sections 18.10.4, 18.11.4, and 18.12.4 for how this applies to LOCK, LOCKT, and LOCKU respectively 🧚

15.1.8.8. NFS4ERR_OPENMODE (Error Code 10038) 🚪

the client tried a READ, WRITE, LOCK, or some other operation that the stateid passed in doesn't allow (like writing to a file that was opened read-only) 🙅‍♀️

15.1.8.9. NFS4ERR_SHARE_DENIED (Error Code 10015) 🙃

an attempt to OPEN a file with a share reservation failed because of a share conflict 💀

15.1.9. Reclaim Errors 🔄

these errors are all about the process of reclaiming locks after a server restart 🌅

15.1.9.1. NFS4ERR_COMPLETE_ALREADY (Error Code 10054) 🙄

the client already sent a successful RECLAIM_COMPLETE operation 💅 sending another one isn't needed, and it gets this error ✨

15.1.9.2. NFS4ERR_GRACE (Error Code 10013) ⏳

the server was in its recovery or grace period 🕰️ the locking request wasn't a reclaim, so it couldn't be granted during that time 😭

15.1.9.3. NFS4ERR_NO_GRACE (Error Code 10033) 🚨

a reclaim of client state was tried in a situation where the server can't guarantee that conflicting state hasn't been handed to another client 😬 that can happen when the reclaim is done outside the server's grace period, after the client already did a RECLAIM_COMPLETE operation, or when earlier operations left the server unable to tell whether a reclaim-interfering edge condition exists 🫣

15.1.9.4. NFS4ERR_RECLAIM_BAD (Error Code 10034) 🚫

the server decided a reclaim from the client isn't valid, meaning the lock being reclaimed couldn't possibly have existed before the server restarted 🧐 the server doesn't have to make that call, and usually it just trusts the client to only reclaim locks it got before the restart 🫶 but when the server has reliable info to make the call, this error says the reclaim was rejected as invalid 💜 This is different from NFS4ERR_RECLAIM_CONFLICT (see Section 15.1.9.5), where the server can only tell that some reclaim is invalid, but can't figure out which request is the bad one 🤷‍♀️

15.1.9.5. NFS4ERR_RECLAIM_CONFLICT (Error Code 10035) ⚡

the client's reclaim ran into a conflict and can't be satisfied 😵 this might mean the client is misbehaving, though not necessarily the one getting the error 👀 the misbehavior could belong to the client that set up the lock this client conflicted with 🕵️‍♀️ See also Section 15.1.9.4 for the related error, NFS4ERR_RECLAIM_BAD 🎀

15.1.10. pNFS Errors 🧩

this section covers pNFS-related errors, including the ones tied to using NFSv4.1 to talk to a data server 🌈

15.1.10.1. NFS4ERR_BADIOMODE (Error Code 10049) 🙃

an invalid or inappropriate layout iomode was specified 🙈 for example, say a client's LAYOUTGET operation asked for an iomode of LAYOUTIOMODE4_RW, but the server can't or won't let the client send write requests to data servers 🥲 the server can reply with NFS4ERR_BADIOMODE 💌 and then the client would send another LAYOUTGET with an iomode of LAYOUTIOMODE4_READ 🦋

15.1.10.2. NFS4ERR_BADLAYOUT (Error Code 10050) 💔

the layout specified is invalid somehow 😬 for LAYOUTCOMMIT, this means the layout isn't held by the client or isn't of mode LAYOUTIOMODE4_RW 💅 for LAYOUTGET, it means a layout matching the client's minimum length spec can't be granted 🥺

15.1.10.3. NFS4ERR_LAYOUTTRYLATER (Error Code 10058) ⏰

layouts are temporarily unavailable for the file 🫠 the client should re-send later (but not with the same slot ID and sequence ID; one or both MUST be different on the re-send) ✨

15.1.10.4. NFS4ERR_LAYOUTUNAVAILABLE (Error Code 10059) 🚫

comes back when layouts aren't available for the current file system or that specific file 😔

15.1.10.5. NFS4ERR_NOMATCHING_LAYOUT (Error Code 10060) 🔍

comes back when layouts get recalled and the client has no layouts matching the spec of the ones being recalled 🙃

15.1.10.6. NFS4ERR_PNFS_IO_HOLE (Error Code 10075) 🕳️

the pNFS client tried to read from or write to an illegal hole in a file on a data server that's using sparse packing 🫣 See Section 13.4.4 🧚

15.1.10.7. NFS4ERR_PNFS_NO_LAYOUT (Error Code 10080) 🙅‍♀️

the pNFS client tried to read from or write to a file (using a request to a data server) without holding a valid layout 💀 this includes the case where the client had a layout, but its iomode doesn't allow a WRITE 😤

15.1.10.8. NFS4ERR_RETURNCONFLICT (Error Code 10086) ↩️

a layout is unavailable because someone tried to do LAYOUTGET before a pending LAYOUTRETURN on the file got received 🌀 See Section 12.5.5.2.1.3 📖

15.1.10.9. NFS4ERR_UNKNOWN_LAYOUTTYPE (Error Code 10062) ❓

the client asked for a layout type the server doesn't support 🙈

15.1.11. Session Use Errors 🎟️

this section is about errors you run into when using sessions, meaning when a request uses a Sequence (that's either SEQUENCE or CB_SEQUENCE) operation 💜

15.1.11.1. NFS4ERR_BADSESSION (Error Code 10052) 👻

the session ID you sent isn't known to the server the operation is addressed to 😵

15.1.11.2. NFS4ERR_BADSLOT (Error Code 10053) 🎰

the requester sent a Sequence operation that tried to use a slot the replier doesn't have in its slot table 🫨 it's possible the slot got retired 🥀

15.1.11.3. NFS4ERR_BAD_HIGH_SLOT (Error Code 10077) 📈

the highest_slot argument in a Sequence operation goes over the replier's enforced highest_slotid 🙅‍♀️

15.1.11.4. NFS4ERR_CB_PATH_DOWN (Error Code 10048) 🛤️

there's a problem contacting the client over the callback path 📞 this error's job has mostly been taken over by status flags in the reply to the SEQUENCE operation (see Section 18.46) ✨

15.1.11.5. NFS4ERR_DEADSESSION (Error Code 10078) 💀

the session is a persistent session that's dead, so it won't accept new requests or do new operations on existing requests (in the case where a request was partly executed before a server restart) ⚰️

15.1.11.6. NFS4ERR_CONN_NOT_BOUND_TO_SESSION (Error Code 10055) 🔗

a Sequence operation got sent on a connection that isn't associated with the specified session, and the client asked for that connection association to be enforced with SP4_MACH_CRED or SP4_SSV state protection 🙅‍♀️

15.1.11.7. NFS4ERR_SEQ_FALSE_RETRY (Error Code 10076) 🔁

the requester sent a Sequence operation with a slot ID and sequence ID that are in the reply cache, but the replier noticed the retried request isn't the same as the original 👀 See Section 2.10.6.1.3.1 📖

15.1.11.8. NFS4ERR_SEQ_MISORDERED (Error Code 10063) 🌪️

the requester sent a Sequence operation with an invalid sequence ID 😩

15.1.12. Session Management Errors 🗂️

this section covers errors tied to requests used in session management 💅

15.1.12.1. NFS4ERR_BACK_CHAN_BUSY (Error Code 10057) 📵

someone tried to destroy a session, but it can't be destroyed because the server still has callback requests outstanding 🙅‍♀️

15.1.12.2. NFS4ERR_BAD_SESSION_DIGEST (Error Code 10051) 🔑

the digest used in a SET_SSV request isn't valid 🚫

15.1.13. Client Management Errors 🪪

this section covers errors tied to requests used to create and manage client IDs 💖

15.1.13.1. NFS4ERR_CLIENTID_BUSY (Error Code 10074) 🐝

the DESTROY_CLIENTID operation found sessions and/or unexpired state still associated with the client ID it's trying to destroy 😤

15.1.13.2. NFS4ERR_CLID_INUSE (Error Code 10017) 👯

while processing an EXCHANGE_ID operation, the server got a co_ownerid field that matches an existing client with valid leased state, but the principal sending the EXCHANGE_ID differs from the principal that set up the existing client 😳 this is a collision (most likely by chance) between clients 🎲 the client should recover by changing the co_ownerid and re-sending EXCHANGE_ID (but not with the same slot ID and sequence ID; one or both MUST be different on the re-send) 🫶

15.1.13.3. NFS4ERR_ENCR_ALG_UNSUPP (Error Code 10079) 🔐

an EXCHANGE_ID was sent that specified state protection via SSV, and the set of encryption algorithms the client offered didn't include any the server supports 🙈

15.1.13.4. NFS4ERR_HASH_ALG_UNSUPP (Error Code 10072) #️⃣

an EXCHANGE_ID was sent that specified state protection via SSV, and the set of hashing algorithms the client offered didn't include any the server supports 🥲

15.1.13.5. NFS4ERR_STALE_CLIENTID (Error Code 10022) 🥀

a client ID the server doesn't recognize got passed to an operation 👀 unlike NFSv4.0, client IDs aren't passed explicitly to the server in ordinary locking operations, so they can't cause this error 🙅‍♀️ instead, when there's a server restart, it first shows up as an error on the associated session, and the staleness of the client ID gets detected when the client tries to associate it with a new session ✨

15.1.14. Delegation Errors ✨

okay this section is all about the errors you get when you ask for a delegation or hand one back 🦋 it's giving paperwork energy but make it cute 🎀

15.1.14.1. NFS4ERR_DELEG_ALREADY_WANTED (Error Code 10056) 💅

this one happens when the client asks for a delegation it already said it wanted 🙈 like, you already put your name on the list bestie, you don't need to ask twice 💀

15.1.14.2. NFS4ERR_DIRDELEG_UNAVAIL (Error Code 10084) 🌸

this error shows up when the server can't or won't hand over the directory delegation you asked for 😭 maybe it's booked, maybe it's just not feeling it today 💅

15.1.14.3. NFS4ERR_RECALLCONFLICT (Error Code 10061) 🔥

so a recallable object (aka a layout or a delegation) is off limits right now because somebody's already in the middle of a conflicting recall on that exact object 🫶 it's like trying to get into the club while the bouncer is busy with someone else 💀

15.1.14.4. NFS4ERR_REJECT_DELEG (Error Code 10085) 👑

the callback operation that's supposed to deal with a brand new delegation said nope and rejected it 😭 rude but iconic 💅

---

15.1.15. Attribute Handling Errors 🎀

this section is the tea on errors that are specifically about handling attributes in NFSv4 💖

15.1.15.1. NFS4ERR_ATTRNOTSUPP (Error Code 10032) 🦋

this means the server doesn't support an attribute you asked for 🙈 and this error MUST NOT be returned by the GETATTR operation 🫶 so don't you dare send it from GETATTR, periodt 💅

15.1.15.2. NFS4ERR_BADOWNER (Error Code 10039) 🌈

this one comes up when an owner or owner_group attribute value, or the who field of an ACE inside an ACL attribute value, can't be translated into a local representation 😭 the server just can't figure out who that is 💀

15.1.15.3. NFS4ERR_NOT_SAME (Error Code 10027) ✨

the VERIFY operation sends this back to say the attributes it compared were NOT the same as the ones the client sent in the request 🧚 the vibes just don't match 💅

15.1.15.4. NFS4ERR_SAME (Error Code 10009) 💜

this is the mirror image of the last one 🪞 the NVERIFY operation returns it to say the attributes it compared WERE the same as what the client sent 🎀 twins 💖

---

15.1.16. Obsoleted Errors 🍓

okay so these errors are retired and they MUST NOT be generated by any NFSv4.1 operation 💀 there are a few reasons this happened, and basically it's giving glow-up 🌸

• the job the error used to do has been taken over by one of the status bits that the SEQUENCE operation returns ✨
• the new session structure and the change in locking made the error unnecessary 🦋
• NFSv4.1 got a restructure of some errors, and that got rid of certain ones entirely 🫶

15.1.16.1. NFS4ERR_BAD_SEQID (Error Code 10026) 🙈

the sequence number (seqid) in a locking request is neither the next one expected nor the last one processed 😭 in NFSv4.1 these seqids just get ignored, no drama 💅

15.1.16.2. NFS4ERR_LEASE_MOVED (Error Code 10031) 🚚

this one was for when a lease being renewed belongs to a file system that got migrated to a new server 🧳 it's been superseded by the SEQ4_STATUS_LEASE_MOVED status bit, so check out Section 18.46 for the deets 🌈

15.1.16.3. NFS4ERR_NXIO (Error Code 5) 💀

this is the I/O error that says "no such device or address" 😭 it was for block and character device access stuff, but NFSv4.1 isn't a device-access protocol, so it doesn't apply here 🙅‍♀️

15.1.16.4. NFS4ERR_RESTOREFH (Error Code 10030) 🧚

the RESTOREFH operation has no saved filehandle (the one SAVEFH stashes) to work with 🥲 in NFSv4.1 this got replaced by NFS4ERR_NOFILEHANDLE 💖

15.1.16.5. NFS4ERR_STALE_STATEID (Error Code 10023) 🕰️

this one happened when a stateid from an earlier server instance got used 😭 but in NFSv4.1 it's moot, because every operation that takes a stateid MUST be preceded by the SEQUENCE operation 🫶 and the session infrastructure that supports SEQUENCE catches the old server instance on its own, so we're good ✨

---

15.2. Operations and Their Valid Errors 💅

this section has a table that lists the valid error returns for each protocol operation 🎀 the NFS4_OK error (which just means no error) isn't listed, but it's understood that every operation can return it, with two big exceptions 👑

• the operations that MUST NOT be implemented are OPEN_CONFIRM, RELEASE_LOCKOWNER, RENEW, SETCLIENTID, and SETCLIENTID_CONFIRM 💀
• the invalid operation is ILLEGAL 🙅‍♀️

Valid Error Returns for Each Protocol Operation

Okay bestie, the part where we hit the big cheat-sheet table 💖 This table lists which errors each operation is allowed to hand back, so it's basically the ultimate ✨ who-can-do-what ✨ guide. 🎀

   +----------------------+--------------------------------------------+
   | Operation            | Errors                                     |
   +----------------------+--------------------------------------------+
   | ACCESS               | NFS4ERR_ACCESS, NFS4ERR_BADXDR,            |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_FHEXPIRED, NFS4ERR_INVAL,          |
   |                      | NFS4ERR_IO, NFS4ERR_MOVED,                 |
   |                      | NFS4ERR_NOFILEHANDLE,                      |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS                       |
   | BACKCHANNEL_CTL      | NFS4ERR_BADXDR, NFS4ERR_DEADSESSION,       |
   |                      | NFS4ERR_DELAY, NFS4ERR_INVAL,              |
   |                      | NFS4ERR_NOENT, NFS4ERR_OP_NOT_IN_SESSION,  |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_TOO_MANY_OPS                       |
   | BIND_CONN_TO_SESSION | NFS4ERR_BADSESSION, NFS4ERR_BADXDR,        |
   |                      | NFS4ERR_BAD_SESSION_DIGEST,                |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_INVAL, NFS4ERR_NOT_ONLY_OP,        |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_TOO_MANY_OPS  |
   | CLOSE                | NFS4ERR_ADMIN_REVOKED, NFS4ERR_BADXDR,     |
   |                      | NFS4ERR_BAD_STATEID, NFS4ERR_DEADSESSION,  |
   |                      | NFS4ERR_DELAY, NFS4ERR_EXPIRED,            |
   |                      | NFS4ERR_FHEXPIRED, NFS4ERR_LOCKS_HELD,     |
   |                      | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,       |
   |                      | NFS4ERR_OLD_STATEID,                       |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONG_CRED   |
   | COMMIT               | NFS4ERR_ACCESS, NFS4ERR_BADXDR,            |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_FHEXPIRED, NFS4ERR_IO,             |
   |                      | NFS4ERR_ISDIR, NFS4ERR_MOVED,              |
   |                      | NFS4ERR_NOFILEHANDLE,                      |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_SYMLINK, NFS4ERR_TOO_MANY_OPS,     |
   |                      | NFS4ERR_WRONG_TYPE                         |
   | CREATE               | NFS4ERR_ACCESS, NFS4ERR_ATTRNOTSUPP,       |
   |                      | NFS4ERR_BADCHAR, NFS4ERR_BADNAME,          |
   |                      | NFS4ERR_BADOWNER, NFS4ERR_BADTYPE,         |
   |                      | NFS4ERR_BADXDR, NFS4ERR_DEADSESSION,       |
   |                      | NFS4ERR_DELAY, NFS4ERR_DQUOT,              |
   |                      | NFS4ERR_EXIST, NFS4ERR_FHEXPIRED,          |
   |                      | NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_MLINK,  |
   |                      | NFS4ERR_MOVED, NFS4ERR_NAMETOOLONG,        |
   |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOSPC,       |
   |                      | NFS4ERR_NOTDIR, NFS4ERR_OP_NOT_IN_SESSION, |
   |                      | NFS4ERR_PERM, NFS4ERR_REP_TOO_BIG,         |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP, NFS4ERR_ROFS,  |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS,                      |
   |                      | NFS4ERR_UNSAFE_COMPOUND                    |
   | CREATE_SESSION       | NFS4ERR_BADXDR, NFS4ERR_CLID_INUSE,        |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_INVAL, NFS4ERR_NOENT,              |
   |                      | NFS4ERR_NOT_ONLY_OP, NFS4ERR_NOSPC,        |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SEQ_MISORDERED,                    |
   |                      | NFS4ERR_SERVERFAULT,                       |
   |                      | NFS4ERR_STALE_CLIENTID, NFS4ERR_TOOSMALL,  |
   |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONG_CRED   |
   | DELEGPURGE           | NFS4ERR_BADXDR, NFS4ERR_DEADSESSION,       |
   |                      | NFS4ERR_DELAY, NFS4ERR_NOTSUPP,            |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_TOO_MANY_OPS, |
   |                      | NFS4ERR_WRONG_CRED                         |
   | DELEGRETURN          | NFS4ERR_ADMIN_REVOKED, NFS4ERR_BADXDR,     |
   |                      | NFS4ERR_BAD_STATEID, NFS4ERR_DEADSESSION,  |
   |                      | NFS4ERR_DELAY, NFS4ERR_DELEG_REVOKED,      |
   |                      | NFS4ERR_EXPIRED, NFS4ERR_FHEXPIRED,        |
   |                      | NFS4ERR_INVAL, NFS4ERR_MOVED,              |
   |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTSUPP,     |
   |                      | NFS4ERR_OLD_STATEID,                       |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONG_CRED   |
   | DESTROY_CLIENTID     | NFS4ERR_BADXDR, NFS4ERR_CLIENTID_BUSY,     |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_NOT_ONLY_OP, NFS4ERR_REP_TOO_BIG,  |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT,                       |
   |                      | NFS4ERR_STALE_CLIENTID,                    |
   |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONG_CRED   |
   | DESTROY_SESSION      | NFS4ERR_BACK_CHAN_BUSY,                    |
   |                      | NFS4ERR_BADSESSION, NFS4ERR_BADXDR,        |
   |                      | NFS4ERR_CB_PATH_DOWN,                      |
   |                      | NFS4ERR_CONN_NOT_BOUND_TO_SESSION,         |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_NOT_ONLY_OP, NFS4ERR_REP_TOO_BIG,  |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT,                       |
   |                      | NFS4ERR_STALE_CLIENTID,                    |
   |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONG_CRED   |
   | EXCHANGE_ID          | NFS4ERR_BADCHAR, NFS4ERR_BADXDR,           |
   |                      | NFS4ERR_CLID_INUSE, NFS4ERR_DEADSESSION,   |
   |                      | NFS4ERR_DELAY, NFS4ERR_ENCR_ALG_UNSUPP,    |
   |                      | NFS4ERR_HASH_ALG_UNSUPP, NFS4ERR_INVAL,    |
   |                      | NFS4ERR_NOENT, NFS4ERR_NOT_ONLY_OP,        |
   |                      | NFS4ERR_NOT_SAME, NFS4ERR_REP_TOO_BIG,     |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_TOO_MANY_OPS  |
   | FREE_STATEID         | NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID,       |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_LOCKS_HELD, NFS4ERR_OLD_STATEID,   |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_TOO_MANY_OPS, |
   |                      | NFS4ERR_WRONG_CRED                         |
   | GET_DIR_DELEGATION   | NFS4ERR_ACCESS, NFS4ERR_BADXDR,            |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_DIRDELEG_UNAVAIL,                  |
   |                      | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,          |
   |                      | NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_MOVED,  |
   |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTDIR,      |
   |                      | NFS4ERR_NOTSUPP,                           |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |

Each row pairs one operation with its errors, so you can scan down the left column to find the op you care about and then read across the right to see the full list. 🦋

   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONG_TYPE   |
   | RECLAIM_COMPLETE     | NFS4ERR_BADXDR, NFS4ERR_COMPLETE_ALREADY,  |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_FHEXPIRED, NFS4ERR_INVAL,          |
   |                      | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,       |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONG_CRED,  |
   |                      | NFS4ERR_WRONG_TYPE                         |
   | RELEASE_LOCKOWNER    | NFS4ERR_NOTSUPP                            |
   | REMOVE               | NFS4ERR_ACCESS, NFS4ERR_BADCHAR,           |
   |                      | NFS4ERR_BADNAME, NFS4ERR_BADXDR,           |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_FHEXPIRED, NFS4ERR_FILE_OPEN,      |
   |                      | NFS4ERR_GRACE, NFS4ERR_INVAL, NFS4ERR_IO,  |
   |                      | NFS4ERR_MOVED, NFS4ERR_NAMETOOLONG,        |
   |                      | NFS4ERR_NOENT, NFS4ERR_NOFILEHANDLE,       |
   |                      | NFS4ERR_NOTDIR, NFS4ERR_NOTEMPTY,          |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP, NFS4ERR_ROFS,  |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS                       |
   | RENAME               | NFS4ERR_ACCESS, NFS4ERR_BADCHAR,           |
   |                      | NFS4ERR_BADNAME, NFS4ERR_BADXDR,           |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_DQUOT, NFS4ERR_EXIST,              |
   |                      | NFS4ERR_FHEXPIRED, NFS4ERR_FILE_OPEN,      |
   |                      | NFS4ERR_GRACE, NFS4ERR_INVAL, NFS4ERR_IO,  |
   |                      | NFS4ERR_MLINK, NFS4ERR_MOVED,              |
   |                      | NFS4ERR_NAMETOOLONG, NFS4ERR_NOENT,        |
   |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOSPC,       |
   |                      | NFS4ERR_NOTDIR, NFS4ERR_NOTEMPTY,          |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP, NFS4ERR_ROFS,  |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONGSEC,    |
   |                      | NFS4ERR_XDEV                               |
   | RENEW                | NFS4ERR_NOTSUPP                            |
   | RESTOREFH            | NFS4ERR_DEADSESSION, NFS4ERR_FHEXPIRED,    |
   |                      | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,       |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONGSEC     |
   | SAVEFH               | NFS4ERR_DEADSESSION, NFS4ERR_FHEXPIRED,    |
   |                      | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE,       |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS                       |
   | SECINFO              | NFS4ERR_ACCESS, NFS4ERR_BADCHAR,           |
   |                      | NFS4ERR_BADNAME, NFS4ERR_BADXDR,           |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_FHEXPIRED, NFS4ERR_INVAL,          |
   |                      | NFS4ERR_MOVED, NFS4ERR_NAMETOOLONG,        |
   |                      | NFS4ERR_NOENT, NFS4ERR_NOFILEHANDLE,       |
   |                      | NFS4ERR_NOTDIR, NFS4ERR_OP_NOT_IN_SESSION, |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS                       |
   | SECINFO_NO_NAME      | NFS4ERR_ACCESS, NFS4ERR_BADXDR,            |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_FHEXPIRED, NFS4ERR_INVAL,          |
   |                      | NFS4ERR_MOVED, NFS4ERR_NOENT,              |
   |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTDIR,      |
   |                      | NFS4ERR_NOTSUPP,                           |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS                       |
   | SEQUENCE             | NFS4ERR_BADSESSION, NFS4ERR_BADSLOT,       |
   |                      | NFS4ERR_BADXDR, NFS4ERR_BAD_HIGH_SLOT,     |
   |                      | NFS4ERR_CONN_NOT_BOUND_TO_SESSION,         |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SEQUENCE_POS,                      |
   |                      | NFS4ERR_SEQ_FALSE_RETRY,                   |
   |                      | NFS4ERR_SEQ_MISORDERED,                    |
   |                      | NFS4ERR_TOO_MANY_OPS                       |
   | SET_SSV              | NFS4ERR_BADXDR,                            |
   |                      | NFS4ERR_BAD_SESSION_DIGEST,                |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_INVAL, NFS4ERR_OP_NOT_IN_SESSION,  |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_TOO_MANY_OPS                       |
   | SETATTR              | NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED,     |
   |                      | NFS4ERR_ATTRNOTSUPP, NFS4ERR_BADCHAR,      |
   |                      | NFS4ERR_BADOWNER, NFS4ERR_BADXDR,          |
   |                      | NFS4ERR_BAD_STATEID, NFS4ERR_DEADSESSION,  |
   |                      | NFS4ERR_DELAY, NFS4ERR_DELEG_REVOKED,      |
   |                      | NFS4ERR_DQUOT, NFS4ERR_EXPIRED,            |
   |                      | NFS4ERR_FBIG, NFS4ERR_FHEXPIRED,           |
   |                      | NFS4ERR_GRACE, NFS4ERR_INVAL, NFS4ERR_IO,  |
   |                      | NFS4ERR_LOCKED, NFS4ERR_MOVED,             |
   |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOSPC,       |
   |                      | NFS4ERR_OLD_STATEID, NFS4ERR_OPENMODE,     |
   |                      | NFS4ERR_OP_NOT_IN_SESSION, NFS4ERR_PERM,   |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP, NFS4ERR_ROFS,  |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS,                      |
   |                      | NFS4ERR_UNKNOWN_LAYOUTTYPE,                |
   |                      | NFS4ERR_WRONG_TYPE                         |
   | SETCLIENTID          | NFS4ERR_NOTSUPP                            |
   | SETCLIENTID_CONFIRM  | NFS4ERR_NOTSUPP                            |
   | TEST_STATEID         | NFS4ERR_BADXDR, NFS4ERR_DEADSESSION,       |
   |                      | NFS4ERR_DELAY, NFS4ERR_OP_NOT_IN_SESSION,  |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_TOO_MANY_OPS  |
   | VERIFY               | NFS4ERR_ACCESS, NFS4ERR_ATTRNOTSUPP,       |
   |                      | NFS4ERR_BADCHAR, NFS4ERR_BADXDR,           |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,          |
   |                      | NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_MOVED,  |
   |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOT_SAME,    |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS,                      |
   |                      | NFS4ERR_UNKNOWN_LAYOUTTYPE,                |
   |                      | NFS4ERR_WRONG_TYPE                         |
   | WANT_DELEGATION      | NFS4ERR_BADXDR, NFS4ERR_DEADSESSION,       |

okay the table keeps going and it's giving a lot 💅✨ every row is a little promise about which errors an operation can throw, so we're reading it exactly as written 🫶

okay the list keeps going and it's giving very long 💅

   |                      | NFS4ERR_DELAY,                             |
   |                      | NFS4ERR_DELEG_ALREADY_WANTED,              |
   |                      | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,          |
   |                      | NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_MOVED,  |
   |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTSUPP,     |
   |                      | NFS4ERR_NO_GRACE,                          |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_RECALLCONFLICT,                    |
   |                      | NFS4ERR_RECLAIM_BAD,                       |
   |                      | NFS4ERR_RECLAIM_CONFLICT,                  |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP,                |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_TOO_MANY_OPS, NFS4ERR_WRONG_TYPE   |
   | WRITE                | NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED,     |
   |                      | NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID,       |
   |                      | NFS4ERR_DEADSESSION, NFS4ERR_DELAY,        |
   |                      | NFS4ERR_DELEG_REVOKED, NFS4ERR_DQUOT,      |
   |                      | NFS4ERR_EXPIRED, NFS4ERR_FBIG,             |
   |                      | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE,          |
   |                      | NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_ISDIR,  |
   |                      | NFS4ERR_LOCKED, NFS4ERR_MOVED,             |
   |                      | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOSPC,       |
   |                      | NFS4ERR_OLD_STATEID, NFS4ERR_OPENMODE,     |
   |                      | NFS4ERR_OP_NOT_IN_SESSION,                 |
   |                      | NFS4ERR_PNFS_IO_HOLE,                      |
   |                      | NFS4ERR_PNFS_NO_LAYOUT,                    |
   |                      | NFS4ERR_REP_TOO_BIG,                       |
   |                      | NFS4ERR_REP_TOO_BIG_TO_CACHE,              |
   |                      | NFS4ERR_REQ_TOO_BIG,                       |
   |                      | NFS4ERR_RETRY_UNCACHED_REP, NFS4ERR_ROFS,  |
   |                      | NFS4ERR_SERVERFAULT, NFS4ERR_STALE,        |
   |                      | NFS4ERR_SYMLINK, NFS4ERR_TOO_MANY_OPS,     |
   |                      | NFS4ERR_WRONG_TYPE                         |
   +----------------------+--------------------------------------------+

Table 6

15.3. Callback Operations and Their Valid Errors ✨

okay so this section is basically the cheat sheet 📋 for every callback operation and the errors it's allowed to throw back. fr fr it's super handy. the code NFS4_OK (aka "no error") isn't listed in the table, but it's understood to be returnable by every callback op except CB_ILLEGAL 💅

look at this table, it's giving organized chaos 🎀

         Valid Error Returns for Each Protocol Callback Operation

   +-------------------------+-----------------------------------------+
   | Callback Operation      | Errors                                  |
   +-------------------------+-----------------------------------------+
   | CB_GETATTR              | NFS4ERR_BADHANDLE, NFS4ERR_BADXDR,      |
   |                         | NFS4ERR_DELAY, NFS4ERR_INVAL,           |
   |                         | NFS4ERR_OP_NOT_IN_SESSION,              |
   |                         | NFS4ERR_REP_TOO_BIG,                    |
   |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,           |
   |                         | NFS4ERR_REQ_TOO_BIG,                    |
   |                         | NFS4ERR_RETRY_UNCACHED_REP,             |
   |                         | NFS4ERR_SERVERFAULT,                    |
   |                         | NFS4ERR_TOO_MANY_OPS,                   |
   | CB_ILLEGAL              | NFS4ERR_BADXDR, NFS4ERR_OP_ILLEGAL      |
   | CB_LAYOUTRECALL         | NFS4ERR_BADHANDLE, NFS4ERR_BADIOMODE,   |
   |                         | NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID,    |
   |                         | NFS4ERR_DELAY, NFS4ERR_INVAL,           |
   |                         | NFS4ERR_NOMATCHING_LAYOUT,              |
   |                         | NFS4ERR_NOTSUPP,                        |
   |                         | NFS4ERR_OP_NOT_IN_SESSION,              |
   |                         | NFS4ERR_REP_TOO_BIG,                    |
   |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,           |
   |                         | NFS4ERR_REQ_TOO_BIG,                    |
   |                         | NFS4ERR_RETRY_UNCACHED_REP,             |
   |                         | NFS4ERR_TOO_MANY_OPS,                   |
   |                         | NFS4ERR_UNKNOWN_LAYOUTTYPE,             |
   |                         | NFS4ERR_WRONG_TYPE                      |
   | CB_NOTIFY               | NFS4ERR_BADHANDLE, NFS4ERR_BADXDR,      |
   |                         | NFS4ERR_BAD_STATEID, NFS4ERR_DELAY,     |
   |                         | NFS4ERR_INVAL, NFS4ERR_NOTSUPP,         |
   |                         | NFS4ERR_OP_NOT_IN_SESSION,              |
   |                         | NFS4ERR_REP_TOO_BIG,                    |
   |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,           |
   |                         | NFS4ERR_REQ_TOO_BIG,                    |
   |                         | NFS4ERR_RETRY_UNCACHED_REP,             |
   |                         | NFS4ERR_SERVERFAULT,                    |
   |                         | NFS4ERR_TOO_MANY_OPS                    |
   | CB_NOTIFY_DEVICEID      | NFS4ERR_BADXDR, NFS4ERR_DELAY,          |
   |                         | NFS4ERR_INVAL, NFS4ERR_NOTSUPP,         |
   |                         | NFS4ERR_OP_NOT_IN_SESSION,              |
   |                         | NFS4ERR_REP_TOO_BIG,                    |
   |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,           |
   |                         | NFS4ERR_REQ_TOO_BIG,                    |
   |                         | NFS4ERR_RETRY_UNCACHED_REP,             |
   |                         | NFS4ERR_SERVERFAULT,                    |
   |                         | NFS4ERR_TOO_MANY_OPS                    |
   | CB_NOTIFY_LOCK          | NFS4ERR_BADHANDLE, NFS4ERR_BADXDR,      |
   |                         | NFS4ERR_BAD_STATEID, NFS4ERR_DELAY,     |
   |                         | NFS4ERR_NOTSUPP,                        |
   |                         | NFS4ERR_OP_NOT_IN_SESSION,              |
   |                         | NFS4ERR_REP_TOO_BIG,                    |
   |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,           |
   |                         | NFS4ERR_REQ_TOO_BIG,                    |
   |                         | NFS4ERR_RETRY_UNCACHED_REP,             |
   |                         | NFS4ERR_SERVERFAULT,                    |
   |                         | NFS4ERR_TOO_MANY_OPS                    |
   | CB_PUSH_DELEG           | NFS4ERR_BADHANDLE, NFS4ERR_BADXDR,      |
   |                         | NFS4ERR_DELAY, NFS4ERR_INVAL,           |
   |                         | NFS4ERR_NOTSUPP,                        |
   |                         | NFS4ERR_OP_NOT_IN_SESSION,              |
   |                         | NFS4ERR_REJECT_DELEG,                   |
   |                         | NFS4ERR_REP_TOO_BIG,                    |
   |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,           |
   |                         | NFS4ERR_REQ_TOO_BIG,                    |
   |                         | NFS4ERR_RETRY_UNCACHED_REP,             |
   |                         | NFS4ERR_SERVERFAULT,                    |
   |                         | NFS4ERR_TOO_MANY_OPS,                   |
   |                         | NFS4ERR_WRONG_TYPE                      |
   | CB_RECALL               | NFS4ERR_BADHANDLE, NFS4ERR_BADXDR,      |
   |                         | NFS4ERR_BAD_STATEID, NFS4ERR_DELAY,     |
   |                         | NFS4ERR_OP_NOT_IN_SESSION,              |
   |                         | NFS4ERR_REP_TOO_BIG,                    |
   |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,           |
   |                         | NFS4ERR_REQ_TOO_BIG,                    |
   |                         | NFS4ERR_RETRY_UNCACHED_REP,             |
   |                         | NFS4ERR_SERVERFAULT,                    |
   |                         | NFS4ERR_TOO_MANY_OPS                    |
   | CB_RECALL_ANY           | NFS4ERR_BADXDR, NFS4ERR_DELAY,          |
   |                         | NFS4ERR_INVAL,                          |
   |                         | NFS4ERR_OP_NOT_IN_SESSION,              |
   |                         | NFS4ERR_REP_TOO_BIG,                    |
   |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,           |
   |                         | NFS4ERR_REQ_TOO_BIG,                    |
   |                         | NFS4ERR_RETRY_UNCACHED_REP,             |
   |                         | NFS4ERR_TOO_MANY_OPS                    |
   | CB_RECALLABLE_OBJ_AVAIL | NFS4ERR_BADXDR, NFS4ERR_DELAY,          |
   |                         | NFS4ERR_INVAL, NFS4ERR_NOTSUPP,         |
   |                         | NFS4ERR_OP_NOT_IN_SESSION,              |
   |                         | NFS4ERR_REP_TOO_BIG,                    |
   |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,           |
   |                         | NFS4ERR_REQ_TOO_BIG,                    |
   |                         | NFS4ERR_RETRY_UNCACHED_REP,             |
   |                         | NFS4ERR_SERVERFAULT,                    |
   |                         | NFS4ERR_TOO_MANY_OPS                    |
   | CB_RECALL_SLOT          | NFS4ERR_BADXDR, NFS4ERR_BAD_HIGH_SLOT,  |
   |                         | NFS4ERR_DELAY,                          |
   |                         | NFS4ERR_OP_NOT_IN_SESSION,              |
   |                         | NFS4ERR_REP_TOO_BIG,                    |
   |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,           |
   |                         | NFS4ERR_REQ_TOO_BIG,                    |
   |                         | NFS4ERR_RETRY_UNCACHED_REP,             |
   |                         | NFS4ERR_TOO_MANY_OPS                    |
   | CB_SEQUENCE             | NFS4ERR_BADSESSION, NFS4ERR_BADSLOT,    |
   |                         | NFS4ERR_BADXDR, NFS4ERR_BAD_HIGH_SLOT,  |
   |                         | NFS4ERR_CONN_NOT_BOUND_TO_SESSION,      |
   |                         | NFS4ERR_DELAY, NFS4ERR_REP_TOO_BIG,     |
   |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,           |
   |                         | NFS4ERR_REQ_TOO_BIG,                    |
   |                         | NFS4ERR_RETRY_UNCACHED_REP,             |
   |                         | NFS4ERR_SEQUENCE_POS,                   |
   |                         | NFS4ERR_SEQ_FALSE_RETRY,                |
   |                         | NFS4ERR_SEQ_MISORDERED,                 |
   |                         | NFS4ERR_TOO_MANY_OPS                    |
   | CB_WANTS_CANCELLED      | NFS4ERR_BADXDR, NFS4ERR_DELAY,          |
   |                         | NFS4ERR_NOTSUPP,                        |
   |                         | NFS4ERR_OP_NOT_IN_SESSION,              |
   |                         | NFS4ERR_REP_TOO_BIG,                    |
   |                         | NFS4ERR_REP_TOO_BIG_TO_CACHE,           |
   |                         | NFS4ERR_REQ_TOO_BIG,                    |
   |                         | NFS4ERR_RETRY_UNCACHED_REP,             |
   |                         | NFS4ERR_SERVERFAULT,                    |
   |                         | NFS4ERR_TOO_MANY_OPS                    |
   +-------------------------+-----------------------------------------+

Table 7 🧚

15.4. Errors and the Operations That Use Them 🦋

+-----------------------------------+-------------------------------+
| Error                             | Operations                    |
+-----------------------------------+-------------------------------+
| NFS4ERR_ACCESS                    | ACCESS, COMMIT, CREATE,       |
|                                   | GETATTR, GET_DIR_DELEGATION,  |
|                                   | LAYOUTCOMMIT, LAYOUTGET,      |
|                                   | LINK, LOCK, LOCKT, LOCKU,     |
|                                   | LOOKUP, LOOKUPP, NVERIFY,     |
|                                   | OPEN, OPENATTR, READ,         |
|                                   | READDIR, READLINK, REMOVE,    |
|                                   | RENAME, SECINFO,              |
|                                   | SECINFO_NO_NAME, SETATTR,     |
|                                   | VERIFY, WRITE                 |
| NFS4ERR_ADMIN_REVOKED             | CLOSE, DELEGRETURN,           |
|                                   | LAYOUTCOMMIT, LAYOUTGET,      |
|                                   | LAYOUTRETURN, LOCK, LOCKU,    |
|                                   | OPEN, OPEN_DOWNGRADE, READ,   |
|                                   | SETATTR, WRITE                |
| NFS4ERR_ATTRNOTSUPP               | CREATE, LAYOUTCOMMIT,         |
|                                   | NVERIFY, OPEN, SETATTR,       |
|                                   | VERIFY                        |
| NFS4ERR_BACK_CHAN_BUSY            | DESTROY_SESSION               |
| NFS4ERR_BADCHAR                   | CREATE, EXCHANGE_ID, LINK,    |
|                                   | LOOKUP, NVERIFY, OPEN,        |
|                                   | REMOVE, RENAME, SECINFO,      |
|                                   | SETATTR, VERIFY               |
| NFS4ERR_BADHANDLE                 | CB_GETATTR, CB_LAYOUTRECALL,  |
|                                   | CB_NOTIFY, CB_NOTIFY_LOCK,    |
|                                   | CB_PUSH_DELEG, CB_RECALL,     |
|                                   | PUTFH                         |
| NFS4ERR_BADIOMODE                 | CB_LAYOUTRECALL,              |
|                                   | LAYOUTCOMMIT, LAYOUTGET       |
| NFS4ERR_BADLAYOUT                 | LAYOUTCOMMIT, LAYOUTGET       |
| NFS4ERR_BADNAME                   | CREATE, LINK, LOOKUP, OPEN,   |
|                                   | REMOVE, RENAME, SECINFO       |
| NFS4ERR_BADOWNER                  | CREATE, OPEN, SETATTR         |
| NFS4ERR_BADSESSION                | BIND_CONN_TO_SESSION,         |
|                                   | CB_SEQUENCE, DESTROY_SESSION, |
|                                   | SEQUENCE                      |
| NFS4ERR_BADSLOT                   | CB_SEQUENCE, SEQUENCE         |
| NFS4ERR_BADTYPE                   | CREATE                        |
| NFS4ERR_BADXDR                    | ACCESS, BACKCHANNEL_CTL,      |
|                                   | BIND_CONN_TO_SESSION,         |
|                                   | CB_GETATTR, CB_ILLEGAL,       |
|                                   | CB_LAYOUTRECALL, CB_NOTIFY,   |
|                                   | CB_NOTIFY_DEVICEID,           |
|                                   | CB_NOTIFY_LOCK,               |
|                                   | CB_PUSH_DELEG, CB_RECALL,     |
|                                   | CB_RECALLABLE_OBJ_AVAIL,      |
|                                   | CB_RECALL_ANY,                |
|                                   | CB_RECALL_SLOT, CB_SEQUENCE,  |
|                                   | CB_WANTS_CANCELLED, CLOSE,    |
|                                   | COMMIT, CREATE,               |
|                                   | CREATE_SESSION, DELEGPURGE,   |
|                                   | DELEGRETURN,                  |
|                                   | DESTROY_CLIENTID,             |
|                                   | DESTROY_SESSION, EXCHANGE_ID, |
|                                   | FREE_STATEID, GETATTR,        |
|                                   | GETDEVICEINFO, GETDEVICELIST, |
|                                   | GET_DIR_DELEGATION, ILLEGAL,  |
|                                   | LAYOUTCOMMIT, LAYOUTGET,      |
|                                   | LAYOUTRETURN, LINK, LOCK,     |
|                                   | LOCKT, LOCKU, LOOKUP,         |
|                                   | NVERIFY, OPEN, OPENATTR,      |
|                                   | OPEN_DOWNGRADE, PUTFH, READ,  |
|                                   | READDIR, RECLAIM_COMPLETE,    |
|                                   | REMOVE, RENAME, SECINFO,      |
|                                   | SECINFO_NO_NAME, SEQUENCE,    |
|                                   | SETATTR, SET_SSV,             |
|                                   | TEST_STATEID, VERIFY,         |
|                                   | WANT_DELEGATION, WRITE        |
| NFS4ERR_BAD_COOKIE                | GETDEVICELIST, READDIR        |
| NFS4ERR_BAD_HIGH_SLOT             | CB_RECALL_SLOT, CB_SEQUENCE,  |
|                                   | SEQUENCE                      |
| NFS4ERR_BAD_RANGE                 | LOCK, LOCKT, LOCKU            |
| NFS4ERR_BAD_SESSION_DIGEST        | BIND_CONN_TO_SESSION, SET_SSV |
| NFS4ERR_BAD_STATEID               | CB_LAYOUTRECALL, CB_NOTIFY,   |
|                                   | CB_NOTIFY_LOCK, CB_RECALL,    |
|                                   | CLOSE, DELEGRETURN,           |
|                                   | FREE_STATEID, LAYOUTGET,      |
|                                   | LAYOUTRETURN, LOCK, LOCKU,    |
|                                   | OPEN, OPEN_DOWNGRADE, READ,   |
|                                   | SETATTR, WRITE                |
| NFS4ERR_CB_PATH_DOWN              | DESTROY_SESSION               |
| NFS4ERR_CLID_INUSE                | CREATE_SESSION, EXCHANGE_ID   |
| NFS4ERR_CLIENTID_BUSY             | DESTROY_CLIENTID              |
| NFS4ERR_COMPLETE_ALREADY          | RECLAIM_COMPLETE              |
| NFS4ERR_CONN_NOT_BOUND_TO_SESSION | CB_SEQUENCE, DESTROY_SESSION, |
|                                   | SEQUENCE                      |
| NFS4ERR_DEADLOCK                  | LOCK                          |
| NFS4ERR_DEADSESSION               | ACCESS, BACKCHANNEL_CTL,      |
|                                   | BIND_CONN_TO_SESSION, CLOSE,  |
|                                   | COMMIT, CREATE,               |
|                                   | CREATE_SESSION, DELEGPURGE,   |
|                                   | DELEGRETURN,                  |
|                                   | DESTROY_CLIENTID,             |
|                                   | DESTROY_SESSION, EXCHANGE_ID, |
|                                   | FREE_STATEID, GETATTR,        |
|                                   | GETDEVICEINFO, GETDEVICELIST, |
|                                   | GET_DIR_DELEGATION,           |
|                                   | LAYOUTCOMMIT, LAYOUTGET,      |
|                                   | LAYOUTRETURN, LINK, LOCK,     |
|                                   | LOCKT, LOCKU, LOOKUP,         |
|                                   | LOOKUPP, NVERIFY, OPEN,       |
|                                   | OPENATTR, OPEN_DOWNGRADE,     |
|                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
|                                   | READ, READDIR, READLINK,      |
|                                   | RECLAIM_COMPLETE, REMOVE,     |
|                                   | RENAME, RESTOREFH, SAVEFH,    |
|                                   | SECINFO, SECINFO_NO_NAME,     |
|                                   | SEQUENCE, SETATTR, SET_SSV,   |
|                                   | TEST_STATEID, VERIFY,         |
|                                   | WANT_DELEGATION, WRITE        |
| NFS4ERR_DELAY                     | ACCESS, BACKCHANNEL_CTL,      |
|                                   | BIND_CONN_TO_SESSION,         |
|                                   | CB_GETATTR, CB_LAYOUTRECALL,  |
|                                   | CB_NOTIFY,                    |
|                                   | CB_NOTIFY_DEVICEID,           |
|                                   | CB_NOTIFY_LOCK,               |
|                                   | CB_PUSH_DELEG, CB_RECALL,     |
|                                   | CB_RECALLABLE_OBJ_AVAIL,      |
|                                   | CB_RECALL_ANY,                |
|                                   | CB_RECALL_SLOT, CB_SEQUENCE,  |
|                                   | CB_WANTS_CANCELLED, CLOSE,    |
|                                   | COMMIT, CREATE,               |
|                                   | CREATE_SESSION, DELEGPURGE,   |
|                                   | DELEGRETURN,                  |
|                                   | DESTROY_CLIENTID,             |
|                                   | DESTROY_SESSION, EXCHANGE_ID, |
|                                   | FREE_STATEID, GETATTR,        |
|                                   | GETDEVICEINFO, GETDEVICELIST, |
|                                   | GET_DIR_DELEGATION,           |
|                                   | LAYOUTCOMMIT, LAYOUTGET,      |
|                                   | LAYOUTRETURN, LINK, LOCK,     |
|                                   | LOCKT, LOCKU, LOOKUP,         |
|                                   | LOOKUPP, NVERIFY, OPEN,       |
|                                   | OPENATTR, OPEN_DOWNGRADE,     |
|                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
|                                   | READ, READDIR, READLINK,      |
|                                   | RECLAIM_COMPLETE, REMOVE,     |
|                                   | RENAME, SECINFO,              |
|                                   | SECINFO_NO_NAME, SEQUENCE,    |
|                                   | SETATTR, SET_SSV,             |
|                                   | TEST_STATEID, VERIFY,         |
|                                   | WANT_DELEGATION, WRITE        |
| NFS4ERR_DELEG_ALREADY_WANTED      | OPEN, WANT_DELEGATION         |
| NFS4ERR_DELEG_REVOKED             | DELEGRETURN, LAYOUTCOMMIT,    |
|                                   | LAYOUTGET, LAYOUTRETURN,      |
|                                   | OPEN, READ, SETATTR, WRITE    |
| NFS4ERR_DENIED                    | LOCK, LOCKT                   |
| NFS4ERR_DIRDELEG_UNAVAIL          | GET_DIR_DELEGATION            |
| NFS4ERR_DQUOT                     | CREATE, LAYOUTGET, LINK,      |
|                                   | OPEN, OPENATTR, RENAME,       |
|                                   | SETATTR, WRITE                |
| NFS4ERR_ENCR_ALG_UNSUPP           | EXCHANGE_ID                   |
| NFS4ERR_EXIST                     | CREATE, LINK, OPEN, RENAME    |
| NFS4ERR_EXPIRED                   | CLOSE, DELEGRETURN,           |
|                                   | LAYOUTCOMMIT, LAYOUTRETURN,   |
|                                   | LOCK, LOCKU, OPEN,            |
|                                   | OPEN_DOWNGRADE, READ,         |
|                                   | SETATTR, WRITE                |
| NFS4ERR_FBIG                      | LAYOUTCOMMIT, OPEN, SETATTR,  |
|                                   | WRITE                         |
| NFS4ERR_FHEXPIRED                 | ACCESS, CLOSE, COMMIT,        |

bestie, the table gets cut off right at NFS4ERR_FHEXPIRED 🥺 so the rest of that row and everything after it picks up in the next part 🫶✨

okay bestie, the table is still going and it's giving chaotic 💅

   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SETATTR, SET_SSV,             |
   |                                   | TEST_STATEID, VERIFY,         |
   |                                   | WANT_DELEGATION, WRITE        |
   | NFS4ERR_PERM                      | CREATE, OPEN, SETATTR         |
   | NFS4ERR_PNFS_IO_HOLE              | READ, WRITE                   |
   | NFS4ERR_PNFS_NO_LAYOUT            | READ, WRITE                   |
   | NFS4ERR_RECALLCONFLICT            | LAYOUTGET, WANT_DELEGATION    |
   | NFS4ERR_RECLAIM_BAD               | LAYOUTCOMMIT, LOCK, OPEN,     |
   |                                   | WANT_DELEGATION               |
   | NFS4ERR_RECLAIM_CONFLICT          | LAYOUTCOMMIT, LOCK, OPEN,     |
   |                                   | WANT_DELEGATION               |
   | NFS4ERR_REJECT_DELEG              | CB_PUSH_DELEG                 |
   | NFS4ERR_REP_TOO_BIG               | ACCESS, BACKCHANNEL_CTL,      |
   |                                   | BIND_CONN_TO_SESSION,         |
   |                                   | CB_GETATTR, CB_LAYOUTRECALL,  |
   |                                   | CB_NOTIFY,                    |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG, CB_RECALL,     |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_RECALL_ANY,                |
   |                                   | CB_RECALL_SLOT, CB_SEQUENCE,  |
   |                                   | CB_WANTS_CANCELLED, CLOSE,    |
   |                                   | COMMIT, CREATE,               |
   |                                   | CREATE_SESSION, DELEGPURGE,   |
   |                                   | DELEGRETURN,                  |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION, EXCHANGE_ID, |
   |                                   | FREE_STATEID, GETATTR,        |
   |                                   | GETDEVICEINFO, GETDEVICELIST, |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SEQUENCE, SETATTR, SET_SSV,   |
   |                                   | TEST_STATEID, VERIFY,         |
   |                                   | WANT_DELEGATION, WRITE        |
   | NFS4ERR_REP_TOO_BIG_TO_CACHE      | ACCESS, BACKCHANNEL_CTL,      |
   |                                   | BIND_CONN_TO_SESSION,         |
   |                                   | CB_GETATTR, CB_LAYOUTRECALL,  |
   |                                   | CB_NOTIFY,                    |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG, CB_RECALL,     |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_RECALL_ANY,                |
   |                                   | CB_RECALL_SLOT, CB_SEQUENCE,  |
   |                                   | CB_WANTS_CANCELLED, CLOSE,    |
   |                                   | COMMIT, CREATE,               |
   |                                   | CREATE_SESSION, DELEGPURGE,   |
   |                                   | DELEGRETURN,                  |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION, EXCHANGE_ID, |
   |                                   | FREE_STATEID, GETATTR,        |
   |                                   | GETDEVICEINFO, GETDEVICELIST, |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SEQUENCE, SETATTR, SET_SSV,   |
   |                                   | TEST_STATEID, VERIFY,         |
   |                                   | WANT_DELEGATION, WRITE        |
   | NFS4ERR_REQ_TOO_BIG               | ACCESS, BACKCHANNEL_CTL,      |
   |                                   | BIND_CONN_TO_SESSION,         |
   |                                   | CB_GETATTR, CB_LAYOUTRECALL,  |
   |                                   | CB_NOTIFY,                    |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG, CB_RECALL,     |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_RECALL_ANY,                |
   |                                   | CB_RECALL_SLOT, CB_SEQUENCE,  |
   |                                   | CB_WANTS_CANCELLED, CLOSE,    |
   |                                   | COMMIT, CREATE,               |
   |                                   | CREATE_SESSION, DELEGPURGE,   |
   |                                   | DELEGRETURN,                  |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION, EXCHANGE_ID, |
   |                                   | FREE_STATEID, GETATTR,        |
   |                                   | GETDEVICEINFO, GETDEVICELIST, |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SEQUENCE, SETATTR, SET_SSV,   |
   |                                   | TEST_STATEID, VERIFY,         |
   |                                   | WANT_DELEGATION, WRITE        |
   | NFS4ERR_RETRY_UNCACHED_REP        | ACCESS, BACKCHANNEL_CTL,      |
   |                                   | BIND_CONN_TO_SESSION,         |
   |                                   | CB_GETATTR, CB_LAYOUTRECALL,  |
   |                                   | CB_NOTIFY,                    |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG, CB_RECALL,     |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_RECALL_ANY,                |
   |                                   | CB_RECALL_SLOT, CB_SEQUENCE,  |
   |                                   | CB_WANTS_CANCELLED, CLOSE,    |
   |                                   | COMMIT, CREATE,               |
   |                                   | CREATE_SESSION, DELEGPURGE,   |
   |                                   | DELEGRETURN,                  |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION, EXCHANGE_ID, |
   |                                   | FREE_STATEID, GETATTR,        |
   |                                   | GETDEVICEINFO, GETDEVICELIST, |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SEQUENCE, SETATTR, SET_SSV,   |
   |                                   | TEST_STATEID, VERIFY,         |
   |                                   | WANT_DELEGATION, WRITE        |
   | NFS4ERR_ROFS                      | CREATE, LINK, LOCK, LOCKT,    |
   |                                   | OPEN, OPENATTR,               |
   |                                   | OPEN_DOWNGRADE, REMOVE,       |
   |                                   | RENAME, SETATTR, WRITE        |
   | NFS4ERR_SAME                      | NVERIFY                       |
   | NFS4ERR_SEQUENCE_POS              | CB_SEQUENCE, SEQUENCE         |
   | NFS4ERR_SEQ_FALSE_RETRY           | CB_SEQUENCE, SEQUENCE         |
   | NFS4ERR_SEQ_MISORDERED            | CB_SEQUENCE, CREATE_SESSION,  |
   |                                   | SEQUENCE                      |
   | NFS4ERR_SERVERFAULT               | ACCESS, BIND_CONN_TO_SESSION, |
   |                                   | CB_GETATTR, CB_NOTIFY,        |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG, CB_RECALL,     |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_WANTS_CANCELLED, CLOSE,    |

   |                                   | COMMIT, CREATE,               |
   |                                   | CREATE_SESSION, DELEGPURGE,   |
   |                                   | DELEGRETURN,                  |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION, EXCHANGE_ID, |
   |                                   | FREE_STATEID, GETATTR,        |
   |                                   | GETDEVICEINFO, GETDEVICELIST, |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKU, LOOKUP, LOOKUPP,       |
   |                                   | NVERIFY, OPEN, OPENATTR,      |
   |                                   | OPEN_DOWNGRADE, PUTFH,        |
   |                                   | PUTPUBFH, PUTROOTFH, READ,    |
   |                                   | READDIR, READLINK,            |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SETATTR, TEST_STATEID,        |
   |                                   | VERIFY, WANT_DELEGATION,      |
   |                                   | WRITE                         |
   | NFS4ERR_SHARE_DENIED              | OPEN                          |
   | NFS4ERR_STALE                     | ACCESS, CLOSE, COMMIT,        |
   |                                   | CREATE, DELEGRETURN, GETATTR, |
   |                                   | GETFH, GET_DIR_DELEGATION,    |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, READ, READDIR,         |
   |                                   | READLINK, RECLAIM_COMPLETE,   |
   |                                   | REMOVE, RENAME, RESTOREFH,    |
   |                                   | SAVEFH, SECINFO,              |
   |                                   | SECINFO_NO_NAME, SETATTR,     |
   |                                   | VERIFY, WANT_DELEGATION,      |
   |                                   | WRITE                         |
   | NFS4ERR_STALE_CLIENTID            | CREATE_SESSION,               |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION               |
   | NFS4ERR_SYMLINK                   | COMMIT, LAYOUTCOMMIT, LINK,   |
   |                                   | LOCK, LOCKT, LOOKUP, LOOKUPP, |
   |                                   | OPEN, READ, WRITE             |
   | NFS4ERR_TOOSMALL                  | CREATE_SESSION,               |
   |                                   | GETDEVICEINFO, LAYOUTGET,     |
   |                                   | READDIR                       |
   | NFS4ERR_TOO_MANY_OPS              | ACCESS, BACKCHANNEL_CTL,      |
   |                                   | BIND_CONN_TO_SESSION,         |
   |                                   | CB_GETATTR, CB_LAYOUTRECALL,  |
   |                                   | CB_NOTIFY,                    |
   |                                   | CB_NOTIFY_DEVICEID,           |
   |                                   | CB_NOTIFY_LOCK,               |
   |                                   | CB_PUSH_DELEG, CB_RECALL,     |
   |                                   | CB_RECALLABLE_OBJ_AVAIL,      |
   |                                   | CB_RECALL_ANY,                |
   |                                   | CB_RECALL_SLOT, CB_SEQUENCE,  |
   |                                   | CB_WANTS_CANCELLED, CLOSE,    |
   |                                   | COMMIT, CREATE,               |
   |                                   | CREATE_SESSION, DELEGPURGE,   |
   |                                   | DELEGRETURN,                  |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION, EXCHANGE_ID, |
   |                                   | FREE_STATEID, GETATTR,        |
   |                                   | GETDEVICEINFO, GETDEVICELIST, |
   |                                   | GET_DIR_DELEGATION,           |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, LOCKU, LOOKUP,         |
   |                                   | LOOKUPP, NVERIFY, OPEN,       |
   |                                   | OPENATTR, OPEN_DOWNGRADE,     |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | READ, READDIR, READLINK,      |
   |                                   | RECLAIM_COMPLETE, REMOVE,     |
   |                                   | RENAME, RESTOREFH, SAVEFH,    |
   |                                   | SECINFO, SECINFO_NO_NAME,     |
   |                                   | SEQUENCE, SETATTR, SET_SSV,   |
   |                                   | TEST_STATEID, VERIFY,         |
   |                                   | WANT_DELEGATION, WRITE        |
   | NFS4ERR_UNKNOWN_LAYOUTTYPE        | CB_LAYOUTRECALL,              |
   |                                   | GETDEVICEINFO, GETDEVICELIST, |
   |                                   | LAYOUTCOMMIT, LAYOUTGET,      |
   |                                   | LAYOUTRETURN, NVERIFY,        |
   |                                   | SETATTR, VERIFY               |
   | NFS4ERR_UNSAFE_COMPOUND           | CREATE, OPEN, OPENATTR        |
   | NFS4ERR_WRONGSEC                  | LINK, LOOKUP, LOOKUPP, OPEN,  |
   |                                   | PUTFH, PUTPUBFH, PUTROOTFH,   |
   |                                   | RENAME, RESTOREFH             |
   | NFS4ERR_WRONG_CRED                | CLOSE, CREATE_SESSION,        |
   |                                   | DELEGPURGE, DELEGRETURN,      |
   |                                   | DESTROY_CLIENTID,             |
   |                                   | DESTROY_SESSION,              |
   |                                   | FREE_STATEID, LAYOUTCOMMIT,   |
   |                                   | LAYOUTRETURN, LOCK, LOCKT,    |
   |                                   | LOCKU, OPEN_DOWNGRADE,        |
   |                                   | RECLAIM_COMPLETE              |
   | NFS4ERR_WRONG_TYPE                | CB_LAYOUTRECALL,              |
   |                                   | CB_PUSH_DELEG, COMMIT,        |
   |                                   | GETATTR, LAYOUTGET,           |
   |                                   | LAYOUTRETURN, LINK, LOCK,     |
   |                                   | LOCKT, NVERIFY, OPEN,         |
   |                                   | OPENATTR, READ, READLINK,     |
   |                                   | RECLAIM_COMPLETE, SETATTR,    |
   |                                   | VERIFY, WANT_DELEGATION,      |
   |                                   | WRITE                         |
   | NFS4ERR_XDEV                      | LINK, RENAME                  |
   +-----------------------------------+-------------------------------+

                                  Table 8

16. NFSv4.1 Procedures ✨

Okay so bestie, here's the deal: both procedures, NULL and COMPOUND, MUST be implemented. No skipping either one, periodt 💅

16.1. Procedure 0: NULL - No Operation 🌸

16.1.1. ARGUMENTS

void;

16.1.2. RESULTS

void;

16.1.3. DESCRIPTION

Okay this one is the classic NULL procedure 🫶 It takes a void argument, gives back a void response, and literally does nothing. That's the whole point! People sometimes use it to measure how much overhead it costs to process a request, so the server SHOULD make sure it doesn't do any unnecessary work when handling it. Lowkey genius, no cap 🧚

16.1.4. ERRORS

None. Nothing can go wrong when you do nothing, iykyk 🙈

16.2. Procedure 1: COMPOUND - Compound Operations 🦋

16.2.1. ARGUMENTS ✨

okay bestie, this is the full menu of ops for COMPOUND 💅 every single one gets its own little slot in the enum, and the union is how each op carries its own args 🎀

   enum nfs_opnum4 {
    OP_ACCESS              = 3,
    OP_CLOSE               = 4,
    OP_COMMIT              = 5,
    OP_CREATE              = 6,
    OP_DELEGPURGE          = 7,
    OP_DELEGRETURN         = 8,
    OP_GETATTR             = 9,
    OP_GETFH               = 10,
    OP_LINK                = 11,
    OP_LOCK                = 12,
    OP_LOCKT               = 13,
    OP_LOCKU               = 14,
    OP_LOOKUP              = 15,
    OP_LOOKUPP             = 16,
    OP_NVERIFY             = 17,
    OP_OPEN                = 18,
    OP_OPENATTR            = 19,
    OP_OPEN_CONFIRM        = 20, /* Mandatory not-to-implement */
    OP_OPEN_DOWNGRADE      = 21,
    OP_PUTFH               = 22,
    OP_PUTPUBFH            = 23,
    OP_PUTROOTFH           = 24,
    OP_READ                = 25,
    OP_READDIR             = 26,
    OP_READLINK            = 27,
    OP_REMOVE              = 28,
    OP_RENAME              = 29,
    OP_RENEW               = 30, /* Mandatory not-to-implement */
    OP_RESTOREFH           = 31,
    OP_SAVEFH              = 32,
    OP_SECINFO             = 33,
    OP_SETATTR             = 34,
    OP_SETCLIENTID         = 35, /* Mandatory not-to-implement */
    OP_SETCLIENTID_CONFIRM = 36, /* Mandatory not-to-implement */
    OP_VERIFY              = 37,
    OP_WRITE               = 38,
    OP_RELEASE_LOCKOWNER   = 39, /* Mandatory not-to-implement */

   /* new operations for NFSv4.1 */
    OP_BACKCHANNEL_CTL     = 40,
    OP_BIND_CONN_TO_SESSION = 41,
    OP_EXCHANGE_ID         = 42,
    OP_CREATE_SESSION      = 43,
    OP_DESTROY_SESSION     = 44,
    OP_FREE_STATEID        = 45,
    OP_GET_DIR_DELEGATION  = 46,
    OP_GETDEVICEINFO       = 47,
    OP_GETDEVICELIST       = 48,
    OP_LAYOUTCOMMIT        = 49,
    OP_LAYOUTGET           = 50,
    OP_LAYOUTRETURN        = 51,
    OP_SECINFO_NO_NAME     = 52,
    OP_SEQUENCE            = 53,
    OP_SET_SSV             = 54,
    OP_TEST_STATEID        = 55,
    OP_WANT_DELEGATION     = 56,
    OP_DESTROY_CLIENTID    = 57,
    OP_RECLAIM_COMPLETE    = 58,
    OP_ILLEGAL             = 10044
   };
   union nfs_argop4 switch (nfs_opnum4 argop) {
    case OP_ACCESS:        ACCESS4args opaccess;
    case OP_CLOSE:         CLOSE4args opclose;
    case OP_COMMIT:        COMMIT4args opcommit;
    case OP_CREATE:        CREATE4args opcreate;
    case OP_DELEGPURGE:    DELEGPURGE4args opdelegpurge;
    case OP_DELEGRETURN:   DELEGRETURN4args opdelegreturn;
    case OP_GETATTR:       GETATTR4args opgetattr;
    case OP_GETFH:         void;
    case OP_LINK:          LINK4args oplink;
    case OP_LOCK:          LOCK4args oplock;
    case OP_LOCKT:         LOCKT4args oplockt;
    case OP_LOCKU:         LOCKU4args oplocku;
    case OP_LOOKUP:        LOOKUP4args oplookup;
    case OP_LOOKUPP:       void;
    case OP_NVERIFY:       NVERIFY4args opnverify;
    case OP_OPEN:          OPEN4args opopen;
    case OP_OPENATTR:      OPENATTR4args opopenattr;

    /* Not for NFSv4.1 */
    case OP_OPEN_CONFIRM:  OPEN_CONFIRM4args opopen_confirm;

    case OP_OPEN_DOWNGRADE:
                           OPEN_DOWNGRADE4args opopen_downgrade;

    case OP_PUTFH:         PUTFH4args opputfh;
    case OP_PUTPUBFH:      void;
    case OP_PUTROOTFH:     void;
    case OP_READ:          READ4args opread;
    case OP_READDIR:       READDIR4args opreaddir;
    case OP_READLINK:      void;
    case OP_REMOVE:        REMOVE4args opremove;
    case OP_RENAME:        RENAME4args oprename;

    /* Not for NFSv4.1 */
    case OP_RENEW:         RENEW4args oprenew;

    case OP_RESTOREFH:     void;
    case OP_SAVEFH:        void;
    case OP_SECINFO:       SECINFO4args opsecinfo;
    case OP_SETATTR:       SETATTR4args opsetattr;

    /* Not for NFSv4.1 */
    case OP_SETCLIENTID: SETCLIENTID4args opsetclientid;
    /* Not for NFSv4.1 */
    case OP_SETCLIENTID_CONFIRM: SETCLIENTID_CONFIRM4args
                                   opsetclientid_confirm;
    case OP_VERIFY:        VERIFY4args opverify;
    case OP_WRITE:         WRITE4args opwrite;

    /* Not for NFSv4.1 */
    case OP_RELEASE_LOCKOWNER:
                           RELEASE_LOCKOWNER4args
                           oprelease_lockowner;

    /* Operations new to NFSv4.1 */
    case OP_BACKCHANNEL_CTL:
                           BACKCHANNEL_CTL4args opbackchannel_ctl;

    case OP_BIND_CONN_TO_SESSION:
                           BIND_CONN_TO_SESSION4args
                           opbind_conn_to_session;

    case OP_EXCHANGE_ID:   EXCHANGE_ID4args opexchange_id;

    case OP_CREATE_SESSION:
                           CREATE_SESSION4args opcreate_session;

    case OP_DESTROY_SESSION:
                           DESTROY_SESSION4args opdestroy_session;

    case OP_FREE_STATEID:  FREE_STATEID4args opfree_stateid;

    case OP_GET_DIR_DELEGATION:
                           GET_DIR_DELEGATION4args
                                   opget_dir_delegation;

    case OP_GETDEVICEINFO: GETDEVICEINFO4args opgetdeviceinfo;
    case OP_GETDEVICELIST: GETDEVICELIST4args opgetdevicelist;
    case OP_LAYOUTCOMMIT:  LAYOUTCOMMIT4args oplayoutcommit;
    case OP_LAYOUTGET:     LAYOUTGET4args oplayoutget;
    case OP_LAYOUTRETURN:  LAYOUTRETURN4args oplayoutreturn;

    case OP_SECINFO_NO_NAME:
                           SECINFO_NO_NAME4args opsecinfo_no_name;

    case OP_SEQUENCE:      SEQUENCE4args opsequence;
    case OP_SET_SSV:       SET_SSV4args opset_ssv;
    case OP_TEST_STATEID:  TEST_STATEID4args optest_stateid;

    case OP_WANT_DELEGATION:
                           WANT_DELEGATION4args opwant_delegation;

    case OP_DESTROY_CLIENTID:
                           DESTROY_CLIENTID4args
                                   opdestroy_clientid;

    case OP_RECLAIM_COMPLETE:
                           RECLAIM_COMPLETE4args
                                   opreclaim_complete;

    /* Operations not new to NFSv4.1 */
    case OP_ILLEGAL:       void;
   };


   struct COMPOUND4args {
           utf8str_cs      tag;
           uint32_t        minorversion;
           nfs_argop4      argarray<>;
   };

16.2.2. RESULTS ✨

and now the replies, same vibe 🦋 every op gets a matching result slot, and the whole COMPOUND comes back with a status plus its own tag 🫶

   union nfs_resop4 switch (nfs_opnum4 resop) {
    case OP_ACCESS:        ACCESS4res opaccess;
    case OP_CLOSE:         CLOSE4res opclose;
    case OP_COMMIT:        COMMIT4res opcommit;
    case OP_CREATE:        CREATE4res opcreate;
    case OP_DELEGPURGE:    DELEGPURGE4res opdelegpurge;
    case OP_DELEGRETURN:   DELEGRETURN4res opdelegreturn;
    case OP_GETATTR:       GETATTR4res opgetattr;
    case OP_GETFH:         GETFH4res opgetfh;
    case OP_LINK:          LINK4res oplink;
    case OP_LOCK:          LOCK4res oplock;
    case OP_LOCKT:         LOCKT4res oplockt;
    case OP_LOCKU:         LOCKU4res oplocku;
    case OP_LOOKUP:        LOOKUP4res oplookup;
    case OP_LOOKUPP:       LOOKUPP4res oplookupp;
    case OP_NVERIFY:       NVERIFY4res opnverify;
    case OP_OPEN:          OPEN4res opopen;
    case OP_OPENATTR:      OPENATTR4res opopenattr;
    /* Not for NFSv4.1 */
    case OP_OPEN_CONFIRM:  OPEN_CONFIRM4res opopen_confirm;

    case OP_OPEN_DOWNGRADE:
                           OPEN_DOWNGRADE4res
                                   opopen_downgrade;

    case OP_PUTFH:         PUTFH4res opputfh;
    case OP_PUTPUBFH:      PUTPUBFH4res opputpubfh;
    case OP_PUTROOTFH:     PUTROOTFH4res opputrootfh;
    case OP_READ:          READ4res opread;
    case OP_READDIR:       READDIR4res opreaddir;
    case OP_READLINK:      READLINK4res opreadlink;
    case OP_REMOVE:        REMOVE4res opremove;
    case OP_RENAME:        RENAME4res oprename;
    /* Not for NFSv4.1 */
    case OP_RENEW:         RENEW4res oprenew;
    case OP_RESTOREFH:     RESTOREFH4res oprestorefh;
    case OP_SAVEFH:        SAVEFH4res opsavefh;
    case OP_SECINFO:       SECINFO4res opsecinfo;
    case OP_SETATTR:       SETATTR4res opsetattr;
    /* Not for NFSv4.1 */
    case OP_SETCLIENTID: SETCLIENTID4res opsetclientid;

    /* Not for NFSv4.1 */
    case OP_SETCLIENTID_CONFIRM:
                           SETCLIENTID_CONFIRM4res
                                   opsetclientid_confirm;
    case OP_VERIFY:        VERIFY4res opverify;
    case OP_WRITE:         WRITE4res opwrite;

    /* Not for NFSv4.1 */
    case OP_RELEASE_LOCKOWNER:
                           RELEASE_LOCKOWNER4res
                                   oprelease_lockowner;

    /* Operations new to NFSv4.1 */

    case OP_BACKCHANNEL_CTL:
                           BACKCHANNEL_CTL4res
                                   opbackchannel_ctl;

    case OP_BIND_CONN_TO_SESSION:
                           BIND_CONN_TO_SESSION4res
                                    opbind_conn_to_session;

    case OP_EXCHANGE_ID:   EXCHANGE_ID4res opexchange_id;

    case OP_CREATE_SESSION:
                           CREATE_SESSION4res
                                   opcreate_session;

    case OP_DESTROY_SESSION:
                           DESTROY_SESSION4res
                                   opdestroy_session;

    case OP_FREE_STATEID:  FREE_STATEID4res
                                   opfree_stateid;

    case OP_GET_DIR_DELEGATION:
                           GET_DIR_DELEGATION4res
                                   opget_dir_delegation;

    case OP_GETDEVICEINFO: GETDEVICEINFO4res
                                   opgetdeviceinfo;

    case OP_GETDEVICELIST: GETDEVICELIST4res
                                   opgetdevicelist;

    case OP_LAYOUTCOMMIT:  LAYOUTCOMMIT4res oplayoutcommit;
    case OP_LAYOUTGET:     LAYOUTGET4res oplayoutget;
    case OP_LAYOUTRETURN:  LAYOUTRETURN4res oplayoutreturn;

    case OP_SECINFO_NO_NAME:
                           SECINFO_NO_NAME4res
                                   opsecinfo_no_name;

    case OP_SEQUENCE:      SEQUENCE4res opsequence;
    case OP_SET_SSV:       SET_SSV4res opset_ssv;
    case OP_TEST_STATEID:  TEST_STATEID4res optest_stateid;

    case OP_WANT_DELEGATION:
                           WANT_DELEGATION4res
                                   opwant_delegation;

    case OP_DESTROY_CLIENTID:

                           DESTROY_CLIENTID4res
                                   opdestroy_clientid;

    case OP_RECLAIM_COMPLETE:
                           RECLAIM_COMPLETE4res
                                   opreclaim_complete;

    /* Operations not new to NFSv4.1 */
    case OP_ILLEGAL:       ILLEGAL4res opillegal;
   };


   struct COMPOUND4res {
           nfsstat4        status;
           utf8str_cs      tag;
           nfs_resop4      resarray<>;
   };

16.2.3. DESCRIPTION 📦✨

okay so COMPOUND is the procedure that bundles one or more NFSv4 operations into a single RPC request 💅. The server works through them in turn. If an operation runs and its status is NFS4_OK, the next operation in the COMPOUND gets executed 🫶. This keeps going until there's nothing left to run, or until some operation returns a status other than NFS4_OK 😭.

The server might realize it doesn't have the resources to run all or some of the operations in a COMPOUND sequence. See Section 2.10.6.4 for the full tea ☕.

The server will usually pick between two ways of decoding the client's request 🔍. Method one is the traditional one-pass XDR decode. If that hits an XDR decoding error, the RPC XDR decode error is returned 💀. Method two is to make an initial pass that decodes the basic COMPOUND request, and then XDR decode the individual operations. The most interesting one here is decoding attributes ✨. With this method, the server might run into an XDR decode error during the second pass. If it does, the server returns NFS4ERR_BADXDR to signal the decode error 🙈.

The COMPOUND arguments have a minorversion field 🎀. For NFSv4.1, this field's value is 1. If the server receives a COMPOUND procedure with a minorversion value it doesn't support, the server MUST return an error of NFS4ERR_MINOR_VERS_MISMATCH and a zero-length resultdata array 🚫.

The COMPOUND results have a status field 💜. If the results array length is non-zero, this status must match the status of the last operation that was executed inside the COMPOUND procedure. So if an operation hit an error, the status value is the same error value being returned for the operation that failed.

Operations zero and one aren't defined for the COMPOUND procedure 🙅‍♀️. Operation 2 is undefined too, and it's reserved for future definition and use with minor versioning 🌸. If the server gets an operation array containing operation 2 and the minorversion field is zero, the server returns NFS4ERR_OP_ILLEGAL to the client, as described in the next paragraph. If an operation array contains operation 2, the minorversion field is non-zero, and the server doesn't support that minor version, the server returns NFS4ERR_MINOR_VERS_MISMATCH. So NFS4ERR_MINOR_VERS_MISMATCH takes precedence over all other errors, periodt 👑.

The server could also receive a request with an operation that's less than the first legal operation (OP_ACCESS) or greater than the last legal operation (OP_RELEASE_LOCKOWNER) 🧚. In that case, the server's response encodes the opcode OP_ILLEGAL instead of the illegal opcode from the request. The status field in the ILLEGAL return results is set to NFS4ERR_OP_ILLEGAL. The COMPOUND procedure's return results are NFS4ERR_OP_ILLEGAL too 💀.

The definition of the "tag" in the request is left up to the implementor 🦋. It can be used to summarize the content of the Compound request for packet-sniffers and engineers debugging implementations, which is so helpful 🍓. However, the value of "tag" in the response SHOULD be the same value as provided in the request. This applies to the tag field of the CB_COMPOUND procedure as well 🫶.

16.2.3.1. Current Filehandle and Stateid 🌈

The COMPOUND procedure gives you a simple environment for running the operations the client specifies 💖. The first two relate to the filehandle, and the second two relate to the current stateid.

16.2.3.1.1. Current Filehandle 🗂️

The current and saved filehandles get used all over the protocol 🔥. Most operations use the current filehandle implicitly as an argument, and lots of them set the current filehandle as part of their results. Mixing client-specified sequences of operations with current and saved filehandle arguments and results gives you a lot of flexibility 🦋. The best or easiest example of current filehandle usage is a sequence like this:

     PUTFH fh1              {fh1}
     LOOKUP "compA"         {fh2}
     GETATTR                {fh2}
     LOOKUP "compB"         {fh3}
     GETATTR                {fh3}
     LOOKUP "compC"         {fh4}
     GETATTR                {fh4}
     GETFH

                             Figure 2

In this example, the PUTFH (Section 18.19) operation explicitly sets the current filehandle value, while the result of each LOOKUP operation sets the current filehandle value to the resulting file system object 🌟. The client can also slide in GETATTR operations that use the current filehandle as an argument 💅.

The PUTROOTFH (Section 18.21) and PUTPUBFH (Section 18.20) operations also set the current filehandle 🌸. The example above would swap "PUTFH fh1" for PUTROOTFH or PUTPUBFH with no filehandle argument to get the same effect, assuming "compA" sits directly below the root of the namespace 🧚.

There's also a saved filehandle, alongside the current one ✨. The current filehandle gets set as the result of operations like LOOKUP, but the saved filehandle has to be set directly with the SAVEFH operation 🙌. The SAVEFH operation copies the current filehandle value into the saved value. The saved filehandle value gets used together with the current filehandle value for the LINK and RENAME operations 🎀. The RESTOREFH operation copies the saved filehandle value back into the current filehandle value. So the saved filehandle can act as a kind of "scratch" area for the client's series of operations 🧺.

16.2.3.1.2. Current Stateid 🔑

NFSv4.1 added a current stateid and a saved stateid to the COMPOUND processing environment 💜. This lets stateids get passed between operations. The protocol syntax hasn't changed at all, only the semantics of a few operations 🫶.

A "current stateid" is the stateid associated with the current filehandle 🦋. The current stateid can only be changed by an operation that modifies the current filehandle or returns a stateid 🌈.

If an operation returns a stateid, it MUST set the current stateid to the returned value 💖. If an operation sets the current filehandle but doesn't return a stateid, the current stateid MUST be set to the all-zeros special stateid, i.e., (seqid, other) = (0, 0) 🫧. If an operation uses a stateid as an argument but doesn't return one, the current stateid MUST NOT be changed 🚫. For example, PUTFH, PUTROOTFH, and PUTPUBFH change the current server state from {ocfh, (osid)} to {cfh, (0, 0)}, while LOCK changes the current state from {cfh, (osid} to {cfh, (nsid)}. Operations like LOOKUP, which turn a current filehandle and component name into a new current filehandle, also change the current state to {0, 0} 🍓. The SAVEFH and RESTOREFH operations save and restore both the current filehandle and the current stateid as a set 🧚.

The following example is the common case of a simple READ operation with a normal stateid. It shows PUTFH initializing the current stateid to (0, 0). The following READ with stateid (sid1) leaves the current stateid unchanged 💅.

    PUTFH fh1                             - -> {fh1, (0, 0)}
    READ (sid1), 0, 1024      {fh1, (0, 0)} -> {fh1, (0, 0)}

                              Figure 3

This next example does an OPEN with the root filehandle, which generates stateid (sid1) 🔓. The next operation specifies the READ with an argument stateid where (seqid, other) are equal to (1, 0), but the current stateid set by the previous operation is the one actually used when the operation is evaluated 🌸. This allows correct interaction with any existing, potentially conflicting, locks ✨.

    PUTROOTFH                             - -> {fh1, (0, 0)}
    OPEN "compA"              {fh1, (0, 0)} -> {fh2, (sid1)}
    READ (1, 0), 0, 1024      {fh2, (sid1)} -> {fh2, (sid1)}
    CLOSE (1, 0)              {fh2, (sid1)} -> {fh2, (sid2)}

                              Figure 4

This next example is similar to the second one, in how it passes the stateid sid2 generated by the LOCK operation to the next READ operation 🎀. This lets the client explicitly wrap a single I/O operation in a lock and its matching stateid, to guarantee correctness with other client locks 🔒. The example also shows how SAVEFH and RESTOREFH can save a filehandle and stateid, then reuse them later as the current filehandle and stateid for a READ operation 🧺.

    PUTFH fh1                             - -> {fh1, (0, 0)}
    LOCK 0, 1024, (sid1)      {fh1, (sid1)} -> {fh1, (sid2)}
    READ (1, 0), 0, 1024      {fh1, (sid2)} -> {fh1, (sid2)}
    LOCKU 0, 1024, (1, 0)     {fh1, (sid2)} -> {fh1, (sid3)}
    SAVEFH                    {fh1, (sid3)} -> {fh1, (sid3)}

    PUTFH fh2                 {fh1, (sid3)} -> {fh2, (0, 0)}
    WRITE (1, 0), 0, 1024     {fh2, (0, 0)} -> {fh2, (0, 0)}

    RESTOREFH                 {fh2, (0, 0)} -> {fh1, (sid3)}
    READ (1, 0), 1024, 1024   {fh1, (sid3)} -> {fh1, (sid3)}

                              Figure 5

The final example shows a disallowed use of the current stateid 🙅‍♀️. The client is trying to implicitly pass the anonymous special stateid, (0,0), to the READ operation. The server MUST return NFS4ERR_BAD_STATEID in the reply to the READ operation 😭.

    PUTFH fh1                             - -> {fh1, (0, 0)}
    READ (1, 0), 0, 1024      {fh1, (0, 0)} -> NFS4ERR_BAD_STATEID

                              Figure 6

16.2.4. ERRORS 🚨

COMPOUND will of course return every error that each operation on the fore channel can return (see Table 6) 💅. But if COMPOUND returns zero operations, the error COMPOUND returns obviously has nothing to do with an error from an operation 🙈. Here's the list of errors COMPOUND will return when it processes zero operations:

                       COMPOUND Error Returns

+------------------------------+------------------------------------+
| Error                        | Notes                              |
+------------------------------+------------------------------------+
| NFS4ERR_BADCHAR              | The tag argument has a character   |
|                              | the replier does not support.      |
| NFS4ERR_BADXDR               |                                    |
| NFS4ERR_DELAY                |                                    |
| NFS4ERR_INVAL                | The tag argument is not in UTF-8   |
|                              | encoding.                          |
| NFS4ERR_MINOR_VERS_MISMATCH  |                                    |
| NFS4ERR_SERVERFAULT          |                                    |
| NFS4ERR_TOO_MANY_OPS         |                                    |
| NFS4ERR_REP_TOO_BIG          |                                    |
| NFS4ERR_REP_TOO_BIG_TO_CACHE |                                    |
| NFS4ERR_REQ_TOO_BIG          |                                    |
+------------------------------+------------------------------------+

                                Table 9

17. Operations: REQUIRED, RECOMMENDED, or OPTIONAL ✨

okay so these tables are basically the cheat sheet for every NFSv4.1 operation 📋. They show whether each one is REQUIRED, RECOMMENDED, or OPTIONAL to implement, or MUST NOT implement 💅. The MUST NOT implement label is only saved for operations that existed in NFSv4.0 and MUST NOT be implemented in NFSv4.1 🚫.

mostly, the REQUIRED, RECOMMENDED, or OPTIONAL label on operations the client sends is really about what the server has to implement 🎀. The client usually has to implement whatever its operating environment needs 🧚. Like, a read-only NFSv4.1 client has zero reason to implement the WRITE operation, so it doesn't have to 🙈.

the REQUIRED or OPTIONAL label on callback operations the server sends applies to both the client AND the server 👯. Usually the client gets to choose whether to create the backchannel, and the operations it sends on the fore channel are what kick off the server's callbacks 🦋. Partial exception: CB_RECALL_SLOT. The only way a client can dodge supporting that one is by not creating a backchannel 💀.

this is just the summary, so there are subtleties that aren't shown here 🤓. If you're ever unsure what an implementation needs, you absolutely have to check the operation descriptions themselves, along with the other relevant explanatory text in this spec 🔥.

the abbreviations used in the second and third columns of the table are defined like this 🍓:

REQ  REQUIRED to implement

REC  RECOMMEND to implement

OPT  OPTIONAL to implement

MNI  MUST NOT implement

For the NFSv4.1 features that are OPTIONAL 🌸, the operations that support those features are OPTIONAL too, and the server would hand back NFS4ERR_NOTSUPP whenever the client tries to use those operations 💀. If an OPTIONAL feature is supported, a bunch of related operations can become REQUIRED to implement ✨. The third column of the table tells you the feature(s) and whether the operation is REQUIRED or OPTIONAL when that feature is supported 🦋.

the OPTIONAL features and their abbreviations are:

pNFS  Parallel NFS

FDELG  File Delegations
DDELG  Directory Delegations

Operations

+----------------------+------------+--------------+----------------+
| Operation            | REQ, REC,  | Feature      | Definition     |
|                      | OPT, or    | (REQ, REC,   |                |
|                      | MNI        | or OPT)      |                |
+----------------------+------------+--------------+----------------+
| ACCESS               | REQ        |              | Section 18.1   |
| BACKCHANNEL_CTL      | REQ        |              | Section 18.33  |
| BIND_CONN_TO_SESSION | REQ        |              | Section 18.34  |
| CLOSE                | REQ        |              | Section 18.2   |
| COMMIT               | REQ        |              | Section 18.3   |
| CREATE               | REQ        |              | Section 18.4   |
| CREATE_SESSION       | REQ        |              | Section 18.36  |
| DELEGPURGE           | OPT        | FDELG (REQ)  | Section 18.5   |
| DELEGRETURN          | OPT        | FDELG,       | Section 18.6   |
|                      |            | DDELG, pNFS  |                |
|                      |            | (REQ)        |                |
| DESTROY_CLIENTID     | REQ        |              | Section 18.50  |
| DESTROY_SESSION      | REQ        |              | Section 18.37  |
| EXCHANGE_ID          | REQ        |              | Section 18.35  |
| FREE_STATEID         | REQ        |              | Section 18.38  |
| GETATTR              | REQ        |              | Section 18.7   |
| GETDEVICEINFO        | OPT        | pNFS (REQ)   | Section 18.40  |
| GETDEVICELIST        | OPT        | pNFS (OPT)   | Section 18.41  |
| GETFH                | REQ        |              | Section 18.8   |
| GET_DIR_DELEGATION   | OPT        | DDELG (REQ)  | Section 18.39  |
| LAYOUTCOMMIT         | OPT        | pNFS (REQ)   | Section 18.42  |
| LAYOUTGET            | OPT        | pNFS (REQ)   | Section 18.43  |
| LAYOUTRETURN         | OPT        | pNFS (REQ)   | Section 18.44  |
| LINK                 | OPT        |              | Section 18.9   |
| LOCK                 | REQ        |              | Section 18.10  |
| LOCKT                | REQ        |              | Section 18.11  |
| LOCKU                | REQ        |              | Section 18.12  |
| LOOKUP               | REQ        |              | Section 18.13  |
| LOOKUPP              | REQ        |              | Section 18.14  |
| NVERIFY              | REQ        |              | Section 18.15  |
| OPEN                 | REQ        |              | Section 18.16  |
| OPENATTR             | OPT        |              | Section 18.17  |
| OPEN_CONFIRM         | MNI        |              | N/A            |
| OPEN_DOWNGRADE       | REQ        |              | Section 18.18  |
| PUTFH                | REQ        |              | Section 18.19  |
| PUTPUBFH             | REQ        |              | Section 18.20  |
| PUTROOTFH            | REQ        |              | Section 18.21  |
| READ                 | REQ        |              | Section 18.22  |
| READDIR              | REQ        |              | Section 18.23  |
| READLINK             | OPT        |              | Section 18.24  |
| RECLAIM_COMPLETE     | REQ        |              | Section 18.51  |
| RELEASE_LOCKOWNER    | MNI        |              | N/A            |
| REMOVE               | REQ        |              | Section 18.25  |
| RENAME               | REQ        |              | Section 18.26  |
| RENEW                | MNI        |              | N/A            |
| RESTOREFH            | REQ        |              | Section 18.27  |
| SAVEFH               | REQ        |              | Section 18.28  |
| SECINFO              | REQ        |              | Section 18.29  |
| SECINFO_NO_NAME      | REC        | pNFS file    | Section 18.45, |
|                      |            | layout (REQ) | Section 13.12  |
| SEQUENCE             | REQ        |              | Section 18.46  |
| SETATTR              | REQ        |              | Section 18.30  |
| SETCLIENTID          | MNI        |              | N/A            |
| SETCLIENTID_CONFIRM  | MNI        |              | N/A            |
| SET_SSV              | REQ        |              | Section 18.47  |
| TEST_STATEID         | REQ        |              | Section 18.48  |
| VERIFY               | REQ        |              | Section 18.31  |
| WANT_DELEGATION      | OPT        | FDELG (OPT)  | Section 18.49  |
| WRITE                | REQ        |              | Section 18.32  |
+----------------------+------------+--------------+----------------+

Callback Operations

+-------------------------+-----------+-------------+---------------+
| Operation               | REQ, REC, | Feature     | Definition    |
|                         | OPT, or   | (REQ, REC,  |               |
|                         | MNI       | or OPT)     |               |
+-------------------------+-----------+-------------+---------------+
| CB_GETATTR              | OPT       | FDELG (REQ) | Section 20.1  |
| CB_LAYOUTRECALL         | OPT       | pNFS (REQ)  | Section 20.3  |
| CB_NOTIFY               | OPT       | DDELG (REQ) | Section 20.4  |
| CB_NOTIFY_DEVICEID      | OPT       | pNFS (OPT)  | Section 20.12 |
| CB_NOTIFY_LOCK          | OPT       |             | Section 20.11 |
| CB_PUSH_DELEG           | OPT       | FDELG (OPT) | Section 20.5  |
| CB_RECALL               | OPT       | FDELG,      | Section 20.2  |
|                         |           | DDELG, pNFS |               |
|                         |           | (REQ)       |               |
| CB_RECALL_ANY           | OPT       | FDELG,      | Section 20.6  |
|                         |           | DDELG, pNFS |               |
|                         |           | (REQ)       |               |
| CB_RECALL_SLOT          | REQ       |             | Section 20.8  |
| CB_RECALLABLE_OBJ_AVAIL | OPT       | DDELG, pNFS | Section 20.7  |
|                         |           | (REQ)       |               |
| CB_SEQUENCE             | OPT       | FDELG,      | Section 20.9  |
|                         |           | DDELG, pNFS |               |
|                         |           | (REQ)       |               |
| CB_WANTS_CANCELLED      | OPT       | FDELG,      | Section 20.10 |
|                         |           | DDELG, pNFS |               |
|                         |           | (REQ)       |               |
+-------------------------+-----------+-------------+---------------+

---

18. NFSv4.1 Operations 🧚

18.1. Operation 3: ACCESS - Check Access Rights 💅

18.1.1. ARGUMENTS

const ACCESS4_READ      = 0x00000001;
const ACCESS4_LOOKUP    = 0x00000002;
const ACCESS4_MODIFY    = 0x00000004;
const ACCESS4_EXTEND    = 0x00000008;
const ACCESS4_DELETE    = 0x00000010;
const ACCESS4_EXECUTE   = 0x00000020;

struct ACCESS4args {
        /* CURRENT_FH: object */
        uint32_t        access;
};

18.1.2. RESULTS

struct ACCESS4resok {
        uint32_t        supported;
        uint32_t        access;
};

union ACCESS4res switch (nfsstat4 status) {
 case NFS4_OK:
         ACCESS4resok   resok4;
 default:
         void;
};

18.1.3. DESCRIPTION ✨

okay bestie, ACCESS is basically the "wait, am I even allowed to do this??" check 💅 The client packs the rights it wants to check into the bit mask access, and the server checks those permissions. If the status comes back as NFS4_OK, the response has two bit masks: supported 🔍 (the rights the server can verify reliably) and access 🦋 (the rights the user actually has on that filehandle). On success, the current filehandle keeps its value 🌸

Okay but the reply's supported and access fields MUST NOT contain more values than were originally set in the request's access field 🙅‍♀️ Like, if the client only sends ACCESS4_READ and the server supports it, the server MUST NOT set more than ACCESS4_READ in supported, even if it could have reliably checked other stuff 🙈

The reply's access field MUST NOT contain more values than the supported field 💖

The results here are necessarily advisory in nature 🫠 A return of NFS4_OK plus the right bit set doesn't mean that access is guaranteed in the future, because the server can revoke access rights at any time 💀 Plan accordingly, babes.

The following access permissions may be requested:

• ACCESS4_READ: Read data from a file or read a directory 📖
• ACCESS4_LOOKUP: Look up a name in a directory (no meaning for non-directory objects) 🔎
• ACCESS4_MODIFY: Rewrite existing file data or modify existing directory entries ✏️
• ACCESS4_EXTEND: Write new data or add directory entries ➕
• ACCESS4_DELETE: Delete an existing directory entry 🗑️
• ACCESS4_EXECUTE: Execute a regular file (no meaning for a directory) 🚀

On success, the current filehandle retains its value 🎀

ACCESS4_EXECUTE is a challenging semantic to implement, and honestly it's giving main character drama 🎭 because NFS provides remote file access, not remote execution. That leads to the following:

• Whether or not a regular file is executable ought to be the responsibility of the NFS client and not the server. And yet the ACCESS operation is specified to seemingly require a server to own that responsibility 😭
• When a client executes a regular file, it has to read the file from the server. Strictly speaking, the server should not allow the client to read a file being executed unless the user has read permissions on the file. Requiring explicit read permissions on executable files in order to access them over NFS is not going to be acceptable to some users and storage administrators. Historically, NFS servers have allowed a user to READ a file if the user has execute access to the file 🧚

As a practical example, the UNIX specification [52] states that an implementation claiming conformance to UNIX may indicate in the access() programming interface's result that a privileged user has execute rights, even if no execute permission bits are set on the regular file's attributes. It is possible to claim conformance to the UNIX specification and instead not indicate execute rights in that situation, which is true for some operating environments. Suppose the operating environments of the client and server are implementing the access() semantics for privileged users differently, and the ACCESS operation implementations of the client and server follow their respective access() semantics. This can cause undesired behavior 🙃

• Suppose the client's access() interface returns X_OK if the user is privileged and no execute permission bits are set on the regular file's attribute, and the server's access() interface does not return X_OK in that situation. Then the client will be unable to execute files stored on the NFS server that could be executed if stored on a non-NFS file system 🥲
• Suppose the client's access() interface does not return X_OK if the user is privileged, and no execute permission bits are set on the regular file's attribute, and the server's access() interface does return X_OK in that situation. Then:
  • The client will be able to execute files stored on the NFS server that could be executed if stored on a non-NFS file system, unless the client's execution subsystem also checks for execute permission bits 👀
  • Even if the execution subsystem is checking for execute permission bits, there are more potential issues 💅 For example, suppose the client is invoking access() to build a "path search table" of all executable files in the user's "search path", where the path is a list of directories each containing executable files. Suppose there are two files each in separate directories of the search path, such that files have the same component name. In the first directory the file has no execute permission bits set, and in the second directory the file has execute bits set. The path search table will indicate that the first directory has the executable file, but the execute subsystem will fail to execute it. The command shell might fail to try the second file in the second directory. And even if it did, this is a potential performance issue. Clearly, the desired outcome for the client is for the path search table to not contain the first file 🗂️

To deal with the problems described above, the "smart client, stupid server" principle is used 🧠👑 The client owns overall responsibility for determining execute access and relies on the server to parse the execution permissions within the file's mode, acl, and dacl attributes. The rules for the client and server follow:

• If the client is sending ACCESS in order to determine if the user can read the file, the client SHOULD set ACCESS4_READ in the request's access field 📖
• If the client's operating environment only grants execution to the user if the user has execute access according to the execute permissions in the mode, acl, and dacl attributes, then if the client wants to determine execute access, the client SHOULD send an ACCESS request with the ACCESS4_EXECUTE bit set in the request's access field 🚀
• If the client's operating environment grants execution to the user even if the user does not have execute access according to the execute permissions in the mode, acl, and dacl attributes, then if the client wants to determine execute access, it SHOULD send an ACCESS request with both the ACCESS4_EXECUTE and ACCESS4_READ bits set in the request's access field. This way, if any read or execute permission grants the user read or execute access (or if the server interprets the user as privileged), as indicated by the presence of ACCESS4_EXECUTE and/or ACCESS4_READ in the reply's access field, the client will be able to grant the user execute access to the file 🫶
• If the server supports execute permission bits, or some other method for denoting executability (e.g., the suffix of the name of the file might indicate execute), it MUST check only execute permissions, not read permissions, when determining whether or not the reply will have ACCESS4_EXECUTE set in the access field. The server MUST NOT also examine read permission bits when determining whether or not the reply will have ACCESS4_EXECUTE set in the access field. Even if the server's operating environment would grant execute access to the user (e.g., the user is privileged), the server MUST NOT reply with ACCESS4_EXECUTE set in reply's access field unless there is at least one execute permission bit set in the mode, acl, or dacl attributes. In the case of acl and dacl, the "one execute permission bit" MUST be an ACE4_EXECUTE bit set in an ALLOW ACE 🔐
• If the server does not support execute permission bits or some other method for denoting executability, it MUST NOT set ACCESS4_EXECUTE in the reply's supported and access fields. If the client set ACCESS4_EXECUTE in the ACCESS request's access field, and ACCESS4_EXECUTE is not set in the reply's supported field, then the client will have to send an ACCESS request with the ACCESS4_READ bit set in the request's access field 🔄
• If the server supports read permission bits, it MUST only check for read permissions in the mode, acl, and dacl attributes when it receives an ACCESS request with ACCESS4_READ set in the access field. The server MUST NOT also examine execute permission bits when determining whether the reply will have ACCESS4_READ set in the access field or not 📚

Note that if the ACCESS reply has ACCESS4_READ or ACCESS_EXECUTE set, then the user also has permissions to OPEN (Section 18.16) or READ (Section 18.22) the file. In other words, if the client sends an ACCESS request with the ACCESS4_READ and ACCESS_EXECUTE set in the access field (or two separate requests, one with ACCESS4_READ set and the other with ACCESS4_EXECUTE set), and the reply has just ACCESS4_EXECUTE set in the access field (or just one reply has ACCESS4_EXECUTE set), then the user has authorization to OPEN or READ the file 🎉

18.1.4. IMPLEMENTATION ✨

okay bestie, so the client can't just vibe-check the uid, gid, and mode fields in the file attributes, or try to read the ACL attribute and guess its access 💅. Why? Because the server might do uid or gid mapping, or add extra access-control restrictions on top 🙈. It's also possible the server isn't even in the same ID space as the client. In these cases (and maybe others), the client cannot reliably do an access check using only the current file attributes 😭.

In NFSv2, the only reliable way to find out if an operation was allowed was to try it and see if it worked or failed 💀. With the ACCESS operation in NFSv4.1, the client can ask the server whether one or more classes of operations are permitted 🎀. ACCESS exists so clients can check before doing a bunch of operations that would end in an access failure. The OPEN operation is a moment where the server can verify access to the file object and send that info back to the client 🦋. ACCESS is still useful for directory operations, or when the UNIX interface access() gets used on the client 🌸.

The info the server gives back in an ACCESS reply isn't permanent, ngl 😬. It was correct at the exact moment the server did the checks, but not necessarily after that. The server can revoke access permission at any time, so keep that in mind 💖.

The client should use the effective credentials of the user to build the authentication info in the ACCESS request used to figure out access rights. The effective user and group credentials are the ones that get used in later READ and WRITE operations ✨.

Many implementations don't directly support the ACCESS4_DELETE permission, periodt 🙃. Operating systems like UNIX ignore the ACCESS4_DELETE bit if it's set on an access request for a non-directory object. On those systems, delete permission on a file is decided by the access permissions on the directory the file lives in, not the file's own permissions 📁. So the returned mask (which says which access rights can be determined) will have ACCESS4_DELETE set to 0. That tells the client the server couldn't check that particular right 🫶. The client will then ignore the ACCESS4_DELETE bit in the returned access mask 💅.

---

18.2. Operation 4: CLOSE - Close File 🔐

18.2.1. ARGUMENTS

struct CLOSE4args {
        /* CURRENT_FH: object */
        seqid4          seqid;
        stateid4        open_stateid;
};

18.2.2. RESULTS

union CLOSE4res switch (nfsstat4 status) {
 case NFS4_OK:
         stateid4       open_stateid;
 default:
         void;
};

18.2.3. DESCRIPTION

The CLOSE operation releases share reservations for the regular or named attribute file as specified by the current filehandle. The share reservations and other state information released at the server as a result of this CLOSE are only those associated with the supplied stateid. State associated with other OPENs is not affected 🌷.

If byte-range locks are held, the client SHOULD release all locks before sending a CLOSE. The server MAY free all outstanding locks on CLOSE, but some servers may not support the CLOSE of a file that still has byte-range locks held. The server MUST return failure if any locks would exist after the CLOSE 🚨.

The argument seqid MAY have any value, and the server MUST ignore seqid 🙈.

On success, the current filehandle retains its value ✨.

The server MAY require that the combination of principal, security flavor, and, if applicable, GSS mechanism that sent the OPEN request also be the one to CLOSE the file. This might not be possible if credentials for the principal are no longer available. The server MAY allow the machine credential or SSV credential (see Section 18.35) to send CLOSE 🔑.

18.2.4. IMPLEMENTATION

Even though CLOSE returns a stateid, this stateid is not useful to the client and should be treated as deprecated 🥀. CLOSE "shuts down" the state associated with all OPENs for the file by a single open-owner. As noted above, CLOSE will either release all file-locking state or return an error. Therefore, the stateid returned by CLOSE is not useful for operations that follow. To help find any uses of this stateid by clients, the server SHOULD return the invalid special stateid (the "other" value is zero and the "seqid" field is NFS4_UINT32_MAX, see Section 8.2.3) 🔍.

A CLOSE operation may make delegations grantable where they were not previously. Servers may choose to respond immediately if there are pending delegation want requests, or they may respond to the situation at a later time ⏰.

---

18.3. Operation 5: COMMIT - Commit Cached Data 💾

18.3.1. ARGUMENTS

struct COMMIT4args {
        /* CURRENT_FH: file */
        offset4         offset;
        count4          count;
};

18.3.2. RESULTS

struct COMMIT4resok {
        verifier4       writeverf;
};

union COMMIT4res switch (nfsstat4 status) {
 case NFS4_OK:
         COMMIT4resok   resok4;
 default:
         void;
};

18.3.3. DESCRIPTION

The COMMIT operation forces or flushes uncommitted, modified data to stable storage for the file specified by the current filehandle. The flushed data is whatever was previously written with one or more WRITE operations that had the "committed" field of their results field set to UNSTABLE4 📦.

The offset specifies the position within the file where the flush is to begin. An offset value of zero means to flush data starting at the beginning of the file. The count specifies the number of bytes of data to flush. If the count is zero, a flush from the offset to the end of the file is done 📏.

The server returns a write verifier upon successful completion of the COMMIT. The write verifier is used by the client to determine if the server has restarted between the initial WRITE operations and the COMMIT. The client does this by comparing the write verifier returned from the initial WRITE operations with the verifier returned by the COMMIT operation 🔎. The server must vary the value of the write verifier at each server event or instantiation that may lead to a loss of uncommitted data. Most commonly this occurs when the server is restarted, but other events at the server may result in uncommitted data loss as well 💔.

On success, the current filehandle retains its value 🌸.

18.3.4. IMPLEMENTATION

The COMMIT operation is similar in operation and semantics to the POSIX fsync() [25] system interface, which synchronizes a file's state with the disk (file data and metadata is flushed to disk or stable storage) 🧠. COMMIT performs the same operation for a client, flushing any unsynchronized data and metadata on the server to the server's disk or stable storage for the specified file. Like fsync(), it may be that there is some modified data or no modified data to synchronize. The data may have been synchronized by the server's normal periodic buffer synchronization activity. COMMIT should return NFS4_OK, unless there has been an unexpected error 👀.

COMMIT differs from fsync() in that it is possible for the client to flush a range of the file. This is most likely triggered by a buffer-reclamation scheme on the client before the file has been completely written 🧹.

The server implementation of COMMIT is reasonably simple. If the server receives a full file COMMIT request, that is, starting at offset zero and count zero, it should do the equivalent of applying fsync() to the entire file. Otherwise, it should arrange to have the modified data in the range specified by offset and count flushed to stable storage. In both cases, any metadata associated with the file must be flushed to stable storage before returning. It is not an error for there to be nothing to flush on the server. This means that the data and metadata that needed to be flushed have already been flushed or lost during the last server failure 🌧️.

The client implementation of COMMIT is a little more complex 🧩. There are two reasons for wanting to commit a client buffer to stable storage. The first is that the client wants to reuse a buffer. In this case, the offset and count of the buffer are sent to the server in the COMMIT request. The server then flushes any modified data based on the offset and count, and flushes any modified metadata associated with the file. It then returns the status of the flush and the write verifier. The second reason for the client to generate a COMMIT is a full file flush, such as may be done at close. In this case, the client would gather all of the buffers for this file that contain uncommitted data, do the COMMIT operation with an offset of zero and count of zero, and then free all of those buffers. Any other dirty buffers would be sent to the server in the normal fashion 🧺.

After a buffer is written (via the WRITE operation) by the client with the "committed" field in the result of WRITE set to UNSTABLE4, the buffer must be considered as modified by the client until the buffer has either been flushed via a COMMIT operation or written via a WRITE operation with the "committed" field in the result set to FILE_SYNC4 or DATA_SYNC4. This is done to prevent the buffer from being freed and reused before the data can be flushed to stable storage on the server 🛡️.

When a response is returned from either a WRITE or a COMMIT operation and it contains a write verifier that differs from the one previously returned by the server, the client will need to retransmit all of the buffers containing uncommitted data to the server. How this is to be done is up to the implementor 🛠️. If there is only one buffer of interest, then it should be sent in a WRITE request with the FILE_SYNC4 stable parameter. If there is more than one buffer, it might be worthwhile to retransmit all of the buffers in WRITE operations with the stable parameter set to UNSTABLE4, and then retransmit the COMMIT operation to flush all of the data on the server to stable storage. However, if the server repeatably returns from COMMIT a verifier that differs from the one returned by WRITE, the only way to ensure progress is to retransmit all of the buffers with WRITE requests using the FILE_SYNC4 stable parameter 🔁.

The above description applies to page-cache-based systems as well as buffer-cache-based systems. In the former systems, the virtual memory system will need to be modified instead of the buffer cache 🧠✨.

---

18.4. Operation 6: CREATE - Create a Non-Regular File Object 🧚

18.4.1. ARGUMENTS

union createtype4 switch (nfs_ftype4 type) {
 case NF4LNK:
         linktext4 linkdata;
 case NF4BLK:
 case NF4CHR:
         specdata4 devdata;
 case NF4SOCK:
 case NF4FIFO:
 case NF4DIR:
         void;
 default:
         void;  /* server should return NFS4ERR_BADTYPE */
};

struct CREATE4args {
        /* CURRENT_FH: directory for creation */
        createtype4     objtype;
        component4      objname;
        fattr4          createattrs;
};

18.4.2. RESULTS ✨

struct CREATE4resok {
        change_info4    cinfo;
        bitmap4         attrset;        /* attributes set */
};

union CREATE4res switch (nfsstat4 status) {
 case NFS4_OK:
        /* new CURRENTFH: created object */
        CREATE4resok resok4;
 default:
        void;
};

18.4.3. DESCRIPTION 💅

Okay bestie, CREATE makes a file object that is not an ordinary file, inside a directory, with a name you hand it. If you want a regular file or a named attribute, the OPEN operation MUST be used. No exceptions, periodt 🫶

The current filehandle has to be a directory, an object of type NF4DIR. If it's an attribute directory (type NF4ATTRDIR), you get NFS4ERR_WRONG_TYPE. If the current file handle is any other type of object, you get NFS4ERR_NOTDIR 🙈

objname is the name for the new object. objtype decides what kind of object gets made: directory, symlink, etc. If the type you pick is an ordinary file, a named attribute, or a named attribute directory, you'll get NFS4ERR_BADTYPE 💀

If an object with the same name is already in the directory, the server returns NFS4ERR_EXIST 😭

For the directory where the new object was made, the server sends back change_info4 info in cinfo. With the atomic field of the change_info4 data type, the server tells you whether the before and after change attributes were obtained atomically with respect to the file object creation 🦋

If objname has a length of zero, or if it doesn't follow the UTF-8 definition, the error NFS4ERR_INVAL comes back ✨

The current filehandle gets replaced by the new object's filehandle 🎀

createattrs sets the starting attributes for the object. It can include any writable attribute that's valid for the object type. When the operation succeeds, the server sends back an attribute mask showing which attributes were actually set 🌸

If createattrs includes neither the owner attribute nor an ACL with an ACE for the owner, and the server's file system both supports and requires an owner attribute (or an owner ACE), then the server MUST derive the owner (or the owner ACE). That's typically from the principal indicated in the RPC credentials of the call, but the server's operating environment or file system semantics may dictate other methods of derivation. Same energy for groups: if createattrs includes neither the group attribute nor a group ACE, and the server's file system both supports and requires the notion of a group attribute (or group ACE), the server MUST derive the group attribute (or the corresponding owner ACE) for the file. This could be from the RPC call's credentials, such as the group principal if the credentials include it (such as with AUTH_SYS), from the group identifier associated with the principal in the credentials (e.g., POSIX systems have a user database [26] that has a group identifier for every user identifier), inherited from the directory in which the object is created, or whatever else the server's operating environment or file system semantics dictate. This applies to the OPEN operation too 🧚

Conversely, the client might put an owner attribute, group attribute, or ACL in createattrs that the principal indicated in the RPC call's credentials has no permissions to create files for. The error for that is NFS4ERR_PERM. This applies to OPEN too 🚫

If the current filehandle designates a directory for which another client holds a directory delegation, then, unless the delegation is such that the situation can be resolved by sending a notification, the delegation MUST be recalled, and the CREATE operation MUST NOT proceed until the delegation is returned or revoked. Except where this happens very quickly, one or more NFS4ERR_DELAY errors will be returned to requests made while the delegation remains outstanding ⏳

When the current filehandle designates a directory for which one or more directory delegations exist, then, when those delegations request such notifications, NOTIFY4_ADD_ENTRY will be generated as a result of this operation 🔔

If the capability FSCHARSET_CAP4_ALLOWS_ONLY_UTF8 is set (Section 14.4), and a symbolic link is being created, then the content of the symbolic link MUST be in UTF-8 encoding 🔥

18.4.4. IMPLEMENTATION

If the client wants to set attribute values after the create, a SETATTR operation can be added to the COMPOUND request so that the appropriate attributes will be set 💖

---

18.5. Operation 7: DELEGPURGE - Purge Delegations Awaiting Recovery 🌈

18.5.1. ARGUMENTS

struct DELEGPURGE4args {
        clientid4       clientid;
};

18.5.2. RESULTS

struct DELEGPURGE4res {
        nfsstat4        status;
};

18.5.3. DESCRIPTION

This operation purges all of the delegations awaiting recovery for a given client. It's super useful for clients that don't commit delegation information to stable storage, so the server knows conflicting requests don't need to be delayed while it waits on recovery of that info 🧹

The client is not specified by the clientid field of the request. The client SHOULD set the client field to zero, and the server MUST ignore the clientid field. Instead, the server MUST derive the client ID from the value of the session ID in the arguments of the SEQUENCE operation that precedes DELEGPURGE in the COMPOUND request 🎀

The DELEGPURGE operation should be used by clients that record delegation information on stable storage on the client. In this case, after the client recovers all delegations it knows of, it should immediately send a DELEGPURGE operation. Doing so tells the server that no additional delegations for the client will be recovered, which lets it free resources and avoid delaying other clients whose requests conflict with the unrecovered delegations. The set of delegations known to the server and the client might be different. The reason is that after sending a request that resulted in a delegation, the client might experience a failure before it both received the delegation and committed the delegation to the client's stable storage 🫠

The server MAY support DELEGPURGE, but if it does not, it MUST NOT support CLAIM_DELEGATE_PREV and MUST NOT support CLAIM_DELEG_PREV_FH 🦋

---

18.6. Operation 8: DELEGRETURN - Return Delegation 💜

18.6.1. ARGUMENTS

struct DELEGRETURN4args {
        /* CURRENT_FH: delegated object */
        stateid4        deleg_stateid;
};

18.6.2. RESULTS

struct DELEGRETURN4res {
        nfsstat4        status;
};

18.6.3. DESCRIPTION

The DELEGRETURN operation returns the delegation represented by the current filehandle and stateid 📮

Delegations may be returned voluntarily (i.e., before the server has recalled them) or when recalled. In either case, the client must properly propagate state changed under the context of the delegation to the server before returning the delegation 🧾

The server MAY require that the principal, security flavor, and if applicable, the GSS mechanism, combination that acquired the delegation also be the one to send DELEGRETURN on the file. This might not be possible if credentials for the principal are no longer available. The server MAY allow the machine credential or SSV credential (see Section 18.35) to send DELEGRETURN 🔐

---

18.7. Operation 9: GETATTR - Get Attributes 👑

18.7.1. ARGUMENTS

struct GETATTR4args {
        /* CURRENT_FH: object */
        bitmap4         attr_request;
};

18.7.2. RESULTS

struct GETATTR4resok {
        fattr4          obj_attributes;
};

union GETATTR4res switch (nfsstat4 status) {
 case NFS4_OK:
        GETATTR4resok  resok4;
 default:
        void;
};

18.7.3. DESCRIPTION

The GETATTR operation obtains attributes for the file system object specified by the current filehandle. The client sets a bit in the bitmap argument for each attribute value it would like the server to return. The server returns an attribute bitmap showing the attribute values it was able to return, which will include all attributes the client requested that are supported by the server for the target file system. That bitmap is followed by the attribute values, ordered lowest attribute number first 📋

The server MUST return a value for each attribute that the client requests if the attribute is supported by the server for the target file system. If the server does not support a particular attribute on the target file system, then it MUST NOT return the attribute value and MUST NOT set the attribute bit in the result bitmap. The server MUST return an error if it supports an attribute on the target but cannot obtain its value. In that case, no attribute values will be returned 🚨

File systems that are absent should be treated as having support for a very small set of attributes as described in Section 11.3.1, even if previously, when the file system was present, more attributes were supported 👻

All servers MUST support the REQUIRED attributes as specified in Section 5.6, for all file systems, with the exception of absent file systems 💅

On success, the current filehandle retains its value ✨

18.7.4. IMPLEMENTATION

Suppose there is an OPEN_DELEGATE_WRITE delegation held by another client for the file in question, and size and/or change are among the set of attributes being interrogated. The server has two choices. First, the server can obtain the actual current value of these attributes from the client holding the delegation by using the CB_GETATTR callback. Second, the server, particularly when the delegated client is unresponsive, can recall the delegation in question. The GETATTR MUST NOT proceed until one of the following occurs:

• The requested attribute values are returned in the response to CB_GETATTR.
• The OPEN_DELEGATE_WRITE delegation is returned.
• The OPEN_DELEGATE_WRITE delegation is revoked.

Unless one of the above happens very quickly, one or more NFS4ERR_DELAY errors will be returned while a delegation is outstanding 🫶

---

18.8. Operation 10: GETFH - Get Current Filehandle 🍓

18.8.1. ARGUMENTS

/* CURRENT_FH: */
void;

18.8.2. RESULTS

struct GETFH4resok {
        nfs_fh4         object;
};

union GETFH4res switch (nfsstat4 status) {
 case NFS4_OK:
        GETFH4resok     resok4;
 default:
        void;
};

18.8.3. DESCRIPTION

This operation returns the current filehandle value 🔍

On success, the current filehandle retains its value 🌸

As described in Section 2.10.6.4, GETFH is REQUIRED or RECOMMENDED to immediately follow certain operations, and servers are free to reject such operations if the client fails to insert GETFH in the request as REQUIRED or RECOMMENDED. Section 18.16.4.1 provides additional justification for why GETFH MUST follow OPEN 🧠

18.8.4. IMPLEMENTATION

Operations that change the current filehandle, like LOOKUP or CREATE, do not automatically return the new filehandle as a result. For instance, if a client needs to look up a directory entry and obtain its filehandle, then the following request is needed 🦋

PUTFH (directory filehandle)
LOOKUP (entry name)
GETFH

---

18.9. Operation 11: LINK - Create Link to a File 🔗

18.9.1. ARGUMENTS ✨

struct LINK4args {
        /* SAVED_FH: source object */
        /* CURRENT_FH: target directory */
        component4      newname;
};

18.9.2. RESULTS 💅

struct LINK4resok {
        change_info4    cinfo;
};

union LINK4res switch (nfsstat4 status) {
 case NFS4_OK:
         LINK4resok resok4;
 default:
         void;
};

18.9.3. DESCRIPTION 🦋

okay so LINK is basically how you give a file a second name 🫶 It makes an extra newname for the file that the saved filehandle points to (the one you stashed with SAVEFH), and it puts that new name in the directory the current filehandle points to. The file and the target directory have to live on the same file system on the server, or it's a no-go. 🙅‍♀️ When it works, the current filehandle is still the target directory, so you're right where you started, bestie.

If something already exists in the target directory with that same name as newname, the server must return NFS4ERR_EXIST. Not me trying to overwrite your stuff 💀

For the target directory, the server sends back change_info4 data in cinfo. The atomic field of that type tells you whether the before and after change attributes were grabbed atomically with respect to the link creation. Very precise, very iconic ✨

And if newname is zero length, or it doesn't follow the UTF-8 rules, you get NFS4ERR_INVAL. ngl that's just the server being picky about names 🙈

18.9.4. IMPLEMENTATION 🔥

Okay this section is where it gets spicy 🌶️ The server MAY put restrictions on LINK, so a link might not be allowed when the file is open, or when the open was done by certain protocols, or with certain options or access modes. If LINK gets rejected for any of those reasons, you get NFS4ERR_FILE_OPEN back.

Now if a server does have those rules, and one of them is about NFSv4 opens blocking a link, the server needs to recall any delegations that could be hiding the opens that matter for that decision. Why? Because when a client holds a delegation, the server might not have an accurate picture of the opens for that client, since the client can OPEN and CLOSE stuff locally. 👀 LINK should only be delayed until the server can get a definitive answer, and that's it. For example, say there are several delegations and one of them has an open that would block the link. Depending on how the server works, NFS4ERR_FILE_OPEN can be returned to the caller as soon as that one delegation comes back, without waiting for the others. And if those opens aren't tied to any delegation, NFS4ERR_FILE_OPEN can come back right away with no recall needed 🧚

If the current filehandle is a directory that another client holds a directory delegation for, then the delegation MUST be recalled, unless the situation can be fixed just by sending a notification. The operation can't succeed until that delegation is returned or revoked. Unless this happens super fast, requests made while the delegation is still out will get one or more NFS4ERR_DELAY errors. Main character energy? Not really, more like waiting in line 😭

When the current filehandle is a directory with one or more directory delegations, and those delegations asked for notifications, then no recall happens. Instead, a NOTIFY4_ADD_ENTRY gets generated as a result of the LINK operation. Much more chill 🌸

If the current file system supports the numlinks attribute, and other clients have delegations on the file being linked, then those delegations MUST be recalled, and LINK MUST NOT proceed until all of them are returned or revoked. Again, unless this is super quick, requests made during that time get one or more NFS4ERR_DELAY errors.

Changes to any property of the "hard" linked files show up in all of the linked files, so they're all tied together 🎀 And when a link gets made to a file, the file's numlinks value should be one greater than what it was before the LINK operation. Simple math, no cap.

Now, "file and the target directory must reside within the same file system on the server" really just means the fsid fields in the attributes of those objects are the same. If they're on different file systems, you get NFS4ERR_XDEV. Some servers might also return that error when a file system has been internally partitioned and the LINK would break that partition 🧩

On some servers, "." and ".." aren't allowed as newname values. If you send them anyway, you'll get NFS4ERR_BADNAME. Rude 💅

When the current filehandle is a named attribute directory, and the object being linked (the saved filehandle) isn't a named attribute of the same object, the server MUST return NFS4ERR_XDEV. Same deal if the saved filehandle is a named attribute but the current filehandle isn't the right named attribute directory: NFS4ERR_XDEV MUST be returned. No wandering into the wrong folder, periodt 🚫

When the current filehandle is a named attribute directory, and the saved filehandle is a named attribute inside that directory, the server may return NFS4ERR_NOTSUPP. 🙈

If newname is already linked to the file the saved filehandle represents, the server will return NFS4ERR_EXIST.

And last but not least, symbolic links aren't made with LINK. They're created with the CREATE operation, so go there instead 🔗

---

18.10. Operation 12: LOCK - Create Lock 🔐

18.10.1. ARGUMENTS 💜

/*
 * For LOCK, transition from open_stateid and lock_owner
 * to a lock stateid.
 */
struct open_to_lock_owner4 {
        seqid4          open_seqid;
        stateid4        open_stateid;
        seqid4          lock_seqid;
        lock_owner4     lock_owner;
};

/*
 * For LOCK, existing lock stateid continues to request new
 * file lock for the same lock_owner and open_stateid.
 */
struct exist_lock_owner4 {
        stateid4        lock_stateid;
        seqid4          lock_seqid;
};
union locker4 switch (bool new_lock_owner) {
 case TRUE:
        open_to_lock_owner4     open_owner;
 case FALSE:
        exist_lock_owner4       lock_owner;
};

/*
 * LOCK/LOCKT/LOCKU: Record lock management
 */
struct LOCK4args {
        /* CURRENT_FH: file */
        nfs_lock_type4  locktype;
        bool            reclaim;
        offset4         offset;
        length4         length;
        locker4         locker;
};

18.10.2. RESULTS 🍓

struct LOCK4denied {
        offset4         offset;
        length4         length;
        nfs_lock_type4  locktype;
        lock_owner4     owner;
};

struct LOCK4resok {
        stateid4        lock_stateid;
};

union LOCK4res switch (nfsstat4 status) {
 case NFS4_OK:
         LOCK4resok     resok4;
 case NFS4ERR_DENIED:
         LOCK4denied    denied;
 default:
         void;
};

18.10.3. DESCRIPTION 🌈

So LOCK is how you ask for a byte-range lock. The offset and length parameters say which bytes, and the locktype parameter says what kind of lock. If this is a reclaim request, the reclaim parameter will be TRUE. 🔁

Here's a fun fact: you can lock bytes that aren't actually allocated to the file yet 🤯 If you want to lock from a specific offset all the way to end-of-file (no matter how long the file is), set length to NFS4_UINT64_MAX. The server MUST return NFS4ERR_INVAL for these combos of length and offset:

• Length is equal to zero.
• Length is not equal to NFS4_UINT64_MAX, and the sum of length and offset exceeds NFS4_UINT64_MAX.

Now, 32-bit servers are the ones that support locking for byte offsets fitting in 32 bits (so, less than or equal to NFS4_UINT32_MAX). If the client asks for a range that overlaps bytes past offset NFS4_UINT32_MAX but doesn't end at NFS4_UINT64_MAX, then that 32-bit server MUST return NFS4ERR_BAD_RANGE. Too big for the server, sorry bestie 🙃

If the server returns NFS4ERR_DENIED, it sends back the owner, offset, and length of the conflicting lock. 🚧

The locker argument says which lock-owner is tied to the LOCK operation. locker4 is a switched union that tells the server whether the client already has byte-range locking state for that open file and lock-owner. If it does, the argument is just a stateid for the set of locks tied to that open file and lock-owner, along with a lock_seqid value. That lock_seqid MAY be any value, and the server MUST ignore it. If there's no byte-range locking state yet, or the client doesn't have that stateid available, the argument holds the stateid of the open file the lock should be associated with, plus the lock-owner it goes with. The open_to_lock_owner case is for the very first lock a lock-owner makes on a given open file. It's the handoff that moves from the open_stateid's state over to a lock stateid ✨

The following fields of the locker parameter MAY be set to any value by the client, and the server MUST ignore them:

• The clientid field of the lock_owner field of the open_owner field (locker.open_owner.lock_owner.clientid). The server MUST ignore this clientid field because the server MUST derive the client ID from the session ID in the SEQUENCE operation of the COMPOUND request. Honestly the server does the work here, not you 💁‍♀️
• The open_seqid and lock_seqid fields of the open_owner field (locker.open_owner.open_seqid and locker.open_owner.lock_seqid).
• The lock_seqid field of the lock_owner field (locker.lock_owner.lock_seqid).

Quick note: the client ID showing up in a LOCK4denied structure is the actual client that owns the conflicting lock, whether that's the client ID for the current session or a different one. So if the server returns NFS4ERR_DENIED, it MUST set the clientid field of the owner field of the denied field. 🫶

If the current filehandle isn't an ordinary file, an error goes back to the client. If the current filehandle is an object of type NF4DIR, you get NFS4ERR_ISDIR. If it's a symbolic link, you get NFS4ERR_SYMLINK. In every other case, you get NFS4ERR_WRONG_TYPE 🙅‍♀️

On success, the current filehandle keeps its value. Easy 💖

18.10.4. IMPLEMENTATION ✨

okay so if the server can't figure out the exact offset and length of the conflicting byte-range lock, the same offset and length from the arguments should get sent back in the denied results 🧚

LOCK operations go through permission checks and checks against the access type of the file, bestie 🔐 But the specific rights and modes needed for each kind of lock reflect how the server-exported file system works, and the protocol doesn't spell them out. Windows 2000 lets you take a write lock on a file that's open for read access, but a POSIX-compliant system doesn't. Not the same vibe at all 💀

okay this part is lowkey important 👀 When the client sends a LOCK for a range the lock-owner already locked (with the same or a different lock type), or for a sub-range of that range, or for a byte-range covering several locks already granted to that lock-owner (in whole or in part), and the server doesn't support that kind of locking (meaning it doesn't do POSIX locking semantics), the server returns NFS4ERR_LOCK_RANGE 😭 Then the client can either return an error or fake the operations itself. To fake them, it can use only LOCK for ranges that don't include any bytes already locked by that lock-owner, plus LOCKU on locks that lock-owner holds (with an exactly matching range and type). Same energy for upgrades (switching from a READ_LT lock to a WRITE_LT lock) and downgrades (switching from a WRITE_LT lock to a READ_LT lock): if the server doesn't support that kind of lock, it returns NFS4ERR_LOCK_NOTSUPP 🙃 Those operations may not perfectly match what's supposed to happen when other clients send conflicting LOCK operations at the same time. Real talk, it's a little messy 🫠

When a client holds an OPEN_DELEGATE_WRITE delegation, it's guaranteed there are no opens by other clients 🦋 So there can't be any conflicting LOCK operations from those clients either. That means the client can handle locking requests locally without doing LOCK operations on the server. BUT, and this is a big but, it then has to be ready to update the lock status on the server by sending the right LOCK and LOCKU operations before it gives the delegation back 💅

When one or more clients hold OPEN_DELEGATE_READ delegations, any LOCK operation on a server doing mandatory locking semantics MUST result in a recall of all those delegations 🚨 The LOCK can't be granted until every one of those delegations is returned or revoked. Unless that happens super fast, the requests made while the delegation is still outstanding will get one or more NFS4ERR_DELAY errors 😭

---

18.11. Operation 13: LOCKT - Test for Lock 🔍

18.11.1. ARGUMENTS 🎀

struct LOCKT4args {
        /* CURRENT_FH: file */
        nfs_lock_type4  locktype;
        offset4         offset;
        length4         length;
        lock_owner4     owner;
};

18.11.2. RESULTS 🌸

union LOCKT4res switch (nfsstat4 status) {
 case NFS4ERR_DENIED:
         LOCK4denied    denied;
 case NFS4_OK:
         void;
 default:
         void;
};

18.11.3. DESCRIPTION 💜

LOCKT is the "is anyone already locking this?" operation, and it tests the lock you specify in the arguments 🕵️‍♀️ If a conflicting lock exists, the owner, offset, length, and type of that conflicting lock come back to you. The owner field in the results includes the client ID of whoever owns the conflicting lock, whether that's the client ID tied to your current session or a totally different client ID. If no lock is held, you only get NFS4_OK back. READ_LT and READW_LT get handled the same way, since the conflicting lock test happens with no regard for blocking or non-blocking. Same goes for WRITE_LT and WRITEW_LT 🧚

The ranges are specified just like they are for LOCK. NFS4ERR_INVAL and NFS4ERR_BAD_RANGE come back under the same circumstances as they do for LOCK ✨

The clientid field of the owner MAY be set to any value by the client and MUST be ignored by the server. The reason the server MUST ignore the clientid field is that the server MUST derive the client ID from the session ID in the SEQUENCE operation of the COMPOUND request. It's giving "trust the session, not the field" 🫶

If the current filehandle isn't an ordinary file, you get an error back. If the current filehandle is an object of type NF4DIR, you get NFS4ERR_ISDIR. If the current filehandle points to a symbolic link, you get NFS4ERR_SYMLINK. Everything else gets NFS4ERR_WRONG_TYPE 🙈

On success, the current filehandle keeps its value 🌈

18.11.4. IMPLEMENTATION 🛠️

If the server can't determine the exact offset and length of the conflicting lock, the same offset and length provided in the arguments should be returned in the denied results 💖

LOCKT uses a lock_owner4 rather than a stateid4 to identify the owner, unlike LOCK. This is because the client doesn't have to open the file to test whether a lock exists, so a stateid might not be available 🦋

As noted in Section 18.10.4, some servers may return NFS4ERR_LOCK_RANGE to certain LOCK operations that don't actually conflict but do overlap ranges already granted to the current lock-owner 🙃

The LOCKT operation's test for conflicting locks SHOULD exclude locks held by the current lock-owner, so it should return NFS4_OK in those cases. Heads up, this means a server might return NFS4_OK to a LOCKT request even though a LOCK for the same range and lock-owner would fail with NFS4ERR_LOCK_RANGE 🫠

When a client holds an OPEN_DELEGATE_WRITE delegation, it may choose (see Section 18.10.4) to handle LOCK requests locally. In that case, LOCKT requests get handled locally too 🍓

---

18.12. Operation 14: LOCKU - Unlock File 🔓

18.12.1. ARGUMENTS 🎀

struct LOCKU4args {
        /* CURRENT_FH: file */
        nfs_lock_type4  locktype;
        seqid4          seqid;
        stateid4        lock_stateid;
        offset4         offset;
        length4         length;
};

18.12.2. RESULTS 🌸

union LOCKU4res switch (nfsstat4 status) {
 case   NFS4_OK:
         stateid4       lock_stateid;
 default:
         void;
};

18.12.3. DESCRIPTION 💜

LOCKU unlocks the byte-range lock described by the parameters 🔓 The client can set the locktype field to any value that's legal for the nfs_lock_type4 enumerated type, and the server MUST accept any legal value for locktype. Any legal value for locktype has no effect on whether LOCKU succeeds or fails. No drama 💅

The ranges are specified just like they are for LOCK. NFS4ERR_INVAL and NFS4ERR_BAD_RANGE come back under the same circumstances as they do for LOCK ✨

The seqid parameter MAY be any value, and the server MUST ignore it 🙈

If the current filehandle isn't an ordinary file, you get an error back. If it's an object of type NF4DIR, you get NFS4ERR_ISDIR. If it points to a symbolic link, you get NFS4ERR_SYMLINK. Everything else gets NFS4ERR_WRONG_TYPE 🌈

On success, the current filehandle keeps its value 🦋

The server MAY require that the principal, the security flavor, and, if applicable, the GSS mechanism that sent a LOCK operation are also the ones that send LOCKU on the file. That might not be possible if the credentials for that principal are no longer available 🥲 The server MAY allow the machine credential or SSV credential (see Section 18.35) to send LOCKU 🔑

18.12.4. IMPLEMENTATION 🛠️

If the area to be unlocked doesn't match exactly a lock the lock-owner actually holds, the server may return the error NFS4ERR_LOCK_RANGE. That includes the case where the area isn't locked at all, where the area is a sub-range of the locked area, where it overlaps the locked area without matching exactly, or where the area includes multiple locks held by the lock-owner. All of these are allowed by POSIX locking [24] semantics. If a client gets this error and wants support for these operations, it should simulate the operation by using LOCKU on ranges matching locks it really holds, possibly followed by LOCK operations for the sub-ranges that aren't being unlocked 🧩

When a client holds an OPEN_DELEGATE_WRITE delegation, it may choose (see Section 18.10.4) to handle LOCK requests locally. In that case, LOCKU operations get handled locally too 💖

---

18.13. Operation 15: LOOKUP - Lookup Filename 🗂️

18.13.1. ARGUMENTS 🎀

struct LOOKUP4args {
        /* CURRENT_FH: directory */
        component4      objname;
};

18.13.2. RESULTS 🌸

struct LOOKUP4res {
        /* New CURRENT_FH: object */
        nfsstat4        status;
};

18.13.3. DESCRIPTION 💜

LOOKUP finds a file system object by looking it up in the directory specified by the current filehandle 🔎 LOOKUP evaluates the component, and if the object exists, the current filehandle gets replaced with the component's filehandle ✨

If the component can't be evaluated, either because it doesn't exist or because the client doesn't have permission to evaluate it, you get an error back and the current filehandle stays the same 🙃

If the component is a zero-length string, or if any component doesn't follow the UTF-8 definition, you get NFS4ERR_INVAL 🙈

18.13.4. IMPLEMENTATION 🛠️

If the client wants a multi-component lookup, it can build a COMPOUND request like this one (and grab each filehandle along the way) 🧚

      PUTFH  (directory filehandle)
      LOOKUP "pub"
      GETFH
      LOOKUP "foo"
      GETFH
      LOOKUP "bar"
      GETFH

Unlike NFSv3, NFSv4.1 lets LOOKUP requests cross mountpoints on the server 🌍 The client can detect a mountpoint crossing by comparing the fsid attribute of the directory with the fsid attribute of the directory that was looked up. If the fsids are different, the new directory is a server mountpoint. UNIX clients that detect a mountpoint crossing need to mount the server's file system. This needs to happen to keep the file object identity checking mechanisms that UNIX clients rely on 🐧

Servers that limit NFS access to "shared" or "exported" file systems should provide a pseudo file system where the exported file systems can be integrated, so clients can browse the server's namespace 🗺️ A client's view of a pseudo file system is limited to paths that lead to exported file systems 🦋

Note: previous versions of the protocol gave special meaning to the names "." and "..". NFSv4.1 gives no special meaning to these names. The LOOKUPP operator must be used to look up a parent directory 👑

LOOKUP does not follow symbolic links. The client is responsible for all parsing of filenames, including filenames modified by symbolic links encountered during the lookup process 💅

If the current filehandle isn't a directory but a symbolic link, you get the error NFS4ERR_SYMLINK. For all other non-directory file types, you get NFS4ERR_NOTDIR 🙈

---

18.14. Operation 16: LOOKUPP - Lookup Parent Directory 👆

18.14.1. ARGUMENTS 🎀

/* CURRENT_FH: object */
void;

18.14.2. RESULTS 🌸

struct LOOKUPP4res {
        /* new CURRENT_FH: parent directory */
        nfsstat4        status;
};

18.14.3. DESCRIPTION 💅

okay so the current filehandle is supposed to be pointing at a regular directory or a named attribute directory 📂. LOOKUPP then sets the current filehandle to the parent directory of that one ✨. if there's no parent (aka you're at the root or top of the server's file tree 🌳), an NFS4ERR_NOENT error must be returned. so NFS4ERR_NOENT is what you'll get when the current filehandle is at the root or top of the server's file tree 💀.

LOOKUPP crosses mountpoints too, just like LOOKUP does 🚪✨. iconic consistency, honestly.

if the current filehandle isn't a directory or a named attribute directory, you get NFS4ERR_NOTDIR 🙅‍♀️.

now the security flavor drama 😤. if the requester's security flavor doesn't match what's configured for the parent directory, the server SHOULD return NFS4ERR_WRONGSEC in the LOOKUPP response (a future minor revision of NFSv4 may upgrade this to MUST 👀). BUT if the server does that, it MUST support the SECINFO_NO_NAME operation (Section 18.45) so the client can gracefully figure out the correct security flavor 🫶.

last one 🎀: if the current filehandle is a named attribute directory that's tied to a file system object through OPENATTR (i.e., not a sub-directory of a named attribute directory), LOOKUPP SHOULD return the filehandle of that associated file system object.

18.14.4. IMPLEMENTATION 🛠️

okay implementation tea ☕. named attribute directories are basically detached from the namespace, and this property should be safely represented in the client's operating environment 🫧. LOOKUPP on a named attribute directory might hand back the filehandle of the associated file, and passing that up to apps can be sketchy 😬, because a lot of apps expect an object's parent to always be a directory. so the client may want to hide the parent of named attribute directories (shown as .. in UNIX) or represent the named attribute directory as its own parent (which is what's typically done for the file system root directory in UNIX) 🦋.

18.15. Operation 17: NVERIFY - Verify Difference in Attributes ✨

18.15.1. ARGUMENTS

struct NVERIFY4args {
        /* CURRENT_FH: object */
        fattr4          obj_attributes;
};

18.15.2. RESULTS

struct NVERIFY4res {
        nfsstat4        status;
};

18.15.3. DESCRIPTION 🪄

this op is used to prefix a sequence of operations that only runs if one or more attributes have changed on some file system object. if all the attributes match, then the error NFS4ERR_SAME MUST be returned.

on success, the current filehandle keeps its value 💖.

18.15.4. IMPLEMENTATION 🧠

this is seriously useful as a cache validation operator. if the object's attributes have changed, the operations that follow can grab fresh data for it. for example, to check whether a file has changed and grab new data if it has:

      SEQUENCE
      PUTFH fh
      NVERIFY attrbits attrs
      READ 0 32767

contrast that with NFSv3 🐢. NFSv3 would send a GETATTR in one request/reply round trip first, and then, if the attributes showed the client's cache was stale, send a READ in another request/reply round trip. so much slower, bestie.

in the case that a RECOMMENDED attribute is specified in the NVERIFY operation and the server doesn't support that attribute for the file system object, the error NFS4ERR_ATTRNOTSUPP is returned to the client 💀.

when the attribute rdattr_error or any set-only attribute (e.g., time_modify_set) is specified, the error NFS4ERR_INVAL is returned to the client 🙅.

18.16. Operation 18: OPEN - Open a Regular File 🚪

18.16.1. ARGUMENTS 🎁

/*
 * Various definitions for OPEN
 */
enum createmode4 {
        UNCHECKED4      = 0,
        GUARDED4        = 1,
        /* Deprecated in NFSv4.1. */
        EXCLUSIVE4      = 2,
        /*
         * New to NFSv4.1. If session is persistent,
         * GUARDED4 MUST be used.  Otherwise, use
         * EXCLUSIVE4_1 instead of EXCLUSIVE4.
         */
        EXCLUSIVE4_1    = 3
};
struct creatverfattr {
         verifier4      cva_verf;
         fattr4         cva_attrs;
};

union createhow4 switch (createmode4 mode) {
 case UNCHECKED4:
 case GUARDED4:
         fattr4         createattrs;
 case EXCLUSIVE4:
         verifier4      createverf;
 case EXCLUSIVE4_1:
         creatverfattr  ch_createboth;
};

enum opentype4 {
        OPEN4_NOCREATE  = 0,
        OPEN4_CREATE    = 1
};

union openflag4 switch (opentype4 opentype) {
 case OPEN4_CREATE:
         createhow4     how;
 default:
         void;
};

/* Next definitions used for OPEN delegation */
enum limit_by4 {
        NFS_LIMIT_SIZE          = 1,
        NFS_LIMIT_BLOCKS        = 2
        /* others as needed */
};

struct nfs_modified_limit4 {
        uint32_t        num_blocks;
        uint32_t        bytes_per_block;
};

union nfs_space_limit4 switch (limit_by4 limitby) {
 /* limit specified as file size */
 case NFS_LIMIT_SIZE:
         uint64_t               filesize;
 /* limit specified by number of blocks */
 case NFS_LIMIT_BLOCKS:
         nfs_modified_limit4    mod_blocks;
} ;
/*
 * Share Access and Deny constants for open argument
 */
const OPEN4_SHARE_ACCESS_READ   = 0x00000001;
const OPEN4_SHARE_ACCESS_WRITE  = 0x00000002;
const OPEN4_SHARE_ACCESS_BOTH   = 0x00000003;

const OPEN4_SHARE_DENY_NONE     = 0x00000000;
const OPEN4_SHARE_DENY_READ     = 0x00000001;
const OPEN4_SHARE_DENY_WRITE    = 0x00000002;
const OPEN4_SHARE_DENY_BOTH     = 0x00000003;


/* new flags for share_access field of OPEN4args */
const OPEN4_SHARE_ACCESS_WANT_DELEG_MASK        = 0xFF00;
const OPEN4_SHARE_ACCESS_WANT_NO_PREFERENCE     = 0x0000;
const OPEN4_SHARE_ACCESS_WANT_READ_DELEG        = 0x0100;
const OPEN4_SHARE_ACCESS_WANT_WRITE_DELEG       = 0x0200;
const OPEN4_SHARE_ACCESS_WANT_ANY_DELEG         = 0x0300;
const OPEN4_SHARE_ACCESS_WANT_NO_DELEG          = 0x0400;
const OPEN4_SHARE_ACCESS_WANT_CANCEL            = 0x0500;

const
 OPEN4_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_RESRC_AVAIL
 = 0x10000;

const
 OPEN4_SHARE_ACCESS_WANT_PUSH_DELEG_WHEN_UNCONTENDED
 = 0x20000;

enum open_delegation_type4 {
        OPEN_DELEGATE_NONE      = 0,
        OPEN_DELEGATE_READ      = 1,
        OPEN_DELEGATE_WRITE     = 2,
        OPEN_DELEGATE_NONE_EXT  = 3 /* new to v4.1 */
};

enum open_claim_type4 {
        /*
         * Not a reclaim.
         */
        CLAIM_NULL              = 0,

        CLAIM_PREVIOUS          = 1,
        CLAIM_DELEGATE_CUR      = 2,
        CLAIM_DELEGATE_PREV     = 3,
        /*
         * Not a reclaim.
         *
         * Like CLAIM_NULL, but object identified
         * by the current filehandle.
         */
        CLAIM_FH                = 4, /* new to v4.1 */

        /*
         * Like CLAIM_DELEGATE_CUR, but object identified
         * by current filehandle.
         */
        CLAIM_DELEG_CUR_FH      = 5, /* new to v4.1 */

        /*
         * Like CLAIM_DELEGATE_PREV, but object identified
         * by current filehandle.
         */
        CLAIM_DELEG_PREV_FH     = 6 /* new to v4.1 */
};

struct open_claim_delegate_cur4 {
        stateid4        delegate_stateid;
        component4      file;
};

union open_claim4 switch (open_claim_type4 claim) {
 /*
  * No special rights to file.
  * Ordinary OPEN of the specified file.
  */
 case CLAIM_NULL:
        /* CURRENT_FH: directory */
        component4      file;
 /*
  * Right to the file established by an
  * open previous to server reboot.  File
  * identified by filehandle obtained at
  * that time rather than by name.
  */
 case CLAIM_PREVIOUS:
        /* CURRENT_FH: file being reclaimed */
        open_delegation_type4   delegate_type;

 /*
  * Right to file based on a delegation
  * granted by the server.  File is
  * specified by name.
  */
 case CLAIM_DELEGATE_CUR:
        /* CURRENT_FH: directory */
        open_claim_delegate_cur4        delegate_cur_info;

 /*
  * Right to file based on a delegation
  * granted to a previous boot instance
  * of the client.  File is specified by name.
  */
 case CLAIM_DELEGATE_PREV:
         /* CURRENT_FH: directory */
        component4      file_delegate_prev;

 /*
  * Like CLAIM_NULL.  No special rights
  * to file.  Ordinary OPEN of the
  * specified file by current filehandle.
  */
 case CLAIM_FH: /* new to v4.1 */
        /* CURRENT_FH: regular file to open */
        void;

 /*
  * Like CLAIM_DELEGATE_PREV.  Right to file based on a
  * delegation granted to a previous boot
  * instance of the client.  File is identified by
  * by filehandle.
  */
 case CLAIM_DELEG_PREV_FH: /* new to v4.1 */
        /* CURRENT_FH: file being opened */
        void;

 /*
  * Like CLAIM_DELEGATE_CUR.  Right to file based on
  * a delegation granted by the server.
  * File is identified by filehandle.
  */
 case CLAIM_DELEG_CUR_FH: /* new to v4.1 */
         /* CURRENT_FH: file being opened */
         stateid4       oc_delegate_stateid;

};
/*
 * OPEN: Open a file, potentially receiving an OPEN delegation
 */
struct OPEN4args {
        seqid4          seqid;
        uint32_t        share_access;
        uint32_t        share_deny;
        open_owner4     owner;
        openflag4       openhow;
        open_claim4     claim;
};

18.16.2. RESULTS ✨

Okay so, if the OPEN works out, here's what the server sends back 💖 The delegation stuff is giving main character energy, so let's look at the structs first:

   struct open_read_delegation4 {
    stateid4 stateid;    /* Stateid for delegation*/
    bool     recall;     /* Pre-recalled flag for
                            delegations obtained
                            by reclaim (CLAIM_PREVIOUS) */

    nfsace4 permissions; /* Defines users who don't
                            need an ACCESS call to
                            open for read */
   };

   struct open_write_delegation4 {
    stateid4 stateid;      /* Stateid for delegation */
    bool     recall;       /* Pre-recalled flag for
                              delegations obtained
                              by reclaim
                              (CLAIM_PREVIOUS) */

    nfs_space_limit4
              space_limit; /* Defines condition that
                              the client must check to
                              determine whether the
                              file needs to be flushed
                              to the server on close.  */

    nfsace4   permissions; /* Defines users who don't
                              need an ACCESS call as
                              part of a delegated
                              open. */
   };
   enum why_no_delegation4 { /* new to v4.1 */
           WND4_NOT_WANTED         = 0,
           WND4_CONTENTION         = 1,
           WND4_RESOURCE           = 2,
           WND4_NOT_SUPP_FTYPE     = 3,
           WND4_WRITE_DELEG_NOT_SUPP_FTYPE = 4,
           WND4_NOT_SUPP_UPGRADE   = 5,
           WND4_NOT_SUPP_DOWNGRADE = 6,
           WND4_CANCELLED          = 7,
           WND4_IS_DIR             = 8
   };

   union open_none_delegation4 /* new to v4.1 */
   switch (why_no_delegation4 ond_why) {
           case WND4_CONTENTION:
                   bool ond_server_will_push_deleg;
           case WND4_RESOURCE:
                   bool ond_server_will_signal_avail;
           default:
                   void;
   };

   union open_delegation4
   switch (open_delegation_type4 delegation_type) {
           case OPEN_DELEGATE_NONE:
                   void;
           case OPEN_DELEGATE_READ:
                   open_read_delegation4 read;
           case OPEN_DELEGATE_WRITE:
                   open_write_delegation4 write;
           case OPEN_DELEGATE_NONE_EXT: /* new to v4.1 */
                   open_none_delegation4 od_whynone;
   };

   /*
    * Result flags
    */

   /* Client must confirm open */
   const OPEN4_RESULT_CONFIRM      = 0x00000002;
   /* Type of file locking behavior at the server */
   const OPEN4_RESULT_LOCKTYPE_POSIX = 0x00000004;
   /* Server will preserve file if removed while open */
   const OPEN4_RESULT_PRESERVE_UNLINKED = 0x00000008;
   /*
    * Server may use CB_NOTIFY_LOCK on locks
    * derived from this open
    */
   const OPEN4_RESULT_MAY_NOTIFY_LOCK = 0x00000020;

   struct OPEN4resok {
    stateid4       stateid;      /* Stateid for open */
    change_info4   cinfo;        /* Directory Change Info */
    uint32_t       rflags;       /* Result flags */
    bitmap4        attrset;      /* attribute set for create*/
    open_delegation4 delegation; /* Info on any open
                                    delegation */
   };

   union OPEN4res switch (nfsstat4 status) {
    case NFS4_OK:
           /* New CURRENT_FH: opened file */
           OPEN4resok      resok4;
    default:
           void;
   };

Basically, a successful OPEN hands back a stateid for the open, the directory change info (cinfo), some result flags (rflags), the attribute set (attrset) for any create, and info on any delegation 🦋 The result flags are a whole vibe too: the client might have to confirm the open, the server might use POSIX-style locking, it might keep a removed file alive while it's still open, or it might send CB_NOTIFY_LOCK on locks that came from this open 🔒

---

18.16.3. DESCRIPTION 💅

Okay, bestie, so OPEN opens a regular file in a directory using either a name or a filehandle 📂 It can also create the file if you give it a name and say you want one 🌸 Whether the file gets created, and how, is decided by the openhow parameter. That's a switched union (data type opengflag4) that switches on opentype, which is either OPEN4_NOCREATE or OPEN4_CREATE. If it's OPEN4_CREATE, another switched union kicks in (data type createhow4) with four creation methods: UNCHECKED4, GUARDED4, EXCLUSIVE4, or EXCLUSIVE4_1. And if opentype is OPEN4_CREATE, the claim field of the claim field MUST be one of CLAIM_NULL, CLAIM_DELEGATE_CUR, or CLAIM_DELEGATE_PREV, because those claim methods include a component of a file name 🧚

Upon success (which might mean a brand new file got made), the current filehandle gets swapped for the one belonging to the created or existing object ✨

If the current filehandle is a named attribute directory, OPEN will create or open a named attribute file instead. Heads up though: exclusive create of a named attribute is not supported 🙈 So if the createmode is EXCLUSIVE4 or EXCLUSIVE4_1 and the current filehandle is a named attribute directory, the server will return EINVAL.

UNCHECKED4 is the chill one. It says the file should be created if no file with that name exists, and running into an existing regular file with that name is not an error 🍓 For this kind of create, createattrs sets the starting attributes for the file. Those can include any writable attribute that's valid for regular files. When an UNCHECKED4 create finds an existing file, the attributes in createattrs are ignored, with one exception: if createattrs has the size attribute set to zero, the existing file gets truncated 💀

GUARDED4 is the cautious one 🧐 The server checks for a duplicate object by name before it creates anything. If there's a duplicate, you get NFS4ERR_EXIST. If the object doesn't exist, the request goes through just like UNCHECKED4.

For UNCHECKED4 and GUARDED4, when the operation succeeds, the server sends back an attribute mask that shows which attributes it actually set on the object 🎀

EXCLUSIVE4_1 and EXCLUSIVE4 are the strict ones 👑 They mean the server should follow exclusive creation semantics, using the verifier to make sure the target is created exclusively. The server should check for a duplicate object by name. If the object doesn't exist, the server creates it and stores the verifier with it. If the object does exist and the stored verifier matches the verifier the client sent, the server uses the existing object as the newly created one. But if the stored verifier doesn't match, you get an NFS4ERR_EXIST error 😭

If you're using EXCLUSIVE4, and the server uses attributes to store the exclusive create verifier, the server tells you which attributes it used by setting the matching bits in the attribute mask it returns in the results 🦋 EXCLUSIVE4 doesn't let you set attributes at file creation, which is kind of a bummer. So after a successful OPEN via EXCLUSIVE4, the client MUST send a SETATTR to set attributes to a known state 💅

In NFSv4.1, EXCLUSIVE4 got deprecated in favor of EXCLUSIVE4_1 🥀 Unlike EXCLUSIVE4, EXCLUSIVE4_1 lets you provide attributes. But because the server might use attributes on the target object to store the verifier, the set of allowable attributes could be smaller than what SETATTR allows. The allowable attributes for EXCLUSIVE4_1 are listed in the suppattr_exclcreat (Section 5.8.1.14) attribute. If the client tries to set an attribute in cva_attrs that isn't in suppattr_exclcreat, the server MUST return NFS4ERR_INVAL 🚫 The response field, attrset, tells you both which attributes the server set from cva_attrs and which attributes it used to store the verifier. As described in Section 18.16.4, the client can compare cva_attrs.attrmask with attrset to figure out which attributes were used for the verifier 🔍

Now, with persistent sessions and pNFS in the mix, under some conditions EXCLUSIVE4 MUST NOT be used by the client or supported by the server 🙅‍♀️ Here's the cheat sheet for the appropriate and mandated exclusive create methods for NFSv4.1 implementations:

                 Required methods for exclusive create

 +----------------+-----------+---------------+----------------------+
 | Persistent     | Server    | Server        | Client Allowed       |
 | Reply Cache    | Supports  | REQUIRED      |                      |
 | Enabled        | pNFS      |               |                      |
 +----------------+-----------+---------------+----------------------+
 | no             | no        | EXCLUSIVE4_1  | EXCLUSIVE4_1         |
 |                |           | and           | (SHOULD) or          |
 |                |           | EXCLUSIVE4    | EXCLUSIVE4 (SHOULD   |
 |                |           |               | NOT)                 |
 | no             | yes       | EXCLUSIVE4_1  | EXCLUSIVE4_1         |
 | yes            | no        | GUARDED4      | GUARDED4             |
 | yes            | yes       | GUARDED4      | GUARDED4             |
 +----------------+-----------+---------------+----------------------+

                               Table 10

If CREATE_SESSION4_FLAG_PERSIST is set in the results of CREATE_SESSION, the reply cache is persistent (see Section 18.36) 💾 If the EXCHGID4_FLAG_USE_PNFS_MDS flag is set in the results from EXCHANGE_ID, the server is a pNFS server (see Section 18.35) 🌈 And if the client tries to use EXCLUSIVE4 on a persistent session, or on a session derived from an EXCHGID4_FLAG_USE_PNFS_MDS client ID, the server MUST return NFS4ERR_INVAL 🚨

With persistent sessions, exclusive create semantics are fully achievable through GUARDED4, so EXCLUSIVE4 or EXCLUSIVE4_1 MUST NOT be used 🫶 When pNFS is in use, the layout_hint attribute might not be supported after the file is created. Only the EXCLUSIVE4_1 and GUARDED methods of exclusive file creation allow the atomic setting of attributes ⭐

For the target directory, the server returns change_info4 information in cinfo. With the atomic field of the change_info4 data type, the server will indicate whether the before and after change attributes were obtained atomically with respect to the link creation 🔗

The OPEN operation brings the Windows share reservation capability through the share_access and share_deny fields of the OPEN arguments 🪟 The client specifies the required share_access and share_deny modes at OPEN. Clients that don't directly support SHAREs (i.e., UNIX) are expected to use OPEN4_SHARE_DENY_NONE as the deny value. If there's an existing SHARE reservation that conflicts with the OPEN request, the server returns NFS4ERR_SHARE_DENIED 🚧 For more SHARE semantics tea, check out Section 9.7 ☕

For each OPEN, the client provides a value for the owner field of the OPEN argument. The owner field is of data type open_owner4 and contains a field called clientid and a field called owner. The client can set clientid to any value, and the server MUST ignore it 🙃 Instead, the server MUST derive the client ID from the session ID of the SEQUENCE operation in the COMPOUND request 🧠

The "seqid" field of the request isn't used in NFSv4.1, but it MAY be any value, and the server MUST ignore it 🌙

Sometimes a client is recovering state after a server failure. In that case, the claim field of the OPEN argument signals that the request is meant to reclaim state it previously held 🔄

The "claim" field of the OPEN argument is what specifies the file to be opened and the state information the client claims to possess 📜 There are seven claim types as follows:

   +----------------------+--------------------------------------------+
   | open type            | description                                |
   +----------------------+--------------------------------------------+
   | CLAIM_NULL, CLAIM_FH | For the client, this is a new OPEN request |
   |                      | and there is no previous state associated  |
   |                      | with the file for the client.  With        |
   |                      | CLAIM_NULL, the file is identified by the  |
   |                      | current filehandle and the specified       |
   |                      | component name.  With CLAIM_FH (new to     |
   |                      | NFSv4.1), the file is identified by just   |
   |                      | the current filehandle.                    |
   | CLAIM_PREVIOUS       | The client is claiming basic OPEN state    |
   |                      | for a file that was held previous to a     |
   |                      | server restart.  Generally used when a     |
   |                      | server is returning persistent             |
   |                      | filehandles; the client may not have the   |
   |                      | file name to reclaim the OPEN.             |
   | CLAIM_DELEGATE_CUR,  | The client is claiming a delegation for    |
   | CLAIM_DELEG_CUR_FH   | OPEN as granted by the server.  Generally, |
   |                      | this is done as part of recalling a        |
   |                      | delegation.  With CLAIM_DELEGATE_CUR, the  |
   |                      | file is identified by the current          |
   |                      | filehandle and the specified component     |
   |                      | name.  With CLAIM_DELEG_CUR_FH (new to     |
   |                      | NFSv4.1), the file is identified by just   |
   |                      | the current filehandle.                    |
   | CLAIM_DELEGATE_PREV, | The client is claiming a delegation        |
   | CLAIM_DELEG_PREV_FH  | granted to a previous client instance;     |
   |                      | used after the client restarts.  The       |
   |                      | server MAY support CLAIM_DELEGATE_PREV     |
   |                      | and/or CLAIM_DELEG_PREV_FH (new to         |
   |                      | NFSv4.1).  If it does support either claim |
   |                      | type, CREATE_SESSION MUST NOT remove the   |
   |                      | client's delegation state, and the server  |
   |                      | MUST support the DELEGPURGE operation.     |
   +----------------------+--------------------------------------------+

Okay so that table is the whole claim-type lineup 🎀 Now, if an OPEN shows up during the grace period, the server says NFS4ERR_GRACE 🙈 unless it's one of these VIP exceptions:

• CLAIM_PREVIOUS is the reclaim-after-restart queen 👑. These OPENs are devoted to getting your opens back after a server restart, and they're typically only valid during the grace period.
• CLAIM_DELEGATE_CUR and CLAIM_DELEG_CUR_FH are valid both during and after the grace period 💖. The delegation they hang off already guarantees no other client has locks that need reclaiming, so the server need not return NFS4ERR_GRACE for these during the grace period. Iconic, no notes.

For any OPEN request, the server may return an OPEN delegation, which lets further opens and closes get handled locally on the client (see Section 10.4) ✨. Big reminder: delegation is 100% the server's call 💅 The client should never assume that delegation will or will not be granted in a particular instance. It should always be prepared for either case 🫶. There's one partial exception: the reclaim (CLAIM_PREVIOUS) case, where a delegation type is claimed. Here, delegation will always be granted, although the server may specify an immediate recall in the delegation structure 😭.

The rflags returned by a successful OPEN let the server tell you how the open file is supposed to be handled 🦋.

• OPEN4_RESULT_CONFIRM is deprecated and MUST NOT be returned by an NFSv4.1 server 💀.
• OPEN4_RESULT_LOCKTYPE_POSIX indicates that the server's byte-range locking behavior supports the complete set of POSIX locking techniques [24]. From this, the client can choose to manage byte-range locking state in a way to handle a mismatch of byte-range locking management 🔒.
• OPEN4_RESULT_PRESERVE_UNLINKED indicates that the server will preserve the open file if the client (or any other client) removes the file as long as it is open. Furthermore, the server promises to preserve the file through the grace period after server restart, thereby giving the client the opportunity to reclaim its open 🌸.
• OPEN4_RESULT_MAY_NOTIFY_LOCK indicates that the server may attempt CB_NOTIFY_LOCK callbacks for locks on this file. This flag is a hint only, and may be safely ignored by the client 🍓.

If the component is of zero length, NFS4ERR_INVAL will be returned. The component is also subject to the normal UTF-8, character support, and name checks. See Section 14.5 for further discussion 🧚.

When an OPEN is done and the specified open-owner already has the resulting filehandle open, the result is to "OR" together the new share and deny status together with the existing status. In this case, only a single CLOSE needs to be done, even though multiple OPENs were completed 💜. When such an OPEN is done, checking of share reservations for the new OPEN proceeds normally, with no exception for the existing OPEN held by the same open-owner. In this case, the stateid returned has an "other" field that matches that of the previous open, while the "seqid" field is incremented to reflect the change in status due to the new open ✨.

If the underlying file system at the server is only accessible in a read-only mode and the OPEN request has specified ACCESS_WRITE or ACCESS_BOTH, the server will return NFS4ERR_ROFS to indicate a read-only file system 🙅‍♀️.

---

As with the CREATE operation, the server MUST derive the owner, owner ACE, group, or group ACE if any of the four attributes are required and supported by the server's file system 👑. For an OPEN with the EXCLUSIVE4 createmode, the server has no choice, since such OPEN calls do not include the createattrs field 🤷‍♀️. Conversely, if createattrs (UNCHECKED4 or GUARDED4) or cva_attrs (EXCLUSIVE4_1) is specified, and includes an owner, owner_group, or ACE that the principal in the RPC call's credentials does not have authorization to create files for, then the server may return NFS4ERR_PERM 🚫.

In the case of an OPEN that specifies a size of zero (e.g., truncation) and the file has named attributes, the named attributes are left as is and are not removed 🧸.

NFSv4.1 gives more precise control to clients over acquisition of delegations via the following new flags for the share_access field of OPEN4args 🎯:

• OPEN4_SHARE_ACCESS_WANT_READ_DELEG
• OPEN4_SHARE_ACCESS_WANT_WRITE_DELEG
• OPEN4_SHARE_ACCESS_WANT_ANY_DELEG
• OPEN4_SHARE_ACCESS_WANT_NO_DELEG
• OPEN4_SHARE_ACCESS_WANT_CANCEL
• OPEN4_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_RESRC_AVAIL
• OPEN4_SHARE_ACCESS_WANT_PUSH_DELEG_WHEN_UNCONTENDED

If (share_access & OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) is not zero, then the client will have specified one and only one of:

• OPEN4_SHARE_ACCESS_WANT_READ_DELEG
• OPEN4_SHARE_ACCESS_WANT_WRITE_DELEG
• OPEN4_SHARE_ACCESS_WANT_ANY_DELEG
• OPEN4_SHARE_ACCESS_WANT_NO_DELEG
• OPEN4_SHARE_ACCESS_WANT_CANCEL

Otherwise, the client is neither indicating a desire nor a non-desire for a delegation, and the server MAY or MAY not return a delegation in the OPEN response 🤞.

If the server supports the new _WANT_ flags and the client sends one or more of the new flags, then in the event the server does not return a delegation, it MUST return a delegation type of OPEN_DELEGATE_NONE_EXT. The field ond_why in the reply indicates why no delegation was returned, and it will be one of:

• WND4_NOT_WANTED: The client specified OPEN4_SHARE_ACCESS_WANT_NO_DELEG. 🙅‍♀️
• WND4_CONTENTION: There is a conflicting delegation or open on the file. ⚔️
• WND4_RESOURCE: Resource limitations prevent the server from granting a delegation. 🪫
• WND4_NOT_SUPP_FTYPE: The server does not support delegations on this file type. 🚫
• WND4_WRITE_DELEG_NOT_SUPP_FTYPE: The server does not support OPEN_DELEGATE_WRITE delegations on this file type. ✍️
• WND4_NOT_SUPP_UPGRADE: The server does not support atomic upgrade of an OPEN_DELEGATE_READ delegation to an OPEN_DELEGATE_WRITE delegation. ⬆️
• WND4_NOT_SUPP_DOWNGRADE: The server does not support atomic downgrade of an OPEN_DELEGATE_WRITE delegation to an OPEN_DELEGATE_READ delegation. ⬇️
• WND4_CANCELED: The client specified OPEN4_SHARE_ACCESS_WANT_CANCEL and now any "want" for this file object is cancelled. 🧹
• WND4_IS_DIR: The specified file object is a directory, and the operation is OPEN or WANT_DELEGATION, which do not support delegations on directories. 📁

OPEN4_SHARE_ACCESS_WANT_READ_DELEG, OPEN_SHARE_ACCESS_WANT_WRITE_DELEG, or OPEN_SHARE_ACCESS_WANT_ANY_DELEG mean, respectively, that the client wants an OPEN_DELEGATE_READ, OPEN_DELEGATE_WRITE, or any delegation, regardless of which of OPEN4_SHARE_ACCESS_READ, OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH is set 🎀. If the client has an OPEN_DELEGATE_READ delegation on a file and requests an OPEN_DELEGATE_WRITE delegation, then the client is requesting atomic upgrade of its OPEN_DELEGATE_READ delegation to an OPEN_DELEGATE_WRITE delegation. If the client has an OPEN_DELEGATE_WRITE delegation on a file and requests an OPEN_DELEGATE_READ delegation, then the client is requesting atomic downgrade to an OPEN_DELEGATE_READ delegation. A server MAY support atomic upgrade or downgrade. If it does, then a returned delegation_type of OPEN_DELEGATE_READ or OPEN_DELEGATE_WRITE that is different from the delegation type the client currently has indicates a successful upgrade or downgrade 🥳. If the server does not support atomic delegation upgrade or downgrade, then ond_why will be set to WND4_NOT_SUPP_UPGRADE or WND4_NOT_SUPP_DOWNGRADE 😤.

OPEN4_SHARE_ACCESS_WANT_NO_DELEG means that the client wants no delegation 🙅‍♀️.

OPEN4_SHARE_ACCESS_WANT_CANCEL means that the client wants no delegation and wants to cancel any previously registered "want" for a delegation 🧹.

The client may set one or both of OPEN4_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_RESRC_AVAIL and OPEN4_SHARE_ACCESS_WANT_PUSH_DELEG_WHEN_UNCONTENDED. However, they will have no effect unless one of the following is set:

• OPEN4_SHARE_ACCESS_WANT_READ_DELEG
• OPEN4_SHARE_ACCESS_WANT_WRITE_DELEG
• OPEN4_SHARE_ACCESS_WANT_ANY_DELEG

If the client specifies OPEN4_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_RESRC_AVAIL, then it wishes to register a "want" for a delegation, in the event the OPEN results do not include a delegation 🔔. If so, and the server denies the delegation due to insufficient resources, the server MAY later inform the client, via the CB_RECALLABLE_OBJ_AVAIL operation, that the resource limitation condition has eased 📣. The server will tell the client that it intends to send a future CB_RECALLABLE_OBJ_AVAIL operation by setting delegation_type in the results to OPEN_DELEGATE_NONE_EXT, ond_why to WND4_RESOURCE, and ond_server_will_signal_avail set to TRUE. If ond_server_will_signal_avail is set to TRUE, the server MUST later send a CB_RECALLABLE_OBJ_AVAIL operation 💜.

Okay so if the client sets OPEN4_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_UNCONTENDED 🦋, it's basically saying "hey, I'd love a delegation 🥺 if the OPEN results don't come with one." That's registering a "want" for a delegation ✨. If the server says no because of contention, it MAY later send the client a CB_PUSH_DELEG operation to say the contention has eased 💅. The server signals that plan by setting delegation_type to OPEN_DELEGATE_NONE_EXT, ond_why to WND4_CONTENTION, and ond_server_will_push_deleg to TRUE 💜. And once ond_server_will_push_deleg is TRUE, the server MUST later send that CB_PUSH_DELEG operation, no excuses 🔥.

If the client already registered a want for a delegation on a file and then asks to register another want on that same file, the server MUST return a brand new error: NFS4ERR_DELEG_ALREADY_WANTED 🙈. Wanting a different type of delegation on the same file? Then the client MUST cancel the existing delegation WANT first 💅.

18.16.4. IMPLEMENTATION ✨

Without a persistent session, the client does an exclusive create by setting how to EXCLUSIVE4 or EXCLUSIVE4_1 🧚. In that case, the client hands over a verifier that's reasonably expected to be unique 🔑. Something like a client identifier (maybe its network address) plus a unique number the client generated (maybe the RPC transaction identifier) can work well.

If the object doesn't exist, the server creates it and stores the verifier in stable storage 💾. For file systems that can't store arbitrary file attributes, the server may use one or more elements of the object's metadata to hold the verifier. The verifier MUST be stored in stable storage so a retransmitted request doesn't erroneously fail 🙏. Exclusive create is used because exclusive semantics are critical to the application 👑. So exclusive CREATE doesn't rely only on the server's reply cache to store the verifier 🫶. A nonpersistent reply cache doesn't survive a crash, and the session and reply cache may get deleted after a network partition that outlasts the lease time, which opens failure windows 💀.

An NFSv4.1 server SHOULD NOT store the verifier in any of the file's RECOMMENDED or REQUIRED attributes 🎀. If it does anyway, the server SHOULD use time_modify_set or time_access_set to store the verifier. The server SHOULD NOT store the verifier in these attributes either:

• acl (access control is best set up at creation)
• dacl (same vibe)
• mode (same vibe)
• owner (same vibe)
• owner_group (same vibe)
• retentevt_set (you may want retention established at creation)
• retention_hold (same vibe)
• retention_set (same vibe)
• sacl (auditing control is best set up at creation)
• size (on some servers, size only takes a limited range of values)
• mode_set_masked (same as mode)
• time_creation (a meaningful creation time should be set when the file is created)

Another option for the server is to use a named attribute to store the verifier 🌸.

Because the EXCLUSIVE4 create method doesn't specify initial attributes when it's processed, the server:

• SHOULD set the owner of the file to the one matching the credential in the request's RPC header.
• SHOULD NOT leave the file's access control to anyone but the owner of the file.

If the server can't support exclusive create semantics, maybe because it can't commit the verifier to stable storage, it should fail the OPEN request with the error NFS4ERR_NOTSUPP 😭.

During an exclusive CREATE request, if the object already exists, the server rebuilds the object's verifier and compares it with the verifier in the request 🔍. If they match, the server treats the request as a success. That's presumed to be a duplicate of an earlier successful request whose reply got lost, and the server's duplicate request cache didn't catch it. If the verifiers don't match, the request gets rejected with the status NFS4ERR_EXIST 🚫.

After a successful exclusive create, the attrset response tells you which attributes were used to store the verifier 👀. If EXCLUSIVE4 was used, the attributes set in attrset were used for the verifier. If EXCLUSIVE4_1 was used, the client figures out the verifier attributes by comparing attrset with cva_attrs.attrmask. Any bits set in the former but not the latter identify the attributes that store the verifier. The client MUST immediately send a SETATTR to set the attributes used to store the verifier 🏃‍♀️. Until it does, those attributes can't be relied upon. And the subsequent SETATTR MUST NOT occur in the same COMPOUND request as the OPEN.

Unless a persistent session is used, the GUARDED4 attribute doesn't give you exactly once semantics ⚠️. If a reply gets lost and the server doesn't detect the retransmission, the operation can fail with NFS4ERR_EXIST even though the create actually worked. The client would use this behavior when the application hasn't asked for an exclusive create but does want the file truncated when it's opened. If the client times out and retransmits the create request, the client can use GUARDED4 to guard against a sequence like create, write, create (retransmitted).

For SHARE reservations, the value of the expression (share_access & ~OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) MUST be one of OPEN4_SHARE_ACCESS_READ, OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH. If it isn't, the server MUST return NFS4ERR_INVAL 🙅‍♀️. The value of share_deny MUST be one of OPEN4_SHARE_DENY_NONE, OPEN4_SHARE_DENY_READ, OPEN4_SHARE_DENY_WRITE, or OPEN4_SHARE_DENY_BOTH. If it isn't, the server MUST return NFS4ERR_INVAL too.

Based on the share_access value (OPEN4_SHARE_ACCESS_READ, OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH), the client should check that the requester has the right access to do the operation 🔐. That usually comes from applying the ACL access rules to the file for the current requester. But just like with the ACCESS operation, the client should not second-guess the server's decisions, since access rights can change and may be subject to server administrative controls outside the ACL framework 🤷‍♀️. If the requester's READ or WRITE operation isn't authorized (depending on the share_access value), the server MUST return NFS4ERR_ACCESS 🚫.

Here's a tricky one 🧐: if the client ID wasn't created with the EXCHGID4_FLAG_BIND_PRINC_STATEID capability set in the reply to EXCHANGE_ID, the server MUST NOT impose any requirement that READs and WRITEs sent for an open file have the same credentials as the OPEN itself. The server is REQUIRED to perform access checking on the READs and WRITEs themselves. Otherwise, if the EXCHANGE_ID reply did have EXCHGID4_FLAG_BIND_PRINC_STATEID set, then the credentials used in the OPEN request MUST match those used in the READs and WRITEs, and the stateids in the READs and WRITEs MUST match or be derived from the stateid in the reply to OPEN. There's one exception 🎀: if SP4_SSV or SP4_MACH_CRED state protection is used, and the spo_must_allow result of EXCHANGE_ID includes the READ and/or WRITE operations, then the machine or SSV credential will be allowed to send READ and/or WRITE. See Section 18.35.

If the component given to OPEN is a symbolic link, the error NFS4ERR_SYMLINK gets returned to the client 🔗. If it's a directory, you get NFS4ERR_ISDIR 📁. If the component is neither of those but still not an ordinary file, the error NFS4ERR_WRONG_TYPE comes back. And if the current filehandle isn't a directory, the error NFS4ERR_NOTDIR gets returned 🙃.

The OPEN4_RESULT_PRESERVE_UNLINKED result flag lets a client skip the usual trick of renaming an open file to ".nfs<unique value>" after removing it 🙌. After the server returns OPEN4_RESULT_PRESERVE_UNLINKED, if a client sends a REMOVE operation that would drop the file's link count to zero, the server SHOULD report a value of zero for the numlinks attribute on the file.

If another client holds a delegation on the file being opened that conflicts with the open (sometimes depending on the share_access or share_deny value), that delegation MUST be recalled 📣. The operation can't proceed until each such delegation is returned or revoked. Unless that happens super quickly, one or more NFS4ERR_DELAY errors will be returned to requests made while the delegation is still outstanding ⏳. For an OPEN_DELEGATE_WRITE delegation, any open by a different client conflicts. For an OPEN_DELEGATE_READ delegation, only opens with one of these characteristics count as conflicting:

• The value of share_access includes the bit OPEN4_SHARE_ACCESS_WRITE.
• The value of share_deny specifies OPEN4_SHARE_DENY_READ or OPEN4_SHARE_DENY_BOTH.
• OPEN4_CREATE is specified together with UNCHECKED4, the size attribute is specified as zero (for truncation), and an existing file gets truncated.

If OPEN4_CREATE is specified, the file doesn't exist, and the current filehandle designates a directory that another client holds a directory delegation on, then the delegation MUST be recalled, unless the delegation is such that the situation can be resolved by sending a notification 📬. The operation can't proceed until the delegation is returned or revoked. Unless that happens very quickly, one or more NFS4ERR_DELAY errors will be returned to requests made while the delegation is outstanding.

If OPEN4_CREATE is specified, the file doesn't exist, and the current filehandle designates a directory with one or more directory delegations, then, when those delegations ask for such notifications, NOTIFY4_ADD_ENTRY gets generated as a result of this operation 🔔.

18.16.4.1. Warning to Client Implementors ⚠️

OPEN is kind of like LOOKUP in that it generates a filehandle for the client to use. But unlike LOOKUP, OPEN creates server state on the filehandle 🧠. Normally, the client can only release that state with a CLOSE operation. CLOSE uses the current filehandle to figure out which file to close. So the client MUST follow every OPEN operation with a GETFH operation in the same COMPOUND procedure 🫶. That gives the client the filehandle it needs so CLOSE can be used properly.

Just waiting for the lease on the file to expire is not enough 🙅‍♀️. The server may keep the state indefinitely as long as no other client tries to make a conflicting access to the same file.

See also Section 2.10.6.4 💅.

18.17. Operation 19: OPENATTR - Open Named Attribute Directory ✨

18.17.1. ARGUMENTS

struct OPENATTR4args {
        /* CURRENT_FH: object */
        bool    createdir;
};

18.17.2. RESULTS

okay so the RESULTS for OPENATTR are super simple, bestie 🎀 if it works (NFS4_OK), your new CURRENT_FH is the named attribute directory 📁✨

   struct OPENATTR4res {
           /*
            * If status is NFS4_OK,
            *   new CURRENT_FH: named attribute
            *                   directory
            */
           nfsstat4        status;
   };

18.17.3. DESCRIPTION

so OPENATTR is how you grab the filehandle for the named attribute directory attached to your current filehandle 🗂️💅 the result is a filehandle pointing at an object of type NF4ATTRDIR. from there you can use READDIR and LOOKUP to get filehandles for each named attribute on the original file system object, and those ones are objects of type NF4NAMEDATTR 🧚

the createdir argument is where it gets fun. it's how the client says whether the named attribute directory should be made as part of OPENATTR 🙈 some clients send createdir as FALSE just to check if the object has any named attributes at all. if there are none, you get NFS4ERR_NOENT back 💀

if createdir is TRUE and there's no named attribute directory yet, one gets created and its filehandle becomes the current filehandle 🌸 if createdir is TRUE and the directory already exists, nothing goes wrong. no error, and the existing directory's filehandle becomes the current filehandle 🫶

ngl the creation part is a whole vibe: it only works if the server has set up named attribute support in this particular way. the server isn't required to do that by this definition. 🦋

if your current filehandle is already an object of type NF4NAMEDATTR (a named attribute) or NF4ATTRDIR (a named attribute directory), you get NFS4ERR_WRONG_TYPE 😭 named attributes and named attribute directories MUST NOT have their own named attributes. no nesting, bestie 🚫

18.17.4. IMPLEMENTATION

if the server doesn't support named attributes for the current filehandle, the client gets NFS4ERR_NOTSUPP 🥀

18.18. Operation 21: OPEN_DOWNGRADE - Reduce Open File Access ✨

18.18.1. ARGUMENTS

   struct OPEN_DOWNGRADE4args {
           /* CURRENT_FH: opened file */
           stateid4        open_stateid;
           seqid4          seqid;
           uint32_t        share_access;
           uint32_t        share_deny;
   };

18.18.2. RESULTS

   struct OPEN_DOWNGRADE4resok {
           stateid4        open_stateid;
   };

   union OPEN_DOWNGRADE4res switch(nfsstat4 status) {
    case NFS4_OK:
           OPEN_DOWNGRADE4resok    resok4;
    default:
            void;
   };

18.18.3. DESCRIPTION

this one is all about lowering your access and deny states on an open 🔽💖 it's needed when the same open-owner has opened the same file more than once with different access and deny values. closing one of those opens can mean the share_access and share_deny flags need to drop the bits that belonged to opens that are no longer in effect 🧹

okay here's the valid-values tea 🍓

• the expression (share_access & ~OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) can only be OPEN4_SHARE_ACCESS_READ, OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH. anything else and the server MUST reply with NFS4ERR_INVAL 🙅‍♀️
• share_deny can only be OPEN4_SHARE_DENY_NONE, OPEN4_SHARE_DENY_READ, OPEN4_SHARE_DENY_WRITE, or OPEN4_SHARE_DENY_BOTH. anything else and the server MUST reply with NFS4ERR_INVAL 🙅‍♀️

once those checks pass, the server swaps the current access and deny modes on the file for share_access and share_deny, with a couple of rules attached 👑

• the bits in share_access SHOULD equal the union of the share_access bits (not including OPEN4_SHARE_WANT_* bits) specified for some subset of the OPENs in effect for the current open-owner on the current file ✨
• the bits in share_deny SHOULD equal the union of the share_deny bits specified for some subset of the OPENs in effect for the current open-owner on the current file 🫧

if those rules get broken, the server SHOULD return NFS4ERR_INVAL 🙃 the big payoff: since the share_access and share_deny bits should be subsets of what's already granted, OPEN_DOWNGRADE can't get denied because of conflicting share reservations. the only exception is a defect in the client or server implementation 🫠

the seqid argument isn't used in NFSv4.1. it MAY be any value, and the server MUST ignore it 🤷‍♀️

on success, the current filehandle keeps its value 🎀

18.18.4. IMPLEMENTATION

an OPEN_DOWNGRADE can make OPEN_DELEGATE_READ delegations grantable where they weren't before 🌈 servers may choose to answer right away if there are pending delegation want requests, or they can handle it later on 🕰️

18.19. Operation 22: PUTFH - Set Current Filehandle 🧚

18.19.1. ARGUMENTS

   struct PUTFH4args {
           nfs_fh4         object;
   };

18.19.2. RESULTS

   struct PUTFH4res {
           /*
            * If status is NFS4_OK,
            *    new CURRENT_FH: argument to PUTFH
            */
           nfsstat4        status;
   };

18.19.3. DESCRIPTION

PUTFH swaps out the current filehandle for whatever filehandle you pass in as the argument, and it clears the current stateid 🧹

if the security mechanism the requester is using doesn't meet what the filehandle requires, the server MUST return NFS4ERR_WRONGSEC 🔐 bestie, that's a hard no.

check out Section 16.2.3.1.1 for more on the current filehandle 📖 and Section 16.2.3.1.2 for more on the current stateid 📖

18.19.4. IMPLEMENTATION

this is how a client sets up the context for the file accessing operations that come after it in the same COMPOUND request 🎬

18.20. Operation 23: PUTPUBFH - Set Public Filehandle 🌐

18.20.1. ARGUMENT

   void;

18.20.2. RESULT

   struct PUTPUBFH4res {
           /*
            * If status is NFS4_OK,
            *   new CURRENT_FH: public fh
            */
           nfsstat4        status;
   };

18.20.3. DESCRIPTION

PUTPUBFH replaces the current filehandle with the one representing the public filehandle of the server's namespace 🌍 this one can be different from the "root" filehandle that's tied to some other directory on the server.

it also clears the current stateid 🧹

the public filehandle is the idea behind RFC 2054 [42], RFC 2055 [43], and RFC 2224 [53]. for NFSv4.1 the goal is for the public filehandle (through PUTPUBFH) to give you WebNFS server compatibility with NFSv3 🫶

the public filehandle and the root filehandle (through PUTROOTFH) SHOULD be equivalent 👯 if they aren't, then the directory for the public filehandle MUST be a descendant of the directory for the root filehandle 🌳

see Section 16.2.3.1.1 for the current filehandle details and Section 16.2.3.1.2 for the current stateid details 📖

18.20.4. IMPLEMENTATION

same deal as always: this sets up the context for the file accessing operations that follow in the same COMPOUND request 🎬

with the NFSv3 public filehandle, the client could say whether a LOOKUP pathname should be read as an absolute path from the server's root or relative to the public filehandle 🧭 RFC 2224 [53] has more on that. NFSv4.1 can't do that through LOOKUP, because the component separators you'd need for absolute vs. relative aren't allowed in NFSv4 🙅‍♀️ so the client has to build its request so that PUTROOTFH or PUTPUBFH tells the server whether to evaluate an NFS URL absolutely or relatively 🛣️

heads up: RFC 2224 [53] has warnings about absolute evaluation and about how much of its namespace the server might expose. those warnings apply to NFSv4.1 too 👀 because of server implementation details, an NFSv3 absolute public filehandle lookup might act differently from an NFSv4.1 absolute resolution. 😬

there's a security negotiation form in RFC 2755 [54] that uses the public filehandle and overloads the pathname. that doesn't exist in NFSv4.1, since filehandles don't carry special meaning there and don't give you the same framework as NFSv3 🙃 clients should use the security negotiation mechanisms in Section 2.6 instead 🔐

18.21. Operation 24: PUTROOTFH - Set Root Filehandle 🌳

18.21.1. ARGUMENTS

   void;

18.21.2. RESULTS

   struct PUTROOTFH4res {
           /*
            * If status is NFS4_OK,
            *   new CURRENT_FH: root fh
            */
           nfsstat4        status;
   };

18.21.3. DESCRIPTION

PUTROOTFH replaces the current filehandle with the one representing the root of the server's namespace 🌱 from there, a LOOKUP can find any other filehandle on the server 🔍 this might be different from the "public" filehandle tied to some other directory on the server.

PUTROOTFH also clears the current stateid 🧹

see Section 16.2.3.1.1 for more on the current filehandle 📖 and Section 16.2.3.1.2 for more on the current stateid 📖

18.21.4. IMPLEMENTATION

this is another context-setter for the file accessing operations later in the same COMPOUND request 🎀

18.22. Operation 25: READ - Read from File 📖

18.22.1. ARGUMENTS

   struct READ4args {
           /* CURRENT_FH: file */
           stateid4        stateid;
           offset4         offset;
           count4          count;
   };

18.22.2. RESULTS

   struct READ4resok {
           bool            eof;
           opaque          data<>;
   };

   union READ4res switch (nfsstat4 status) {
    case NFS4_OK:
            READ4resok     resok4;
    default:
            void;
   };

18.22.3. DESCRIPTION ✨

okay so READ is the op that grabs data out of the regular file sitting in the current filehandle 📂💖

the client tells the server where to start (that's the offset, and offset zero means the very beginning of the file 🌸) and how many bytes it wants (the count). if offset is greater than or equal to the size of the file, you get NFS4_OK back with a data length of zero and eof set to TRUE. that's giving "nothing left here babe" 💀. and the READ still has to pass access permissions checking, no skipping that 🔐

if the client asks for a count of zero, the READ succeeds and you get zero bytes back, still subject to access permissions checking, obviously 🙈. the server MAY choose to return fewer bytes than the client asked for, so the client needs to check for that and handle it properly 🫶

unless special stateids are being used, the stateid for a READ is the value you got back from an earlier byte-range lock or share reservation request, or the stateid tied to a delegation 🔑. it tells the server who the owners are (if there are any) and lets the server double check that the locks are still valid, like they haven't been revoked 🕵️‍♀️

eof comes back TRUE if the read ended right at end-of-file (formally, in a correctly formed READ, offset + count equals the size of the file), or if the READ ran past the end (offset + count is greater than the size of the file). otherwise it's FALSE. and a successful READ of an empty file will always return eof as TRUE 🌈

if the current filehandle isn't an ordinary file, the client gets an error 🙅‍♀️. if it's an object of type NF4DIR, that's NFS4ERR_ISDIR. if it's a symbolic link, that's NFS4ERR_SYMLINK. everything else gets NFS4ERR_WRONG_TYPE.

for a READ with a stateid where all bits are set to zero, the server MAY let the READ go ahead subject to mandatory byte-range locks or the current share deny modes for the file. for a READ with a stateid where all bits are set to one, the server MAY let READ operations skip the locking checks at the server 👑

on success, the current filehandle keeps its value ✅

18.22.4. IMPLEMENTATION

if the server gives back a "short read" (fewer data than requested, with eof set to FALSE), the client should send another READ to get the rest 🍓. a server can return less data than requested for a few reasons. the file might've been truncated, by another client or on the server itself, so the size changed from what the requesting client thinks, which shrinks the amount of data available 😭. it's also possible the server reduces the transfer size and returns a short read result, and server resource exhaustion can cause one too 💀

if mandatory byte-range locking is in effect for the file, and the byte-range being read is WRITE_LT locked by an owner not associated with the stateid, the server will return the NFS4ERR_LOCKED error 🔒. the client should try to get the right READ_LT via the LOCK operation before re-attempting the READ. once the READ completes, the client should release the byte-range lock via LOCKU 🎀

if another client has an OPEN_DELEGATE_WRITE delegation for the file being read, that delegation must be recalled, and the operation can't proceed until the delegation is returned or revoked. unless that happens super quickly, requests made while the delegation is still outstanding will get one or more NFS4ERR_DELAY errors 🕰️. normally delegations won't get recalled because of a READ, since the recall happens as a result of an earlier OPEN. but since a READ can be done with a special stateid, the server needs to check for that case even though the client should have done an OPEN beforehand 🧐

---

18.23. Operation 26: READDIR - Read Directory 📖

18.23.1. ARGUMENTS

struct READDIR4args {
        /* CURRENT_FH: directory */
        nfs_cookie4     cookie;
        verifier4       cookieverf;
        count4          dircount;
        count4          maxcount;
        bitmap4         attr_request;
};

18.23.2. RESULTS

struct entry4 {
        nfs_cookie4     cookie;
        component4      name;
        fattr4          attrs;
        entry4          *nextentry;
};

struct dirlist4 {
        entry4          *entries;
        bool            eof;
};
struct READDIR4resok {
        verifier4       cookieverf;
        dirlist4        reply;
};

union READDIR4res switch (nfsstat4 status) {
 case NFS4_OK:
         READDIR4resok  resok4;
 default:
         void;
};

18.23.3. DESCRIPTION

READDIR pulls a variable number of entries out of a file system directory 📚. it returns the attributes the client asked for on each entry, plus the info the client needs to request more entries in a later READDIR 💅

the arguments carry a cookie that says where the READDIR should start in the directory. a cookie of zero means start at the beginning. for later requests, the client uses a cookie that the server gave back on a previous READDIR 🍪

the request's cookieverf field should be set to 0 (zero) when the request's cookie field is zero, which is the first read of the directory 🌱. on later requests, the cookieverf field must match the cookieverf returned by the READDIR where the cookie was acquired. if the server determines that the cookieverf is no longer valid for the directory, the error NFS4ERR_NOT_SAME must be returned 🙈

the dircount field of the request is a hint for the max number of bytes of directory info that should come back. that counts the total length of the directory entry names and the cookie values for those entries. the length is the XDR encoding of the data (names and cookies), not the length in the server's native format 📏

the maxcount field of the request is the max total size of all the data returned inside the READDIR4resok structure, XDR overhead included. the server MAY return less data. if the server can't fit even a single directory entry within the maxcount limit, the error NFS4ERR_TOOSMALL MUST be returned to the client 🙅

finally, the request's attr_request field is the list of attributes to return for each directory entry the server supplies 🎀

a successful reply is a list of directory entries. each one has its name, a cookie value for that entry, and the associated attributes as requested. the eof flag has a value of TRUE if there are no more entries in the directory 🏁

the cookie value only means something to the server, and it works as a cursor for the directory entry. the client uses it on later READDIR operations so it can keep reading the directory. it's kinda like a READ offset, but the client MUST NOT interpret it as one 🙅‍♀️. ideally, the cookie value SHOULD NOT change if the directory is modified, since the client may be caching these values 🧁

sometimes the server runs into an error while getting the attributes for one directory entry. instead of failing the entire READDIR, the server can return the attribute rdattr_error (Section 5.8.1.12) 💖. that way the server tells the client about the failure without failing the whole operation, which is great when it's just a transient hiccup ✨. but the client must request the fattr4_rdattr_error attribute for this to work. if the client doesn't ask for it, the server has no choice but to return failure for the entire READDIR operation 😬

for some file system environments, the directory entries "." and ".." have special meaning, and for others they don't. if the server supports these special entries within a directory, they SHOULD NOT be returned to the client as part of the READDIR response 🚫. to make some client environments happy, the cookie values zero, 1, and 2 are reserved. the UNIX client uses these values when it combines the server's response with its local representations, so the app gets a fully formed UNIX directory presentation 🐧

for READDIR arguments, cookie values of one and two SHOULD NOT be used, and for READDIR results, cookie values of zero, one, and two SHOULD NOT be returned 🙈

on success, the current filehandle keeps its value ✅

18.23.4. IMPLEMENTATION

server directory representations can be wildly different from each other, and a client's programming interfaces may be tied to the local OS in a way that doesn't map well to NFS 🧩. so the dircount and maxcount fields exist to give the server hints from the client. if the client is aggressive about attribute collection during a READDIR, the server gets an idea of how to limit the encoded response 🧠

if dircount is zero, the server bounds the reply's size based on the request's maxcount field 📐

the cookieverf can be used by the server to help manage cookie values that might go stale. it should be a rare occurrence that a server can't keep reading a directory with the provided cookie/cookieverf pair, and the server SHOULD make every effort to avoid that, since the app at the client might not be able to handle that kind of failure 😰

the cookieverf also protects the client from using READDIR cookie values that might be stale 💅 say the file system got migrated. the new server might or might not be able to use the same cookie values the previous server used. with the client providing the cookieverf, the server can send the appropriate response 🔄. this stops the case where the server accepts a cookie value even though the underlying directory has changed, and the response is invalid from the client's point of view

since some servers won't return "." and ".." entries the way previous versions of the NFS protocol did, a client that needs those entries present in READDIR responses has to fabricate them itself 🛠️

18.24. Operation 27: READLINK - Read Symbolic Link 🔗

18.24.1. ARGUMENTS

/* CURRENT_FH: symlink */
void;

18.24.2. RESULTS

struct READLINK4resok {
        linktext4       link;
};
union READLINK4res switch (nfsstat4 status) {
 case NFS4_OK:
         READLINK4resok resok4;
 default:
         void;
};

18.24.3. DESCRIPTION

READLINK reads the data associated with a symbolic link 🔗. depending on the value of the UTF-8 capability attribute (Section 14.4), the data comes back encoded in UTF-8 💜. whether it was created by an NFS client or locally on the server, the data in a symbolic link isn't interpreted when it's created (except possibly to check for proper UTF-8 encoding). it's just stored 💅

on success, the current filehandle keeps its value 💖

18.24.4. IMPLEMENTATION ✨

okay so a symbolic link is basically a pointer to another file 🦋 ngl the server doesn't really read what's inside it, it just stores the data. a client could totally stash a pathname in there that means nothing to the server's OS 💀 READLINK just hands the data back so the client can interpret it. if different implementations want to share access to symlinks, they gotta agree on how to read the data in them, periodt 💅

READLINK is only allowed on objects of type NF4LNK 🎀 if the object isn't NF4LNK, the server should return the error NFS4ERR_WRONG_TYPE.

18.25. Operation 28: REMOVE - Remove File System Object 🗑️

18.25.1. ARGUMENTS

struct REMOVE4args {
        /* CURRENT_FH: directory */
        component4      target;
};

18.25.2. RESULTS

struct REMOVE4resok {
        change_info4    cinfo;
};
union REMOVE4res switch (nfsstat4 status) {
 case NFS4_OK:
         REMOVE4resok   resok4;
 default:
         void;
};

18.25.3. DESCRIPTION

the REMOVE operation deletes a directory entry called filename, from the directory the current filehandle points to 🧚 if that entry was the last reference to the file system object, the object may be destroyed. the directory can be either NF4DIR or NF4ATTRDIR.

for the directory where the filename got removed, the server sends back change_info4 info in cinfo. with the atomic field of change_info4, the server tells you whether the before and after change attributes were grabbed atomically with respect to the removal 👀

if the target has a length of zero, or if the target doesn't follow the UTF-8 definition (and the server is enforcing UTF-8 encoding; see Section 14.4), you'll get NFS4ERR_INVAL 🙈

on success, the current filehandle keeps its value 🌸

18.25.4. IMPLEMENTATION

okay this one is a whole story 📖 NFSv3 needed two different operators: RMDIR for removing directories and REMOVE for removing non-directories. that meant clients could skip checking the file type when handed a non-directory delete call (like unlink() [27] in POSIX) to remove a directory, and the converse too (like rmdir() on a non-directory), because they knew the server would check the file type. NFSv4.1 REMOVE can delete any directory entry no matter its file type 🔥 so if you're implementing an NFSv4.1 client's entry points for the unlink() and rmdir() system calls, you should first check the file type against the types that system call is allowed to remove, before sending a REMOVE operation. or you can build a COMPOUND call that includes a LOOKUP/VERIFY sequence to verify the file type right before REMOVE, in the same COMPOUND call 🫶

the concept of last reference is server specific 🤷‍♀️ but if the numlinks field in the object's previous attributes was 1, the client should not rely on using a filehandle to get to the object. likewise, the client should not rely on the resources (disk space, directory entry, and so on) that used to belong to the object becoming free right away 😭 so if a client needs to keep accessing a file after using REMOVE on it, it should make sure the file will still be accessible. the old traditional move is to RENAME the file to a new hidden name 🕵️‍♀️ but the NFSv4.1 OPEN operation MAY return a result flag, OPEN4_RESULT_PRESERVE_UNLINKED, which tells the client the file will be preserved if it has an outstanding open (see Section 18.16) ✨

if the server finds the file is still open when the REMOVE arrives:

• the server SHOULD NOT delete the file's directory entry if the file was opened with OPEN4_SHARE_DENY_WRITE or OPEN4_SHARE_DENY_BOTH 🛑
• if the file was NOT opened with OPEN4_SHARE_DENY_WRITE or OPEN4_SHARE_DENY_BOTH, the server SHOULD delete the file's directory entry. however, until the last CLOSE of the file, the server MAY continue to allow access to the file through its filehandle 🌈
• the server MUST NOT delete the directory entry if the reply from OPEN had the flag OPEN4_RESULT_PRESERVE_UNLINKED set 💜

the server MAY have its own restrictions on removing a file while it's open 🔒 it might disallow such a REMOVE (or a removal that happens as part of RENAME). the things that affect those restrictions include:

• whether certain access protocols (not just NFS) are holding the file open 🧐
• whether particular options, access modes, or policies on the server are enabled ⚙️

if a file has an outstanding OPEN and that stops the removal of the file's directory entry, you'll get NFS4ERR_FILE_OPEN 🚫

sometimes the determination above can't be made definitively, because delegations are being held. in that case they MUST be recalled so the REMOVE can keep processing 🧶 when a delegation is held, the server has no reliable knowledge of the status of OPENs for that client. so unless there are files opened with the particular deny modes by clients without delegations, the call can't be decided until delegations are recalled, and the operation can't go ahead until each sufficient delegation has been returned or revoked, so the server can make a correct call 💅

whenever delegations get recalled, the server will probably return one or more NFS4ERR_DELAY errors while those delegations are still outstanding 🕰️

if the current filehandle designates a directory that another client holds a directory delegation for, then, unless the situation can be resolved by sending a notification, the directory delegation MUST be recalled, and the operation MUST NOT proceed until the delegation is returned or revoked. unless this happens super quick, one or more NFS4ERR_DELAY errors will come back to requests made while the delegation is still outstanding 😤

when the current filehandle designates a directory with one or more directory delegations, then, if those delegations asked for these notifications, NOTIFY4_REMOVE_ENTRY gets generated as a result of this operation 📬

one thing to note: when a removal happens as a result of a RENAME, NOTIFY4_REMOVE_ENTRY is only generated if the removal happens as a separate operation. if the removal is integrated and atomic with RENAME, the notification of the removal is folded into the notification for the RENAME. check out the discussion of the NOTIFY4_RENAME_ENTRY notification in Section 20.4 💖

18.26. Operation 29: RENAME - Rename Directory Entry 🦋

18.26.1. ARGUMENTS

struct RENAME4args {
        /* SAVED_FH: source directory */
        component4      oldname;
        /* CURRENT_FH: target directory */
        component4      newname;
};

18.26.2. RESULTS

struct RENAME4resok {
        change_info4    source_cinfo;
        change_info4    target_cinfo;
};
union RENAME4res switch (nfsstat4 status) {
 case NFS4_OK:
        RENAME4resok    resok4;
 default:
        void;
};

18.26.3. DESCRIPTION

the RENAME operation renames the object called oldname, in the source directory the saved filehandle points to (as set by the SAVEFH operation), to newname in the target directory the current filehandle points to 🌷 the operation is required to be atomic to the client. source and target directories MUST reside on the same file system on the server. on success, the current filehandle will still be the target directory 🫧

if the target directory already has an entry named newname, the source object MUST be compatible with the target: either both are non-directories, or both are directories and the target MUST be empty 🧐 if they're compatible, the existing target gets removed before the rename happens, or, even better, the target gets removed atomically as part of the rename. see Section 18.25.4 for what the client and server do whenever a target gets removed. but note that when the removal is done atomically with the rename, certain parts of the removal described there get folded into the rename. for example, the notification of the removal won't be a NOTIFY4_REMOVE_ENTRY. instead it shows up as part of the NOTIFY4_ADD_ENTRY or NOTIFY4_RENAME_ENTRY generated by the rename 📮

if the source object and the target aren't compatible, or if the target is a directory that isn't empty, the server returns NFS4ERR_EXIST 🙅‍♀️

if oldname and newname both refer to the same file (like if they're hard links of each other), then, unless the file is open (see Section 18.26.4), RENAME MUST perform no action and return NFS4_OK 🫶

for both directories involved in the RENAME, the server returns change_info4 info. with the atomic field of change_info4, the server tells you whether the before and after change attributes were grabbed atomically with respect to the rename 👀

if oldname refers to a named attribute and the saved and current filehandles refer to different file system objects, the server returns NFS4ERR_XDEV, just like it would if the saved and current filehandles were directories on different file systems 🚧

if oldname or newname has a length of zero, or if oldname or newname doesn't follow the UTF-8 definition, you'll get NFS4ERR_INVAL 💀

18.26.4. IMPLEMENTATION ✨

okay so the server is allowed (it MAY!) to put limits on RENAME 🙈 It can say no to a rename if the file being renamed is open, or if that open came from particular protocols, or with particular options or access modes. Similar limits can apply when a file already exists with the target name and that file is open. When RENAME gets turned down for any of these reasons, the error NFS4ERR_FILE_OPEN comes back. 💅

Now here's a fun one. If oldname and rename point to the same file, and that file is open in a way that would get RENAME rejected with NFS4ERR_FILE_OPEN if they were different files, then RENAME SHOULD be rejected with NFS4ERR_FILE_OPEN too. 🫶

okay this part is the most dramatic, bestie 😭 If the server does have restrictions like that, and they include NFSv4 opens that stop a rename from happening, then the server needs to recall any delegations that could be hiding the opens that matter for the decision. Why? Because when a client holds a delegation, the server might not have an accurate count of that client's opens, since the client can run OPENs and CLOSEs locally. 🦋 The RENAME operation only needs to be delayed until a definitive result can be obtained. For example, if there are multiple delegations and one of them sets up an open that would block the rename (given the server's semantics), NFS4ERR_FILE_OPEN may be returned to the caller as soon as that delegation comes back, without waiting for the other delegations to be returned. And if those opens aren't tied to delegations at all, NFS4ERR_FILE_OPEN can be returned right away with no delegation recall being done. 🎀

If the current filehandle or the saved filehandle designates a directory that another client holds a directory delegation for, then, unless the situation can be resolved by sending a notification, the delegation MUST be recalled. The operation can't proceed until that delegation is returned or revoked. Unless this happens super quickly, one or more NFS4ERR_DELAY errors will come back to requests made while the delegation is still outstanding. 💀

When the current and saved filehandles are the same and they designate a directory with one or more directory delegations, then, if those delegations ask for such notifications, a notification of type NOTIFY4_RENAME_ENTRY gets generated as a result of this operation. When oldname and rename refer to the same file, no notification is generated (because, as Section 18.26.3 states, the server MUST take no action). 🌸 When a file is removed because it has the same name as the target, and that removal is done atomically with the rename, a NOTIFY4_REMOVE_ENTRY notification will not be generated. Instead, the deletion of the file gets reported as part of the NOTIFY4_RENAME_ENTRY notification. ✨

When the current and saved filehandles are not the same:

• If the current filehandle designates a directory with one or more directory delegations, then, if those delegations ask for such notifications, NOTIFY4_ADD_ENTRY will be generated as a result of this operation. When a file is removed because it has the same name as the target, and that removal is done atomically with the rename, a NOTIFY4_REMOVE_ENTRY notification will not be generated. Instead, the deletion of the file gets reported as part of the NOTIFY4_ADD_ENTRY notification. 🍓
• If the saved filehandle designates a directory with one or more directory delegations, then, if those delegations ask for such notifications, NOTIFY4_REMOVE_ENTRY will be generated as a result of this operation. 🧚

If the object being renamed has file delegations held by clients other than the one doing the RENAME, the delegations MUST be recalled. The operation can't proceed until each one is returned or revoked. 🌈 Note that for multiply linked files, the delegation recall requirement applies even if the delegation was obtained through a different name than the one being renamed. In every case where delegations get recalled, the server is likely to return one or more NFS4ERR_DELAY errors while the delegation(s) stay outstanding, although it might not do that if the delegations come back quickly. 👑

The RENAME operation must be atomic to the client. ✨ The statement "source and target directories MUST reside on the same file system on the server" means that the fsid fields in the attributes for the directories are the same. If they live on different file systems, the error NFS4ERR_XDEV is returned. 🙅‍♀️

Based on the value of the fh_expire_type attribute for the object, the filehandle may or may not expire on a RENAME. 🦋 That said, server implementors are strongly encouraged to try to keep filehandles from expiring in this fashion. 🫶

On some servers, the file names . and .. are illegal as either oldname or newname, and they'll result in the error NFS4ERR_BADNAME. 🙈 On top of that, many servers check whether oldname or newname is an alias for the source directory. Those servers return the error NFS4ERR_INVAL in those cases. 💅

If either the source or target filehandles are not directories, the server will return NFS4ERR_NOTDIR. 💖

---

18.27. Operation 31: RESTOREFH - Restore Saved Filehandle ✨

18.27.1. ARGUMENTS

   /* SAVED_FH: */
   void;

18.27.2. RESULTS

   struct RESTOREFH4res {
           /*
            * If status is NFS4_OK,
            *     new CURRENT_FH: value of saved fh
            */
           nfsstat4        status;
   };

18.27.3. DESCRIPTION

okay so RESTOREFH sets the current filehandle and stateid back to whatever was saved in the saved filehandle and stateid 🧚 If there's no saved filehandle, the server returns the error NFS4ERR_NOFILEHANDLE.

See Section 16.2.3.1.1 for more details on the current filehandle.

See Section 16.2.3.1.2 for more details on the current stateid.

18.27.4. IMPLEMENTATION

Operations like OPEN and LOOKUP use the current filehandle to represent a directory and swap it out for a new filehandle. 🌸 If the previous filehandle was saved with a SAVEFH operator, then the previous filehandle can be restored as the current filehandle. This is commonly used to get post-operation attributes for the directory, e.g.,

         PUTFH (directory filehandle)
         SAVEFH
         GETATTR attrbits     (pre-op dir attrs)
         CREATE optbits "foo" attrs
         GETATTR attrbits     (file attributes)
         RESTOREFH
         GETATTR attrbits     (post-op dir attrs)

---

18.28. Operation 32: SAVEFH - Save Current Filehandle 🎀

18.28.1. ARGUMENTS

   /* CURRENT_FH: */
   void;

18.28.2. RESULTS

   struct SAVEFH4res {
           /*
            * If status is NFS4_OK,
            *    new SAVED_FH: value of current fh
            */
           nfsstat4        status;
   };

18.28.3. DESCRIPTION

The SAVEFH operation saves the current filehandle and stateid. 💅 If a previous filehandle was saved, it's no longer accessible, so it's gone bye-bye. 😭 The saved filehandle can be restored as the current filehandle with the RESTOREFH operator.

On success, the current filehandle keeps its value. ✨

See Section 16.2.3.1.1 for more details on the current filehandle.

See Section 16.2.3.1.2 for more details on the current stateid.

18.28.4. IMPLEMENTATION

---

18.29. Operation 33: SECINFO - Obtain Available Security 🔐

18.29.1. ARGUMENTS

   struct SECINFO4args {
           /* CURRENT_FH: directory */
           component4      name;
   };

18.29.2. RESULTS

   /*
    * From RFC 2203
    */
   enum rpc_gss_svc_t {
           RPC_GSS_SVC_NONE        = 1,
           RPC_GSS_SVC_INTEGRITY   = 2,
           RPC_GSS_SVC_PRIVACY     = 3
   };

   struct rpcsec_gss_info {
           sec_oid4        oid;
           qop4            qop;
           rpc_gss_svc_t   service;
   };

   /* RPCSEC_GSS has a value of '6' - See RFC 2203 */
   union secinfo4 switch (uint32_t flavor) {
    case RPCSEC_GSS:
            rpcsec_gss_info        flavor_info;
    default:
            void;
   };

   typedef secinfo4 SECINFO4resok<>;

   union SECINFO4res switch (nfsstat4 status) {
    case NFS4_OK:
           /* CURRENTFH: consumed */
            SECINFO4resok resok4;
    default:
            void;
   };

18.29.3. DESCRIPTION

okay so SECINFO is how the client asks, "which security flavors can I use for this directory filehandle and file name pair?" 🦋 SECINFO should use the same access methodology as LOOKUP when it checks the name. So if the requester doesn't have the right access to LOOKUP the name, then SECINFO MUST act the same way and return NFS4ERR_ACCESS. 🙈

The result is an array of the security mechanisms the server has available, in the server's order of preference, with the most preferred one first. ✨ The client can pick whatever mechanism it both wants and supports, or it can go down the server's preference list and take the first one it supports. 💖 Each array entry is a secinfo4 structure. The field 'flavor' will hold a value of AUTH_NONE, AUTH_SYS (as defined in RFC 5531 [3]), or RPCSEC_GSS (as defined in RFC 2203 [4]). The field flavor can also be any other security flavor registered with IANA. 🌈

For the flavors AUTH_NONE and AUTH_SYS, no extra security information is returned. 🍓 The same goes for lots of (if not most) other security flavors, including AUTH_DH. 🌸 For a return value of RPCSEC_GSS, a security triple comes back with the mechanism object identifier (OID, as defined in RFC 2743 [7]), the quality of protection (as defined in RFC 2743 [7]), and the service type (as defined in RFC 2203 [4]). 💜 It's possible for SECINFO to return multiple entries with flavor equal to RPCSEC_GSS, each with different security triple values. 🧚

On success, the current filehandle is consumed (see Section 2.6.3.1.1.8). If the next operation after SECINFO tries to use the current filehandle, that operation will fail with the status NFS4ERR_NOFILEHANDLE. 💀

If the name has a length of zero, or if the name doesn't follow the UTF-8 definition (assuming UTF-8 capabilities are enabled; see Section 14.4), the error NFS4ERR_INVAL gets returned. 🙅‍♀️

See Section 2.6 for additional information on the use of SECINFO. 🫶

18.29.4. IMPLEMENTATION ✨

okay bestie, so SECINFO is the operation the NFS client reaches for when some other NFS operation comes back with NFS4ERR_WRONGSEC 🙈. that error is basically the server going "nope, my security policy is different from what you're using rn" 💅. so the client has to grab a list of possible security flavors and pick the one that best fits its policies 🫶.

the server's security policies decide when a request gets NFS4ERR_WRONGSEC 🔥. check out Table 8 for the list of operations that can return it. also, when READDIR returns attributes, the rdattr_error (Section 5.8.1.12) can contain NFS4ERR_WRONGSEC 👀. now for the drama 😭: CREATE and REMOVE MUST NOT return NFS4ERR_WRONGSEC. the logic for CREATE is that unless the target name already exists, it can't have its own separate security policy. the parent's policy was already checked when its filehandle was injected into the COMPOUND request's operations stream 💖. for similar reasons, an OPEN operation that creates the target MUST NOT return NFS4ERR_WRONGSEC. if the target name does exist, it might have a separate policy, but that doesn't matter because CREATE MUST return NFS4ERR_EXIST 🙅‍♀️.

for REMOVE, the target might have its own policy, but it's about to be deleted, so the parent's policy wins out 👑. RENAME and LINK MAY return NFS4ERR_WRONGSEC, but that error only applies to the saved filehandle (see Section 2.6.3.1.2) 🦋. any NFS4ERR_WRONGSEC error on the current filehandle used by LINK and RENAME MUST be returned by the PUTFH, PUTPUBFH, PUTROOTFH, or RESTOREFH operation that injected the current filehandle ✨.

except for LINK and RENAME, the operations that can return NFS4ERR_WRONGSEC are the spots where the client can inject a filehandle into the "current filehandle" on the server 🧚. that filehandle either comes from the client (PUTFH, PUTPUBFH, PUTROOTFH), gets generated by a name-to-filehandle translation (LOOKUP and OPEN), or gets generated from the saved filehandle via RESTOREFH 🌸. as Section 2.6.3.1.1.1 says, a put filehandle operation followed by SAVEFH MUST NOT return NFS4ERR_WRONGSEC 🙈. so RESTOREFH is allowed to return NFS4ERR_WRONGSEC under certain conditions (see Section 2.6.3.1.1), which is how security policies get honored 🔐.

READDIR won't directly return NFS4ERR_WRONGSEC 🫣. but if the READDIR request asked for attributes, the READDIR request's security triple might not match that of a directory entry. when that happens and the client asked for the rdattr_error attribute, the server returns NFS4ERR_WRONGSEC in rdattr_error for that entry 💜.

to fix an NFS4ERR_WRONGSEC error, the client does the following 💅:

• for LOOKUP and OPEN, the client uses SECINFO with the same current filehandle and name that was in the original LOOKUP or OPEN, so it can enumerate the available security triples 🍓.
• for rdattr_error, the client uses SECINFO with the same current filehandle that was in the original READDIR. the name passed to SECINFO is the directory entry's name (as returned from READDIR) that had NFS4ERR_WRONGSEC in its rdattr_error attribute 🌈.
• for PUTFH, PUTROOTFH, PUTPUBFH, RESTOREFH, LINK, and RENAME, the client uses SECINFO_NO_NAME { style = SECINFO_STYLE4_CURRENT_FH } 🎀. the client prefixes the SECINFO_NO_NAME operation with the appropriate PUTFH, PUTPUBFH, or PUTROOTFH operation that provides the filehandle originally provided by the PUTFH, PUTPUBFH, PUTROOTFH, or RESTOREFH operation.

NOTE: in NFSv4.0, the client was required to use SECINFO and had to reconstruct the parent of the original filehandle and the component name of the original filehandle. the introduction of SECINFO_NO_NAME in NFSv4.1 means no reconstruction is needed 🙌.

• for LOOKUPP, the client uses SECINFO_NO_NAME { style = SECINFO_STYLE4_PARENT } and provides the filehandle that equals the filehandle originally provided to LOOKUPP 🧁.

for the recommendations on which security flavor SECINFO and SECINFO_NO_NAME should use, see Section 21 💖.

---

18.30. Operation 34: SETATTR - Set Attributes 🎀

18.30.1. ARGUMENTS

   struct SETATTR4args {
           /* CURRENT_FH: target object */
           stateid4        stateid;
           fattr4          obj_attributes;
   };

18.30.2. RESULTS

   struct SETATTR4res {
           nfsstat4        status;
           bitmap4         attrsset;
   };

18.30.3. DESCRIPTION

SETATTR changes one or more attributes of a file system object ✨. the new attributes come with a bitmap, and the attributes that follow the bitmap are listed in bit order 🦋.

the stateid argument for SETATTR provides the byte-range locking context needed for SETATTR requests that set the size attribute 🔒. setting the size attribute changes the file's data, so it has the same locking requirements as a matching WRITE 💅. any SETATTR that sets the size attribute is incompatible with a share reservation that specifies OPEN4_SHARE_DENY_WRITE 🙅‍♀️. the area between the old end-of-file and the new end-of-file counts as modified, just like it would if that area had been the target of a WRITE. this matters when checking for conflicts with byte-range locks, in cases where a server implements mandatory byte-range locking behavior 🔥. a valid stateid SHOULD always be specified 💜. when the file size attribute is not set, the special stateid made of all bits equal to zero MAY be passed 🌸.

on either success or failure, the server returns the attrsset bitmask to show what attributes (if any) were successfully set 🍓. the attrsset in the response is a subset of the attrmask field of the obj_attributes field in the argument 🧚. on success, the current filehandle keeps its value 🫶.

18.30.4. IMPLEMENTATION

okay so if the request asks to set the owner attribute, the server SHOULD let the operation succeed when the current owner of the object matches the value in the request 🙌. some servers might prohibit setting the owner attribute unless the requester has privilege to do so 🔐. if the server is lenient about matching owner values in this one case, the client implementation can be simpler when it creates an object (for example, an exclusive create via OPEN) and then does a SETATTR 🧁.

the file size attribute is how you request changes to a file's size 📏. a value of zero truncates the file. a value less than the current size discards the data from the new size to the end of the file. a size greater than the current size adds logically zeroed data bytes to the end of the file 🌈. servers are free to do this with unallocated bytes (holes) or with allocated data bytes set to zero 🤷‍♀️. clients should not make any assumptions about how a server implements this, beyond the bytes in the affected byte-range returned by READ being zeroed 🙈. servers MUST support extending the file size via SETATTR 💅.

SETATTR is not guaranteed to be atomic 😬. a failed SETATTR might partially change a file's attributes, which is why the reply always includes the status and the list of attributes that were set 💖.

if the object whose attributes are changing has a file delegation held by a client other than the one doing the SETATTR, the delegation(s) must be recalled. the operation can't change an attribute until each such delegation is returned or revoked 🔁. whenever delegations get recalled, the server is likely to return one or more NFS4ERR_DELAY errors while the delegation(s) stay outstanding, although it might not do that if the delegations come back quickly ⏳.

if the object whose attributes are being set is a directory, and another client holds a directory delegation for that directory, then (if enabled) asynchronous notifications get generated when the set of attributes changed has a non-null intersection with the set of attributes that notification was requested for 🔔. NOTIFY4_CHANGE_DIR_ATTRS notifications go to the appropriate client(s), but the SETATTR isn't delayed while waiting for those notifications to be sent 💜.

if the object whose attributes are being set is a member of a directory that another client holds a directory delegation for, then asynchronous notifications get generated when the set of attributes changed has a non-null intersection with the set of attributes that notification was requested for 🦋. NOTIFY4_CHANGE_CHILD_ATTRS notifications go to the appropriate clients, but the SETATTR still isn't delayed waiting on them 🌸.

changing a file's size with SETATTR indirectly changes the time_modify and change attributes ⏰. a client must account for this, since size changes can result in data deletion 🫣.

time_access_set and time_modify_set are write-only attributes built as a switched union, so the client can tell the server how to set the time values 🧚. if the switched union specifies SET_TO_CLIENT_TIME4, the client has provided an nfstime4 to use for the operation 🕰️. if the switch union doesn't specify SET_TO_CLIENT_TIME4, the server uses its current time for the SETATTR operation ✨.

if server and client times differ, programs that compare client time to file times can break 💀. a time synchronization protocol should be used to limit client/server time skew 🎀.

a COMPOUND containing a VERIFY operation that specifies only the change attribute, immediately followed by a SETATTR, gives the client a way to ask for something that emulates the SETATTR guard mechanism of NFSv3 🫶. the guard mechanism exists to avoid changing file attributes based on stale information. so delays between checking the guard condition and setting the attributes can compromise that, and so can the matching delay in the NFSv4 emulation 😬. therefore, NFSv4.1 servers SHOULD take care to avoid such delays, as far as possible, when executing such a request 🍓.

if the server doesn't support an attribute the client requested, the server SHOULD return NFS4ERR_ATTRNOTSUPP 🙅‍♀️.

SETATTR returns a mask of the attributes it actually set, in all cases 🌈. that mask MUST NOT include attribute bits the client didn't ask to be set 🚫. if the attribute masks in the request and reply are equal, the status field in the reply MUST be NFS4_OK 👑.

---

18.31. Operation 37: VERIFY - Verify Same Attributes 🧚

18.31.1. ARGUMENTS

   struct VERIFY4args {
           /* CURRENT_FH: object */
           fattr4          obj_attributes;
   };

18.31.2. RESULTS

   struct VERIFY4res {
           nfsstat4        status;
   };

18.31.3. DESCRIPTION 💅

Okay so VERIFY is the op where the client basically goes "wait, are these attributes still what I think they are?" before the rest of the COMPOUND request keeps going 🧚 If any of the attributes don't match, the error NFS4ERR_NOT_SAME must be returned. ✨ The current filehandle keeps its value after a successful VERIFY 🎀

18.31.4. IMPLEMENTATION 🫶

One possible use of VERIFY is this little series of operations 🦋 The client is checking that the file being removed is actually the file it expects to be removed, which helps stop an accidental deletion 🙈

         PUTFH (directory filehandle)
         LOOKUP (file name)
         VERIFY (filehandle == fh)
         PUTFH (directory filehandle)
         REMOVE (file name)

ngl this doesn't stop a second client from removing the file and making a brand new one in the middle of the sequence 💀 But it does help avoid the unintended result, so it's still giving protection 💜

If a RECOMMENDED attribute is specified in the VERIFY operation and the server doesn't support that attribute for the file system object, the error NFS4ERR_ATTRNOTSUPP is returned to the client 🌸

When the attribute rdattr_error or any set-only attribute (e.g., time_modify_set) is specified, the error NFS4ERR_INVAL is returned to the client 🚫✨

---

18.32. Operation 38: WRITE - Write to File 🌈

18.32.1. ARGUMENTS 🍓

   enum stable_how4 {
           UNSTABLE4       = 0,
           DATA_SYNC4      = 1,
           FILE_SYNC4      = 2
   };

   struct WRITE4args {
           /* CURRENT_FH: file */
           stateid4        stateid;
           offset4         offset;
           stable_how4     stable;
           opaque          data<>;
   };

18.32.2. RESULTS 👑

   struct WRITE4resok {
           count4          count;
           stable_how4     committed;
           verifier4       writeverf;
   };

   union WRITE4res switch (nfsstat4 status) {
    case NFS4_OK:
            WRITE4resok    resok4;
    default:
            void;
   };

18.32.3. DESCRIPTION 🔥

WRITE is the op for writing data to a regular file 📝 The target file is whatever the current filehandle points to. The offset says where the data should go, and an offset of zero means the write starts at the very beginning of the file 🌟 The count is encoded as part of the opaque data parameter, and it's the number of bytes that are supposed to be written. If the count is zero, the WRITE succeeds and returns a count of zero, as long as the permissions check passes 🧁 The server MAY write fewer bytes than the client asked for 🫧

The client uses the stable parameter to tell the server how the data should be processed. If stable is FILE_SYNC4, the server MUST commit the written data plus all file system metadata to stable storage before it returns results. This matches the NFSv2 protocol semantics, and any other behavior is a protocol violation, so no vibes-based storage allowed 🙅‍♀️ If stable is DATA_SYNC4, the server MUST commit all of the data to stable storage, plus enough of the metadata to retrieve that data, before returning 💾 The server implementor is free to do DATA_SYNC4 the same way as FILE_SYNC4, just with a possible performance drop 🐢 If stable is UNSTABLE4, the server is free to commit any part of the data and metadata to stable storage, including all of it or none of it, before replying to the client 🌊 There's no guarantee about whether or when uncommitted data gets committed later. The only promises the server makes are that it won't destroy any data without changing the value of writeverf, and that it won't commit the data and metadata at a lower level than the client asked for 🫶

Except when special stateids are used, the stateid for a WRITE request is a value that came back from an earlier byte-range LOCK or OPEN request, or it's the stateid tied to a delegation 🔑 The stateid identifies the associated owners (if there are any), and the server uses it to check that the associated locks are still valid, e.g., they haven't been revoked 👀

If the WRITE succeeds, these results come back. The count result is the number of bytes actually written to the file. The server may write fewer bytes than requested. If it does, the actual number of bytes written, starting at location offset, is what gets returned 💅

The server also tells the client how committed the data and metadata are, using the committed field. Per Table 11:

o The server MAY commit the data at a stronger level than requested.

o The server MUST commit the data at a level at least as high as that committed.

Here are the valid combos of the stable field in the request and the committed field in the reply 🎀

            +------------+-----------------------------------+
            | stable     | committed                         |
            +------------+-----------------------------------+
            | UNSTABLE4  | FILE_SYNC4, DATA_SYNC4, UNSTABLE4 |
            | DATA_SYNC4 | FILE_SYNC4, DATA_SYNC4            |
            | FILE_SYNC4 | FILE_SYNC4                        |
            +------------+-----------------------------------+

                                 Table 11

The last part of the result is writeverf 🧚 It's the write verifier, a cookie the client uses to tell whether the server's instance state has changed (say, the server restarted) between a WRITE and a later WRITE or COMMIT. This cookie MUST stay unchanged during a single instance of the NFSv4.1 server, and it MUST be unique between instances of the NFSv4.1 server 🌸 If the cookie changes, the client MUST assume that any data written with an UNSTABLE4 committed value and an old writeverf in the reply has been lost and needs to be recovered 😭

If a client writes data with stable set to UNSTABLE4 and the reply comes back with committed set to DATA_SYNC4 or UNSTABLE4, the client will follow up later with a COMMIT operation to sync the outstanding asynchronous data and metadata with the server's stable storage, barring client error ✨ It's possible that, because of a client crash or some other error, a later COMMIT never reaches the server 💀

For a WRITE with a stateid where all bits are zero, the server MAY let the WRITE go through subject to mandatory byte-range locks or the current share deny modes for the file 🔓 For a WRITE with a stateid where all bits are equal to 1, the server MUST NOT let the WRITE bypass locking checks at the server. Otherwise it's treated as if a stateid of all bits equal to zero were used 🙅‍♀️

On success, the current filehandle keeps its value 🦋

18.32.4. IMPLEMENTATION 💅

okay so it's totally possible for the server to write fewer bytes than the client asked for 🫠 in that case the server SHOULD NOT return an error unless it wrote literally zero bytes. if it writes less than the number specified, the client is gonna need to send another WRITE for the rest 📝

it's assumed that writing data to a file updates the time_modified and change attributes of that file 👀 but these attributes SHOULD NOT be changed unless the actual contents changed. so a WRITE with count set to zero SHOULD NOT update time_modified or change. no fake updates, bestie 💅

Stable storage is persistent storage that survives:

1. Repeated power failures.
2. Hardware failures (of any board, power supply, etc.).
3. Repeated software crashes and restarts.

This definition does not address failure of the stable storage module itself. 🫶

the verifier is defined so a client can tell apart different instances of an NFSv4.1 protocol server that might've lost cached, uncommitted data 🔍 most of the time that means the client can spot server restarts. this info is required so the client can safely figure out whether the server could've lost cached data. ngl this is the whole point 😭 if the server fails unexpectedly and the client has uncommitted data from earlier WRITE requests (made with the stable argument set to UNSTABLE4, and the committed result also came back as UNSTABLE4), the server might not have flushed that cached data to stable storage. the burden of recovery is on the client, so the client will need to retransmit the data to the server 🔁

a suggested verifier would be the time the server was last started, but only if restarting the server results in lost buffers ⏰

the reply's committed field lets the client do way more effective caching 🧠 if the server is committing every WRITE request to stable storage, then it SHOULD return with committed set to FILE_SYNC4, no matter what the stable field said in the arguments. a server that uses an NVRAM accelerator may choose to do this ⚡ and the client can use it to make its cache work harder by tossing out cached data that's already been committed on the server 🗑️✨

some implementations might return NFS4ERR_NOSPC instead of NFS4ERR_DQUOT when a user's quota is exceeded 🙈

in the case that the current filehandle is of type NF4DIR, the server will return NFS4ERR_ISDIR 📁 if the current file is a symbolic link, the error NFS4ERR_SYMLINK gets returned 🔗 otherwise, if the current filehandle doesn't designate an ordinary file, the server will return NFS4ERR_WRONG_TYPE ❌

locking drama incoming 🎭

if mandatory byte-range locking is in effect for the file, and the matching byte-range of the data being written is locked as READ_LT or WRITE_LT by an owner that isn't associated with the stateid, the server MUST return NFS4ERR_LOCKED. if that happens, the client MUST check whether the owner tied to the stateid used with the WRITE operation has a conflicting READ_LT lock overlapping the byte-range that was supposed to be written 👀 if the stateid's owner has no conflicting READ_LT lock, then the client SHOULD try to get the right write byte-range lock via the LOCK operation before it re-attempts the WRITE 🔐 and when the WRITE completes, the client SHOULD release that byte-range lock via LOCKU 🔓

if the stateid's owner had a conflicting READ_LT lock, then the client has no choice but to return an error to the application that attempted the WRITE 💀 the reason is that since the stateid's owner had a READ_LT lock, either the server tried to temporarily upgrade that READ_LT lock to a WRITE_LT lock, or the server has no upgrade capability at all 🙃 if the server tried to upgrade the READ_LT lock and failed, it's pointless for the client to re-attempt the upgrade with the LOCK operation, because another client might also be trying to upgrade 🥴 if two clients are blocked trying to upgrade the same lock, the clients deadlock 😵‍💫 and if the server has no upgrade capability, then trying a LOCK operation to upgrade is pointless too 🫠

if one or more other clients have delegations for the file being written, those delegations MUST be recalled, and the operation can't proceed until those delegations are returned or revoked 📜 except where this happens super quickly, one or more NFS4ERR_DELAY errors will be returned to requests made while the delegation is still outstanding ⏳ normally, delegations won't get recalled because of a WRITE operation, since the recall would happen because of an earlier OPEN 🗝️ but since it's possible for a WRITE to be done with a special stateid, the server needs to check for this case even though the client should have done an OPEN previously 🧐

---

18.33. Operation 40: BACKCHANNEL_CTL - Backchannel Control ✨

18.33.1. ARGUMENT

typedef opaque gsshandle4_t<>;

struct gss_cb_handles4 {
        rpc_gss_svc_t           gcbp_service; /* RFC 2203 */
        gsshandle4_t            gcbp_handle_from_server;
        gsshandle4_t            gcbp_handle_from_client;
};

union callback_sec_parms4 switch (uint32_t cb_secflavor) {
case AUTH_NONE:
        void;
case AUTH_SYS:
        authsys_parms   cbsp_sys_cred; /* RFC 1831 */
case RPCSEC_GSS:
        gss_cb_handles4 cbsp_gss_handles;
};

struct BACKCHANNEL_CTL4args {
        uint32_t                bca_cb_program;
        callback_sec_parms4     bca_sec_parms<>;
};

18.33.2. RESULT

struct BACKCHANNEL_CTL4res {
        nfsstat4                bcr_status;
};

18.33.3. DESCRIPTION

okay so BACKCHANNEL_CTL is the op that swaps out the backchannel's callback program number 🔄 and it adds RPCSEC_GSS handles for the backchannel to use (it doesn't replace them, so no yeeting your old ones 🙅‍♀️)

the arguments of BACKCHANNEL_CTL are basically a subset of the CREATE_SESSION parameters 🎀 in BACKCHANNEL_CTL, the bca_cb_program and bca_sec_parms fields line up with csa_cb_program and csa_sec_parms in the arguments of CREATE_SESSION (Section 18.36) 💜

BACKCHANNEL_CTL MUST appear in a COMPOUND that starts with SEQUENCE 🧚

if the RPCSEC_GSS handle identified by gcbp_handle_from_server doesn't exist on the server, the server MUST return NFS4ERR_NOENT 💀

and if an RPCSEC_GSS handle is using the SSV context (see Section 2.10.9), then, because every SSV RPCSEC_GSS handle shares one common SSV GSS context, there are security considerations specific to this situation, discussed in Section 2.10.10 🔐 these are serious, so go read them 🫶

---

18.34. Operation 41: BIND_CONN_TO_SESSION - Associate Connection with Session 🦋

18.34.1. ARGUMENT

enum channel_dir_from_client4 {
 CDFC4_FORE             = 0x1,
 CDFC4_BACK             = 0x2,
 CDFC4_FORE_OR_BOTH     = 0x3,
 CDFC4_BACK_OR_BOTH     = 0x7
};

struct BIND_CONN_TO_SESSION4args {
 sessionid4     bctsa_sessid;

 channel_dir_from_client4
                bctsa_dir;

 bool           bctsa_use_conn_in_rdma_mode;
};

18.34.2. RESULT

enum channel_dir_from_server4 {
 CDFS4_FORE     = 0x1,
 CDFS4_BACK     = 0x2,
 CDFS4_BOTH     = 0x3
};

struct BIND_CONN_TO_SESSION4resok {
 sessionid4     bctsr_sessid;

 channel_dir_from_server4
                bctsr_dir;

 bool           bctsr_use_conn_in_rdma_mode;
};

union BIND_CONN_TO_SESSION4res
 switch (nfsstat4 bctsr_status) {

 case NFS4_OK:
  BIND_CONN_TO_SESSION4resok
                bctsr_resok4;

 default:       void;
};

18.34.3. DESCRIPTION

BIND_CONN_TO_SESSION is how you hook extra connections up to a session 🧵 it MUST be used on the connection being associated with the session, and it MUST be the only operation in the COMPOUND procedure 🙌 if SP4_NONE (Section 18.35) state protection is in use, any principal, security flavor, or RPCSEC_GSS context MAY be used to invoke the operation 🌈 if SP4_MACH_CRED is used, RPCSEC_GSS MUST be used with the integrity or privacy services, using the principal that created the client ID 🔑 and if SP4_SSV is used, RPCSEC_GSS with the SSV GSS mechanism (Section 2.10.9) and integrity or privacy MUST be used 🛡️

if the client picked SP4_NONE state protection when the client ID was created, the client isn't required to use BIND_CONN_TO_SESSION to associate a connection with the session, unless it wants to associate that connection with the backchannel 🫧 with SP4_NONE protection, just sending a COMPOUND request with a SEQUENCE operation is enough to associate the connection with the session named in SEQUENCE ✨

the field bctsa_dir tells the server whether the client wants the connection associated with the fore channel, the backchannel, or both 🧭 the value CDFC4_FORE_OR_BOTH means the client wants both channels but will happily accept just the fore channel 🍓 the value CDFC4_BACK_OR_BOTH means the client wants both channels but will accept just the backchannel 🍓 the server replies in bctsr_dir with which channel(s) the connection ended up on 📬 if the client specified CDFC4_FORE, the server MUST return CDFS4_FORE 💅 if the client specified CDFC4_BACK, the server MUST return CDFS4_BACK 💅 if the client specified CDFC4_FORE_OR_BOTH, the server MUST return CDFS4_FORE or CDFS4_BOTH 💅 and if the client specified CDFC4_BACK_OR_BOTH, the server MUST return CDFS4_BACK or CDFS4_BOTH 💅

to figure out bctsa_use_conn_in_rdma_mode you'll want to check the description of the csa_use_conn_in_rdma_mode argument in the CREATE_SESSION operation (Section 18.36), and for bctsr_use_conn_in_rdma_mode check the description of csr_use_conn_in_rdma_mode in that same section 📖🫶

invoking BIND_CONN_TO_SESSION on a connection that's already associated with the specified session has no effect, and the server MUST respond with NFS4_OK 🥰 the exception is if the client is demanding changes to the set of channels the connection is associated with, and in that case the server MUST return NFS4ERR_INVAL 🙅‍♀️

18.34.4. IMPLEMENTATION 🔥

okay bestie, picture this: a session's channel loses all of its connections 😭. Whether the client needs BIND_CONN_TO_SESSION to plug in a new one depends on the client ID's state protection and what type of channel it is. And if the server restarted and didn't keep the reply cache in stable storage, it won't recognize the session ID at all 💀. At that point the client has to call EXCHANGE_ID to make a brand new client ID and session. ✨

Now the spicy part 🌶️. Suppose SP4_SSV state protection is in use, and BIND_CONN_TO_SESSION is in the spo_must_enforce set when the client ID was created (Section 18.35). Then things get messy if SET_SSV gets sent, no response comes back, and the last connection on that client ID drops. Per the sessions model, the client MUST retry the SET_SSV. But it needs a new connection to do that, and it MUST associate that connection with the session through a BIND_CONN_TO_SESSION authenticated with the SSV GSS mechanism. The problem is that RPCSEC_GSS message integrity codes use a subkey derived from the SSV as the key, and the SSV may have changed already 😩. There are several ways to recover, but here's one general strategy:

• The client reconnects.
• The client assumes that the SET_SSV was executed, and so sends BIND_CONN_TO_SESSION with the subkey (derived from the new SSV, i.e., what SET_SSV would have set the SSV to) used as the key for the RPCSEC_GSS credential message integrity codes.
• If the request succeeds, this means that the original attempted SET_SSV did execute successfully. The client re-sends the original SET_SSV, which the server will reply to via the reply cache.
• If the server returns an RPC authentication error, this means that the server's current SSV was not changed (and the SET_SSV was likely not executed). The client then tries BIND_CONN_TO_SESSION with the subkey derived from the old SSV as the key for the RPCSEC_GSS message integrity codes.
• The attempted BIND_CONN_TO_SESSION with the old SSV should succeed. If so, the client re-sends the original SET_SSV. If the original SET_SSV was not executed, then the server executes it. If the original SET_SSV was executed but failed, the server will return the SET_SSV from the reply cache.

---

18.35. Operation 42: EXCHANGE_ID - Instantiate Client ID 🦋

The EXCHANGE_ID operation swaps long-hand client and server identifiers (owners), and creates a client ID. 🫶

18.35.1. ARGUMENT

   const EXCHGID4_FLAG_SUPP_MOVED_REFER    = 0x00000001;
   const EXCHGID4_FLAG_SUPP_MOVED_MIGR     = 0x00000002;

   const EXCHGID4_FLAG_BIND_PRINC_STATEID  = 0x00000100;

   const EXCHGID4_FLAG_USE_NON_PNFS        = 0x00010000;
   const EXCHGID4_FLAG_USE_PNFS_MDS        = 0x00020000;
   const EXCHGID4_FLAG_USE_PNFS_DS         = 0x00040000;

   const EXCHGID4_FLAG_MASK_PNFS           = 0x00070000;

   const EXCHGID4_FLAG_UPD_CONFIRMED_REC_A = 0x40000000;
   const EXCHGID4_FLAG_CONFIRMED_R         = 0x80000000;
   struct state_protect_ops4 {
           bitmap4 spo_must_enforce;
           bitmap4 spo_must_allow;
   };

   struct ssv_sp_parms4 {
           state_protect_ops4      ssp_ops;
           sec_oid4                ssp_hash_algs<>;
           sec_oid4                ssp_encr_algs<>;
           uint32_t                ssp_window;
           uint32_t                ssp_num_gss_handles;
   };

   enum state_protect_how4 {
           SP4_NONE = 0,
           SP4_MACH_CRED = 1,
           SP4_SSV = 2
   };

   union state_protect4_a switch(state_protect_how4 spa_how) {
           case SP4_NONE:
                   void;
           case SP4_MACH_CRED:
                   state_protect_ops4      spa_mach_ops;
           case SP4_SSV:
                   ssv_sp_parms4           spa_ssv_parms;
   };

   struct EXCHANGE_ID4args {
           client_owner4           eia_clientowner;
           uint32_t                eia_flags;
           state_protect4_a        eia_state_protect;
           nfs_impl_id4            eia_client_impl_id<1>;
   };

18.35.2. RESULT

   struct ssv_prot_info4 {
    state_protect_ops4     spi_ops;
    uint32_t               spi_hash_alg;
    uint32_t               spi_encr_alg;
    uint32_t               spi_ssv_len;
    uint32_t               spi_window;
    gsshandle4_t           spi_handles<>;
   };
   union state_protect4_r switch(state_protect_how4 spr_how) {
    case SP4_NONE:
            void;
    case SP4_MACH_CRED:
            state_protect_ops4     spr_mach_ops;
    case SP4_SSV:
            ssv_prot_info4         spr_ssv_info;
   };

   struct EXCHANGE_ID4resok {
    clientid4        eir_clientid;
    sequenceid4      eir_sequenceid;
    uint32_t         eir_flags;
    state_protect4_r eir_state_protect;
    server_owner4    eir_server_owner;
    opaque           eir_server_scope<NFS4_OPAQUE_LIMIT>;
    nfs_impl_id4     eir_server_impl_id<1>;
   };

   union EXCHANGE_ID4res switch (nfsstat4 eir_status) {
   case NFS4_OK:
    EXCHANGE_ID4resok      eir_resok4;

   default:
    void;
   };

18.35.3. DESCRIPTION 💜

The client uses EXCHANGE_ID to register a particular client owner with the server. The client ID it returns is needed for any request that creates state on the server, and it acts as the parent object for sessions the client creates. To confirm the client ID, it must first be used, along with the returned eir_sequenceid, as arguments to CREATE_SESSION. If the flag EXCHGID4_FLAG_CONFIRMED_R is set in the result, eir_flags, then eir_sequenceid MUST be ignored, as it has no relevancy. 🙈

EXCHANGE_ID MAY be sent in a COMPOUND procedure that starts with SEQUENCE. However, when a client talks to a server for the first time, it won't have a session yet, so SEQUENCE won't be possible. If EXCHANGE_ID is sent without a preceding SEQUENCE, then it MUST be the only operation in the COMPOUND procedure's request. If it isn't, the server MUST return NFS4ERR_NOT_ONLY_OP. 🚫

The eia_clientowner field is made up of a co_verifier field and a co_ownerid string. As noted in Section 2.4, the co_ownerid describes the client, and the co_verifier is the incarnation of the client. An EXCHANGE_ID sent with a new incarnation of the client will lead to the server removing lock state of the old incarnation. Whereas an EXCHANGE_ID sent with the current incarnation and co_ownerid will result in an error or an update of the client ID's properties, depending on the arguments to EXCHANGE_ID. 🔄

A server MUST NOT use the same client ID for two different incarnations of an eir_clientowner. 🛑

Along with the client ID and sequence ID, the server returns a server owner (eir_server_owner) and server scope (eir_server_scope). The first one is used for network trunking, as described in Section 2.10.5. The second one lets clients figure out when client IDs from one server can be recognized by another after file system migration (see Section 11.7.7). 🌍

The client ID returned by EXCHANGE_ID is only unique relative to the combination of eir_server_owner.so_major_id and eir_server_scope. So if two servers hand back the same client ID, it's on the client to tell them apart using eir_server_owner.so_major_id and eir_server_scope. If two different servers claim matching server_owner.so_major_id and eir_server_scope, the client can use the verification techniques from Section 2.10.5 to check whether the servers are actually distinct. If they are, the client needs to note the destination network addresses of the connections used with each server, and use the network address as the final discriminator. 🕵️‍♀️

The server, identified by the so_major_id of the server owner together with the server scope, has to track several properties of each client ID it hands out. These properties apply to the client ID and every session tied to it, and they come from the arguments and results of EXCHANGE_ID. The client ID properties include:

• The capabilities expressed by the following bits, which come from the results of EXCHANGE_ID:
  • EXCHGID4_FLAG_SUPP_MOVED_REFER
  • EXCHGID4_FLAG_SUPP_MOVED_MIGR
  • EXCHGID4_FLAG_BIND_PRINC_STATEID
  • EXCHGID4_FLAG_USE_NON_PNFS
  • EXCHGID4_FLAG_USE_PNFS_MDS
  • EXCHGID4_FLAG_USE_PNFS_DS

These properties may be updated by subsequent EXCHANGE_ID requests on confirmed client IDs, though the server MAY refuse to change them. 🧐

• The state protection method used, one of SP4_NONE, SP4_MACH_CRED, or SP4_SSV, as set by the spa_how field of the arguments to EXCHANGE_ID. Once the client ID is confirmed, this property cannot be updated by subsequent EXCHANGE_ID requests. 🔐
• For SP4_MACH_CRED or SP4_SSV state protection:
  • The list of operations (spo_must_enforce) that MUST use the specified state protection. This list comes from the results of EXCHANGE_ID.
  • The list of operations (spo_must_allow) that MAY use the specified state protection. This list comes from the results of EXCHANGE_ID.

Once the client ID is confirmed, these properties cannot be updated by subsequent EXCHANGE_ID requests.

• For SP4_SSV protection:
  • The OID of the hash algorithm. This property is represented by one of the algorithms in the ssp_hash_algs field of the EXCHANGE_ID arguments. Once the client ID is confirmed, this property cannot be updated by subsequent EXCHANGE_ID requests.
  • The OID of the encryption algorithm. This property is represented by one of the algorithms in the ssp_encr_algs field of the EXCHANGE_ID arguments. Once the client ID is confirmed, this property cannot be updated by subsequent EXCHANGE_ID requests.
  • The length of the SSV. This property is represented by the spi_ssv_len field in the EXCHANGE_ID results. Once the client ID is confirmed, this property cannot be updated by subsequent EXCHANGE_ID requests.

There are REQUIRED and RECOMMENDED relationships among the length of the key of the encryption algorithm ("key length"), the length of the output of the hash algorithm ("hash length"), and the length of the SSV ("SSV length"). 📐

• key length MUST be <= hash length. This is because the keys used for the encryption algorithm are actually subkeys derived from the SSV, and the derivation is via the hash algorithm. Choosing an encryption algorithm with a key length longer than the hash algorithm's output would require padding, and that would weaken the encryption algorithm.
• hash length SHOULD be <= SSV length. This is because the SSV is a key used to derive subkeys via an HMAC, and it is recommended that the key used as input to an HMAC be at least as long as the length of the HMAC's hash algorithm's output (see Section 3 of RFC2104 [11]).
• key length SHOULD be <= SSV length. This is a transitive result of the above two invariants.
• key length SHOULD be >= hash length / 2. This is because the subkey derivation is via an HMAC, and it is recommended that if the HMAC has to be truncated, it should not be truncated to less than half the hash length (see Section 4 of RFC2104 [11]).

Okay so the property list keeps going, and this next bit is where the SSV and the flags get really interesting 🦋✨

• Number of concurrent versions of the SSV the client and server will support (Section 2.10.9) 🎀. This property is represented by spi_window in the EXCHANGE_ID results 💅. The property may be updated by subsequent EXCHANGE_ID requests.

• The client's implementation ID, as represented by the eia_client_impl_id field of the arguments 🌸. The property may be updated by subsequent EXCHANGE_ID requests.
• The server's implementation ID, as represented by the eir_server_impl_id field of the reply 🧚. The property may be updated by replies to subsequent EXCHANGE_ID requests.

The eia_flags in the arguments and the eir_flags in the results let the client and server tell each other what they're capable of, plus how the client ID is going to be used 🫶. Setting or clearing a bit in the arguments' flags does not force the server to set or clear that same bit in the results 💀 (no pressure, server 🙈). Bits that aren't defined above cannot be set in the eia_flags field. If they are, the server MUST reject the operation with NFS4ERR_INVAL 😭.

The EXCHGID4_FLAG_UPD_CONFIRMED_REC_A bit can only be set in eia_flags, and it's always off in eir_flags 🙅‍♀️. The EXCHGID4_FLAG_CONFIRMED_R bit can only be set in eir_flags, and it's always off in eia_flags 💜. If the server recognizes the co_ownerid and co_verifier as mapping to a confirmed client ID, it sets EXCHGID4_FLAG_CONFIRMED_R in eir_flags. That flag is how a client can tell whether the client ID it's trying to create already exists and is confirmed 👑.

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is set in eia_flags, the client is trying to update the properties of an existing confirmed client ID 🔥. (If it wants to update an unconfirmed client ID instead, it MUST NOT set EXCHGID4_FLAG_UPD_CONFIRMED_REC_A.) In that case it is RECOMMENDED that the client send the update EXCHANGE_ID operation in the same COMPOUND as a SEQUENCE, so the EXCHANGE_ID gets executed exactly once ⭐. Whether the client can update the properties of the client ID depends on two things: the state protection it picked when the client ID was created, and the principal and security flavor it uses when sending the EXCHANGE_ID request 🌈. The situations described in items 6, 7, 8, or 9 of the second numbered list of Section 18.35.4 will apply. And note that if the operation succeeds and returns a client ID that is already confirmed, the server MUST set the EXCHGID4_FLAG_CONFIRMED_R bit in eir_flags 🫶.

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is not set in eia_flags, then the client is either establishing a brand-new client ID, trying to trunk data communication to the server (Section 2.10.5), or updating the properties of an unconfirmed client ID 🦋. The situations described in items 1, 2, 3, 4, or 5 of the second numbered list of Section 18.35.4 will apply. Note that if the operation succeeds and returns a client ID that was previously confirmed, the server MUST set the EXCHGID4_FLAG_CONFIRMED_R bit in eir_flags 💖.

When the EXCHGID4_FLAG_SUPP_MOVED_REFER flag bit is set, the client is saying it can handle an NFS4ERR_MOVED error as part of a referral sequence 🧚. When that bit isn't set, it's still legal for the server to do a referral sequence 😌. Hot take though: a server may use the fact that the client can't correctly respond to a referral to avoid referring that particular client 👀. For instance, it may act as a proxy for that file system, at some cost in performance, although it is not obligated to do so 🍓. If the server will potentially perform a referral, it MUST set EXCHGID4_FLAG_SUPP_MOVED_REFER in eir_flags ✨.

When EXCHGID4_FLAG_SUPP_MOVED_MIGR is set, the client is saying it can handle an NFS4ERR_MOVED error as part of a file system migration sequence 🌸. If that bit isn't set, it's still legal for the server to indicate that a file system has moved, when that actually happens 🙈. A server may also use the fact that the client can't handle a migration when it schedules which file systems to migrate, so it avoids migrating ones that are actively being used 🔥. It may also hide actual migrations from clients that can't deal with them, by acting as a proxy for a migrated file system for particular clients, at some cost in performance, although it is not obligated to do so 💅. If the server will potentially perform a migration, it MUST set EXCHGID4_FLAG_SUPP_MOVED_MIGR in eir_flags 🎀.

When EXCHGID4_FLAG_BIND_PRINC_STATEID is set, the client is asking the server to bind the stateid to the principal 🔐. That means when a principal creates a stateid, it has to be the one that uses the stateid 💜. If the server will perform binding, it returns EXCHGID4_FLAG_BIND_PRINC_STATEID. The server MAY return EXCHGID4_FLAG_BIND_PRINC_STATEID even if the client didn't request it 🌈. If an update to the client ID changes the value of the EXCHGID4_FLAG_BIND_PRINC_STATEID client ID property, the effect only applies to new stateids ⭐. Existing stateids (and all stateids with the same "other" field) that were created with stateid to principal binding in force will keep binding in force. Existing stateids (and all stateids with the same "other" field) that were created with stateid to principal binding not in force will keep binding not in force 🦋.

The EXCHGID4_FLAG_USE_NON_PNFS, EXCHGID4_FLAG_USE_PNFS_MDS, and EXCHGID4_FLAG_USE_PNFS_DS bits are described in Section 13.1, and they convey the roles the client ID is meant for in a pNFS environment 🌟. The server MUST set one of the acceptable combinations of these bits (roles) in eir_flags, as specified in Section 13.1 💅. Note that the same client owner/server owner pair can have multiple roles 👑. Multiple roles can be associated with the same client ID or with different client IDs 🧚. So if a client sends EXCHANGE_ID from the same client owner to the same server owner multiple times, but specifies different pNFS roles each time, the server might return different client IDs 🙃. Since different pNFS roles might have different client IDs, the client may ask for different properties for each role/client ID 🫶.

The spa_how field of the eia_state_protect field specifies how the client wants to protect its client, locking, and session states from unauthorized changes (Section 2.10.8.3) 🔐:

• SP4_NONE. The client does not request the NFSv4.1 server to enforce state protection. The NFSv4.1 server MUST NOT enforce state protection for the returned client ID 🙅‍♀️.
• SP4_MACH_CRED. If spa_how is SP4_MACH_CRED, then the client MUST send the EXCHANGE_ID request with RPCSEC_GSS as the security flavor, and with a service of RPC_GSS_SVC_INTEGRITY or RPC_GSS_SVC_PRIVACY. If SP4_MACH_CRED is specified, the client wants to use an RPCSEC_GSS-based machine credential to protect its state 🛡️. The server MUST note the principal the EXCHANGE_ID operation was sent with, and the GSS mechanism used. These notes collectively comprise the machine credential 🎀.

After the client ID is confirmed, as long as the lease associated with the client ID is unexpired, a subsequent EXCHANGE_ID operation that uses the same eia_clientowner.co_owner as the first EXCHANGE_ID MUST also use the same machine credential as the first EXCHANGE_ID 💖. The server returns the same client ID for the subsequent EXCHANGE_ID as the one returned from the first EXCHANGE_ID 🦋.

• SP4_SSV. If spa_how is SP4_SSV, then the client MUST send the EXCHANGE_ID request with RPCSEC_GSS as the security flavor, and with a service of RPC_GSS_SVC_INTEGRITY or RPC_GSS_SVC_PRIVACY. If SP4_SSV is specified, the client wants to use the SSV to protect its state ✨. The server records the credential used in the request as the machine credential (as defined above) for the eia_clientowner.co_owner. The CREATE_SESSION operation that confirms the client ID MUST use the same machine credential 💅.

When a client specifies SP4_MACH_CRED or SP4_SSV, it also provides two lists of operations, each expressed as a bitmap 🌈. The first list is spo_must_enforce, and it consists of those operations the client MUST send (subject to the server confirming the list of operations in the result of EXCHANGE_ID) with the machine credential (if SP4_MACH_CRED protection is specified) or the SSV-based credential (if SP4_SSV protection is used) 🔥. The client MUST send these operations with RPCSEC_GSS credentials that specify the RPC_GSS_SVC_INTEGRITY or RPC_GSS_SVC_PRIVACY security service 🫶. Typically, the first list includes EXCHANGE_ID, CREATE_SESSION, DELEGPURGE, DESTROY_SESSION, BIND_CONN_TO_SESSION, and DESTROY_CLIENTID 🧚. The client SHOULD NOT specify in this list any operations that require a filehandle, because the server's access policies MAY conflict with the client's choice, and then the client would be unable to access a subset of the server's namespace 🙈.

Heads up, bestie: if SP4_SSV protection is specified and the client indicates that CREATE_SESSION must be protected with SP4_SSV, the SSV can't exist without a confirmed client ID. So the first CREATE_SESSION MUST instead be sent using the machine credential, and the server MUST accept the machine credential 🚨.

There's a matching result, also called spo_must_enforce, listing the operations for which the server will require SP4_MACH_CRED or SP4_SSV protection 👑. Normally the server's result equals the client's argument, but the result MAY be different 💜. If the client requests one or more operations in the set { EXCHANGE_ID, CREATE_SESSION, DELEGPURGE, DESTROY_SESSION, BIND_CONN_TO_SESSION, DESTROY_CLIENTID }, then the result spo_must_enforce MUST include the operations the client requested from that set 🎀.

If spo_must_enforce in the results has BIND_CONN_TO_SESSION set, then connection binding enforcement is enabled, and the client MUST use the machine credential (if SP4_MACH_CRED protection is used) or the SSV credential (if SP4_SSV protection is used) on calls to BIND_CONN_TO_SESSION 🔐.

The second list is spo_must_allow, and it consists of those operations the client wants the option of sending with the machine credential or the SSV-based credential, even if the object the operations are performed on isn't owned by the machine or SSV credential 🦋. The corresponding result, also called spo_must_allow, consists of the operations the server will allow the client to use SP4_SSV or SP4_MACH_CRED credentials with. Normally the server's result equals the client's argument, but the result MAY be different ✨.

The whole point of spo_must_allow is to solve this little conundrum 😩. Say the client ID is confirmed with EXCHGID4_FLAG_BIND_PRINC_STATEID, and it calls OPEN with the RPCSEC_GSS credentials of a normal user 🌸. Now the user's credentials expire and can't be renewed (e.g., a Kerberos ticket granting ticket expires, and the user has logged off and won't be getting a new ticket granting ticket) 💀. The client won't be able to send CLOSE without the user's credentials. That means the client has to either leave the state on the server, or re-send EXCHANGE_ID with a new verifier to clear all state 😭. Unless, of course, the client includes CLOSE on the spo_must_allow list and the server agrees 🙌.

The SP4_SSV protection parameters also have:

ssp_hash_algs:

Okay bestie, so this is the list of hash algorithms the client offers for the internal SSV GSS mechanism and for SET_SSV 💅 Each one gets named with an OID (object identifier) 🔥 For a server, the REQUIRED ones are id-sha1, id-sha224, id-sha256, id-sha384, and id-sha512 [28]. The server picks one and tells you which with spi_hash_alg, a field inside spr_ssv_prot_info ✨ That field is an index into ssp_hash_algs, so it's basically pointing at an item on your list 🦋 If the server doesn't support any of the algorithms you offered, it returns NFS4ERR_HASH_ALG_UNSUPP 💀 And if ssp_hash_algs is empty, the server MUST return NFS4ERR_INVAL. No exceptions 🙈

ssp_encr_algs:

This is the set of encryption algorithms the client offers to give privacy protection to the internal SSV GSS mechanism 🎀 Each one is also an OID. The REQUIRED algorithm for a server is id-aes256-CBC 👑 The RECOMMENDED ones are id-aes192-CBC and id-aes128-CBC [29] ✨ The one the server picks comes back in spi_encr_alg, which is an index into ssp_encr_algs. If the server doesn't support any of the offered algorithms, it returns NFS4ERR_ENCR_ALG_UNSUPP 😭 If ssp_encr_algs is empty, the server MUST return NFS4ERR_INVAL. Now the tea: because of the earlier rules about how key length and hash length relate, some pairings of RECOMMENDED and REQUIRED encryption algorithms with hash algorithms either SHOULD NOT or MUST NOT be used 🙅‍♀️ Table 12 has the full rundown of the illegal and discouraged combos, so check it before you pair anything 💅

+-------------------+----------------------+------------------------+
| Encryption        | MUST NOT be combined | SHOULD NOT be combined |
| Algorithm         | with                 | with                   |
+-------------------+----------------------+------------------------+
| id-aes128-CBC     |                      | id-sha384, id-sha512   |
| id-aes192-CBC     | id-sha1              | id-sha512              |
| id-aes256-CBC     | id-sha1, id-sha224   |                        |
+-------------------+----------------------+------------------------+

                             Table 12

ssp_window:

This is how many SSV versions the client wants the server to keep around 🦋 (every successful call to SET_SSV makes a brand new version of the SSV). If ssp_window is zero, the server MUST return NFS4ERR_INVAL. The server answers with spi_window, which MUST NOT exceed ssp_window, and it MUST be at least one 🌸 Any request on the backchannel or the fore channel that uses an SSV version outside that window fails with an ONC RPC authentication error 😭 and the requester has to retry it with the same slot ID and sequence ID. Rough, but that's the deal.

ssp_num_gss_handles:

This is how many RPCSEC_GSS handles the server should make that are based on the GSS SSV mechanism (Section 2.10.9) ✨ It is NOT the total number of RPCSEC_GSS handles for the client ID, bestie. Later calls to EXCHANGE_ID will add more handles 🧚 The server sends back a list of handles in spi_handles. If the client asks for at least one handle and the server can't create it, the server MUST return an error 💀 The handles in spi_handles aren't usable until the client ID is confirmed. That could happen right away if EXCHANGE_ID returns EXCHGID4_FLAG_CONFIRMED_R, or after successful confirmation from CREATE_SESSION 🫶

A client ID can span all the connections that are connected to a server sharing the same eir_server_owner.so_major_id, but the RPCSEC_GSS handles in spi_handles can only be used on connections to a server that returns the same eir_server_owner.so_major_id and eir_server_owner.so_minor_id on each connection 👑 It's totally allowed for the client to set ssp_num_gss_handles to zero, since the client can make more handles with another EXCHANGE_ID call 🍓

Because every SSV RPCSEC_GSS handle shares a common SSV GSS context, there are security considerations specific to this situation, which are discussed in Section 2.10.10. Take them seriously, main character energy 🥹

The seq_window (see Section 5.2.3.1 of RFC2203 [4]) of each RPCSEC_GSS handle in spi_handle MUST be the same as the seq_window of the RPCSEC_GSS handle used for the credential of the RPC request that the EXCHANGE_ID request was sent with 🔒

The arguments include an array of up to one element called eia_client_impl_id. If eia_client_impl_id is present, it has the information that identifies the client's implementation 🌈 In the same way, the results include an array of up to one element called eir_server_impl_id, which identifies the server's implementation 🎀 Servers MUST accept a zero-length eia_client_impl_id array, and clients MUST accept a zero-length eir_server_impl_id array ✨

Picture diagnostic software that pulls this info to figure out interoperability problems, performance workload behaviors, or general usage statistics 🔍 That's the whole vibe. Since this information is only for planning or general diagnosis, the client and server MUST NOT interpret this implementation identity information in a way that affects interoperational behavior of the implementation 🙅‍♀️ Why? Because if they did, they might use fewer protocol capabilities than the peer can actually support, or they might refuse to interoperate entirely 😭

Some implementations will ignore that rule and read the identity info anyway, so implementations MUST allow the users of the NFSv4 client and server to set the contents of the sent nfs_impl_id structure to any value 💖 Let the users choose, periodt 💅

18.35.4. IMPLEMENTATION ✨

okay so, the server keeps a client record for every client, and each one is a 5-tuple 💅 Here's what's in it:

1. co_ownerid: the client identifier string, pulled from the eia_clientowner structure of the EXCHANGE_ID4args structure 🫶
2. co_verifier: a client-specific value used to show incarnations (so a client restart counts as a brand new incarnation) 🔄 It also comes from the eia_clientowner structure of EXCHANGE_ID4args.
3. principal: the principal that was defined in the RPC header's credential and/or verifier when the client record was made 🔐
4. client ID: the shorthand client identifier. The server generates it and returns it in the eir_clientid field of the EXCHANGE_ID4resok structure 🎀
5. confirmed: a private field on the server that says whether a client record has been confirmed or not 👀 A record is confirmed once a CREATE_SESSION operation succeeds in confirming it. If not, it's unconfirmed. Unconfirmed records come from an EXCHANGE_ID call, and any unconfirmed record that isn't confirmed within a lease period SHOULD be removed 🧹

Next up, these identifiers stand for special values in the records 🌸

• ownerid_arg: the value of the eia_clientowner.co_ownerid subfield of the EXCHANGE_ID4args structure in the current request.
• verifier_arg: the value of the eia_clientowner.co_verifier subfield of the EXCHANGE_ID4args structure in the current request.
• old_verifier_arg: a value of the eia_clientowner.co_verifier field from a client record that came in a previous request. This is NOT the same as verifier_arg 🙅‍♀️
• principal_arg: the value of the RPCSEC_GSS principal for the current request.
• old_principal_arg: a value of the principal from a client record, as defined by the RPC header's credential or verifier in a previous request. This is distinct from principal_arg.
• clientid_ret: the value of the eir_clientid field the server will return in the EXCHANGE_ID4resok structure for the current request.
• old_clientid_ret: the value of the eir_clientid field the server returned in the EXCHANGE_ID4resok structure for a previous request. This is distinct from clientid_ret.
• confirmed: the client ID has been confirmed.
• unconfirmed: the client ID has not been confirmed.

Okay, real talk: EXCHANGE_ID is non-idempotent, so we have to think about retries happening because of a client restart, a network partition, a malfunctioning router, etc. 😭 Retries are spotted by the value of the eia_clientowner field of EXCHANGE_ID4args, and the scenarios below explain how to handle each one.

The scenarios are all described in terms of the client record(s) a server has for a given co_ownerid. Also, if the client ID was created with SP4_SSV state protection and EXCHANGE_ID is listed in spo_must_allow, then the server MUST authorize EXCHANGE_IDs with the SSV principal as well as the principal that created the client ID 🔥

1. New Owner ID

If the server has no client records whose eia_clientowner.co_ownerid matches ownerid_arg, and EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is not set in the EXCHANGE_ID, then a new shorthand client ID (let's call it clientid_ret) gets generated, and this unconfirmed record gets added to the server's state ✨

{ ownerid_arg, verifier_arg, principal_arg, clientid_ret,
unconfirmed }

After that, the server returns clientid_ret. 🎀

2. Non-Update on Existing Client ID

If the server has this confirmed record, and the request does NOT have EXCHGID4_FLAG_UPD_CONFIRMED_REC_A set, then the request is a retry from a faulty router or a lost connection, or the client is just checking whether it can do trunking 🧐

{ ownerid_arg, verifier_arg, principal_arg, clientid_ret,
confirmed }

Since the record is confirmed, the client must have gotten the server's reply to the first EXCHANGE_ID request. And since the server has a confirmed record and EXCHGID4_FLAG_UPD_CONFIRMED_REC_A isn't set, the server returns the same result it gave when the client ID's properties were last updated (or, if they were never updated, the result from when the client ID was created). The only possible exception is eir_server_owner.so_minor_id. The confirmed record stays unchanged 💖

3. Client Collision

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is not set, and the server has this confirmed record, then this request is probably a chance collision between the eia_clientowner.co_ownerid values of two different clients 💀

{ ownerid_arg, *, old_principal_arg, old_clientid_ret,
confirmed }

If there's no state tied to old_clientid_ret, or there is state but its lease has expired, then this case is basically the same as the New Owner ID case in Paragraph 1. The confirmed record gets deleted, old_clientid_ret and its lock state get deleted, a new shorthand client ID gets generated, and this unconfirmed record gets added to the server's state:

{ ownerid_arg, verifier_arg, principal_arg, clientid_ret,
unconfirmed }

After that, the server returns clientid_ret. 🌈

If old_clientid_ret has an unexpired lease with state, then none of the state for old_clientid_ret gets changed or deleted. The server returns NFS4ERR_CLID_INUSE to tell the client it should retry with a different value for the eia_clientowner.co_ownerid subfield of EXCHANGE_ID4args. The client record doesn't change 🙅‍♀️

4. Replacement of Unconfirmed Record

If the EXCHGID4_FLAG_UPD_CONFIRMED_REC_A flag is not set, and the server has this unconfirmed record, then the client is trying EXCHANGE_ID again on an unconfirmed client ID. This could be a retry, a client restart before the client ID gets confirmed (meaning before CREATE_SESSION was called), or some other reason 🔁

{ ownerid_arg, *, *, old_clientid_ret, unconfirmed }

The properties of old_clientid_ret might be different from what's in the current EXCHANGE_ID. Whether or not the properties are being updated, the server deletes the unconfirmed record to clear up any ambiguity, generates a new client ID (clientid_ret), and sets up this unconfirmed record:

{ ownerid_arg, verifier_arg, principal_arg, clientid_ret,
unconfirmed }

5. Client Restart

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is not set, and the server has this confirmed client record, then this request is probably from a previously confirmed client that restarted 🔄

{ ownerid_arg, old_verifier_arg, principal_arg,
old_clientid_ret, confirmed }

The previous incarnation of the same client won't be sending requests anymore, so once the new client ID is confirmed by CREATE_SESSION, byte-range locks and share reservations should be released right away 🧹 The new incarnation shouldn't have to wait for the lease on the previous incarnation to expire. Session state should also be removed, because if the client had kept that info across the restart, this request wouldn't have been sent. If the server supports neither the CLAIM_DELEGATE_PREV nor the CLAIM_DELEG_PREV_FH claim types, the associated delegations should be purged too. Otherwise, the delegations are kept, and recovery goes ahead according to Section 10.2.1 🙌

After processing, clientid_ret is returned to the client, and this client record is added:

{ ownerid_arg, verifier_arg, principal_arg, clientid_ret,
unconfirmed }

The earlier confirmed record still exists, so the same ownerid_arg is in both a confirmed and an unconfirmed state at the same time 👀 The number of states can collapse down to one once the server gets an applicable CREATE_SESSION or EXCHANGE_ID.

• If the server later gets a successful CREATE_SESSION that confirms clientid_ret, then the server atomically destroys the confirmed record and makes the unconfirmed record confirmed, as described in Section 18.36.4 💅
• If the server instead later gets an EXCHANGE_ID with the client owner equal to ownerid_arg, one strategy is to just delete the unconfirmed record and process the EXCHANGE_ID as described in the whole of Section 18.35.4 🧚

6. Update

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is set, and the server has this confirmed record, then this request is an attempt at an update ✨

{ ownerid_arg, verifier_arg, principal_arg, clientid_ret,
confirmed }

Since the record is confirmed, the client must have gotten the server's reply to the first EXCHANGE_ID request. The server allows the update, and the client record is left intact 🫶

7. Update but No Confirmed Record

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is set, and the server has no confirmed record for ownerid_arg, then the server returns NFS4ERR_NOENT and leaves any unconfirmed record intact 🙈

8. Update but Wrong Verifier

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is set, and the server has this confirmed record, then this request is an illegal attempt at an update. Maybe it's a retry from a previous client incarnation 🚫

{ ownerid_arg, old_verifier_arg, *, clientid_ret, confirmed }

The server returns NFS4ERR_NOT_SAME and leaves the client record intact.

9. Update but Wrong Principal

If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is set, and the server has this confirmed record, then this request is an illegal attempt at an update by a principal that isn't authorized 🛑

{ ownerid_arg, verifier_arg, old_principal_arg, clientid_ret,
confirmed }

The server returns NFS4ERR_PERM and leaves the client record intact.

---

18.36. Operation 43: CREATE_SESSION - Create New Session and Confirm Client ID 🧚

18.36.1. ARGUMENT ✨

okay so this is the whole packet the client sends when it wants a shiny new session 💖 the channel attributes, the flag bits, and the args all live together in here 🎀

   struct channel_attrs4 {
           count4                  ca_headerpadsize;
           count4                  ca_maxrequestsize;
           count4                  ca_maxresponsesize;
           count4                  ca_maxresponsesize_cached;
           count4                  ca_maxoperations;
           count4                  ca_maxrequests;
           uint32_t                ca_rdma_ird<1>;
   };

   const CREATE_SESSION4_FLAG_PERSIST              = 0x00000001;
   const CREATE_SESSION4_FLAG_CONN_BACK_CHAN       = 0x00000002;
   const CREATE_SESSION4_FLAG_CONN_RDMA            = 0x00000004;

   struct CREATE_SESSION4args {
           clientid4               csa_clientid;
           sequenceid4             csa_sequence;

           uint32_t                csa_flags;

           channel_attrs4          csa_fore_chan_attrs;
           channel_attrs4          csa_back_chan_attrs;

           uint32_t                csa_cb_program;
           callback_sec_parms4     csa_sec_parms<>;
   };

18.36.2. RESULT 🎀

and here's what the server sends back 🧚 it's basically the session ID, the sequence echo, the flags it agreed to, and the channel attrs it actually picked 💅

   struct CREATE_SESSION4resok {
           sessionid4              csr_sessionid;
           sequenceid4             csr_sequence;

           uint32_t                csr_flags;

           channel_attrs4          csr_fore_chan_attrs;
           channel_attrs4          csr_back_chan_attrs;
   };
   union CREATE_SESSION4res switch (nfsstat4 csr_status) {
   case NFS4_OK:
           CREATE_SESSION4resok    csr_resok4;
   default:
           void;
   };

18.36.3. DESCRIPTION 🦋

so basically, the client uses CREATE_SESSION to make brand new session objects on the server 🌸 it's giving main character energy ✨

CREATE_SESSION can be sent with or without a preceding SEQUENCE operation in the same COMPOUND procedure. If it comes after a SEQUENCE, the session it creates has no direct relation to the session named in that SEQUENCE, even though the two might belong to the same client ID 🙈 If CREATE_SESSION is sent without a preceding SEQUENCE, then it MUST be the only operation in the COMPOUND procedure's request. If it isn't, the server MUST return NFS4ERR_NOT_ONLY_OP 💀

Besides making a session, CREATE_SESSION does a couple of extra things too 👑

• the first session created with a new client ID is what confirms that client's state on the server 🔥 the server hands back the parameter values for that new session
• the connection CREATE_SESSION was sent over gets associated with the session's fore channel 🦋

Here's how the arguments and results work, bestie 🫶

csa_clientid: this is the client ID the new session will be tied to 💖 the matching result is csr_sessionid, which is the new session's ID.

csa_sequence: every client ID serializes CREATE_SESSION through a per-client ID sequence number (see Section 18.36.4) ✨ the result, csr_sequence, MUST be equal to csa_sequence. no cap, it has to match 💅

In the next three arguments, the client offers a value that the session is supposed to have. Except where stated otherwise, it is RECOMMENDED that the server accept the value. If it's not acceptable, the server MAY use a different value 🎀 either way, the server MUST return the value the session will actually use, which is either what the client offered or what the server is insisting on 🌈

csa_flags: this field holds a list of flag bits. here they are, iykyk 🍓

CREATE_SESSION4_FLAG_PERSIST: if this bit is set, the client wants the server to provide a persistent reply cache. For sessions where only idempotent operations will be used (a read-only session, for example), clients SHOULD NOT set CREATE_SESSION4_FLAG_PERSIST 🙈 if the server does not or cannot provide a persistent reply cache, the server MUST NOT set CREATE_SESSION4_FLAG_PERSIST in the field csr_flags.

If the server is a pNFS metadata server, then for the reasons described in Section 12.5.2 it SHOULD support CREATE_SESSION4_FLAG_PERSIST if it supports the layout_hint (Section 5.12.4) attribute 🧚

CREATE_SESSION4_FLAG_CONN_BACK_CHAN: if this bit is set in csa_flags, the client is asking that the connection CREATE_SESSION arrived on also get associated with the session's backchannel, in addition to its fore channel 💜 if the server agrees, it sets CREATE_SESSION4_FLAG_CONN_BACK_CHAN in the result field csr_flags. If CREATE_SESSION4_FLAG_CONN_BACK_CHAN isn't set in csa_flags, then CREATE_SESSION4_FLAG_CONN_BACK_CHAN MUST NOT be set in csr_flags 🌸

CREATE_SESSION4_FLAG_CONN_RDMA: if this bit is set in csa_flags, and the connection CREATE_SESSION arrived on is currently in non-RDMA mode but is capable of running in RDMA mode, then the client is asking the server to "step up" to RDMA mode on that connection ⭐ if the server agrees, it sets CREATE_SESSION4_FLAG_CONN_RDMA in csr_flags. If CREATE_SESSION4_FLAG_CONN_RDMA isn't set in csa_flags, then CREATE_SESSION4_FLAG_CONN_RDMA MUST NOT be set in csr_flags. Note that once the server agrees to step up, it and the client MUST exchange all future traffic on the connection with RPC RDMA framing and not Record Marking ([8]) 😭

csa_fore_chan_attrs, csa_fore_chan_attrs: these two fields are the attributes for the fore channel (which carries requests from the client to the server) and the backchannel (which carries callback requests from the server to the client), in that order 🦋 the results come back in matching structures called csr_fore_chan_attrs and csr_back_chan_attrs. The results set the attributes for each channel, and they apply to every later use of that channel in the session 💅 each structure has these fields:

ca_headerpadsize: this is the maximum amount of padding the requester is willing to apply so write payloads line up on some boundary at the replier. For each channel, the server

• will reply in ca_headerpadsize with its preferred value, or zero if padding is not in use, and
• MAY decrease this value but MUST NOT increase it.

ca_maxrequestsize: this is the maximum size of a COMPOUND or CB_COMPOUND request that will be sent. It represents the XDR encoded size of the request, including the RPC headers (security flavor credentials and verifiers included), but it excludes any RPC transport framing headers 🎀 picture a request coming over a non-RDMA TCP/IP connection with a single Record Marking header in front of it. The maximum count encoded in that header will be ca_maxrequestsize. If a requester sends a request that goes over ca_maxrequestsize, the error NFS4ERR_REQ_TOO_BIG will be returned, as described in Section 2.10.6.4 😭 for each channel, the server MAY decrease this value but MUST NOT increase it.

ca_maxresponsesize: this is the maximum size of a COMPOUND or CB_COMPOUND reply the requester will accept from the replier, including RPC headers (see the ca_maxrequestsize definition). For each channel, the server MAY decrease this value, but MUST NOT increase it. however, if the client picks a value for ca_maxresponsesize so small that a replier on a channel could never send a response, the server SHOULD return NFS4ERR_TOOSMALL in the CREATE_SESSION reply 💀 After the session is created, if a requester sends a request whose reply would be bigger than this value, the replier will return NFS4ERR_REP_TOO_BIG, per the description in Section 2.10.6.4.

ca_maxresponsesize_cached: this one is like ca_maxresponsesize, but it's the maximum size of a reply that will be stored in the reply cache (Section 2.10.6.1) 🧚 For each channel, the server MAY decrease this value, but MUST NOT increase it. If, in the CREATE_SESSION reply, a channel's ca_maxresponsesize_cached is less than its ca_maxresponsesize, that's a clue for the requester that it needs to be selective about which replies it tells the replier to cache. For example, large replies from nonidempotent operations (like COMPOUND requests with a READ operation) should not be cached 🙈 The requester decides which replies to cache with an argument to SEQUENCE (the sa_cachethis field, see Section 18.46) or CB_SEQUENCE (the csa_cachethis field, see Section 20.9). After the session is created, if a requester sends a request whose reply would be bigger than ca_maxresponsesize_cached, the replier will return NFS4ERR_REP_TOO_BIG_TO_CACHE, per the description in Section 2.10.6.4 ✨

ca_maxoperations: this is the maximum number of operations the replier will accept in a COMPOUND or CB_COMPOUND 🔥 For the backchannel, the server MUST NOT change the value the client offers. For the fore channel, the server MAY change the requested value. After the session is created, if a requester sends a COMPOUND or CB_COMPOUND with more operations than ca_maxoperations, the replier MUST return NFS4ERR_TOO_MANY_OPS 💅

ca_maxrequests: this is the maximum number of concurrent COMPOUND or CB_COMPOUND requests the requester will send on the session 🦋 subsequent requests will each be given a slot identifier by the requester, within the range zero to ca_maxrequests - 1 inclusive. For the backchannel, the server MUST NOT change the value the client offers. For the fore channel, the server MAY change the requested value.

ca_rdma_ird: this array has a maximum of one element. If it has one element, then that element is the inbound RDMA read queue depth (IRD) 🌈 For each channel, the server MAY decrease this value, but MUST NOT increase it.

csa_cb_program: this is the ONC RPC program number the server MUST use in any callbacks it sends through the backchannel to the client 👑 The server MUST specify an ONC RPC program number equal to csa_cb_program and an ONC RPC version number equal to 4 in callbacks sent to the client. If a CB_COMPOUND is sent to the client, the server MUST use a minor version number of 1. There is no corresponding result 💖

csa_sec_parms

The csa_sec_parms field is basically the list of security outfits the server is allowed to wear when it talks back to the client on the backchannel 💅🎀 There are three flavors on the menu: AUTH_NONE, AUTH_SYS, and RPCSEC_GSS.

If AUTH_NONE is specified for a credential, the client is authorizing the server to use AUTH_NONE on all callbacks for the session 🌸 If AUTH_SYS is specified, the server can use AUTH_SYS on all callbacks, using the credential in cbsp_sys_cred 🧚 And if RPCSEC_GSS is specified, the server can use the RPCSEC_GSS context in cbsp_gss_parms as the RPCSEC_GSS context in the credential of the RPC header of callbacks to the client ✨ There's no corresponding result, so nothing to see there 🙈

The backchannel's RPCSEC_GSS context is described with a pair of gsshandle4_t values. A gsshandle4_t is an RPCSEC_GSS handle, and it's exactly the same data type as the handle field of the rpc_gss_init_res data type from Section 5.2.3.1, "Context Creation Response - Successful Acceptance", of [4]. Very iconic of it to match like that 💖

The first handle, gcbp_handle_from_server, is the fore handle the server gave the client when the RPCSEC_GSS context was created on the server 🔥 The client got it either in the handle field of rpc_gss_init_res or as one of the elements of the spi_handles field in the reply to EXCHANGE_ID. The second handle, gcbp_handle_from_client, is the back handle the client maps the RPCSEC_GSS context to. The server can immediately use gcbp_handle_from_client in the RPCSEC_GSS credential in callback RPCs. In other words, that value can be used as the handle field in rpc_gss_cred_t (see Section 5, "Elements of the RPCSEC_GSS Security Protocol", of [4]) in callback RPCs.

The server MUST use the RPCSEC_GSS security service specified in gcbp_service. Put simply, it MUST set the service field of the rpc_gss_cred_t data type in the RPCSEC_GSS credential to the value of gcbp_service (see Section 5.3.1, "RPC Request Header", of [4]). No skipping this step, bestie 😤

If the RPCSEC_GSS handle identified by gcbp_handle_from_server doesn't exist on the server, the server will return NFS4ERR_NOENT 💔

Within each element of csa_sec_parms, the fore and back RPCSEC_GSS contexts MUST share the same GSS context and MUST have the same seq_window (see Section 5.2.3.1 of RFC2203 [4]). Their RPCSEC_GSS sequence numbers are independent of each other, though. The fore and back context state don't affect each other as far as the seq_num field goes (see the seq_num field in the rpc_gss_cred_t data type of Sections 5 and 5.3.1 of [4]) 🦋

If an RPCSEC_GSS handle is using the SSV context (see Section 2.10.9), then each SSV RPCSEC_GSS handle shares a common SSV GSS context. That means there are security considerations specific to this situation, discussed in Section 2.10.10. Please read that one carefully, it's seriously important 😭

Last bit: once the session is created, the first SEQUENCE or CB_SEQUENCE received on a slot MUST have a sequence ID equal to 1. If it doesn't, the replier MUST return NFS4ERR_SEQ_MISORDERED. Rude, but that's the rule 💅

18.36.4. IMPLEMENTATION ✨

okay so, to describe a possible implementation, we're reusing the same notation for client records that we met in EXCHANGE_ID, with one extra bit 🎀

• clientid_arg: The value of the csa_clientid field of the CREATE_SESSION4args structure of the current request.

ngl, CREATE_SESSION is non-idempotent, so we have to think about retries 😭 They can happen because of a client restart, a network partition, a malfunctioning router, etc. For each client ID created by EXCHANGE_ID, the server keeps a separate reply cache (called the CREATE_SESSION reply cache). It's kinda like the session reply cache used for SEQUENCE operations, with two differences 💅

• First, this is a reply cache just for detecting and processing CREATE_SESSION requests for a given client ID.
• Second, the client ID reply cache has exactly one slot (so the CREATE_SESSION request doesn't carry a slot number). That means at most one CREATE_SESSION request for a given client ID can be outstanding.

As we said before, CREATE_SESSION can be sent with or without a preceding SEQUENCE operation. Even if a SEQUENCE comes before CREATE_SESSION, the server MUST maintain the CREATE_SESSION reply cache. It's separate from the reply cache for the session associated with a SEQUENCE 🦋 If CREATE_SESSION was originally sent by itself, the client MAY send a retry of the CREATE_SESSION operation within a COMPOUND preceded by a SEQUENCE. If CREATE_SESSION was originally sent in a COMPOUND that started with a SEQUENCE, then the client SHOULD send a retry in a COMPOUND that starts with a SEQUENCE that has the same session ID as the SEQUENCE of the original request. However, the client MAY send a retry in a COMPOUND that either has no preceding SEQUENCE, or has a preceding SEQUENCE that refers to a different session than the original CREATE_SESSION. This might be necessary if the client sends a CREATE_SESSION in a COMPOUND preceded by a SEQUENCE with session ID X, and session X no longer exists. Regardless, any retry of CREATE_SESSION, with or without a preceding SEQUENCE, MUST use the same value of csa_sequence as the original 🫶

Every retry needs the same csa_sequence as the original, no exceptions 💀

After the client gets a reply to an EXCHANGE_ID operation that contains a new, unconfirmed client ID, the server expects the client to follow up with a CREATE_SESSION operation to confirm the client ID. The server expects the value of csa_sequenceid in the arguments to that CREATE_SESSION to equal the value of eir_sequenceid returned in the results of the EXCHANGE_ID that returned the unconfirmed client ID. Before the server replies to that EXCHANGE_ID operation, it initializes the client ID slot to be equal to eir_sequenceid - 1 (accounting for underflow), and records a contrived CREATE_SESSION result with a "cached" result of NFS4ERR_SEQ_MISORDERED. With the client ID slot set up like that, processing of the CREATE_SESSION operation is divided into four phases 🌸

1. Client record look up. The server looks up the client ID in its client record table. If the server has no records with client ID equal to clientid_arg, then most likely the client's state was purged during a period of inactivity, possibly because connectivity was lost. NFS4ERR_STALE_CLIENTID is returned, and no changes are made to any client records on the server. Otherwise, the server moves on to phase 2.
2. Sequence ID processing. If csa_sequenceid equals the sequence ID in the client ID's slot, then this is a replay of the previous CREATE_SESSION request, and the server returns the cached result. If csa_sequenceid is not equal to the sequence ID in the slot, and is more than one greater (accounting for wraparound), then the server returns the error NFS4ERR_SEQ_MISORDERED and does not change the slot. If csa_sequenceid equals the slot's sequence ID + 1 (accounting for wraparound), then the slot's sequence ID is set to csa_sequenceid, and CREATE_SESSION processing goes on to the next phase. A subsequent new CREATE_SESSION call over the same client ID MUST use a csa_sequenceid that is one greater than the sequence ID in the slot 🔥
3. Client ID confirmation. If this would be the first session for the client ID, the CREATE_SESSION operation serves to confirm the client ID. Otherwise, the client ID confirmation phase is skipped and only the session creation phase happens. Any case in which there is more than one record with identical values for client ID is a server implementation error 🙈 The potentially valid cases work out like this:
   • Successful Confirmation

If the server has the following unconfirmed record, then this is the expected confirmation of an unconfirmed record.

{ ownerid, verifier, principal_arg, clientid_arg,
unconfirmed }

As noted in Section 18.35.4, the server might also have the following confirmed record.

{ ownerid, old_verifier, principal_arg, old_clientid,
confirmed }

The server schedules the replacement of both records with:

{ ownerid, verifier, principal_arg, clientid_arg, confirmed
}

Processing of CREATE_SESSION continues on to session creation. Once the session is successfully created, the scheduled client record replacement is committed. If the session is not successfully created, then no changes are made to any client records on the server. ✨

• Unsuccessful Confirmation

If the server has the following record, then either the client has changed principals after the previous EXCHANGE_ID request, or there has been a chance collision between shorthand client identifiers.

{ *, *, old_principal_arg, clientid_arg, * }

Neither of these cases is permissible. Processing stops and NFS4ERR_CLID_INUSE is returned to the client. No changes are made to any client records on the server. 😭

4. Session creation. The server has confirmed the client ID, either in this CREATE_SESSION operation or in a previous one. The server examines the remaining fields of the arguments.

The server creates the session by recording the parameter values used (including whether the CREATE_SESSION4_FLAG_PERSIST flag is set and has been accepted by the server), and by allocating space for the session reply cache (if there isn't enough space, the server returns NFS4ERR_NOSPC). For each slot in the reply cache, the server sets the sequence ID to zero, and records an entry containing a COMPOUND reply with zero operations and the error NFS4ERR_SEQ_MISORDERED. This way, if the first SEQUENCE request sent has a sequence ID equal to zero, the server can just return what's in the reply cache: NFS4ERR_SEQ_MISORDERED. The client initializes its reply cache for receiving callbacks in the same way, and similarly, the first CB_SEQUENCE operation on a slot after session creation MUST have a sequence ID of one 💖

If the session state is created successfully, the server associates the session with the client ID the client provided.

When a request that had CREATE_SESSION4_FLAG_CONN_RDMA set needs to be retried, the retry MUST be done on a new connection that is in non-RDMA mode. If the properties of the new connection are different enough that the arguments to CREATE_SESSION need to change, then a non-retry MUST be sent. The server will eventually dispose of any session that was created on the original connection 🧚

On the backchannel, the client and server might want lots of slots, in some cases even more than the fore channel, so they can handle situations where the network link has high latency and is the main bottleneck for responding to recalls. If so, and if the client gives the backchannel too few slots, the server might limit the number of recallable objects it hands out to the client 🍓

Implementing RPCSEC_GSS callback support needs changes to both the client and server implementations of RPCSEC_GSS. One possible set of changes includes:

• Adding a data structure that wraps the GSS-API context with a reference count.
• New functions to increment and decrement the reference count. If the reference count drops to zero, the wrapper data structure and the GSS-API context it refers to would be freed.
• Changing RPCSEC_GSS to create the wrapper data structure upon receiving a GSS-API context from gss_accept_sec_context() and gss_init_sec_context(). The reference count would be initialized to 1.
• Adding a function to map an existing RPCSEC_GSS handle to a pointer to the wrapper data structure. The reference count would be incremented.
• Adding a function to create a new RPCSEC_GSS handle from a pointer to the wrapper data structure. The reference count would be incremented.
• Replacing calls from RPCSEC_GSS that free GSS-API contexts with calls that decrement the reference count on the wrapper data structure.

---

18.37. Operation 44: DESTROY_SESSION - Destroy a Session 💜

18.37.1. ARGUMENT

struct DESTROY_SESSION4args {
        sessionid4      dsa_sessionid;
};

18.37.2. RESULT

struct DESTROY_SESSION4res {
        nfsstat4        dsr_status;
};

18.37.3. DESCRIPTION 💅✨

okay so DESTROY_SESSION is the op that closes a session and tosses its reply cache 🗑️ any connections still hanging onto that session get disassociated right away 🔌 and if a connection has zero sessions left, the server MAY close it 🫶 but locks, delegations, layouts, wants, and the lease? totally untouched, because those are tied to the client ID, not the session 🔒

now the rules, bestie 📜 DESTROY_SESSION MUST be sent over a connection that's associated with the session being destroyed. if the client ID was made with SP4_MACH_CRED state protection, the RPCSEC_GSS principal who created the session MUST be the one destroying it, and they have to use RPCSEC_GSS privacy or integrity 🔐 if it was SP4_SSV instead, RPCSEC_GSS using the SSV mechanism (Section 2.10.9) MUST be used, with integrity or privacy 🧚

the COMPOUND ordering stuff is where it gets spicy 🌶️ if the COMPOUND starts with SEQUENCE and the sessionids in SEQUENCE and DESTROY_SESSION match, then:

• DESTROY_SESSION MUST be the final operation in the COMPOUND request. 💀
• it's advisable to avoid putting DESTROY_SESSION in a COMPOUND with other state-modifying operations, because it'll nuke the reply cache 🙈
• since the session and its reply cache are gone, a client that retries might get an error back even though the original request actually worked 😭

if the sessionids are different instead, DESTROY_SESSION can show up anywhere in the COMPOUND except the first spot ✨ and those two sessionids can belong to different client IDs, which is very main character energy 👑

if the COMPOUND doesn't start with SEQUENCE and DESTROY_SESSION isn't the only op, the server MUST return NFS4ERR_NOT_ONLY_OP 🚫

last thing, the backchannel 🔙 if there's one on the session and the server still has CB_COMPOUND operations for it that haven't been replied to, the server MAY refuse to destroy the session and return an error. if that happens and the backchannel is down, the server SHOULD return NFS4ERR_CB_PATH_DOWN so the client knows it needs to fix the backchannel first 🛠️ otherwise, CB_BACK_CHAN_BUSY SHOULD be returned to say there are CB_COMPOUNDs that still need replies 📬 and the client SHOULD reply to every outstanding CB_COMPOUND before re-sending DESTROY_SESSION 🫶🌸

---

18.38. Operation 45: FREE_STATEID - Free Stateid with No Locks 🧹

18.38.1. ARGUMENT 🎀

struct FREE_STATEID4args {
        stateid4        fsa_stateid;
};

18.38.2. RESULT 🍓

struct FREE_STATEID4res {
        nfsstat4        fsr_status;
};

18.38.3. DESCRIPTION 🦋

okay so FREE_STATEID is how you free a stateid that has zero locks left on it (that includes opens, byte-range locks, delegations, and layouts) 🫧 maybe the client sent LOCKU, or maybe the server revoked things. if there are still valid locks of any kind on that stateid, you get NFS4ERR_LOCKS_HELD and the stateid does NOT get freed 🙅‍♀️

when you free a stateid that had revoked locks attached, you're basically telling the server "yep, I acknowledge I lost those locks" 🫶 and once the server has every revoked bit acknowledged, it can let that client reclaim locks again without running into the edge cases in Section 8.4.2 💖

and once FREE_STATEID works on a stateid, any later use of that stateid gets NFS4ERR_BAD_STATEID 💀 it's gone, bestie 👋

---

18.39. Operation 46: GET_DIR_DELEGATION - Get a Directory Delegation 📂✨

18.39.1. ARGUMENT 🎀

typedef nfstime4 attr_notice4;

struct GET_DIR_DELEGATION4args {
        /* CURRENT_FH: delegated directory */
        bool            gdda_signal_deleg_avail;
        bitmap4         gdda_notification_types;
        attr_notice4    gdda_child_attr_delay;
        attr_notice4    gdda_dir_attr_delay;
        bitmap4         gdda_child_attributes;
        bitmap4         gdda_dir_attributes;
};

18.39.2. RESULT 🍓

struct GET_DIR_DELEGATION4resok {
        verifier4       gddr_cookieverf;
        /* Stateid for get_dir_delegation */
        stateid4        gddr_stateid;
        /* Which notifications can the server support */
        bitmap4         gddr_notification;
        bitmap4         gddr_child_attributes;
        bitmap4         gddr_dir_attributes;
};

enum gddrnf4_status {
        GDD4_OK         = 0,
        GDD4_UNAVAIL    = 1
};

union GET_DIR_DELEGATION4res_non_fatal
 switch (gddrnf4_status gddrnf_status) {
 case GDD4_OK:
  GET_DIR_DELEGATION4resok      gddrnf_resok4;
 case GDD4_UNAVAIL:
  bool                          gddrnf_will_signal_deleg_avail;
};
union GET_DIR_DELEGATION4res
 switch (nfsstat4 gddr_status) {
 case NFS4_OK:
  GET_DIR_DELEGATION4res_non_fatal      gddr_res_non_fatal4;
 default:
  void;
};

18.39.3. DESCRIPTION 🦋

okay so GET_DIR_DELEGATION is how a client asks for a directory delegation 📂 the directory is whatever the current filehandle points to 👀 the client can also say "hey, tell me when the directory changes in certain ways" by setting bits in a bitmap 🔔 the server is allowed to say no, and in that case it returns NFS4ERR_DIRDELEG_UNAVAIL 🙅‍♀️ if it says yes, it hands back a cookie verifier for that directory 🍪 and if that verifier changes while the client is holding the delegation, the delegation gets recalled UNLESS the client asked to be notified about that event 🫶

the server also returns a directory delegation stateid, gddr_stateid 🆔 that stateid shows up in callbacks about the delegation, like notifications and recalls 📞 the client uses it to give the delegation back, either willingly or when it gets recalled ✨ giving it back is done with DELEGRETURN 🎀

the server might not be able to do notifications for every event 🤷‍♀️ if the client asks for one it can't handle, the server MUST tell the client as much in the GET_DIR_DELEGATION reply by leaving that bit unset in the supported notifications bitmask, gddr_notification 🚩 and the server MUST NOT add bits to gddr_notification that the client never asked for 🙅‍♀️

GET_DIR_DELEGATION works on both normal directories and named attribute directories 📁🌈

now the "want" thing 🫧 if the client sets gdda_signal_deleg_avail to TRUE, it's basically registering a "want" for a directory delegation with the server 💌 if there's no delegation available right now, and the server supports and will honor the "want", the results have gddrnf_will_signal_deleg_avail set to TRUE and no error comes back 🥳 in that case the client should expect a future CB_RECALLABLE_OBJ_AVAIL operation saying a directory delegation is ready 📬 if the server won't honor the "want", or can't, it returns NFS4ERR_DIRDELEG_UNAVAIL 🥲 and if a delegation is available right away, the server SHOULD return it in the response to the operation instead of making the client wait for a callback ⚡💖

one more thing, the "already have one" case 👑 if the client makes a request for a directory delegation while it already holds one for that directory (this includes when it was recalled but not returned yet, or revoked by the server), the server MUST reply with gddr_status set to NFS4_OK, gddrnf_status set to GDD4_UNAVAIL, and gddrnf_will_signal_deleg_avail set to FALSE 🙅‍♀️ the delegation the client already had stays intact and its state doesn't change 🔒 and the current stateid isn't changed either (see Section 16.2.3.1.2 for what the current stateid is) 🧚✨

18.39.4. IMPLEMENTATION 🧚

okay so directory delegations are all about keeping namespace info consistent in your cache 💅, and they do it with synchronous callbacks. no synchronous callbacks, no directory delegations, periodt. asynchronous notifications are the lowkey bonus that can cut network traffic and make clients feel faster in certain situations ✨

notifications are described as potential changes to the directory. a client asks for them by setting one or more bits in gdda_notification_types. wanna hear about entries being added? set NOTIFY4_ADD_ENTRY. removals? NOTIFY4_REMOVE_ENTRY. renames? NOTIFY4_RENAME_ENTRY. changes to directory attributes? NOTIFY4_CHANGE_DIR_ATTRIBUTES. cookie verifier changes? NOTIFY4_CHANGE_COOKIE_VERIFIER. each one gets its own bit in the gdda_notification_types field 🎀

the client can also ask for NOTIFY4_CHANGE_CHILD_ATTRIBUTES so its attribute cache stays fresh 🌸 but heads up, changes to child attributes do not recall the delegation. if you're into directory entry caching or negative name caching, set gdda_notification_types to match what you need and the server will tell you about every change that would otherwise invalidate your name cache. what you ask for can depend on the directory's size, how often it changes, and which apps are poking at it. figuring out every condition where a client might want a notification is out of scope for this spec 🦋

for attribute notifications, the client sets bits in the gdda_dir_attributes bitmap to say which attributes it wants updates on. if the server doesn't support notifications for a given attribute, it SHOULD NOT put that attribute in the supported attribute bitmap in the reply (gddr_dir_attributes). the client also sets the attributes of directory entries it wants in gdda_child_attributes, and the server reports back in gddr_child_attributes which of those it will actually notify on 💜

the client also picks timing with a delay factor: gdda_child_attr_delay is for changes to directory entries, and gdda_dir_attr_delay is for changes to the directory itself 🍓 a delay of zero means "tell me the second it changes." a delay of N seconds means the server makes a best-effort promise that attribute updates are synced within N seconds. if the server doesn't support the delay the client asked for, or the delay would cause a ton of notifications and eat up server resources, the server should not commit to sending attribute notifications. so it must not set the matching bit in the gddr_child_attributes and gddr_dir_attributes bitmaps in the response 🙈

the client MUST use a security tuple (Section 2.6.1) that the directory or its applicable ancestor (Section 2.6) is exported with. if it doesn't, the server MUST return NFS4ERR_WRONGSEC to the operation that both precedes GET_DIR_DELEGATION and sets the current filehandle (see Section 2.6.3.1). 💀

the directory delegation covers every entry in the directory except the parent entry 🌈 so if a directory and its parent both hold directory delegations, changes to the parent won't send a notification for the child, even though the child's parent entry points to that parent directory. iykyk 😭

---

18.40. Operation 47: GETDEVICEINFO - Get Device Information 🔥

18.40.1. ARGUMENT

struct GETDEVICEINFO4args {
        deviceid4       gdia_device_id;
        layouttype4     gdia_layout_type;
        count4          gdia_maxcount;
        bitmap4         gdia_notify_types;
};

18.40.2. RESULT

struct GETDEVICEINFO4resok {
        device_addr4    gdir_device_addr;
        bitmap4         gdir_notification;
};

union GETDEVICEINFO4res switch (nfsstat4 gdir_status) {
case NFS4_OK:
        GETDEVICEINFO4resok     gdir_resok4;
case NFS4ERR_TOOSMALL:
        count4                  gdir_mincount;
default:
        void;
};

18.40.3. DESCRIPTION

GETDEVICEINFO is the op that hands back pNFS storage device address info for a given device ID ✨ the client says which device it means by sending gdia_device_id and gdia_layout_type, which together uniquely identify the device. it also sends gdia_maxcount to cap how many bytes the result can be. that limit covers everything in the GETDEVICEINFO4resok structure, XDR overhead included 🎀 the server can send back less than that. if it can't fit anything under gdia_maxcount, it returns NFS4ERR_TOOSMALL. but if gdia_maxcount is zero, NFS4ERR_TOOSMALL MUST NOT be returned 🫶

the da_layout_type field of the gdir_device_addr the server returns MUST be equal to the gdia_layout_type the client sent. if it isn't, the client SHOULD ignore the response as invalid and act as if the server returned an error, even if the client does support the layout type that came back 💅

the client also sends a notification bitmap, gdia_notify_types, for the device ID mapping notifications it wants. the server must support device ID notifications for the request to have affect. the mask is built the same way as the bitmap for file attributes (Section 3.3.7). the bit positions come from the notify_device_type4 enumeration (Section 20.12). only two of its values apply to GETDEVICEINFO right now: NOTIFY_DEVICEID4_CHANGE and NOTIFY_DEVICEID4_DELETE (see Section 20.12) 👑

the notification bitmap only applies to the device ID you named. if a client sends GETDEVICEINFO for the same deviceID more than once, the server uses the last notification bitmap for later notifications. if the bitmap is zero or empty, notifications for that device ID are turned off 🦋

if the client just wants to update or turn off notifications, it MAY send a GETDEVICEINFO with gdia_maxcount set to zero. in that case, if the device ID is valid, the da_addr_body field of the gdir_device_addr in the reply will have zero length 🧚

if gdia_device_id is an unknown device ID, the server returns NFS4ERR_NOENT. otherwise the device address info comes back in gdir_device_addr. and if the server supports device ID mapping notifications, the gdir_notification result holds a bitmap of which notifications it will actually send to the client (via CB_NOTIFY_DEVICEID, see Section 20.12) 💖

if NFS4ERR_TOOSMALL comes back, the results also include gdir_mincount, which is the minimum size needed to get the device info 🍓

18.40.4. IMPLEMENTATION

besides updating or turning off notifications, setting gdia_maxcount to zero has one other use: validating a device ID 🌸

the client SHOULD request a notification for changes to, or deletion of, a device ID to device address mapping. that way the server can let the client switch to a new mapping gracefully, without pending I/O failing abruptly, and without forcing layouts that use the device ID to be recalled or revoked 💜

GETDEVICEINFO (and GETDEVICELIST) can race with CB_NOTIFY_DEVICEID. that means CB_NOTIFY_DEVICEID can show up before the client gets and processes the response to GETDEVICEINFO or GETDEVICELIST. the race analysis relies on one fact: the server MUST NOT delete a device ID that a layout the client has refers to 🙈

• CB_NOTIFY_DEVICEID deletes a device ID. if the client thinks it has layouts that refer to that device ID, then layouts referring to the deleted device ID might have been revoked. the client should send a TEST_STATEID request using the stateid for each layout that might have been revoked. if TEST_STATEID shows any layouts were revoked, the client must recover from layout revocation as described in Section 12.5.6. if TEST_STATEID shows at least one layout was not revoked, the client should send a GETDEVICEINFO on the supposedly deleted device ID to check that it really was deleted ⭐

if GETDEVICEINFO says the device ID doesn't exist, the client assumes the server is faulty and recovers by sending an EXCHANGE_ID operation. if GETDEVICEINFO says the device ID does exist, then the server is faulty for sending a wrong device ID deletion notification. but how faulty it is doesn't require the client to create a new client ID 🌈

if the client has no layouts that refer to the device ID, no harm is done. the client should mark the device ID as deleted. once GETDEVICEINFO or GETDEVICELIST results show the device ID really was deleted, it should be removed from the client's cache ✨

• CB_NOTIFY_DEVICEID says a device ID's device addressing mappings have changed. the client should assume the results of its in-progress GETDEVICEINFO will be stale for that device ID once they arrive, so it should send another GETDEVICEINFO on the device ID 🔥

---

18.41. Operation 48: GETDEVICELIST - Get All Device Mappings for a File System 🦋

18.41.1. ARGUMENT

struct GETDEVICELIST4args {
        /* CURRENT_FH: object belonging to the file system */
        layouttype4     gdla_layout_type;

        /* number of deviceIDs to return */
        count4          gdla_maxdevices;

        nfs_cookie4     gdla_cookie;
        verifier4       gdla_cookieverf;
};

18.41.2. RESULT

struct GETDEVICELIST4resok {
        nfs_cookie4             gdlr_cookie;
        verifier4               gdlr_cookieverf;
        deviceid4               gdlr_deviceid_list<>;
        bool                    gdlr_eof;
};

union GETDEVICELIST4res switch (nfsstat4 gdlr_status) {
case NFS4_OK:
        GETDEVICELIST4resok     gdlr_resok4;
default:
        void;
};

18.41.3. DESCRIPTION 🧚

okay so this operation is how the client lists out every single device ID that the server's file system uses 🗂️✨

The client hands over a current filehandle for a file object that belongs to the file system (so basically any file object sharing the same fsid as the current filehandle) plus the layout type in gdia_layout_type. Since listing everything can take more than one call (kinda like READDIR (Section 18.23) 💅), the client also sends gdia_cookie and gdia_cookieverf to mark where it is in the list. 🔖

Wanna start from the very beginning of the file system's device mappings? Set gdla_cookie to zero 🌸 and then the server MUST ignore gdla_cookieverf, since it doesn't matter when the cookie is zero. The client uses gdla_maxdevices to cap how many device IDs come back. If gdla_maxdevices is zero, the server MUST return NFS4ERR_INVAL 🙈. The server MAY return fewer device IDs than you asked for, so don't panic if the list comes back short!

The successful reply carries a new cookie, gdlr_cookie, and a cookie verifier, gdlr_cookieverf, which you use for the next GETDEVICELIST call 🦋. If gdlr_eof comes back TRUE, that means the server has no more entries in its device list 🎀. Each item inside gdlr_deviceid_list is a device ID.

---

18.41.4. IMPLEMENTATION 🍓

Here's a real-life example: pNFS clients and servers that use LAYOUT4_BLOCK_VOLUME layouts. In those setups it may be helpful for a client to check whether each device is accessible right when it first touches the file system 👑

---

18.42. Operation 49: LAYOUTCOMMIT - Commit Writes Made Using a Layout 💜

18.42.1. ARGUMENT 🎀

union newtime4 switch (bool nt_timechanged) {
case TRUE:
        nfstime4           nt_time;
case FALSE:
        void;
};

union newoffset4 switch (bool no_newoffset) {
case TRUE:
        offset4           no_offset;
case FALSE:
        void;
};

struct LAYOUTCOMMIT4args {
        /* CURRENT_FH: file */
        offset4                 loca_offset;
        length4                 loca_length;
        bool                    loca_reclaim;
        stateid4                loca_stateid;
        newoffset4              loca_last_write_offset;
        newtime4                loca_time_modify;
        layoutupdate4           loca_layoutupdate;
};

18.42.2. RESULT 🌈

union newsize4 switch (bool ns_sizechanged) {
case TRUE:
        length4         ns_size;
case FALSE:
        void;
};

struct LAYOUTCOMMIT4resok {
        newsize4                locr_newsize;
};

union LAYOUTCOMMIT4res switch (nfsstat4 locr_status) {
case NFS4_OK:
        LAYOUTCOMMIT4resok      locr_resok4;
default:
        void;
};

18.42.3. DESCRIPTION 💅

LAYOUTCOMMIT is the operation that commits changes to the layout tied to the current filehandle, the client ID (which comes from the session ID in the preceding SEQUENCE), a byte-range, and a stateid ✨. Layouts can be split up, so you can commit just a smaller piece of a layout you got from LAYOUTGET. That byte-range goes in loca_offset and loca_length. This byte-range MUST overlap with one or more existing layouts previously granted via LAYOUTGET (Section 18.43), and each of those needs an iomode of LAYOUTIOMODE4_RW 🔥. If any held layout segment has an iomode that isn't LAYOUTIOMODE4_RW, the server should return NFS4ERR_BAD_IOMODE. And if the client doesn't hold matching layout segments for that byte-range, the server should return NFS4ERR_BAD_LAYOUT 💀.

So LAYOUTCOMMIT basically says "hey, I'm done writing with this layout from before" 🫶. The client might've only written part of what it asked for, and that's totally fine: LAYOUTCOMMIT lets it commit or discard provisionally allocated space and tell the server about a new end-of-file. The layout is still valid after this, so the client can keep referencing it by client ID, filehandle, byte-range, layout type, and stateid 🦋.

Now for the reclaim situation 🙈. If loca_reclaim is TRUE, the client is trying to commit changes to a layout after the metadata server restarted, while the server is still in its recovery grace period (see Section 12.7.4). This can be needed when the client has uncommitted writes to provisionally allocated byte-ranges that it sent to the storage devices before the restart. In that case, the layout the client gives us MUST be a subset of a writable layout the client held right before the restart. The value of loca_stateid MUST be one the metadata server handed out before it restarted. The metadata server is free to accept or reject the request based on its own internal consistency checks. If the layout doesn't pass those checks, the server MUST reject the request with NFS4ERR_RECLAIM_BAD ⚠️.

A successful LAYOUTCOMMIT with loca_reclaim set to TRUE does NOT give the client a layout for the file 🙅‍♀️. It just commits the changes to the layout described in loca_layoutupdate. If the client wants an actual layout, it must send a LAYOUTGET request after the server's grace period is over. And if the metadata server gets a LAYOUTCOMMIT with loca_reclaim set to TRUE while it is NOT in its recovery grace period, it MUST reject the request with NFS4ERR_NO_GRACE.

Setting loca_reclaim to TRUE is required if and only if the committed layout was acquired before the metadata server restart 🎀. If the client is committing a layout that it got during the metadata server's grace period, it MUST set the "reclaim" field to FALSE.

loca_stateid is a layout stateid, the kind returned by earlier successful layout operations (see Section 12.5.3) 🔖.

loca_last_write_offset tells the server the offset of the last byte the client wrote before the LAYOUTCOMMIT 👑. This value is never equal to the file's size (at most it's one byte less than the size), and it MUST be less than or equal to NFS4_MAXFILEOFF. Also, loca_last_write_offset MUST overlap the range described by loca_offset and loca_length. The metadata server can use this to figure out whether the file's size needs updating. If the server does update the size because of LAYOUTCOMMIT, it must return the new size (locr_newsize.ns_size) in the results 🌸.

loca_time_modify is the client's way of suggesting a modification time 💜. The metadata server can use that suggestion, or it can use the time of the LAYOUTCOMMIT operation instead. If the server goes with the client's time, it should make sure that time doesn't flow backwards ⏳. Want to force an exact time? Then the client should send a SETATTR operation in a COMPOUND right after LAYOUTCOMMIT (see Section 12.5.4 for more details). And if the client wants to see the resulting modification time, it should build the COMPOUND so that a GETATTR follows the LAYOUTCOMMIT 🦋.

loca_layoutupdate is where the client passes layout-specific updates to the metadata server 🧠. For instance, it can say which byte-ranges of the original layout were actually used and which ones can be deallocated. There is no NFSv4.1 file layout-specific layoutupdate4 structure.

Block device layouts are way more detailed than object or file layouts 🧱, because those two hide the nitty-gritty of block allocation behind their storage protocols. At the very least, the client needs to tell the server about changes to the end-of-file location and, if it wants, its view of the modification time. For block/volume layouts, it also has to specify exactly which blocks were used.

If the layout in the arguments doesn't exist, the server returns the error NFS4ERR_BADLAYOUT 😭. The layout being committed can also get rejected if it doesn't match an existing layout with an iomode of LAYOUTIOMODE4_RW.

On success, the current filehandle keeps its value and the current stateid keeps its value too ✨.

18.42.4. IMPLEMENTATION 🌸

The client MAY also use LAYOUTCOMMIT with loca_reclaim set to TRUE to pass along hints about modified file attributes, or to report layout-type specific info like I/O errors for object-based storage layouts. It's the same stuff that gets reported during normal operation 🔧. Doing this may help the metadata server recover files more efficiently after a restart. For example, some file system implementations might need expansive recovery of file system objects if the metadata server doesn't get a positive signal from every client holding a LAYOUTIOMODE4_RW layout saying that it finished all its writes 😬. Sending a LAYOUTCOMMIT (if required) and then following it with LAYOUTRETURN can give that signal, which allows for graceful and efficient recovery 💖.

If loca_reclaim is TRUE, the metadata server is free to either examine or ignore the value in loca_stateid 🤷‍♀️. The metadata server implementation might or might not encode information in its layout stateid that lets it run a consistency check on the LAYOUTCOMMIT request 🔍.

---

18.43. Operation 50: LAYOUTGET - Get Layout Information 🧚

18.43.1. ARGUMENT 🍓

struct LAYOUTGET4args {
        /* CURRENT_FH: file */
        bool                    loga_signal_layout_avail;
        layouttype4             loga_layout_type;
        layoutiomode4           loga_iomode;
        offset4                 loga_offset;
        length4                 loga_length;
        length4                 loga_minlength;
        stateid4                loga_stateid;
        count4                  loga_maxcount;
};

18.43.2. RESULT 💜

struct LAYOUTGET4resok {
        bool               logr_return_on_close;
        stateid4           logr_stateid;
        layout4            logr_layout<>;
};

union LAYOUTGET4res switch (nfsstat4 logr_status) {
case NFS4_OK:
        LAYOUTGET4resok     logr_resok4;
case NFS4ERR_LAYOUTTRYLATER:
        bool                logr_will_signal_layout_avail;
default:
        void;
};

18.43.3. DESCRIPTION 🌈

LAYOUTGET is how a client asks the metadata server for a layout so it can read or write the file behind the filehandle, covering the byte-range given by offset and length 🦋. Layouts are identified by the client ID (derived from the session ID in the preceding SEQUENCE), the current filehandle, the layout type (loga_layout_type), and the layout stateid (loga_stateid). How loga_iomode gets used depends on the layout type, but it should reflect the client's data access intent 🎯.

Here's the important bit 🚨: if the metadata server is in a grace period and it does NOT persist layouts and device ID to device address mappings, then it MUST return NFS4ERR_GRACE (see Section 8.4.2.1).

okay so LAYOUTGET is the op where the client asks the metadata server for layout info for a byte-range, and what comes back is a layout 🗺️✨ this is literally the main character of pNFS rn 💅. The client actually sends two ranges, and both start at the offset in the loga_offset field. The first range runs from loga_offset to loga_offset + loga_length - 1 inclusive, and that's the desired range, the dream coverage the client would love to have 🌸. The second runs from loga_offset to loga_offset + loga_minlength - 1 inclusive, and that's the required range, the bare minimum it actually needs 🥺. So loga_minlength MUST be less than or equal to loga_length, no exceptions bestie.

When a length field is set to NFS4_UINT64_MAX, it's basically the client saying "give me everything, I don't care how big the file is" 🦋. If loga_length is NFS4_UINT64_MAX, that's a desire for a layout from loga_offset all the way to end-of-file, regardless of the file's length. If loga_minlength is NFS4_UINT64_MAX, that same thing becomes a requirement instead of a wish 👑.

The next bit is about how these lengths and the offset interact, and the minimums that come with them. These are the rules, so read closely 🧚

• If loga_length is less than loga_minlength, the metadata server MUST return NFS4ERR_INVAL.
• If loga_minlength is zero, the client is telling the metadata server it'll take any layout at offset loga_offset or less that the metadata server has "readily available" 🍓. "Readily" is subjective and depends on the layout type and the pNFS server implementation. For example, some metadata servers might have to pre-allocate stable storage when they get a request for a range of a file that goes beyond the file's current length. If loga_minlength is zero and loga_length is greater than zero, that tells the server what range the client would prefer to have. If loga_length and loga_minlength are both zero, then the client is saying it wants a layout of any length, with the ending offset of the range no less than the value of loga_offset, and the starting offset at or below loga_offset. If the metadata server does not have a layout that is readily available, then it MUST return NFS4ERR_LAYOUTTRYLATER 😭.
• If the sum of loga_offset and loga_minlength exceeds NFS4_UINT64_MAX, and loga_minlength is not NFS4_UINT64_MAX, the error NFS4ERR_INVAL MUST result.
• If the sum of loga_offset and loga_length exceeds NFS4_UINT64_MAX, and loga_length is not NFS4_UINT64_MAX, the error NFS4ERR_INVAL MUST result.

After the metadata server has done the checks above on loga_offset, loga_minlength, and loga_offset, it MUST return a layout according to the rules in Table 13. Table 13 is the big one, so here it is 👀

Acceptable layouts based on loga_minlength. Note: u64m = NFS4_UINT64_MAX; a_off = loga_offset; a_minlen = loga_minlength.

   +-----------+-----------+----------+----------+---------------------+
   | Layout    | Layout    | Layout   | Layout   | Layout length of    |
   | iomode of | a_minlen  | iomode   | offset   | reply               |
   | request   | of        | of reply | of reply |                     |
   |           | request   |          |          |                     |
   +-----------+-----------+----------+----------+---------------------+
   | _READ     | u64m      | MAY be   | MUST be  | MUST be >= file     |
   |           |           | _READ    | <= a_off | length - layout     |
   |           |           |          |          | offset              |
   | _READ     | u64m      | MAY be   | MUST be  | MUST be u64m        |
   |           |           | _RW      | <= a_off |                     |
   | _READ     | > 0 and < | MAY be   | MUST be  | MUST be >= MIN(file |
   |           | u64m      | _READ    | <= a_off | length, a_minlen +  |
   |           |           |          |          | a_off) - layout     |
   |           |           |          |          | offset              |
   | _READ     | > 0 and < | MAY be   | MUST be  | MUST be >= a_off -  |
   |           | u64m      | _RW      | <= a_off | layout offset +     |
   |           |           |          |          | a_minlen            |
   | _READ     | 0         | MAY be   | MUST be  | MUST be > 0         |
   |           |           | _READ    | <= a_off |                     |
   | _READ     | 0         | MAY be   | MUST be  | MUST be > 0         |
   |           |           | _RW      | <= a_off |                     |
   | _RW       | u64m      | MUST be  | MUST be  | MUST be u64m        |
   |           |           | _RW      | <= a_off |                     |
   | _RW       | > 0 and < | MUST be  | MUST be  | MUST be >= a_off -  |
   |           | u64m      | _RW      | <= a_off | layout offset +     |
   |           |           |          |          | a_minlen            |
   | _RW       | 0         | MUST be  | MUST be  | MUST be > 0         |
   |           |           | _RW      | <= a_off |                     |
   +-----------+-----------+----------+----------+---------------------+

Table 13

If loga_minlength is not zero and the metadata server can't return a layout according to Table 13, then the metadata server MUST return the error NFS4ERR_BADLAYOUT 🚫. If loga_minlength is zero and the metadata server cannot or will not return a layout according to Table 13, then it MUST return the error NFS4ERR_LAYOUTTRYLATER. As long as loga_length is greater than loga_minlength or equal to zero, the metadata server SHOULD return a layout according to the rules in Table 14 ✨.

Table 14 is the "desired" one, the wishlist 🎀. The rules of Table 13 MUST be applied first, though.

Desired layouts based on loga_length. The rules of Table 13 MUST be applied first. Note: u64m = NFS4_UINT64_MAX; a_off = loga_offset; a_len = loga_length.

   +------------+------------+-----------+-----------+-----------------+
   | Layout     | Layout     | Layout    | Layout    | Layout length   |
   | iomode of  | a_len of   | iomode of | offset of | of reply        |
   | request    | request    | reply     | reply     |                 |
   +------------+------------+-----------+-----------+-----------------+
   | _READ      | u64m       | MAY be    | MUST be   | SHOULD be u64m  |
   |            |            | _READ     | <= a_off  |                 |
   | _READ      | u64m       | MAY be    | MUST be   | SHOULD be u64m  |
   |            |            | _RW       | <= a_off  |                 |
   | _READ      | > 0 and <  | MAY be    | MUST be   | SHOULD be >=    |
   |            | u64m       | _READ     | <= a_off  | a_off - layout  |
   |            |            |           |           | offset + a_len  |
   | _READ      | > 0 and <  | MAY be    | MUST be   | SHOULD be >=    |
   |            | u64m       | _RW       | <= a_off  | a_off - layout  |
   |            |            |           |           | offset + a_len  |
   | _READ      | 0          | MAY be    | MUST be   | SHOULD be >     |
   |            |            | _READ     | <= a_off  | a_off - layout  |
   |            |            |           |           | offset          |
   | _READ      | 0          | MAY be    | MUST be   | SHOULD be >     |
   |            |            | _READ     | <= a_off  | a_off - layout  |
   |            |            |           |           | offset          |
   | _RW        | u64m       | MUST be   | MUST be   | SHOULD be u64m  |
   |            |            | _RW       | <= a_off  |                 |
   | _RW        | > 0 and <  | MUST be   | MUST be   | SHOULD be >=    |
   |            | u64m       | _RW       | <= a_off  | a_off - layout  |
   |            |            |           |           | offset + a_len  |
   | _RW        | 0          | MUST be   | MUST be   | SHOULD be >     |
   |            |            | _RW       | <= a_off  | a_off - layout  |
   |            |            |           |           | offset          |
   +------------+------------+-----------+-----------+-----------------+

Table 14

The loga_stateid field specifies a valid stateid. If the client doesn't currently hold a layout, loga_stateid represents a stateid reflecting the matching open, byte-range lock, or delegation stateid 🔐. Once the client holds a layout on the file, loga_stateid MUST be a stateid as returned from a previous LAYOUTGET or LAYOUTRETURN operation, or provided by a CB_LAYOUTRECALL operation (see Section 12.5.3) 💜.

The loga_maxcount field gives the maximum layout size, in bytes, that the client can handle. If the size of the layout structure is bigger than what maxcount specifies, the metadata server will return the NFS4ERR_TOOSMALL error 🙈.

The returned layout is an array called logr_layout, where each element is of type layout4. If a file has a single striping pattern, logr_layout SHOULD contain just one entry. Otherwise, if the requested range overlaps more than one striping pattern, logr_layout will contain as many entries as needed 🧩. The elements of logr_layout MUST be sorted in ascending order of the value of the lo_offset field of each element. There MUST be no gaps or overlaps in the range between two successive elements of logr_layout. The lo_iomode field in each element of logr_layout MUST be the same 🫶.

Table 13 and Table 14 both talk about a returned layout iomode, offset, and length. Because the returned layout is encoded in the logr_layout array, those need a bit more explaining, so here we go 🌈

iomode

The returned layout iomode in Table 13 and Table 14 is equal to the value of the lo_iomode field in each element of logr_layout. As Table 13 and Table 14 show, the metadata server MAY return a layout with an lo_iomode different from the requested iomode (the loga_iomode field of the request). If it does, it MUST ensure that the lo_iomode is more permissive than the loga_iomode requested. For example, this lets an implementation upgrade LAYOUTIOMODE4_READ requests to LAYOUTIOMODE4_RW requests at its discretion, within the limits of the layout type specific protocol 🦋. Either LAYOUTIOMODE4_READ or LAYOUTIOMODE4_RW MUST be returned.

offset

The returned layout offset in Table 13 and Table 14 is always equal to the lo_offset field of the first element of logr_layout 🎯.

length

Setting the returned layout length is a little messy, because the special layout length value NFS4_UINT64_MAX might be involved 🙃. For a logr_layout array of N elements, the lo_length field in the first N-1 elements MUST NOT be NFS4_UINT64_MAX. The lo_length field of the last element of logr_layout can be NFS4_UINT64_MAX under some conditions, described in the list below.

• If an applicable rule of Table 13 states that the metadata server MUST return a layout of length NFS4_UINT64_MAX, then the lo_length field of the last element of logr_layout MUST be NFS4_UINT64_MAX.
• If an applicable rule of Table 13 states that the metadata server MUST NOT return a layout of length NFS4_UINT64_MAX, then the lo_length field of the last element of logr_layout MUST NOT be NFS4_UINT64_MAX.
• If an applicable rule of Table 14 states that the metadata server SHOULD return a layout of length NFS4_UINT64_MAX, then the lo_length field of the last element of logr_layout SHOULD be NFS4_UINT64_MAX.
• When the returned layout length from Table 13 and Table 14 is not NFS4_UINT64_MAX, the returned layout length is equal to the sum of the lo_length fields of each element of logr_layout ✨

The logr_return_on_close result field is basically a bestie reminder telling the client to give the layout back before it closes the file 💅. When the metadata server sets this to TRUE, it MUST be prepared to recall the layout in case the client fails to return it before close 😭. For a metadata server that already knows a layout has to come back before a close, this return value lets it tell the client what to do, so one extra step just disappears from the client and server dance 🧚.

The logr_stateid stateid gets sent back to the client so it can use it in later layout operations 🫶. See Sections 8.2, 12.5.3, and 12.5.5.2 for more deets and requirements 📚.

The format of the returned layout (lo_content) depends on the layout type 🎀. The layout type (lo_content.loc_type) of each layout in the array the metadata server sends back (logr_layout) MUST be equal to the loga_layout_type the client asked for 🌸. If it isn't equal, the client SHOULD ignore the response as invalid and act like the metadata server returned an error, even if the client does support the returned layout type 🙈.

If neither the requested file nor its containing file system supports layouts, the metadata server MUST return NFS4ERR_LAYOUTUNAVAILABLE 💀. If the layout type isn't supported, the metadata server MUST return NFS4ERR_UNKNOWN_LAYOUTTYPE 😭. If layouts are supported but no layout matches the client's layout identification, the metadata server MUST return NFS4ERR_BADLAYOUT 💔. If an invalid loga_iomode is specified, or a loga_iomode of LAYOUTIOMODE4_ANY is specified, the metadata server MUST return NFS4ERR_BADIOMODE 🍓.

If the layout for the file is unavailable because of transient conditions, like file sharing prohibiting layouts, the metadata server MUST return NFS4ERR_LAYOUTTRYLATER ⏳.

If the layout request gets rejected because of an overlapping layout recall, the metadata server MUST return NFS4ERR_RECALLCONFLICT ✨. See Section 12.5.5.2 for details.

If the layout conflicts with a mandatory byte-range lock held on the file, and the storage devices have no way to enforce mandatory locks other than restricting layouts, the metadata server SHOULD return NFS4ERR_LOCKED 🔒.

If the client sets loga_signal_layout_avail to TRUE, it's registering a "want" for a layout with the client, in case the layout can't be obtained because of resource exhaustion 🦋. If the metadata server supports and will honor the "want", the results will have logr_will_signal_layout_avail set to TRUE. If so, the client should expect a CB_RECALLABLE_OBJ_AVAIL operation to say a layout is available 🎉.

On success, the current filehandle keeps its value, and the current stateid gets updated to match the value returned in the results 👑.

18.43.4. IMPLEMENTATION 💅

Typically, LAYOUTGET gets called as part of a COMPOUND request right after an OPEN operation, and it leaves the client with location information for the file 🗺️. This requires loga_stateid to be set to the special stateid that tells the metadata server to use the current stateid, which OPEN sets (see Section 16.2.3.1.2) 🔑. A client can also hold a layout across multiple OPENs 🌈. The client specifies a layout type that limits what kind of layout the metadata server will return, which stops metadata servers from granting layouts the client can't use 🙌.

As Table 13 and Table 14 show, the LAYOUTGET spec gives a pNFS client and server a lot of flexibility 🧚‍♀️. A pNFS client has several strategies for sending LAYOUTGET. Here are some examples:

• Okay so, if LAYOUTGET comes right after OPEN in the same COMPOUND request, and the OPEN asks for OPEN4_SHARE_ACCESS_READ access, the client might go for a _READ layout with loga_offset set to 0, loga_minlength set to 0, and loga_length set to NFS4_UINT64_MAX. If the file has space allocated to it, that space is striped over one or more storage devices, and there's either no conflicting layout or the idea of one doesn't apply to the pNFS server's layout type or implementation, then the metadata server might return a layout starting at offset 0 with a length equal to the length of the file, if that isn't NFS4_UINT64_MAX. If the file's length isn't a multiple of the pNFS server's stripe width (see Section 13.2 for a formal definition), the metadata server might round the returned layout's length up 📏.
• If LAYOUTGET comes right after OPEN in the same COMPOUND request, and the OPEN asks for OPEN4_SHARE_ACCESS_WRITE access and doesn't truncate the file, the client might go for a _RW layout with loga_offset set to 0, loga_minlength set to 0, and loga_length set to the file's current length (if known) or NFS4_UINT64_MAX ✍️. Like the previous case, under some conditions the metadata server might return a layout covering the entire length of the file or beyond 💖.
• Same as above, but this time the OPEN truncates the file 🪄. The client might expect to write from offset zero, so loga_offset and loga_minlength are set to 0, and loga_length is set to the value of threshold4_write_iosize. The metadata server might return a layout from offset zero with a length at least as long as threshold4_write_iosize 🌟.
• A process on the client asks to read from offset 10000 for length 50000. The client is using buffered I/O with buffer sizes of 4096 bytes 🧁. The client plans to map the process's request into a series of READ requests starting at offset 8192. The end offset needs to be higher than 10000 + 50000 = 60000, and the next offset that's a multiple of 4096 is 61440. The difference between 61440 and the layout's starting offset is 53248 (which is 4096 times 15) 🧮. Since threshold4_read_iosize is less than 53248, the client sends a LAYOUTGET request with loga_offset set to 8192, loga_minlength set to 53248, and loga_length set to the file's length (if known) minus 8192, or NFS4_UINT64_MAX (if the file's length isn't known) 💅. Because this LAYOUTGET request goes over the metadata server's threshold, it grants the layout, possibly with an initial offset of zero, and an end offset of at least 8192 + 53248 - 1 = 61439. Ideally, though, it's a layout with an offset aligned on the stripe width and a length that's a multiple of the stripe width 🎀.
• Same as above, but the client isn't using buffered I/O. Instead, all internal I/O requests go straight to the server 🚀. The LAYOUTGET request has loga_offset equal to 10000 and loga_minlength set to 50000. The value of loga_length is set to the length of the file. The metadata server is free to return a layout that fully overlaps the requested range, with a starting offset and length aligned on the stripe width 🦄.
• Again, a process on the client asks to read from offset 10000 for length 50000 (so a range with a starting offset of 10000 and an ending offset of 69999), and buffered I/O is in use 📖. The client expects the server might not be able to return a layout for the full I/O range 😬. The client plans to map the process's request into a series of thirteen READ requests starting at offset 8192, each with length 4096, for a total length of 53248 (which equals 13 * 4096). That fully contains the range the client's process wants to read 🧩. Because threshold4_read_iosize equals 4096, it's practical and reasonable for the client to use several LAYOUTGET operations to finish the series of READs ⭐. The client sends a LAYOUTGET request with loga_offset set to 8192, loga_minlength set to 4096, and loga_length set to 53248 or higher. The server will grant a layout, possibly with an initial offset of zero, and an end offset of at least 8192 + 4096 - 1 = 12287. Ideally it's a layout with an offset aligned on the stripe width and a length that's a multiple of the stripe width 🌸. This lets the client make forward progress, and it can send more LAYOUTGET operations for the rest of the range 🏃‍♀️.
• An NFS client detects a sequential read pattern, so it sends a LAYOUTGET operation that goes well beyond any current or pending read requests to the server 📚. The server might also detect this pattern and grant the LAYOUTGET request 👀. Once the client reads from an offset of the file that's 50% of the way through the range of the last layout it received, the client sends more operations from an offset that's 50% of the way through the last layout it received, so I/O doesn't stall waiting for a layout 😤. The client keeps requesting layouts with byte-ranges well in advance of the byte-ranges of recent read requests from processes running on the client 🦋.
• Same as above, but the client fails to detect the pattern while the server does. The next time the metadata server gets a LAYOUTGET, it returns a layout with a length well beyond loga_minlength 💫.
• A client is using buffered I/O and has a long queue of write-behinds to process, and it also detects a sequential write pattern 🌊. It sends a LAYOUTGET for a layout that spans the range of the queued write-behinds and well beyond, including ranges past the file's current length 🙈. The client keeps sending LAYOUTGET operations once the write-behind queue reaches 50% of the maximum queue length 🫶.

Once the client has obtained a layout referring to a particular device ID, the metadata server MUST NOT delete the device ID until the layout is returned or revoked 🔐.

CB_NOTIFY_DEVICEID can race with LAYOUTGET 🏁. One race scenario is that LAYOUTGET returns a device ID the client doesn't have device address mappings for. The metadata server sends a CB_NOTIFY_DEVICEID to add the device ID to the client's awareness, and meanwhile the client sends GETDEVICEINFO on the device ID. This scenario is discussed in Section 18.40.4 🧐. Another scenario is that the client processes CB_NOTIFY_DEVICEID before it processes the results from LAYOUTGET. The client will send a GETDEVICEINFO on the device ID. If the GETDEVICEINFO results arrive before the client gets the LAYOUTGET results, there's no longer a race 🏆. If the LAYOUTGET results arrive before the GETDEVICEINFO results, the client can either wait for the GETDEVICEINFO results or send another one to get possibly more up-to-date device address mappings for the device ID 💜.

18.44. Operation 51: LAYOUTRETURN - Release Layout Information ✨

18.44.1. ARGUMENT ✨

Okay, here's the argument struct for LAYOUTRETURN, the same constants also get reused for CB_LAYOUTRECALL 🎀

   /* Constants used for LAYOUTRETURN and CB_LAYOUTRECALL */
   const LAYOUT4_RET_REC_FILE      = 1;
   const LAYOUT4_RET_REC_FSID      = 2;
   const LAYOUT4_RET_REC_ALL       = 3;

   enum layoutreturn_type4 {
           LAYOUTRETURN4_FILE = LAYOUT4_RET_REC_FILE,
           LAYOUTRETURN4_FSID = LAYOUT4_RET_REC_FSID,
           LAYOUTRETURN4_ALL  = LAYOUT4_RET_REC_ALL
   };
   struct layoutreturn_file4 {
           offset4         lrf_offset;
           length4         lrf_length;
           stateid4        lrf_stateid;
           /* layouttype4 specific data */
           opaque          lrf_body<>;
   };

   union layoutreturn4 switch(layoutreturn_type4 lr_returntype) {
           case LAYOUTRETURN4_FILE:
                   layoutreturn_file4      lr_layout;
           default:
                   void;
   };


   struct LAYOUTRETURN4args {
           /* CURRENT_FH: file */
           bool                    lora_reclaim;
           layouttype4             lora_layout_type;
           layoutiomode4           lora_iomode;
           layoutreturn4           lora_layoutreturn;
   };

18.44.2. RESULT 🦋

Here's what you get back. If the status is good, you get an optional stateid, and it only shows up when lrs_present is TRUE 💅

   union layoutreturn_stateid switch (bool lrs_present) {
   case TRUE:
           stateid4                lrs_stateid;
   case FALSE:
           void;
   };

   union LAYOUTRETURN4res switch (nfsstat4 lorr_status) {
   case NFS4_OK:
           layoutreturn_stateid    lorr_stateid;
   default:
           void;
   };

18.44.3. DESCRIPTION 🌸

Okay bestie, so this operation is how the client gives one or more layouts back to the server 💖 Each layout is identified by the client ID (which comes from the session ID in the SEQUENCE operation right before it), plus lora_layout_type and lora_iomode. When lr_returntype is LAYOUTRETURN4_FILE, the layout is pinned down further by the current filehandle, lrf_offset, lrf_length, and lrf_stateid. If lrf_length is NFS4_UINT64_MAX, then every byte of the layout, starting at lrf_offset, gets returned ✨ When lr_returntype is LAYOUTRETURN4_FSID, the current filehandle picks out the file system, and every layout matching the client ID, the fsid of that file system, lora_layout_type, and lora_iomode comes back. When lr_returntype is LAYOUTRETURN4_ALL, every layout matching the client ID, lora_layout_type, and lora_iomode is returned, and the current filehandle isn't used at all 🫶 After this call, the client MUST NOT use the returned layout(s) and the associated storage protocol to access the file data. That part is super important, no cap 😭

If the set of layouts you're returning with LAYOUTRETURN4_FSID or LAYOUTRETURN4_ALL is empty, then no error happens, which is so chill 🧚 With LAYOUTRETURN4_FILE, the byte range you name still gets returned even if it's a piece of a layout you got with LAYOUTGET, a mix of several LAYOUTGET layouts, or a mix that includes some of those layouts plus subdivisions of them. And if the byte range doesn't cover any bytes where you actually hold a layout for that file, client ID, layout type, and mode, you still don't get an error. For the "bulk" return stuff, check out Section 12.5.5.2.1.5 🍓

The layout you're returning can be a subset or a superset of what CB_LAYOUTRECALL asked for. But if it's a subset, the recall isn't finished until the whole recalled scope has come back 👑 Recalled scope means the byte range for LAYOUTRETURN4_FILE, or the use of LAYOUTRETURN4_FSID, or the use of LAYOUTRETURN4_ALL. There must be a LAYOUTRETURN with a matching scope to complete the return, even if every current layout range was already returned one by one 💀

For every lr_returntype value, an iomode of LAYOUTIOMODE4_ANY means that all layouts matching the other arguments to LAYOUTRETURN (that's client ID, lora_layout_type, and either the current filehandle plus a range, or the fsid derived from the current filehandle, or LAYOUTRETURN4_ALL) are being returned 🎀

When lr_returntype is LAYOUTRETURN4_FILE, the lrf_stateid the client sends has to be a layout stateid, the kind that came back from earlier layout operations. Note that the "seqid" field of lrf_stateid MUST NOT be zero. Check out Sections 8.2, 12.5.3, and 12.5.5.2 for more on this 🔥

Returning a layout (or all of them) does NOT wipe out the mapping from storage device ID to storage device address. That mapping sticks around until it gets changed or deleted through device ID notification callbacks 🌈 Of course, if no remaining layouts point at a device ID anymore, the server is free to delete that device ID without a notification callback. That happens when notifications aren't in effect 🦋

If lora_reclaim is TRUE, the client is trying to return a layout that was acquired before the metadata server restarted, during the metadata server's grace period. When you return layouts that were acquired during the grace period, the client MUST set lora_reclaim to FALSE. lora_reclaim MUST also be set to FALSE when lr_layoutreturn is LAYOUTRETURN4_FSID or LAYOUTRETURN4_ALL. For more details, look at LAYOUTCOMMIT (Section 18.42) 💖

Layouts can be returned when they're recalled, or voluntarily (that is, before the server has recalled them). Either way, the client must properly propagate state changed under the context of the layout to the storage device(s) or to the metadata server before returning the layout 🫶

If the client returns the layout in response to a CB_LAYOUTRECALL where the lor_recalltype field of the clora_recall field was LAYOUTRECALL4_FILE, the client should use the lor_stateid value from CB_LAYOUTRECALL as the value for lrf_stateid. Otherwise, it should use logr_stateid (from a previous LAYOUTGET result) or lorr_stateid (from a previous LAYRETURN result). This is done to show the point in time (in terms of layout stateid transitions) when the recall was sent 🧚 The client uses the precise lora_recallstateid value and MUST NOT set the stateid's seqid to zero; otherwise, NFS4ERR_BAD_STATEID MUST be returned. NFS4ERR_OLD_STATEID can be returned if the client is using an old seqid and the server knows the client shouldn't be using it. For example, the client uses the seqid on slot 1 of the session, gets the response with the new seqid, and then uses the slot to send another request with the old seqid 💅

If a client doesn't return a layout in a timely manner, the metadata server SHOULD use its control protocol with the storage devices to fence the client from accessing the data referenced by the layout 🔥 See Section 12.5.5 for more details.

If the LAYOUTRETURN request sets lora_reclaim to TRUE after the metadata server's grace period, NFS4ERR_NO_GRACE is returned 🙈

If the LAYOUTRETURN request sets lora_reclaim to TRUE and lr_returntype is LAYOUTRETURN4_FSID or LAYOUTRETURN4_ALL, NFS4ERR_INVAL is returned 💀

If the client sets lr_returntype to LAYOUTRETURN4_FILE, then the lrs_stateid field represents the layout stateid as updated for this operation's processing. The current stateid is also updated to match that returned value 🌸 If the last byte of any layout for the current file, client ID, and layout type is being returned, and there are no pending CB_LAYOUTRECALL operations left that need a LAYOUTRETURN, then lrs_present MUST be FALSE, and no stateid comes back. On top of that, the COMPOUND request's current stateid gets set to the all-zeroes special stateid (see Section 16.2.3.1.2). The server MUST reject with NFS4ERR_BAD_STATEID any further use of the current stateid in that COMPOUND until the current stateid is re-established by a later stateid-returning operation 😭

On success, the current filehandle keeps its value 🎀

If the EXCHGID4_FLAG_BIND_PRINC_STATEID capability is set on the client ID (see Section 18.35), the server will require that the principal, security flavor, and (if applicable) the GSS mechanism combination that acquired the layout is also the one that sends LAYOUTRETURN. That might not be possible if the credentials for that principal aren't available anymore 🥺 The server will allow the machine credential or SSV credential (see Section 18.35) to send LAYOUTRETURN if the LAYOUTRETURN operation code was set in the spo_must_allow result of EXCHANGE_ID 👑

18.44.4. IMPLEMENTATION 🧚

The final LAYOUTRETURN operation that responds to a CB_LAYOUTRECALL callback MUST be serialized with any outstanding, intersecting LAYOUTRETURN operations 💜 Here's the tricky bit: while a client is returning the layout for some recalled range, the server might recall a superset of that range (for example, LAYOUTRECALL4_ALL). In that case, the final return for the bigger recall has to block until the smaller one is done 🫶

Returning all layouts in a file system with LAYOUTRETURN4_FSID is typically done in response to a CB_LAYOUTRECALL for that file system, as the final return operation. Likewise, LAYOUTRETURN4_ALL is used in response to a recall callback for all layouts. It's possible the client already gave back some outstanding layouts through individual LAYOUTRETURN calls, and then the call for LAYOUTRETURN4_FSID or LAYOUTRETURN4_ALL marks the end of the LAYOUTRETURN sequence. See Section 12.5.5.1 for more details ✨

Once the client has returned all layouts referring to a particular device ID, the server MAY delete the device ID 🦋

---

18.45. Operation 52: SECINFO_NO_NAME - Get Security on Unnamed Object 🌈

18.45.1. ARGUMENT 💅

   enum secinfo_style4 {
           SECINFO_STYLE4_CURRENT_FH       = 0,
           SECINFO_STYLE4_PARENT           = 1
   };

   /* CURRENT_FH: object or child directory */
   typedef secinfo_style4 SECINFO_NO_NAME4args;

18.45.2. RESULT 🍓

   /* CURRENTFH: consumed if status is NFS4_OK */
   typedef SECINFO4res SECINFO_NO_NAME4res;

18.45.3. DESCRIPTION ✨

okay bestie, SECINFO_NO_NAME is basically SECINFO's cousin 💅 The client uses it to ask which RPC authentication flavors are valid for one specific file object 🦋 The only real difference is that SECINFO_NO_NAME only works on objects that get accessed by filehandle 🎀

There are two styles, picked with the secinfo_style4 enumeration 🌸 Pass SECINFO_STYLE4_CURRENT_FH and you're asking what security is needed for the current filehandle. Pass SECINFO_STYLE4_PARENT and you're asking about the current filehandle's parent instead 👑 For the PARENT style, SECINFO is supposed to use the same access check that LOOKUPP uses when it decides whether you can traverse up to the parent directory. So if you don't have access to LOOKUPP the parent, SECINFO_NO_NAME has to act the same way and hand back NFS4ERR_ACCESS 😭

If PUTFH, PUTPUBFH, PUTROOTFH, or RESTOREFH returns NFS4ERR_WRONGSEC, the client fixes it by sending a COMPOUND request with PUTFH, PUTPUBFH, or PUTROOTFH right before SECINFO_NO_NAME in the SECINFO_STYLE4_CURRENT_FH style 🧚 Check out Section 2.6 for how to handle NFS4ERR_WRONGSEC errors coming back from PUTFH, PUTROOTFH, PUTPUBFH, or RESTOREFH.

If you pick SECINFO_STYLE4_PARENT and there's no parent directory, SECINFO_NO_NAME MUST return NFS4ERR_NOENT. That's a hard rule, bestie 🙈

On success, the current filehandle gets consumed (see Section 2.6.3.1.1.8) 💀 So if the next operation after SECINFO_NO_NAME tries to use the current filehandle, that operation will fail with NFS4ERR_NOFILEHANDLE.

Everything else about SECINFO_NO_NAME is the same as SECINFO, so go read the SECINFO discussion (Section 18.29.3) 💜

18.45.4. IMPLEMENTATION 🫶

Same deal here: check the SECINFO discussion (Section 18.29.4) and you're set 🌈

---

18.46. Operation 53: SEQUENCE - Supply Per-Procedure Sequencing and Control 🔥

18.46.1. ARGUMENT 💖

struct SEQUENCE4args {
        sessionid4     sa_sessionid;
        sequenceid4    sa_sequenceid;
        slotid4        sa_slotid;
        slotid4        sa_highest_slotid;
        bool           sa_cachethis;
};

18.46.2. RESULT 🦋

const SEQ4_STATUS_CB_PATH_DOWN                  = 0x00000001;
const SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING      = 0x00000002;
const SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED       = 0x00000004;
const SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED     = 0x00000008;
const SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED    = 0x00000010;
const SEQ4_STATUS_ADMIN_STATE_REVOKED           = 0x00000020;
const SEQ4_STATUS_RECALLABLE_STATE_REVOKED      = 0x00000040;
const SEQ4_STATUS_LEASE_MOVED                   = 0x00000080;
const SEQ4_STATUS_RESTART_RECLAIM_NEEDED        = 0x00000100;
const SEQ4_STATUS_CB_PATH_DOWN_SESSION          = 0x00000200;
const SEQ4_STATUS_BACKCHANNEL_FAULT             = 0x00000400;
const SEQ4_STATUS_DEVID_CHANGED                 = 0x00000800;
const SEQ4_STATUS_DEVID_DELETED                 = 0x00001000;
struct SEQUENCE4resok {
        sessionid4      sr_sessionid;
        sequenceid4     sr_sequenceid;
        slotid4         sr_slotid;
        slotid4         sr_highest_slotid;
        slotid4         sr_target_highest_slotid;
        uint32_t        sr_status_flags;
};

union SEQUENCE4res switch (nfsstat4 sr_status) {
case NFS4_OK:
        SEQUENCE4resok  sr_resok4;
default:
        void;
};

18.46.3. DESCRIPTION 💅

okay so SEQUENCE is basically the server's bouncer 🧚 it handles session request control and the reply cache semantics, and it's giving total organisation, no cap.

SEQUENCE MUST appear as the first operation of any COMPOUND it shows up in. If it pops up anywhere past the first spot, the server hands back NFS4ERR_SEQUENCE_POS 💀. Anything that isn't SEQUENCE, BIND_CONN_TO_SESSION, EXCHANGE_ID, CREATE_SESSION, or DESTROY_SESSION MUST NOT be the first operation in a COMPOUND, and if one tries to start a COMPOUND anyway, it MUST yield NFS4ERR_OP_NOT_IN_SESSION 🙈

If SEQUENCE arrives on a connection that isn't tied to the session through CREATE_SESSION or BIND_CONN_TO_SESSION, and connection association enforcement is turned on (see Section 18.35), then the server returns NFS4ERR_CONN_NOT_BOUND_TO_SESSION 🔒 rude.

The sa_sessionid argument tells the server which session this request is for 🎀. The sr_sessionid result MUST equal sa_sessionid, so the ID always matches up.

The sa_slotid argument is the index into the reply cache for this request 🍓. The sa_sequenceid field is the sequence number of the request for that reply cache entry (slot). The sr_slotid result MUST equal sa_slotid, and the sr_sequenceid result MUST equal sa_sequenceid. Matching receipts, iconic ✨

The sa_highest_slotid argument is the highest slot ID the client has a request outstanding on, and it could be equal to sa_slotid 🦋 The server gives back two "highest_slotid" values, sr_highest_slotid and sr_target_highest_slotid, and they do different jobs:

• sr_highest_slotid is the highest slot ID the server will accept in a future SEQUENCE operation. It SHOULD NOT be less than the value of sa_highest_slotid (but see Section 2.10.6.1 for an exception) 👑
• sr_target_highest_slotid is the highest slot ID the server would prefer the client use on a future SEQUENCE operation 🌈

If sa_cachethis is TRUE, the client is asking the server to cache the entire reply in the server's reply cache, so the server MUST cache the reply (see Section 2.10.6.1.3) 💜 The server MAY cache the reply if sa_cachethis is FALSE. And if the server does not cache the entire reply, it MUST still record that it executed the request at the specified slot and sequence ID. Never forget the receipt, bestie 🫶

The response to SEQUENCE has a word of status flags (sr_status_flags) that tells the client what's up with its lock state and communication paths 🔥 Heads up: any status bits about lock state MAY be reset when lock state is lost because of a server restart (even if the session is persistent across restarts, since session persistence does not imply lock state persistence) or when a new client instance gets established 😭

Here's the flag tea:

SEQ4_STATUS_CB_PATH_DOWN 🚩 When set, the client has no working backchannel path for any session tied to the client ID, so it has to re-establish one. This bit stays set on all SEQUENCE responses on all sessions tied to the client ID until at least one backchannel is available on any session tied to that client ID. If the client fails to re-establish a backchannel for the client ID, it is subject to having recallable state revoked 💀

SEQ4_STATUS_CB_PATH_DOWN_SESSION 🚩 When set, the session has no working backchannel. There are two reasons this can be set without SEQ4_STATUS_CB_PATH_DOWN. The first is that a callback operation specific to the session (e.g., CB_RECALL_SLOT, see Section 20.8) needs to be sent. The second is that the server did send a callback operation, but the connection was lost before the reply came back. The server cannot be sure whether or not the client received the callback operation, so, per the rules on request retry, the server MUST retry the callback operation over the same session 🔁 This bit is the client's cue that it needs to associate a connection to the session's backchannel. It stays set on all SEQUENCE responses of the session until a connection is associated with the session's backchannel. If the client fails to re-establish a backchannel for the session, it is subject to having recallable state revoked.

SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING ⏳ When set, all GSS contexts or RPCSEC_GSS handles assigned to the session's backchannel will expire within a period equal to the lease time. This bit stays set on all SEQUENCE replies until at least one of the following is true:

• All SSV RPCSEC_GSS handles on the session's backchannel have been destroyed and all non-SSV GSS contexts have expired.
• At least one more SSV RPCSEC_GSS handle has been added to the backchannel.
• The expiration time of at least one non-SSV GSS context of an RPCSEC_GSS handle is beyond the lease period from the current time (relative to the time of when a SEQUENCE response was sent).

SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED 🥀 When set, all non-SSV GSS contexts and all SSV RPCSEC_GSS handles assigned to the session's backchannel have expired or have been destroyed. This bit stays set on all SEQUENCE replies until at least one non-expired non-SSV GSS context for the session's backchannel has been established, or at least one SSV RPCSEC_GSS handle has been assigned to the backchannel.

SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED 😱 When set, the lease has expired, and as a result the server released all of the client's locking state. This status bit stays set on all SEQUENCE replies until the loss of all such locks has been acknowledged by using FREE_STATEID (see Section 18.38), or by establishing a new client instance. To do that, you destroy all sessions (via DESTROY_SESSION), destroy the client ID (via DESTROY_CLIENTID), and then invoke EXCHANGE_ID and CREATE_SESSION to establish a new client ID 🧹

SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED 🫠 When set, some subset of the client's locks have been revoked because the lease period expired and then another client's conflicting LOCK operation came in. This status bit stays set on all SEQUENCE replies until the loss of all such locks has been acknowledged by using FREE_STATEID.

SEQ4_STATUS_ADMIN_STATE_REVOKED 🛑 When set, one or more locks have been revoked without the lease period expiring, due to administrative action. This status bit stays set on all SEQUENCE replies until the loss of all such locks has been acknowledged by using FREE_STATEID.

SEQ4_STATUS_RECALLABLE_STATE_REVOKED 📦 When set, one or more recallable objects have been revoked without the lease period expiring, because the client failed to return them when recalled. This can be a consequence of there being no working backchannel and the client failing to re-establish a backchannel per the SEQ4_STATUS_CB_PATH_DOWN, SEQ4_STATUS_CB_PATH_DOWN_SESSION, or SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED status flags. This status bit stays set on all SEQUENCE replies until the loss of all such locks has been acknowledged by using FREE_STATEID.

SEQ4_STATUS_LEASE_MOVED 🧳 When set, responsibility for lease renewal has been transferred to one or more new servers. This condition continues until the client receives an NFS4ERR_MOVED error and the server receives the subsequent GETATTR for the fs_locations or fs_locations_info attribute for an access to each file system whose lease has moved to a new server. See Section 11.7.7.1.

SEQ4_STATUS_RESTART_RECLAIM_NEEDED 🔄 When set, a server restart means the client must reclaim its locking state. Until the client sends a global RECLAIM_COMPLETE (Section 18.51), every SEQUENCE operation will return SEQ4_STATUS_RESTART_RECLAIM_NEEDED.

SEQ4_STATUS_BACKCHANNEL_FAULT 🚨 The server has hit an unrecoverable fault with the backchannel (e.g., it has lost track of the sequence ID for a slot in the backchannel). The client MUST stop sending more requests on the session's fore channel, wait for all outstanding requests to complete on the fore and back channel, and then destroy the session. Big yikes, but that's the move 😤

SEQ4_STATUS_DEVID_CHANGED 🔀 The client is using device ID notifications, and the server has changed a device ID mapping the client holds. This flag stays present until the client has obtained the new mapping with GETDEVICEINFO.

SEQ4_STATUS_DEVID_DELETED 🗑️ The client is using device ID notifications, and the server has deleted a device ID mapping the client holds. This flag stays in effect until the client sends a GETDEVICEINFO on the device ID with a null value in the argument gdia_notify_types.

Now the big question: how does the sa_sequenceid argument compare to the cached sequence ID on the slot? There are three cases, and it's giving sorting hat energy 🎩

• If the difference between sa_sequenceid and the server's cached sequence ID at the slot ID is two (2) or more, or if sa_sequenceid is less than the cached sequence ID (accounting for wraparound of the unsigned sequence ID value), then the server MUST return NFS4ERR_SEQ_MISORDERED 🙅‍♀️
• If sa_sequenceid and the cached sequence ID are the same, this is a retry, and the server replies with what is recorded in the reply cache. The lease is possibly renewed as described below 🔁
• If sa_sequenceid is one greater (accounting for wraparound) than the cached sequence ID, then this is a new request, and the slot's sequence ID is incremented. The operations subsequent to SEQUENCE, if any, are processed. If there are no other operations, the only other effects are to cache the SEQUENCE reply in the slot, maintain the session's activity, and possibly renew the lease ✨

If the client reuses a slot ID and sequence ID for a completely different request, the server MAY treat the request as if it is a retry of what it has already executed 🫣 The server MAY however detect the client's illegal reuse and return NFS4ERR_SEQ_FALSE_RETRY. Not the vibe.

If SEQUENCE returns an error, then the state of the slot (sequence ID, cached reply) MUST NOT change, and the associated lease MUST NOT be renewed 🛑

If SEQUENCE returns NFS4_OK, then the associated lease MUST be renewed (see Section 8.3), except if SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED is returned in sr_status_flags. Lease renewal, but make it conditional 💜

18.46.4. IMPLEMENTATION 💅

The server MUST maintain a mapping of session ID to client ID, bestie 🫶 It needs this so it can validate any operation that comes after SEQUENCE and takes a stateid as an argument and/or result.

Okay so, if the client has a persistent session, a SEQUENCE that arrives after a server restart might run into requests that were already performed and saved in a persistent reply cache before the restart. In that case SEQUENCE still gets processed successfully, but any request that was never performed and recorded gets rejected with NFS4ERR_DEADSESSION 💀

Depending on which operations in the COMPOUND finished before the restart, those operations also get their replies sent from the server reply cache 🎀 This part is wild ngl. When these operations establish locking state, that state belongs to the previous server instance and the previous client ID, even though the restart logically happened after those operations and wiped that state out 😭 For a partially executed COMPOUND, processing might reach an operation that wasn't processed during the earlier server instance. That makes it a brand new operation that can't be performed on the existing session, so NFS4ERR_DEADSESSION gets returned from that operation 🥀

18.47. Operation 54: SET_SSV - Update SSV for a Client ID ✨

18.47.1. ARGUMENT

struct ssa_digest_input4 {
        SEQUENCE4args sdi_seqargs;
};

struct SET_SSV4args {
        opaque          ssa_ssv<>;
        opaque          ssa_digest<>;
};

18.47.2. RESULT

struct ssr_digest_input4 {
        SEQUENCE4res sdi_seqres;
};

struct SET_SSV4resok {
        opaque          ssr_digest<>;
};

union SET_SSV4res switch (nfsstat4 ssr_status) {
case NFS4_OK:
        SET_SSV4resok   ssr_resok4;
default:
        void;
};

18.47.3. DESCRIPTION

This operation is how you update the SSV for a client ID 🔑 Before SET_SSV is called the first time on a client ID, the SSV is zero. The SSV is the key used for the SSV GSS mechanism (Section 2.10.9).

SET_SSV MUST be preceded by a SEQUENCE operation in the same COMPOUND. It MUST NOT be used if the client did not opt for SP4_SSV state protection when the client ID was created (see Section 18.35). In that case the server returns NFS4ERR_INVAL. 🙅‍♀️

The field ssa_digest is computed as the output of the HMAC (RFC 2104 [11]), using the subkey derived from SSV4_SUBKEY_MIC_I2T and the current SSV as the key (see Section 2.10.9 for a description of subkeys), and an XDR encoded value of data type ssa_digest_input4. The field sdi_seqargs is equal to the arguments of the SEQUENCE operation for the COMPOUND procedure that SET_SSV is within.

The argument ssa_ssv is XORed with the current SSV to produce the new SSV. The argument ssa_ssv SHOULD be generated randomly. 🎲

In the response, ssr_digest is the output of the HMAC using the subkey derived from SSV4_SUBKEY_MIC_T2I and the new SSV as the key, and an XDR encoded value of data type ssr_digest_input4. The field sdi_seqres is equal to the results of the SEQUENCE operation for the COMPOUND procedure that SET_SSV is within.

As noted in Section 18.35, the client and server can maintain multiple concurrent versions of the SSV ✨ The client and server each MUST maintain an internal SSV version number. It gets set to one the first time SET_SSV executes on the server and the client receives the first SET_SSV reply. Every subsequent SET_SSV bumps the internal SSV version number up by one. This version number corresponds to the smpt_ssv_seq, smt_ssv_seq, sspt_ssv_seq, and ssct_ssv_seq fields of the SSV GSS mechanism tokens (see Section 2.10.9).

18.47.4. IMPLEMENTATION

When the server receives ssa_digest, it MUST verify the digest by computing it the same way the client did and comparing it with ssa_digest. If the server gets a different result, that's an error, NFS4ERR_BAD_SESSION_DIGEST. This error might happen because another SET_SSV from the same client ID changed the SSV. If so, the client recovers by sending SET_SSV again with a recomputed digest based on the subkey of the new SSV. If the transport connection drops after the SET_SSV request is sent but before the SET_SSV reply is received, there are special considerations for recovery if the client has no more connections associated with sessions associated with the client ID of the SSV. See Section 18.34.4.

Clients SHOULD NOT send an ssa_ssv that is equal to a previous ssa_ssv, nor equal to a previous or current SSV. That includes an ssa_ssv equal to zero, since the SSV is initialized to zero when the client ID is created. 🚫

Clients SHOULD send SET_SSV with RPCSEC_GSS privacy. Servers MUST support RPCSEC_GSS with privacy for any COMPOUND that has { SEQUENCE, SET_SSV }. 🔐

A client SHOULD NOT send SET_SSV with the SSV GSS mechanism's credential, because the whole purpose of SET_SSV is to seed the SSV from non-SSV credentials. Instead, SET_SSV SHOULD be sent with the credential of a user that is accessing the client ID for the first time (Section 2.10.8.3). However, if the client does send SET_SSV with SSV credentials, the digest protecting the arguments uses the value of the SSV before ssa_ssv is XORed in, and the digest protecting the results uses the value of the SSV after ssa_ssv is XORed in. 🧩

18.48. Operation 55: TEST_STATEID - Test Stateids for Validity ✨

18.48.1. ARGUMENT

struct TEST_STATEID4args {
        stateid4        ts_stateids<>;
};

18.48.2. RESULT

struct TEST_STATEID4resok {
        nfsstat4        tsr_status_codes<>;
};

union TEST_STATEID4res switch (nfsstat4 tsr_status) {
    case NFS4_OK:
        TEST_STATEID4resok tsr_resok4;
    default:
        void;
};

18.48.3. DESCRIPTION

TEST_STATEID is how you check whether a set of stateids is still valid 🕵️‍♀️ You can use it any time, but the client should definitely use it when it gets a hint that one or more of its stateids were invalidated by lock revocation. That happens when SEQUENCE comes back with one of these sr_status_flags set:

• SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED
• SEQ4_STATUS_EXPIRED_ADMIN_STATE_REVOKED
• SEQ4_STATUS_EXPIRED_RECALLABLE_STATE_REVOKED

The client can use TEST_STATEID one or more times to test the validity of its stateids. Each use lets it test a large set of stateids at once. It also avoids problems where earlier stateids in a COMPOUND request interfere with checking the later ones, which is what would happen if each stateid got tested by its own separate operation in the COMPOUND request. 🧠

For each stateid, the server returns the status code that would be returned if that stateid were used in normal operation. Returning that status isn't an error, and it doesn't stop COMPOUND processing. The stateid checks work the same way they would for normal operations, with a few exceptions:

• There is no check for the type of stateid object, which normal use of a stateid would do.
• There is no reference to the current filehandle.
• Special stateids are always considered invalid. They result in the error code NFS4ERR_BAD_STATEID. 🚫

All stateids are interpreted as being associated with the client for the current session. Any possible association with a previous instance of the client (as stale stateids) is not considered. 👀

The valid status values in the returned status_code array are NFS4ERR_OK, NFS4ERR_BAD_STATEID, NFS4ERR_OLD_STATEID, NFS4ERR_EXPIRED, NFS4ERR_ADMIN_REVOKED, and NFS4ERR_DELEG_REVOKED. 🌈

18.48.4. IMPLEMENTATION

For a discussion of stateid structure, lifetime, and validation, see Sections 8.2.2 and 8.2.4. 📚

18.49. Operation 56: WANT_DELEGATION - Request Delegation ✨

18.49.1. ARGUMENT

union deleg_claim4 switch (open_claim_type4 dc_claim) {
/*
 * No special rights to object.  Ordinary delegation
 * request of the specified object.  Object identified
 * by filehandle.
 */
case CLAIM_FH: /* new to v4.1 */
        /* CURRENT_FH: object being delegated */
        void;

/*
 * Right to file based on a delegation granted
 * to a previous boot instance of the client.
 * File is specified by filehandle.
 */
case CLAIM_DELEG_PREV_FH: /* new to v4.1 */
        /* CURRENT_FH: object being delegated */
        void;

/*
 * Right to the file established by an open previous
 * to server reboot.  File identified by filehandle.
 * Used during server reclaim grace period.
 */
case CLAIM_PREVIOUS:
        /* CURRENT_FH: object being reclaimed */
        open_delegation_type4   dc_delegate_type;
};

struct WANT_DELEGATION4args {
        uint32_t        wda_want;
        deleg_claim4    wda_claim;
};

18.49.2. RESULT

union WANT_DELEGATION4res switch (nfsstat4 wdr_status) {
case NFS4_OK:
        open_delegation4 wdr_resok4;
default:
        void;
};

18.49.3. DESCRIPTION 🎀

Okay bestie, so if the description says a specific error code has to come back for a specific condition, and more than one condition applies at once, the server MAY return any of the mandated error codes. 💅

This operation lets a client:

• Get a delegation on all types of files except directories. ✨
• Register a "want" for a delegation on the specified file object, and get notified through a callback when the delegation is available. The server MAY support notifications of availability via callbacks. If the server doesn't support registering wants, it MUST NOT return an error to say so. Instead it MUST return with ond_why set to WND4_CONTENTION or WND4_RESOURCE, and ond_server_will_push_deleg or ond_server_will_signal_avail set to FALSE. When the server says it'll notify the client by callback, it will either hand over the delegation using a CB_PUSH_DELEG operation, or cancel its promise by sending a CB_WANTS_CANCELLED operation. 🦋
• Cancel a want for a delegation. 🫶

The client SHOULD NOT set OPEN4_SHARE_ACCESS_READ and SHOULD NOT set OPEN4_SHARE_ACCESS_WRITE in wda_want. If it does, the server MUST ignore them. 🙈

The meanings of the following flags in wda_want are the same as they are in OPEN, except as noted below.

• OPEN4_SHARE_ACCESS_WANT_READ_DELEG
• OPEN4_SHARE_ACCESS_WANT_WRITE_DELEG
• OPEN4_SHARE_ACCESS_WANT_ANY_DELEG
• OPEN4_SHARE_ACCESS_WANT_NO_DELEG. Unlike the OPEN operation, this flag SHOULD NOT be set by the client in the arguments to WANT_DELEGATION, and MUST be ignored by the server. 🔥
• OPEN4_SHARE_ACCESS_WANT_CANCEL
• OPEN4_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_RESRC_AVAIL
• OPEN4_SHARE_ACCESS_WANT_PUSH_DELEG_WHEN_UNCONTENDED

The handling of the above flags in WANT_DELEGATION is the same as in OPEN. Info about the delegation, and/or the promises the server is making about future callbacks, is the same as what's described in the open_delegation4 structure. 🌸

The successful results of WANT_DELEGATION are of data type open_delegation4, which is the same data type as the "delegation" field in the results of the OPEN operation (see Section 18.16.3). The server constructs wdr_resok4 the same way it constructs OPEN's "delegation", with one difference: WANT_DELEGATION MUST NOT return a delegation type of OPEN_DELEGATE_NONE. 👑

If ((wda_want & OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) & ~OPEN4_SHARE_ACCESS_WANT_NO_DELEG) is zero, then the client is indicating no explicit desire or non-desire for a delegation, and the server MUST return NFS4ERR_INVAL. 💀

The client uses the OPEN4_SHARE_ACCESS_WANT_CANCEL flag in the WANT_DELEGATION operation to cancel a previously requested want for a delegation. Note that if the server is in the process of sending the delegation (via CB_PUSH_DELEG) at the time the client sends a cancellation of the want, the delegation might still be pushed to the client. 🧚

If WANT_DELEGATION fails to return a delegation, and the server returns NFS4_OK, the server MUST set the delegation type to OPEN4_DELEGATE_NONE_EXT, and set od_whynone, as described in Section 18.16. Write delegations are not available for file types that are not writable. This includes file objects of types NF4BLK, NF4CHR, NF4LNK, NF4SOCK, and NF4FIFO. If the client requests OPEN4_SHARE_ACCESS_WANT_WRITE_DELEG without OPEN4_SHARE_ACCESS_WANT_READ_DELEG on an object with one of the aforementioned file types, the server must set wdr_resok4.od_whynone.ond_why to WND4_WRITE_DELEG_NOT_SUPP_FTYPE. 🍓

18.49.4. IMPLEMENTATION 🌈

A request for a conflicting delegation is not normally intended to trigger the recall of the existing delegation. Servers may choose to treat some clients as having higher priority, so that their wants will trigger recall of an existing delegation, although that is expected to be an unusual situation. 🙌

Servers will generally recall delegations assigned by WANT_DELEGATION on the same basis as those assigned by OPEN. CB_RECALL will generally be done only when other clients perform operations inconsistent with the delegation. The normal response to aging of delegations is to use CB_RECALL_ANY, so the client gets the chance to keep the delegations that are most useful from its point of view. 💜

18.50. Operation 57: DESTROY_CLIENTID - Destroy a Client ID 💖

18.50.1. ARGUMENT ✨

struct DESTROY_CLIENTID4args {
        clientid4       dca_clientid;
};

18.50.2. RESULT ✨

struct DESTROY_CLIENTID4res {
        nfsstat4        dcr_status;
};

18.50.3. DESCRIPTION 🔥

The DESTROY_CLIENTID operation destroys the client ID. If there are sessions (both idle and non-idle), opens, locks, delegations, layouts, and/or wants (Section 18.49) associated with the unexpired lease of the client ID, the server MUST return NFS4ERR_CLIENTID_BUSY. DESTROY_CLIENTID MAY be preceded with a SEQUENCE operation, as long as the client ID derived from the session ID of SEQUENCE is not the same as the client ID to be destroyed. If the client IDs are the same, then the server MUST return NFS4ERR_CLIENTID_BUSY. 😭

If DESTROY_CLIENTID is not prefixed by SEQUENCE, it MUST be the only operation in the COMPOUND request (otherwise, the server MUST return NFS4ERR_NOT_ONLY_OP). If the operation is sent without a SEQUENCE preceding it, a client that retransmits the request may receive an error in response, because the original request might have been successfully executed. 🙈

18.50.4. IMPLEMENTATION 🦋

DESTROY_CLIENTID lets a server immediately reclaim the resources consumed by an unused client ID, and also forget that it ever generated the client ID. By forgetting that it ever generated the client ID, the server can safely reuse the client ID on a future EXCHANGE_ID operation. 🧠✨

18.51. Operation 58: RECLAIM_COMPLETE - Indicates Reclaims Finished 🌸

18.51.1. ARGUMENT ✨

struct RECLAIM_COMPLETE4args {
        /*
         * If rca_one_fs TRUE,
         *
         *    CURRENT_FH: object in
         *    file system reclaim is
         *    complete for.
         */
        bool            rca_one_fs;
};

18.51.2. RESULTS ✨

struct RECLAIM_COMPLETE4res {
        nfsstat4        rcr_status;
};

18.51.3. DESCRIPTION 🔥

A RECLAIM_COMPLETE operation is used to indicate that the client has reclaimed all of the locking state it will recover, when it is recovering state due to either a server restart or the transfer of a file system to another server. There are two types of RECLAIM_COMPLETE operations:

• When rca_one_fs is FALSE, a global RECLAIM_COMPLETE is being done. This indicates that recovery of all locks that the client held on the previous server instance has been completed.
• When rca_one_fs is TRUE, a file system-specific RECLAIM_COMPLETE is being done. This indicates that recovery of locks for a single fs (the one designated by the current filehandle) due to a file system transition has been completed. Presence of a current filehandle is only required when rca_one_fs is set to TRUE. 🍓

Once a RECLAIM_COMPLETE is done, there can be no further reclaim operations for locks whose scope is defined as having completed recovery. Once the client sends RECLAIM_COMPLETE, the server will not allow the client to do subsequent reclaims of locking state for that scope, and if these are attempted, it will return NFS4ERR_NO_GRACE. 🚫

Whenever a client establishes a new client ID, and before it does the first non-reclaim operation that obtains a lock, it MUST send a RECLAIM_COMPLETE with rca_one_fs set to FALSE, even if there are no locks to reclaim. If non-reclaim locking operations are done before the RECLAIM_COMPLETE, an NFS4ERR_GRACE error will be returned. 👑

Similarly, when the client accesses a file system on a new server, before it sends the first non-reclaim operation that obtains a lock on this new server, it MUST send a RECLAIM_COMPLETE with rca_one_fs set to TRUE and current filehandle within that file system, even if there are no locks to reclaim. If non-reclaim locking operations are done on that file system before the RECLAIM_COMPLETE, an NFS4ERR_GRACE error will be returned. 💅

Any locks not reclaimed at the point at which RECLAIM_COMPLETE is done become non-reclaimable. The client MUST NOT attempt to reclaim them, either during the current server instance or in any subsequent server instance, or on another server to which responsibility for that file system is transferred. If the client were to do so, it would be violating the protocol by representing itself as owning locks that it does not own, and so it has no right to reclaim. See Section 8.4.3 for a discussion of edge conditions related to lock reclaim. ⚠️

By sending a RECLAIM_COMPLETE, the client indicates readiness to proceed with normal non-reclaim locking operations. The client should be aware that such operations may temporarily result in NFS4ERR_GRACE errors until the server is ready to end its grace period. 🌈

18.51.4. IMPLEMENTATION 🧚

Servers will typically use the information about when reclaim activity is complete to shorten the grace period. When the server maintains in persistent storage a list of clients that might have had locks, it is in a position to use the fact that all such clients have done a RECLAIM_COMPLETE to end the grace period and begin normal operations (i.e., grant requests for new locks) sooner than it otherwise might. ⏰

Latency can be minimized by doing a RECLAIM_COMPLETE as part of the COMPOUND request in which the last lock-reclaiming operation is done. When there are no reclaims to be done, RECLAIM_COMPLETE should be done immediately, so the grace period can end as soon as possible. 💖

RECLAIM_COMPLETE should only be done once for each server instance or occasion of the transition of a file system. If it is done a second time, the error NFS4ERR_COMPLETE_ALREADY will result. Note that because of the session feature's retry protection, retries of COMPOUND requests containing a RECLAIM_COMPLETE operation will not result in this error. 🛡️

When a RECLAIM_COMPLETE is sent, the client effectively acknowledges any locks not yet reclaimed as lost. This allows the server to re-enable the client to recover locks, if the occurrence of edge conditions (as described in Section 8.4.3) had caused the server to disable the client from recovering locks. 🫶

18.52. Operation 10044: ILLEGAL - Illegal Operation 🙃

18.52.1. ARGUMENTS ✨

void;

18.52.2. RESULTS ✨

struct ILLEGAL4res {
        nfsstat4        status;
};

18.52.3. DESCRIPTION 🔥

This operation is a placeholder for encoding a result to handle the case of the client sending an operation code within COMPOUND that is not supported. See the COMPOUND procedure description for more details. 🧩

The status field of ILLEGAL4res MUST be set to NFS4ERR_OP_ILLEGAL. 🚨

18.52.4. IMPLEMENTATION 🧠

A client will probably not send an operation with code OP_ILLEGAL, but if it does, the response will be ILLEGAL4res, just as it would be with any other invalid operation code. Note that if the server gets an illegal operation code that is not OP_ILLEGAL, and if the server checks for legal operation codes during the XDR decode phase, then the ILLEGAL4res would not be returned. 🥹

19. NFSv4.1 Callback Procedures 🎀

Okay bestie, this is where the server gets to call the client back 📞 The procedures used for callbacks are defined in the following sections. Quick heads up: "client" and "server" mean NFS clients and servers here, even though for one callback RPC the roles are literally flipped 🙃 iykyk.

Both procedures, CB_NULL and CB_COMPOUND, MUST be implemented. 💅

19.1. Procedure 0: CB_NULL - No Operation 🌸

19.1.1. ARGUMENTS

void;

19.1.2. RESULTS

void;

19.1.3. DESCRIPTION

CB_NULL is the standard ONC RPC NULL procedure, with the usual void argument and void response. There's no real functionality here, but the server uses it to check that a path exists for RPCs going from the server to the client 🛤️✨

19.1.4. ERRORS

None. 🫶

19.2. Procedure 1: CB_COMPOUND - Compound Operations 🧩

19.2.1. ARGUMENTS

   enum nfs_cb_opnum4 {
           OP_CB_GETATTR           = 3,
           OP_CB_RECALL            = 4,
   /* Callback operations new to NFSv4.1 */
           OP_CB_LAYOUTRECALL      = 5,
           OP_CB_NOTIFY            = 6,
           OP_CB_PUSH_DELEG        = 7,
           OP_CB_RECALL_ANY        = 8,
           OP_CB_RECALLABLE_OBJ_AVAIL = 9,
           OP_CB_RECALL_SLOT       = 10,
           OP_CB_SEQUENCE          = 11,
           OP_CB_WANTS_CANCELLED   = 12,
           OP_CB_NOTIFY_LOCK       = 13,
           OP_CB_NOTIFY_DEVICEID   = 14,

           OP_CB_ILLEGAL           = 10044
   };

   union nfs_cb_argop4 switch (unsigned argop) {
    case OP_CB_GETATTR:
         CB_GETATTR4args           opcbgetattr;
    case OP_CB_RECALL:
         CB_RECALL4args            opcbrecall;
    case OP_CB_LAYOUTRECALL:
         CB_LAYOUTRECALL4args      opcblayoutrecall;
    case OP_CB_NOTIFY:
         CB_NOTIFY4args            opcbnotify;
    case OP_CB_PUSH_DELEG:
         CB_PUSH_DELEG4args        opcbpush_deleg;
    case OP_CB_RECALL_ANY:
         CB_RECALL_ANY4args        opcbrecall_any;
    case OP_CB_RECALLABLE_OBJ_AVAIL:
         CB_RECALLABLE_OBJ_AVAIL4args opcbrecallable_obj_avail;
    case OP_CB_RECALL_SLOT:
         CB_RECALL_SLOT4args       opcbrecall_slot;
    case OP_CB_SEQUENCE:
         CB_SEQUENCE4args          opcbsequence;
    case OP_CB_WANTS_CANCELLED:
         CB_WANTS_CANCELLED4args   opcbwants_cancelled;
    case OP_CB_NOTIFY_LOCK:
         CB_NOTIFY_LOCK4args       opcbnotify_lock;
    case OP_CB_NOTIFY_DEVICEID:
         CB_NOTIFY_DEVICEID4args   opcbnotify_deviceid;
    case OP_CB_ILLEGAL:            void;
   };
   struct CB_COMPOUND4args {
           utf8str_cs      tag;
           uint32_t        minorversion;
           uint32_t        callback_ident;
           nfs_cb_argop4   argarray<>;
   };

19.2.2. RESULTS

   union nfs_cb_resop4 switch (unsigned resop) {
    case OP_CB_GETATTR:    CB_GETATTR4res  opcbgetattr;
    case OP_CB_RECALL:     CB_RECALL4res   opcbrecall;

    /* new NFSv4.1 operations */
    case OP_CB_LAYOUTRECALL:
                           CB_LAYOUTRECALL4res
                                           opcblayoutrecall;

    case OP_CB_NOTIFY:     CB_NOTIFY4res   opcbnotify;

    case OP_CB_PUSH_DELEG: CB_PUSH_DELEG4res
                                           opcbpush_deleg;

    case OP_CB_RECALL_ANY: CB_RECALL_ANY4res
                                           opcbrecall_any;

    case OP_CB_RECALLABLE_OBJ_AVAIL:
                           CB_RECALLABLE_OBJ_AVAIL4res
                                   opcbrecallable_obj_avail;

    case OP_CB_RECALL_SLOT:
                           CB_RECALL_SLOT4res
                                           opcbrecall_slot;

    case OP_CB_SEQUENCE:   CB_SEQUENCE4res opcbsequence;

    case OP_CB_WANTS_CANCELLED:
                           CB_WANTS_CANCELLED4res
                                   opcbwants_cancelled;

    case OP_CB_NOTIFY_LOCK:
                           CB_NOTIFY_LOCK4res
                                           opcbnotify_lock;

    case OP_CB_NOTIFY_DEVICEID:
                           CB_NOTIFY_DEVICEID4res
                                           opcbnotify_deviceid;

    /* Not new operation */
    case OP_CB_ILLEGAL:    CB_ILLEGAL4res  opcbillegal;
   };

   struct CB_COMPOUND4res {
           nfsstat4 status;
           utf8str_cs      tag;
           nfs_cb_resop4   resarray<>;
   };

19.2.3. DESCRIPTION

CB_COMPOUND is the bundler 🎁 It packs one or more callback procedures into a single RPC request. The main callback RPC program has two main procedures, CB_NULL and CB_COMPOUND, and every other operation rides inside CB_COMPOUND as a wrapper.

While CB_COMPOUND is being processed, the client might realize it doesn't have the resources to run some or all of the operations in the sequence. Check out Section 2.10.6.4 for the details 👀

The minorversion field of the arguments MUST be the same as the minorversion of the COMPOUND procedure used to create the client ID and session. For NFSv4.1, minorversion MUST be set to 1. 🔢

The results include a "status" field. This status MUST be equal to the status of the last operation that was executed within CB_COMPOUND. So if an operation hit an error, the "status" value is that same error code, the one returned for the operation that failed 💀

The "tag" field is handled the same way as in the COMPOUND procedure (see Section 16.2.3). 🏷️

Illegal operation codes are handled the same way they are for the COMPOUND procedure. ✋

19.2.4. IMPLEMENTATION

CB_COMPOUND strings individual operations together into one RPC request, and the client works through them in turn. If an operation is executed and its status is NFS4_OK, the next operation runs. This keeps going until there are no operations left, or one of them returns a status other than NFS4_OK. 🚂

19.2.5. ERRORS

Obviously CB_COMPOUND can return every error that each operation on the backchannel can return (see Table 7). BUT if CB_COMPOUND processes zero operations, then the error from COMPOUND has nothing to do with any operation's error. The errors CB_COMPOUND can return when it processes zero operations include:

                         CB_COMPOUND error returns

   +------------------------------+------------------------------------+
   | Error                        | Notes                              |
   +------------------------------+------------------------------------+
   | NFS4ERR_BADCHAR              | The tag argument has a character   |
   |                              | the replier does not support.      |
   | NFS4ERR_BADXDR               |                                    |
   | NFS4ERR_DELAY                |                                    |
   | NFS4ERR_INVAL                | The tag argument is not in UTF-8   |
   |                              | encoding.                          |
   | NFS4ERR_MINOR_VERS_MISMATCH  |                                    |
   | NFS4ERR_SERVERFAULT          |                                    |
   | NFS4ERR_TOO_MANY_OPS         |                                    |
   | NFS4ERR_REP_TOO_BIG          |                                    |
   | NFS4ERR_REP_TOO_BIG_TO_CACHE |                                    |
   | NFS4ERR_REQ_TOO_BIG          |                                    |
   +------------------------------+------------------------------------+

Table 15

---

20. NFSv4.1 Callback Operations 🦋

20.1. Operation 3: CB_GETATTR - Get Attributes 📋

20.1.1. ARGUMENT

   struct CB_GETATTR4args {
           nfs_fh4 fh;
           bitmap4 attr_request;
   };

20.1.2. RESULT

   struct CB_GETATTR4resok {
           fattr4  obj_attributes;
   };

   union CB_GETATTR4res switch (nfsstat4 status) {
    case NFS4_OK:
            CB_GETATTR4resok       resok4;
    default:
            void;
   };

20.1.3. DESCRIPTION

CB_GETATTR is how the server asks for the current modified state of a file that has an OPEN_DELEGATE_WRITE delegation. Only the size and change attributes are guaranteed to be serviced by the client. For the full scoop on how client and server work together around CB_GETATTR, see Section 10.4.3. 🔍

If the filehandle given isn't one the client holds an OPEN_DELEGATE_WRITE delegation for, you get an NFS4ERR_BADHANDLE error back. 🚫

20.1.4. IMPLEMENTATION

The client returns attrmask bits and their values only for the change attribute, plus any attributes it may change (time_modify and size). 📝

20.2. Operation 4: CB_RECALL - Recall a Delegation 🔙

20.2.1. ARGUMENT

   struct CB_RECALL4args {
           stateid4        stateid;
           bool            truncate;
           nfs_fh4         fh;
   };

20.2.2. RESULT

   struct CB_RECALL4res {
           nfsstat4        status;
   };

20.2.3. DESCRIPTION

CB_RECALL kicks off the process of recalling a delegation and returning it to the server 📤

The truncate flag is an optimization for a regular file that's about to be truncated to zero. When it is TRUE, the client is freed from having to propagate modified data for the file to the server, since that data is irrelevant anyway 🗑️

If the handle given isn't one the client holds a delegation for, you get an NFS4ERR_BADHANDLE error. 🙅

If the stateid given isn't one corresponding to an OPEN delegation for the file in the filehandle, NFS4ERR_BAD_STATEID is returned. 🆔

20.2.4. IMPLEMENTATION

The client SHOULD reply to the callback immediately. 🏃‍♀️ Replying doesn't complete the recall, except when the reply's status field is neither NFS4ERR_DELAY nor NFS4_OK. The recall isn't complete until the delegation is returned using a DELEGRETURN operation. 🎬

20.3. Operation 5: CB_LAYOUTRECALL - Recall Layout from Client 🗺️

20.3.1. ARGUMENT

   /*
    * NFSv4.1 callback arguments and results
    */

   enum layoutrecall_type4 {
           LAYOUTRECALL4_FILE = LAYOUT4_RET_REC_FILE,
           LAYOUTRECALL4_FSID = LAYOUT4_RET_REC_FSID,
           LAYOUTRECALL4_ALL  = LAYOUT4_RET_REC_ALL
   };

   struct layoutrecall_file4 {
           nfs_fh4         lor_fh;
           offset4         lor_offset;
           length4         lor_length;
           stateid4        lor_stateid;
   };
   union layoutrecall4 switch(layoutrecall_type4 lor_recalltype) {
   case LAYOUTRECALL4_FILE:
           layoutrecall_file4 lor_layout;
   case LAYOUTRECALL4_FSID:
           fsid4              lor_fsid;
   case LAYOUTRECALL4_ALL:
           void;
   };

   struct CB_LAYOUTRECALL4args {
           layouttype4             clora_type;
           layoutiomode4           clora_iomode;
           bool                    clora_changed;
           layoutrecall4           clora_recall;
   };

20.3.2. RESULT

   struct CB_LAYOUTRECALL4res {
           nfsstat4        clorr_status;
   };

20.3.3. DESCRIPTION 💌

okay so CB_LAYOUTRECALL is how the server tells the client to start giving layouts back 🫶. The client then begins returning them with LAYOUTRETURN. There are three flavors of recall, picked with a layoutrecall_type4 value: one specific layout of one specific file (LAYOUTRECALL4_FILE), everything in one file system ID (LAYOUTRECALL4_FSID), or everything across all file systems (LAYOUTRECALL4_ALL) 💅✨

How the operation behaves depends on which flavor you get. Here's the breakdown 🍓

LAYOUTRECALL4_FILE

so for a layout to match the recall, these fields have to line up with the layout: clora_type, clora_iomode, lor_fh, and the byte-range given by lor_offset and lor_length 🎀 there are two special values to know about. clora_iomode can be LAYOUTIOMODE4_ANY, which is a wild card that matches any iomode the layout was originally returned with. And if lor_length is NFS4_UINT64_MAX, it means the maximum possible file size 🙈 if a matching layout turns up, it MUST be returned using the LAYOUTRETURN operation (see Section 18.44). for example, if clora_iomode is LAYOUTIOMODE4_ANY, lor_offset is zero, and lor_length is NFS4_UINT64_MAX, then the entire layout gets returned 🧚

NFS4ERR_NOMATCHING_LAYOUT only shows up when the client holds no layouts for the file, or when it doesn't have any overlapping layouts for what the recall specified 💀

LAYOUTRECALL4_FSID and LAYOUTRECALL4_ALL

if it's LAYOUTRECALL4_FSID, the fsid tells you which file system's outstanding layouts MUST be returned 🌸 if it's LAYOUTRECALL4_ALL, then every outstanding layout MUST be returned 🔥 these two also say that every storage device ID to storage device address mapping in the affected file system(s) gets recalled too 👑 the matching LAYOUTRETURN, either LAYOUTRETURN4_FSID or LAYOUTRETURN4_ALL, is how the client tells the server it has thrown out those device mappings 🫶 for the "bulk" recall stuff, go check Section 12.5.5.2.1.5 🦋

NFS4ERR_NOMATCHING_LAYOUT here is only returned when the client holds no layouts AND has no valid deviceid mappings 😭

Now here's the other part. The client also changes what it does based on the clora_changed field 🌈 the server uses this field to give extra context about why the layout is being recalled. A FALSE value for clora_changed means nothing about the layout is expected to change, so the client may write modified data to the storage devices involved. That has to happen before the layout goes back via LAYOUTRETURN ✨ a TRUE value means the server IS changing the layout. Examples of why: the metadata server is restriping the file, or a permanent error hit a storage device and the metadata server wants to hand over a new layout for the file 💜 so a TRUE clora_changed means some kind of change is happening, and the client SHOULD NOT write and commit modified data to the storage devices. In that case the client writes and commits data through the metadata server instead 🫶

For how the lor_stateid field in the arguments gets built, see Section 12.5.3. Heads up: the "seqid" field of lor_stateid MUST NOT be zero 🙅‍♀️ for more on that, check Sections 8.2, 12.5.3, and 12.5.5.2 🎀

20.3.4. IMPLEMENTATION 🧚

okay so CB_LAYOUTRECALL is kind of like CB_RECALL (the one for file delegations) in that the client replies to the request BEFORE it actually gives back the layouts through LAYOUTRETURN 💅 the reply comes back right away, but the operation isn't really done (still pending) until every affected layout has been returned to the server with LAYOUTRETURN ⏳

before the client sends a layout back with LAYOUTRETURN, it should wait for the responses from any in-process or in-flight READ, WRITE, or COMMIT operations that use that recalled layout 🌸

if the client is sitting on modified data that's affected by a recalled layout, it has a few options for getting that data to the server 🍓 it can always write through the metadata server. honestly, though, it might not have a choice when clora_changed is TRUE and the server has no new layout to offer 😭 but if clora_changed is FALSE, the client may be able to write the modified data straight to the storage device, and it needs to do that before the layout goes back via LAYOUTRETURN ✨ if the client gets a new layout that covers the byte-range of the modified data, writing to the storage devices is also an option 🦋 just remember that the client must first return the original layout before it can get a new one 🙈

when modified data gets written while the layout is held, the client must use LAYOUTCOMMIT operations at the right time, and LAYOUTCOMMIT has to happen before the LAYOUTRETURN 🫶 if there's a big pile of modified data outstanding, the client can send LAYOUTRETURNs for portions of the recalled layout 🌈 that way the server can keep an eye on how the client is doing and whether it's sticking to the original recall request 👀 but the LAYOUTRETURN that comes last in a sequence MUST specify the full range being recalled (see Section 12.5.5.1 for details) 👑

one more important one 🔥 if a server needs to delete a device ID and there are layouts pointing at that device ID, CB_LAYOUTRECALL MUST be invoked so the client returns all layouts referring to the device ID first, and only then can the server delete the device ID 💅 if the client doesn't return the affected layouts, the server MAY revoke them 💀

---

20.4. Operation 6: CB_NOTIFY - Notify Client of Directory Changes 🔔✨

20.4.1. ARGUMENT 💖

/*
 * Directory notification types.
 */
enum notify_type4 {
        NOTIFY4_CHANGE_CHILD_ATTRS = 0,
        NOTIFY4_CHANGE_DIR_ATTRS = 1,
        NOTIFY4_REMOVE_ENTRY = 2,
        NOTIFY4_ADD_ENTRY = 3,
        NOTIFY4_RENAME_ENTRY = 4,
        NOTIFY4_CHANGE_COOKIE_VERIFIER = 5
};

/* Changed entry information.  */
struct notify_entry4 {
        component4      ne_file;
        fattr4          ne_attrs;
};

/* Previous entry information */
struct prev_entry4 {
        notify_entry4   pe_prev_entry;
        /* what READDIR returned for this entry */
        nfs_cookie4     pe_prev_entry_cookie;
};

struct notify_remove4 {
        notify_entry4   nrm_old_entry;
        nfs_cookie4     nrm_old_entry_cookie;
};

struct notify_add4 {
        /*
         * Information on object
         * possibly renamed over.
         */
        notify_remove4      nad_old_entry<1>;
        notify_entry4       nad_new_entry;
        /* what READDIR would have returned for this entry */
        nfs_cookie4         nad_new_entry_cookie<1>;
        prev_entry4         nad_prev_entry<1>;
        bool                nad_last_entry;
};
struct notify_attr4 {
        notify_entry4   na_changed_entry;
};

struct notify_rename4 {
        notify_remove4  nrn_old_entry;
        notify_add4     nrn_new_entry;
};

struct notify_verifier4 {
        verifier4       nv_old_cookieverf;
        verifier4       nv_new_cookieverf;
};

/*
 * Objects of type notify_<>4 and
 * notify_device_<>4 are encoded in this.
 */
typedef opaque notifylist4<>;

struct notify4 {
        /* composed from notify_type4 or notify_deviceid_type4 */
        bitmap4         notify_mask;
        notifylist4     notify_vals;
};

struct CB_NOTIFY4args {
        stateid4    cna_stateid;
        nfs_fh4     cna_fh;
        notify4     cna_changes<>;
};

20.4.2. RESULT 🎀

struct CB_NOTIFY4res {
        nfsstat4    cnr_status;
};

20.4.3. DESCRIPTION ✨

okay so CB_NOTIFY is the server's way of tapping clients on the shoulder 🫶. It tells them when stuff changes inside a directory they've been delegated 📁. You set up the notifications when the delegation happens, using GET_DIR_DELEGATION, and then they travel over the backchannel 💜. The server only sends them after the original request has been fully processed on its end, and it sends an array of notifications for whatever might have changed in the directory. Each one is a list of pairs made of bitmaps and values 🎀. If you want the full scoop on how NFSv4.1 bitmaps work, go read Section 3.3.7, bestie 🤓.

If the server has more notifications than can fit in one CB_COMPOUND request, it SHOULD send a sequence of serial CB_COMPOUND requests 🧚. The point is to keep the client's view of the directory from getting totally scrambled 🌀. For example, say the server tells you a file named "foo" was added and then that "foo" was removed. The client has to receive those notifications in the same order the operations happened on the server, or it'll be giving confusion 😭.

If the client holding the delegation makes changes in the directory that add or remove files or sub-directories, the server will notify that client about the resulting change(s) 💅. But if that client is only changing attributes or cookie verifiers, the server doesn't need to send notifications to it 🙈. Here's what the server sends for each operation:

NOTIFY4_ADD_ENTRY

The server sends info about the brand new directory entry, along with that entry's cookie ✨. The entry info (data type notify_add4) has the component name and attributes 🍓. The server sends this type when a file is actually being created, when an entry gets added because of a rename across directories (more on that below), and when a hard link is made to an existing file 🔗. If the entry goes at the end of the directory, the server sets nad_last_entry to TRUE 🏁. If the file lands with at least one entry in front of it, the server also returns the previous entry's info (nad_prev_entry, a variable-length array of up to one element, where zero length means there's no previous entry) along with its cookie 👑. This helps clients figure out exactly where the entry belongs in their file name and directory caches 🗺️. If the new entry's cookie is available, it shows up in nad_new_entry_cookie, which is another variable-length array of up to one element 🦋. And if adding the entry knocks out another entry (which can only happen in the rename case) atomically with the addition, that removed entry's info gets reported in nad_old_entry 💀.

NOTIFY4_REMOVE_ENTRY

The server sends info about the directory entry that's getting deleted 🗑️. It also sends the cookie for the deleted entry, so clients can find the cached info for that entry and deal with it 🧹.

NOTIFY4_RENAME_ENTRY

The server sends info about both the old entry and the new entry, including the name and attributes of each one 💖. If the rename ends up deleting an entry (like when a file gets renamed over another one), that gets reported in nrn_new_new_entry.nad_old_entry 😱. This notification is only sent if both entries are in the same directory 🏠. If the rename crosses directories, the server sends a remove notification to one directory and an add notification to the other, as long as both have a directory delegation 🌈.

NOTIFY4_CHANGE_CHILD_ATTRS/NOTIFY4_CHANGE_DIR_ATTRS

The client uses the attribute mask to tell the server which attributes it wants notifications for ✨. You can ask for change notifications on the directory's attributes and on any file's attributes in the directory, using two separate attribute masks 🎀. You can't ask for change attribute notifications on one specific file, though. One attribute mask covers all the files in the directory 🙅‍♀️. When an attribute changes, the server sends back the values of the changed attributes 🔥. Notifications might not make sense for some file system-wide attributes, so it's up to the server to decide which subset it wants to support 🤷‍♀️. The client can also negotiate how often it gets attribute notifications, by letting the server know how often it wants to hear about a change 💜. The server returns the supported notification frequencies, or says that no notification is allowed for directory or child attributes, by setting the dir_notif_delay and dir_entry_notif_delay attributes respectively 🌸.

NOTIFY4_CHANGE_COOKIE_VERIFIER

If the cookie verifier changes while a client is holding a delegation, the server notifies that client 📣. That way the client can invalidate its cookies and re-send a READDIR to get the new set of cookies 🔄.

---

20.5. Operation 7: CB_PUSH_DELEG - Offer Previously Requested Delegation to Client 🎁

20.5.1. ARGUMENT

struct CB_PUSH_DELEG4args {
        nfs_fh4          cpda_fh;
        open_delegation4 cpda_delegation;

};

20.5.2. RESULT

struct CB_PUSH_DELEG4res {
        nfsstat4 cpdr_status;
};

20.5.3. DESCRIPTION

CB_PUSH_DELEG is how the server does two things at once 🎯. First, it tells the client that the delegation it asked for (the one it marked with a want, either through an OPEN or a WANT_DELEGATION operation) is ready. Second, it offers that delegation to the client right then and there 🎉. The client has three options. It can accept by returning NFS4_OK 💅. It can hit pause and delay the decision by returning NFS4ERR_DELAY ⏳. Or it can permanently say no by returning NFS4ERR_REJECT_DELEG 🙅‍♀️. When the client rejects it like that, the want it set up earlier is permanently deleted, and the delegation becomes available for another client to grab 🦋.

20.5.4. IMPLEMENTATION

If the client returns NFS4ERR_DELAY and there's a conflicting delegation request, the server MAY process that other request at the expense of the client that returned NFS4ERR_DELAY 😬. The client's want isn't cancelled, but it MAY get processed behind other delegation requests or registered wants 🕰️.

When a client returns a status other than NFS4_OK, NFS4ERR_DELAY, or NFS4ERR_REJECT_DELAY, the want stays pending 📌. The server may still decide to cancel the want by sending a CB_WANTS_CANCELLED 🧹.

---

20.6. Operation 8: CB_RECALL_ANY - Keep Any N Recallable Objects 🧺

20.6.1. ARGUMENT

const RCA4_TYPE_MASK_RDATA_DLG          = 0;
const RCA4_TYPE_MASK_WDATA_DLG          = 1;
const RCA4_TYPE_MASK_DIR_DLG            = 2;
const RCA4_TYPE_MASK_FILE_LAYOUT        = 3;
const RCA4_TYPE_MASK_BLK_LAYOUT         = 4;
const RCA4_TYPE_MASK_OBJ_LAYOUT_MIN     = 8;
const RCA4_TYPE_MASK_OBJ_LAYOUT_MAX     = 9;
const RCA4_TYPE_MASK_OTHER_LAYOUT_MIN   = 12;
const RCA4_TYPE_MASK_OTHER_LAYOUT_MAX   = 15;
struct  CB_RECALL_ANY4args      {
        uint32_t        craa_objects_to_keep;
        bitmap4         craa_type_mask;
};

20.6.2. RESULT

struct CB_RECALL_ANY4res {
        nfsstat4        crar_status;
};

20.6.3. DESCRIPTION

The server might realise it can't hold all the state for recallable objects, like delegations and layouts, without running out of resources 😭. When that happens, it's not the ideal move, but the server is totally free to recall individual objects to lighten the load 🧹.

Here's the thing about delegations: their whole purpose is to cut out client-server chatter 🦋. So the server can't assume that an object nobody's touched lately is no longer useful. Not seeing any use just means the delegation is quietly keeping potential operations from ever reaching the server, which is the entire point 💅. Layouts are a bit different. A layout's usefulness does show up when storage devices get I/O requests, but nothing requires a storage device to tell the metadata server about past or present use of a layout 🙈. That means the metadata server probably has no clue which layouts would be good candidates to recall when resources are low.

So if you want an effective reclaim scheme for these objects, the server's knowledge of available resources has to decide when objects must be recalled, while the clients pick which actual objects to hand back 🎀.

Server implementations can handle resource allocation differently ✨. One server might share resources across all classes of recallable objects, while another might keep separate pools for layouts and delegations, or split things up even further by type of delegation 🌈.

When a resource pool is over-utilised, the server can send a CB_RECALL_ANY to clients holding recallable objects of the types involved 📣. This lets the server say "keep this many, return the rest." A mask says which types of objects are being limited, and the client picks which of those objects to give back, based on its own sense of what's still useful 🫶.

There are a bunch of bits defined here 🍓. Some of them come with ranges, and it's up to the definition of the storage protocol to spell out how those get used 🧚. Certain ranges are reserved for object-based storage protocols, and others for experimental storage protocols. Any RFC defining a storage protocol needs to specify how particular bits in its range are used 💜. For example, it could map layout attributes (read vs. write, size of area) to a specific bit, or it could define a field in the layout where the server makes the associated bit position available to the client 🔥.

Here's the full breakdown of the type mask bits:

• RCA4_TYPE_MASK_RDATA_DLG
  • The client is to return OPEN_DELEGATE_READ delegations on non-directory file objects.
• RCA4_TYPE_MASK_WDATA_DLG
  • The client is to return OPEN_DELEGATE_WRITE delegations on regular file objects.
• RCA4_TYPE_MASK_DIR_DLG
  • The client is to return directory delegations.
• RCA4_TYPE_MASK_FILE_LAYOUT
  • The client is to return layouts of type LAYOUT4_NFSV4_1_FILES.
• RCA4_TYPE_MASK_BLK_LAYOUT
  • See [41] for a description.
• RCA4_TYPE_MASK_OBJ_LAYOUT_MIN to RCA4_TYPE_MASK_OBJ_LAYOUT_MAX
  • See [40] for a description.
• RCA4_TYPE_MASK_OTHER_LAYOUT_MIN to RCA4_TYPE_MASK_OTHER_LAYOUT_MAX
  • This range is reserved for telling the client to recall layouts of experimental or site-specific layout types (see Section 3.3.13).

If a bit is set in the type mask that matches an undefined type of recallable object, NFS4ERR_INVAL MUST be returned 🚫. But if a bit is set for a defined type of object that the client doesn't support, NFS4ERR_INVAL MUST NOT be returned 🙅‍♀️. Future minor versions of NFSv4 might expand the set of valid type mask bits 🔮.

CB_RECALL_ANY gives a count of objects the client may keep, not a count it must return 💅. That's on purpose, to dodge a sneaky race condition. If a CB_RECALL_ANY with a count of objects to free crossed paths with client-originated operations that return layouts or delegations, the client and server would end up with different ideas about how many to return, and the client could accidentally free too many 😬.

The server can send another CB_RECALL_ANY with a lower count if resource demands call for it, even before it's heard back from the client about a previous CB_RECALL_ANY with the same type mask 🏃‍♀️. Sure, these might arrive in a different order than they were sent, but any reordering of the callback stream is harmless 🫠. The client's job is to bring the size of the recallable object set in line with each CB_RECALL_ANY it receives. Until that's done, the obligation can't be cancelled or changed by any later CB_RECALL_ANY for the same type mask 🔒. So if the server sends two CB_RECALL_ANYs, the result is the same as if only the lower count had been sent, no matter what order they showed up in ✨. Also, a server can't cancel a CB_RECALL_ANY by sending another one with a higher count 🙅‍♀️. And if a CB_RECALL_ANY comes in with a count that's already within the limit, or above a limit the client is still working to reach, that callback does nothing 🤷‍♀️.

Servers are generally free to deny recallable objects when they don't have enough resources 🚪. But heads up: that policy quietly favours objects that already exist over newly requested ones, so resources might not get used as well as they could 😤. To avoid that, servers are strongly advised to start sending CB_RECALL_ANY callbacks at a point somewhat below where they stop handing out new delegations and layouts 📉. That way the client can purge its less-used objects whenever it makes sense, and keep getting fresh resources freed up by object returns 🌸.

20.6.4. IMPLEMENTATION

The client can choose to return any type of object specified by the mask 🧺. If a server wants to limit how many objects of a particular type are in use, it should only put that type in the mask it sends 🎯. If the client doesn't return the requested objects, it's up to the server to deal with that, usually by sending specific recalls (i.e., CB_RECALL operations) to properly limit resource usage 🛠️. The server should give the client enough time to return objects before moving on to specific recalls, and that time should not be less than the lease period ⏰.

20.7. Operation 9: CB_RECALLABLE_OBJ_AVAIL - Signal Resources for Recallable Objects

20.7.1. ARGUMENT

The argument here is just a typedef of the CB_RECALL_ANY argument, so it's giving reuse energy ♻️.

typedef CB_RECALL_ANY4args CB_RECALLABLE_OBJ_AVAIL4args;

20.7.2. RESULT

struct CB_RECALLABLE_OBJ_AVAIL4res {
        nfsstat4        croa_status;
};

20.7.3. DESCRIPTION

CB_RECALLABLE_OBJ_AVAIL is how the server tells the client that it now has resources to grant recallable objects that might've been turned down earlier by OPEN, WANT_DELEGATION, GET_DIR_DELEG, or LAYOUTGET 🎉. It's the glow-up announcement ✨.

The argument craa_objects_to_keep is the total number of recallable objects, of the types listed in the argument type_mask, that the server thinks it can let the client have. That total includes the ones the client already holds 🧮. If a client tries to grab more recallable objects than the server said it could have, it risks getting some recalled 😬.

The server isn't obligated to reserve the gap between what the client currently holds and craa_objects_to_keep 🙅‍♀️. And delaying the reply to CB_RECALLABLE_OBJ_AVAIL doesn't stop the server from using those resources for something else. In fact, if a client replies slowly, the server might decide it has lost capacity to manage recallable objects, and cancel or shrink whatever reservation it's been holding for that client 🐌. So if the client wants more recallable objects, it needs to reply quickly to CB_RECALLABLE_OBJ_AVAIL and then send the operations to acquire them 🏃‍♀️💨.

20.8. Operation 10: CB_RECALL_SLOT - Change Flow Control Limits

20.8.1. ARGUMENT

struct CB_RECALL_SLOT4args {
        slotid4       rsa_target_highest_slotid;
};

20.8.2. RESULT

struct CB_RECALL_SLOT4res {
        nfsstat4   rsr_status;
};

20.8.3. DESCRIPTION

CB_RECALL_SLOT asks the client to return session slots and, where it applies, transport credits (for example, RDMA credits for connections tied to the operations channel) on the session's fore channel 🎀. It carries rsa_target_highest_slotid, which is the target highest slot ID the server wants for the session 🎯. The client MUST then work toward bringing the session's highest slot ID down to that target value 💪.

If the session only has non-RDMA connections tied to its operations channel, the client just needs to wait for all outstanding requests with a slot ID greater than rsa_target_highest_slotid to finish. Then it sends a single COMPOUND containing one SEQUENCE operation, with the sa_highestslot field set to rsa_target_highest_slotid 🧾. If there are RDMA-based connections tied to the operations channel, the client also needs to send enough zero-length "RDMA Send" messages to bring the total RDMA credit count down to rsa_target_highest_slotid + 1 or below 📉.

20.8.4. IMPLEMENTATION

If the client doesn't reduce its highest slot on the fore channel to what the server asked for, the server can force the issue 😤. It does this by asserting flow control on the receive side of all connections bound to the fore channel, and then finishing all outstanding requests sitting in slots greater than rsa_target_highest_slotid. Once that's done, the server can reopen flow control. From then on, any time the client sends a new request on a slot greater than rsa_target_highest_slotid, the server can return NFS4ERR_BADSLOT 🚫.

20.9. Operation 11: CB_SEQUENCE - Supply Backchannel Sequencing and Control

20.9.1. ARGUMENT

struct referring_call4 {
        sequenceid4     rc_sequenceid;
        slotid4         rc_slotid;
};
struct referring_call_list4 {
        sessionid4      rcl_sessionid;
        referring_call4 rcl_referring_calls<>;
};

struct CB_SEQUENCE4args {
        sessionid4           csa_sessionid;
        sequenceid4          csa_sequenceid;
        slotid4              csa_slotid;
        slotid4              csa_highest_slotid;
        bool                 csa_cachethis;
        referring_call_list4 csa_referring_call_lists<>;
};

20.9.2. RESULT

struct CB_SEQUENCE4resok {
        sessionid4         csr_sessionid;
        sequenceid4        csr_sequenceid;
        slotid4            csr_slotid;
        slotid4            csr_highest_slotid;
        slotid4            csr_target_highest_slotid;
};

union CB_SEQUENCE4res switch (nfsstat4 csr_status) {
case NFS4_OK:
        CB_SEQUENCE4resok   csr_resok4;
default:
        void;
};

20.9.3. DESCRIPTION

okay so CB_SEQUENCE is the op that keeps the books for the backchannel on whatever session the request got sent on 📒✨ It carries the session ID the request belongs to, plus the slot ID and sequence ID the server uses for session request control and exactly once semantics 🔥 It also carries the exchanged slot ID maxima, which are how the reply cache size gets adjusted 🧚

In each CB_COMPOUND request, CB_SEQUENCE MUST appear once and MUST be the first operation 💅 If CB_SEQUENCE shows up in any spot after the first one, the error NFS4ERR_SEQUENCE_POS MUST be returned 🙈 And if the first operation is anything other than CB_SEQUENCE, you get NFS4ERR_OP_NOT_IN_SESSION MUST be returned, no exceptions bestie 💀

See Section 18.46.3 for how slots get processed. It's a whole thing, but it's worth the read 📖🦋

If csa_cachethis is TRUE, the server is basically asking the client to save the reply in the callback reply cache 🫶 And the client MUST cache the reply (see Section 2.10.6.1.3). Non-negotiable, periodt 👑

Now the csa_referring_call_lists array. This is the list of COMPOUND requests the client already sent to the server, each one identified by session ID, slot ID, and sequence ID 🌸 Those earlier requests created state, and some operation(s) in the same CB_COMPOUND as that list are pointing back at that state. The session ID is in there because leased state is tied to a client ID, and a single client ID can have multiple sessions 🎀 See Section 2.10.6.3 for the details.

The value of csa_sequenceid, compared with the sequence ID cached on the slot, lands in one of three cases:

• If the difference between csa_sequenceid and the client's cached sequence ID at the slot ID is two (2) or more, or if csa_sequenceid is less than the cached sequence ID (accounting for wraparound of the unsigned sequence ID value), then the client MUST return NFS4ERR_SEQ_MISORDERED. Oof, out of order 😭
• If csa_sequenceid and the cached sequence ID are the same, this is a retry, and the client returns the CB_COMPOUND request's cached reply 🔁✨
• If csa_sequenceid is one greater (accounting for wraparound) than the cached sequence ID, then this is a new request, and the slot's sequence ID gets incremented 🚀 The operations after CB_SEQUENCE, if there are any, get processed. If there are no other operations, the only other effects are caching the CB_SEQUENCE reply in the slot, maintaining the session's activity, and, when the server receives the CB_SEQUENCE reply, renewing the lease of state related to the client ID. Main character energy 👑

If the server reuses a slot ID and sequence ID for a completely different request, the client MAY treat the request as if it's a retry of what it already executed 🦋 The client MAY however detect the server's illegal reuse and return NFS4ERR_SEQ_FALSE_RETRY. Not me being the one who reused the ID 🙅‍♀️

If CB_SEQUENCE returns an error, then the state of the slot (sequence ID, cached reply) MUST NOT change 🔒 For the conditions when NFS4ERR_RETRY_UNCACHED_REP might come back, check Section 2.10.6.1.3.

The client returns two "highest_slotid" values: csr_highest_slotid and csr_target_highest_slotid 🧚‍♀️ The first one is the highest slot ID the client will accept in a future CB_SEQUENCE operation, and it SHOULD NOT be less than the value of csa_highest_slotid (but see Section 2.10.6.1 for an exception). The second one is the highest slot ID the client would prefer the server use on a future CB_SEQUENCE operation 🍓

---

20.10. Operation 12: CB_WANTS_CANCELLED - Cancel Pending Delegation Wants

20.10.1. ARGUMENT

struct CB_WANTS_CANCELLED4args {
        bool cwca_contended_wants_cancelled;
        bool cwca_resourced_wants_cancelled;
};

20.10.2. RESULT

struct CB_WANTS_CANCELLED4res {
        nfsstat4        cwcr_status;
};

20.10.3. DESCRIPTION

CB_WANTS_CANCELLED is how the server tells the client that some or all of the wants it registered for recallable delegations and layouts have been cancelled 📣💜

If cwca_contended_wants_cancelled is TRUE, that means the server will not be pushing any delegations to the client that become available after contention passes 🙅‍♀️

If cwca_resourced_wants_cancelled is TRUE, that means the server will not notify the client when there are resources on the server to grant delegations or layouts 😔

After getting a CB_WANTS_CANCELLED, the client is free to try acquiring the delegations or layouts it was waiting for, and it can possibly re-register its wants 🦋✨

20.10.4. IMPLEMENTATION

So here's the tricky part 🤓 When a client has an OPEN, WANT_DELEGATION, or GET_DIR_DELEGATION request outstanding and a CB_WANTS_CANCELLED gets sent, the server may need to make it clear whether a promise to signal delegation availability happened before the CB_WANTS_CANCELLED (and is therefore covered by it), or after it (in which case it wasn't covered). The server can make that distinction by putting the appropriate requests into the list of referring calls in the associated CB_SEQUENCE 🧩

---

20.11. Operation 13: CB_NOTIFY_LOCK - Notify Client of Possible Lock Availability

20.11.1. ARGUMENT

struct CB_NOTIFY_LOCK4args {
    nfs_fh4     cnla_fh;
    lock_owner4 cnla_lock_owner;
};

20.11.2. RESULT

struct CB_NOTIFY_LOCK4res {
        nfsstat4        cnlr_status;
};

20.11.3. DESCRIPTION

The server can use this op to say that a byte-range lock for the given file and lock-owner might be available 🔓 The client had requested it earlier with an unsuccessful LOCK operation, so this is like a little "psst, try again" ✨

This callback exists to help cut down the latency of blocking locks 🏃‍♀️ The server might notice that a client has been polling for a blocking byte-range lock and that the client may now be able to get it. If the server supports this callback for a given file, it MUST set the OPEN4_RESULT_MAY_NOTIFY_LOCK flag when responding to successful opens for that file 🚩 Supporting it doesn't commit the server to actually using CB_NOTIFY_LOCK, though. The client can just treat the flag as a hint for how often to poll for locks derived from that open 🧠

Now for upgrades 👀 If an OPEN operation results in an upgrade, where the stateid returned has an "other" value matching a stateid already allocated, but with a new "seqid" showing the lock being represented has changed, then the OPEN4_RESULT_MAY_NOTIFY_LOCK flag in the response to that new OPEN decides how things get handled from that point on. And when parallel OPENs happen on the same file and open-owner, the ordering of the "seqid" fields in the returned stateids (taking wraparound into account) is what picks the controlling value of the OPEN4_RESULT_MAY_NOTIFY_LOCK flag 🎯

20.11.4. IMPLEMENTATION

The server MUST NOT grant the byte-range lock to the client unless and until it gets a LOCK operation from the client 🙅‍♀️ And the client that receives this callback can't assume it now has the lock, or that a later LOCK for it will succeed. Not vibes-based locking, bestie 💀

The server is not required to implement this callback, and even if it does, it isn't required to use it in any particular case 🤷‍♀️ So the client must still rely on polling for blocking locks, as described in Section 9.6 🔁

Same energy on the client side: it isn't required to implement this callback, and even if it does, it's still free to ignore it 🫠 So the server MUST NOT assume that the client will act based on the callback 🙈

---

20.12. Operation 14: CB_NOTIFY_DEVICEID - Notify Client of Device ID Changes

20.12.1. ARGUMENT

/*
 * Device notification types.
 */
enum notify_deviceid_type4 {
        NOTIFY_DEVICEID4_CHANGE = 1,
        NOTIFY_DEVICEID4_DELETE = 2
};

/* For NOTIFY4_DEVICEID4_DELETE */
struct notify_deviceid_delete4 {
        layouttype4     ndd_layouttype;
        deviceid4       ndd_deviceid;
};

/* For NOTIFY4_DEVICEID4_CHANGE */
struct notify_deviceid_change4 {
        layouttype4     ndc_layouttype;
        deviceid4       ndc_deviceid;
        bool            ndc_immediate;
};

struct CB_NOTIFY_DEVICEID4args {
        notify4 cnda_changes<>;
};

20.12.2. RESULT

struct CB_NOTIFY_DEVICEID4res {
        nfsstat4        cndr_status;
};

20.12.3. DESCRIPTION

CB_NOTIFY_DEVICEID is how the server pings clients about changes to pNFS device IDs 📬💖 Registering for these notifications is optional, and you do it through GETDEVICEINFO. The notifications go out over the backchannel once the original request has been processed on the server ✨ The server sends an array of notifications, cnda_changes, as a list of pairs of bitmaps and values. For how NFSv4.1 bitmaps work, see Section 3.3.7 🧮

Like CB_NOTIFY (Section 20.4.3), the server might have more notifications than fit in one CB_COMPOUND, so it may need multiple CB_COMPOUNDs 📦 The difference is that serialization isn't a problem here. Unlike directory entries, device IDs can't be re-used after being deleted (Section 12.2.10), so there's no ordering drama 🙌

Every device ID notification has a device ID and a layout type 🎀 The layout type matters because two different layout types can share the same device ID, and that common device ID can have completely different mappings for each layout type. Wild, right? 🤯

The server will send the following notifications:

• NOTIFY_DEVICEID4_CHANGE: A previously provided device-ID-to-device-address mapping has changed, and the client uses GETDEVICEINFO to get the updated mapping 🔄 The notification is encoded in a value of data type notify_deviceid_change4. That type also has a boolean field, ndc_immediate. If it's TRUE, the change will be enforced immediately, so the client might not be able to finish any pending I/O to the device ID 😬 If ndc_immediate is FALSE, then for an indefinite time, the client can complete pending I/O. Once pending I/O is done, the client SHOULD get the new device-ID-to-device-address mappings before sending new I/O requests to the storage devices addressed by the device ID 📥
• NOTIFY4_DEVICEID_DELETE: Deletes a device ID from the mappings 🗑️ This notification MUST NOT be sent if the client has a layout that refers to the device ID 🚫 In other words, if the server is sending a delete device ID notification, one of the following has to be true for layouts associated with the layout type:
  • The client never had a layout referring to that device ID.
  • The client has returned all layouts referring to that device ID.
  • The server has revoked all layouts referring to that device ID.

The notification is encoded in a value of data type notify_deviceid_delete4. And after a server deletes a device ID, it MUST NOT reuse that device ID for the same layout type until the client ID is deleted 🔒✨

---

20.13. Operation 10044: CB_ILLEGAL - Illegal Callback Operation

20.13.1. ARGUMENT

        void;

20.13.2. RESULT

/*
 * CB_ILLEGAL: Response for illegal operation numbers
 */
struct CB_ILLEGAL4res {
        nfsstat4        status;
};

20.13.3. DESCRIPTION

This one is kinda the placeholder operation 🧸 It exists so the client can encode a result when the server sends an operation code within CB_COMPOUND that isn't defined in the NFSv4.1 specification. See Section 19.2.3 for more details 🔍

The status field of CB_ILLEGAL4res MUST be set to NFS4ERR_OP_ILLEGAL 💅

20.13.4. IMPLEMENTATION ✨

okay so a server will probably not send an operation with code OP_CB_ILLEGAL 🙈 but if it does, the response will be CB_ILLEGAL4res, just like it would be with any other invalid operation code. ngl that's pretty consistent 💅 Also, if the client gets an illegal operation code that is not OP_ILLEGAL, and the client checks for legal operation codes during the XDR decode phase, then an instance of data type CB_ILLEGAL4res will not be returned 🫶

21. Security Considerations 🔐

okay bestie, this part is super serious, so get comfy 💜 Historically, NFS's authentication model assumed the entire machine was the NFS client, and the NFS server trusted that client to authenticate the end-user. The server only shared its files with specific clients, identified by the client's source network address. In that world, the AUTH_SYS RPC security flavor just told the NFS server which end-user was using the client. When processing NFS responses, the client made sure the responses came from the same network address and port number the request was sent to. Honestly, that model is easy to build, simple to deploy, and simple to use, but it is unsafe 😭 So NFSv4.1 implementations are REQUIRED to support a security model with end-to-end authentication, where an end-user on a client mutually authenticates (using cryptographic schemes that don't expose passwords or keys in the clear on the network) to a principal on an NFS server ✨ Integrity and privacy of NFS requests and responses are also a big deal here. All of this (end-to-end mutual authentication, integrity, and privacy) is covered in Section 2.2.1.1.1. There are also specific considerations when using Kerberos V5, which are in Section 2.2.1.1.1.2.1.1 🔥

Note that being REQUIRED to implement does not mean REQUIRED to use 👀 AUTH_SYS can still be used by NFSv4.1 clients and servers. But AUTH_SYS is merely an OPTIONAL security flavor in NFSv4.1, so interoperability via AUTH_SYS is not assured 💀

For reasons like less admin overhead, better performance, and/or lower CPU usage, users of NFSv4.1 implementations might want to skip security mechanisms that protect the integrity of each remote procedure call and response. But going without integrity leaves the user open to a man-in-the-middle between the NFS client and server, who could modify the RPC request and/or the response. Implementations are free to offer the option of weaker security mechanisms, but there are three operations in particular that warrant the implementation overriding user choices 🌸

• The first two such operations are SECINFO and SECINFO_NO_NAME. It is RECOMMENDED that the client send both of them protected with a security flavor that has integrity protection, such as RPCSEC_GSS with either the rpc_gss_svc_integrity or rpc_gss_svc_privacy service. Without integrity protection wrapping SECINFO and SECINFO_NO_NAME and their results, a man-in-the-middle could change the results so the client picks a weaker algorithm from the set the server allows, which could make the client and/or server vulnerable to further attacks 😱
• The third operation that SHOULD use integrity protection is any GETATTR for the fs_locations and fs_locations_info attributes, to lower the severity of a man-in-the-middle attack. The attack has two steps. First, the attacker modifies the unprotected results of some operation to return NFS4ERR_MOVED. Second, when the client follows up with a GETATTR for the fs_locations or fs_locations_info attributes, the attacker changes the results so the client moves its traffic to a server the attacker controls 🚨 With integrity protection, this attack is mitigated 🛡️

Compared to previous NFS versions, NFSv4.1 has extra security considerations for pNFS (see Sections 12.9 and 13.12), for locking and session state (see Section 2.10.8.3), and for state recovery during the grace period (see Section 8.4.2.1.1). For locking and session state, if SP4_SSV state protection is being used, Section 2.10.10 has specific security considerations for the NFSv4.1 client and server 🫧

22. IANA Considerations 🦋

this section uses terms that are defined in [55] 📚

22.1. Named Attribute Definitions

IANA created a registry called the "NFSv4 Named Attribute Definitions Registry" 🌈

NFSv4.1 lets a file be associated with zero or more named attributes 💖 The namespace identifiers for these attributes are defined as string names. The protocol doesn't define how the namespace for these file attributes gets assigned. The IANA registry helps with interoperability where common interests exist. Application developers are allowed to define and use attributes as they need to, but they're encouraged to register them with IANA ✨

Registered named attributes are presumed to apply to all minor versions of NFSv4, including ones defined after the registration. If a named attribute is meant to be limited to specific minor versions, that will be clearly stated in the registry's assignment 🍓

All assignments to the registry are made on a First Come First Served basis, per Section 4.1 of [55]. The policy for each assignment is Specification Required, per Section 4.1 of [55] 💅

In theory, under the NFSv4.1 specification, the name of a named attribute can be up to 2^32 - 1 bytes long. But in practice, NFSv4.1 clients and servers won't be able to handle a string that long 😭 IANA should reject any assignment request with a named attribute that exceeds 128 UTF-8 characters. So the IESG has room to set up bases of assignment for Experimental Use and Standards Action, the prefixes "EXPE" and "STDS" are Reserved. The named attribute with a zero-length name is also Reserved 🙈

The prefix "PRIV" is designated for Private Use. A site that wants to use unregistered named attributes without risking a clash with an assignment in IANA's registry should use the prefix "PRIV" in all of its named attributes 🔒

Some NFSv4.1 clients and servers have case-insensitive semantics, so the fifteen additional lower case and mixed case permutations of each of "EXPE", "PRIV", and "STDS" are Reserved (for example, "expe", "expE", "exPe", and so on are Reserved). Similarly, IANA must not allow two assignments that would conflict if both named attributes were converted to a common case 🎀

The registry of named attributes is a list of assignments. Each assignment has three fields:

1. A US-ASCII string name that is the actual name of the attribute. This name must be unique. This string name can be 1 to 128 UTF-8 characters long.
2. A reference to the specification of the named attribute. The reference can consume up to 256 bytes (or more if IANA permits).
3. The point of contact of the registrant. The point of contact can consume up to 256 bytes (or more if IANA permits). 🫶

22.1.1. Initial Registry

There is no initial registry 🫥

22.1.2. Updating Registrations

The registrant is always permitted to update the point of contact field 💜 Any other change will require Expert Review or IESG Approval ⭐

22.2. Device ID Notifications

IANA created a registry called the "NFSv4 Device ID Notifications Registry" 🧚

New notification types could be added to the CB_NOTIFY_DEVICEID operation (see Section 20.12) 👀 This can happen through changes to the operations that register notifications, or by adding new operations to NFSv4. That requires a new minor version of NFSv4, and a Standards Track document from the IETF. Another way to add a notification is to specify a new layout type (see Section 22.4) ✨

So all assignments to the registry are made on a Standards Action basis per Section 4.1 of [55], with Expert Review required 🔥

The registry is a list of assignments, and each assignment has five fields 🌸

1. The name of the notification type. This name must have the prefix "NOTIFY_DEVICEID4_". This name must be unique.
2. The value of the notification. IANA will assign this number, and the registrant's request will use TBD1 instead of an actual value. IANA MUST use a whole number that is no higher than 2^32-1, and should be the next available value. The value assigned must be unique. A Designated Expert must be used to make sure that when the name of the notification type and its value are added to the NFSv4.1 notify_deviceid_type4 enumerated data type in the NFSv4.1 XDR description ([13]), the result is still a valid XDR description 🙌
3. The Standards Track RFC(s) that describe the notification. If the RFC(s) haven't been published yet, the registrant will use RFCTBD2, RFCTBD3, and so on instead of an actual RFC number 📝
4. How the RFC introduces the notification. This is shown by a single US-ASCII value. If the value is N, it means a minor revision to the NFSv4 protocol. If the value is L, it means a new pNFS layout type. Other values can be used with IESG Approval 🍭
5. The minor versions of NFSv4 that are allowed to use the notification. These are numeric values, but IANA won't allocate and assign them. The author of the relevant RFCs, with IESG Approval, assigns these numbers. Each time a new minor version of NFSv4 is approved, a Designated Expert should review the registry to make recommended updates as needed 🌈

22.2.1. Initial Registry

The initial registry is in Table 16. Note that the next available value is zero 🦋

   +-------------------------+-------+---------+-----+----------------+
   | Notification Name       | Value | RFC     | How | Minor Versions |
   +-------------------------+-------+---------+-----+----------------+
   | NOTIFY_DEVICEID4_CHANGE | 1     | RFC5661 | N   | 1              |
   | NOTIFY_DEVICEID4_DELETE | 2     | RFC5661 | N   | 1              |
   +-------------------------+-------+---------+-----+----------------+

           Table 16: Initial Device ID Notification Assignments

22.2.2. Updating Registrations

The update of a registration will require IESG Approval on the advice of a Designated Expert 💖

22.3. Object Recall Types 🎀

okay so IANA made a registry called the "NFSv4 Recallable Object Types Registry" 📋✨

new object types can get added to CB_RECALL_ANY (see Section 20.6) 💅 and there are two ways to do it: either change the operations that add recallable types, or add brand new operations to NFSv4. both of those need a new minor version of NFSv4 🦋 plus a Standards Track document from IETF 👑. the other way to add a recallable object is to specify a new layout type (see Section 22.4) 🧚

all assignments to the registry are made on a Standards Action basis per Section 4.1 of [55], with Expert Review required 🫶

recallable object types are 32-bit unsigned numbers 🔢 there are no Reserved values, and values 12 through 15 inclusive are designated for Private Use 🌈

the registry is a list of assignments, and each assignment has five fields 💜

1. the name of the recallable object type. this name has to start with the prefix "RCA4_TYPE_MASK_" and it has to be unique 🏷️
2. the value of the recallable object type. IANA is the one who assigns this number, so the registrant puts TBD1 in the request instead of a real value. IANA MUST use a whole number that can be no higher than 2^32-1, and should be the next available value. the value has to be unique too. a Designated Expert must make sure that when the name and value get added to the NFSv4 XDR description [13], the result is still a valid XDR description 🧐
3. the Standards Track RFC(s) that describe the recallable object type. if the RFC(s) aren't published yet, the registrant uses RFCTBD2, RFCTBD3, etc. instead of an actual RFC number 📝
4. how the RFC introduces the recallable object type. this is a single US-ASCII value. N means a minor revision to the NFSv4 protocol, and L means a new pNFS layout type. other values can be used with IESG Approval 🌸
5. the minor versions of NFSv4 that are allowed to use the recallable object type. these are numeric values, but IANA won't allocate or assign them. the author of the relevant RFCs with IESG Approval assigns these numbers instead 🙈 every time a new minor version of NFSv4 gets approved, a Designated Expert should review the registry to make recommended updates as needed ✨

22.3.1. Initial Registry 🍓

the initial registry is in Table 17 🎀 note that the next available value is five 💖

+-------------------------------+-------+--------+-----+------------+
| Recallable Object Type Name   | Value | RFC    | How | Minor      |
|                               |       |        |     | Versions   |
+-------------------------------+-------+--------+-----+------------+
| RCA4_TYPE_MASK_RDATA_DLG      | 0     | RFC    | N   | 1          |
|                               |       | 5661   |     |            |
| RCA4_TYPE_MASK_WDATA_DLG      | 1     | RFC    | N   | 1          |
|                               |       | 5661   |     |            |
| RCA4_TYPE_MASK_DIR_DLG        | 2     | RFC    | N   | 1          |
|                               |       | 5661   |     |            |
| RCA4_TYPE_MASK_FILE_LAYOUT    | 3     | RFC    | N   | 1          |
|                               |       | 5661   |     |            |
| RCA4_TYPE_MASK_BLK_LAYOUT     | 4     | RFC    | L   | 1          |
|                               |       | 5661   |     |            |
| RCA4_TYPE_MASK_OBJ_LAYOUT_MIN | 8     | RFC    | L   | 1          |
|                               |       | 5661   |     |            |
| RCA4_TYPE_MASK_OBJ_LAYOUT_MAX | 9     | RFC    | L   | 1          |
|                               |       | 5661   |     |            |
+-------------------------------+-------+--------+-----+------------+

           Table 17: Initial Recallable Object Type Assignments

22.3.2. Updating Registrations 🔄

the update of a registration will require IESG Approval on the advice of a Designated Expert 🧑‍⚖️

22.4. Layout Types 🧩

IANA created a registry called the "pNFS Layout Types Registry" 🗂️

all assignments to the registry are made on a Standards Action basis, with Expert Review required 💅

layout types are 32-bit numbers 🔢 the value zero is Reserved. values in the range 0x80000000 to 0xFFFFFFFF inclusive are designated for Private Use, and IANA will assign numbers from the range 0x00000001 to 0x7FFFFFFF inclusive 🌈

the registry is a list of assignments, each containing five fields 💜

1. the name of the layout type. this name has to start with the prefix "LAYOUT4_" and it has to be unique 🏷️
2. the value of the layout type. IANA assigns this number, so the request from the registrant uses TBD1 instead of an actual value. the assigned value has to be unique. a Designated Expert must make sure that when the name and value get added to the NFSv4.1 layouttype4 enumerated data type in the NFSv4.1 XDR description ([13]), the result is still a valid XDR description 🧐
3. the Standards Track RFC(s) that describe the notification. if the RFC(s) aren't published yet, the registrant uses RFCTBD2, RFCTBD3, etc. instead of an actual RFC number. together, the RFC(s) must adhere to the guidelines listed in Section 22.4.3 📖
4. how the RFC introduces the layout type. this is a single US-ASCII value. N means a minor revision to the NFSv4 protocol, and L means a new pNFS layout type. other values can be used with IESG Approval 🌸
5. the minor versions of NFSv4 that are allowed to use the notification. these are numeric values, but IANA won't allocate or assign them. the author of the relevant RFCs with IESG Approval assigns these numbers 🙈 every time a new minor version of NFSv4 gets approved, a Designated Expert should review the registry to make recommended updates as needed ✨

22.4.1. Initial Registry 🍓

the initial registry is in Table 18 🎀

 +-----------------------+-------+----------+-----+----------------+
 | Layout Type Name      | Value | RFC      | How | Minor Versions |
 +-----------------------+-------+----------+-----+----------------+
 | LAYOUT4_NFSV4_1_FILES | 0x1   | RFC 5661 | N   | 1              |
 | LAYOUT4_OSD2_OBJECTS  | 0x2   | RFC 5664 | L   | 1              |
 | LAYOUT4_BLOCK_VOLUME  | 0x3   | RFC 5663 | L   | 1              |
 +-----------------------+-------+----------+-----+----------------+

                 Table 18: Initial Layout Type Assignments

22.4.2. Updating Registrations 🔄

the update of a registration will require IESG Approval on the advice of a Designated Expert 🧑‍⚖️

22.4.3. Guidelines for Writing Layout Type Specifications 📜

okay bestie, here's the step-by-step for getting a new pNFS layout type accepted as a Standards Track RFC 🫶 the author has to follow these steps:

1. the author devises the new layout specification 💡
2. the new layout type specification MUST, at a minimum:
   • define the contents of the layout-type-specific fields of the following data types:
     • the da_addr_body field of the device_addr4 data type 🧩
     • the loh_body field of the layouthint4 data type 🧩
     • the loc_body field of layout_content4 data type (which in turn is the lo_content field of the layout4 data type) 🧩
     • the lou_body field of the layoutupdate4 data type 🧩
   • describe or define the storage access protocol used to access the storage devices 🗝️
   • describe whether revocation of layouts is supported 🔐
   • at a minimum, describe the methods of recovery from:
     1. failure and restart for client, server, storage device 🔁
     2. lease expiration from the perspective of the active client, server, storage device ⏰
     3. loss of layout state resulting in fencing of client access to storage devices (for an example, see Section 12.7.3) 🚧
   • include an IANA considerations section, which will in turn include:
     • a request to IANA for a new layout type per Section 22.4 📮
     • a list of requests to IANA for any new recallable object types for CB_RECALL_ANY; each entry is to be presented in the form described in Section 22.3 📋
     • a list of requests to IANA for any new notification values for CB_NOTIFY_DEVICEID; each entry is to be presented in the form described in Section 22.2 📋
   • include a security considerations section. this section MUST explain how the NFSv4.1 authentication, authorization, and access-control models are preserved. that is, if a metadata server would restrict a READ or WRITE operation, how would pNFS via the layout similarly restrict a corresponding input or output operation? 🛡️😭
3. the author documents the new layout specification as an Internet-Draft 📝
4. the author submits the Internet-Draft for review through the IETF standards process as defined in "The Internet Standards Process--Revision 3" (BCP 9). the new layout specification will be submitted for eventual publication as a Standards Track RFC 🚀
5. the layout specification progresses through the IETF standards process 🌈

22.5. Path Variable Definitions 🧚‍♀️

this section is all about the IANA stuff for the variable substitution feature for location names, the one described in Section 11.10.3 💖 as it explains there, the variables that get substituted are made of a domain name and a specific name inside that domain, with a colon in between 🫧 there are two sets of IANA considerations here:

1. the list of variable names 🏷️
2. for each variable name, the list of possible values 🔢

so yeah, there will be one registry for the list of variable names, and possibly one registry for listing the values of each variable name 🗂️✨

22.5.1. Path Variables Registry 🗃️

IANA created a registry called the "NFSv4 Path Variables Registry" 🎀

22.5.1.1. Path Variable Values

Okay so variable names are shaped like ${, then a domain name, then a colon (:), then a domain-specific chunk of the name, then } 💅. When the domain is ietf.org, every variable name must be registered with IANA on a Standards Action basis, with Expert Review required 📜. Path variables with registered domain names that are neither part of ietf.org nor equal to it are handed out on a Hierarchical Allocation basis, which means the domain owner runs the show and IANA stays out of it 🙈. The only exception is if the domain owner decides to register a variable name from their domain, and then IANA does it on a First Come First Serve basis.

For registrants who don't have their own domain, IANA will accept requests to register variables with the prefix ${FCFS.ietf.org: on a First Come First Serve basis ✨. Assignments on a First Come First Serve basis don't need Expert Review, unless the registrant also wants IANA to set up a registry for the values of the registered variable 🦋.

The registry is a list of assignments, and each one has three fields 🎀.

1. The name of the variable. The name must start with ${, followed by a registered domain name, followed by :, or it must start with ${FCFS.ietf.org. The name must be no more than 64 UTF-8 characters long. The name must be unique 💖.
2. For assignments made on a Standards Action basis, the Standards Track RFC(s) that describe the variable. If the RFC(s) haven't been published yet, the registrant uses RFCTBD1, RFCTBD2, and so on instead of an actual RFC number. Note that the RFCs don't have to be part of an NFS minor version 🌸. For assignments made on a First Come First Serve basis, there's an explanation of the purpose of the variable (consuming no more than 1024 bytes, or more if IANA permits). A reference to the explanation can be substituted 🧚.
3. The point of contact, including an email address. The point of contact can consume up to 256 bytes (or more if IANA permits). For assignments made on a Standards Action basis, the point of contact is always IESG 👑.

22.5.1.1.1. Initial Registry

The initial registry is in Table 19 ✨.

      +------------------------+----------+------------------+
      | Variable Name          | RFC      | Point of Contact |
      +------------------------+----------+------------------+
      | ${ietf.org:CPU_ARCH}   | RFC 5661 | IESG             |
      | ${ietf.org:OS_TYPE}    | RFC 5661 | IESG             |
      | ${ietf.org:OS_VERSION} | RFC 5661 | IESG             |
      +------------------------+----------+------------------+

              Table 19: Initial List of Path Variables

IANA has created registries for the values of the variable names ${ietf.org:CPU_ARCH} and ${ietf.org:OS_TYPE} 🌈. See Sections 22.5.2 and 22.5.3.

For the values of the variable ${ietf.org:OS_VERSION}, no registry is needed, because the specifics of its values change depending on the value of ${ietf.org:OS_TYPE} 🙃. So values for ${ietf.org:OS_VERSION} are on a Hierarchical Allocation basis and are of no concern to IANA 💅.

22.5.1.1.2. Updating Registrations

Updating an assignment made on a Standards Action basis requires IESG Approval, on the advice of a Designated Expert 📝.

The registrant can always update the point of contact of an assignment made on a First Come First Serve basis 💜. Any other update requires Expert Review 🔥.

22.5.2. Values for the ${ietf.org:CPU_ARCH} Variable

IANA created a registry called the "NFSv4 ${ietf.org:CPU_ARCH} Value Registry" 🍓.

Assignments to this registry are made on a First Come First Serve basis. The zero-length value of ${ietf.org:CPU_ARCH} is Reserved 🔒. Values with a prefix of "PRIV" are designated for Private Use 🦋.

The registry is a list of assignments, and each one has three fields 🎀.

1. A value of the ${ietf.org:CPU_ARCH} variable. The value must be 1 to 32 UTF-8 characters long. The value must be unique ✨.
2. An explanation (consuming no more than 1024 bytes, or more if IANA permits) of what CPU architecture the value denotes. A reference to the explanation can be substituted 🌸.
3. The point of contact, including an email address. The point of contact can consume up to 256 bytes (or more if IANA permits) 💖.

22.5.2.1. Initial Registry

There is no initial registry 🫶.

22.5.2.2. Updating Registrations

The registrant is free to update the assignment, i.e., change the explanation and/or point-of-contact fields 💅.

22.5.3. Values for the ${ietf.org:OS_TYPE} Variable

IANA created a registry called the "NFSv4 ${ietf.org:OS_TYPE} Value Registry" 🧚.

Assignments to this registry are made on a First Come First Serve basis. The zero-length value of ${ietf.org:OS_TYPE} is Reserved 🔒. Values with a prefix of "PRIV" are designated for Private Use 🌈.

The registry is a list of assignments, and each one has three fields 🎀.

1. A value of the ${ietf.org:OS_TYPE} variable. The value must be 1 to 32 UTF-8 characters long. The value must be unique 💜.
2. An explanation (consuming no more than 1024 bytes, or more if IANA permits) of what CPU architecture the value denotes. A reference to the explanation can be substituted 👑.
3. The point of contact, including an email address. The point of contact can consume up to 256 bytes (or more if IANA permits) 💖.

22.5.3.1. Initial Registry

There is no initial registry 🙈.

22.5.3.2. Updating Registrations

The registrant is free to update the assignment, i.e., change the explanation and/or point of contact fields 💅.

23. References

23.1. Normative References

[1] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.

[2] Eisler, M., Ed., "XDR: External Data Representation Standard", STD 67, RFC 4506, May 2006.

[3] Thurlow, R., "RPC: Remote Procedure Call Protocol Specification Version 2", RFC 5531, May 2009.

[4] Eisler, M., Chiu, A., and L. Ling, "RPCSEC_GSS Protocol Specification", RFC 2203, September 1997.

[5] Zhu, L., Jaganathan, K., and S. Hartman, "The Kerberos Version 5 Generic Security Service Application Program Interface (GSS-API) Mechanism Version 2", RFC 4121, July 2005.

[6] The Open Group, "Section 3.191 of Chapter 3 of Base Definitions of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version (www.opengroup.org), ISBN 1931624232", 2004.

[7] Linn, J., "Generic Security Service Application Program Interface Version 2, Update 1", RFC 2743, January 2000.

[8] Talpey, T. and B. Callaghan, "Remote Direct Memory Access Transport for Remote Procedure Call", RFC 5666, January 2010.

[9] Talpey, T. and B. Callaghan, "Network File System (NFS) Direct Data Placement", RFC 5667, January 2010.

[10] Recio, R., Metzler, B., Culley, P., Hilland, J., and D. Garcia, "A Remote Direct Memory Access Protocol Specification", RFC 5040, October 2007.

[11] Krawczyk, H., Bellare, M., and R. Canetti, "HMAC: Keyed-Hashing for Message Authentication", RFC 2104, February 1997.

[12] Eisler, M., "RPCSEC_GSS Version 2", RFC 5403, February 2009.

[13] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "Network File System (NFS) Version 4 Minor Version 1 External Data Representation Standard (XDR) Description", RFC 5662, January 2010.

[14] The Open Group, "Section 3.372 of Chapter 3 of Base Definitions of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version (www.opengroup.org), ISBN 1931624232", 2004.

[15] Eisler, M., "IANA Considerations for Remote Procedure Call (RPC) Network Identifiers and Universal Address Formats", RFC 5665, January 2010.

[16] The Open Group, "Section 'read()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version (www.opengroup.org), ISBN 1931624232", 2004.

[17] The Open Group, "Section 'readdir()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version (www.opengroup.org), ISBN 1931624232", 2004.

[18] The Open Group, "Section 'write()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version (www.opengroup.org), ISBN 1931624232", 2004.

[19] Hoffman, P. and M. Blanchet, "Preparation of Internationalized Strings ("stringprep")", RFC 3454, December 2002.

[20] The Open Group, "Section 'chmod()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version (www.opengroup.org), ISBN 1931624232", 2004.

[21] International Organization for Standardization, "Information Technology - Universal Multiple-octet coded Character Set (UCS) - Part 1: Architecture and Basic Multilingual Plane", ISO Standard 10646-1, May 1993.

[22] Alvestrand, H., "IETF Policy on Character Sets and Languages", BCP 18, RFC 2277, January 1998.

[23] Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep Profile for Internationalized Domain Names (IDN)", RFC 3491, March 2003.

[24] The Open Group, "Section 'fcntl()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version (www.opengroup.org), ISBN 1931624232", 2004.

[25] The Open Group, "Section 'fsync()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version (www.opengroup.org), ISBN 1931624232", 2004.

[26] The Open Group, "Section 'getpwnam()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version (www.opengroup.org), ISBN 1931624232", 2004.

[27] The Open Group, "Section 'unlink()' of System Interfaces of The Open Group Base Specifications Issue 6 IEEE Std 1003.1, 2004 Edition, HTML Version (www.opengroup.org), ISBN 1931624232", 2004.

[28] Schaad, J., Kaliski, B., and R. Housley, "Additional Algorithms and Identifiers for RSA Cryptography for use in the Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile", RFC 4055, June 2005.

[29] National Institute of Standards and Technology, "Cryptographic Algorithm Object Registration", URL http://csrc.nist.gov/groups/ST/crypto_apps_infra/csor/algorithms.html, November 2007.

23.2. Informative References 📚✨

okay bestie, these are the background reads the spec points to for extra context 🧚 nfs history, xdr stuff, storage, the works 💅

• [30] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame, C., Eisler, M., and D. Noveck, "Network File System (NFS) version 4 Protocol", RFC 3530, April 2003.
• [31] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS Version 3 Protocol Specification", RFC 1813, June 1995.
• [32] Eisler, M., "LIPKEY - A Low Infrastructure Public Key Mechanism Using SPKM", RFC 2847, June 2000.
• [33] Eisler, M., "NFS Version 2 and Version 3 Security Issues and the NFS Protocol's Use of RPCSEC_GSS and Kerberos V5", RFC 2623, June 1999.
• [34] Juszczak, C., "Improving the Performance and Correctness of an NFS Server", USENIX Conference Proceedings, June 1990.
• [35] Reynolds, J., Ed., "Assigned Numbers: RFC 1700 is Replaced by an On-line Database", RFC 3232, January 2002.
• [36] Srinivasan, R., "Binding Protocols for ONC RPC Version 2", RFC 1833, August 1995.
• [37] Werme, R., "RPC XID Issues", USENIX Conference Proceedings, February 1996.
• [38] Nowicki, B., "NFS: Network File System Protocol specification", RFC 1094, March 1989.
• [39] Bhide, A., Elnozahy, E., and S. Morgan, "A Highly Available Network Server", USENIX Conference Proceedings, January 1991.
• [40] Halevy, B., Welch, B., and J. Zelenka, "Object-Based Parallel NFS (pNFS) Operations", RFC 5664, January 2010.
• [41] Black, D., Glasgow, J., and S. Fridella, "Parallel NFS (pNFS) Block/Volume Layout", RFC 5663, January 2010.
• [42] Callaghan, B., "WebNFS Client Specification", RFC 2054, October 1996.
• [43] Callaghan, B., "WebNFS Server Specification", RFC 2055, October 1996.
• [44] IESG, "IESG Processing of RFC Errata for the IETF Stream", July 2008.
• [45] Shepler, S., "NFS Version 4 Design Considerations", RFC 2624, June 1999.
• [46] The Open Group, "Protocols for Interworking: XNFS, Version 3W, ISBN 1-85912-184-5", February 1998.
• [47] Floyd, S. and V. Jacobson, "The Synchronization of Periodic Routing Messages", IEEE/ACM Transactions on Networking 2(2), pp. 122-136, April 1994.
• [48] Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and E. Zeidner, "Internet Small Computer Systems Interface (iSCSI)", RFC 3720, April 2004.
• [49] Snively, R., "Fibre Channel Protocol for SCSI, 2nd Version (FCP-2)", ANSI/INCITS 350-2003, Oct 2003.
• [50] Weber, R., "Object-Based Storage Device Commands (OSD)", ANSI/INCITS 400-2004, July 2004, <http://www.t10.org/ftp/t10/drafts/osd/osd-r10.pdf>.
• [51] Carns, P., Ligon III, W., Ross, R., and R. Thakur, "PVFS: A Parallel File System for Linux Clusters.", Proceedings of the 4th Annual Linux Showcase and Conference, 2000.
• [52] The Open Group, "The Open Group Base Specifications Issue 6, IEEE Std 1003.1, 2004 Edition", 2004.
• [53] Callaghan, B., "NFS URL Scheme", RFC 2224, October 1997.
• [54] Chiu, A., Eisler, M., and B. Callaghan, "Security Negotiation for WebNFS", RFC 2755, January 2000.
• [55] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 5226, May 2008.

---

Appendix A. Acknowledgments 🏆💖

this is the shoutout section, and the people in it absolutely earned every single bit of it 🫶

The initial text for the SECINFO extensions were edited by Mike Eisler with contributions from Peng Dai, Sergey Klyushin, and Carl Burnett.

The initial text for the SESSIONS extensions were edited by Tom Talpey, Spencer Shepler, Jon Bauman with contributions from Charles Antonelli, Brent Callaghan, Mike Eisler, John Howard, Chet Juszczak, Trond Myklebust, Dave Noveck, John Scott, Mike Stolarchuk, and Mark Wittle.

Initial text relating to multi-server namespace features, including the concept of referrals, were contributed by Dave Noveck, Carl Burnett, and Charles Fan with contributions from Ted Anderson, Neil Brown, and Jon Haswell.

The initial text for the Directory Delegations support were contributed by Saadia Khan with input from Dave Noveck, Mike Eisler, Carl Burnett, Ted Anderson, and Tom Talpey.

The initial text for the ACL explanations were contributed by Sam Falkner and Lisa Week.

The pNFS work was inspired by the NASD and OSD work done by Garth Gibson. Gary Grider has also been a champion of high-performance parallel I/O. Garth Gibson and Peter Corbett started the pNFS effort with a problem statement document for the IETF that formed the basis for the pNFS work in NFSv4.1.

The initial text for the parallel NFS support was edited by Brent Welch and Garth Goodson. Additional authors for those documents were Benny Halevy, David Black, and Andy Adamson. Additional input came from the informal group that contributed to the construction of the initial pNFS drafts; specific acknowledgment goes to Gary Grider, Peter Corbett, Dave Noveck, Peter Honeyman, and Stephen Fridella.

Fredric Isaman found several errors in draft versions of the ONC RPC XDR description of the NFSv4.1 protocol.

Audrey Van Belleghem provided, in numerous ways, essential co-ordination and management of the process of editing the specification documents.

Richard Jernigan gave feedback on the file layout's striping pattern design.

Several formal inspection teams were formed to review various areas of the protocol. All the inspections found significant errors and room for improvement. NFSv4.1's inspection teams were:

• ACLs, with the following inspectors: Sam Falkner, Bruce Fields, Rahul Iyer, Saadia Khan, Dave Noveck, Lisa Week, Mario Wurzl, and Alan Yoder.
• Sessions, with the following inspectors: William Brown, Tom Doeppner, Robert Gordon, Benny Halevy, Fredric Isaman, Rick Macklem, Trond Myklebust, Dave Noveck, Karen Rochford, John Scott, and Peter Shah.
• Initial pNFS inspection, with the following inspectors: Andy Adamson, David Black, Mike Eisler, Marc Eshel, Sam Falkner, Garth Goodson, Benny Halevy, Rahul Iyer, Trond Myklebust, Spencer Shepler, and Lisa Week.
• Global namespace, with the following inspectors: Mike Eisler, Dan Ellard, Craig Everhart, Fredric Isaman, Trond Myklebust, Dave Noveck, Theresa Raj, Spencer Shepler, Renu Tewari, and Robert Thurlow.
• NFSv4.1 file layout type, with the following inspectors: Andy Adamson, Marc Eshel, Sam Falkner, Garth Goodson, Rahul Iyer, Trond Myklebust, and Lisa Week.
• NFSv4.1 locking and directory delegations, with the following inspectors: Mike Eisler, Pranoop Erasani, Robert Gordon, Saadia Khan, Eric Kustarz, Dave Noveck, Spencer Shepler, and Amy Weaver.
• EXCHANGE_ID and DESTROY_CLIENTID, with the following inspectors: Mike Eisler, Pranoop Erasani, Robert Gordon, Benny Halevy, Fredric Isaman, Saadia Khan, Ricardo Labiaga, Rick Macklem, Trond Myklebust, Spencer Shepler, and Brent Welch.
• Final pNFS inspection, with the following inspectors: Andy Adamson, Mike Eisler, Mark Eshel, Sam Falkner, Jason Glasgow, Garth Goodson, Robert Gordon, Benny Halevy, Dean Hildebrand, Rahul Iyer, Suchit Kaura, Trond Myklebust, Anatoly Pinchuk, Spencer Shepler, Renu Tewari, Lisa Week, and Brent Welch.

A review team worked together to generate the tables of assignments of error sets to operations and make sure that each such assignment had two or more people validating it. Participating in the process were Andy Adamson, Mike Eisler, Sam Falkner, Garth Goodson, Robert Gordon, Trond Myklebust, Dave Noveck, Spencer Shepler, Tom Talpey, Amy Weaver, and Lisa Week.

Jari Arkko, David Black, Scott Bradner, Lisa Dusseault, Lars Eggert, Chris Newman, and Tim Polk provided valuable review and guidance.

Olga Kornievskaia found several errors in the SSV specification.

Ricardo Labiaga found several places where the use of RPCSEC_GSS was underspecified.

Those who provided miscellaneous comments include: Andy Adamson, Sunil Bhargo, Alex Burlyga, Pranoop Erasani, Bruce Fields, Vadim Finkelstein, Jason Goldschmidt, Vijay K. Gurbani, Sergey Klyushin, Ricardo Labiaga, James Lentini, Anshul Madan, Daniel Muntz, Daniel Picken, Archana Ramani, Jim Rees, Mahesh Siddheshwar, Tom Talpey, and Peter Varga.

---

Authors' Addresses 💌

these are the humans who wrote it, so you know who to thank (or DM) 🦋

Spencer Shepler (editor) Storspeed, Inc. 7808 Moonflower Drive Austin, TX 78750 USA

Phone: +1-512-402-5811 ext 8530 EMail: shepler@storspeed.com

Mike Eisler (editor) NetApp 5765 Chase Point Circle Colorado Springs, CO 80919 USA

Phone: +1-719-599-9026 EMail: mike@eisler.com URI: http://www.eisler.com

David Noveck (editor) NetApp 1601 Trapelo Road, Suite 16 Waltham, MA 02451 USA

Phone: +1-781-768-5347 EMail: dnoveck@netapp.com