RFC 7530
Network File System (NFS) Version 4 Protocol
Network File System (NFS) Version 4 Protocol 🎀
okay bestie, here's the paperwork so you know exactly who's who 💅
- Internet Engineering Task Force (IETF) 🌸
- RFC: 7530 ✨
- Title: Network File System (NFS) Version 4 Protocol
- Authors: T. Haynes, Ed. (Primary Data) and D. Noveck, Ed. (Dell) 🫶
- Category: Standards Track 👑
- ISSN: 2070-1721
- Date: March 2015 🍓
- Obsoletes: RFC 3530 💀
Abstract ✨
okay so NFS version 4 is a distributed file system protocol, and honestly it's the glow-up of NFS version 2 (RFC 1094) and version 3 (RFC 1813) 🦋. Unlike the earlier versions, NFSv4 still does regular file access, but it also bakes in file locking and the MOUNT protocol, which is main character energy 🔥. On top of that it adds strong security (plus a way to negotiate it), COMPOUND operations, client caching, and internationalization 🌈. And they clearly cared about it running smoothly on the Internet too 🌐.
this document, together with the companion External Data Representation (XDR) description document, RFC 7531, obsoletes RFC 3530 as the definition of the NFS version 4 protocol. so RFC 7531 is the bestie that goes along with it 🧚
Status of This Memo 🌷
this is an Internet Standards Track document 💖.
it's a product of the Internet Engineering Task Force (IETF), so it represents the consensus of the IETF community. it went through public review and got approved for publication by the Internet Engineering Steering Group (IESG) ✨. if you want more info on Internet Standards, check out Section 2 of RFC 5741 📚.
for the current status of this document, any errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc7530 🎀.
Copyright Notice 🦋
Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved 💜.
this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the date this document was published. please read these carefully, because they spell out your rights and restrictions for this document 🙈. any Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License 🫶.
this document may contain material from IETF Documents or IETF Contributions that were published or made publicly available before November 10, 2008 🌸. the people who control the copyright in some of that material may not have given the IETF Trust the right to allow modifications outside the IETF Standards Process. so unless you get an adequate license from whoever controls that copyright, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process 😭. the one exception is formatting it for publication as an RFC, or translating it into languages other than English 🌈.
Table of Contents 📖
1. Introduction ....................................................8
1.1. Requirements Language ......................................8
1.2. NFS Version 4 Goals ........................................8
1.3. Definitions in the Companion Document RFC 7531 Are
Authoritative ..............................................9
1.4. Overview of NFSv4 Features .................................9
1.4.1. RPC and Security ....................................9
1.4.2. Procedure and Operation Structure ..................10
1.4.3. File System Model ..................................10
1.4.4. OPEN and CLOSE .....................................12
1.4.5. File Locking .......................................12
1.4.6. Client Caching and Delegation ......................13
1.5. General Definitions .......................................14
1.6. Changes since RFC 3530 ....................................16
1.7. Changes between RFC 3010 and RFC 3530 .....................16
2. Protocol Data Types ............................................18
2.1. Basic Data Types ..........................................18
2.2. Structured Data Types .....................................21
3. RPC and Security Flavor ........................................25
3.1. Ports and Transports ......................................25
3.1.1. Client Retransmission Behavior .....................26
3.2. Security Flavors ..........................................27
3.2.1. Security Mechanisms for NFSv4 ......................27
3.3. Security Negotiation ......................................28
3.3.1. SECINFO ............................................29
3.3.2. Security Error .....................................29
3.3.3. Callback RPC Authentication ........................29
4. Filehandles ....................................................30
4.1. Obtaining the First Filehandle ............................30
4.1.1. Root Filehandle ....................................31
4.1.2. Public Filehandle ..................................31
4.2. Filehandle Types ..........................................31
4.2.1. General Properties of a Filehandle .................32
4.2.2. Persistent Filehandle ..............................32
4.2.3. Volatile Filehandle ................................33
4.2.4. One Method of Constructing a Volatile Filehandle ...34
4.3. Client Recovery from Filehandle Expiration ................35
5. Attributes .....................................................35
5.1. REQUIRED Attributes .......................................37
5.2. RECOMMENDED Attributes ....................................37
5.3. Named Attributes ..........................................37
5.4. Classification of Attributes ..............................39
5.5. Set-Only and Get-Only Attributes ..........................40
5.6. REQUIRED Attributes - List and Definition References ......40
5.7. RECOMMENDED Attributes - List and Definition References ...41
5.8. Attribute Definitions .....................................42
5.8.1. Definitions of REQUIRED Attributes .................42
5.8.2. Definitions of Uncategorized RECOMMENDED
Attributes .........................................45
5.9. Interpreting owner and owner_group ........................51
5.10. Character Case Attributes ................................53
6. Access Control Attributes ......................................54
6.1. Goals .....................................................54
6.2. File Attributes Discussion ................................55
6.2.1. Attribute 12: acl ..................................55
6.2.2. Attribute 33: mode .................................70
6.3. Common Methods ............................................71
6.3.1. Interpreting an ACL ................................71
6.3.2. Computing a mode Attribute from an ACL .............72
6.4. Requirements ..............................................73
6.4.1. Setting the mode and/or ACL Attributes .............74
6.4.2. Retrieving the mode and/or ACL Attributes ..........75
6.4.3. Creating New Objects ...............................75okay the table of contents keeps going and it's giving organized chaos 💅✨
7. NFS Server Namespace ...........................................77
7.1. Server Exports ............................................77
7.2. Browsing Exports ..........................................77
7.3. Server Pseudo-File System .................................78
7.4. Multiple Roots ............................................79
7.5. Filehandle Volatility .....................................79
7.6. Exported Root .............................................79
7.7. Mount Point Crossing ......................................79
7.8. Security Policy and Namespace Presentation ................80
8. Multi-Server Namespace .........................................81
8.1. Location Attributes .......................................81
8.2. File System Presence or Absence ...........................81
8.3. Getting Attributes for an Absent File System ..............83
8.3.1. GETATTR within an Absent File System ...............83
8.3.2. READDIR and Absent File Systems ....................84
8.4. Uses of Location Information ..............................84
8.4.1. File System Replication ............................85
8.4.2. File System Migration ..............................86
8.4.3. Referrals ..........................................86
8.5. Location Entries and Server Identity ......................87
8.6. Additional Client-Side Considerations .....................88
8.7. Effecting File System Referrals ...........................89
8.7.1. Referral Example (LOOKUP) ..........................89
8.7.2. Referral Example (READDIR) .........................93
8.8. The Attribute fs_locations ................................96
9. File Locking and Share Reservations ............................98
9.1. Opens and Byte-Range Locks ................................99
9.1.1. Client ID ..........................................99
9.1.2. Server Release of Client ID .......................102
9.1.3. Use of Seqids .....................................103
9.1.4. Stateid Definition ................................104
9.1.5. Lock-Owner ........................................110
9.1.6. Use of the Stateid and Locking ....................110
9.1.7. Sequencing of Lock Requests .......................113
9.1.8. Recovery from Replayed Requests ...................114
9.1.9. Interactions of Multiple Sequence Values ..........114
9.1.10. Releasing State-Owner State ......................115
9.1.11. Use of Open Confirmation .........................116
9.2. Lock Ranges ..............................................117
9.3. Upgrading and Downgrading Locks ..........................117
9.4. Blocking Locks ...........................................118
9.5. Lease Renewal ............................................119
9.6. Crash Recovery ...........................................120
9.6.1. Client Failure and Recovery .......................120
9.6.2. Server Failure and Recovery .......................120
9.6.3. Network Partitions and Recovery ...................122
9.7. Recovery from a Lock Request Timeout or Abort ............130
9.8. Server Revocation of Locks ...............................130
9.9. Share Reservations .......................................132
9.10. OPEN/CLOSE Operations ...................................132
9.10.1. Close and Retention of State Information .........133
9.11. Open Upgrade and Downgrade ..............................134
9.12. Short and Long Leases ...................................135
9.13. Clocks, Propagation Delay, and Calculating Lease
Expiration ..............................................135
9.14. Migration, Replication, and State .......................136
9.14.1. Migration and State ..............................136
9.14.2. Replication and State ............................137
9.14.3. Notification of Migrated Lease ...................137
9.14.4. Migration and the lease_time Attribute ...........138
10. Client-Side Caching ..........................................139
10.1. Performance Challenges for Client-Side Caching ..........139
10.2. Delegation and Callbacks ................................140
10.2.1. Delegation Recovery ..............................142
10.3. Data Caching ............................................147
10.3.1. Data Caching and OPENs ...........................147
10.3.2. Data Caching and File Locking ....................148
10.3.3. Data Caching and Mandatory File Locking ..........150
10.3.4. Data Caching and File Identity ...................150
10.4. Open Delegation .........................................151
10.4.1. Open Delegation and Data Caching .................154
10.4.2. Open Delegation and File Locks ...................155
10.4.3. Handling of CB_GETATTR ...........................155
10.4.4. Recall of Open Delegation ........................158
10.4.5. OPEN Delegation Race with CB_RECALL ..............160
10.4.6. Clients That Fail to Honor Delegation Recalls ....161
10.4.7. Delegation Revocation ............................162
10.5. Data Caching and Revocation .............................162
10.5.1. Revocation Recovery for Write Open Delegation ....163
10.6. Attribute Caching .......................................164
10.7. Data and Metadata Caching and Memory-Mapped Files .......166
10.8. Name Caching ............................................168
10.9. Directory Caching .......................................169
11. Minor Versioning .............................................170
12. Internationalization .........................................170
12.1. Introduction ............................................170
12.2. Limitations on Internationalization-Related
Processing in the NFSv4 Context .........................172
12.3. Summary of Server Behavior Types ........................173
12.4. String Encoding .........................................173
12.5. Normalization ...........................................174
12.6. Types with Processing Defined by Other Internet Areas ...175
12.7. Errors Related to UTF-8 .................................177
12.8. Servers That Accept File Component Names That
Are Not Valid UTF-8 Strings .............................1771. Introduction ✨
1.1. Requirements Language
okay bestie, first things first 💅 the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119], except where "REQUIRED" and "RECOMMENDED" are used as qualifiers to distinguish classes of attributes, as described in Sections 1.4.3.2 and 5 of this document 🎀
1.2. NFS Version 4 Goals
so the Network File System version 4 (NFSv4) protocol is a further revision of the NFS protocol, which was already defined by versions 2 [RFC1094] and 3 [RFC1813] 🌸 it keeps the essential stuff from earlier versions: easy recovery, independence from transport protocols, operating systems, and file systems, simplicity, and good performance ✨ the NFSv4 revision has these goals:
- Improved access and good performance on the Internet. 🦋
The protocol is designed to transit firewalls easily, perform well where latency is high and bandwidth is low, and scale to very large numbers of clients per server. it's giving internet-ready 🌈
- Strong security with negotiation built into the protocol. 🔐
The protocol builds on the work of the Open Network Computing (ONC) Remote Procedure Call (RPC) working group in supporting the RPCSEC_GSS protocol (see both [RFC2203] and [RFC5403]). Additionally, the NFSv4 protocol provides a mechanism that lets clients and servers negotiate security, and it requires clients and servers to support a minimal set of security schemes 👑
- Good cross-platform interoperability. 🧚
The protocol features a file system model that provides a useful, common set of features. it doesn't unduly favor one file system or operating system over another, which is so fair 💜
- Designed for protocol extensions. 🍓
The protocol is designed to accept standard extensions that do not compromise backward compatibility. future-proof era, bestie 🫶
This document, together with the companion External Data Representation (XDR) description document [RFC7531], obsoletes [RFC3530] as the authoritative document describing NFSv4 💖 it does not introduce any over-the-wire protocol changes, in the sense that previously valid requests remain valid. so your old requests are still iconic and still valid ✨
1.3. Definitions in the Companion Document RFC 7531 Are Authoritative
the "Network File System (NFS) Version 4 External Data Representation Standard (XDR) Description" [RFC7531] has the definitions, written in XDR description language, of the constructs the protocol uses 📜 this document reproduces several of those constructs just to explain things, but the reader is warned that the reproduced constructs might contain errors outside of [RFC7531] 🙈
heads up, bestie: if any part of this document is inconsistent with [RFC7531], then [RFC7531] is the one to trust. no cap, it wins 👑
1.4. Overview of NFSv4 Features
okay so before we dive in, let's do a quick sweep of the major NFSv4 features 🌷 this gives context for people who already know the earlier NFS versions and for people who are brand new to NFS. if you're new, some fundamentals are still expected: you should be familiar with XDR and RPC as described in [RFC4506] and [RFC5531], and a basic knowledge of file systems and distributed file systems is expected too 🧠
1.4.1. RPC and Security
as with earlier versions of NFS, the XDR and RPC mechanisms used by NFSv4 are the ones defined in [RFC4506] and [RFC5531] 🎀 to meet end-to-end security requirements, the RPCSEC_GSS framework (version 1 in [RFC2203] and version 2 in [RFC5403]) is used to extend the basic RPC security 🔐 with RPCSEC_GSS, various mechanisms can offer authentication, integrity, and privacy to the NFSv4 protocol 🦋 Kerberos V5 is used as described in [RFC4121] to provide one security framework, and with RPCSEC_GSS, other mechanisms may also be specified and used for NFSv4 security 🌈
to enable in-band security negotiation, NFSv4 added a new operation that gives the client a way to ask the server about its policies on which security mechanisms must be used to access the server's file system resources 💅 with that, the client can securely match a security mechanism that meets the policies set at both the client and the server. it's giving perfectly matched energy 💖
1.4.2. Procedure and Operation Structure ✨
okay so the big glow-up in NFSv4 is the COMPOUND procedure 💅 earlier versions were all about separate calls for each thing, but NFSv4 only has two RPC procedures, NULL and COMPOUND. The real action happens inside COMPOUND, where the operations line up much more closely with the old traditional NFS procedures. it's giving streamlined 🦋
With COMPOUND, the client can build requests that are simple or super complex 🧚 That means fewer RPCs for logical file system operations. For example, even without talking to the server first, a client can read data from a file in one request by stringing together LOOKUP, OPEN, and READ in a single COMPOUND RPC. Old versions couldn't do that kind of single request at all 😭
The COMPOUND model is very simple, bestie. There's no logical OR or AND between operations. The server evaluates the operations in the order they were sent. Once one operation returns a failing result, evaluation stops, and the results of every operation that was evaluated get sent back to the client 🌸
NFSv4 still has the client refer to a file or directory on the server with a "filehandle" 🎀 COMPOUND has a way to pass a filehandle from one operation to the next inside the sequence. There's a current filehandle and a saved filehandle. Most operations act on the current filehandle as the file system object they work on. The saved filehandle is temporary storage inside a COMPOUND, and it's also an extra operand for certain operations. main character energy for filehandles, ngl ✨
1.4.3. File System Model
The general file system model for NFSv4 is the same as before 🍓 The server's file system is hierarchical, and the regular files inside it are treated as opaque byte streams. The slight twist is that file and directory names are encoded with UTF-8, so the basics of internationalization are covered 🌈
NFSv4 doesn't need a separate protocol for the initial mapping between a pathname and a filehandle. The older MOUNT protocol is out, and the server gives you a root filehandle instead. That root represents the logical root, or top, of the file system tree the server provides 👑 If the server offers multiple file systems, it glues them together with pseudo-file systems. Those pseudo-file systems cover any gaps in the pathnames between the real file systems 🧚
1.4.3.1. Filehandle Types
In earlier NFS versions, the server's filehandle was guaranteed to stay valid, or persistent, for as long as the file system object it referred to existed. For some server implementations, that was really hard to pull off 💀 So NFSv4 relaxed the requirement by adding another filehandle type called volatile 🙈 With persistent and volatile filehandle types, the server can match what its file system and operating environment can actually do. The client knows which type the server is handing out, so it can be ready to deal with the semantics of each one 🫶
1.4.3.2. Attribute Types
NFSv4 has a rich, extensible attribute structure for file objects. It's split into REQUIRED, RECOMMENDED, and named attributes (see Section 5) 💜
A bunch of the REQUIRED attributes (though not all of them) come from NFSv3. You can check the definition of the fattr3 data type in [RFC1813] if you're curious 🔥 An example of a REQUIRED attribute is the file object's type (Section 5.8.1.2), which lets regular files be told apart from directories (also known as folders in some operating environments) and other kinds of objects. REQUIRED attributes are covered in Section 5.1 ✨
An example of a RECOMMENDED attribute is the acl (Section 6.2.1). It defines an Access Control List (ACL) on a file object, and it's basically a glow-up on the access control model from NFSv3 💅 The ACL lets you specify particular sets of permissions for individual users and groups. ACL inheritance also lets permissions and restrictions propagate down a directory tree as new file system objects get created 🌳 RECOMMENDED attributes are discussed in Section 5.2 🎀
A named attribute is an opaque byte stream linked to a directory or file and referred to by a string name. The point is for client applications to attach their own application-specific data to a regular file or directory 🦋 NFSv4.1 tightens things up compared to NFSv4.0 by restricting which operations are allowed on named attributes, so people don't end up building implementations that don't play nicely together 🙅♀️ Named attributes are covered in Section 5.3 💖
1.4.3.3. Multi-Server Namespace
A single-server namespace is the file system hierarchy the server shows for remote access. It's a proper subset of all the file systems available locally 🧚 NFSv4 has a bunch of features that let namespaces cross server boundaries, and that make it easier to move support for individual file systems between servers without disrupting anything ✨ All of these are built on attributes that let one file system point to alternative or new locations for itself. So just as a client can move across local file systems on one server, it can now move over to a remote file system on a different server 🌈
These attributes can be combined with absent file systems, which give you specifications for other locations but no actual file system content. That opens up some important facilities:
- Location attributes can go with absent file systems to set up referrals, where one server points the client to a file system provided by another server. That's how you build big multi-server namespaces 🗺️
- Location attributes can be provided for present file systems to list alternative file system instances or replicas. Those get used if the current instance becomes unavailable 🛟
- Location attributes can be provided when a file system that used to be present becomes absent. That's what makes non-disruptive migration of file systems to alternative servers possible 🚚
1.4.4. OPEN and CLOSE
NFSv4 introduces the OPEN and CLOSE operations 🔑 OPEN is the one-stop shop where file lookup, creation, and share semantics (see Section 9.9) all come together. CLOSE handles releasing the state that OPEN accumulated 💅
1.4.5. File Locking
In NFSv4, byte-range file locking is part of the NFS protocol itself 🔒 The locking support is set up so that an RPC callback mechanism isn't required. That's a big departure from the earlier NFS file locking protocol, Network Lock Manager (NLM) [RFC1813] 🙈 The state tied to file locks is kept at the server under a lease-based model. The server defines a single lease period for all the state held by an NFS client ⏳
If the client doesn't renew its lease within that defined period, the server may release all the state associated with the client's lease 😭 The client can renew its lease with the RENEW operation, or implicitly through other operations (mostly READ) 💖
1.4.6. Client Caching and Delegation
Caching for files, attributes, and directories in NFSv4 is similar to previous versions 🍓 Attributes and directory information are cached for a duration the client decides on. When a predefined timeout ends, the client asks the server whether the related file system object has been updated 🕰️
For file data, the client checks its cache validity when the file gets opened. It sends a query to the server to find out whether the file has changed. Based on the answer, the client decides whether to keep or release its data cache for that file. Also, when the file is closed, any modified data gets written back to the server 💾
If an application wants to serialize access to file data, it should use file locking on the file data ranges in question 🔐
The big new caching addition in NFSv4 is that the server can delegate certain responsibilities to the client 👑 When the server grants a delegation for a file to a client, the client is guaranteed certain semantics about sharing that file with other clients. At OPEN, the server may give the client either a read delegation (OPEN_DELEGATE_READ) or a write delegation (OPEN_DELEGATE_WRITE) for the file (see Section 10.4) ✨ If the client gets an OPEN_DELEGATE_READ delegation, it's assured that no other client can write to the file for as long as the delegation lasts. If the client gets an OPEN_DELEGATE_WRITE delegation, it's assured that no other client has read or write access to the file 🫶
Delegations can be recalled by the server. If another client asks for access to the file in a way that conflicts with the granted delegation, the server can notify the original client and recall the delegation. This requires that a callback path exists between the server and the client. If that callback path doesn't exist, delegations can't be granted. The essence of a delegation is that it lets the client handle operations such as OPEN, CLOSE, LOCK, LOCKU, READ, or WRITE locally, without immediate interaction with the server 🦋
1.5. General Definitions ✨
okay so before we dive in, here's the glossary bestie. these definitions are just there to give you the right context while you read, so keep them in your back pocket 💅
- Absent File System: a file system is "absent" when a namespace component doesn't have a backing file system behind it 🫥
- Anonymous Stateid: a special locking object. it's defined in Section 9.1.4.3 🎀
- Byte: in this document, a byte is an octet, which is a datum exactly 8 bits long 🔢
- Client: the client is the entity that accesses the NFS server's resources 💜. it can be an application that has the logic to talk to the NFS server directly, or it can be the traditional operating system client that provides remote file system services for a bunch of applications.
when we're talking about byte-range locking, the client is also the entity that keeps track of a set of locks on behalf of one or more applications. this client is the one in charge of crash or failure recovery for the locks it manages 🛠️
also, multiple clients can share the same transport and connection, and multiple clients can live on the same network node. no cap, that's a lot of clients 🦋
- Client ID: the client ID is a 64-bit quantity that works as a unique, shorthand reference to a client-supplied verifier and ID. the server is the one responsible for supplying it 🪪
- File System: the file system is the collection of objects on a server that share the same
fsidattribute (see Section 5.8.1.9) 🗂️ - Lease: a lease is an interval of time that the server defines, during which the client is irrevocably granted a lock 🔒. when a lease period ends, the lock can get revoked if the lease hasn't been extended. and if a conflicting lock gets granted after the lease interval, the lock has to be revoked. no exceptions, bestie 💀
every lease a server grants has the same fixed duration. the fixed interval was chosen on purpose, to save the server from the headache of keeping state about variable-length leases across server failures 🧠
- Lock: the term "lock" refers to record (byte-range) locks as well as share reservations, unless it's specifically said otherwise 🔐
- Lock-Owner: each byte-range lock is tied to a specific lock-owner and an open-owner. the lock-owner is made of a client ID and an opaque owner string. the client shows this to the server to prove ownership of the byte-range lock whenever it's needed ✨
- Open-Owner: each open file is tied to a specific open-owner, which is made of a client ID and an opaque owner string. the client presents it to the server to establish ownership of the open whenever it's needed 🌸
- READ Bypass Stateid: another special locking object, defined in Section 9.1.4.3 🦋
- Server: the "server" is the entity responsible for coordinating client access to a set of file systems 👑
- Stable Storage: NFSv4 servers have to be able to bounce back with zero data loss from multiple power failures (including cascading ones, aka several power failures in quick succession), operating system failures, and hardware failures of components other than the storage medium itself (like the disk or non-volatile RAM) 😭
these are some of the stable storage setups that are allowed for an NFS server, iykyk:
- Media commit of data. the modified data has been successfully written to the disk media, like the actual disk platter 💿
- An immediate reply disk drive with battery-backed on-drive intermediate storage or an uninterruptible power system (UPS) 🔋
- Server commit of data with battery-backed intermediate storage and recovery software 🛡️
- Cache commit with a UPS and recovery software 🧯
- Stateid: a stateid is a 128-bit quantity returned by a server. it uniquely identifies the open and locking states the server provides for a specific open-owner, or a lock-owner/open-owner pair, for a specific file and type of lock 🆔
- Verifier: a verifier is a 64-bit quantity generated by the client. the server can use it to figure out whether the client has restarted and lost all of its previous lock state 🔄
1.6. Changes since RFC 3530
the main changes from RFC 3530 [RFC3530] are giving major glow-up energy 🌈
- the XDR definition got moved over to a companion document, [RFC7531] 📦
- the IETF intellectual property statements were updated to the latest version 📜
- there's a restructured and more complete explanation of multi-server namespace features 🗺️
- the handling of domain names was updated to reflect Internationalized Domain Names in Applications (IDNA) [RFC5891] 🌍
- the previously required LIPKEY and SPKM-3 security mechanisms have been removed 🗑️
- some clarification was added about a client re-establishing callback information to the new server when state has been migrated 📞
- a third edge case was added for courtesy locks and network partitions 🧩
- the definition of stateid got strengthened 💪
1.7. Changes between RFC 3010 and RFC 3530
okay the history lesson is actually so cute ngl. the definition of the NFSv4 protocol in [RFC3530] replaced and obsoleted the one in [RFC3010]. some parts stayed the same, but others had real substantive changes. those changes reflect implementation experience and further review of the protocol, which is very mature behavior 🧠✨
this list isn't everything, just the most notable changes and additions:
- the state model added an
open_owner4identifier. this was done to accommodate POSIX-based clients and the model they use for file locking 🐧. for POSIX clients, anopen_owner4corresponds to a file descriptor that may be shared among a set of processes, and thelock_owner4identifier corresponds to a process that's locking a file 📂 - added clarifications and error conditions for handling the owner and group attributes. these attributes are string based (not the numeric uid/gid of previous versions of NFS), so translations may not be available, and that's why the changes were made 🔤
- added clarifications for the ACL and mode attributes to address evaluation and partial support 🎨
- for identifiers that are defined as XDR opaque, limits were set on their size 📏
- added the
mounted_on_fileidattribute so POSIX clients can correctly construct local mounts 🪄 - modified the
SETCLIENTID/SETCLIENTID_CONFIRMoperations so they deal correctly with confirmation details, plus the ability to specify new client callback information. clarification was also added on the callback information itself 📞 - added a new operation,
RELEASE_LOCKOWNER, to let the server know alock_owner4won't be used by the client anymore 👋 - added
RENEWoperation changes to identify the client correctly and allow for additional error returns 🔁 - verified the error return possibilities for all operations ✅
- removed the use of the
pathname4data type fromLOOKUPandOPEN, in favor of having the client construct a sequence ofLOOKUPoperations to get the same effect 🧭
2. Protocol Data Types
the syntax and semantics for describing the data types of the NFSv4 protocol are defined in the XDR [RFC4506] and RPC [RFC5531] documents. the next sections build on the XDR data types to define types and structures specific to this protocol 🧱. quick reminder, the size constants and authoritative definitions live in [RFC7531] 📌
2.1. Basic Data Types ✨
okay so Table 1 is basically the cheat sheet for every base NFSv4 data type 💅 each row gives you the name, the typedef or enum, and what it's used for. 🍓
+-----------------+-------------------------------------------------+
| Data Type | Definition |
+-----------------+-------------------------------------------------+
| int32_t | typedef int int32_t; |
| | |
| uint32_t | typedef unsigned int uint32_t; |
| | |
| int64_t | typedef hyper int64_t; |
| | |
| uint64_t | typedef unsigned hyper uint64_t; |
| | |
| attrlist4 | typedef opaque attrlist4<>; |
| | |
| | Used for file/directory attributes. |
| | |
| bitmap4 | typedef uint32_t bitmap4<>; |
| | |
| | Used in attribute array encoding. |
| | |
| changeid4 | typedef uint64_t changeid4; |
| | |
| | Used in the definition of change_info4. |
| | |
| clientid4 | typedef uint64_t clientid4; |
| | |
| | Shorthand reference to client identification. |
| | |
| count4 | typedef uint32_t count4; |
| | |
| | Various count parameters (READ, WRITE, COMMIT). |
| | |
| length4 | typedef uint64_t length4; |
| | |
| | Describes LOCK lengths. |
| | |
| mode4 | typedef uint32_t mode4; |
| | |
| | Mode attribute data type. |
| | |
| nfs_cookie4 | typedef uint64_t nfs_cookie4; |
| | |
| | Opaque cookie value for READDIR. |
| | |
| nfs_fh4 | typedef opaque nfs_fh4<NFS4_FHSIZE>; |
| | |
| | Filehandle definition. |
| | |
| nfs_ftype4 | enum nfs_ftype4; |
| | |
| | Various defined file types. |
| | |
| nfsstat4 | enum nfsstat4; |
| | |
| | Return value for operations. |
| | |
| nfs_lease4 | typedef uint32_t nfs_lease4; |
| | |
| | Duration of a lease in seconds. |
| | |
| offset4 | typedef uint64_t offset4; |
| | |
| | Various offset designations (READ, WRITE, LOCK, |
| | COMMIT). |
| | |
| qop4 | typedef uint32_t qop4; |
| | |
| | Quality of protection designation in SECINFO. |
| | |
| sec_oid4 | typedef opaque sec_oid4<>; |
| | |
| | Security Object Identifier. The sec_oid4 data |
| | type is not really opaque. Instead, it |
| | contains an ASN.1 OBJECT IDENTIFIER as used by |
| | GSS-API in the mech_type argument to |
| | GSS_Init_sec_context. See [RFC2743] for |
| | details. |
| | |
| seqid4 | typedef uint32_t seqid4; |
| | |
| | Sequence identifier used for file locking. |
| | |
| utf8string | typedef opaque utf8string<>; |
| | |
| | UTF-8 encoding for strings. |
| | |
| utf8str_cis | typedef utf8string utf8str_cis; |
| | |
| | Case-insensitive UTF-8 string. |
| | |
| utf8str_cs | typedef utf8string utf8str_cs; |
| | |
| | Case-sensitive UTF-8 string. |
| | |
| utf8str_mixed | typedef utf8string utf8str_mixed; |
| | |
| | UTF-8 strings with a case-sensitive prefix and |
| | a case-insensitive suffix. |
| | |
| component4 | typedef utf8str_cs component4; |
| | |
| | Represents pathname components. |
| | |
| linktext4 | typedef opaque linktext4<>; |
| | |
| | Symbolic link contents ("symbolic link" is |
| | defined in an Open Group [openg_symlink] |
| | standard). |
| | |
| ascii_REQUIRED4 | typedef utf8string ascii_REQUIRED4; |
| | |
| | String is sent as ASCII and thus is |
| | automatically UTF-8. |
| | |
| pathname4 | typedef component4 pathname4<>; |
| | |
| | Represents pathname for fs_locations. |
| | |
| nfs_lockid4 | typedef uint64_t nfs_lockid4; |
| | |
| verifier4 | typedef opaque verifier4[NFS4_VERIFIER_SIZE]; |
| | |
| | Verifier used for various operations (COMMIT, |
| | CREATE, OPEN, READDIR, WRITE) |
| | NFS4_VERIFIER_SIZE is defined as 8. |
+-----------------+-------------------------------------------------+
Table 1: Base NFSv4 Data Types2.2. Structured Data Types 🎀
2.2.1. nfstime4 🕰️
struct nfstime4 {
int64_t seconds;
uint32_t nseconds;
};so nfstime4 is how we count time 🧚 it's a number of seconds plus a number of nanoseconds, counted from midnight (0 hour) on January 1, 1970 in Coordinated Universal Time (UTC). a positive seconds value means a date after that moment, and a negative one means a date before it. either way, nseconds gets added onto seconds to get the final time. example bestie: if you want something one-half second before 0 hour January 1, 1970, seconds is negative one (-1) and nseconds is one-half second (500000000). 🙈 and nseconds values above 999,999,999 are considered invalid, so don't even try it 💀
this type is what carries time and date info around. a server converts to and from its own local time representation when it processes these values, keeping as much accuracy as it can. if the timestamps stored for a file system object are less precise than what's defined here, you can lose some precision, which is lowkey unavoidable. an adjunct time maintenance protocol is recommended to reduce client and server time skew. ⏰
2.2.2. time_how4 ⏱️
enum time_how4 {
SET_TO_SERVER_TIME4 = 0,
SET_TO_CLIENT_TIME4 = 1
};2.2.3. settime4 🌸
union settime4 switch (time_how4 set_it) {
case SET_TO_CLIENT_TIME4:
nfstime4 time;
default:
void;
};these definitions are the attribute definitions you use to set time values ✨ if set_it is SET_TO_SERVER_TIME4, the server uses its own local time representation for the time value. if it's SET_TO_CLIENT_TIME4, you send a full nfstime4 along with it. 🫶
2.2.4. specdata4 🦋
struct specdata4 {
uint32_t specdata1; /* major device number */
uint32_t specdata2; /* minor device number */
};this one carries extra info for the device file types NF4CHR and NF4BLK, so it's only relevant when you're dealing with those 🔥
2.2.5. fsid4 👑
struct fsid4 {
uint64_t major;
uint64_t minor;
};fsid4 is the file system identifier, and it's a REQUIRED attribute, so no skipping it 💅
2.2.6. fs_location4 🌈
struct fs_location4 {
utf8str_cis server<>;
pathname4 rootpath;
};2.2.7. fs_locations4 🧭
struct fs_locations4 {
pathname4 fs_root;
fs_location4 locations<>;
};fs_location4 and fs_locations4 are the types behind the fs_locations RECOMMENDED attribute, which is what supports migration and replication. so if you're moving or copying file systems around, this is the stuff you need, periodt 🚚💜
2.2.8. fattr4 ✨
Okay bestie, this is the file attribute bundle 📦. Here's the struct:
struct fattr4 {
bitmap4 attrmask;
attrlist4 attr_vals;
};The fattr4 structure is how we represent file and directory attributes 🗂️. The bitmap is a counted array of 32-bit integers that holds bit values. To find which integer in the array holds bit n, you compute (n / 32), and its position inside that integer is (n mod 32) 🧮.
0 1
+-----------+-----------+-----------+--
| count | 31 .. 0 | 63 .. 32 |
+-----------+-----------+-----------+--2.2.9. change_info4 💕
struct change_info4 {
bool atomic;
changeid4 before;
changeid4 after;
};This one is used with the CREATE, LINK, REMOVE, and RENAME operations 🔄. It lets the client know the value of the change attribute for the directory where the target file system object lives. It's giving "here's what changed, babe" 💅.
2.2.10. clientaddr4 🌸
struct clientaddr4 {
/* see struct rpcb in RFC 1833 */
string r_netid<>; /* network id */
string r_addr<>; /* universal address */
};The clientaddr4 structure gets used in the SETCLIENTID operation in one of two ways. Either it says where the client using a client ID lives, or it shows up as part of the callback registration 📍. The r_netid and r_addr fields hold a network id and a universal address, respectively. The network id and universal address concepts, plus the formats for TCP over IPv4 and TCP over IPv6, are defined in [RFC5665], specifically Tables 2 and 3 and Sections 5.2.3.3 and 5.2.3.4 📚.
2.2.11. cb_client4 🦋
struct cb_client4 {
unsigned int cb_program;
clientaddr4 cb_location;
};The client uses this structure to tell the server its callback address. It has the program number and the client address, so the server knows exactly where to call back 📞.
2.2.12. nfs_client_id4 🎀
struct nfs_client_id4 {
verifier4 verifier;
opaque id<NFS4_OPAQUE_LIMIT>;
};This structure is part of the arguments for the SETCLIENTID operation 🪪.
2.2.13. open_owner4 🧚
struct open_owner4 {
clientid4 clientid;
opaque owner<NFS4_OPAQUE_LIMIT>;
};This structure identifies the owner of open state 🔑.
2.2.14. lock_owner4 🔒
struct lock_owner4 {
clientid4 clientid;
opaque owner<NFS4_OPAQUE_LIMIT>;
};This one identifies the owner of file locking state. Same vibe as open_owner4, just for locks 💜.
2.2.15. open_to_lock_owner4 🍓
struct open_to_lock_owner4 {
seqid4 open_seqid;
stateid4 open_stateid;
seqid4 lock_seqid;
lock_owner4 lock_owner;
};This structure is used for the first LOCK operation done for an open_owner4 🌈. It carries both the open_stateid and the lock_owner, so the transition goes from a valid open_stateid sequence over to the new lock_stateid sequence. The best part is that this skips confirming the lock_owner/lock_seqid pair, because it's already tied to established state in the form of the open_stateid/open_seqid. Iconic efficiency 👑.
2.2.16. stateid4 ✨
struct stateid4 {
uint32_t seqid;
opaque other[NFS4_OTHER_SIZE];
};This structure is used for the various state-sharing mechanisms between the client and server 🤝. For the client, it's read-only, no touching 🙈. The server is required to increment the seqid field monotonically at each transition of the stateid. This matters a lot because the client looks at the seqid in OPEN stateids to figure out the order the server processed OPEN requests in. Main character timeline stuff 🕰️.
3. RPC and Security Flavor 🛡️
The NFSv4 protocol is an RPC application that uses RPC version 2 and the XDR, as defined in [RFC5531] and [RFC4506] 💻. The RPCSEC_GSS security flavors, as defined in version 1 ([RFC2203]) and version 2 ([RFC5403]), MUST be implemented as the mechanism for delivering stronger security for the NFSv4 protocol 🔐. However, deployment of RPCSEC_GSS is optional, so it's a glow-up, not a mandatory dress code ✨.
3.1. Ports and Transports 🚪
Historically, NFSv2 and NFSv3 servers have lived on port 2049 🏠. The registered port 2049 [RFC3232] for the NFS protocol SHOULD be the default configuration. Using the registered port means the NFS client won't need to use the RPC binding protocols described in [RFC1833], and that lets NFS get through firewalls 🧱➡️🦋.
Where an NFSv4 implementation supports operation over the IP network protocol, the supported transport layer between NFS and IP MUST be an IETF standardized transport protocol that is specified to avoid network congestion. Such transports include TCP and the Stream Control Transmission Protocol (SCTP) 🌊. To boost interoperability, an NFSv4 implementation MUST support operation over the TCP transport protocol 🫶.
If TCP is the transport, the client and server SHOULD use persistent connections. This prevents TCP's congestion control from getting weakened by short-lived connections, and it improves performance over the Wide Area Network (WAN) by skipping the need for SYN handshakes 🚀.
As noted in Section 19, the authentication model for NFSv4 has moved from machine-based to principal-based 🪄. But changing the authentication model doesn't force a technical change to the TCP connection management model, which can stay whole machine-based instead of moving to a per-user model. NFS over TCP client implementations have traditionally multiplexed traffic for multiple users over a common TCP connection between an NFS client and server. That's been true whether the NFS client uses AUTH_SYS, AUTH_DH, RPCSEC_GSS, or any other flavor 🧃. NFS over TCP server implementations have assumed this model too, so they scale their TCP connection management with the number of expected client machines. NFSv4 is intended to leave this connection management model alone. NFSv4 clients that break this assumption can expect scaling issues on the server, and that means reduced service 😬.
3.1.1. Client Retransmission Behavior 🔁
When an NFSv4 server processes a request received over a reliable transport such as TCP, it MUST NOT silently drop the request, except when the established transport connection has been broken 🙅♀️. Given that contract between NFSv4 clients and servers, clients MUST NOT retry a request unless one or both of the following are true:
- The transport connection has been broken
- The procedure being retried is the NULL procedure
Reliable transports like TCP don't always tell their peer right away when the other side has broken the connection (for example, when an NFS server reboots) 🥲. So the NFSv4 client may want to actively "probe" the connection to see whether it's been broken. Using the NULL procedure is one recommended way to do that. When a client hits a remote procedure call timeout (of some arbitrary implementation-specific amount), instead of retrying the call, it could issue a NULL procedure call to the server. If the server has died, the transport connection break will eventually be indicated to the NFSv4 client 💀. The client can then reconnect and retry the original request. If the NULL procedure call gets a response, the connection hasn't broken. The client can choose to wait longer for the original request's response, or it can break the transport connection, reconnect, and then re-send the original request 🧠.
For callbacks from the server to the client, the same rules apply, but the server doing the callback becomes the client, and the client receiving the callback becomes the server 🔄.
3.2. Security Flavors 🎀
Traditional RPC implementations have included AUTH_NONE, AUTH_SYS, AUTH_DH, and AUTH_KRB4 as security flavors. With [RFC2203], there's an additional security flavor called RPCSEC_GSS, which uses the functionality of GSS-API [RFC2743]. This lets the RPC layer use various security mechanisms without the extra implementation overhead of adding RPC security flavors 🧩. For NFSv4, the RPCSEC_GSS security flavor MUST be used to enable the mandatory-to-implement security mechanism. Other flavors, such as AUTH_NONE, AUTH_SYS, and AUTH_DH, MAY be implemented as well 💖.
3.2.1. Security Mechanisms for NFSv4 🔐
RPCSEC_GSS, via GSS-API, supports multiple mechanisms that provide security services. For interoperability, NFSv4 clients and servers MUST support the Kerberos V5 security mechanism 🗝️.
Using RPCSEC_GSS means picking a mechanism, a quality of protection (QOP), and a service (authentication, integrity, or privacy). For the mandated security mechanisms, NFSv4 specifies that a QOP of zero is used, and it leaves it up to the mechanism, or the mechanism's configuration, to map QOP zero to an appropriate level of protection. Each mandated mechanism specifies a minimum set of cryptographic algorithms for integrity and privacy. NFSv4 clients and servers MUST be implemented on operating environments that comply with the required cryptographic algorithms of each required mechanism 🧁.
3.2.1.1. Kerberos V5 as a Security Triple 👑
The Kerberos V5 GSS-API mechanism described in [RFC4121] MUST be implemented with the RPCSEC_GSS services specified in Table 2. Both client and server MUST support each of the pseudo-flavors 💅.
+--------+-------+----------------------+-----------------------+
| Number | Name | Mechanism's OID | RPCSEC_GSS service |
+--------+-------+----------------------+-----------------------+
| 390003 | krb5 | 1.2.840.113554.1.2.2 | rpc_gss_svc_none |
| 390004 | krb5i | 1.2.840.113554.1.2.2 | rpc_gss_svc_integrity |
| 390005 | krb5p | 1.2.840.113554.1.2.2 | rpc_gss_svc_privacy |
+--------+-------+----------------------+-----------------------+
Table 2: Mapping Pseudo-Flavor to ServiceThe pseudo-flavor is shown here as a mapping aid for the implementer 🗺️. NFSv4 has its own way to negotiate security and it understands the GSS-API mechanism, so the pseudo-flavor isn't actually needed. It's only needed for NFSv3, since security negotiation there happens through the MOUNT protocol described in [RFC2623] 📜.
When this document was specified, the Advanced Encryption Standard (AES) with HMAC-SHA1 was a required algorithm set for Kerberos V5 ✨. In contrast, when NFSv4.0 was first specified in [RFC3530], weaker algorithm sets were REQUIRED for Kerberos V5, and they were REQUIRED in the NFSv4.0 specification too, because the Kerberos V5 specification at that time didn't specify stronger algorithms 🕰️. The NFSv4 specification doesn't specify required algorithms for Kerberos V5. Instead, implementers are expected to track the evolution of the Kerberos V5 standard, in case stronger algorithms get specified later 👀.
3.2.1.1.1. Security Considerations for Cryptographic Algorithms in Kerberos V5 🚨
When you deploy NFSv4, the strength of the security you get depends on your existing Kerberos V5 infrastructure. The algorithms of Kerberos V5 aren't directly exposed to or selectable by the client or server, so the user of NFSv4 has to do some due diligence to make sure security is acceptable wherever it's needed 🧐. Guidance is provided in [RFC6649] as to why weak algorithms should be disabled by default 💀.
3.3. Security Negotiation 🔐✨
Okay bestie, so the NFSv4 server might be offering a bunch of security mechanisms, and the client needs a way to figure out which one to use 🤔 That's the whole negotiation thing! The NFS server can have lots of entry points in its file system namespace that NFS clients can use, and each of those entry points can have different security mechanisms, or even several at once. Main character server energy 👑
The security negotiation between client and server SHOULD happen over a secure channel 🔒 Why? So nobody in the middle can sneak in, mess with the negotiation, and trick the client and server into picking a weaker level of security than they need or want. Lowkey terrifying, so keep it locked down! See Section 19 for more on this 🫶
3.3.1. SECINFO 🕵️♀️
The SECINFO operation lets the client figure out, for each filehandle, which security triple (see [RFC2743] and Section 16.31.4) to use when accessing the server 💅 Usually the client won't need SECINFO, except during the very first chat with the server, or when it runs into a new security policy while it's wandering around the namespace. Either of those moments forces the client to negotiate a brand new security triple 🦋
3.3.2. Security Error 🚨
Okay so here's the plan. Every NFSv4 client and server MUST support a minimum set of security, which is Kerberos V5 under RPCSEC_GSS 🔑 So the NFS client starts talking to the server using one of those minimal security triples. Then, while they're chatting, the client might get the NFS error NFS4ERR_WRONGSEC 😭 That error is the server basically saying, "hey, this security triple isn't good enough to get into my file system stuff." The client then has to figure out what security triples the server has available and pick one that works for it 🎀 Check Section 16.31 for how the client responds to NFS4ERR_WRONGSEC and uses SECINFO, it's giving detailed instructions 💖
3.3.3. Callback RPC Authentication 📞
Except as noted elsewhere in this section, the callback RPC (described later) MUST mutually authenticate the NFS server to the principal that acquired the client ID (also described later), using the security flavor of the original SETCLIENTID operation used 🫶 Like, the server and the principal have to vouch for each other, no skipping!
For AUTH_NONE, there are no principals, so this is a non-issue 🙈
AUTH_SYS has no notion of mutual authentication or a server principal. So the callback from the server just uses the AUTH_SYS credential that the user had when they set up the delegation ✨
For AUTH_DH, one commonly used convention is that the server uses the credential corresponding to this AUTH_DH principal, which looks like this:
unix.host@domainHere, host and domain are variables standing for the name of the server host and the directory services domain it lives in, like a Network Information System domain or a DNS domain 🌸
Regardless of which security mechanism under RPCSEC_GSS is being used, the NFS server MUST identify itself in GSS-API using a GSS_C_NT_HOSTBASED_SERVICE name type 💜 GSS_C_NT_HOSTBASED_SERVICE names look like this:
service@hostnameFor NFS, the "service" element is:
nfsSecurity mechanism implementations will convert nfs@hostname into different forms depending on the mechanism 🧚 For Kerberos V5, the following form is RECOMMENDED:
nfs/hostnameFor Kerberos V5, nfs/hostname would be a server principal in the Kerberos Key Distribution Center database 🔥 This is the same principal the client got a GSS-API context for when it issued the SETCLIENTID operation, so the realm name for the server principal must be the same for the callback as it was for the SETCLIENTID. No realm drama allowed 💅
4. Filehandles 📁🎀
The filehandle in the NFS protocol is a per-server unique identifier for a file system object 🦋 The contents of the filehandle are opaque to the client, so the server is the one responsible for translating the filehandle into an internal representation of the file system object. The client just holds it and hands it back, no peeking inside 🙈
4.1. Obtaining the First Filehandle 🚪
The operations of the NFS protocol are defined in terms of one or more filehandles, so the client needs a filehandle to start talking to the server 💖 With the NFSv2 protocol [RFC1094] and the NFSv3 protocol [RFC1813], there was an ancillary protocol for getting that first filehandle. The MOUNT protocol, RPC program number 100005, is the one that turns a string-based file system pathname into a filehandle that the NFS protocols can then use ✨
The MOUNT protocol has security problems and doesn't play nicely with firewalls 😬 That's one reason the public filehandle got introduced in [RFC2054] and [RFC2055]. With the public filehandle plus the LOOKUP operation in NFSv2 and NFSv3, it was shown that the MOUNT protocol isn't actually needed for a solid client and server relationship 💅
So NFSv4 doesn't use an ancillary protocol to translate string-based pathnames into filehandles. Instead, there are two special filehandles that serve as starting points for the NFS client 🌈
4.1.1. Root Filehandle 🌳
The first special filehandle is the root filehandle. It's the "conceptual" root of the file system namespace at the NFS server 👑 The client uses or starts with the root filehandle by using the PUTROOTFH operation. PUTROOTFH tells the server to set the current filehandle to the root of the server's file tree. Once PUTROOTFH has been used, the client can walk the whole server file tree with the LOOKUP operation 🧚 For the full scoop on the server namespace, see Section 7.
4.1.2. Public Filehandle 🌍
The second special filehandle is the public filehandle. Unlike the root filehandle, the public filehandle may be bound to, or represent, an arbitrary file system object at the server 🦋 The server is responsible for this binding. The public filehandle and the root filehandle might refer to the same file system object, but it's up to the administrative software at the server and the server administrator's policies to decide how the public filehandle is bound to a server file system object. The client may not make any assumptions about this binding 🙈 The client uses the public filehandle via the PUTPUBFH operation.
4.2. Filehandle Types 🗂️
In the NFSv2 and NFSv3 protocols, there was one type of filehandle with a single set of semantics. The main one was that it stayed persistent across a server reboot 💜 That's why this type is called "persistent" in NFSv4. Persistent filehandles keep the same semantics as before. NFSv4 also brings in a new type called the volatile filehandle, which tries to fit certain server environments 🌸
The volatile filehandle type was added to deal with server functionality or implementation issues that make a correct persistent filehandle infeasible 😭 Some server environments just don't provide a file system level invariant that can be used to build a persistent filehandle. Either the underlying file system doesn't provide that invariant, or the server's file system programming interfaces don't give access to it. Volatile filehandles can make things like hierarchical storage management or file system reorganization or migration easier to implement on the server side. However, volatile filehandles make life harder for the client 💀
Since the client needs to handle persistent and volatile filehandles differently, there's a file attribute defined that the client can use to figure out which filehandle types the server is returning ✨
4.2.1. General Properties of a Filehandle 🧠
The filehandle has all the info the server needs to tell one individual file apart from another 🔍 To the client, the filehandle is opaque. The client stores filehandles to use in a later request, and it can compare two filehandles from the same server for equality by doing a byte-by-byte comparison. But the client MUST NOT otherwise interpret the contents of filehandles 🚫 If two filehandles from the same server are equal, they MUST refer to the same file. However, it's not required that two different filehandles refer to different file system objects. Servers SHOULD try to keep a one-to-one correspondence between filehandles and file system objects, but there may be situations where the mapping isn't one-to-one 💅 Clients MUST use filehandle comparisons only to improve performance, not for correct behavior. All clients need to be ready for situations where it can't be determined whether two different filehandles denote the same object. In those cases they need to avoid assuming that the objects are different, because that might cause incorrect behavior 🫶 More on filehandle and attribute comparison in the context of data caching is in Section 10.3.4.
As an example, if two different pathnames, when traversed at the server, end up at the same file system object, the server SHOULD return the same filehandle for each path 🦋 This can happen if a hard link is used to create two filenames that refer to the same underlying file object and its associated data. For example, if paths /a/b/c and /a/d/c refer to the same file, the server SHOULD return the same filehandle for both pathname traversals 💖
4.2.2. Persistent Filehandle 💎
A persistent filehandle has a fixed value for the lifetime of the file system object it refers to 🔒 Once the server creates the filehandle for a file system object, the server MUST accept the same filehandle for that object for the object's entire lifetime. If the server restarts or reboots, the NFS server must honor the same filehandle value it used in its previous instantiation. Similarly, if the file system is migrated, the new NFS server must honor the same filehandle as the old NFS server 🌸
A persistent filehandle becomes stale or invalid when its file system object is removed 😭 When the server gets a persistent filehandle that refers to a deleted object, it MUST return an error of NFS4ERR_STALE. A filehandle may also go stale when the file system containing the object is no longer available. That can happen if the file system lives on removable media and the media isn't available at the server anymore, if the whole file system has been destroyed, or if it's simply been removed from the server's namespace (like being unmounted in a UNIX environment) 🥀
4.2.3. Volatile Filehandle 🦋
okay so a volatile filehandle does NOT have the same longevity as a persistent one 😭 the server can decide it's no longer valid at lots of different points in time. if the server can definitively tell that the object was removed, it should send back NFS4ERR_STALE to the client (same as with persistent filehandles 💅). in every other case where the server decides a volatile filehandle can't be used anymore, it should return NFS4ERR_FHEXPIRED.
the REQUIRED attribute "fh_expire_type" is how the client finds out what kind of filehandle the server is giving for a particular file system ✨ it's a bitmask with these values:
FH4_PERSISTENT: this one means a persistent filehandle, valid until the object is removed from the file system. the server will not returnNFS4ERR_FHEXPIREDfor it.FH4_PERSISTENTis defined as the value where none of the bits listed below are set 🌸FH4_VOLATILE_ANY: the filehandle may expire at any time, except where it's specifically excluded (i.e.,FH4_NOEXPIRE_WITH_OPEN) 🍓FH4_NOEXPIRE_WITH_OPEN: only allowed whenFH4_VOLATILE_ANYis set. if this bit is set, the meaning ofFH4_VOLATILE_ANYgets qualified so that the filehandle doesn't expire while it's open 🔒FH4_VOL_MIGRATION: the filehandle will expire as a result of migration. ifFH4_VOLATILE_ANYis set,FH4_VOL_MIGRATIONis redundant 🧚FH4_VOL_RENAME: the filehandle will expire during a rename. that includes a rename by the requesting client OR a rename by any other client. ifFH4_VOLATILE_ANYis set,FH4_VOL_RENAMEis redundant 💜
so servers that hand out volatile filehandles which can expire while open (that's when FH4_VOL_MIGRATION or FH4_VOL_RENAME is set, or when FH4_VOLATILE_ANY is set and FH4_NOEXPIRE_WITH_OPEN is not set) should deny a RENAME or REMOVE that would affect an OPEN file, or any of the components leading to that OPEN file 🙅♀️ and on top of that, the server SHOULD deny ALL RENAME or REMOVE requests during the grace period after a server restart. it's giving protection 👑
ngl the bits FH4_VOL_MIGRATION and FH4_VOL_RENAME are super useful because they let the client figure out that expiration happened whenever that specific event occurs, without the server ever sending an explicit expiration error 🫶 FH4_VOLATILE_ANY doesn't give that kind of info. say the server will expire many, but not all, filehandles on migration (like, everything except the ones that are open). then FH4_VOLATILE_ANY together with FH4_NOEXPIRE_WITH_OPEN is the better pick, because the client can't assume every filehandle will expire at migration, and extra expirations can happen later when a file gets CLOSEd, separate in time from the migration event itself 🦋
4.2.4. One Method of Constructing a Volatile Filehandle ✨
a volatile filehandle is opaque to the client, but the server could build one that holds:
[volatile bit = 1 | server boot time | slot | generation number]slotis an index in the server volatile filehandle tablegeneration numberis the generation number for the table entry/slot
when the client shows up with a volatile filehandle, the server runs these checks, and they assume the volatile bit check already passed 🔍 if the server boot time is less than the current server boot time, return NFS4ERR_FHEXPIRED. if the slot is out of range, return NFS4ERR_BADHANDLE. if the generation number doesn't match, return NFS4ERR_FHEXPIRED.
when the server reboots, the table is gone, because it's volatile 💀 and if the volatile bit is 0, then it's actually a persistent filehandle with a different structure following it 🌈
4.3. Client Recovery from Filehandle Expiration 🌸
if it's possible, the client should recover when it gets an NFS4ERR_FHEXPIRED error 💖 the client has to take on some extra responsibility so it can get ready to recover from a volatile filehandle expiring. if the server returns persistent filehandles, the client doesn't need any of these extra steps 🎀
for volatile filehandles, the client will most commonly need to store the component names leading up to and including the file system object in question 📝 with those names, the client should be able to recover by finding a filehandle in the namespace that's still available, or by starting over at the root of the server's file system namespace 🗺️
if the expired filehandle points to an object that was removed from the file system, the client obviously won't be able to recover from it 😭
it's also possible the expired filehandle points to a file that got renamed. if another client did the renaming, the original client may not be able to recover. BUT if the client itself is renaming the file and the file is open, the client may be able to recover 🙌 the client can work out the new pathname from the processing of the rename request, and then regenerate the new filehandle from that new pathname. the client could also use the COMPOUND operation mechanism to build a set of operations like this:
RENAME A B
LOOKUP B
GETFHheads up, though: the COMPOUND procedure does NOT provide atomicity. this example just cuts down the overhead of recovering from an expired filehandle ⚠️
5. Attributes 💅
okay so attributes need to be handled in a flexible way, because we need extensibility and better interoperability with non-UNIX platforms 🌍 the NFSv3 fattr3 structure has a fixed list of attributes that not every client and server can support or even cares about. fattr3 can't be extended as new needs come up, and it has no way to say "not supported" 💀 with NFSv4.0, the client can ask what attributes the server supports and build requests with only those (or a subset of them) 🎀
so attributes are split into three groups: REQUIRED, RECOMMENDED, and named ✨ both REQUIRED and RECOMMENDED attributes get a specific, well-defined encoding in the NFSv4.0 protocol and are identified by number. you request them by setting a bit in the bit vector sent in the GETATTR request, and the server's response includes a bit vector listing which attributes came back 🧚 new REQUIRED or RECOMMENDED attributes may be added to the NFSv4 protocol as part of a new minor version, by publishing a Standards Track RFC that allocates a new attribute number value and defines the encoding for it. see Section 11 for more on that 📜
named attributes are accessed with the OPENATTR operation, which opens up a hidden directory of attributes attached to a file system object 🙈 OPENATTR takes a filehandle for the object and returns the filehandle for the attribute hierarchy. the filehandle for the named attributes is a directory object you can reach with LOOKUP or READDIR, and it contains files whose names are the named attributes and whose data bytes are the value of each attribute. for example:
+----------+-----------+---------------------------------+
| LOOKUP | "foo" | ; look up file |
| GETATTR | attrbits | |
| OPENATTR | | ; access foo's named attributes |
| LOOKUP | "x11icon" | ; look up specific attribute |
| READ | 0,4096 | ; read stream of bytes |
+----------+-----------+---------------------------------+named attributes are meant for data that applications need, not for what an NFS client implementation needs 🫶 NFS implementers are strongly encouraged to define their new attributes as RECOMMENDED attributes by taking them through the IETF Standards Track process 🦋
the REQUIRED set is deliberately small, since servers need to do whatever it takes to support those 💪 a server should support as many RECOMMENDED attributes as it possibly can. but by definition, the server isn't required to support all of them. an attribute only counts as REQUIRED if the data is needed by a large number of clients AND isn't reasonably computable by the client when the server doesn't provide it 🔥
heads up: the hidden directory returned by OPENATTR is just a convenience for protocol processing. the client shouldn't make any assumptions about how the server implements named attributes, or whether the underlying file system at the server has a named attribute directory at all 🙈 because of that, operations like SETATTR and GETATTR on the named attribute directory are undefined 🚫
5.1. REQUIRED Attributes 👑
these attributes MUST be supported by every NFSv4.0 client and server so there's a minimum level of interoperability 🫶 the server MUST store and return these attributes, and the client MUST be able to function with an attribute set limited to just these. with only the REQUIRED attributes, some client functionality can be impaired or limited in some ways 😬 a client can ask for any of these by setting a bit in the GETATTR request, and for each such bit that's set, the server MUST return the matching attribute value 💖
5.2. RECOMMENDED Attributes 🌈
these attributes are understood well enough to deserve support in the NFSv4.0 protocol 🌸 however, they might not be supported on all clients and servers. a client MAY ask for any of these by setting a bit in the GETATTR request, but it MUST handle the case where the server doesn't return them 🙏 a client MAY also ask for the set of attributes the server supports, but it SHOULD NOT request attributes the server doesn't support 🚫
a server should be tolerant of requests for unsupported attributes and just not return them, instead of treating the request as an error 🫶 it's expected that servers will support all the attributes they comfortably can, and only fail to support ones that are hard to support in their operating environments 💅 a server should provide attributes whenever it doesn't have to "tell lies" to the client. for example, a file modification time should either be an accurate time or not be supported by the server at all 🕰️ it's tough for clients sometimes, but a client is better placed to decide whether and how to fabricate or construct an attribute, or whether to just do without it 🧚
5.3. Named Attributes ✨
okay so named attributes are the side quest of the attribute world 🧚 they aren't encoded directly in the NFSv4 protocol, so they get a string name instead of a number. what they hold is an uninterpreted stream of bytes stored with the file system object. to get into the namespace you use the OPENATTR operation, which hands you a filehandle for a virtual "named attribute directory" 📂 from there you can browse and edit using operations you'd normally use on regular directories. READDIR gives you the list of named attributes, LOOKUP and OPEN can pick out one specific attribute, and creating a new one can happen when an OPEN asks for file creation 🦋
once you've done an OPEN, you can read and write named attributes with normal READ and WRITE using the filehandles and stateids that OPEN returned 💅
named attributes and the named attribute directory can have their own (non-named) attributes too 💖 each of these objects must have all of the REQUIRED attributes and may have additional RECOMMENDED attributes. the set for named attributes and the named attribute directory does not need to be, and typically won't be, as big as the set for other objects in that file system 🌸
named attributes might be the target of delegations 👀 but granting delegations is up to the server, so a server doesn't need to support delegations on named attributes. ngl that's a very chill rule
it is RECOMMENDED that servers support arbitrary named attributes ✨ a client should not depend on being able to store any named attributes in the server's file system, though. if a server does support them, a client that can handle them should be able to copy a file's data and metadata with complete transparency from one location to another. that would mean names allowed for regular directory entries are also valid for named attribute names 🫶
okay now the main drama. in NFSv4.0, the structure of named attribute directories is restricted in a bunch of ways 💀 the goal is to stop non-interoperable implementations from showing up, where some servers support a fully general hierarchical directory structure for named attributes and others support a limited but adequate one. in that kind of messy environment, clients or applications might end up depending on non-portable extensions. the restrictions are:
- CREATE is not allowed in a named attribute directory. Thus, such objects as symbolic links and special files are not allowed to be named attributes. Further, directories may not be created in a named attribute directory, so no hierarchical structure of named attributes for a single object is allowed.
- If
OPENATTRis done on a named attribute directory or on a named attribute, the server MUST return an error. - Doing a RENAME of a named attribute to a different named attribute directory or to an ordinary (i.e., non-named-attribute) directory is not allowed.
- Creating hard links between named attribute directories or between named attribute directories and ordinary directories is not allowed.
names of attributes will not be controlled by this document or other IETF Standards Track documents 🎀 see Section 20 for further discussion, it's a whole thing 🧐
5.4. Classification of Attributes 🍓
so every attribute you access with SETATTR and GETATTR (aka the REQUIRED and RECOMMENDED attributes) falls into one of three buckets 🪣
- per-server attributes, where the value is the same for every file object that shares the same server.
- per-file system attributes, where the value is the same for some or all file objects that share the same server and fsid attribute (Section 5.8.1.9). more on when that sharing actually kicks in below.
- per-file system object attributes.
how per-file system attributes get handled depends on the specific attribute and on the setting of the homogeneous (Section 5.8.2.12) attribute. the rules go like this:
- the values of the attributes
supported_attrs,fsid,homogeneous,link_support, andsymlink_supportare always common to all objects within the given file system. - for other attributes, different values may be returned for different file system objects if the attribute
homogeneousis supported within the file system in question and has the value false.
here's the sorting 🫧 note that time_access_set and time_modify_set aren't listed, because they're write-only attributes that correspond to time_access and time_modify and they're only used in a special instance of SETATTR 🙈
- The per-server attribute is:
lease_time
- The per-file system attributes are:
supported_attrs, fh_expire_type, link_support, symlink_support, unique_handles, aclsupport, cansettime, case_insensitive, case_preserving, chown_restricted, files_avail, files_free, files_total, fs_locations, homogeneous, maxfilesize, maxname, maxread, maxwrite, no_trunc, space_avail, space_free, space_total, and time_delta
- The per-file system object attributes are:
type, change, size, named_attr, fsid, rdattr_error, filehandle, acl, archive, fileid, hidden, maxlink, mimetype, mode, numlinks, owner, owner_group, rawdev, space_used, system, time_access, time_backup, time_create, time_metadata, time_modify, and mounted_on_fileid
for quota_avail_hard, quota_avail_soft, and quota_used, check their definitions below for the right classification 🔥
5.5. Set-Only and Get-Only Attributes 🌈
some REQUIRED and RECOMMENDED attributes are set-only, meaning you can set them with SETATTR but you can't retrieve them with GETATTR. similarly, some REQUIRED and RECOMMENDED attributes are get-only, meaning you can retrieve them with GETATTR but you can't set them with SETATTR. if a client tries to set a get-only attribute or get a set-only attribute, the server MUST return NFS4ERR_INVAL 🙅♀️ not allowed, bestie
5.6. REQUIRED Attributes - List and Definition References 📋
the list of REQUIRED attributes lives in Table 3 ✨ here's what the columns mean:
- Name: The name of the attribute.
- ID: The number assigned to the attribute. In the event of conflicts between the assigned number and [RFC7531], the latter is authoritative, but in such an event, it should be resolved with errata to this document and/or [RFC7531]. See [IESG_ERRATA] for the errata process.
- Data Type: The XDR data type of the attribute.
- Acc: Access allowed to the attribute. R means read-only (GETATTR may retrieve, SETATTR may not set). W means write-only (SETATTR may set, GETATTR may not retrieve). R W means read/write (GETATTR may retrieve, SETATTR may set).
- Defined in: The section of this specification that describes the attribute.
+-----------------+----+------------+-----+-------------------+
| Name | ID | Data Type | Acc | Defined in |
+-----------------+----+------------+-----+-------------------+
| supported_attrs | 0 | bitmap4 | R | Section 5.8.1.1 |
| type | 1 | nfs_ftype4 | R | Section 5.8.1.2 |
| fh_expire_type | 2 | uint32_t | R | Section 5.8.1.3 |
| change | 3 | changeid4 | R | Section 5.8.1.4 |
| size | 4 | uint64_t | R W | Section 5.8.1.5 |
| link_support | 5 | bool | R | Section 5.8.1.6 |
| symlink_support | 6 | bool | R | Section 5.8.1.7 |
| named_attr | 7 | bool | R | Section 5.8.1.8 |
| fsid | 8 | fsid4 | R | Section 5.8.1.9 |
| unique_handles | 9 | bool | R | Section 5.8.1.10 |
| lease_time | 10 | nfs_lease4 | R | Section 5.8.1.11 |
| rdattr_error | 11 | nfsstat4 | R | Section 5.8.1.12 |
| filehandle | 19 | nfs_fh4 | R | Section 5.8.1.13 |
+-----------------+----+------------+-----+-------------------+
Table 3: REQUIRED Attributes5.7. RECOMMENDED Attributes - List and Definition References ✨
okay so the RECOMMENDED attributes are all lined up in Table 4 🎀 the column headers mean the same thing they did in Table 3, so if you forget, peek at Section 5.6 💅
+-------------------+----+-----------------+-----+------------------+
| Name | ID | Data Type | Acc | Defined in |
+-------------------+----+-----------------+-----+------------------+
| acl | 12 | nfsace4<> | R W | Section 6.2.1 |
| aclsupport | 13 | uint32_t | R | Section 6.2.1.2 |
| archive | 14 | bool | R W | Section 5.8.2.1 |
| cansettime | 15 | bool | R | Section 5.8.2.2 |
| case_insensitive | 16 | bool | R | Section 5.8.2.3 |
| case_preserving | 17 | bool | R | Section 5.8.2.4 |
| chown_restricted | 18 | bool | R | Section 5.8.2.5 |
| fileid | 20 | uint64_t | R | Section 5.8.2.6 |
| files_avail | 21 | uint64_t | R | Section 5.8.2.7 |
| files_free | 22 | uint64_t | R | Section 5.8.2.8 |
| files_total | 23 | uint64_t | R | Section 5.8.2.9 |
| fs_locations | 24 | fs_locations4 | R | Section 5.8.2.10 |
| hidden | 25 | bool | R W | Section 5.8.2.11 |
| homogeneous | 26 | bool | R | Section 5.8.2.12 |
| maxfilesize | 27 | uint64_t | R | Section 5.8.2.13 |
| maxlink | 28 | uint32_t | R | Section 5.8.2.14 |
| maxname | 29 | uint32_t | R | Section 5.8.2.15 |
| maxread | 30 | uint64_t | R | Section 5.8.2.16 |
| maxwrite | 31 | uint64_t | R | Section 5.8.2.17 |
| mimetype | 32 | ascii_ | R W | Section 5.8.2.18 |
| | | REQUIRED4<> | | |
| mode | 33 | mode4 | R W | Section 6.2.2 |
| mounted_on_fileid | 55 | uint64_t | R | Section 5.8.2.19 |
| no_trunc | 34 | bool | R | Section 5.8.2.20 |
| numlinks | 35 | uint32_t | R | Section 5.8.2.21 |
| owner | 36 | utf8str_mixed | R W | Section 5.8.2.22 |
| owner_group | 37 | utf8str_mixed | R W | Section 5.8.2.23 |
| quota_avail_hard | 38 | uint64_t | R | Section 5.8.2.24 |
| quota_avail_soft | 39 | uint64_t | R | Section 5.8.2.25 |
| quota_used | 40 | uint64_t | R | Section 5.8.2.26 |
| rawdev | 41 | specdata4 | R | Section 5.8.2.27 |
| space_avail | 42 | uint64_t | R | Section 5.8.2.28 |
| space_free | 43 | uint64_t | R | Section 5.8.2.29 |
| space_total | 44 | uint64_t | R | Section 5.8.2.30 |
| space_used | 45 | uint64_t | R | Section 5.8.2.31 |
| system | 46 | bool | R W | Section 5.8.2.32 |
| time_access | 47 | nfstime4 | R | Section 5.8.2.33 |
| time_access_set | 48 | settime4 | W | Section 5.8.2.34 |
| time_backup | 49 | nfstime4 | R W | Section 5.8.2.35 |
| time_create | 50 | nfstime4 | R W | Section 5.8.2.36 |
| time_delta | 51 | nfstime4 | R | Section 5.8.2.37 |
| time_metadata | 52 | nfstime4 | R | Section 5.8.2.38 |
| time_modify | 53 | nfstime4 | R | Section 5.8.2.39 |
| time_modify_set | 54 | settime4 | W | Section 5.8.2.40 |
+-------------------+----+-----------------+-----+------------------+Table 4: RECOMMENDED Attributes 🌸
5.8. Attribute Definitions 🦋
5.8.1. Definitions of REQUIRED Attributes
5.8.1.1. Attribute 0: supported_attrs
this is the bit vector that gets back every REQUIRED and RECOMMENDED attribute that's supported for this object 🧚 the scope of this attribute covers all objects with a matching fsid, so it's bigger than just one file, bestie 👑
5.8.1.2. Attribute 1: type
this one designates the type of an object using one of a bunch of special constants 💖
- NF4REG designates a regular file.
- NF4DIR designates a directory.
- NF4BLK designates a block device special file.
- NF4CHR designates a character device special file.
- NF4LNK designates a symbolic link.
- NF4SOCK designates a named socket special file.
- NF4FIFO designates a fifo special file.
- NF4ATTRDIR designates a named attribute directory.
- NF4NAMEDATTR designates a named attribute.
when we're in the explanatory text and the operation descriptions, these phrases get used with the meanings below, so keep them straight 🎀
- The phrase "is a directory" means that the object's type attribute is NF4DIR or NF4ATTRDIR.
- The phrase "is a special file" means that the object's type attribute is NF4BLK, NF4CHR, NF4SOCK, or NF4FIFO.
- The phrase "is a regular file" means that the object's type attribute is NF4REG or NF4NAMEDATTR.
- The phrase "is a symbolic link" means that the object's type attribute is NF4LNK.
5.8.1.3. Attribute 2: fh_expire_type
the server uses this to tell the client how filehandle expiration behaves 🔥 see Section 4 for the extra details.
5.8.1.4. Attribute 3: change
this is a value the server creates so the client can tell whether file data, directory contents, or attributes of the object have changed 🦋 the server MAY return the object's time_metadata attribute for this value, but only if the file system object can't be updated more often than the resolution of time_metadata. ✨
5.8.1.5. Attribute 4: size
the size of the object in bytes 🍓
5.8.1.6. Attribute 5: link_support
TRUE, if the object's file system supports hard links. 🔗
5.8.1.7. Attribute 6: symlink_support
TRUE, if the object's file system supports symbolic links. 🌈
5.8.1.8. Attribute 7: named_attr
TRUE, if this object has named attributes. in other words, this object has a non-empty named attribute directory 💜
5.8.1.9. Attribute 8: fsid
this is the unique file system identifier for the file system holding this object 🫶 the fsid attribute has major and minor components, and each of them is of data type uint64_t.
5.8.1.10. Attribute 9: unique_handles
TRUE, if two distinct filehandles are guaranteed to refer to two different file system objects. no mix-ups, periodt 💅
5.8.1.11. Attribute 10: lease_time
the duration of the lease at the server, in seconds ⭐
5.8.1.12. Attribute 11: rdattr_error
the error returned when an attempt to retrieve attributes during a READDIR operation goes wrong 💀
5.8.1.13. Attribute 19: filehandle
the filehandle of this object (mainly for READDIR requests) 🧚
5.8.2. Definitions of Uncategorized RECOMMENDED Attributes
okay so here are the definitions for most of the RECOMMENDED attributes ✨ collections that share a common category get defined in other sections, so don't panic if one's missing here 🌸
5.8.2.1. Attribute 14: archive
TRUE, if this file has been archived since the time of the last modification (deprecated in favor of time_backup). it's giving old-school energy 🕰️
5.8.2.2. Attribute 15: cansettime
TRUE, if the server is able to change the times for a file system object as specified in a SETATTR operation. ⏰
5.8.2.3. Attribute 16: case_insensitive
TRUE, if filename comparisons on this file system are case insensitive. this only refers to comparisons, and not to the case in which filenames are stored 🔤
5.8.2.4. Attribute 17: case_preserving
TRUE, if the filename case on this file system is preserved. this only refers to how filenames are stored, and not to how they are compared. filenames stored in mixed case might be compared using either case-insensitive or case-sensitive comparisons, so it's a lil messy 🙈
5.8.2.5. Attribute 18: chown_restricted
if TRUE, the server will reject any request to change either the owner or the group associated with a file if the caller is not a privileged user (for example, "root" in UNIX operating environments or the "Take Ownership" privilege in Windows 2000). no sneaking ownership changes past the server, bestie 🚫
5.8.2.6. Attribute 20: fileid
a number that uniquely identifies the file within the file system 🆔
5.8.2.7. Attribute 21: files_avail
file slots available to this user on the file system containing this object. this should be the smallest relevant limit 📁
5.8.2.8. Attribute 22: files_free
free file slots on the file system containing this object. this should be the smallest relevant limit 🗂️
5.8.2.9. Attribute 23: files_total
total file slots on the file system containing this object 📚
5.8.2.10. Attribute 24: fs_locations
this is where this file system may be found 🗺️ if the server returns NFS4ERR_MOVED as an error, this attribute MUST be supported, so don't skip it, bestie 😭
the server specifies the rootpath for a given server by returning a path made of zero path components 🌱
5.8.2.11. Attribute 25: hidden
TRUE, if the file is considered hidden with respect to the Windows API. shh, it's a secret 🤫
5.8.2.12. Attribute 26: homogeneous
TRUE, if this object's file system is homogeneous, i.e., all objects in the file system (all objects on the server with the same fsid) have common values for all per-file system attributes. everything's the same, no drama 🧁
5.8.2.13. Attribute 27: maxfilesize
the maximum supported file size for the file system of this object 🐘
5.8.2.14. Attribute 28: maxlink
the maximum number of hard links for this object 🔗
5.8.2.15. Attribute 29: maxname
the maximum filename size supported for this object 📝
5.8.2.16. Attribute 30: maxread
the maximum amount of data the READ operation will return for this object 📖
5.8.2.17. Attribute 31: maxwrite
the maximum amount of data the WRITE operation will accept for this object. this attribute SHOULD be supported if the file is writable. without it, the client can end up wasting bandwidth or missing out on the best performance, and nobody wants that 🚀
5.8.2.18. Attribute 32: mimetype
the MIME media type/subtype of this object 🎨
5.8.2.19. Attribute 55: mounted_on_fileid 🦋
okay so mounted_on_fileid is basically fileid's sibling, except if the target filehandle is the root of a file system, it gives you the fileid of the underlying directory instead 🫶
UNIX-based operating environments hook a file system into the namespace by mounting it onto an existing file object (the mount point, usually a directory) of another file system. When you read the mount point's parent directory with readdir() [readdir_api], the results are directory entries, each with a component name and a fileid. The fileid on the mount point's directory entry will be different from the one the stat() [stat] system call returns. ngl this is the plot twist 😭: stat() is returning the fileid of the root of the mounted file system, while readdir() is returning the fileid that stat() would have returned before any file systems were mounted on the mount point.
Unlike NFSv3, NFSv4.0 lets a client's LOOKUP request cross other file systems. The client detects the file system crossing whenever the filehandle argument of LOOKUP has an fsid attribute different from that of the filehandle returned by LOOKUP. A UNIX-based client will call this a "mount point crossing" 🧚 UNIX also has a legacy scheme for letting a process figure out its current working directory. It relies on readdir() of a mount point's parent and stat() of the mount point returning fileids as we just described. The mounted_on_fileid attribute corresponds to the fileid that readdir() would have returned, as described previously.
While an NFSv4.0 client could just fabricate a fileid matching what mounted_on_fileid provides (and if the server doesn't support mounted_on_fileid, the client has no choice), there's a risk the client generates a fileid that conflicts with one already assigned to another object in the file system. 😬 Instead, if the server can provide mounted_on_fileid, the potential for client operational problems in this area is eliminated ✨
If the server detects that there is nothing mounted on top of the target file object, then the mounted_on_fileid value it returns is the same as the fileid attribute.
The mounted_on_fileid attribute is RECOMMENDED, so the server SHOULD provide it if possible, and for a UNIX-based server this is straightforward 💅 Usually mounted_on_fileid gets requested during a READDIR operation, and in that case it's trivial (at least for UNIX-based servers) to return it, since it equals the fileid of a directory entry returned by readdir(). If mounted_on_fileid is requested in a GETATTR operation, the server should obey an invariant: it returns a value equal to the file object's entry in the object's parent directory, i.e., what readdir() would have returned. Some operating environments allow a series of two or more file systems to be mounted onto a single mount point. In that case, for the server to obey the aforementioned invariant, it will need to find the base mount point, and not the intermediate mount points 🌸
5.8.2.20. Attribute 34: no_trunc 🙈
If this attribute is TRUE, then if the client uses a filename longer than name_max, an error will be returned instead of the name being truncated. no more silent cut-offs, bestie 💀
5.8.2.21. Attribute 35: numlinks 🔗
Number of hard links to this object.
5.8.2.22. Attribute 36: owner 👑
The string name of the owner of this object.
5.8.2.23. Attribute 37: owner_group 🫂
The string name of the group ownership of this object.
5.8.2.24. Attribute 38: quota_avail_hard 🍓
The value in bytes that represents the amount of additional disk space beyond the current allocation that can be allocated to this file or directory before further allocations will be refused. It is understood that this space may be consumed by allocations to other files or directories.
5.8.2.25. Attribute 39: quota_avail_soft 🌈
The value in bytes that represents the amount of additional disk space that can be allocated to this file or directory before the user may reasonably be warned. It is understood that this space may be consumed by allocations to other files or directories, though there may exist server-side rules as to which other files or directories.
5.8.2.26. Attribute 40: quota_used 💜
The value in bytes that represents the amount of disk space used by this file or directory, and possibly a number of other similar files or directories, where the set of "similar" meets at least the criterion that allocating space to any file or directory in the set will reduce the quota_avail_hard of every other file or directory in the set.
Note that there may be a number of distinct but overlapping sets of files or directories for which a quota_used value is maintained, e.g., "all files with a given owner", "all files with a given group owner", etc. The server is at liberty to choose any of those sets when providing the content of the quota_used attribute, but should do so in a repeatable way. The rule may be configured per file system, or may be "choose the set with the smallest quota" 🧠
5.8.2.27. Attribute 41: rawdev 🎀
Raw device number of file of type NF4BLK or NF4CHR. The device number is split into major and minor numbers. If the file's type attribute is not NF4BLK or NF4CHR, this attribute SHOULD NOT be returned, and any value returned SHOULD NOT be considered useful.
5.8.2.28. Attribute 42: space_avail 🌷
Disk space in bytes available to this user on the file system containing this object -- this should be the smallest relevant limit.
5.8.2.29. Attribute 43: space_free 🫧
Free disk space in bytes on the file system containing this object -- this should be the smallest relevant limit.
5.8.2.30. Attribute 44: space_total 🍰
Total disk space in bytes on the file system containing this object.
5.8.2.31. Attribute 45: space_used 🧁
Number of file system bytes allocated to this object.
5.8.2.32. Attribute 46: system 🪟
TRUE, if this file is a "system" file with respect to the Windows operating environment.
5.8.2.33. Attribute 47: time_access ⏰
Represents the time of last access to the object by a READ operation sent to the server. The notion of what is an "access" depends on the server's operating environment and/or the server's file system semantics. For example, for servers obeying Portable Operating System Interface (POSIX) semantics, time_access would be updated only by the READ and READDIR operations, and not any of the operations that modify the content of the object [read_api], [readdir_api], [write_api]. Of course, setting the corresponding time_access_set attribute is another way to modify the time_access attribute 💭
Whenever the file object resides on a writable file system, the server should make its best efforts to record time_access into stable storage. However, to mitigate the performance effects of doing so, and most especially whenever the server is satisfying the read of the object's content from its cache, the server MAY cache access time updates and lazily write them to stable storage. It is also acceptable to give administrators of the server the option to disable time_access updates 🙌
5.8.2.34. Attribute 48: time_access_set 🎯
Sets the time of last access to the object. SETATTR use only.
5.8.2.35. Attribute 49: time_backup 📦
The time of last backup of the object.
5.8.2.36. Attribute 50: time_create 🐣
The time of creation of the object. This attribute does not have any relation to the traditional UNIX file attribute "ctime" ("change time"). not the same vibe, so don't mix them up 🙃
5.8.2.37. Attribute 51: time_delta 🔍
Smallest useful server time granularity.
5.8.2.38. Attribute 52: time_metadata 🗂️
The time of last metadata modification of the object.
5.8.2.39. Attribute 53: time_modify 🖊️
The time of last modification to the object.
5.8.2.40. Attribute 54: time_modify_set ✏️
Sets the time of last modification to the object. SETATTR use only.
5.9. Interpreting owner and owner_group 💖
okay so the RECOMMENDED attributes owner and owner_group (plus the users and groups that show up as the who field inside nfs4ace structures in the acl attribute) are all written as UTF-8 strings ✨. The reason is that this format doesn't tie you to one specific way of representing things on the client or the server, which is lowkey genius 🧚. Note that Section 6.1 of [RFC2624] gives more rationale if you want the deep dive. Each side is expected to keep its own local version of owners and groups for storage or for showing things to apps through APIs that expect that. So when these attributes travel between client and server, the protocol says the local version gets translated into a string shaped like identifier@dns_domain 🎀. That way, clients and servers that don't share a local format can still interoperate, because they both speak the same syntax and both can read it 🫶.
Security principals work the same way, just with a twist 🦋. Different security mechanisms can represent them differently. Servers usually translate those representations into one common format, typically the one their local storage uses, so they can tell which users each principal belongs to. When those local identifiers get translated into the form of the owner attribute for files created by those principals, they show who each set of principals belongs to, all in one shared format 💅.
The translation used to read owner and group strings isn't specified by the protocol, so people can get creative with solutions 🌈. For example, a local translation table might map a numeric identifier to the user@dns_domain syntax. A name service can also do the job. A server can offer a more general service that isn't limited to any particular translation (which would only translate a limited set of possible strings) by storing the owner and owner_group attributes in local storage without translating them at all. Or it can mix methods, storing the entire string for attributes that have no translation available while using the local representation for the ones that do 👑.
Servers that don't support every possible user and group string SHOULD return an error (NFS4ERR_BADOWNER) when they get a string with no translation. That goes for a value to be set by a SETATTR of the owner or owner_group attributes, or part of the value of the acl attribute 😭. When a server accepts a user or group string as valid on a SETATTR, it's promising to return that same string (see below) when a matching GETATTR happens, as long as nothing else has changed that attribute in the meantime. There are some internationalization exceptions where this just isn't possible, and those are covered below. Configuration changes (including changes to how the string maps to the local representation) and badly built name translations (the ones with aliasing) can make that promise impossible to keep. Servers should make a real effort to avoid changing these attribute values when nothing has actually changed about ownership or ACLs 🙈.
The "dns_domain" part of the owner string is meant to be a DNS domain name, like user@example.org 🍓. Servers should accept a set of users for at least one domain. A server can treat other domains as having no valid translations. A more general service becomes possible when a server can accept users for multiple domains, or for all domains, subject to security constraints ✨.
As an implementation tip, both clients and servers can provide a way to configure the "dns_domain" part of the owner string 💜. For example, the NFS server's host might have the DNS domain name lab.example.org, while the user names are defined in example.org. Without any such configuration, or as a default, the server's current DNS domain name should be the value used for "dns_domain" 🌸.
As mentioned above, it's really desirable for a server to return the same user@domain or group@domain string when it accepts one in an attribute and that attribute is fetched later 🔥. Internationalization issues make that impossible in some situations, and the client needs to know about them. See Section 12 for a detailed discussion of these issues 🧐.
When there's no translation available to the client or server, the attribute value gets built without the "@" 💀. So if the "@" is missing from the owner or owner_group attribute, that means no translation was available at the sender, and the receiver shouldn't use that string as the basis for translating into its own internal format. Even though it can't be translated, the string may still be useful. A client can use the attribute string to display ownership locally 🦋.
To be more compatible with NFSv3, which identified users and groups by 32-bit unsigned user identifiers and group identifiers, owner and group strings made of ASCII-encoded decimal numeric values with no leading zeros can get a special interpretation. This is for clients and servers that choose to support it 🎀. The receiver may treat such a user or group string as representing the same user that an NFSv3 uid or gid with that numeric value would represent 🧚.
A server SHOULD reject such a numeric value if the security mechanism is using Kerberos 🙅♀️. That's because in that situation, the client will already need to form user@domain strings. For any other security mechanism, the server SHOULD accept such numeric values. As an implementation note, the server could make that acceptance configurable 💅. If the server doesn't support numeric values, or if it's configured to turn them off, then it MUST return an NFS4ERR_BADOWNER error 😤. If the security mechanism is using Kerberos and the client tries the special form anyway, then the server SHOULD return an NFS4ERR_BADOWNER error when there's a valid translation for the user or owner designated that way. In that case, the client must use the proper user@domain string instead of the special form, for compatibility 🫶.
The client MUST always accept numeric values if the security mechanism is not RPCSEC_GSS 🌈. A client can figure out whether a server supports numeric identifiers by first trying to send one. If that attempt gets rejected with an NFS4ERR_BADOWNER error, then the client should only use named identifiers of the form user@dns_domain 🍓.
The owner string nobody may be used to designate an anonymous user 👻. That user gets associated with a file created by a security principal that can't be mapped to the owner attribute through normal means 🥺.
5.10. Character Case Attributes 🔤
With the case_insensitive and case_preserving attributes in mind, case-insensitive comparisons of Unicode characters SHOULD use Unicode Default Case Folding, as defined in Chapter 3 of the Unicode Standard [UNICODE] 🌟. They MAY override that behavior for specific selected characters with the case folding defined in the SpecialCasing.txt [SPECIALCASING] file. For more on that, see Section 3.13 of the Unicode Standard 🧠.
The SpecialCasing.txt file swaps Default Case Folding for locale- and context-dependent case folding in specific situations ✨. One example of locale- and context-dependent case folding is that LATIN CAPITAL LETTER I (I, U+0049) gets default case folded to LATIN SMALL LETTER I (i, U+0069). But several languages (e.g., Turkish) treat an "I" with a dot as a different letter from an "I" without a dot 🙈. So in those languages, unless an I comes before a dot_above, the I (U+0049) character should be case folded to a different character instead, which is LATIN SMALL LETTER DOTLESS I (U+0131) 💜.
The [UNICODE] and [SPECIALCASING] references in this RFC are for version 7.0.0 of the Unicode standard, since that was the latest version when this RFC got published 📚. Implementations SHOULD always use the latest version of Unicode (http://www.unicode.org/versions/latest/) 🦋.
6. Access Control Attributes 🔐
Access Control Lists (ACLs) are file attributes that specify fine-grained access control 👑. This section covers the acl, aclsupport, and mode file attributes, plus how they interact with each other 💅. Note that file attributes may apply to any file system object 🌸.
6.1. Goals 🎯
ACLs and modes are two well-established models for specifying permissions, and they're both kinda iconic in their own way ✨. This section specifies requirements that try to meet the following goals:
- If a server supports the
modeattribute, it should provide reasonable semantics to clients that only set and retrieve themodeattribute. - If a server supports ACL attributes, it should provide reasonable semantics to clients that only set and retrieve those attributes.
- On servers that support the
modeattribute, if ACL attributes have never been set on an object, either explicitly or via inheritance, the behavior should be traditional UNIX-like behavior 🐧. - On servers that support the
modeattribute, if the ACL attributes have been previously set on an object, either explicitly or via inheritance:- Setting only the
modeattribute should effectively control the traditional UNIX-like permissions of read, write, and execute onowner,owner_group, and other 💖. - Setting only the
modeattribute should provide reasonable security. For example, setting a mode of000should be enough to ensure that future opens for read or write by any principal fail, regardless of a previously existing or inherited ACL 😤.
- Setting only the
- When a
modeattribute is set on an object, the ACL attributes may need to be modified so they don't conflict with the new mode 🥹. In such cases, it's desirable for the ACL to keep as much information as possible. That includes information about inheritance, AUDIT and ALARM access control entries (ACEs), and permissions that are granted and denied without conflicting with the new mode 🫶.
6.2. File Attributes Discussion 📝
Support for each of the ACL attributes is RECOMMENDED and not required, since file systems accessed using NFSv4 might not support ACLs 🙃.
6.2.1. Attribute 12: acl 💅✨
okay bestie, the NFSv4.0 ACL attribute is an array of ACEs (access control entries) attached to each file system object 🦋 Clients can read AND write the acl attribute, but the server is the one actually using it for access control 👑 If a client just wants to check access without reading or changing any data or metadata, it can use the OPEN or ACCESS operations 🫶
The NFS ACE structure is defined like this:
typedef uint32_t acetype4;
typedef uint32_t aceflag4;
typedef uint32_t acemask4;
struct nfsace4 {
acetype4 type;
aceflag4 flag;
acemask4 access_mask;
utf8str_mixed who;
};Now the processing rules, which are kind of the main character moment of this whole section 🎀 To figure out if a request succeeds, the server goes through each nfsace4 entry in order. Only ACEs whose "who" matches the requester get considered. Each ACE gets processed until all of the bits of the requester's access have been ALLOWED. Once a bit (see below) has been ALLOWED by an ACCESS_ALLOWED_ACE, it is no longer considered in the processing of later ACEs. If an ACCESS_DENIED_ACE shows up where the requester's access still has unALLOWED bits in common with the "access_mask" of the ACE, the request is denied 🚫 And when the whole ACL has been processed, if there are bits in the requester's mask that were never ALLOWED or DENIED, access is denied too 💀
Unlike the ALLOW and DENY ACE types, the ALARM and AUDIT ACE types don't change a requester's access at all. They're there to trigger events when a requester tries to get in 🔔 So AUDIT and ALARM ACEs only get processed after the ALLOW and DENY ACEs are done 🧚
The NFSv4.0 ACL model is seriously rich 🌈 Some server platforms have access control that goes beyond the UNIX-style mode attribute, but it's not as rich as the NFS ACL model. So that users can still use that more limited functionality, the server may support the acl attributes by mapping between its ACL model and the NFSv4.0 ACL model. Servers must ensure that the ACL they actually store or enforce is at least as strict as the NFSv4 ACL that was set. It's tempting to do this by rejecting any ACL that falls outside the small set that can be represented accurately, but that can make ACLs unusable without special client-side knowledge of the server's mapping, which kind of defeats the whole point of having a common NFSv4 ACL protocol 😤 So servers should accept every ACL that they can without compromising security. To help with that, servers may make a special exception, for unsupported permission bits, to the rule that bits not ALLOWED or DENIED by an ACL must be denied. For example, a UNIX-style server might choose to silently allow read attribute permissions even though an ACL doesn't explicitly allow them. (An ACL that explicitly denies permission to read attributes should still result in a denial 🙅♀️)
The situation gets complicated because a server may have multiple modules that enforce ACLs. For example, enforcement for NFSv4.0 access may be different from, but not weaker than, enforcement for local access, and both may be different from enforcement for access through other protocols such as Server Message Block (SMB) [MS-SMB]. So it may be useful for a server to accept an ACL even if not all of its modules can support it 🧩
The guiding principle for NFSv4 access is that the server must not accept ACLs that give an appearance of more restricted access to a file than what's actually enforced. Bestie, this one is non-negotiable 🔥
6.2.1.1. ACE Type
The constants used for the type field (acetype4) are these:
const ACE4_ACCESS_ALLOWED_ACE_TYPE = 0x00000000;
const ACE4_ACCESS_DENIED_ACE_TYPE = 0x00000001;
const ACE4_SYSTEM_AUDIT_ACE_TYPE = 0x00000002;
const ACE4_SYSTEM_ALARM_ACE_TYPE = 0x00000003;All four ACE types are permitted in the acl attribute 🌸
+------------------------------+--------------+---------------------+
| Value | Abbreviation | Description |
+------------------------------+--------------+---------------------+
| ACE4_ACCESS_ALLOWED_ACE_TYPE | ALLOW | Explicitly grants |
| | | the access defined |
| | | in acemask4 to the |
| | | file or directory. |
| | | |
| ACE4_ACCESS_DENIED_ACE_TYPE | DENY | Explicitly denies |
| | | the access defined |
| | | in acemask4 to the |
| | | file or directory. |
| | | |
| ACE4_SYSTEM_AUDIT_ACE_TYPE | AUDIT | LOG (in a system- |
| | | dependent way) any |
| | | access attempt to a |
| | | file or directory |
| | | that uses any of |
| | | the access methods |
| | | specified in |
| | | acemask4. |
| | | |
| ACE4_SYSTEM_ALARM_ACE_TYPE | ALARM | Generate a system |
| | | ALARM (system |
| | | dependent) when any |
| | | access attempt is |
| | | made to a file or |
| | | directory for the |
| | | access methods |
| | | specified in |
| | | acemask4. |
+------------------------------+--------------+---------------------+The "Abbreviation" column is the nickname each type goes by for the rest of this section, so keep an eye on it 👀
6.2.1.2. Attribute 13: aclsupport
A server doesn't have to support all of the above ACE types 🤷♀️ This attribute tells you which ACE types are supported for the current file system. The bitmask constants used to represent the definitions above inside the aclsupport attribute are:
const ACL4_SUPPORT_ALLOW_ACL = 0x00000001;
const ACL4_SUPPORT_DENY_ACL = 0x00000002;
const ACL4_SUPPORT_AUDIT_ACL = 0x00000004;
const ACL4_SUPPORT_ALARM_ACL = 0x00000008;Servers that support either the ALLOW or DENY ACE type SHOULD support both the ALLOW and DENY ACE types 🎀
Clients should not attempt to set an ACE unless the server claims support for that ACE type. If the server gets a request to set an ACE that it can't store, it MUST reject the request with NFS4ERR_ATTRNOTSUPP 🚫 If the server gets a request to set an ACE that it can store but can't enforce, the server SHOULD reject the request with NFS4ERR_ATTRNOTSUPP 😭
6.2.1.3. ACE Access Mask
Last but definitely not least, the access mask 💖 The bitmask constants used for the access mask field are:
const ACE4_READ_DATA = 0x00000001;
const ACE4_LIST_DIRECTORY = 0x00000001;
const ACE4_WRITE_DATA = 0x00000002;
const ACE4_ADD_FILE = 0x00000002;
const ACE4_APPEND_DATA = 0x00000004;
const ACE4_ADD_SUBDIRECTORY = 0x00000004;
const ACE4_READ_NAMED_ATTRS = 0x00000008;
const ACE4_WRITE_NAMED_ATTRS = 0x00000010;
const ACE4_EXECUTE = 0x00000020;
const ACE4_DELETE_CHILD = 0x00000040;
const ACE4_READ_ATTRIBUTES = 0x00000080;
const ACE4_WRITE_ATTRIBUTES = 0x00000100;
const ACE4_DELETE = 0x00010000;
const ACE4_READ_ACL = 0x00020000;
const ACE4_WRITE_ACL = 0x00040000;
const ACE4_WRITE_OWNER = 0x00080000;
const ACE4_SYNCHRONIZE = 0x00100000;Some masks have coincident values, which is kind of a plot twist 🙈 For example, ACE4_READ_DATA and ACE4_LIST_DIRECTORY are both 0x00000001. The mask entries ACE4_LIST_DIRECTORY, ACE4_ADD_FILE, and ACE4_ADD_SUBDIRECTORY are meant to be used with directory objects, while ACE4_READ_DATA, ACE4_WRITE_DATA, and ACE4_APPEND_DATA are meant to be used with non-directory objects 🧚✨
6.2.1.3.1. Discussion of Mask Attributes ✨
okay so this is where we get into what each access mask bit actually does 💖 I'll list the operations each one touches, then the tea on what it means.
ACE4_READ_DATA 🦋
- Operation(s) affected:
- READ
- OPEN
- Discussion: Permission to read the data of the file. Servers SHOULD let a user read the file's data when only the
ACE4_EXECUTEbit is set, bestie.
ACE4_LIST_DIRECTORY 📂
- Operation(s) affected:
- READDIR
- Discussion: Permission to list what's inside a directory. 🌸
ACE4_WRITE_DATA ✍️
- Operation(s) affected:
- WRITE
- OPEN
- SETATTR of size
- Discussion: Permission to modify a file's data. 💅
ACE4_ADD_FILE 🧚
- Operation(s) affected:
- CREATE
- LINK
- OPEN
- RENAME
- Discussion: Permission to add a new file to a directory. The CREATE operation is affected when nfs_ftype4 is
NF4LNK,NF4BLK,NF4CHR,NF4SOCK, orNF4FIFO. (NF4DIRisn't listed because it's covered byACE4_ADD_SUBDIRECTORY, ngl that's so organized 🗂️) OPEN is affected when it's used to create a regular file. LINK and RENAME are always affected. 🔥
ACE4_APPEND_DATA 📝
- Operation(s) affected:
- WRITE
- OPEN
- SETATTR of size
- Discussion: This is the ability to modify a file's data, but only starting at EOF. It's how you get append-only files: give a user or group
ACE4_APPEND_DATAand deny themACE4_WRITE_DATA, and boom, they can add to the end but can't rewrite the middle 🙈 If a file has an ACL like that and a WRITE request comes in for somewhere other than EOF, the server should returnNFS4ERR_ACCESS. Main character energy for the file, honestly ✨
ACE4_ADD_SUBDIRECTORY 🌳
- Operation(s) affected:
- CREATE
- RENAME
- Discussion: Permission to create a subdirectory in a directory. CREATE is affected when nfs_ftype4 is
NF4DIR. RENAME is always affected. 💖
ACE4_READ_NAMED_ATTRS 🏷️
- Operation(s) affected:
- OPENATTR
- Discussion: Permission to read the named attributes of a file, or to look up the named attributes directory. OPENATTR is affected when it's not being used to create a named attribute directory. That's when 1) createdir is TRUE but a named attribute directory already exists, or 2) createdir is FALSE. 🫶
ACE4_WRITE_NAMED_ATTRS 🎀
- Operation(s) affected:
- OPENATTR
- Discussion: Permission to write the named attributes of a file, or to create a named attribute directory. OPENATTR is affected when it's being used to create a named attribute directory, which is when createdir is TRUE and none exists yet. Being able to check whether one exists depends on being able to look it up, so users also need
ACE4_READ_NAMED_ATTRSto create one. It's a whole chain, iykyk 🔗
ACE4_EXECUTE (the read-side one) 🏃♀️
- Operation(s) affected:
- READ
- Discussion: Permission to execute a file.
Servers SHOULD let a user read the data of the file when only the ACE4_EXECUTE bit is set. Honestly it makes sense, because there's literally no way to execute a file without reading its contents 🤷♀️ Servers may treat ACE4_EXECUTE and ACE4_READ_DATA the same when deciding whether to permit a READ, but they SHOULD still let the two bits be set independently in ACLs. They MUST tell them apart when replying to ACCESS operations. In particular, servers SHOULD NOT silently flip one bit on when the other is set, because then the client couldn't correctly enforce the difference between read and execute permissions. That would be so messy 😭
Here's an example. After a SETATTR of this ACL:
nfsuser:ACE4_EXECUTE:ALLOWa later GETATTR of ACL for that file SHOULD return:
nfsuser:ACE4_EXECUTE:ALLOWand not:
nfsuser:ACE4_EXECUTE/ACE4_READ_DATA:ALLOWACE4_EXECUTE (the directory-side one) 🚪
- Operation(s) affected:
- LOOKUP
- OPEN
- REMOVE
- RENAME
- LINK
- CREATE
- Discussion: Permission to traverse or search a directory. It's the "walk through the hallway" bit 🚶♀️
ACE4_DELETE_CHILD 🗑️
- Operation(s) affected:
- REMOVE
- RENAME
- Discussion: Permission to delete a file or directory inside a directory. See Section 6.2.1.3.2 for how ACE4_DELETE and ACE4_DELETE_CHILD interact. It's a whole relationship, bestie 💜
ACE4_READ_ATTRIBUTES 🔍
- Operation(s) affected:
- GETATTR of file system object attributes
- VERIFY
- NVERIFY
- READDIR
- Discussion: The ability to read the basic (non-ACL) attributes of a file. On a UNIX system, those are the stat-level attributes. With this bit, the entity can run "ls -l" and stat ✨ If a READDIR operation requests attributes, this mask must be allowed for the READDIR to succeed.
ACE4_WRITE_ATTRIBUTES ⏰
- Operation(s) affected:
- SETATTR of time_access_set, time_backup, time_create, time_modify_set, mimetype, hidden, and system
- Discussion: Permission to change the times associated with a file or directory to an arbitrary value. It also covers changing the mimetype, hidden, and system attributes. A user with ACE4_WRITE_DATA or ACE4_WRITE_ATTRIBUTES can set a file's times to the current server time, which is giving "good enough" 🕰️
ACE4_DELETE 💀
- Operation(s) affected:
- REMOVE
- Discussion: Permission to delete the file or directory. See Section 6.2.1.3.2 for how ACE4_DELETE and ACE4_DELETE_CHILD interact. 😤
ACE4_READ_ACL 👀
- Operation(s) affected:
- GETATTR of acl
- NVERIFY
- VERIFY
- Discussion: Permission to read the ACL. 📖
ACE4_WRITE_ACL 🔐
- Operation(s) affected:
- SETATTR of acl and mode
- Discussion: Permission to write the acl and mode attributes. Big power move 👑
ACE4_WRITE_OWNER 🫅
- Operation(s) affected:
- SETATTR of owner and owner_group
- Discussion: Permission to write the owner and owner_group attributes. On UNIX systems, this is the ability to run chown() and chgrp(). 🦋
ACE4_SYNCHRONIZE 🔄
- Operation(s) affected:
- NONE
- Discussion: Permission to use the file object as a synchronization primitive for interprocess communication. This permission isn't enforced or interpreted by the NFSv4.0 server on the client's behalf.
Typically, ACE4_SYNCHRONIZE only means something on local file systems, meaning file systems not accessed via NFSv4.0. The bit exists because some operating environments, like Windows, use ACE4_SYNCHRONIZE 🪟
Here's a fun one. If a client copies a file that has ACE4_SYNCHRONIZE set from a local file system to an NFSv4.0 server, and later copies it back from the server to a local file system, the client will probably want the bit set on that second copy, if it was set on the original. The first copy won't have the permission set unless the NFSv4.0 server can set the ACE4_SYNCHRONIZE bit. The second copy won't have it set unless the server can retrieve the bit. It's a bit of a bestie chain of custody situation 🧳
Server implementations don't have to offer the granularity this list of masks suggests. For example, POSIX-based systems might not tell ACE4_APPEND_DATA (the ability to append to a file) apart from ACE4_WRITE_DATA (the ability to modify existing contents). Both masks would be tied to one "write" permission. When a server like that returns attributes to the client, it would show both ACE4_APPEND_DATA and ACE4_WRITE_DATA if and only if the write permission is enabled. Two for one, no cap 🎁
If a server gets a SETATTR request it can't implement accurately, it should err on the side of more restricted access, except in the execute and read cases we already talked about. For example, say a server can't tell overwriting data apart from appending new data, like we just covered. If a client submits an ALLOW ACE where ACE4_APPEND_DATA is set but ACE4_WRITE_DATA isn't (or the other way around), the server should either turn off ACE4_APPEND_DATA or reject the request with NFS4ERR_ATTRNOTSUPP. Safety first, periodt 🛡️
6.2.1.3.2. ACE4_DELETE versus ACE4_DELETE_CHILD 🗝️
okay this one is the drama of deletion, and I'm obsessed 😭 Two access mask bits control whether a directory entry can be deleted: ACE4_DELETE on the object itself (the "target"), and ACE4_DELETE_CHILD on the directory that contains it (the "parent").
Lots of systems also use the "sticky bit" (MODE4_SVTX) on a directory, which only lets a user unlink if they own either the target or the parent. On some of those systems, the decision also depends on whether the target is writable. Layers, bestie 🧅
Servers SHOULD allow unlink if either ACE4_DELETE is permitted on the target or ACE4_DELETE_CHILD is permitted on the parent. (Note that this still holds even if the parent or the target explicitly denies the other permission. Wild, I know 🙃)
If the ACLs in question neither explicitly ALLOW nor DENY either of the above, and MODE4_SVTX isn't set on the parent, then the server SHOULD allow the removal if and only if ACE4_ADD_FILE is permitted. When MODE4_SVTX is set, the server may also require the remover to own either the parent or the target, or may require the target to be writable. 🫶
This lets servers support something pretty close to traditional UNIX-style semantics, with ACE4_ADD_FILE taking the place of the write bit. It's giving nostalgia 🕹️
6.2.1.4. ACE flag 🚩
Here are the bitmask constants for the flag field:
const ACE4_FILE_INHERIT_ACE = 0x00000001;
const ACE4_DIRECTORY_INHERIT_ACE = 0x00000002;
const ACE4_NO_PROPAGATE_INHERIT_ACE = 0x00000004;
const ACE4_INHERIT_ONLY_ACE = 0x00000008;
const ACE4_SUCCESSFUL_ACCESS_ACE_FLAG = 0x00000010;
const ACE4_FAILED_ACCESS_ACE_FLAG = 0x00000020;
const ACE4_IDENTIFIER_GROUP = 0x00000040;A server doesn't have to support any of these flags at all. If the server has flags that are similar to these but not exactly the same, the implementation may define a mapping between the protocol-defined flags and its own flags. 🗺️
For example, say a client tries to set an ACE with ACE4_FILE_INHERIT_ACE set but not ACE4_DIRECTORY_INHERIT_ACE. If the server doesn't support any form of ACL inheritance, it should reject the request with NFS4ERR_ATTRNOTSUPP. If the server has a single "inherit ACE" flag that applies to both files and directories, it may reject the request (so the client has to set both the file and directory inheritance flags). It may also accept the request and quietly turn on the ACE4_DIRECTORY_INHERIT_ACE flag. Options, options 💁♀️
6.2.1.4.1. Discussion of Flag Bits ✨
Okay bestie, let's talk flag bits 🎀 These little guys control how an ACE behaves, so pay attention 💅
ACE4_FILE_INHERIT_ACE: any non-directory file in any subdirectory will get this ACE inherited 🍓ACE4_DIRECTORY_INHERIT_ACE: this one can go on a directory and says that the ACE should be added to each new directory that gets created 🌸 If this flag is set in an ACE inside an ACL attribute that's going on a non-directory file system object, the operation trying to set that ACL SHOULD fail withNFS4ERR_ATTRNOTSUPP. Not it 💀ACE4_INHERIT_ONLY_ACE: this can go on a directory, but it doesn't apply to that directory itself 🦋 ALLOW and DENY ACEs with this bit set don't affect access to the directory, and AUDIT and ALARM ACEs with this bit set don't trigger log or alarm events. These only kick in once they're applied (with this bit cleared) to newly created files and directories, as the two flags above describe ✨ If this flag is on an ACE but neitherACE4_DIRECTORY_INHERIT_ACEnorACE4_FILE_INHERIT_ACEis there, then an operation trying to set that attribute SHOULD fail withNFS4ERR_ATTRNOTSUPP. Main character energy has limits, periodt 🙈ACE4_NO_PROPAGATE_INHERIT_ACE: also goes on a directory 🧚 It tells the server that inheritance of this ACE should stop at newly created child directories.ACE4_SUCCESSFUL_ACCESS_ACE_FLAGandACE4_FAILED_ACCESS_ACE_FLAG(SUCCESS and FAILED for short) 🔥 These two bits can only be set on ACE4_SYSTEM_AUDIT_ACE_TYPE (AUDIT) and ACE4_SYSTEM_ALARM_ACE_TYPE (ALARM) ACE types. Here's the tea: while the server is going through the file's ACL, if it hits an AUDIT or ALARM ACE that matches the principal trying to OPEN, it makes a note of that and notes whether SUCCESS and/or FAILED was set on it. Once ACL processing is done, the server checks whether the operation succeeded or failed. If it succeeded and SUCCESS was set on the matching AUDIT or ALARM ACE, the matching AUDIT or ALARM event happens. If it failed and FAILED was set on that matching ACE, the matching event happens too. You can set either or both of SUCCESS and FAILED, but if you set neither, the AUDIT or ALARM ACE is not useful at all 💅
The processing I just described also applies to ACCESS operations, even when they return NFS4_OK. For AUDIT and ALARM purposes, an ACCESS operation counts as a "failure" if it fails to return a bit that was requested and supported ✨
ACE4_IDENTIFIER_GROUP: this one says the "who" is a GROUP as defined under UNIX, or a GROUP ACCOUNT as defined under Windows 👑 Clients and servers MUST ignore theACE4_IDENTIFIER_GROUPflag on ACEs whose who value equals one of the special identifiers in Section 6.2.1.5. No exceptions, no cap 🫶
6.2.1.5. ACE Who 🌈
Okay so the who field of an ACE is an identifier for the principal (or principals) the ACE applies to 🦋 It can point at a user or a group, and the flag bit ACE4_IDENTIFIER_GROUP is what tells you which one 💖
There are a few special identifiers that everybody needs to understand universally, not just inside one particular DNS domain 🎀 Some of them can't be figured out when an NFS client is accessing the server, but they do mean something when a local process is accessing the file. Being able to display and modify these permissions is allowed over NFS, even if none of the server's access methods understand the identifiers 🌸
+---------------+---------------------------------------------------+
| Who | Description |
+---------------+---------------------------------------------------+
| OWNER | The owner of the file. |
| GROUP | The group associated with the file. |
| EVERYONE | The world, including the owner and owning group. |
| INTERACTIVE | Accessed from an interactive terminal. |
| NETWORK | Accessed via the network. |
| DIALUP | Accessed as a dialup user to the server. |
| BATCH | Accessed from a batch job. |
| ANONYMOUS | Accessed without any authentication. |
| AUTHENTICATED | Any authenticated user (opposite of ANONYMOUS). |
| SERVICE | Access from a system service. |
+---------------+---------------------------------------------------+
Table 5: Special IdentifiersTo avoid any mix-ups 🙈 these special identifiers get an appended "@" and should show up in the form xxxx@ (with no domain name after the "@"). For example, ANONYMOUS@ ✨
The ACE4_IDENTIFIER_GROUP flag MUST be ignored on entries with these special identifiers 💅 When you encode entries with these special identifiers, the ACE4_IDENTIFIER_GROUP flag SHOULD be set to zero 🍓
6.2.1.5.1. Discussion of EVERYONE@ 💖
It's really important to know that EVERYONE@ is NOT the same as the UNIX "other" entity 😭 That's because, by definition, UNIX "other" doesn't include the owner or owning group of a file. EVERYONE@ means literally everyone, owner and owning group included 🫶
6.2.2. Attribute 33: mode 🎀
Okay, the NFSv4.0 mode attribute is based on the UNIX mode bits 🦋 Here are the bits that are defined:
const MODE4_SUID = 0x800; /* set user id on execution */
const MODE4_SGID = 0x400; /* set group id on execution */
const MODE4_SVTX = 0x200; /* save text even after use */
const MODE4_RUSR = 0x100; /* read permission: owner */
const MODE4_WUSR = 0x080; /* write permission: owner */
const MODE4_XUSR = 0x040; /* execute permission: owner */
const MODE4_RGRP = 0x020; /* read permission: group */
const MODE4_WGRP = 0x010; /* write permission: group */
const MODE4_XGRP = 0x008; /* execute permission: group */
const MODE4_ROTH = 0x004; /* read permission: other */
const MODE4_WOTH = 0x002; /* write permission: other */
const MODE4_XOTH = 0x001; /* execute permission: other */Bits MODE4_RUSR, MODE4_WUSR, and MODE4_XUSR apply to the principal identified in the owner attribute 👑 Bits MODE4_RGRP, MODE4_WGRP, and MODE4_XGRP apply to principals identified in the owner_group attribute, but only if they're NOT identified in the owner attribute 🌈 Bits MODE4_ROTH, MODE4_WOTH, and MODE4_XOTH apply to any principal that doesn't match the owner attribute AND doesn't have a group matching the owner_group attribute 🧚
Bits in the mode other than the ones above aren't defined by this protocol 💀 A server MUST NOT return bits other than those defined above in a GETATTR or READDIR operation, and it MUST return NFS4ERR_INVAL if bits other than those defined above are set in a SETATTR, CREATE, OPEN, VERIFY, or NVERIFY operation. Strict, but honestly iconic 💅
6.3. Common Methods 🌸
The requirements in this section will be referred to in future sections, especially Section 6.4 ✨
6.3.1. Interpreting an ACL 🦋
6.3.1.1. Server Considerations
The server uses the algorithm described in Section 6.2.1 to figure out whether an ACL allows access to an object 🔥 However, the ACL isn't always the only thing that decides access. For example:
- In the case of a file system exported as read-only, the server may deny write permissions even though an object's ACL grants them 🙈
- Server implementations MAY grant ACE4_WRITE_ACL and ACE4_READ_ACL permissions, to prevent a situation where there's no valid way to ever modify the ACL 🫶
- All servers will allow a user to read the data of the file when only the execute permission is granted. (So if the ACL denies the user ACE4_READ_DATA access and allows the user ACE4_EXECUTE, the server will still let them read the data of the file 🍓)
- Many servers have the notion of owner-override, where the owner of the object is allowed to override accesses that the ACL denies 🌈 This can be helpful, for example, to let users keep access to files they already have open when the permissions have changed ✨
- Many servers have the notion of a "superuser" with privileges beyond an ordinary user 👑 The superuser may be able to read or write data or metadata in ways the ACL wouldn't allow 💖
6.3.1.2. Client Considerations
Clients SHOULD NOT do their own access checks based on their interpretation of the ACL 🙅 Instead they should use the OPEN and ACCESS operations to do access checks 🎀 That way the client can act on the server's decision about whether access should be granted, based on the server's own reading of the ACL 🦋
Clients must be aware that there are situations where an object's ACL defines a certain access even though the server won't have enough information to enforce it 😭 For example, the server has no way of telling whether a particular OPEN is a user opening a file for read access or is part of executing the file. In those situations, the client needs to do its part in enforcing access as the ACL defines it 🔥 To do that, the client sends the appropriate ACCESS operation (or uses a cached previous determination) before servicing the user's or application's request, so it can work out whether that user or application should get the requested access 💅 For examples of ACLs defining accesses the server doesn't enforce, see Section 6.3.1.1 🌈
6.3.2. Computing a mode Attribute from an ACL 🧚
Here's a method you can use to calculate the MODE4_R*, MODE4_W*, and MODE4_X* bits of a mode attribute, based on an ACL ✨
First, for each of the special identifiers OWNER@, GROUP@, and EVERYONE@, evaluate the ACL in order, considering only ALLOW and DENY ACEs for the identifier EVERYONE@ and for the identifier under consideration. The result of that evaluation is an NFSv4 ACL mask showing exactly which bits are permitted to that identifier 🍓
Then translate the calculated mask for OWNER@, GROUP@, and EVERYONE@ into mode bits for the user, group, and other, respectively, like this:
- Set the read bit (MODE4_RUSR, MODE4_RGRP, or MODE4_ROTH) if and only if ACE4_READ_DATA is set in the corresponding mask 📖
- Set the write bit (MODE4_WUSR, MODE4_WGRP, or MODE4_WOTH) if and only if ACE4_WRITE_DATA and ACE4_APPEND_DATA are both set in the corresponding mask 🫶
- Set the execute bit (MODE4_XUSR, MODE4_XGRP, or MODE4_XOTH) if and only if ACE4_EXECUTE is set in the corresponding mask 🔥
6.3.2.1. Discussion
Some server implementations also add bits permitted to named users and groups to the group bits (MODE4_RGRP, MODE4_WGRP, and MODE4_XGRP) 👀
Implementations are discouraged from doing this 🙈 It's been found to confuse users, who see members of a file's group denied access even though the mode bits appear to allow it. (DENY ACEs can cause that same confusion too, but DENY ACEs are expected to be used less often 💀)
The same user confusion shows up when fetching the mode, and also if setting the mode doesn't actually control permissions for the owner, group, and other users. That's what motivates some of the requirements that follow 💖
6.4. Requirements 💅
okay so the server that supports both mode and ACL has to keep the MODE4_*USR, MODE4_*GRP, and MODE4_*OTH bits in sync with the ACEs whose who fields are "OWNER@", "GROUP@", and "EVERYONE@" 🎀 that way the client sees the same semantically equivalent permissions whether it asks for just the ACL or for the owner, owner_group, and mode attributes ✨
many requirements point back to Section 6.3.2, but notice that the methods have behaviors written with "SHOULD" 🙈 that's on purpose, so existing implementations that compute the mode from the withdrawn POSIX ACL draft ([P1003.1e]), instead of from actual permissions on owner, group, and other, don't get invalidated 🫶
6.4.1. Setting the mode and/or ACL Attributes
6.4.1.1. Setting mode and Not ACL
when any of the nine low-order mode bits change because the mode attribute was set, and no ACL attribute is explicitly set, the acl attribute must be modified to match the updated value of those bits 😭 this has to happen even if the low-order bits end up the same after the mode is set as they were before 💀
any AUDIT or ALARM ACEs are unaffected by changes to the mode, just so you know 🌸
when the permission bits are subject to change, the acl attribute MUST be modified so that the mode computed via the method in Section 6.3.2 gives back the low-order nine bits (MODE4_R*, MODE4_W*, MODE4_X*) of the mode attribute as modified by the change attribute 👑 the ACL attributes SHOULD also be modified so that:
- if
MODE4_RGRPis not set, entities explicitly listed in the ACL other than OWNER@ and EVERYONE@ SHOULD NOT be granted ACE4_READ_DATA 🦋 - if
MODE4_WGRPis not set, entities explicitly listed in the ACL other than OWNER@ and EVERYONE@ SHOULD NOT be granted ACE4_WRITE_DATA or ACE4_APPEND_DATA 🍓 - if
MODE4_XGRPis not set, entities explicitly listed in the ACL other than OWNER@ and EVERYONE@ SHOULD NOT be granted ACE4_EXECUTE 🧚
access mask bits other than the ones listed above, showing up in ALLOW ACEs, MAY also be disabled 🌈
ACEs with the flag ACE4_INHERIT_ONLY_ACE set don't affect the permissions of the ACL itself, and neither do ACEs of the types AUDIT and ALARM 💜 so it's honestly desirable to leave these ACEs alone when you're modifying the ACL attributes ✨
also, the requirement can be met by throwing out the acl and replacing it with an ACL that represents the mode and only the mode 💅 that's permitted, but it's preferable for a server to keep as much of the ACL as it can without breaking the requirements above 🫶 tossing the ACL makes it effectively impossible for a file created with a mode attribute to inherit an ACL (see Section 6.4.3) 💀
6.4.1.2. Setting ACL and Not mode
when you set the acl and don't set the mode attribute, the permission bits of the mode have to be derived from the ACL 🔥 in this case the ACL attribute SHOULD be set as given 🎀 the nine low-order bits of the mode attribute (MODE4_R*, MODE4_W*, MODE4_X*) MUST be modified to match the result of the method in Section 6.3.2 💖 the three high-order bits of the mode (MODE4_SUID, MODE4_SGID, MODE4_SVTX) SHOULD remain unchanged 🌸
6.4.1.3. Setting Both ACL and mode
when you set both the mode and the acl attribute in the same operation, the attributes MUST be applied in this order: mode, then ACL 👑 the mode-related attribute gets set as given, then the ACL attribute gets set as given, which may change the final mode, as described above in Section 6.4.1.2 ✨
6.4.2. Retrieving the mode and/or ACL Attributes
this section only applies to servers that support both the mode and ACL attributes 🫶
some server implementations have a concept of "objects without ACLs" 🧚 that means all permissions are granted and denied according to the mode attribute, and no ACL attribute is stored for that object 💅 if an ACL attribute is requested from a server like that, the server SHOULD return an ACL that doesn't conflict with the mode, aka the ACL returned SHOULD represent the nine low-order bits of the mode attribute (MODE4_R*, MODE4_W*, MODE4_X*) as described in Section 6.3.2 🦋
for other server implementations, the ACL attribute is always present for every object 🌈 those servers SHOULD store at least the three high-order bits of the mode attribute (MODE4_SUID, MODE4_SGID, MODE4_SVTX) 💜 the server SHOULD return a mode attribute if one is requested, and the low-order nine bits of the mode (MODE4_R*, MODE4_W*, MODE4_X*) MUST match the result of applying the method in Section 6.3.2 to the ACL attribute 🍓
6.4.3. Creating New Objects
if a server supports any ACL attributes, it may use the ACL attributes on the parent directory to compute an initial ACL attribute for a newly created object 🧚 this gets called the inherited ACL in this section 🌸 adding one or more ACEs to the inherited ACL that are based on ACEs in the parent directory's ACL is called inheriting an ACE in this section ✨
whether or not the mode and ACL attributes are present, the behavior of CREATE and OPEN SHOULD be:
- if just the mode is given in the call:
in this case, inheritance SHOULD take place, but the mode MUST be applied to the inherited ACL as described in Section 6.4.1.1, which modifies the ACL 💖
- if just the ACL is given in the call:
in this case, inheritance SHOULD NOT take place, the ACL as defined in the CREATE or OPEN gets set without modification, and the mode gets modified as in Section 6.4.1.2 🎀
- if both mode and ACL are given in the call:
in this case, inheritance SHOULD NOT take place, and both attributes get set as described in Section 6.4.1.3 👑
- if neither mode nor ACL is given in the call:
when an object is created without any initial attributes at all, e.g., an OPEN operation with an opentype4 of OPEN4_CREATE and a createmode4 of EXCLUSIVE4, inheritance SHOULD NOT take place 😭 instead, the server SHOULD set permissions to deny all access to the newly created object 🙈 it's expected that the appropriate client will set the desired attributes in a later SETATTR operation, and the server SHOULD allow that operation to succeed, regardless of what permissions the object was created with 🫶 for example, an empty ACL denies all permissions, but the server should allow the owner's SETATTR to succeed even though WRITE_ACL is implicitly denied 💅
in other cases, inheritance SHOULD take place, and no modifications to the ACL happen 🌈 the mode attribute, if supported, MUST be as computed via the method in Section 6.3.2, with the MODE4_SUID, MODE4_SGID, and MODE4_SVTX bits clear 💜 if there are no inheritable ACEs on the parent directory, the rules for creating acl attributes are implementation defined 🦋
6.4.3.1. The Inherited ACL
if the object being created is not a directory, the inherited ACL SHOULD NOT inherit ACEs from the parent directory ACL unless the ACE4_FILE_INHERIT_FLAG is set 🍓
if the object being created is a directory, the inherited ACL should inherit all inheritable ACEs from the parent directory, i.e., the ones that have the ACE4_FILE_INHERIT_ACE or ACE4_DIRECTORY_INHERIT_ACE flag set 🧚 if the inheritable ACE has ACE4_FILE_INHERIT_ACE set but ACE4_DIRECTORY_INHERIT_ACE is clear, the inherited ACE on the newly created directory MUST have the ACE4_INHERIT_ONLY_ACE flag set 🌸 that keeps the directory from being affected by ACEs meant for non-directories ✨
when a new directory is created, the server MAY split any inherited ACE that is both inheritable and effective (meaning it has neither ACE4_INHERIT_ONLY_ACE nor ACE4_NO_PROPAGATE_INHERIT_ACE set) into two ACEs 💖 one has no inheritance flags, and the other has ACE4_INHERIT_ONLY_ACE set 🎀 this makes it simpler to change the effective permissions on the directory without modifying the ACE that gets inherited to the new directory's children 👑
7. NFS Server Namespace 🦋
7.1. Server Exports
on a UNIX server, the namespace describes all the files reachable by pathnames under the root directory or "/" 🌈 on a Windows server, the namespace is all the files on disks named by mapped disk letters 💜 NFS server administrators rarely make the entire server's file system namespace available to NFS clients 🍓 more often, portions of the namespace are made available via an "export" feature ✨ in previous versions of the NFS protocol, the root filehandle for each export is obtained through the MOUNT protocol 🧚 the client sends a string that identifies an object in the exported namespace, and the server returns the root filehandle for it 🌸 the MOUNT protocol supports an EXPORTS procedure that enumerates the server's exports 💅
7.2. Browsing Exports
the NFSv4 protocol provides a root filehandle that clients can use to get filehandles for these exports via a multi-component LOOKUP 🫶 a super common user experience is a graphical user interface (perhaps a file "Open" dialog window) to find a file by browsing progressively through a directory tree 💖 the client must be able to move from one export to another via single-component, progressive LOOKUP operations 👑
this style of browsing isn't well supported by the NFSv2 and NFSv3 protocols 😭 the client expects all LOOKUP operations to stay within a single-server file system, so the device attribute won't change, for example 🙈 that stops a client from taking namespace paths that span exports 💀
an automounter on the client can get a snapshot of the server's namespace using the EXPORTS procedure of the MOUNT protocol 🎀 if it understands the server's pathname syntax, it can create an image of the server's namespace on the client 🦋 the parts of the namespace that aren't exported by the server get filled in with a "pseudo-file system" that lets the user browse from one mounted file system to another 🌈 the catch is this representation of the server's namespace on the client is static 🥲 if the server administrator adds a new export, the client won't know about it 😭
7.3. Server Pseudo-File System
NFSv4 servers avoid this namespace inconsistency by presenting all the exports within the framework of a single-server namespace 🧚 an NFSv4 client uses LOOKUP and READDIR operations to browse seamlessly from one export to another ✨ portions of the server namespace that aren't exported get bridged via a "pseudo-file system" that gives a view of exported directories only 💜 a pseudo-file system has a unique fsid and behaves like a normal, read-only file system 🫶
based on how the server's namespace is built, multiple pseudo-file systems may exist 🍓 for example:
/a pseudo-file system
/a/b real file system
/a/b/c pseudo-file system
/a/b/c/d real file systemeach of the pseudo-file systems is considered a separate entity, so each one gets its own unique fsid 💅✨
7.4. Multiple Roots 🧚
Okay so DOS and Windows get described as having "multiple roots," bc file systems usually show up as disk letters over there. MacOS does its own thing and shows file systems as top-level names 🍓. NFSv4 servers for these platforms can build a pseudo-file system above those root names, so disk letters or volume names just end up being directory names in the pseudo-root. Main character energy for the namespace 💅✨
7.5. Filehandle Volatility 🦋
The whole point of the server's pseudo-file system is that it's a logical view of the file system(s) the server makes available. So it's most likely built dynamically when the server first gets instantiated, and it might not have an on-disk counterpart that persistent filehandles could be built from. ngl that's a bit of a plot twist 🙈 It's preferable for the server to give persistent filehandles for the pseudo-file system, but the NFS client should still expect those filehandles to be volatile. To check, look at the fh_expire_type attribute for the filehandles in question. If they're volatile, the NFS client must be ready to recover a filehandle value (for example, with a multi-component LOOKUP) whenever it gets an NFS4ERR_FHEXPIRED error. Be prepared, bestie 🫶
7.6. Exported Root 🌸
If the server's root file system is exported, you might think "cool, no pseudo-file system needed!" 🙃 Literally wrong. Imagine a server with these file systems:
/ disk1 (exported)
/a disk2 (not exported)
/a/b disk3 (exported)Since disk2 isn't exported, disk3 can't be reached with simple LOOKUPs. The server has to bridge that gap with a pseudo-file system 🌉
7.7. Mount Point Crossing 🎀
Sometimes the server's file system environment is set up so that one file system has a directory that's "covered" or mounted upon by a second file system. For example:
/a/b (file system 1)
/a/b/c/d (file system 2)The pseudo-file system for this server might end up looking like:
/ (placeholder/not exported)
/a/b (file system 1)
/a/b/c/d (file system 2)It's the server's job to present a complete pseudo-file system to the client 👑 If the client sends a LOOKUP request for the path /a/b/c/d, the server responds with the filehandle of the file system /a/b/c/d. In previous versions of the NFS protocol, the server would have responded with the filehandle of the directory /a/b/c/d inside the file system /a/b. Different vibes, same path 😭
The NFS client can tell it's crossing a server mount point when the value of the fsid attribute changes ✨
7.8. Security Policy and Namespace Presentation 🔐
NFSv4 clients can change the security mechanisms they use, after figuring out what's allowed by using SECINFO. So the server SHOULD NOT show a different view of the namespace depending on which security mechanism a client is using. It should show a consistent view, and return NFS4ERR_WRONGSEC if someone tries to access data with an inappropriate security mechanism. Consistency is iconic 💅
Sometimes security means you need to hide that a particular file system exists, not just the data inside it. In that case, the server can apply the security policy of a shared resource to the components of that resource's ancestors. For example:
/ (placeholder/not exported)
/a/b (file system 1)
/a/b/MySecretProject (file system 2)The /a/b/MySecretProject directory is a real file system, and it's the shared resource. Say the security policy for /a/b/MySecretProject is Kerberos with integrity, and you want to limit anyone knowing this file system exists. The server should apply that same security policy to /a/b. That way, knowing a file system exists can be locked down when you want it to be 🔒
If the server's resources use multiple, disjoint security mechanisms, applying that kind of policy would leave the higher-level file system unreachable with any security flavor. So that setup doesn't work with hiding existence (as opposed to contents) from clients who use multiple disjoint sets of security flavors 🙅♀️
In other situations, a good policy is for the security of an object in the server's namespace to include the union of all security mechanisms of all its direct descendants. A common, convenient move, unless strong security requirements say otherwise, is to make the whole pseudo-file system accessible by every valid security mechanism 🌈
When there's concern about data security on the network, clients should use strong security mechanisms to access the pseudo-file system, so they can prevent man-in-the-middle attacks. Lowkey the most important part of this whole section 🚨
8. Multi-Server Namespace 🌍
NFSv4 supports attributes that let a namespace stretch beyond the boundaries of a single server. It is RECOMMENDED that clients and servers support building these multi-server namespaces. Using them is optional, though, and for lots of purposes single-server namespaces are perfectly fine 💖 Multi-server namespaces can still bring a lot of perks, because they separate a file system's logical position in the namespace from the (possibly changing) logistical and administrative stuff that decides which servers particular file systems live on 🦋
8.1. Location Attributes 📍
NFSv4 has RECOMMENDED attributes that let file systems on one server be associated with one or more instances of that file system on other servers. These attributes point at those file system instances by naming a server address target (either a DNS name that stands for one or more IP addresses, or a literal IP address), plus the path of that file system within the associated single-server namespace.
The fs_locations RECOMMENDED attribute lets you say where the data for a given file system can be found 🗺️
8.2. File System Presence or Absence 🫥
A given location in an NFSv4 namespace (usually a multi-server namespace, but not always) can have a bunch of file system instance locations attached to it through the fs_locations attribute. There might also be an actual current file system at that location, reachable through normal namespace operations (like LOOKUP). When that's the case, the file system is "present" at that spot in the namespace, and clients will typically use it. The other locations from the location-related attributes are saved for when the main location isn't available anymore 🙌
When there's no actual file system at the namespace location, the file system is "absent." An absent file system has no files or directories except the root. Any reference to it will result in an error, NFS4ERR_MOVED, except for accessing a small set of attributes that help figure out alternative locations. Note that if the server ever returns the error NFS4ERR_MOVED, it MUST support the fs_locations attribute ✅
The error name makes it sound like a file system was once present and then became absent. That's just one possibility, though 👀 A position in the namespace can be permanently absent, with the file system(s) named by the location attributes being the only "real" version. The name NFS4ERR_MOVED reflects an earlier, narrower idea of what it does, but the error comes back whenever the referenced file system is absent, whether it moved or never existed at all 🤷♀️
Except for GETATTR-type operations (more on those later), when the current filehandle at the start of an operation is inside an absent file system, that operation doesn't get performed. The error NFS4ERR_MOVED comes back instead, to say the file system is absent on the current server 🚫
Because a GETFH can't succeed when the current filehandle is inside an absent file system, filehandles in an absent file system can't be transferred to the client. If a client does have filehandles inside an absent file system, it got them back when the file system was present, and then the file system became absent later 🕰️
Heads up: since the check for the current filehandle being inside an absent file system happens at the start of every operation, operations that change the current filehandle so it lands inside an absent file system won't error out. That's what makes combos like PUTFH-GETATTR and LOOKUP-GETATTR work for grabbing attribute info, especially location attribute info, which we'll get into below 💜
8.3. Getting Attributes for an Absent File System 🔍
When a file system is absent, most attributes aren't available. But the client still needs access to the small set of attributes that are available, especially the one that tells it the correct current locations for this file system: fs_locations 🧭
8.3.1. GETATTR within an Absent File System ✨
okay so GETATTR gets a special exception 💅 you can grab attributes for a filehandle sitting inside an absent file system, but only if your attribute mask includes at least the fs_locations bit 🫶 that bit tells the server you actually care about the absent file system. if you don't ask for it, GETATTR just throws NFS4ERR_MOVED at you 💀
when you GETATTR on an absent file system, the list of supported attributes is super limited 😭 lots of attributes, including the ones that are normally REQUIRED, just won't be around. besides fs_locations, these ones SHOULD be available on absent file systems 🌸 for the RECOMMENDED ones, they should be available at least as much as they are on present file systems ✨
fsid: this one should be provided so the client can figure out where the file system boundaries are, especially the line between present and absent file systems 🧚 this value must be different from any other fsid on the current server, and it doesn't need any particular relationship to fsids on whatever destination the client might get sent to 🦋mounted_on_fileid: for objects at the top of an absent file system, this one needs to be available 🎀 the fileid lives in the present parent file system, so there should be no need to reference the absent file system to give you this info 💜
other attributes SHOULD NOT be made available for absent file systems, even when it's possible to provide them 🙈 the server should not assume more info is always better, and it should avoid gratuitously providing extra information. overshare much? 💅
when a GETATTR includes a bitmask for fs_locations, but the bitmask also includes attributes that aren't supported, GETATTR won't error out 🌈 instead it returns the mask of the actual supported attributes along with the results.
VERIFY/NVERIFY is kind of like GETATTR here 🧚 if the attribute mask doesn't include fs_locations, you get NFS4ERR_MOVED. the difference is that any attribute in the mask that isn't supported for an absent file system (and yes, that includes some normally REQUIRED attributes) will also cause an NFS4ERR_MOVED result 😭
8.3.2. READDIR and Absent File Systems 🍓
a READDIR done while the current filehandle is inside an absent file system will give you NFS4ERR_MOVED, since READDIR doesn't get the GETATTR exception 💀 no cap.
you can still fetch attributes for an absent file system through a READDIR on a directory in a present file system, as long as that directory contains the root directories of one or more absent file systems 👑 here's how it goes:
- if the attribute set you ask for includes
fs_locations, the attributes get fetched normally, and you won't get anNFS4ERR_MOVEDindication even if you requestrdattr_error🫶 - if the attribute set doesn't include
fs_locations, but you do requestrdattr_error, then each directory entry for the root of an absent file system reportsNFS4ERR_MOVEDas the value ofrdattr_error🙈 - if the attribute set includes neither
fs_locationsnorrdattr_error, then the root of an absent file system showing up in the directory makes the READDIR fail with anNFS4ERR_MOVEDerror 😭 - an attribute being unavailable because the file system is absent doesn't throw any error, even if it's one that's ordinarily REQUIRED ✨ the attributes returned for the root directory of the absent file system are just limited to the ones that are actually available 🌸
8.4. Uses of Location Information 🌍
okay the location-bearing attribute fs_locations, together with the possibility of absent file systems, unlocks a bunch of important stuff for reliable, manageable, and scalable data access 🔥
when a file system is present, these attributes can give you alternative locations to get to the same data, for when server failures, communication problems, or other stuff makes continued access impossible or just impractical 😭 sometimes you can even use several alternative locations at once for higher-performance access 🚀 providing those alternative locations is called "replication", even though sometimes the replicas aren't actually there and they're just different paths to the same data 🫠
when a file system is present and then becomes absent, clients can get the chance to keep accessing their data at an alternative location 🦋 moving the file system contents to the new spot is called "migration" 💅 see Section 8.4.2 for details.
alternative locations can be physical replicas of the file system data, or alternative communication paths to the same server, or (with some server clustering) another server giving access to the same physical file system ✨ how the client handles the transition depends on what kind of new access path it is and whether the data actually got migrated. we'll go over all of this in detail below 🧚
when a file system wasn't previously present, specifying its location lets a file system on one server get associated with a namespace defined by another server 🌈 that's how you get a general multi-server namespace facility 👑 putting that location in place of an absent file system is called a "referral" 💜
because client support for location-related attributes is OPTIONAL, a server may (but doesn't have to) hide migration and referral events from those clients 🎀 it can do that by acting as a proxy, for example 🫶
8.4.1. File System Replication 🪞
the fs_locations attribute gives alternative locations you can use to access data in place of the current file system instance, or in addition to it 💖 the first time you touch a file system, the client should grab the set of alternative locations by checking the fs_locations attribute 🔥
if server failures, communication problems, or other difficulties make the current file system impossible or impractical to reach, the client can use those alternative locations to keep getting its data 🌸 multiple locations can even be used at the same time for higher performance, by exploiting multiple paths between the client and the target file system ⭐
multiple server addresses might correspond to the same actual server 🧚 that's true whether they come from one entry with a DNS name that represents a set of IP addresses, or from multiple entries that each have their own server address 🦋
8.4.2. File System Migration 🦢
when a file system is present and becomes absent, clients can get the chance to keep accessing their data at an alternative location, as specified by fs_locations ✨ usually a client is busy accessing the file system, gets an NFS4ERR_MOVED error, and then uses fs_locations to find where the data lives now 🫶
migration is super helpful for load balancing or general resource reallocation 💅 the protocol doesn't say how the file system gets moved between servers 🙃 lots of different server-to-server transfer mechanisms might get used, and the server implementer gets to pick 🎀 NFSv4 does specify how the migration event gets communicated between client and server 💜
when an alternative location is designated as a migration target, it must designate the same data 🔥 if file systems are writable, a change made on the original file system must be visible on all migration targets 🌈 if a file system isn't writable but is a read-only copy (maybe one that gets periodically updated) of a writable file system, similar requirements apply to propagating updates 🧚 any change visible in the original file system must already be in effect on all migration targets 😭 that way, when a client moves over to the migration target, it won't see any reversion in file system state 💖
8.4.3. Referrals 🌸
referrals let you put a file system somewhere in the namespace basically without caring where it physically sits on a given server 🌍 this lets one server, or a set of servers, present a multi-server namespace that covers file systems on lots of different servers 🔥 likely uses include site-wide or organization-wide namespaces, or even knitting those together into a truly global namespace ✨
referrals happen when a client figures out, the first time it references a spot in the current namespace, that the spot is part of a new file system and that the file system is absent 👑 usually that shows up as an NFS4ERR_MOVED error, and the client can then find the actual location or locations by fetching fs_locations 🫶
the location-related attribute can point at a single file system location or at multiple ones, and the client picks based on what it needs 🦋
using multi-server namespaces is enabled by NFSv4, but it's not required 🙈 how much you use them and their scope depend on the applications you run and what your sysadmins prefer 💅
multi-server namespaces can be set up by a single server that hands out a big set of referrals to all of the included file systems 🧚 or a single multi-server namespace can be split up administratively, with separate referral file systems (on separate servers) for each separately administered chunk of the namespace 🌈 the top-level referral file system, or any segment, can use replicated referral file systems for higher availability 💜
generally, multi-server namespaces are mostly uniform, meaning the same data a client sees at a given location in the namespace is what every client sees at that location 🫶
8.5. Location Entries and Server Identity ✨
okay so, remember how we said earlier that one location entry can point at a server using a DNS name 🌸 and that name can secretly stand for a bunch of different IP addresses? And also that several location entries can point at the very same server? Yeah, that's the vibe here.
When a server has multiple addresses, the client may assume that for each file system in the namespace of a given server network address, there exist file systems at corresponding namespace locations for each of the other server network addresses. It may do this even if nothing is explicitly listed in fs_locations 💅 Those corresponding file system locations can be used as alternative locations, just like the ones that are spelled out in the fs_locations attribute. Iconic behavior, honestly.
If a single location entry designates multiple server IP addresses, the client should choose a single one to use. Now, when two server addresses are designated by a single location entry and they correspond to different servers, this normally indicates some sort of misconfiguration 🙈 so the client should avoid using such location entries when alternatives are available. When they are not, clients should pick one of the IP addresses and use it, without using others that are not directed to the same server. Not me having to pick just one 😭
8.6. Additional Client-Side Considerations 🎀
When clients make use of servers that implement referrals, replication, and migration, care should be taken that a user who mounts a given file system that includes a referral or a relocated file system continues to see a coherent picture of that user-side file system 🦋 even though it contains a bunch of server-side file systems that may be on different servers. Main character energy for the namespace, fr.
One important issue is upward navigation from the root of a server-side file system to its parent (specified as .. in UNIX), in the case where it transitions to that file system as a result of referral, migration, or a transition as a result of replication. When the client is at such a point and needs to ascend to the parent, it must go back to the parent as seen within the multi-server namespace rather than sending a LOOKUPP operation to the server, which would result in the parent within that server's single-server namespace. 🙃 So the client needs to remember the filehandles that represent such file system roots and use these instead of issuing a LOOKUPP operation to the current server. This will allow the client to present to applications a consistent namespace, where upward navigation and downward navigation are consistent 🧚 no weird surprises, periodt.
Another issue is refresh of referral locations, and this one is a little spicy 🌶️ When referrals are used extensively, they may change as server configurations change.
Another issue concerns refresh of referral locations. When referrals are used extensively, they may change as server configurations change. It is expected that clients will cache information related to traversing referrals so that future client-side requests are resolved locally without server communication. This is usually rooted in client-side name lookup caching. Clients should periodically purge this data for referral points in order to detect changes in location information. So basically: cache it, but don't hold on to it forever 🌈
A potential problem exists if a client were to allow an open-owner to have state on multiple file systems on a server, because it is unclear how the sequence numbers associated with open-owners are to be dealt with in the event of transparent state migration 😬 A client can avoid this whole mess if it ensures that any use of an open-owner is confined to a single file system. Keep it simple, bestie 💜
A server MAY decline to migrate state associated with open-owners that span multiple file systems. In cases in which the server chooses not to migrate such state, the server MUST return NFS4ERR_BAD_STATEID when the client uses those stateids on the new server.
The server MUST return NFS4ERR_STALE_STATEID when the client uses those stateids on the old server, regardless of whether migration has occurred or not. ⭐
8.7. Effecting File System Referrals 🦋
Referrals are effected when an absent file system is encountered and one or more alternative locations are made available by the fs_locations attribute. The client will typically get an NFS4ERR_MOVED error, fetch the appropriate location information, and proceed to access the file system on a different server, even though it retains its logical position within the original namespace. It's giving "I moved but I'm still in the same spot" 💖 Referrals differ from migration events in that they happen only when the client has not previously referenced the file system in question, so there is nothing to transition. Referrals can only come into effect when an absent file system is encountered at its root. Very specific, very important 🍓
The examples given in the sections below are somewhat artificial, in that an actual client will not typically do a multi-component lookup but will have cached information regarding the upper levels of the name hierarchy. However, these examples are chosen to make the required behavior clear and easy to put within the scope of a small number of requests, without getting unduly into details of how specific clients might choose to cache things. Real clients are a little more chill about it than the examples, but the examples make the point so clearly 🫶
8.7.1. Referral Example (LOOKUP) 🦋
Okay so let's imagine this COMPOUND gets sent in an environment where /this/is/the/path is absent from the target server 👀. There are a few reasons that could happen. The file system might have moved, or the target server might be mainly (or only) there to point clients at the servers where the various file systems actually live 💅.
PUTROOTFH
LOOKUP "this"
LOOKUP "is"
LOOKUP "the"
LOOKUP "path"
GETFH
GETATTR(fsid, fileid, size, time_modify)Here's what happens with that chain:
PUTROOTFH-->NFS_OK. The current fh is now the root of the pseudo-fs ✨.LOOKUP "this"-->NFS_OK. The current fh is for/thisand is within the pseudo-fs.LOOKUP "is"-->NFS_OK. The current fh is for/this/isand is within the pseudo-fs.LOOKUP "the"-->NFS_OK. The current fh is for/this/is/theand is within the pseudo-fs.LOOKUP "path"-->NFS_OK. The current fh is for/this/is/the/pathand is within a new, absent file system, but... the client will never see the value of that fh 🙈.GETFH-->NFS4ERR_MOVED. Big oof, this one fails 💀. The current fh is in an absent file system at the start of the operation, and the spec makes no exception forGETFH.GETATTR(fsid, fileid, size, time_modify). Not executed, because theGETFHfailure stops processing of the COMPOUND.
So after that GETFH failure, the client has to figure out where the root of the absent file system is and where to find it, meaning the server and the path relative to that server's root fh 🔍. Notice that the client here didn't grab filehandles or attributes (like fsid) for the intermediate directories. That means it isn't sure where the absent file system actually starts. Maybe /this/is/the is the root of the moved file system, and the LOOKUP "path" worked only because the file system wasn't absent during that operation but got moved between the last LOOKUP and the GETFH. (COMPOUND is not atomic, remember 🫶.) And even if we had the fsids for every intermediate directory, we still couldn't tell that /this/is/the/path was the root of a new file system, since we don't have its fsid yet.
To get the info we need, let's re-send the chain of LOOKUPs with GETFHs and GETATTRs, at least to collect the fsids so we know exactly where the file system boundaries are 🧐. The client could also ask for fs_locations at the same time, but usually it already has a good guess about the boundaries (based on where NFS4ERR_MOVED showed up and where it didn't), so grabbing fs_locations isn't necessary.
OP01: PUTROOTFH --> NFS_OK- The current fh is at the root of the pseudo-fs.
OP02: GETATTR(fsid) --> NFS_OK- Just for completeness 💖. Normally, clients already know the fsid of the pseudo-fs as soon as they connect to a server.
OP03: LOOKUP "this" --> NFS_OKOP04: GETATTR(fsid) --> NFS_OK- Grabbing the current fsid to see where the file system boundaries are. The fsid is the one for the pseudo-fs here, so there's no boundary yet.
OP05: GETFH --> NFS_OK- The current fh is for
/thisand is within the pseudo-fs.
- The current fh is for
OP06: LOOKUP "is" --> NFS_OK- The current fh is for
/this/isand is within the pseudo-fs.
- The current fh is for
OP07: GETATTR(fsid) --> NFS_OK- Checking the fsid again. Still the pseudo-fs, so still no boundary.
OP08: GETFH --> NFS_OK- The current fh is for
/this/isand is within the pseudo-fs.
- The current fh is for
OP09: LOOKUP "the" --> NFS_OK- The current fh is for
/this/is/theand is within the pseudo-fs.
- The current fh is for
OP10: GETATTR(fsid) --> NFS_OK- Checking the fsid once more. Pseudo-fs, no boundary.
OP11: GETFH --> NFS_OK- The current fh is for
/this/is/theand is within the pseudo-fs.
- The current fh is for
OP12: LOOKUP "path" --> NFS_OK- The current fh is for
/this/is/the/pathand is within a new, absent file system, but... - The client will never see the value of that fh 🙈.
- The current fh is for
OP13: GETATTR(fsid, fs_locations) --> NFS_OK- We're grabbing the fsid so we know where the file system boundaries are. This time the fsid is different from the parent directory's (which we got back in OP10). Heads up, though: the fsid we're given won't necessarily survive at the new location. That fsid could be different there, and the fsid we have for this file system might even be a valid fsid for a totally different file system on that new server 🤯.
- In this particular case, we're pretty sure what moved is
/this/is/the/pathand not/this/is/the, because we have the fsid of the latter and it matches the pseudo-fs, which presumably can't move. But in other examples we might not have that kind of info to lean on (for instance,/this/is/themight be a non-pseudo-file system separate from/this/is/the/path). So we need some other reliable source of info about where the moved file system's boundary is. If, say, the file system/this/ishad moved, we'd be looking at a migration instead of a referral, and once the boundaries of the migrated file system were clear, we could fetchfs_locations📦. - We're fetching
fs_locationsbecause gettingNFS4ERR_MOVEDat this point means this is most likely a referral, and we need the destination. Even if/this/is/thewere a file system that migrated, we'd still need the location info for that file system 💜.
OP14: GETFH --> NFS4ERR_MOVED- Fails, because the current fh is in an absent file system at the start of the operation, and the spec makes no exception for
GETFH. This means the server will never send the client a filehandle from inside an absent file system 🔒.
- Fails, because the current fh is in an absent file system at the start of the operation, and the spec makes no exception for
So, putting it all together: the client knows the root of the absent file system is /this/is/the/path, because that's where the fsid changed (between "the" and "path") 🎯. The fs_locations attribute also tells the client the actual location of the absent file system, so the referral can go through 🦋. The server gives the client the bare minimum about the absent file system, which keeps the chances of conflicting info (between the referring server and the file system's home) super low 🫶. No filehandles and very few attributes come from the referring server, so the client should treat whatever it does receive as temporary info whose only job is enabling the referral ✨.
8.7.2. Referral Example (READDIR) 🌸
Another situation where a client might run into referrals is when it does a READDIR on a directory where some of the subdirectories are roots of absent file systems 📂.
Say that directory gets read like this:
PUTROOTFH
LOOKUP "this"
LOOKUP "is"
LOOKUP "the"
READDIR(fsid, size, time_modify, mounted_on_fileid)In this case, rdattr_error isn't requested, fs_locations isn't requested, and some attributes can't be provided. So the result is an NFS4ERR_MOVED error on the READDIR, with the detailed results looking like this:
PUTROOTFH-->NFS_OK. The current fh is at the root of the pseudo-fs.LOOKUP "this"-->NFS_OK. The current fh is for/thisand is within the pseudo-fs.LOOKUP "is"-->NFS_OK. The current fh is for/this/isand is within the pseudo-fs.LOOKUP "the"-->NFS_OK. The current fh is for/this/is/theand is within the pseudo-fs.READDIR(fsid, size, time_modify, mounted_on_fileid)-->NFS4ERR_MOVED. Note that the same error would've come back if/this/is/thehad migrated, but here it's returned because the directory contains the root of an absent file system 🤓.
Now let's re-send with rdattr_error:
PUTROOTFH
LOOKUP "this"
LOOKUP "is"
LOOKUP "the"
READDIR(rdattr_error, fsid, size, time_modify, mounted_on_fileid)The results are:
PUTROOTFH-->NFS_OK. The current fh is at the root of the pseudo-fs.LOOKUP "this"-->NFS_OK. The current fh is for/thisand is within the pseudo-fs.LOOKUP "is"-->NFS_OK. The current fh is for/this/isand is within the pseudo-fs.LOOKUP "the"-->NFS_OK. The current fh is for/this/is/theand is within the pseudo-fs.READDIR(rdattr_error, fsid, size, time_modify, mounted_on_fileid)-->NFS_OK. The attributes for the directory entry named "path" will only contain rdattr_error with the valueNFS4ERR_MOVED, along with an fsid value and a value for mounted_on_fileid ✨.
Next up, let's do another READDIR to grab fs_locations. (We could've just used a GETATTR directly, like in Section 8.7.1, but hey, READDIR works too 💅.)
PUTROOTFH
LOOKUP "this"
LOOKUP "is"
LOOKUP "the"
READDIR(rdattr_error, fs_locations, mounted_on_fileid, fsid, size,
time_modify)The results would be:
PUTROOTFH-->NFS_OK. The current fh is at the root of the pseudo-fs.LOOKUP "this"-->NFS_OK. The current fh is for/thisand is within the pseudo-fs.LOOKUP "is"-->NFS_OK. The current fh is for/this/isand is within the pseudo-fs.LOOKUP "the"-->NFS_OK. The current fh is for/this/is/theand is within the pseudo-fs.READDIR(rdattr_error, fs_locations, mounted_on_fileid, fsid, size, time_modify)-->NFS_OK. The attributes come back as described below 🎀.
For the directory entry named "path", the attributes will only contain:
- rdattr_error (value:
NFS_OK) - fs_locations
- mounted_on_fileid (value: unique fileid within referring file system)
- fsid (value: unique value within referring server)
The attributes for entry "path" won't include size or time_modify, because those attributes aren't available inside an absent file system 🙈💖.
8.8. The Attribute fs_locations 🦋
okay so fs_locations is literally the attribute that tells you where a file system lives, and it's defined by both fs_location4 (Section 2.2.6) and fs_locations4 (Section 2.2.7) 💅. It gives you a server name plus the path to the root of the file system in that server's namespace. When a bunch of servers have matching file systems at the same path in their namespaces, you can give an array of server names. Each entry is a UTF-8 string, and it's one of three things: a traditional DNS host name, an IPv4 address, an IPv6 address, or a zero-length string. A zero-length string SHOULD be used to mean the current address being used for the RPC. That's giving main character energy 🎀. It is not a requirement that every server sharing the same rootpath gets listed in one fs_location4 instance. The array is just there for convenience 🌸. Servers that share the same rootpath can also show up in separate fs_location4 entries inside the fs_locations attribute ✨.
The fs_locations4 data type and the fs_locations attribute hold an array of these locations 🧚. Every server might build its namespace differently, so there's an fs_root field. The path in fs_root is the location of the file system in the current server's namespace, meaning the server the fs_locations attribute came from 👑. The fs_root is meant to help the client by clearly pointing at the root of the file system whose locations are being reported, no matter which object in that file system the current filehandle is pointing at 💜. Basically, fs_root is just the pathname the client used to reach the object on the current server, which is the object the fs_locations attribute applies to 🫶.
When you ask for fs_locations and there are no alternative file system locations, the server SHOULD return a zero-length array of fs_location4 structures, together with a valid fs_root 🍓.
Let's do an example, because this part is confusing without one 🙈. Say there's a replicated file system on two servers, servA and servB. On servA it lives at /a/b/c, and on servB it lives at /x/y/z. If the client asks for the fs_locations value for the directory at /a/b/c/d, it might not know that the file system's root is at /a/b/c in servA's namespace. When the client switches to servB, it has to figure out that the directory it first referenced on servA is now /x/y/z/d on servB 😭. To make that work, the fs_locations attribute from servA would have an fs_root value of /a/b/c and two entries in fs_locations. One entry is for itself (servA), and the other is for servB with a path of /x/y/z. With that info, the client can swap /x/y/z in for /a/b/c at the start of its access path and build /x/y/z/d to use on the new server ✨.
There's no rule that every rootpath has to have the same number of components. There's also no relationship between how many components a rootpath or fs_root has, and none of the components in any rootpath or fs_root have to match 💖. In the example above, we could have added a third element to the locations array, with server equal to "servC" and rootpath equal to "/I/II", and a fourth element with server equal to "servD" and rootpath equal to "/aleph/beth/gimel/daleth/he" 🌈.
The relationship between an fs_root and a rootpath is simple: the client swaps the pathname listed in fs_root for the current server with the one listed in the rootpath for the new server 🦋.
Here's another example, this time for a referred or migrated file system. Say a file system lives on serv1 at /az/buky/vedi/glagoli. The client finds that the object at glagoli has migrated (or is a referral). The client gets the fs_locations attribute, which has an fs_root of /az/buky/vedi/glagoli and one element in the locations array, with server equal to serv2 and rootpath equal to /izhitsa/fita. The client replaces /az/buky/vedi/glagoli with /izhitsa/fita and uses that pathname on serv2 💅.
So, bestie, the server MUST return an fs_root that is equal to the path the client used to reach the object the fs_locations attribute applies to. Otherwise the client can't work out the new path to use on the new server 😱. No cap, this is the whole point 👑.
9. File Locking and Share Reservations 🔐
okay so locking makes NFS stateful, full stop 🥺. Once you add share reservations, the protocol relies on state way more than the old NFS plus NLM (Network Lock Manager) [xnfs] combo ever did. Three things keep that state manageable 💜:
- clear division between client and server
- ability to reliably detect inconsistency in state between client and server
- simple and robust recovery mechanisms
In this model, the server owns the state information 👑. The client asks for changes to locks, and the server replies with the changes it made. Changes to locking state that the client didn't start are rare 🌸. The client gets prompt notice of those changes and can update its view of the locking state to match the server ✨.
Individual pieces of state that the server creates and hands to the client on request are represented by 128-bit stateids 🧚. These might stand for a particular open file, a set of byte-range locks held by a particular owner, or a recallable delegation of privileges to access a file in particular ways or at a particular location 🎀.
In every case, you go from the most general info, which represents a client as a whole, to the eventual lightweight stateid that most client and server locking interactions use 🦋. The details change with the type of object, but it always starts with a client ID 💖.
To support Win32 share reservations, you have to atomically OPEN or CREATE files and apply the right locks in the same operation 🔥. A separate share/unshare operation wouldn't let you implement the Win32 OpenFile API correctly 🙈. To get share semantics right, the old NFS mechanisms used when a file is opened or created (LOOKUP, CREATE, ACCESS) have to be replaced 😭. NFSv4 has an OPEN operation that subsumes the NFSv3 way of doing LOOKUP, CREATE, and ACCESS ✨. But since lots of operations need a filehandle, the traditional LOOKUP sticks around to map a filename to a filehandle without setting up any state on the server 💅. The server manages the policy for granting access or changing files based on the client's state 👑. These mechanisms can cover everything from advisory-only locking all the way up to full mandatory locking 🌈.
9.1. Opens and Byte-Range Locks 🔒
okay so manipulating a byte-range lock is rare compared to READ and WRITE operations 💖. It's also assumed that server restarts and network partitions are relatively rare 🌸. That's why READ and WRITE need a lightweight way to say whether they hold a lock 🫶. A byte-range lock request carries the heavyweight info needed to establish a lock and uniquely identify the lock's owner 🧚.
The following sections describe the move from that heavyweight info to the eventual stateid that most client and server locking and lease interactions use 🦋.
9.1.1. Client ID ✨
okay bestie, let's talk about how the server figures out who is asking for locks 🔐💖 For every LOCK request, the client must identify itself to the server. This is done in a way that makes correct lock identification and crash recovery possible 🧠 To set up that identity, you run a SETCLIENTID operation and then a SETCLIENTID_CONFIRM operation. Both are required, so no skipping steps, periodt 💅
Here's the lowkey iconic part: when a new incarnation of a client establishes its identity, it immediately breaks any leased state the previous incarnation might have left on the server 😭 The new client doesn't have to sit around waiting for those leases to expire. Breaking that lease state means the server removes all lock state, all share reservation state, and, where the server doesn't support the CLAIM_DELEGATE_PREV claim type, all delegation state tied to the same client with the same identity 🦋 If you want the details on delegation state recovery, check out Section 10.2.1 🌸
Owners of opens and owners of byte-range locks are separate entities 🎀 They stay separate even if the exact same opaque arrays are used to name each kind of owner. The protocol tells them apart: open-owners are represented by open_owner4 structures, and lock-owners are represented by lock_owner4 structures.
Both kinds of owner are made of a clientid plus an opaque owner string. For each client, the set of distinct owner values used with that client is the set of owners of that type, for that client 🧚
Each open is tied to one specific open-owner. Each byte-range lock is tied to a lock-owner and an open-owner, and that open-owner is the one linked to the open file that the LOCK operation was done under ✨
Client identification is wrapped up in this structure:
struct nfs_client_id4 {
verifier4 verifier;
opaque id<NFS4_OPAQUE_LIMIT>;
};The first field, verifier, is a client incarnation verifier. It's how the server detects client reboots 🔄 The server only starts canceling the client's leased state if the verifier is different from the one it previously recorded for that client. That client is identified by the second field of the structure, id.
The second field, id, is a variable-length string that uniquely defines the client 🫶
There are a few things to keep in mind when the client generates the id string:
- The string should be unique, so that multiple clients don't present the same string. If two clients do, the fallout can range from one client getting an error all the way to one client having its leased state abruptly and unexpectedly canceled 😱
- The string should be picked so that later incarnations (like reboots) of the same client present the same string. The implementer is warned against an approach that requires the string to be recorded in a local file 📁 Why? Because that rules out environments with no local disk, where all file access goes through an NFSv4 server 💀
- The string should be different for each server network address the client accesses, rather than the same across all of them 🌐 The reason is that the client may not be able to tell whether the same server is listening on multiple network addresses. If the client sends SETCLIENTID with the same id string to each address of such a server, the server will think it's the same client, and every successive SETCLIENTID will kick off the process of removing the client's previous leased state 🙈
- The algorithm for generating the string should not assume the client's network address won't change. That includes changes between incarnations and even changes while the client is still running in its current incarnation 🦋 So if the client only includes its own and the server's network addresses in the id string, there's a real risk. Once the client gives up that address, another client using a similar algorithm could generate a conflicting id string 😬
With all of that in mind, a well-generated id string is one that includes:
- The server's network address.
- The client's network address.
- For a user-level NFSv4 client, extra information that distinguishes it from other user-level clients running on the same host, such as a universally unique identifier (UUID) 🆔
- Additional information that tends to be unique, such as one or more of:
- The client machine's serial number. For privacy reasons, it's best to run some one-way function on the serial number 🔒
- A MAC address. For privacy reasons, it's best to run some one-way function on the MAC address 🔒
- The timestamp of when the NFSv4 software was first installed on the client. This is subject to the earlier caution about information stored in a file, because that file might only be reachable over NFSv4 🌀
- A true random number. But since this number ought to be the same between client incarnations, it has the same problem as the software-installation timestamp 🎲
🚨 Security moment: as a security measure, the server MUST NOT cancel a client's leased state if the principal that established the state for a given id string is not the same as the principal issuing the SETCLIENTID. This one is non-negotiable, bestie 🛡️💜
Quick side note: SETCLIENTID (Section 16.33) and SETCLIENTID_CONFIRM (Section 16.34) also have a secondary job. They establish the information the server needs to make callbacks to the client so delegations can work 📞 It's allowed to change this information through SETCLIENTID and SETCLIENTID_CONFIRM within the same incarnation of the client, and doing so doesn't remove the client's leased state 🌈
Once a SETCLIENTID and SETCLIENTID_CONFIRM sequence has successfully completed, the client switches to the shorthand client identifier, of type clientid4, instead of the longer and less compact nfs_client_id4 structure 👑 This shorthand (a client ID) is assigned by the server, and it should be chosen so it won't conflict with a client ID the server assigned before. That holds across server restarts and reboots too 🔁 If a client ID shows up and the server doesn't recognize it (like after a server reboot), the server rejects the request with the error NFS4ERR_STALE_CLIENTID 💔 When that happens, the client must get a new client ID through the SETCLIENTID operation, then carry on with any other recovery needed for the server reboot case (see Section 9.6.2) 🧚♀️
The client must also use SETCLIENTID when it gets an NFS4ERR_STALE_STATEID error on a stateid derived from its current client ID. That's because this error also signals a server reboot, which has invalidated the existing client ID (see Section 9.6.2 for details) ✨
For the complete specification of these operations, see the detailed descriptions of SETCLIENTID (Section 16.33.4) and SETCLIENTID_CONFIRM (Section 16.34.4) 📖💅
9.1.2. Server Release of Client ID
Okay, so if the server finds that the client holds no associated state for its client ID, the server may choose to release that client ID 🗑️ The server might do this for an inactive client so resources aren't eaten up by clients that only show up now and then 🕰️ If the client contacts the server after the release, the server must make sure the client gets the appropriate error, so it knows to run the SETCLIENTID/SETCLIENTID_CONFIRM sequence to get a new identity 🔁
It should be super clear that the server has to be very hesitant to release a client ID 😭 The work the client has to do to recover from that is exactly the same burden as if the server had failed and restarted. Typically, a server wouldn't release a client ID unless there had been no activity from that client for many minutes ⏳
Here's the tea 🍵: if the id string in a SETCLIENTID request is built properly, and the client makes sure to use the same principal for every successive SETCLIENTID, then, barring an active denial-of-service attack, NFS4ERR_CLID_INUSE should never be returned 🙌
That said, client bugs, server bugs, or a deliberate change in the principal owner of the id string can cause NFS4ERR_CLID_INUSE in rare cases 🐛 One example is a client that changes security flavors, where under the new flavor there's no mapping to the previous owner 🔀
When that happens, the server gets a SETCLIENTID for a client ID that currently has no state, or has state but the lease has expired. In that case, rather than returning NFS4ERR_CLID_INUSE, the server MUST allow the SETCLIENTID, and it will confirm the new client ID if the appropriate SETCLIENTID_CONFIRM follows 🦋💖
9.1.3. Use of Seqids
In several contexts, 32-bit sequence values called "seqids" are used as part of managing locking state 🔢 They show up in these situations:
- To provide an ordering of locking-related operations associated with a particular lock-owner or open-owner. See Section 9.1.7 for a detailed explanation 📋
- To define an ordered set of instances of a set of locks that share a particular set of ownership characteristics. See Section 9.1.4.2 for a detailed explanation 🔐
Successive seqid values for the same object are normally found by adding one to the current value 🌸 This pattern keeps going until the seqid is incremented past NFS4_UINT32_MAX. At that point, one (rather than zero) is the next seqid value 🔄
When you compare two seqid values to figure out which is later, you have to think about wraparound 🌀 Often the values are simple enough that plain numeric comparisons work. For example, if both seqid values are less than one million, the higher one can be considered later 🥰 But if one value is at or near NFS_UINT32_MAX and the other is less than one million, then implementations can reasonably decide that the lower value has wrapped around one more time. So even though it's numerically lower, it's actually later 🧠✨
Implementations can compare seqids even with potential wraparound by assuming that the chain of increments from one to the other is shorter than 231 💅 So if the difference between the two seqids is less than 231, the lower seqid is treated as earlier. If the difference is greater than or equal to 2**31, then it can be assumed the lower seqid has wrapped around one more time, and it can be treated as later 👑💜
9.1.4. Stateid Definition ✨
Okay bestie, so whenever the server grants a lock of any type (opens, byte-range locks, and delegations too 💅), it sends back a unique stateid. That stateid stands for a whole bundle of locks (often just one) that are on the same file, of the same type, and share the same ownership vibes 🦋. So opens of the same file by different open-owners each get their own stateid, and each set of byte-range locks owned by a specific lock-owner gets its own stateid too. Delegations have stateids as well, so they can be referenced 🎀. A stateid is basically a shorthand for a lock or a set of locks. Give the server one, and it can figure out the associated state-owner (or the open-owner/lock-owner pair) and the filehandle. When you use a stateid, the current filehandle must be the one associated with that stateid 🔥.
All stateids tied to a given client ID share a common lease. The lease is how the server keeps track of the claim those stateids and their objects have to being maintained. See Section 9.5 for the lease tea ☕.
Each stateid must be unique to the server. Lots of operations take a stateid but not a clientid, so the server must be able to infer the client from the stateid 🕵️♀️.
9.1.4.1. Stateid Types
With the exception of special stateids (see Section 9.1.4.3), each stateid represents locking objects of one of a set of types defined by the NFSv4 protocol. Quick heads up: whenever we talk about a guarantee here, there are situations like a client restart or lock revocation that can void it 🙈.
- Stateids may represent opens of files.
Each stateid in this case represents the OPEN state for a given client ID/open-owner/filehandle triple. Such stateids are subject to change (with consequent incrementing of the stateid's seqid) in response to OPENs that result in upgrade and OPEN_DOWNGRADE operations 🌸.
- Stateids may represent sets of byte-range locks.
All locks held on a particular file by a particular owner, and all gotten under the aegis of a particular open file, are associated with a single stateid. The seqid gets incremented whenever LOCK and LOCKU operations affect that set of locks 🧚.
- Stateids may represent file delegations, which are recallable guarantees by the server to the client that other clients will not reference, or will not modify, a particular file until the delegation is returned.
A stateid represents a single delegation held by a client for a particular filehandle 👑.
9.1.4.2. Stateid Structure
Stateids are split into two fields: a 96-bit other field that identifies the specific set of locks, and a 32-bit seqid sequence value. Except for special stateids (see Section 9.1.4.3), a particular value of the other field denotes a set of locks of the same type (byte-range locks, opens, or delegations), for a specific file or directory, sharing the same ownership characteristics. The seqid picks out a specific instance of that set. It goes up whenever the set changes, whether that's locks getting added or deleted, the byte-range they apply to changing, or one or more locks getting upgraded or downgraded 💖.
When a set of locks is first created, the server returns a stateid with a seqid value of one. On later operations that modify the set, the server is required to advance the seqid field by one whenever it returns a stateid for the same state-owner/file/type combination, and the operation is one that might actually change the set of locks designated. In that case, the server returns a stateid with an other field that matches what was used before for that state-owner/file/type combination, plus an incremented seqid field ✨.
Both the client and the server compare seqids. The client uses those comparisons to figure out the order of operations, and the server uses them to decide whether NFS4ERR_OLD_STATEID should be returned. In all cases, seqid wraparound needs to be taken into account, as discussed in Section 9.1.3 🍓.
9.1.4.3. Special Stateids
Stateid values whose other field is either all zeros or all ones are reserved. They MUST NOT be assigned by the server, but they have special meanings defined by the protocol. The meaning depends on whether other is all zeros or all ones, and on the specific value of seqid 🫶.
The following combinations of other and seqid are defined in NFSv4:
- Anonymous Stateid: When
otherandseqidare both zero, the stateid is treated as a special anonymous stateid. It can be used in READ, WRITE, and SETATTR requests to indicate the absence of any open state associated with the request. When an anonymous stateid value is used, and an existing open denies the form of access requested, then access will be denied to the request 🚫. - READ Bypass Stateid: When
otherandseqidare both all ones, the stateid is a special READ bypass stateid. When this value is used in WRITE or SETATTR, it is treated like the anonymous value. When used in READ, the server MAY grant access, even if access would normally be denied to READ requests 🦋.
If a stateid value is used that has all zeros or all ones in the other field but does not match one of the cases above, the server MUST return the error NFS4ERR_BAD_STATEID 💀.
Special stateids are a different breed. Unlike other stateids, they aren't associated with individual client IDs or filehandles, so they can be used with all valid client IDs and filehandles 🌈.
9.1.4.4. Stateid Lifetime and Validation
Stateids must remain valid until either a client restart or a server restart, or until the client returns all of the locks associated with the stateid through an operation such as CLOSE or DELEGRETURN. If the locks are lost due to revocation, then as long as the client ID is valid, the stateid stays a valid designation of that revoked state. Stateids associated with byte-range locks are the exception. They remain valid even if a LOCKU frees all remaining locks, so long as the open file they're associated with stays open 🎀.
It should be noted that there are situations where the client's locks become invalid without the client asking for them to be returned. These include lease expiration and a number of forms of lock revocation within the lease period. It's important to note that in these situations the stateid remains valid, and the client can use it to determine the disposition of the associated lost locks 🔍.
An other value must never be reused for a different purpose (i.e., a different filehandle, owner, or type of locks) within the context of a single client ID. A server may keep the other value for the same purpose beyond the point where it could otherwise be freed, but if it does so, it must maintain seqid continuity with previous values 👑.
One mechanism that may be used to meet the requirement that the server recognize invalid and out-of-date stateids is for the server to split the other field of the stateid into two fields:
- An index into a table of locking-state structures.
- A generation number that is incremented on each allocation of a table entry for a particular use.
And then the server stores the following in each table entry:
- The client ID with which the stateid is associated.
- The current generation number for the (at most one) valid stateid sharing this index value.
- The filehandle of the file on which the locks are taken.
- An indication of the type of stateid (open, byte-range lock, file delegation).
- The last seqid value returned corresponding to the current
othervalue. - An indication of the current status of the locks associated with this stateid, in particular whether they have been revoked and, if so, for what reason.
With this information, an incoming stateid can be validated, and the appropriate error returned when necessary. Special and non-special stateids are handled separately. (See Section 9.1.4.3 for a discussion of special stateids.) 🧩
When a stateid is being tested and the other field is all zeros or all ones, a check that the other and seqid fields match a defined combination for a special stateid is done, and the results are determined as follows:
- If the
otherandseqidfields do not match a defined combination associated with a special stateid, the errorNFS4ERR_BAD_STATEIDis returned. - If the combination is valid in general but is not appropriate to the context in which the stateid is used (e.g., an all-zero stateid is used when an open stateid is required in a LOCK operation), the error
NFS4ERR_BAD_STATEIDis also returned. - Otherwise, the check is completed and the special stateid is accepted as valid ✅.
When a stateid is being tested and the other field is neither all zeros nor all ones, the following procedure could be used to validate an incoming stateid and return an appropriate error when necessary. This assumes the other field would be divided into a table index and an entry generation. Note that the terms "earlier" and "later" used in connection with seqid comparison are to be understood as explained in Section 9.1.3 🫧.
- If the table index field is outside the range of the associated table, return
NFS4ERR_BAD_STATEID. - If the selected table entry is of a different generation than the one specified in the incoming stateid, return
NFS4ERR_BAD_STATEID. - If the selected table entry does not match the current filehandle, return
NFS4ERR_BAD_STATEID. - If the stateid represents revoked state or state lost as a result of lease expiration, then return
NFS4ERR_EXPIRED,NFS4ERR_BAD_STATEID, orNFS4ERR_ADMIN_REVOKED, as appropriate. - If the stateid type is not valid for the context in which the stateid appears, return
NFS4ERR_BAD_STATEID. A stateid may be valid in general but invalid for a particular operation. For example, a stateid that doesn't represent byte-range locks is passed to the non-from_open case of LOCK or to LOCKU, or a stateid that does not represent an open is passed to CLOSE or OPEN_DOWNGRADE. In such cases, the server MUST returnNFS4ERR_BAD_STATEID🚨. - If the
seqidfield is not zero and it is later than the current sequence value corresponding to the currentotherfield, returnNFS4ERR_BAD_STATEID. - If the
seqidfield is earlier than the current sequence value corresponding to the currentotherfield, returnNFS4ERR_OLD_STATEID. - Otherwise, the stateid is valid. The table entry should contain any additional information about the type of stateid and about that particular type, such as the associated set of locks (e.g., open-owner and lock-owner information), as well as information on the specific locks themselves, such as open modes and byte ranges 🌷.
9.1.4.5. Stateid Use for I/O Operations ✨
Okay so, clients doing Input/Output (I/O) operations have to pick the right stateid 💅 based on the locks they're holding (including opens and delegations) and on the different kinds of state-owners sending the I/O requests. Lowkey, SETATTR operations that change the file size count as I/O here too 🫶
These rules are listed from highest priority to lowest, and they decide which stateid gets used. While following them, the client only considers locks it has actually been notified about through an appropriate operation response or callback 📬
- If the client holds a delegation for the file in question, the delegation stateid SHOULD be used. 🦋
- Otherwise, if the entity corresponding to the lock-owner (e.g., a process) sending the I/O has a byte-range lock stateid for the associated open file, then the byte-range lock stateid for that lock-owner and open file SHOULD be used. 🌸
- If there is no byte-range lock stateid, then the OPEN stateid for the current open-owner, i.e., the OPEN stateid for the open file in question, SHOULD be used. 🎀
- Finally, if none of the above apply, then a special stateid SHOULD be used. 💜
Ignoring these rules may result in situations where the server doesn't have the info it needs to process the request properly 😭 For example, when mandatory byte-range locks are in effect, if the stateid doesn't point to the proper lock-owner through a lock stateid, a request might get avoidably rejected 💀
The server, however, should not try to enforce these ordering rules and should use whatever information is available to properly process I/O requests. Especially, when a client has a delegation for a given file, it SHOULD take note of this fact in processing a request, even if it is sent with a special stateid 🔥
9.1.4.6. Stateid Use for SETATTR Operations 🎀
For SETATTR, a stateid is present. Outside of the cases that set the file size, the client may send either a special stateid or, when a delegation is held for the file in question, a delegation stateid 🦋 While the server SHOULD validate the stateid and may use it to optimize figuring out whether a delegation is held, it SHOULD note the presence of a delegation even when a special stateid is sent, and it MUST accept a valid delegation stateid when sent 🫶
9.1.5. Lock-Owner 🔒
When requesting a lock, the client has to present the server with the client ID and an identifier for the owner of the requested lock. These two fields make up the lock-owner, and they're defined like this:
- A client ID returned by the server as part of the client's use of the
SETCLIENTIDoperation. - A variable-length opaque array used to uniquely define the owner of a lock managed by the client. This may be a thread id, process id, or other unique value.
When the server grants the lock, it responds with a unique stateid ✨ The stateid is used as a shorthand reference to the lock-owner, since the server will be keeping track of the correspondence between them 🧚
9.1.6. Use of the Stateid and Locking 🔐
All READ, WRITE, and SETATTR operations contain a stateid. For the purposes of this section, SETATTR operations that change the size attribute of a file are treated as if they're writing the area between the old and new size (i.e., the range truncated or added to the file by means of the SETATTR), even where SETATTR isn't explicitly mentioned in the text 👀 The stateid passed to one of these operations must be one that represents an OPEN (e.g., via the open-owner), a set of byte-range locks, or a delegation. It may also be a special stateid representing anonymous access or the READ bypass stateid 🌈
If the state-owner performs a READ or WRITE in a situation where it has established a lock or share reservation on the server (any OPEN constitutes a share reservation), the stateid (previously returned by the server) must be used to show what locks, including both byte-range locks and share reservations, are held by the state-owner 🔑 If no state is established by the client (either a byte-range lock or a share reservation), the anonymous stateid is used. Regardless of whether an anonymous stateid or a stateid returned by the server is used, if there's a conflicting share reservation or mandatory byte-range lock held on the file, the server MUST refuse to service the READ or WRITE operation 🚫
Share reservations are set up by OPEN operations, and by their nature they're mandatory. When the OPEN denies READ or WRITE operations, that denial gets those operations rejected with the error NFS4ERR_LOCKED 🙈 Byte-range locks can be implemented by the server as either mandatory or advisory. Or the server can decide between mandatory and advisory behavior based on the file being accessed. For example, some UNIX-based servers support a "mandatory lock bit" on the mode attribute, so if it's set, byte-range locks are required on the file before I/O is possible 💅 When byte-range locks are advisory, they only prevent the granting of conflicting lock requests and have no effect on READs or WRITEs. Mandatory byte-range locks, though, prevent conflicting I/O operations. When they're attempted, they get rejected with NFS4ERR_LOCKED 🙅♀️ When the client gets NFS4ERR_LOCKED on a file it knows it has the proper share reservation for, it'll need to issue a LOCK request on the region of the file that includes the region the I/O was supposed to hit, with an appropriate locktype (i.e., READ*_LT for a READ operation, WRITE*_LT for a WRITE operation) 🎯
With NFSv3, there was no notion of a stateid, so there was no way to tell whether the application process of the client sending the READ or WRITE operation had also acquired the appropriate byte-range lock on the file. So there was no way to implement mandatory locking. With the stateid construct, that barrier is gone 🥳
Note that for UNIX environments that support mandatory file locking, the difference between advisory and mandatory locking is subtle 🤓 In fact, advisory and mandatory byte-range locks are exactly the same as far as the APIs and implementation requirements are concerned. If the mandatory lock attribute is set on the file, the server checks whether the lock-owner has an appropriate shared (read) or exclusive (write) byte-range lock on the region it wishes to read or write to. If there's no appropriate lock, the server checks whether there's a conflicting lock (which can be done by attempting to acquire the conflicting lock on behalf of the lock-owner and, if successful, releasing the lock after the READ or WRITE is done). If there is one, the server returns NFS4ERR_LOCKED 🫠
For Windows environments, there are no advisory byte-range locks, so the server always checks for byte-range locks during I/O requests 🪟
So the NFSv4 LOCK operation doesn't need to tell advisory and mandatory byte-range locks apart. It's the NFSv4 server's processing of the READ and WRITE operations that introduces the distinction 💖
Every stateid other than the special stateid values noted in this section, whether it's returned by an OPEN-type operation (i.e., OPEN, OPEN_DOWNGRADE) or by a LOCK-type operation (i.e., LOCK or LOCKU), defines an access mode for the file (i.e., READ, WRITE, or READ-WRITE). That access mode is established by the original OPEN that began the stateid sequence, and it's modified by any subsequent OPENs and OPEN_DOWNGRADEs within that stateid sequence 🌸 When a READ, WRITE, or SETATTR that specifies the size attribute is done, the operation is checked against the access mode to verify that it's appropriate given the OPEN it's associated with 🔍
For WRITE-type operations (i.e., WRITEs and SETATTRs that set size), the server must verify that the access mode allows writing, and it returns an NFS4ERR_OPENMODE error if it doesn't 🙅♀️ For READ, the server may perform the corresponding check on the access mode. Or it may choose to allow READ on opens for WRITE only, to accommodate clients whose write implementation may unavoidably do reads (e.g., due to buffer cache constraints) 🧃 Even if READs are allowed in these circumstances, the server MUST still check for locks that conflict with the READ (e.g., another open specifying denial of READs) 🚨 Note that a server that does enforce the access mode check on READs doesn't need to explicitly check for conflicting share reservations, since the existence of an OPEN for read access guarantees that no conflicting share reservation can exist 🙌
A READ bypass stateid MAY allow READ operations to bypass locking checks at the server 🦋 However, WRITE operations with a READ bypass stateid MUST NOT bypass locking checks, and they're treated exactly the same as if an anonymous stateid were used 🚫
A lock may not be granted while a READ or WRITE operation using one of the special stateids is being performed and the range of the lock request conflicts with the range of that operation 😬 For the purposes of this paragraph, a conflict happens when a shared lock is requested while a WRITE operation is being performed, or when an exclusive lock is requested while either a READ or a WRITE operation is being performed. A SETATTR that sets size is treated similarly to a WRITE, as discussed above 💜
9.1.7. Sequencing of Lock Requests 🔒✨
Okay so locking is literally not like most NFS operations 🙈 because it needs "at-most-one" semantics, and ONC RPC doesn't give you that on its own. ONC RPC over a reliable transport is not enough, bestie, because a sequence of locking requests may span multiple TCP connections 😭. So if things get retransmitted or reordered, lock and unlock requests need a well-defined, consistent behavior. The fix is that every lock request carries a sequence number, a consecutively increasing integer 💅. Different state-owners get their own sequences. The server keeps track of the last sequence number it received (that's L) and the response it sent back. The server SHOULD assign a seqid value of one to the first request issued for any given state-owner 🌸. After that you just increment, with wraparound as described in Section 9.1.3 🔄.
Heads up, for requests that contain a sequence number, each state-owner should have no more than one outstanding request 🫶.
When a request comes in, its sequence number (r) gets compared to the last one received (L). It only gets processed past the seqid checking if it has the correct next sequence, normally L + 1 💖. With a properly functioning client, the response to (r) must have been received before the last request (L) was sent. If a duplicate of the last request (r == L) shows up, the stored response is returned 🔁. If the sequence value is any other value, it gets rejected with the error NFS4ERR_BAD_SEQID 🙅♀️. And the sequence history gets reinitialized whenever the SETCLIENTID/SETCLIENTID_CONFIRM sequence changes the client verifier ✨.
It is critical that the server maintain the last response sent to the client 🚨. This gives a more reliable cache of duplicate non-idempotent requests than the traditional cache described in [Chet]. The traditional duplicate request cache uses a least recently used algorithm to remove unneeded requests. But the last lock request and response on a given state-owner must be cached for as long as the lock state exists on the server 🧠.
The client MUST advance the sequence number for the CLOSE, LOCK, LOCKU, OPEN, OPEN_CONFIRM, and OPEN_DOWNGRADE operations 💅. This is true even if the previous operation that used the sequence number received an error. The only exception is if the previous operation received one of these errors:
NFS4ERR_STALE_CLIENTIDNFS4ERR_STALE_STATEIDNFS4ERR_BAD_STATEIDNFS4ERR_BAD_SEQIDNFS4ERR_BADXDRNFS4ERR_RESOURCENFS4ERR_NOFILEHANDLENFS4ERR_MOVED
9.1.8. Recovery from Replayed Requests 🔁
As described above, the sequence number is per state-owner 🎀. As long as the server maintains the last sequence number received and follows the methods described above, there are no risks of a Byzantine router re-sending old requests 🙌. The server only needs to maintain the (state-owner, sequence number) state for as long as there are open files or closed files with locks outstanding.
LOCK, LOCKU, OPEN, OPEN_DOWNGRADE, and CLOSE each contain a sequence number. So the risk of replaying these operations and getting undesired effects is non-existent while the server maintains the state-owner state 💯.
9.1.9. Interactions of Multiple Sequence Values 🧩
Some operations may have multiple sources of data for request sequence checking and retransmission determination 👀. Some operations have multiple sequence values tied to multiple types of state-owners. On top of that, these operations may also have a stateid with its own seqid value that gets checked for validity.
As noted above, there may be multiple sequence values to check 🫠. The following rules should be followed by the server when processing these multiple sequence values within a single operation:
- When a sequence value associated with a state-owner is unavailable for checking because the state-owner is unknown to the server, it takes no part in the comparison.
- When any of the state-owner sequence values are invalid, NFS4ERR_BAD_SEQID is returned. When a stateid sequence is checked,
NFS4ERR_BAD_STATEIDorNFS4ERR_OLD_STATEIDis returned as appropriate, but NFS4ERR_BAD_SEQID has priority 👑. - When any one of the sequence values matches a previous request for a state-owner, it is treated as a retransmission and not re-executed. When the type of the operation does not match the one originally used, NFS4ERR_BAD_SEQID is returned. When the server can determine that the request differs from the original, it may return NFS4ERR_BAD_SEQID.
- When multiple sequence values match previous operations but the operations are not the same, NFS4ERR_BAD_SEQID is returned.
- When there are no sequence values available for comparison and the operation is an
OPEN, the server indicates to the client that anOPEN_CONFIRMis required. The exception is when the server can conclusively determine that confirmation is not required (e.g., by knowing that no open-owner state has ever been released for the current clientid) ✅.
9.1.10. Releasing State-Owner State 🗑️
When a particular state-owner no longer holds open or file locking state at the server, the server may choose to release the sequence number state associated with that state-owner 🫧. The server may make this choice based on lease expiration, the reclamation of server memory, or other implementation-specific details. Heads up though: when this is done, a retransmitted request (normally identified by a matching state-owner sequence) may not be correctly recognized. That means the client may not get the original response it would have gotten if the state-owner state had not been released 😬.
If the server could be sure that a given state-owner would never again be used by a client, this issue could not come up 🔮. Even when the state-owner state is released and the client later uses that state-owner, retransmitted requests will be detected as invalid and not executed. The client may have a recovery path that is more complicated than simply getting the original response back transparently 🛠️.
Either way, the server can safely release state-owner state (meaning retransmitted requests will not be erroneously acted upon) when the state-owner is not currently being used by the client. That means there are no open files associated with an open-owner and no lock stateids associated with a lock-owner 🙌. The server may choose to hold the state-owner state to simplify the recovery path when retransmissions of currently active requests are received. However, how long it holds this state is implementation specific ⏳.
If a LOCK, LOCKU, OPEN_DOWNGRADE, or CLOSE is retransmitted after the server has released the state-owner state, the server will find that the state-owner has no files open, and an error will be returned to the client 🚫. If the state-owner does have a file open, the stateid will not match, and again an error is returned to the client.
9.1.11. Use of Open Confirmation 🔐
If an OPEN is retransmitted and the open-owner is being used for the first time, or the open-owner state has previously been released by the server, the use of the OPEN_CONFIRM operation prevents incorrect behavior 🎯. When the server first observes the use of the open-owner, it will direct the client to perform the OPEN_CONFIRM for the corresponding OPEN. This sequence establishes the use of an open-owner and its associated sequence number. Since the OPEN_CONFIRM sequence connects a new open-owner on the server with an existing open-owner on a client, the sequence number may have any valid (that is, non-zero) value. The OPEN_CONFIRM step assures the server that the value received is the correct one 🥹. (See Section 16.18 for further details.)
There are a number of situations where requiring confirmation of an OPEN would cause difficulties for the client and server. They would be prevented from acting in a timely fashion on information received, because that information would be provisional and subject to deletion upon non-confirmation 😩. Luckily, these are situations where the server can avoid the need for confirmation when responding to open requests ✨. The two constraints are:
- The server must not bestow a delegation for any open that would require confirmation.
- The server MUST NOT require confirmation on a reclaim-type open (that is, one specifying claim type
CLAIM_PREVIOUSorCLAIM_DELEGATE_PREV) 🙅♀️.
These constraints are related, because reclaim-type opens are the only ones where the server may be required to send a delegation. For CLAIM_NULL, sending the delegation is optional, while for CLAIM_DELEGATE_CUR, no delegation is sent 🧃.
Delegations sent with an open that requires confirmation are troublesome, because recovering from non-confirmation adds undue complexity to the protocol 🌀. Requiring confirmation on reclaim-type opens is also a problem, since the inability to resolve the status of the reclaim until lease expiration may make it hard to determine the set of locks being reclaimed in time (the grace period may expire first) ⏰.
Requiring open confirmation on reclaim-type opens can be avoided because of the nature of the environments where these opens happen 🌷. For CLAIM_PREVIOUS opens, this is immediately after a server reboot, so there should be no time for open-owners to be created, found to be unused, and recycled. For CLAIM_DELEGATE_PREV opens, we are dealing with either a client reboot or a network partition that results in deletion of lease state (and returns NFS4ERR_EXPIRED). A server that supports delegations can be sure that no open-owners for that client have been recycled since client initialization or deletion of lease state. So it can be confident that confirmation will not be required 💖.
9.2. Lock Ranges 🌈
The protocol lets a lock-owner request a lock with a byte range and then either upgrade or unlock a sub-range of that initial lock 🧩. This is expected to be an uncommon type of request. Either way, servers or server file systems may not be able to support sub-range lock semantics 🫤. If a server receives a locking request that represents a sub-range of current locking state for the lock-owner, the server is allowed to return the error NFS4ERR_LOCK_RANGE to signify that it doesn't support sub-range lock operations. So the client should be prepared to receive this error and, if appropriate, report it to the requesting application 📣.
The client is discouraged from combining multiple independent locking ranges that happen to be adjacent into a single request 🙃. That's because the server may not support sub-range requests, and also for reasons related to recovering file locking state after a server failure. As discussed in Section 9.6.2 below, the server may use certain optimizations during recovery. Those optimizations only work effectively when the client's behavior during lock recovery is similar to its locking behavior before the server failure 🦋.
9.3. Upgrading and Downgrading Locks 🦋
okay so if you're holding a write lock on a record, you can ask for an atomic downgrade to a read lock by sending a LOCK request with the type set to READ_LT. if the server supports atomic downgrade, it just works ✨. if it doesn't, you'll get NFS4ERR_LOCK_NOTSUPP 😭. the client should be ready for that error and, if it makes sense, pass it along to the app that asked 💅
going the other way, if you've got a read lock on a record, you can ask for an atomic upgrade to a write lock with a LOCK request whose type is WRITE_LT or WRITEW_LT 🔥. if the server doesn't support atomic upgrade, it returns NFS4ERR_LOCK_NOTSUPP. if the upgrade can happen without any existing conflict, the request succeeds. otherwise the server returns either NFS4ERR_DENIED or NFS4ERR_DEADLOCK. NFS4ERR_DEADLOCK shows up when the client sent LOCK with the type WRITEW_LT and the server has spotted a deadlock 💀. same deal here: the client should be prepared to get these errors and, if appropriate, report them to the requesting application 🫶
9.4. Blocking Locks 🌸
some clients really need blocking locks, and NFSv4 has a bit of a problem with that. the protocol can't rely on a callback mechanism, so there's no way to tell a client "hey, that lock you were denied is free now" 🙈. the client just has to keep polling for the lock nonstop, which is super unfair. so two new lock types, READW and WRITEW, were added to tell the server "heyyy, I want a blocking lock" 🧚. the server should keep an ordered list of pending blocking locks. when the conflicting lock gets released, the server may wait out the lease period so the first waiting client can re-request the lock. once that lease period is over, the next waiting client's request gets the lock. clients are required to poll at an interval small enough that they're likely to grab the lock in a timely way ⏱️
the server is not required to keep a list of pending blocked locks, since the list isn't used for correct operation, just for fairness 🍓. because crash recovery is unordered, you'd need lock state stored in stable storage to guarantee that blocking locks get granted in order 💜
servers may also note the lock types and hold off on returning the denial, giving a conflicting lock extra time to be released so the request can succeed ✨. this way clients don't have to burn themselves out polling for blocking locks as much 🫶. the server should be careful about how long that delay is in case the client retransmits the request 👑
if a server gets a blocking lock request, denies it, and then later gets a non-blocking request for the same lock that's also denied, it should take that lock off its list of pending blocking locks 🎀. clients should send that non-blocking request as a way of saying "this is the last time I'm polling for this lock," which can happen when the process asking for the lock gets interrupted. it's a courtesy to the server so it doesn't wait a whole lease period for nothing before granting other lock requests 🌈. clients aren't required to do this courtesy, though, and servers must not depend on them doing it. also, clients must be prepared for the possibility that this final request gets accepted anyway 🙈
9.5. Lease Renewal 💖
the whole point of a lease is to let the server clean up stale locks held by a client that crashed or is otherwise unreachable 🧹. it is not a mechanism for cache consistency, and lease renewals may not be denied if the lease interval hasn't expired yet 🔒
a client can implicitly show that it's still active, and that the state the server holds for it is still valid, just by doing things. any operation made with a valid clientid (DELEGPURGE, LOCK, LOCKT, OPEN, RELEASE_LOCKOWNER, or RENEW) or a valid stateid (CLOSE, DELEGRETURN, LOCK, LOCKU, OPEN, OPEN_CONFIRM, OPEN_DOWNGRADE, READ, SETATTR, or WRITE) tells the server to renew all of that client's leases, meaning every lease sharing a given client ID 💅. in the stateid case, the stateid must not be one of the special stateids (the anonymous stateid or the READ bypass stateid) 🙅♀️
okay, big note: if a client had restarted or rebooted, it wouldn't be sending these requests without going through theSETCLIENTID/SETCLIENTID_CONFIRMsequence first. that sequence, when it changes the client verifier, tells the server to drop the locking state tied to the client 🧨. andSETCLIENTID/SETCLIENTID_CONFIRMnever renews a lease, so don't count on it for that 🙅♀️
if the server has rebooted, the stateids (NFS4ERR_STALE_STATEID error) or the client ID (NFS4ERR_STALE_CLIENTID error) won't be valid anymore, which stops spurious renewals from sneaking through 🛑
this approach is low-overhead and scales really well 📈. in the typical case, no extra RPCs are needed for lease renewal, and in the worst case, there's one RPC every lease period (a RENEW operation) 🧚. the number of locks a client holds doesn't matter, since all of its state gets involved in the lease renewal action anyway 🦋
since every operation that creates a new lease also renews existing leases, the server must maintain a common lease expiration time for all valid leases belonging to a given client 🫶. that single lease time can then be easily updated whenever there's an implicit lease renewal ✨
9.6. Crash Recovery 🌈
the big requirement in crash recovery is that both the client and the server know when the other one has failed 👀. it's also required that a client sees a consistent view of data across server restarts or reboots 🔥. any READ and WRITE operations that might be sitting in the client's or the network's buffers have to wait until the client has successfully recovered the locks protecting those READ and WRITE operations 💅
9.6.1. Client Failure and Recovery 💀
if a client fails, the server may recover that client's locks once the associated leases have expired 🕰️. conflicting locks from another client can only be granted after that lease expiration 🙈. if the client can restart or reinitialize within the lease period, it may end up waiting out the rest of the lease period before it can get new locks 😭
to keep the delay down when a client restarts, open and lock requests get tied to an instance of the client through a client-supplied verifier 🧚. this verifier is part of the initial SETCLIENTID call the client makes, and the server hands back a client ID as the result of that SETCLIENTID operation 🎀. the client then confirms the client ID with SETCLIENTID_CONFIRM 🫶. after that, the client uses the client ID together with an opaque owner field to identify the open-owner for OPEN 💖. that whole chain of associations is what lets everything be tied back to one particular client's locks 🔗
because the client changes the verifier every time it initializes, the server can compare a new verifier to the verifier tied to the locks it's currently holding and see that they don't match 🕵️♀️. that mismatch signals a new instantiation of the client and the loss of its locking state 🌸. as a result, the server is free to release all the locks held under the old client ID, the one that was derived from the old verifier 🧹
one more thing: the verifier has to have the same uniqueness properties as the verifier used for the COMMIT operation 🔐
9.6.2. Server Failure and Recovery ✨
okay so what happens when the server loses its locking state? 😭 Usually that's a restart or reboot, and it's giving "everyone please calm down and reconnect" 🫶. The server must allow clients time to notice this and re-establish the lost locking state. Clients also need to rebuild their locks without the server turning them away just because it already handed a conflicting grant to someone else 💅. And if there's any chance a client hasn't rebuilt its locks for a file yet, the server must disallow READ and WRITE operations on that file 🙈. This recovery window lasts as long as the lease period, periodt.
A client can tell that the server failed and lost its locking state when it gets one of two errors. NFS4ERR_STALE_STATEID means a stateid got wiped out by a reboot or restart 💀. NFS4ERR_STALE_CLIENTID means a client ID got wiped out the same way. When either one shows up, the client must establish a new client ID (see Section 9.1.1) and then re-establish its locking state, as described below 🔥.
The special handling period for locks, READs, and WRITEs, which lasts as long as the lease period, is called the "grace period" 🌸. During the grace period, clients get their locks and related state back using reclaim-type locking requests. That means LOCK requests with reclaim set to TRUE, and OPEN operations with a claim type of either CLAIM_PREVIOUS or CLAIM_DELEGATE_PREV. Meanwhile, the server must reject READ and WRITE operations and non-reclaim locking requests (that's the other LOCK and OPEN operations) with an error of NFS4ERR_GRACE 🚫.
Here's the twist 🦋. If the server can reliably tell that granting a non-reclaim request won't conflict with any other client's lock reclamation, it doesn't have to return NFS4ERR_GRACE, and the non-reclaim request can go through. For the server to handle READ and WRITE operations during the grace period, it needs to guarantee that no conflict could pop up between an upcoming reclaim locking request and that I/O. If the server can't make that guarantee, it must return NFS4ERR_GRACE to the client.
The simplest way for a server to behave during the grace period is to reject every non-reclaim locking request and every READ and WRITE with NFS4ERR_GRACE 🧚. That said, a server may keep information about granted locks in stable storage. With that info, it can figure out whether a regular lock or a READ or WRITE is safe to process.
For example, if a count of locks on a given file is kept in stable storage, the server can track reclaimed locks for that file. Once all the reclaims have been processed, non-reclaim locking requests may be processed. That way, the server can make sure non-reclaim locking requests won't conflict with potential reclaim requests 🎀. As for I/O, if the server can tell from stable storage (or some similar mechanism) that there are no outstanding reclaim requests for a file, I/O requests for that file can carry on normally ✨.
To say it one more time 💜: a server that lets non-reclaim lock and I/O requests run during the grace period MUST determine that no lock reclaimed later will be rejected, and that no lock reclaimed later would have blocked any I/O operation handled during the grace period.
Clients should be ready to get NFS4ERR_GRACE back for non-reclaim lock and I/O requests 🍓. When that happens, the client should use a retry mechanism for the request. A delay (on the order of several seconds) between retries should be used so the server doesn't get flooded 🌈. More discussion of the general issue is in [Floyd]. The client has to account for servers that can handle I/O and non-reclaim locking requests during the grace period, as well as servers that can't 🫶.
A reclaim-type locking request that arrives outside the server's grace period can only succeed if the server can guarantee that no conflicting lock or I/O request has been granted since the reboot or restart 👑.
A server may pick a new value for the lease period when it restarts ⭐. So once a client has a new client ID, it should refetch the lease_time attribute and use it as the basis for lease renewal for that server's lease. Even so, for this restart event, the server must set up a grace period at least as long as the lease period from the previous server instance. That lets the client state from the previous instance get re-established reliably 💅.
9.6.3. Network Partitions and Recovery 🦋
okay, network partitions 😩. If a partition lasts longer than the lease period the server gave out, the server won't have gotten a lease renewal from the client. When that happens, the server may cancel the lease and free every lock held for that client. Then all the stateids the client had become invalid or stale 💀. Once the client can reach the server again, any I/O it sends with those now-invalid stateids will fail, and the server will return NFS4ERR_EXPIRED. After that error, the client will let the application that held the lock know, suitably 🫶.
9.6.3.1. Courtesy Locks ✨
As a courtesy to the client, or as an optimization, the server may keep holding locks (delegations included) for a client whose recent communication has gone past the lease period, which delays canceling the lease 🌸. If the server gets a lock or I/O request that conflicts with one of these courtesy locks, or if it runs out of resources, the server MAY cause lease cancellation right then. After that, it returns NFS4ERR_EXPIRED whenever one of the stateids tied to the freed locks gets used 😭. If the lease hasn't been canceled and the server gets a lock or I/O request that conflicts with a courtesy lock, these rules apply:
- For a courtesy lock that isn't a delegation, the server MUST free the courtesy lock and grant the new request.
- For a lock or I/O request that conflicts with a delegation being held as a courtesy lock, the server MAY delay resolving the request, but it MUST NOT reject the request. It MUST free the delegation and grant the new request eventually 🔥.
- For a request for a delegation that conflicts with a delegation being held as a courtesy lock, the server MAY grant the new request or not, as it chooses. But if it does grant the conflicting request, the delegation held as a courtesy lock MUST be freed 👑.
If the server hasn't rebooted or canceled the lease by the time the partition heals, and the original client tries to access a courtesy lock that was freed, the server SHOULD send back an NFS4ERR_BAD_STATEID to the client 🙈. If the client tries to access a courtesy lock that was not freed, the server SHOULD mark all of the courtesy locks as implicitly being renewed 🍓.
9.6.3.2. Lease Cancellation 🌈
Leases can get canceled as a result of lease expiration. That can happen right when the lease expires, or later, depending on whether a conflicting lock shows up or the partition runs longer than the server is willing to tolerate 🫶.
When a lease is canceled, all the locking state tied to it is freed. Using any of the associated stateids then returns NFS4ERR_EXPIRED, and so does using the associated clientid 😭.
To recover from this, the client should use SETCLIENTID followed by SETCLIENTID_CONFIRM to establish a new clientid 🧚. Once a lock is obtained with that clientid, a lease will be established again 💅.
9.6.3.3. Client's Reaction to a Freed Lock 💜
There's no way for a client to predict how a given server will act during a network partition 🤷♀️. When the partition heals, the client still has all of its locks, some of its locks, or none of them. The client can look at the different error return values to decide how to respond ✨.
NFS4ERR_EXPIRED: every lock has been freed because of a lease cancellation that happened during the partition. The client should use aSETCLIENTIDto recover 💀.NFS4ERR_ADMIN_REVOKED: the current lock was revoked before, during, or after the partition. The client SHOULD handle this error the way it normally would 🎀.NFS4ERR_BAD_STATEID: the current lock was revoked or released during the partition, and the server did not reboot. Other locks MAY still be renewed. The client doesn't need to do aSETCLIENTID, and instead SHOULD probe with aRENEWcall 🦋.NFS4ERR_RECLAIM_BAD: the current lock was revoked during the partition, and the server rebooted. The server might have no info on the other locks, which may still be renewable 🌸.NFS4ERR_NO_GRACE: the client's locks were revoked during the partition, and the server rebooted. None of the client's locks will be renewable 😭.NFS4ERR_OLD_STATEID: the server has not rebooted. The client SHOULD handle this error the way it normally would 🙈.
9.6.3.4. Edge Conditions 🔥
When a network partition and a server reboot happen together, both the server and the client have jobs to do so the client doesn't reclaim a lock it shouldn't be able to access anymore 👑. The short version:
- Client's responsibility: a client MUST NOT attempt to reclaim any locks that it did not hold at the end of its most recent successfully established client lease 🫶.
- Server's responsibility: a server MUST NOT allow a client to reclaim a lock unless it knows that the lock could not have been followed by a conflicting lock grant. However, when deciding whether a conflicting lock could have been granted, it is permitted to assume that its clients are carrying out their responsibilities, as above 💅.
A server may consider a client's lease "successfully established" once it has received an OPEN operation from that client 🌈.
Everything above is about CLAIM_PREVIOUS reclaims, not CLAIM_DELEGATE_PREV reclaims, which generally don't involve a server reboot ✨. Still, when a server persistently stores delegation information to support CLAIM_DELEGATE_PREV across a period when both the client and the server are down at the same time, similar rules apply 🧚.
The next sections give examples of what can go wrong if these responsibilities get ignored. They also show examples of server implementation strategies that could meet a server's responsibilities 🎀.
9.6.3.4.1. First Server Edge Condition ✨
okay so the first edge condition goes like this 👀
- Client A acquires a lock. 🔐
- Client A and the server get hit by a mutual network partition, so client A can't renew its lease. 💔
- Client A's lease expires, so the server releases the lock.
- Client B acquires a lock that would have conflicted with client A's. 😳
- Client B releases the lock.
- The server reboots. 🔄
- The network partition between client A and the server heals. 🩹
- Client A issues a RENEW operation and gets back an
NFS4ERR_STALE_CLIENTID. 💀 - Client A reclaims its lock within the server's grace period.
So at that last step, the server has wrongly granted client A's lock reclaim 😭 If client B changed the object that the lock was protecting, client A ends up with corrupted object data. Not it. 🙅♀️
9.6.3.4.2. Second Server Edge Condition 🌸
the second known edge condition is giving déjà vu, bestie 🫠
- Client A acquires a lock.
- The server reboots. 🔄
- Client A and the server get hit by a mutual network partition, so client A can't reclaim its lock within the grace period. 💔
- The server's reclaim grace period ends. Client A has no locks recorded on the server anymore.
- Client B acquires a lock that would have conflicted with client A's. 😳
- Client B releases the lock.
- The server reboots a second time. 🔄🔄
- The network partition between client A and the server heals. 🩹
- Client A issues a RENEW operation and gets back an
NFS4ERR_STALE_CLIENTID. 💀 - Client A reclaims its lock within the server's grace period.
Same as the first edge condition, the final step has the server wrongly granting client A's lock reclaim. Main character energy for the wrong reasons 💅
9.6.3.4.3. Handling Server Edge Conditions 🛡️
in both of the examples above, the client tries to reclaim a lock it was holding at the end of its most recent successfully established lease. so the client did its part, periodt ✨
the server, though, messed up 😤 It granted a reclaim even though it had already granted a conflicting lock since that reclaimed lock was last held.
to fix these edge conditions, the server has two options. Option one: it assumes after a reboot that an edge condition has happened, and so it returns NFS4ERR_NO_GRACE for all reclaim attempts. Option two: it records some information in stable storage. How much info it records depends on how harsh the server wants to be whenever these edge conditions happen. The server that is totally tolerant of every edge condition records every lock in stable storage as it's acquired. It only removes that lock record once the client unlocks it and the lock's owner advances the sequence number so the lock release isn't the last stateful event for that owner's sequence. 💾
For the two edge conditions above, the harshest a server can be and still support a grace period for reclaims only requires that the server record some minimal info in stable storage. For example, a server implementation could save, for each client, a record in stable storage containing:
- the client's id string. 🪪
- a boolean that says whether the client's lease expired, or whether there was administrative intervention (see Section 9.8) to revoke a byte-range lock, share reservation, or delegation. 🚫
- a timestamp that gets updated the first time after a server boot or reboot that the client acquires byte-range locking, share reservation, or delegation state on the server. The timestamp doesn't need to be updated on later lock requests until the server reboots again. ⏰
The server implementation would also save the timestamps from the two most recent server reboots in stable storage. 📝
With that record keeping in place, here's how the first edge condition plays out. After the server reboots, the record saying client A's lease expired means another client could have gotten a conflicting record lock, share reservation, or delegation. So the server must reject a reclaim from client A with NFS4ERR_NO_GRACE or NFS4ERR_RECLAIM_BAD. 🙅♀️
For the second edge condition, after the server reboots for a second time, the record showing that the client had an unexpired record lock, share reservation, or delegation before the server's previous incarnation means the server must reject client A's reclaim with NFS4ERR_NO_GRACE or NFS4ERR_RECLAIM_BAD. Same vibe, same answer 🫶
No matter what level or approach the server uses for record keeping, the server MUST implement one of the following strategies. These apply to reclaims of share reservations, byte-range locks, and delegations 📜
- Reject all reclaims with
NFS4ERR_NO_GRACE. This is extremely harsh, but it's necessary if the server doesn't want to record lock state in stable storage. 💀 - Record enough state in stable storage to meet its responsibilities. When in doubt, the server should err on the side of being harsh. 🧊
If, after a server reboot, the server figures out there's unrecoverable damage or corruption in stable storage, then for all affected clients and/or locks, the server MUST return NFS4ERR_NO_GRACE. 🚨
9.6.3.4.4. Client Edge Condition 🦋
a third edge condition hits the client instead of the server 🥺 If the server reboots while the client is in the middle of reclaiming some locks, and then a network partition shows up, the client might end up having reclaimed some locks but not all of them. In that situation, a careful client would assume the non-reclaimed locks were revoked. 🧐
here's the third known edge condition:
- Client A acquires lock 1.
- Client A acquires lock 2.
- The server reboots. 🔄
- Client A issues a RENEW operation and gets back an
NFS4ERR_STALE_CLIENTID. 💀 - Client A reclaims lock 1 within the server's grace period.
- Client A and the server get hit by a mutual network partition, so client A can't reclaim its remaining locks within the grace period. 💔
- The server's reclaim grace period ends.
- Client B acquires a lock that would have conflicted with client A's lock 2. 😳
- Client B releases the lock.
- The server reboots a second time. 🔄🔄
- The network partition between client A and the server heals. 🩹
- Client A issues a RENEW operation and gets back an
NFS4ERR_STALE_CLIENTID. 💀 - Client A reclaims both lock 1 and lock 2 within the server's grace period.
At that last step, the client reclaims lock 2 like it had been held the whole time, even though a conflicting lock was actually granted to client B. 😭
This happens because the client messed up its responsibility. It tried to reclaim lock 2 even though it didn't hold that lock at the end of the lease established by the SETCLIENTID after the first server reboot. (Client A did hold lock 2 on a previous lease, but only the most recent lease counts 🙄)
A server could dodge this by rejecting the reclaim of lock 2. But to do that accurately, it would need to make sure info about each individual lock it held survives a reboot. Server implementations aren't required to do that, so the client must not assume the server will. 🙅♀️
Instead, a client MUST reclaim only the locks it successfully acquired from the previous server instance, skipping any it failed to reclaim before a new reboot. So in the last step above, client A should only reclaim lock 1. ✅
9.6.3.4.5. Client's Handling of Reclaim Errors 🧩
how the client handles the NFS4ERR_NO_GRACE and NFS4ERR_RECLAIM_BAD errors is outside the scope of this specification, since the right strategies depend a lot on the client's operating environment. Still, here's one potential approach 💡
when a client's reclaim fails, it could look at the change attribute of the objects it's trying to reclaim state for, and use that to decide whether to re-establish the state through normal OPEN or LOCK requests. This is fine, as long as the client's operating environment allows it. In other words, the client implementer is advised to document this behavior for their users 📖 The client could also tell the application that its byte-range locks or share reservations (whether they were delegated or not) have been lost, like with a UNIX signal, a GUI pop-up window, etc. 🪟 For how the client should deal with unreclaimed delegations on client state, see Section 10.5.
For more on lock revocation, see Section 9.8. 🔍
9.7. Recovery from a Lock Request Timeout or Abort 🌈
so a lock request can time out, and then the client might decide not to retry it. The client can also abort the request when the process it was issued for gets terminated (for example, in UNIX, due to a signal). 😬 Here's the tea: the server might have received the request and acted on it anyway. That would change the state on the server without the client knowing. It is paramount that the client resynchronizes state with the server before it tries any other operation that takes a seqid and/or a stateid with the same state-owner. Thankfully, this is straightforward to do without a special resynchronize operation 🫶
Since the server keeps track of the last lock request and response received on each state-owner, the client should cache, for each state-owner, the last lock request it sent that didn't get a response. Then, the next time the client does a lock operation for that state-owner, it can send the cached request, if there is one. If the request was one that established state (like a LOCK or OPEN operation), the server will return the cached result or, if it never saw the request, perform it. ✨ The client can then follow up with a request to remove the state (like a LOCKU or CLOSE operation). With this approach, the sequencing and stateid info on the client and server for the given state-owner resynchronize, and in turn, the lock state resynchronizes too. 💖
9.8. Server Revocation of Locks ✨
At any point, the server can revoke locks held by a client, and the client has to be ready for that 🫶. When the client detects that its locks have been, or might have been, revoked, it's on the client to check the state info between itself and the server. Validating locking state means the client must verify or reclaim state for each lock it currently holds.
The first revocation event happens when the server reboots or gets re-initialized 🔄. The client gets an error (NFS4ERR_STALE_STATEID or NFS4ERR_STALE_CLIENTID) and then goes through normal crash recovery, which is covered in the previous section.
The second revocation event is when the lease can't be renewed before it expires ⏰. It's a rare, unusual thing, but the client must be prepared to recover anyway. Both the server and the client can notice a failed renewal, and both can recover without data corruption, which is so important 💅. The server keeps track of the last renewal it handled for each client and knows when the lease will run out. The client has to track its own renewing operations too. By looking at when each request was sent and when its reply came back, the client should bound the time the renewal could have happened on the server, and figure out whether a lease expiration could have happened.
The third revocation event can happen because of administrative intervention within the lease period 👑. Also rare, but the server's administrator might decide to release or revoke a particular lock the client holds. Then the client gets an error of NFS4ERR_ADMIN_REVOKED. In that case, the client may assume that only the state-owner's locks were lost, and it notifies the lock holder appropriately. What the client can't assume is that the lease was renewed just because an operation failed 🙈.
When the client decides the lease period may have expired, it must mark all locks held for the associated lease as "unvalidated" 🚩. That means the client hasn't been able to re-establish or confirm the right lock state with the server. As Section 9.6 explains, the server may grant conflicting locks after a client's lease has expired, so whenever it's possible the lease expired, the client must validate each lock it holds to make sure no conflicting lock was granted. One way to do that is with an I/O request. If there's no relevant I/O pending, a zero-length read specifying the stateid for the lock in question can be synthesized to trigger the renewal. If that response is a success, the client has validated every lock governed by that stateid and re-established the appropriate state with the server 🎉.
If the I/O request isn't successful, then one or more of the locks tied to that stateid were revoked by the server, and the client must notify the owner 💀.
9.9. Share Reservations
A share reservation is a mechanism for controlling access to a file 🎀. It's separate and independent from byte-range locking, so don't mix them up. When a client opens a file, it sends an OPEN operation to the server specifying the access it needs (READ, WRITE, or BOTH) and the access it wants to deny to everyone else (OPEN4_SHARE_DENY_NONE, OPEN4_SHARE_DENY_READ, OPEN4_SHARE_DENY_WRITE, or OPEN4_SHARE_DENY_BOTH). If the OPEN fails, the client fails the application's open request 😭.
Here's the pseudo-code for the semantics, bestie 🧚:
if (request.access == 0)
return (NFS4ERR_INVAL)
else if ((request.access & file_state.deny) ||
(request.deny & file_state.access))
return (NFS4ERR_DENIED)This share reservation checking on OPEN has no exception for an existing OPEN from the same open-owner. No favourites, periodt 💅.
The constants used for the access and deny fields in the OPEN and OPEN_DOWNGRADE operations look like this:
const OPEN4_SHARE_ACCESS_READ = 0x00000001;
const OPEN4_SHARE_ACCESS_WRITE = 0x00000002;
const OPEN4_SHARE_ACCESS_BOTH = 0x00000003;
const OPEN4_SHARE_DENY_NONE = 0x00000000;
const OPEN4_SHARE_DENY_READ = 0x00000001;
const OPEN4_SHARE_DENY_WRITE = 0x00000002;
const OPEN4_SHARE_DENY_BOTH = 0x00000003;9.10. OPEN/CLOSE Operations
For correct share semantics, a client MUST use the OPEN operation to get the initial filehandle and say what access it wants and what access, if any, it wants to deny 🔐. Even if the client plans to use one of the special stateids (the anonymous stateid or the READ bypass stateid), it must still get the filehandle for the regular file through OPEN so the right share semantics can be applied. Clients whose programming interfaces don't have a deny mode for opening a file should ask for a deny mode of OPEN4_SHARE_DENY_NONE.
The OPEN operation with the CREATE flag also takes over the CREATE operation for regular files from previous versions of the NFS protocol. That means a create with a share can happen atomically ⚡.
The CLOSE operation removes all share reservations the open-owner holds on that file. If byte-range locks are held, the client SHOULD release all of them before issuing a CLOSE. The server MAY free all outstanding locks on CLOSE, but some servers might not support closing a file that still has byte-range locks held. If any locks would exist after the CLOSE, the server MUST return failure, NFS4ERR_LOCKS_HELD 🚫.
The LOOKUP operation hands back a filehandle without setting up any lock state on the server 🔍. Without a valid stateid, the server assumes the client has the least access. For example, say one client opened a file with OPEN4_SHARE_DENY_BOTH, and another client reaches that file through a filehandle it got from LOOKUP. That second client could only read the file, and only by using the special READ bypass stateid. It couldn't WRITE the file at all, since it wouldn't have a valid stateid from OPEN, and the special anonymous stateid wouldn't be allowed access 🙅♀️.
9.10.1. Close and Retention of State Information
A CLOSE operation asks for a stateid to be deallocated, so handling a retransmitted CLOSE can get tricky 😬. The state information that would normally tell the server which open file is being referred to might already be gone, and that would produce an NFS4ERR_BAD_STATEID error.
Servers have a few ways to deal with this. For the greatest assurance that the protocol is being used properly, a server should, instead of deallocating the stateid, mark it as close-pending and keep it with that status until it's deallocated later. That way, a retransmitted CLOSE can be recognized, because the stateid points to state information with this distinctive status, and it can be handled without error 🧠.
When a server takes this approach, it should keep the state information until the earliest of these happens:
- Another validly sequenced request for the same open-owner comes in, one that isn't a retransmission.
- The server frees the open-owner because it's been inactive for a period of time.
- All locks for the client are freed as a result of a SETCLIENTID.
Servers can skip this complexity, at the cost of less complete protocol error checking, by just responding NFS4_OK when a CLOSE comes in for an already deallocated stateid. The reasoning is that this case must have been caused by a retransmitted close 🤷♀️. If a server does this, it's desirable to at least log an error when it returns a no-error indication in this situation. If the server keeps a reply-cache mechanism, it can verify that the CLOSE really is a retransmission and avoid error logging in most cases.
9.11. Open Upgrade and Downgrade
When an OPEN is done for a file and the open-owner already has that file open, the result is an upgrade of the open file status kept on the server 🆙. It now includes the access and deny bits from the new OPEN as well as the ones from the existing OPEN. As far as the protocol is concerned, there's one open file, and it holds the union of the access and deny bits from all the OPEN requests that completed. Only a single CLOSE is needed to reset the effects of both OPENs. Keep in mind that when the client issues the OPEN, it might not know the same file is actually being opened. All of this only applies if both OPENs result in the OPENed object being designated by the same filehandle.
When the server chooses to export multiple filehandles for the same file object, and it returns different filehandles on two different OPENs of that file object, the server MUST NOT "OR" the access and deny bits together and coalesce the two open files 🚷. Instead, the server must maintain separate OPENs with separate stateids, and it will require separate CLOSEs to free them.
When multiple open files on the client get merged into a single open file object on the server, closing one of the client's open files may mean the server has to change the access and deny status of the open file. That's because the union of access and deny bits for the remaining opens might be smaller (a proper subset) than before 📉. The OPEN_DOWNGRADE operation makes that change, and the client should use it to update the server so that share reservation requests from other clients get handled properly. The stateid returned has the same "other" field as the one passed to the server. The seqid value in the returned stateid MUST be incremented (Section 9.1.4), even when the access and deny bits for the file haven't changed at all 🔢.
9.12. Short and Long Leases
When picking the time period for the server lease, the usual lease trade-offs apply ⚖️. Short leases are good for fast server recovery, but they cost more RENEW or READ (with zero length) requests. Longer leases are much kinder and gentler to servers handling huge numbers of clients 🥰. The number of RENEW requests drops in proportion to the lease time. The downsides of long leases are slower recovery after a server failure, because the server has to wait for the leases to expire and the grace period to elapse before it grants new lock requests, and more file contention, because if the client fails to send an unlock request, the server has to wait for lease expiration before it can grant new locks 🐢.
Long leases are workable if the server can store lease state in non-volatile memory. On recovery, the server can rebuild the lease state from that memory and keep operating with its clients, so long leases wouldn't be an issue 💜.
9.13. Clocks, Propagation Delay, and Calculating Lease Expiration ⏰
okay so, lease times are granted by the server as a time delta 🫶 so we don't need synchronized clocks, which is honestly iconic. BUT clients and servers still can't drift excessively over the duration of the lock, or things get messy 💀. Then there's propagation delay across the network, which could easily be several hundred milliseconds, plus the chance that requests get lost and need retransmitting 😭
so to account for propagation delay, the client should subtract it from lease times. like if the client estimates the one-way propagation delay as 200 msec, it can assume the lease is already 200 msec old when it gets it ✨ then it takes another 200 msec to get a response back to the server. so the client must send a lock renewal or write data back to the server 400 msec before the lease would expire. the math is giving precision 🧮💅
the server's lease period configuration should take into account the network distance of the clients that will be accessing its resources 🌍 the lease period is expected to account for network propagation delays and other network delay factors for the client population. the protocol doesn't have an automatic way to figure out a good lease period, so the server's administrator may have to tune it by hand 🎀
9.14. Migration, Replication, and State 🦋
okay this section is the drama 🍿 when responsibility for a file system gets transferred to a new server (migration), or the client chooses an alternative server (like when the current one is unresponsive) in the context of file system replication, the handling of shared state (locks, leases, stateids, and client IDs) is different. migration and replication handle it differently, so let's go 💜 for the related stuff on file server state and recovering it, check the subsections of Section 9.6.
in cases where one server is expected to accept opaque values from the client that came from another server, the servers SHOULD encode those opaque values in big-endian byte order 🌈 if they do, the new server can parse values like stateids, directory cookies, filehandles, and so on, even if its native byte order is different from the other servers cooperating in the replication and migration of the file system. bestie, that's a whole glow-up 🫶
9.14.1. Migration and State 🧚
in migration, the servers involved SHOULD transfer all server state from the original server to the new server ✨ this MUST be done in a way that's transparent to the client. the state transfer eases the client's transition when a migration happens. if the servers succeed at transferring all state, the client keeps using stateids assigned by the original server, so the new server must recognize those stateids as valid. same goes for the client ID 🔑 and since responsibility for an entire file system moves in a migration, there's no chance of lock conflicts on the new server from the transfer. ate 💅
leases get transferred too 🦋 the leases going to the new server will usually have a different expiration time from the same client's leases on the old server. to keep the property that all leases on a given server for a given client expire at the same time, the server should push the expiration time forward to the later of the transferred leases or the leases already there. this lets the client renew both classes without any special effort 🙌
the servers may choose not to transfer state upon migration, but this choice is discouraged 🙅♀️ if they don't, and the client presents state from the original server (like in a RENEW operation or a zero-length READ), the client must be ready to get either NFS4ERR_STALE_CLIENTID or NFS4ERR_STALE_STATEID back from the new server 😭 then the client should recover its state the same way it normally would after a server failure. and the new server must take care to allow for state recovery just like it would after a server restart 🫠
a client SHOULD re-establish new callback information with the new server as soon as possible, following the sequences in Sections 16.33 and 16.34 ⚡ this makes sure server operations aren't blocked by not being able to recall delegations. main character energy for the server, honestly 👑
9.14.2. Replication and State 🌸
replication is a different story because client switch-over isn't under the server's control 💀 so leases, stateids, and client IDs do NOT carry validity across a transition from one server to another. the client must re-establish its locks on the new server. this is kind of like re-establishing locks with reclaim-type requests after a server reboot, EXCEPT the server has no way to tell reclaim requests apart from requests for brand-new locks, or to hold off the new ones 🙈 so a client re-establishing a lock on the new server (through a LOCK or OPEN request) might get denied because of a conflicting lock.
ngl, that's not the end of the world, since replication is meant for read-only use of file systems, so denied locks shouldn't be a huge problem in practice 🍓 if an attempt to re-establish a lock on the new server gets denied, the client should treat it as if its original lock had been revoked 🥀
9.14.3. Notification of Migrated Lease 📬
okay so with lease renewal, the client might not be sending requests for a file system that got migrated to another server 😭 that happens because of implicit lease renewal. the client renews leases for all file systems whenever it submits a request to any one file system on the server. so yeah, sneaky 👀
for the client to schedule renewal of leases that may have moved to the new server, it has to find out about the relocation before those leases expire ⏳ to make that happen, all operations that implicitly renew a client's leases (like OPEN, CLOSE, READ, WRITE, RENEW, LOCK, and others) return the error NFS4ERR_LEASE_MOVED if responsibility for any of the leases to be renewed has moved to a new server. this sticks around until the client gets an NFS4ERR_MOVED error AND the server receives the follow-up GETATTR(fs_locations) for an access to each file system that had a lease moved to a new server 💅 by convention, the compound containing the GETATTR(fs_locations) SHOULD append a RENEW operation so the server can identify the client doing the access.
when a client that supports file system migration gets NFS4ERR_LEASE_MOVED, it MUST probe all the file systems on that server where it holds open state 🔍 once it has successfully probed all the migrated file systems, the server MUST resume normal handling of stateful requests from that client ✨
legacy clients don't handle NFS4ERR_LEASE_MOVED correctly, so to support them, the server SHOULD time out after waiting at least two lease periods, and then it resumes normal handling of stateful requests from all clients 🫶 if a client tries to access the migrated files during that, the server MUST reply with NFS4ERR_MOVED 🚫
when the client gets NFS4ERR_MOVED, it can follow the usual process to get the new server info (through the fs_locations attribute) and renew those leases on the new server 🧚 if the server didn't have state transferred to it transparently, the client gets either NFS4ERR_STALE_CLIENTID or NFS4ERR_STALE_STATEID from the new server, like we covered above. then the client can recover its state the same way it does after a server failure 🌈
9.14.4. Migration and the lease_time Attribute ⏱️
so the client can manage its leases properly during a migration, the destination server has to set proper values for the lease_time attribute 🎀
when state is transferred transparently, that state should include the correct value of lease_time ✨ the lease_time on the destination server must NEVER be less than the one on the source, because that would cause premature expiration of leases the source server granted 💀 in a transparent migration, the client has no obligation to refetch lease_time and can keep using the value it previously fetched from the source server 🙌
if state hasn't been transferred transparently (like when the client sees a real or simulated server reboot), the client should fetch lease_time from the new (destination) server and use it for later locking requests. BUT the server must respect a grace period at least as long as the lease_time on the source server 🛡️ this makes sure clients have ample time to reclaim their locks before potentially conflicting non-reclaimed locks get granted. how the new server gets the old server's lease_time is up to the server implementations. the NFSv4 protocol doesn't specify it 🤷♀️
10. Client-Side Caching 💾
client-side caching of data, file attributes, and filenames is essential for good performance with NFS 🚀 distributed cache coherence is a seriously hard problem, and previous NFS versions didn't even try 😭 instead, several NFS client implementations used techniques to reduce the problems that a lack of coherence causes for users. those techniques weren't clearly defined by earlier specs, so it was often unclear what counted as valid or invalid client behavior 🫠
the NFSv4 protocol uses lots of techniques similar to earlier versions 🦋 it does NOT provide distributed cache coherence, but it defines a more limited set of caching guarantees, so locks and share reservations can be used without destructive interference from client-side caching 🔒 ate 💅
on top of that, NFSv4 introduces a delegation mechanism that lets a bunch of decisions normally made by the server get made locally by clients 👑 this is super efficient for the common cases where sharing is infrequent or where sharing is read-only 🌸
10.1. Performance Challenges for Client-Side Caching ✨
Caching techniques from earlier NFS versions have been seriously successful at giving good performance, no cap. But when you have a huge number of clients, some scalability challenges can pop up. It's extra true when clients are spread across the globe, because that usually makes latency for cache revalidation requests go up 😭.
Earlier NFS versions repeat their file data cache validation requests every time a file gets opened, and that can be a real performance bummer 💀. The most common situation is a file that only one client ever touches, so sharing is super rare 🦋.
In that case, checking with the server again and again just to hear "no conflicts, all good" is expensive. A better move for performance is letting a client that keeps reopening a file do it without asking the server, until a potentially conflicting operation from another client actually happens 🎀.
File locking has a similar problem. Sending lock and unlock requests to the server, plus the READ and WRITE requests needed to keep data caching consistent with the locking semantics (see Section 10.3.2), can seriously limit performance. If you're using locks to protect against infrequent conflicts, the penalty is big, and that can scare applications away from locking altogether 🙈.
NFSv4 brings more aggressive caching strategies, with these design goals:
- Compatibility with a large range of server semantics.
- Providing the same caching benefits as previous versions of the NFS protocol when the more aggressive model can't be provided.
- Organizing requirements for aggressive caching so that a large portion of the benefit can still be obtained even when not all of the requirements can be met.
The requirements for the server are discussed in later sections, where specific forms of caching are covered (see Section 10.4) 💜.
10.2. Delegation and Callbacks 🌈
Recallable delegation of server responsibilities for a file to a client is a total performance glow-up 🔥, because it avoids repeated requests to the server when there's no inter-client conflict. With a "callback" RPC from server to client, the server recalls delegated responsibilities when another client starts sharing a delegated file.
A delegation gets passed from the server to the client, specifying the object of the delegation and the type of delegation. There are different types, but each one contains a stateid that represents the delegation when doing operations that depend on it. This stateid is similar to the ones for locks and share reservations, but it differs: the stateid for a delegation is associated with a client ID and can be used on behalf of all the open-owners for that client. A delegation is made to the client as a whole, not to any specific process or thread inside it 🧚.
Callback RPCs may not work in every environment (firewalls, for example), so correct protocol operation doesn't depend on them. Preliminary testing of callback functionality by means of a CB_NULL procedure determines whether callbacks can be supported. The CB_NULL procedure checks the continuity of the callback path. A server makes a preliminary assessment of callback availability for a given client and avoids delegating responsibilities until it has determined that callbacks are supported. Since granting a delegation is always conditional on the absence of conflicting access, clients must not assume that a delegation will be granted, and they must always be prepared for OPENs to be processed without any delegations being granted 🫣.
Once granted, a delegation behaves in most ways like a lock. There's an associated lease that's subject to renewal, along with all the other leases held by that client 💅.
Unlike locks, an operation by a second client to a delegated file will cause the server to recall the delegation through a callback 👑.
On recall, the client holding the delegation must flush modified state (such as modified data) to the server and return the delegation. The conflicting request won't be acted on until the recall is complete. The recall is considered complete when the client returns the delegation, or when the server times out its wait for the delegation to be returned and revokes it as a result of the timeout. In the meantime, the server will either delay responding to conflicting requests or respond to them with NFS4ERR_DELAY. Once the recall resolves, the server has the information it needs to grant or deny the second client's request ⏳.
When a client gets a delegation recall, it may be holding a lot of state that needs flushing to the server. So the server should allow sufficient time for the delegation to come back, since that may involve numerous RPCs to the server. If the server can tell that the client is diligently flushing state as a result of the recall, the server MAY extend the usual time allowed for a recall. However, the time allowed for recall completion should not be unbounded 🍓.
Here's an example: responsibility to mediate opens on a given file is delegated to a client (see Section 10.4). The server won't know which opens are in effect on the client. Without that knowledge, the server can't determine whether the access and deny state for the file allows any particular open until the delegation for the file has been returned 🙈.
A client failure or a network partition can result in failure to respond to a recall callback. In that case, the server will revoke the delegation, and this in turn renders useless any modified state still on the client 😱.
Clients need to be aware that server implementers may enforce practical limitations on the number of delegations issued. Further, since there's no way to determine which delegations to revoke, the server is allowed to revoke any of them. If the server is implemented to revoke another delegation held by that client, the client may be able to determine that a limit has been reached, because each new delegation request results in a revoke. The client could then determine which delegations it may not need and preemptively release them 🫶.
10.2.1. Delegation Recovery ✨
okay bestie, delegation recovery has to handle three situations 🎀
- Client reboot or restart
- Server reboot or restart (see Section 9.6.3.1)
- Network partition (full or callback-only)
So if the client reboots or restarts, confirming a SETCLIENTID with an nfs_client_id4 that has a new verifier4 value releases byte-range locks and share reservations. Delegations, though, get treated a bit differently 💅
Sometimes delegations need to be re-established after a client reboots, because the client may have file data stored locally that was tied to the delegations it held before. The client will need to re-establish the right file state on the server 🧚
To allow this kind of client recovery, the server MAY allow delegations to be retained after other sorts of locks are released. That means requests from other clients that conflict with those delegations have to wait 🙈 A normal recall can take real time for the client to flush changed state to the server, so other clients need to be ready for delays caused by a conflicting delegation. To give clients a chance to get through the reboot (leases aren't renewed during it), the server MAY extend the delegation recovery period beyond the usual lease expiration period. For open delegations, the ones that weren't released get reclaimed with OPEN using a claim type of CLAIM_DELEGATE_PREV. (See Sections 10.5 and 16.16 for the discussions of open delegation and the details of OPEN, respectively.)
A server MAY support CLAIM_DELEGATE_PREV, but if it does, it MUST NOT remove delegations upon SETCLIENTID_CONFIRM. Instead, it MUST make them available for client reclaim using CLAIM_DELEGATE_PREV. The server MUST NOT remove the delegations until either the client does a DELEGPURGE or one lease period has elapsed from the time (whichever is later) of the SETCLIENTID_CONFIRM or the last successful CLAIM_DELEGATE_PREV reclaim. ⏳
Now, that requirement doesn't mean that once the server is no longer obliged to keep delegation information, it should necessarily toss it. Some specific cases:
- When the period is ended by
DELEGPURGE, deleting unreclaimed delegations is appropriate and desirable. - When the period is ended by a lease period elapsing without a successful
CLAIM_DELEGATE_PREVreclaim, and that situation looks like a network partition (i.e., lease expiration has occurred), the server's lease expiration approach, possibly including the use of courtesy locks, would normally keep unreclaimed delegations. Even if lease cancellation happens, the delegation should be reclaimed usingCLAIM_DELEGATE_PREVas part of network partition recovery. - When a period of non-communication is followed by a client reboot, unreclaimed delegations should also be reclaimable using
CLAIM_DELEGATE_PREVas part of client reboot recovery. - When the period is ended by a lease period elapsing without a successful
CLAIM_DELEGATE_PREVreclaim, and lease renewal is happening, the server may well conclude that unreclaimed delegations have been abandoned and treat the situation as one where an impliedDELEGPURGEshould be assumed.
A server that supports CLAIM_DELEGATE_PREV MUST support the DELEGPURGE operation, and similarly, a server that supports DELEGPURGE MUST support CLAIM_DELEGATE_PREV. A server that does not support CLAIM_DELEGATE_PREV MUST return NFS4ERR_NOTSUPP if the client attempts to use that feature or performs a DELEGPURGE operation. 🚫
Support for CLAIM_DELEGATE_PREV is often called providing "client-persistent delegations" 💜 They let the client use persistent storage to hold data it wrote, even across a client restart. It's worth noting that, with the optional exception covered below, this feature requires persistent storage on the client and doesn't add to the persistent storage requirements on the server.
Think of client-persistent delegations as mostly acting like "courtesy locks", with some special semantic tweaks so they can be kept across a client restart, which frees all other sorts of locks. They generally aren't kept across a server restart. The one exception is simultaneous failure of the client and server, which gets its own discussion below.
When the server indicates support for CLAIM_DELEGATE_PREV (implicitly) by returning NFS_OK to DELEGPURGE, a client holding a write delegation can use write-back caching for data headed to the server. It can defer the write-back until the delegation is recalled, even if there are client restarts in between 🍓 Similarly, when the server indicates support for CLAIM_DELEGATE_PREV, a client with a read delegation and an open-for-write subordinate to that delegation can be sure its persistently cached copy of the file is still good after a client restart, without specifically verifying the change attribute.
When the server reboots or restarts, delegations get reclaimed (using the OPEN operation with CLAIM_PREVIOUS) in a similar way to byte-range locks and share reservations. There is a slight semantic difference, though 👑 Normally, if the server decides a delegation shouldn't be granted, it performs the requested action (e.g., OPEN) without granting any delegation. For reclaim, the server grants the delegation, but applies a special designation so the client treats the delegation as having been granted but recalled by the server. Because of this, the client has the duty to write all modified state to the server and then return the delegation. This handling of delegation reclaim balances three principles of the NFSv4 protocol:
- Upon reclaim, a client claiming resources assigned to it by an earlier server instance must be granted those resources.
- The server has unquestionable authority to determine whether delegations are to be granted and, once granted, whether they are to be continued.
- The use of callbacks is not to be depended upon until the client has proven its ability to receive them.
When a client has more than one open associated with a delegation, state for those extra opens can be established using OPEN operations of type CLAIM_DELEGATE_CUR. When these are used to establish opens tied to reclaimed delegations, the server MUST allow them when made within the grace period. 🦋
Situations with a series of client and server restarts, where the client and server never restart at the exact same time, are handled with a mix of CLAIM_DELEGATE_PREV and CLAIM_PREVIOUS reclaim cycles. Persistent storage is only needed on the client. For each server failure there's a CLAIM_PREVIOUS reclaim cycle, and for each client restart there's a CLAIM_DELEGATE_PREV reclaim cycle.
To deal with the possibility of simultaneous failure of client and server (e.g., a data center power outage), the server MAY persistently store delegation information so it can respond to a CLAIM_DELEGATE_PREV reclaim request that comes in from a restarting client. This is the one case in which persistent delegation state can be retained across a server restart. A server is not required to store this information, but if it does, it should do so for write delegations and for read delegations during whose pendency (across multiple client and/or server instances) some open-for-write was done as part of the delegation. When the space to persistently record such information is limited, the server should recall delegations in this class in preference to keeping them active without persistent storage recording. 💾
When a network partition occurs, the server can free delegations once the lease renewal period expires. This is similar to how locks and share reservations behave, and just like for those, it may be modified by support for "courtesy locks", where locks aren't freed in the absence of a conflicting lock request. Here's the twist: locks and share reservations get freed right away when a conflicting request shows up, but for delegations, the server MAY institute a period during which conflicting requests are held off. Eventually, a conflicting request from another client will cause the delegation to be revoked. 😤
A loss of the callback path (e.g., from a later network configuration change) has a similar effect and can also result in revocation of a delegation. A recall request will fail, and revocation of the delegation will result.
A client usually finds out about a revoked delegation when it uses a stateid associated with that delegation and gets back one of the errors NFS4ERR_EXPIRED, NFS4ERR_BAD_STATEID, or NFS4ERR_ADMIN_REVOKED. (NFS4ERR_EXPIRED means all lock state associated with the client has been lost.) It may also find out about the revocation after a client reboot, when it tries to reclaim a delegation and gets NFS4ERR_EXPIRED. Heads up: for a revoked OPEN_DELEGATE_WRITE delegation there are issues, because data may have been modified both by the client whose delegation was revoked and separately by other clients. See Section 10.5.1 for a discussion of those issues. Also, when delegations are revoked, the server writes information about the revoked delegation to stable storage (as described in Section 9.6). This covers the case where a server reboots after revoking a delegation but before the client holding it has been notified 🫶
When a delegation is lost because of a network partition in which all locks associated with the lease are lost, the client will also receive the error NFS4ERR_EXPIRED. You can tell this apart from other revocation situations by seeing that the associated clientid has become invalid, so NFS4ERR_STALE_CLIENTID gets returned when it's used. ✨
When NFS4ERR_EXPIRED is returned, the server MAY retain information about the delegations the client held, deleting only those invalidated by a conflicting request. Keeping that info lets the client recover all non-invalidated delegations using the claim type CLAIM_DELEGATE_PREV, once the SETCLIENTID_CONFIRM is done to recover. Trying to recover a delegation the client has no record of (typically because a conflicting request invalidated it) will result in the error NFS4ERR_BAD_RECLAIM. Once a reclaim has been attempted for all the delegations the client held, it SHOULD do a DELEGPURGE so any remaining server delegation information can be freed 🧹
10.3. Data Caching 🧁
Okay bestie, when apps share access to the same files, they have to be built for the chance that another app is poking at that file too 👀. That's true whether the apps run on different clients or on the very same one.
Share reservations and byte-range locks are the tools NFSv4 gives apps to coordinate, basically mutual exclusion facilities 🔐. So the protocol's data caching has to be built so it never breaks the assumptions people make when they use those tools 💅.
10.3.1. Data Caching and OPENs ✨
To keep from messing up the sharing assumptions apps rely on, NFSv4 clients should not hand cached data to an application, or change it on an application's behalf, when that wouldn't be valid to get or change through a READ or WRITE operation. 🙅♀️
Furthermore, when there's no open delegation (see Section 10.4), two extra rules apply. Lowkey, many NFSv2 and NFSv3 clients already follow these in practice 🙌.
- First, cached data on a client must be revalidated after doing an
OPEN. Revalidating means the client fetches the change attribute from the server and compares it with the cached change attribute. If they're different, the client declares the cached data (as well as the cached attributes) invalid. This makes sure the data for the OPENed file is still correctly reflected in the client's cache 🔍. This validation must be done at least when the client'sOPENincludesDENY=WRITEorBOTH, which ends the stretch where other clients may have had a chance to open the file with WRITE access. Clients may choose to revalidate more often (like at OPENs specifyingDENY=NONE) to match the NFSv3 protocol's practice, for the benefit of users who expect that level of cache revalidation 🌸.
Since the change attribute gets updated for both data and metadata modifications, some implementers might be tempted to use the time_modify attribute instead of change to validate cached data. The goal would be to stop metadata changes from spuriously invalidating clean data 🙈. The implementer is cautioned against this approach. The change attribute is guaranteed to change for each update to the file, but time_modify is only guaranteed to change at the granularity of the time_delta attribute. Using time_modify instead of change for the client's data cache validation logic runs the risk of the client incorrectly marking stale data as valid 💀.
- Second, modified data must be flushed to the server before closing a file OPENed for write. This complements the first rule. If the data isn't flushed at
CLOSE, the revalidation done after the client OPENs a file can't do its job 😭. The other part of flushing before close is that the data must be committed to stable storage at the server before the client requests theCLOSEoperation. If the server reboots or restarts and the file is already CLOSEd, it may not be possible to retransmit the data to be written to the file. That's why this requirement exists 🫶.
10.3.2. Data Caching and File Locking 🔒
For apps that use file locking instead of share reservations to keep file access consistent, there's an analogous set of constraints for client-side data caching. These rules only work if the file locking matches, in an equivalent way, the actual READ and WRITE operations being executed. That's different from locking that's based on pure convention 📜.
Here's an example. You can manipulate a two-megabyte file by splitting it into two one-megabyte regions and protecting access to each region with file locks on byte zero and byte one 🧩. A write lock on byte zero represents the right to do READ and WRITE operations on the first region. A write lock on byte one represents the right to do READ and WRITE operations on the second region. As long as every app working with the file follows this convention, it all works on a local file system ✨. But it may not work with NFSv4 unless clients refrain from data caching 🥲.
The rules for data caching in the file locking environment are:
- First, when a client obtains a file lock for a particular region, the data cache for that region (if any cached data exists) must be revalidated. If the change attribute indicates the file may have been updated since the cached data was obtained, the client must flush or invalidate the cached data for the newly locked region. A client might choose to invalidate all of the non-modified cached data it has for the file, but the only requirement for correct operation is invalidating all of the data in the newly locked region 🧹.
- Second, before releasing a write lock for a region, all modified data for that region must be flushed to the server. The modified data must also be written to stable storage 💾.
Note that flushing data to the server and invalidating cached data must reflect the actual byte ranges locked or unlocked. Rounding them up or down to match client cache block boundaries will cause problems if it isn't done super carefully 🙃. For example, writing a modified block when only half of that block is inside the area being unlocked may cause an invalid modification to the region outside the unlocked area. That region might be part of a region locked by another client 😱. Clients can avoid this by synchronously performing the portions of WRITE operations that overlap the part (initial or final) that isn't a full block. Similarly, invalidating a locked area that isn't a whole number of full buffer blocks would force the client to read one or two partial blocks from the server if the revalidation procedure shows its data may not be valid 📉.
The data written to the server as a prerequisite to unlocking a region must be written, at the server, to stable storage. The client can accomplish this with synchronous writes or by following asynchronous writes with a COMMIT operation. This is required because retransmitting the modified data after a server reboot might conflict with a lock held by another client 🚫.
A client implementation may choose to accommodate applications that use byte-range locking in non-standard ways (for example, using a byte-range lock as a global semaphore) by flushing more data to the server upon a LOCKU than the locked range covers. This may include modified data within files other than the one the unlocks are being done for 🗂️. In such cases, the client must not interfere with applications whose READs and WRITEs are only done within the bounds of record locks that the application holds. For example, an application locks a single byte of a file and proceeds to write that single byte. A client that handles a LOCKU by flushing all modified data to the server could validly write that single byte in response to an unrelated unlock. However, it would not be valid to write the entire block that single byte sits in, since that block includes an area that isn't locked and might be locked by another client 🧱. Client implementations can avoid this problem by dividing files with modified data into two groups: those where all modifications are done to areas covered by an appropriate byte-range lock, and those where there are modifications not covered by a byte-range lock. Any writes done for the first group of files must not include areas that aren't locked and therefore not modified on the client 🎀.
10.3.3. Data Caching and Mandatory File Locking 🛡️
Client-side data caching needs to respect mandatory file locking when it's in effect. Its presence for a given file is indicated when the client gets back NFS4ERR_LOCKED from a READ or WRITE on a file it has an appropriate share reservation for 🚨. When mandatory locking is in effect for a file, the client must check for an appropriate file lock on the data being read or written. If a lock exists for the range being read or written, the client may satisfy the request using its validated cache 🧠. If an appropriate file lock isn't held for the range of the READ or WRITE, the request must not be satisfied from the client's cache, and it must be sent to the server for processing 📨. When a READ or WRITE request partially overlaps a locked region, the request should be subdivided into multiple pieces, with each region (locked or not) treated appropriately ✂️.
10.3.4. Data Caching and File Identity 🪪
When clients cache data, the file data needs to be organized according to the file system object it belongs to. For NFSv3 clients, the typical practice has been to assume, for caching purposes, that distinct filehandles represent distinct file system objects. The client then gets to choose how to organize and maintain the data cache on that basis 🗃️.
In the NFSv4 protocol, there's now the possibility of significant deviations from the "one filehandle per object" model, because a filehandle may be constructed from the object's pathname 🧵. So clients need a reliable method to figure out whether two filehandles designate the same file system object. If clients just assumed that all distinct filehandles denote distinct objects and did their data caching on that basis, caching inconsistencies would pop up between distinct client-side objects that mapped to the same server-side object 😵💫.
The NFSv4 protocol's method for telling filehandles apart takes care of a potential functional regression compared with NFSv3 🫧. Without that method, caching inconsistencies within the same client could happen, and that hasn't been the case in previous versions of the NFS protocol. It's possible to have these inconsistencies with applications running on multiple clients, but that's not the issue being addressed here 🙅♀️.
For data caching purposes, the following steps let an NFSv4 client determine whether two distinct filehandles denote the same server-side object 🔎:
- If
GETATTRdirected to two filehandles returns different values of thefsidattribute, then the filehandles represent distinct objects. - If
GETATTRfor any file with anfsidthat matches thefsidof the two filehandles in question returns aunique_handlesattribute with a value ofTRUE, then the two objects are distinct. - If
GETATTRdirected to the two filehandles doesn't return thefileidattribute for both of the handles, then it can't be determined whether the two objects are the same. Therefore, operations that depend on that knowledge (for example, client-side data caching) can't be done reliably 😬. Note that ifGETATTRdoesn't return thefileidattribute for both filehandles, it won't return it for either of the filehandles, since thefsidfor both filehandles is the same. - If
GETATTRdirected to the two filehandles returns different values for thefileidattribute, then they are distinct objects. - Otherwise, they are the same object 💖.
10.4. Open Delegation ✨
okay so when a file is getting OPENed, the server is allowed to hand the opening client some extra power 💅. It can let that client handle further OPENs and CLOSEs for the file on its own 🦋. Any delegation like this is recallable, bestie, because the circumstances that made it possible can totally change 🔄. Like, the server might get a conflicting OPEN from another client, and then the server must recall the delegation before deciding whether that other OPEN can be granted 😤. Making a delegation is up to the server, so clients should not assume that any particular OPEN either will or will not result in an open delegation 🤷♀️. Here's a typical set of conditions servers might use to decide whether an OPEN should be delegated 📝:
- The client must be able to respond to the server's callback requests. The server will use the
CB_NULLprocedure for a test of callback ability 📞. - The client must have responded properly to previous recalls 🙌.
- There must be no current open conflicting with the requested delegation 🚫.
- There should be no current delegation that conflicts with the delegation being requested 🙅♀️.
- The probability of future conflicting open requests should be low, based on the recent history of the file 🔮.
- The existence of any server-specific semantics of
OPEN/CLOSEthat would make the required handling incompatible with the prescribed handling that the delegated client would apply (see below) 🧩.
There are two types of open delegations: OPEN_DELEGATE_READ and OPEN_DELEGATE_WRITE 👯. An OPEN_DELEGATE_READ delegation lets a client handle, on its own, requests to open a file for reading that do not deny read access to others 📖. It MUST, however, continue to send all requests to open a file for writing to the server ✍️. Multiple OPEN_DELEGATE_READ delegations can be outstanding at the same time and they do not conflict 🤝. An OPEN_DELEGATE_WRITE delegation lets the client handle all opens on its own 👑. Only one OPEN_DELEGATE_WRITE delegation can exist for a given file at a given time, and it is inconsistent with any OPEN_DELEGATE_READ delegations 🚷.
When a single client holds an OPEN_DELEGATE_READ delegation, it is assured that no other client may modify the contents or attributes of the file 🔒. If more than one client holds an OPEN_DELEGATE_READ delegation, then the contents and attributes of that file are not allowed to change 🧊. When a client has an OPEN_DELEGATE_WRITE delegation, it may modify the file data, since no other client will be accessing the file's data 💪. The client holding an OPEN_DELEGATE_WRITE delegation may only affect file attributes that are intimately connected with the file data: size, time_modify, and change 🎀.
When a client has an open delegation, it does not send OPENs or CLOSEs to the server. Instead, it updates the appropriate status internally 🧠. For an OPEN_DELEGATE_READ delegation, opens that cannot be handled locally (opens for write or that deny read access) must be sent to the server 📤.
When an open delegation is made, the response to the OPEN contains an open delegation structure that specifies the following 🎁:
- the type of delegation (read or write)
- space limitation information to control flushing of data on close (
OPEN_DELEGATE_WRITEdelegation only; see Section 10.4.1) 📦 - an
nfsace4specifying read and write permissions 🔐 - a stateid to represent the delegation for READ and WRITE 🪪
The delegation stateid is separate and distinct from the stateid for the OPEN proper 🧁. The standard stateid is associated with a particular open-owner, unlike the delegation stateid, and it will continue to be valid after the delegation is recalled and the file remains open 💖.
When a request internal to the client is made to open a file and open delegation is in effect, it will be accepted or rejected solely on the basis of the following conditions 🎯. Any requirement for other checks to be made by the delegate should result in open delegation being denied, so that the checks can be made by the server itself 🙅♀️.
- The access and deny bits for the request and the file, as described in Section 9.9 🔍.
- The read and write permissions, as determined below 👀.
The nfsace4 passed with the delegation can be used to avoid frequent ACCESS calls 🏃♀️. The permission check should work like this 🪄:
- If the
nfsace4indicates that the open may be done, then it should be granted without reference to the server ✅. - If the
nfsace4indicates that the open may not be done, then anACCESSrequest must be sent to the server to obtain the definitive answer ❓.
The server may return an nfsace4 that is more restrictive than the actual ACL of the file. This includes an nfsace4 that specifies denial of all access 🚫. Note that some common practices, such as mapping the traditional user "root" to the user "nobody", may make it incorrect to return the actual ACL of the file in the delegation response 😬.
The use of delegation, together with various other forms of caching, creates the possibility that no server authentication will ever be performed for a given user, since all of the user's requests might be satisfied locally 🫣. Where the client is depending on the server for authentication, the client should be sure authentication occurs for each user by use of theACCESSoperation 🔑. This should be the case even if anACCESSoperation would not be required otherwise. As mentioned before, the server may enforce frequent authentication by returning annfsace4denying all access with every open delegation 🛡️.
10.4.1. Open Delegation and Data Caching
OPEN delegation cuts out a lot of the message overhead that comes with opening and closing files 🎉. An open made while an open delegation is in effect does not require a validation message to be sent to the server, unless there is a potential for conflict with the requested share mode 🌸. The continued endurance of the "OPEN_DELEGATE_READ delegation" guarantees that no OPEN for write, and therefore no write, has occurred that did not originate from this client 💜. Similarly, when closing a file opened for write while an OPEN_DELEGATE_WRITE delegation is in effect, the data written does not have to be flushed to the server until the open delegation is recalled 🧺. The continued endurance of the open delegation guarantees that no open, and therefore no read or write, has been done by another client ✨.
For the purposes of open delegation, READs and WRITEs done without an OPEN (anonymous and READ bypass stateids) are treated as the functional equivalents of a corresponding type of OPEN 🧚. READs and WRITEs done with an anonymous stateid by another client will force the server to recall an OPEN_DELEGATE_WRITE delegation 📣. A WRITE with an anonymous stateid done by another client will force a recall of OPEN_DELEGATE_READ delegations 🔔. The handling of a READ bypass stateid is identical, except that a READ done with a READ bypass stateid will not force a recall of an OPEN_DELEGATE_READ delegation 🙌.
With delegations, a client can avoid writing data to the server when the CLOSE of a file is serviced 🍓. The file close system call is the usual point where the client is notified of a lack of stable storage for the modified file data generated by the application 😱. At close, file data is written to the server, and through normal accounting the server can determine whether the available file system space for the data has been exceeded (i.e., the server returns NFS4ERR_NOSPC or NFS4ERR_DQUOT) 💀. This accounting includes quotas 📊. Introducing delegations means an alternative method has to be in place for the same type of communication to happen between client and server 🔁.
In the delegation response, the server provides either the limit of the size of the file, or the number of modified blocks and the associated block size 📏. The server must ensure that the client will be able to flush to the server data of a size equal to what was provided in the original delegation 💯. The server must make this assurance for all outstanding delegations 🫶. So the server has to be really careful about managing available space for new or modified data, taking into account available file system space and any applicable quotas 🧠. The server can recall delegations as a result of managing the available file system space 🔙. The client should abide by the server's state space limits for delegations 🚧. If the client exceeds the stated limits for the delegation, the server's behavior is undefined 💀.
Based on server conditions, quotas, or available file system space, the server may grant OPEN_DELEGATE_WRITE delegations with very restrictive space limitations 🧃. The limitations may be defined in a way that always forces modified data to be flushed to the server on close 🚰.
With respect to authentication, flushing modified data to the server after a CLOSE has occurred can be problematic 🙈. For example, the user of the application may have logged off the client, and unexpired authentication credentials may not be present 🔐. In this case, the client may need to take special care to ensure that local unexpired credentials will actually be available 🗝️. One way to do this is to track the expiration time of credentials and flush data well in advance of their expiration ⏰.
10.4.2. Open Delegation and File Locks
When a client holds an OPEN_DELEGATE_WRITE delegation, lock operations may be performed locally 🔓. This includes those required for mandatory file locking 🔐. This works because the delegation implies there can be no conflicting locks 🙆♀️. Similarly, all of the revalidations that would normally be associated with obtaining locks, and the flushing of data associated with releasing locks, need not be done 🧹.
When a client holds an OPEN_DELEGATE_READ delegation, lock operations are not performed locally 🚫. All lock operations, including those requesting non-exclusive locks, are sent to the server for resolution 📨.
10.4.3. Handling of CB_GETATTR ✨
okay so the server needs to do something special for a GETATTR when the target file has an OPEN_DELEGATE_WRITE delegation in effect 👀 why? bc the client holding that delegation may have changed the data, and the server has to show that change to the second client that sent the GETATTR 🫶 so the server goes and asks the delegation holder with CB_GETATTR. ngl the only attributes the server can reliably ask for with CB_GETATTR are size and change 💅
since CB_GETATTR is only there to answer someone else's GETATTR, the server really just needs to know one thing: does the delegation holder have a modified copy of the file? 🤔 if the client's copy isn't modified (data or size), the server can answer the second client straight from the attributes it already has stored locally ✨ but if the file IS modified, the server only needs to know that, and then it answers the second client as if the file had been changed locally on the server 🔥
the change attribute is picked by the server and is opaque to the client, so the two of them have to agree on how to say "hey, this file is modified" 🦋 for size, the client just reports its current view of the file size. for change it's a bit more involved, so hold on bestie 🧚
for the client, when it gets an OPEN_DELEGATE_WRITE delegation, these steps happen:
- the value of the change attribute gets fetched from the server and cached. let's call this value
c💜 - the client makes up a value greater than
cthat means "I'm holding modified data." let's call thisd🌸 - when the client gets a CB_GETATTR for the change attribute, it checks whether it holds modified data. if the file is modified, it returns
das the change value. if the file isn't currently modified, it returnsc🎀
for simplicity of implementation, the client MAY return the same value d for each CB_GETATTR, even if it modifies the file's data or metadata in its cache again between CB_GETATTR calls 🙈 that's fine bc all the server needs is for the client to be able to say "I hold modified data," so d can always just be c + 1 ✨
okay now the tricky part 😭 change is opaque in the sense that the client has no clue what units of time (if any) the server is counting change in 🤷♀️ but it's not opaque in that the client has to treat it as an unsigned integer, and the server has to be able to see the results of the client's changes to that integer. so the server MUST encode the change attribute in network byte order when sending it to the client, the client MUST decode it from network byte order to its native order when receiving it, and the client MUST encode it in network byte order when sending it to the server. this is exactly why change is defined as an unsigned integer and not an opaque array of bytes 💖
for the server, these steps happen when it provides an OPEN_DELEGATE_WRITE delegation:
- upon providing that delegation, the server caches a copy of the change attribute in the data structure it uses to record the delegation. let's call this value
sc📝 - when a second client sends a GETATTR on the same file, the server gets the change attribute from the first client. let's call this value
cc🔍 - if
ccis equal tosc, the file isn't modified, so the server returns the current values for change, time_metadata, and time_modify (for example) to the second client 🌈 - if
ccisn't equal tosc, the file is currently modified at the first client and will most likely be modified at the server at a future time. the server then uses its current time to construct attribute values for time_metadata and time_modify. a new value ofsc, which we'll callnsc, gets computed by the server so thatnsc >= sc + 1✨ it returns the constructed time_metadata, time_modify, andnscvalues to the requester, and replacesscin the delegation record withnsc. to stop time_modify, time_metadata, and change from appearing to go backward (which would happen if the delegation holder fails to write its modified data to the server before the delegation is revoked or returned 😱), the server SHOULD update the file's metadata record with the constructed attribute values. committing the constructed attribute values to stable storage is OPTIONAL, for reasons of reasonable performance 💅
as we talked about earlier in this section, the client MAY return the same cc value on later CB_GETATTR calls, even if the file was modified in the client's cache yet again between those calls 🦋 so the server must assume the file has been modified yet again, and the server MUST take care to ensure that the new nsc it builds and returns is greater than the previous nsc it returned 🔥 an example implementation's delegation record would do this by including a boolean field (let's call it "modified") that's set to FALSE when the delegation is granted, plus an sc value set at grant time to the change attribute value. the modified field gets set to TRUE the first time cc != sc and stays TRUE until the delegation is returned or revoked. the processing for constructing nsc, time_modify, and time_metadata would use this pseudo-code:
if (!modified) {
do CB_GETATTR for change and size;
if (cc != sc)
modified = TRUE;
} else {
do CB_GETATTR for size;
}
if (modified) {
sc = sc + 1;
time_modify = time_metadata = current_time;
update sc, time_modify, time_metadata into file's metadata;
}this returns the attributes the client asked for to the client that sent GETATTR, but makes sure size comes from what CB_GETATTR returned. the server wouldn't update the file's metadata with the client's modified size 🙈
if the file attribute size is different from the server's current value, the server treats it as a modification no matter what the value of the change attribute retrieved via CB_GETATTR is, and responds to the second client just like in the last step 💀
this methodology sorts out clock differences between client and server, plus other scenarios where the use of CB_GETATTR breaks down 🧚
last but not least, the server isn't under any obligation to use CB_GETATTR, so the server MAY simply recall the delegation to avoid its use 🫶
10.4.4. Recall of Open Delegation ✨
okay so these are the moments when the server can recall an open delegation 👀
- Potentially conflicting
OPENrequest (orREAD/WRITEdone with "special" stateid) SETATTRissued by another clientREMOVErequest for the fileRENAMErequest for the file as either source or target of theRENAME
Whether a RENAME of a directory in the path leading to the file triggers a recall depends on the semantics of the server file system 🗂️ If that file system denies such RENAMEs when a file is open, the recall must be performed so the server can figure out whether the file is actually open 🫣
Also, the server may choose to recall open delegations at any time if resource constraints make it advisable to do so. Clients should always be prepared for the possibility of a recall 💅
When a client gets a recall for an open delegation, it needs to update state on the server before giving the delegation back 🎀 The exact same updates have to happen whenever a client decides to return a delegation voluntarily. These are the state items that need to be handled:
- If the file associated with the delegation is no longer open and no previous
CLOSEoperation has been sent to the server, aCLOSEoperation must be sent to the server. - If a file has other open references at the client, then
OPENoperations must be sent to the server. The server will hand back the appropriate stateids for the client to use later, since the delegation stateid will no longer be valid 🦋 TheseOPENrequests use the claim typeCLAIM_DELEGATE_CUR, which lets the client present the delegation stateid so it can establish the right to perform theOPEN. (See Section 16.16 for details.) - If there are granted file locks, the corresponding
LOCKoperations need to be performed. This applies to theOPEN_DELEGATE_WRITEdelegation case only. - For an
OPEN_DELEGATE_WRITEdelegation, if the file is not open for write at the time of the recall, all modified data for the file must be flushed to the server. If the delegation had never existed, the client would have flushed that data before theCLOSEoperation 🧹 - For an
OPEN_DELEGATE_WRITEdelegation, if the file is still open at the time of the recall, any modified data for the file needs to be flushed to the server 💖 - With an
OPEN_DELEGATE_WRITEdelegation in place, the file might have been truncated during the delegation. For example, the truncation could have come from anOPENwithUNCHECKED4and a size attribute value of zero. So if a truncation happened and was never propagated to the server, the truncation must occur before any modified data is written to the server 😱
With an OPEN_DELEGATE_WRITE delegation, file locking adds a few extra rules. To keep the associated invariant exactly right, any modified data in any region whose write lock was released while the OPEN_DELEGATE_WRITE delegation was in effect has to be flushed. Since this delegation means no other client is doing any locking, a simpler approach is to flush all modified data for the file (as described just above) if any write lock was released while the delegation was in effect 🔐
A client doesn't have to wait until a recall (or until it decides to return a delegation voluntarily) to do any of the above, if implementation considerations (like resource availability) make that a better move 🏃♀️ Still, since the actual open state of the file can keep changing, it usually isn't worth sending information about opens and closes to the server except as part of returning the delegation. The one case where clients are likely to act early is closing the open that produced the delegation, because once that close is done, it won't be undone. No matter how the client schedules these things, all of them have to be finished before the delegation is returned, including the close for the open that produced the delegation (when that applies) ⏰ These actions can happen in earlier requests or in earlier operations within the same COMPOUND request.
10.4.5. OPEN Delegation Race with CB_RECALL 🏁
The server tells the client about a recall with a CB_RECALL. One race that can happen is when the delegation gets recalled right away, before the COMPOUND that set up the delegation has made it back to the client 😵💫 A CB_RECALL gives both a stateid and a filehandle the client has no mapping for, so the client can't honor the recall. At that point, the client has two options: it can either not respond, or it can respond with NFS4ERR_BADHANDLE. If it doesn't respond, it runs the risk of the server deciding not to grant it any more delegations 🙈
If it does reply with NFS4ERR_BADHANDLE, then both the client and the server can potentially spot that a race is going on 🕵️♀️ The client can keep a list of pending delegations. When it gets a CB_RECALL for a delegation it doesn't know about, it can cache the stateid and filehandle on a list of pending recalls. When a delegation is handed over, the client would only use it if it isn't on the pending recall list. On the next CB_RECALL, it could return the delegation right away 🚀
The server, for its part, can keep track of when it issues a delegation, and assume that if a client answers a CB_RECALL with NFS4ERR_BADHANDLE, the client hasn't received the delegation yet. The server SHOULD give the client a reasonable amount of time both to receive the delegation and to return it before it revokes the delegation 🕰️ Unlike a failed callback path, the server should periodically probe the client with CB_RECALL to check whether it has received the delegation and is ready to return it.
Once the server decides enough time has gone by, it SHOULD revoke the delegation, and it SHOULD NOT revoke the lease 🙌 While this extended recall is happening, the server SHOULD be renewing the client lease. The whole point is that the client shouldn't have to carry too heavy a burden for a problem the server caused 🫶
10.4.6. Clients That Fail to Honor Delegation Recalls 🙅♀️
A client might not respond to a recall for a bunch of reasons, like the callback path from the server to the client failing. The client might not even know its callback path is broken 😬 That lack of awareness could mean the client only finds out long after the failure that its delegation was revoked, and that another client has changed the data the first client had a delegation for. This is an especially big deal for a client holding an OPEN_DELEGATE_WRITE delegation 💀
The server has a dilemma too, because the client that isn't answering the recall might still be sending other NFS requests, including ones that renew the lease before it expires. If the server doesn't return an error for those lease-renewing operations, it leads the client to believe the delegation is still in force 🤔
Here's how these rules solve that problem 💅
- When the callback path is down, the server MUST NOT revoke the delegation if one of the following occurs:
- The client has issued a
RENEWoperation, and the server has returned anNFS4ERR_CB_PATH_DOWNerror. The server MUST renew the lease for any byte-range locks and share reservations the client has that the server has known about (as opposed to locks and share reservations the client has set up but not yet sent to the server, because of the delegation). The server SHOULD give the client a reasonable amount of time to return its delegations to the server before revoking the client's delegations. - The client has not issued a
RENEWoperation for some period of time after the server tried to recall the delegation. This period of time MUST NOT be less than the value of thelease_timeattribute ⏳
- The client has issued a
- When the client holds a delegation, it can't rely on operations (other than
RENEW) that take a stateid to renew delegation leases across callback path failures. A client that wants to keep its delegations in force across callback path failures has to useRENEWto do it 🎀
10.4.7. Delegation Revocation 💔
When a delegation gets revoked, the applications holding any associated opens on the client need to be notified 📣 Usually that notification shows up as errors returned from READ/WRITE operations, or when a close is attempted on the open file.
If no opens exist for the file when the delegation is revoked, notifying anyone about the revocation isn't necessary. But if there's modified data for the file sitting at the client, the user should be notified. That can be hard, though, since active applications might not be running at the client at all 😭 See Section 10.5.1 for additional details.
10.5. Data Caching and Revocation 🧊
When locks and delegations are revoked, the assumptions that made caching safe are no longer guaranteed 🫠 Any locks or share reservations that have been revoked need their corresponding owner to be notified. That includes applications that have a file open with a delegation that got revoked.
Cached data tied to the revocation has to be removed from the client. If modified data is sitting in the client's cache, that data has to be thrown out of the client without being written to the server 🗑️ Once a lock or delegation is revoked, the assumptions the client was making don't hold anymore. For instance, another client might have been granted a conflicting lock after the first client's lock was revoked. So the data inside the lock range may have been changed by that other client 👀 The first client just can't guarantee to the application what happened to the file after the revocation. Lowkey tragic, honestly 😩
For a lock-owner, notification in lots of cases will just mean returning an error on the next READ or WRITE to the open file, and on all the ones after that, or on the close. But if the client's methods make notification impossible, because errors for certain operations can't be returned, then more drastic steps like signals or process termination might be appropriate 💥 The reasoning is that an invariant the application depends on may have been broken. Depending on how errors are normally handled in the client's operating environment, extra levels of notification (logging, console messages, GUI pop-ups) may also be appropriate 🌈
10.5.1. Revocation Recovery for Write Open Delegation 🦋
Okay bestie, revocation recovery for an OPEN_DELEGATE_WRITE delegation has a super tricky problem: the client might be holding modified data in its cache while the file is not open 😭. So any client that doesn't flush modified data to the server on each close has to make sure the user actually gets told that the write failed because of the revocation 🚨. Some of these messes need a human to fix them, so notification schemes that alert the right user or administrator may be necessary. Logging and console messages are the classic examples 💅.
If there's modified data on the client, it MUST NOT be flushed normally to the server. 🙅♀️ A client may attempt to save a copy of the file data as it was modified during the delegation, under a different name in the file system namespace, to make recovery easier 🎀. Note that when the client can tell the file hasn't been modified by any other client, or when it has a complete cached copy of the file, that saved copy of the client's view can be seriously valuable for recovery 💖. In other cases, recovery from a copy that's partly the client's cached data and partly the server copy as modified by other clients is going to be anything but straightforward 💀. So clients may skip saving file contents in those situations, or mark the results specially to warn users about possible problems ⚠️.
Saving this modified data in delegation revocation situations may be limited to files of a certain size, or might only happen when there's enough free space in the target file system 🧺. Saving may also be restricted to situations where the client has enough buffering resources to keep the cached copy around until it's properly stored to the target file system 🍓.
10.6. Attribute Caching 🌸
The attributes discussed in this section do not include named attributes. 🙈 Individual named attributes are analogous to files, so caching their data needs to be handled just like data caching for regular files. Similarly, LOOKUP results from an OPENATTR directory are to be cached on the same basis as any other pathnames, and the same goes for directory contents 📂.
Clients may cache file attributes obtained from the server and use them to avoid later GETATTR requests ✨. This cache is write-through caching: any modifications to file attributes are always done by requests to the server. That means the modifications should not be done locally and should not be cached 🚫. The exceptions are modifications to attributes that are intimately connected with data caching. So extending a file by writing data to the local data cache is reflected immediately in the size as seen on the client, without that change being immediately reflected on the server 🦄. Normally these changes aren't propagated directly to the server, but when the modified data is flushed to the server, analogous attribute changes are made there too 🔥. When open delegation is in effect, the modified attributes may be returned to the server in the response to a CB_GETATTR call 🎁.
Here's the tea ☕: because attributes are cached locally, the attribute caches on individual clients won't be coherent. Changes made in one order on the server may be seen in a different order on one client, and in a third order on another client 😵💫.
The typical file system application programming interfaces don't give you a way to atomically modify or check attributes for multiple files at the same time. So these rules keep things under control and make the potential incoherency we just talked about reasonably manageable 💅. They're derived from the practice of previous NFS protocols 🧚.
- All attributes for a given file (per-fsid attributes excepted) are cached as a unit at the client, so no non-serializability can arise within the context of a single file 💜.
- An upper time boundary is maintained on how long a client cache entry can be kept without being refreshed from the server ⏰.
- When operations modify attributes at the server, the updated attribute set is requested as part of the containing RPC. This includes directory operations that update attributes indirectly 🪄. This is done by following the modifying operation with a
GETATTRoperation, then using the results of thatGETATTRto update the client's cached attributes 🔁.
Note that if the full set of attributes to be cached is requested by READDIR, the results can be cached by the client on the same basis as attributes obtained via GETATTR 🍭.
A client may validate its cached version of attributes for a file by only fetching the change and time_access attributes. It then assumes that if the change attribute has the same value it had when the attributes were cached, no attributes other than time_access have changed 🕵️♀️. The time_access attribute is fetched too, because many servers operate in environments where the operation that updates change does not update time_access. For example, POSIX file semantics do not update access time when a file is modified by the write system call 🙃. Therefore, a client that wants a current time_access value should fetch it along with change during attribute cache validation and update its cached time_access 🎀.
The client may maintain a cache of modified attributes for those attributes intimately connected with data of modified regular files: size, time_modify, and change 💖. Other than those three attributes, the client MUST NOT maintain a cache of modified attributes 🙅♀️. Instead, attribute changes are immediately sent to the server 📨.
In some operating environments, the equivalent of time_access is expected to be implicitly updated by each read of the content of the file object 📖. If an NFS client is caching the content of a file object, whether it's a regular file, directory, or symbolic link, the client SHOULD NOT update the time_access attribute on the server with each read that's satisfied from cache. That update could be done via SETATTR, or via a small READ or READDIR request 🙅. The reason? It can defeat the performance benefits of caching content, especially since an explicit SETATTR of time_access may alter the change attribute on the server 😩. If the change attribute changes, clients caching the content will think the content has changed and will re-read unmodified data from the server 🔄. The client isn't encouraged to maintain a modified version of time_access in its cache either. That would mean the client either eventually has to write the access time to the server, with bad performance effects, or never updates the server's time_access at all. The result would be an application that caches access time between a close and open of the same file, seeing the access time oscillate between the past and present 🌀. The time_access attribute always means the time of last access to a file by a READ that was satisfied by the server 🕰️. This way, clients will tend to see only time_access changes that go forward in time ⏩✨.
10.7. Data and Metadata Caching and Memory-Mapped Files ✨
Okay so some operating environments let an app map a file's contents straight into its address space 🗺️. Every time the app touches a memory location for a block that isn't loaded yet, a page fault happens and the file gets read. If the block doesn't exist in the file, the block gets allocated and then set up in the app's address space instead. It's a whole process, bestie 💅
As long as every memory-mapped access causes a page fault, the attributes used to detect access and modification (time_access, time_metadata, time_modify, and change) get updated. But in lots of operating environments, when page faults aren't needed, those attributes don't update on reads or updates done through memory access. That's true whether the file is local or being accessed remotely 🙈 A client or server MAY fail to update attributes of a file that's being accessed via memory-mapped I/O. This has several implications:
- If an application on the server has memory mapped a file that a client is also accessing, the client might not get a consistent value of the
changeattribute to figure out whether its cache is stale. A server that knows the file is memory mapped could always pessimistically return updated values forchange, so the app is forced to always get the freshest data and metadata. But because that hurts performance a lot, this behavior is OPTIONAL. 😭 - If the memory-mapped file isn't being modified on the server and is just being read by an app through the memory-mapped interface, the client won't see an updated
time_accessattribute. Still, in many operating environments, neither will any process running on the server. So NFS clients are at no disadvantage compared to local processes 🫶 - If another client is memory mapping the file and that client is holding an
OPEN_DELEGATE_WRITEdelegation, the same issues from the previous two bullets apply. So when a server does aCB_GETATTRto a file the client has modified in its cache, the response fromCB_GETATTRwon't necessarily be accurate. As discussed earlier, the client's obligation is to report that the file has been modified since the delegation was granted. It is not required to report whether it has been modified again between successiveCB_GETATTRcalls. So the server MUST assume that any file the client has modified in cache has been modified again between successiveCB_GETATTRcalls. Depending on the client's memory management system, this weak obligation might not be possible. A client MAY return stale information inCB_GETATTRwhenever the file is memory mapped. 🥺 - Mixing memory mapping and file locking on the same file is problematic 🌀 Check out this scenario, where the page size on each client is 8192 bytes:
- Client A memory maps the first page (8192 bytes) of file X.
- Client B memory maps the first page (8192 bytes) of file X.
- Client A write locks the first 4096 bytes.
- Client B write locks the second 4096 bytes.
- Client A, via a
STOREinstruction, modifies part of its locked region. - Simultaneous to client A, client B issues a
STOREon part of its locked region.
Here's the challenge: each client has to resynchronize to get a correct view of the first page. In many operating environments, the virtual memory management systems on each client only know a page is modified, not that a subset of the page (the part covered by each lock region) has been modified. So it isn't possible for each client to do the right thing, which would be to only write the locked portion of the page to the server. For example, if client A simply writes out the page, and then client B writes out the page, client A's data is lost 💀
It gets worse if mandatory locking is enabled on the file 🔥 When clients A and B issue the STORE instructions, the resulting page faults require a byte-range lock on the entire page. Each client then tries to extend its locked range to the entire page, and that results in a deadlock.
Communicating the NFS4ERR_DEADLOCK error to a STORE instruction is difficult at best.
If a client is locking the entire memory-mapped file, there's no problem with advisory or mandatory byte-range locking, at least until the client unlocks a region in the middle of the file. ✨
Given all of the above, the following are permitted:
- Clients and servers MAY deny memory mapping a file they know has byte-range locks on it.
- Clients and servers MAY deny a byte-range lock on a file they know is memory mapped.
- A client MAY deny memory mapping a file that it knows requires mandatory locking for I/O. If mandatory locking is enabled after the file is opened and mapped, the client MAY deny the application further access to its mapped file.
10.8. Name Caching 🗂️
The results of LOOKUP and READDIR operations can be cached so the client doesn't have to pay for repeat LOOKUP operations. Just like with attribute caching, the various client caches can end up inconsistent with each other 😬 To soften those inconsistencies, and given how typical file system APIs work, there's an upper time boundary on how long a client name cache entry can stick around without checking that another client's directory change operation hasn't made it invalid.
When a client isn't changing a directory that has name cache entries, it needs to periodically fetch attributes for that directory to make sure nobody else is modifying it. After confirming nothing has changed, the expiration time for the associated name cache entries can be bumped to the current time plus the name cache staleness bound 🕰️
When a client is changing a directory, it needs to figure out whether other clients have changed it too. It does this by looking at the change attribute as reported before and after the directory operation, which comes back in the associated change_info4 value for that operation. The server can tell the client whether the change_info4 data was provided atomically with respect to the directory operation. If the change values are provided atomically, the client can compare the pre-operation change value with the change value in its name cache. If that comparison shows another client updated the directory, the name cache for the modified directory gets purged from the client. If the comparison shows no modification, the name cache can be updated on the client to reflect the directory operation, and the timeout gets extended. The post-operation change value then needs to be saved as the basis for future change_info4 comparisons. 💖
Like the scenario above shows, name caching requires the client to revalidate name cache data by checking the change attribute of a directory at the point the name cache item was cached. That means the server has to update the change attribute for directories whenever the contents of the corresponding directory are modified 🔍 For a client to use change_info4 information appropriately and correctly, the server must report the pre- and post-operation change attribute values atomically. When the server can't report the before and after values atomically with respect to the directory operation, it must say so in the change_info4 return value. And when the information isn't reported atomically, the client should not assume that other clients haven't changed the directory. 🚩
10.9. Directory Caching 📂
The results of READDIR operations can be used to avoid later READDIR operations. Just like with attribute and name caching, the client caches can end up inconsistent with each other. To soften those inconsistencies, and given how typical file system APIs work, the following rules should be followed:
- Cached
READDIRinformation for a directory that wasn't obtained in a singleREADDIRoperation must always be a consistent snapshot of the directory contents. To check that, aGETATTRis done before the firstREADDIRand after the lastREADDIRthat contributes to the cache. - An upper time boundary is maintained to show how long a directory cache entry counts as valid before the client has to revalidate the cached information ⏳
The revalidation technique is basically the same as the one for name caching 🫧 When the client isn't changing the directory in question, checking the directory's change attribute with GETATTR is enough. The lifetime of the cache entry can be extended at these checkpoints. When a client is modifying the directory, it needs to use the change_info4 data to figure out whether other clients are modifying the directory too. If it's determined that no other client modifications are happening, the client can update its directory cache to reflect its own changes 🥰
Like we saw before, directory caching requires the client to revalidate directory cache data by inspecting the change attribute of a directory at the point the directory was cached. So the server has to update the change attribute for directories whenever the contents of the corresponding directory are modified. For a client to use change_info4 information appropriately and correctly, the server must report the pre- and post-operation change attribute values atomically. When the server can't report the before and after values atomically with respect to the directory operation, it must indicate that in the change_info4 return value. When the information isn't reported atomically, the client should not assume that other clients haven't changed the directory. 👑
11. Minor Versioning 🌈
To deal with the need for an NFS protocol that can evolve as needed, NFSv4 contains the rules and framework that allow for future minor changes, or versioning. 🦋
The base assumption for minor versioning is that any future accepted minor version has to follow the IETF process and be documented in a Standards Track RFC. So each minor version number corresponds to an RFC. Minor version 0 of the NFSv4 protocol is represented by this RFC. The COMPOUND and CB_COMPOUND procedures support encoding the minor version the client is requesting 🧚
Future minor versions will extend, rather than replace, the XDR for the preceding minor version, just like what happened when NFSv2 moved to NFSv3 and when NFSv3 moved to NFSv4.0 ✨
Detailed rules for constructing minor versions will be addressed in documents that define the early minor versions or, even better, in an RFC that sets up a versioning framework for NFSv4 as a whole 💜
12. Internationalization 🌸
12.1. Introduction 💅
Internationalization is a whole topic with its own vocab (see [RFC6365]) 🌈, and it gets extra messy in NFSv4.0 because of the tangled history and state of NFS implementations 💀. So this section describes what we might call "NFSv4.0 internationalization" (i.e., internationalization as implemented by existing clients and servers) 🎀. It's the base that NFSv4.0 clients can build their own internationalization support on ✨.
This section is based on how existing implementations actually behave 🧚. Each behavior shown is a combo of an NFSv4 server implementation proper and a server-side physical file system 🦋. Servers and physical file systems are often configurable in how they behave, so each configuration that acts differently gets its own separate look 💖.
Quick note: in this section, the key words "MUST", "SHOULD", and "MAY" keep their normal meanings 🫶. BUT, since we're deriving this spec from what implementations do, we're also documenting how the normative terms map back to existing behavior, in the situations where that behavior can be figured out 🔥.
- Behavior implemented by all existing clients or servers is described using "MUST", since new implementations need to follow existing ones to be assured of interoperability 👑. While different behavior might be workable, we haven't found any case where that seems reasonable 🙈.
The converse holds for "MUST NOT": if a type of behavior poses interoperability problems, it MUST NOT be implemented by any existing clients or servers 💀.
- Behavior implemented by most existing clients or servers, where that behavior is more desirable than any alternative, is described using "SHOULD", since new implementations need to follow that existing practice unless there are strong reasons to do otherwise ✨.
The converse holds for "SHOULD NOT" 💅.
- Behavior implemented by some, but not all, existing clients or servers is described using "MAY", meaning new implementations get to choose whether they'll behave that way 🌸. So new implementations get the same flexibility existing ones have 🦋.
- Behavior implemented by all existing clients or servers, as far as is known, but where there's still some uncertainty about the details, is described using "should" 🤔. These cases are mostly about error returns 🙃. New implementations should follow existing practice even though these situations generally don't affect interoperability 🍓.
There are also cases where certain server behaviors aren't known to exist, but also can't be reliably ruled out 😭. Partly this is because so much time has passed since [RFC3530] was published, so the people doing the implementing may no longer be involved in or aware of working group activity 🕰️.
For server behavior that's neither known to exist nor known not to exist, we use "SHOULD NOT" and "MUST NOT" like this, and the same goes for "SHOULD" and "MUST" 🌈.
- Sometimes the potential behavior isn't known to exist, but it's of such a nature that, if it were implemented, interoperability problems would be expected and reported 🚨. That gives us cause to conclude it isn't implemented 🔍. For this kind of behavior, we use "MUST NOT" 🙅♀️. Similarly, we use "MUST" for the contrary behavior 💜.
- Other times the potential behavior isn't known to exist, and while it's undesirable, we can't really draw any conclusions about whether it might exist 🤷♀️. In those cases, we use "SHOULD NOT" 🫠. Similarly, we use "SHOULD" for the contrary behavior 💫.
For a "MAY", "SHOULD", or "SHOULD NOT" that applies to servers, clients need to be aware that some servers may or may not take the specified action 👀. So clients need to be prepared for either outcome 🛡️✨.
12.2. Limitations on Internationalization-Related Processing in the NFSv4 Context 🚧
There are a bunch of noteworthy circumstances that limit how universal internationalization-related processing can be for NFSv4 clients and servers 😤:
- The NFSv4 client is part of a huge set of client-side software components whose design and internal interfaces aren't in the IETF's purview 🙃. That limits how much a particular character encoding can be made standard 📏.
- Server-side handling of file component names is usually implemented inside a server-side physical file system 🗄️. How that file system handles character encoding and normalization isn't something the IETF can specify 🙅♀️.
- Typical UNIX implementation patterns mean the NFSv4 client often has no idea what character encoding is being used 🫥. It can even vary between processes on the same client system 🌀.
- Users may need to access files that were stored earlier with non-UTF-8 encodings, or with UTF-8 encodings that don't match any particular normalization form 📂💖.
12.3. Summary of Server Behavior Types 📋
As mentioned in Section 12.6, servers MAY reject component name strings that aren't valid UTF-8 🙅. This leads to a few types of valid server behavior, outlined below 🧁. When these are combined with the valid normalization-related behaviors described in Section 12.4, you get the combined behaviors listed below 🎀.
- Servers that limit file component names to UTF-8 strings exist, with normalization-related handling as described in Section 12.4 💅. These are best described as "UTF-8-only servers" 👑.
- Servers that don't limit file component names to UTF-8 strings are very common, and they're necessary to deal with clients and applications that aren't oriented toward UTF-8 🌸. These servers ignore normalization-related issues, and there's no way for them to implement either normalization or representation-independent lookups 😭. They're best described as "UTF-8-unaware servers" 🫠, since they treat file component names as uninterpreted strings of bytes and have no knowledge of the characters represented 🧠. See Section 12.7 for details 📖.
- A server can allow component names that aren't valid UTF-8 while still being aware of the structure of UTF-8 strings 🦋. Such servers could implement either normalization or representation-independent lookups, but only apply those techniques to valid UTF-8 strings ✨. Servers like this aren't common, but at least one known server can be configured to behave this way 🔧. This behavior SHOULD NOT be used 🚫. The reason: a filename using one character set might, by coincidence, look like a UTF-8 filename, and the results of UTF-8 normalization or representation-independent lookups are unlikely to be correct in all cases for the other character set 😵💫.
12.4. String Encoding 🔤
Strings that might contain characters outside the ASCII range [RFC20] are generally represented in NFSv4 using the UTF-8 encoding [RFC3629] of Unicode [UNICODE] 🌍💜. See [RFC3629] for the precise encoding and decoding rules 📜.
Some details of protocol treatment depend on the type of string 🎀:
- For strings that are component names, the preferred encoding for any non-ASCII characters is the UTF-8 representation of Unicode 💖.
In many cases, clients have no knowledge of the encoding being used, because encoding happens at the user level under the control of a per-process locale specification 🫥. As a result, it may be impossible for the NFSv4 client to enforce the use of UTF-8 😩. Using non-UTF-8 encodings can be problematic, since it may interfere with access to files stored using other forms of name encoding 🚧. Also, normalization-related processing (see Section 12.5) of a string that isn't encoded in UTF-8 could result in inappropriate name modification or aliasing 💀. And if you have a non-UTF-8 encoded name that accidentally conforms to UTF-8 rules, substituting canonically equivalent strings can change that name drastically 😱.
The kinds of modification and aliasing mentioned here can lead to both false negatives and false positives, depending on the strings in question 🙈. That can result in security issues such as elevation of privilege and denial of service (see [RFC6943] for further discussion) 🚨🛡️. This is serious business, bestie 😭.
- For strings based on domain names, non-ASCII characters MUST be represented using the UTF-8 encoding of Unicode, and additional string format restrictions apply 👑. See Section 12.6 for details 📖.
- The contents of symbolic links (of type
linktext4in the XDR) MUST be treated as opaque data by NFSv4 servers 🫶. Although UTF-8 encoding is often used, it doesn't have to be 🤷♀️. In this respect, the contents of symbolic links are like the contents of regular files, since their encoding is not within the scope of this specification 📂✨. - For other sorts of strings, any non-ASCII characters SHOULD be represented using the UTF-8 encoding of Unicode 🌈.
12.5. Normalization 🫧
The client and server operating environments may differ in their policies and operational methods for character normalization (see [UNICODE] for a discussion of normalization forms) 🧩. This difference can even exist between applications on the same client 😵💫. It makes it harder to provide a single normalization policy for the protocol that allows maximal interoperability 🥲. This issue is similar to character case, where a server may or may not support case-insensitive filename matching, and may or may not preserve character case when storing filenames 🔠. The protocol doesn't mandate a particular behavior, but it does allow for a range of useful behaviors 🌸.
The NFSv4 protocol does not mandate the use of a particular normalization form at this time 🙅♀️. A subsequent minor version of the NFSv4 protocol might specify one ✨. So the server and client can expect that they may receive unnormalized characters within protocol requests and responses 🎀. If the operating environment requires normalization, the implementation will need to normalize the various UTF-8 encoded strings within the protocol before presenting the information to an application (at the client) or to the local file system (at the server) 💅.
Server implementations MAY normalize filenames to conform to a particular normalization form before using the resulting string when looking up or creating a file 🪄. Servers MAY also perform normalization-insensitive string comparisons without modifying the names to match a particular normalization form 🦋. Except in cases where component names are excluded from normalization-related handling because they aren't valid UTF-8 strings, a server MUST make the same choice (whether to normalize or not, the target form of normalization, and whether to do normalization-insensitive string comparisons) in the same way for all accesses to a particular file system 👑🔒. Servers SHOULD NOT reject a filename just because it doesn't conform to a particular normalization form 🙅♀️, since that may deny access to clients that use a different normalization form 💔.
12.6. Types with Processing Defined by Other Internet Areas 🌸
okay so there are two kinds of strings in NFSv4 that are based on domain names 🌐, and the rules for them are defined by other Internet standards. That means the processing has to be consistent across all server operating systems and server file systems 💅
These are the two kinds:
- Server names as they appear in the
fs_locationsattribute. ngl, for most purposes the server is the only one sending these to the client 📨. The exception is using thefs_locationsattribute in a VERIFY or NVERIFY operation 🙈 - Principal suffixes that are used to denote sets of users and groups, and that are in the form of domain names 🎀
The general rules for handling all of these domain-related strings are pretty similar, and they don't change based on whether you're the client or the server 🫶. What does change is the consequences of messing up. The server can report errors when it gets invalid strings 😤, but the client will just ignore an invalid string or swap in a default value 🦋.
The string sent SHOULD be in the form of one or more U-labels as defined by [RFC5890] ✨. If that's impractical, it can instead be in the form of one or more LDH labels [RFC5890] or a UTF-8 domain name that has labels that aren't properly formatted U-labels. The receiver needs to be able to accept domain and server names in any of the formats allowed 🧚. The server MUST reject, using the error NFS4ERR_INVAL, a string that isn't valid UTF-8, or that has an ASCII label that isn't a valid LDH label, or that has an XN-label (one that begins with "xn--") where the characters after "xn--" aren't valid output of the Punycode algorithm [RFC3492] 🚫.
When a domain string is part of id@domain or group@domain, there are two possible approaches 👑:
- The server treats the domain string as a series of U-labels. If the domain string is a series of A-labels or Non-Reserved LDH (NR-LDH) labels, it converts them to U-labels using the Punycode algorithm [RFC3492]. If the domain string is a series of other sorts of LDH labels, the server can use the ToUnicode function defined in [RFC3490] to turn it into labels that generally conform to the U-label syntax. If the domain string is a UTF-8 string that has non-U-labels in it, the server can attempt the ToASCII function defined in [RFC3490] and then the ToUnicode function on the string to get labels that generally conform to the U-label syntax. So, heads up 👀: the domain string returned within a user id on a GETATTR may not match the one sent when the user id was set using SETATTR. When that happens, though, the domain will be in a form that generally conforms to the U-label syntax 🌈.
- The server does not attempt to treat the domain string as a series of U-labels. Specifically, it does not map a domain string that isn't a U-label into a U-label using the methods described above. Because of that, the domain string returned on a GETATTR of the user id MUST be the same as the one used when setting the user id by the SETATTR 🔒.
A server SHOULD use the first method. 💖
For VERIFY and NVERIFY, there are extra string processing requirements that apply to verifying the owner and owner_group attributes. See Section 5.9 🧾.
12.7. Errors Related to UTF-8 🚨
Where the client sends an invalid UTF-8 string, the server MAY return an NFS4ERR_INVAL error. This includes cases where inappropriate prefixes are detected, and where the count includes trailing bytes that don't make up a full Universal Multiple-Octet Coded Character Set (UCS) character 🧩.
Requirements for how a server handles component names that aren't valid UTF-8, when the server does not return NFS4ERR_INVAL in response to them, are described in Section 12.8 📖.
Where the string the client supplied isn't rejected with NFS4ERR_INVAL, but still contains characters the server doesn't support as a value for that string (for example, names with slashes, or characters that don't fit into 16 bits when converted from UTF-8 to a Unicode codepoint), the server should return an NFS4ERR_BADCHAR error 🙅♀️.
Where a UTF-8 string is used as a filename, and the file system supports all of the characters in the name but still doesn't allow that particular name to be used, the server should return the error NFS4ERR_BADNAME 🫠. This includes things like file system prohibitions on "." and ".." as filenames for certain operations, and similar constraints 💅.
12.8. Servers That Accept File Component Names That Are Not Valid UTF-8 Strings 🦋
As stated previously, servers MAY accept component names that aren't valid UTF-8 strings, on all of the physical file systems they export or on only some subset of them 🌷. A typical pattern is for a server to use UTF-8-unaware physical file systems that treat component names as uninterpreted strings of bytes, without having any awareness of the character set being used 🤷♀️.
Such servers SHOULD NOT change the stored representation of component names from what they received on the wire 🙅♀️. They SHOULD use an octet-by-octet comparison of component name strings to decide equivalence, rather than any broader notion of string comparison 🔍. The reason is that the server has no idea what character encoding is being used 🌸.
Still, when such a server uses a broader notion of string equivalence than what's recommended in the paragraph above, the following considerations apply 🚦:
- Outside of 7-bit ASCII, string processing that changes string contents is usually specific to a character set, so it's generally unsafe when the character set is unknown. This processing could change the filename in an unexpected way, making the file inaccessible to the application or client that created or renamed it, and to anyone else expecting the original filename 😱. So that kind of processing should not be performed, because doing it is likely to result in incorrect string modification or aliasing 🚫.
- Unicode normalization is especially dangerous, because that processing assumes the string is UTF-8 💀. When that assumption is false because a different character set was used to create the filename, normalization may corrupt the filename with respect to that character set. That could make the file inaccessible to the application that created it and to others expecting the original filename 🥲. So Unicode normalization SHOULD NOT be performed, because it may cause incorrect string modification or aliasing 🙅♀️.
When the recommendations above aren't followed, the resulting string modification and aliasing can lead to both false negatives and false positives, depending on the strings involved. That can turn into security issues like elevation of privilege and denial of service 😭🚨. See [RFC6943] for further discussion 📚.
13. Error Values 💜
NFS error numbers are assigned to failed operations within a COMPOUND or CB_COMPOUND request 🎀. A COMPOUND request contains a bunch of NFS operations, and their results are encoded in sequence in a COMPOUND reply 🧵. The results of successful operations consist of an NFS4_OK status followed by the encoded results of the operation ✅. If an NFS operation fails, an error status gets entered in the reply, and the COMPOUND request is terminated 🛑. Main character energy for the failed operation, honestly 👑.
13.1. Error Definitions ✨
okay bestie, here's the whole error roster 💅 every error gets a row with its number and where to find its write-up. grab a snack, this one's long 🍓
+-----------------------------+--------+-------------------+
| Error | Number | Description |
+-----------------------------+--------+-------------------+
| NFS4_OK | 0 | Section 13.1.3.1 |
| NFS4ERR_ACCESS | 13 | Section 13.1.6.1 |
| NFS4ERR_ADMIN_REVOKED | 10047 | Section 13.1.5.1 |
| NFS4ERR_ATTRNOTSUPP | 10032 | Section 13.1.11.1 |
| NFS4ERR_BADCHAR | 10040 | Section 13.1.7.1 |
| NFS4ERR_BADHANDLE | 10001 | Section 13.1.2.1 |
| NFS4ERR_BADNAME | 10041 | Section 13.1.7.2 |
| NFS4ERR_BADOWNER | 10039 | Section 13.1.11.2 |
| NFS4ERR_BADTYPE | 10007 | Section 13.1.4.1 |
| NFS4ERR_BADXDR | 10036 | Section 13.1.1.1 |
| NFS4ERR_BAD_COOKIE | 10003 | Section 13.1.1.2 |
| NFS4ERR_BAD_RANGE | 10042 | Section 13.1.8.1 |
| NFS4ERR_BAD_SEQID | 10026 | Section 13.1.8.2 |
| NFS4ERR_BAD_STATEID | 10025 | Section 13.1.5.2 |
| NFS4ERR_CB_PATH_DOWN | 10048 | Section 13.1.12.1 |
| NFS4ERR_CLID_INUSE | 10017 | Section 13.1.10.1 |
| NFS4ERR_DEADLOCK | 10045 | Section 13.1.8.3 |
| NFS4ERR_DELAY | 10008 | Section 13.1.1.3 |
| NFS4ERR_DENIED | 10010 | Section 13.1.8.4 |
| NFS4ERR_DQUOT | 69 | Section 13.1.4.2 |
| NFS4ERR_EXIST | 17 | Section 13.1.4.3 |
| NFS4ERR_EXPIRED | 10011 | Section 13.1.5.3 |
| NFS4ERR_FBIG | 27 | Section 13.1.4.4 |
| NFS4ERR_FHEXPIRED | 10014 | Section 13.1.2.2 |
| NFS4ERR_FILE_OPEN | 10046 | Section 13.1.4.5 |
| NFS4ERR_GRACE | 10013 | Section 13.1.9.1 |
| NFS4ERR_INVAL | 22 | Section 13.1.1.4 |
| NFS4ERR_IO | 5 | Section 13.1.4.6 |
| NFS4ERR_ISDIR | 21 | Section 13.1.2.3 |
| NFS4ERR_LEASE_MOVED | 10031 | Section 13.1.5.4 |
| NFS4ERR_LOCKED | 10012 | Section 13.1.8.5 |
| NFS4ERR_LOCKS_HELD | 10037 | Section 13.1.8.6 |
| NFS4ERR_LOCK_NOTSUPP | 10043 | Section 13.1.8.7 |
| NFS4ERR_LOCK_RANGE | 10028 | Section 13.1.8.8 |
| NFS4ERR_MINOR_VERS_MISMATCH | 10021 | Section 13.1.3.2 |
| NFS4ERR_MLINK | 31 | Section 13.1.4.7 |
| NFS4ERR_MOVED | 10019 | Section 13.1.2.4 |
| NFS4ERR_NAMETOOLONG | 63 | Section 13.1.7.3 |
| NFS4ERR_NOENT | 2 | Section 13.1.4.8 |
| NFS4ERR_NOFILEHANDLE | 10020 | Section 13.1.2.5 |
| NFS4ERR_NOSPC | 28 | Section 13.1.4.9 |
| NFS4ERR_NOTDIR | 20 | Section 13.1.2.6 |
| NFS4ERR_NOTEMPTY | 66 | Section 13.1.4.10 |
| NFS4ERR_NOTSUPP | 10004 | Section 13.1.1.5 |
| NFS4ERR_NOT_SAME | 10027 | Section 13.1.11.3 |
| NFS4ERR_NO_GRACE | 10033 | Section 13.1.9.2 |
| NFS4ERR_NXIO | 6 | Section 13.1.4.11 |
| NFS4ERR_OLD_STATEID | 10024 | Section 13.1.5.5 |
| NFS4ERR_OPENMODE | 10038 | Section 13.1.8.9 |
| NFS4ERR_OP_ILLEGAL | 10044 | Section 13.1.3.3 |
| NFS4ERR_PERM | 1 | Section 13.1.6.2 |
| NFS4ERR_RECLAIM_BAD | 10034 | Section 13.1.9.3 |
| NFS4ERR_RECLAIM_CONFLICT | 10035 | Section 13.1.9.4 |
| NFS4ERR_RESOURCE | 10018 | Section 13.1.3.4 |
| NFS4ERR_RESTOREFH | 10030 | Section 13.1.4.12 |
| NFS4ERR_ROFS | 30 | Section 13.1.4.13 |
| NFS4ERR_SAME | 10009 | Section 13.1.11.4 |
| NFS4ERR_SERVERFAULT | 10006 | Section 13.1.1.6 |
| NFS4ERR_SHARE_DENIED | 10015 | Section 13.1.8.10 |
| NFS4ERR_STALE | 70 | Section 13.1.2.7 |
| NFS4ERR_STALE_CLIENTID | 10022 | Section 13.1.10.2 |
| NFS4ERR_STALE_STATEID | 10023 | Section 13.1.5.6 |
| NFS4ERR_SYMLINK | 10029 | Section 13.1.2.8 |
| NFS4ERR_TOOSMALL | 10005 | Section 13.1.1.7 |
| NFS4ERR_WRONGSEC | 10016 | Section 13.1.6.3 |
| NFS4ERR_XDEV | 18 | Section 13.1.4.14 |
+-----------------------------+--------+-------------------+Table 6: Protocol Error Definitions
13.1.1. General Errors 🫶
this section covers the errors that show up across a whole bunch of different situations. the everyone-can-get-these crew 🌈
13.1.1.1. NFS4ERR_BADXDR (Error Code 10036) 💀
so the arguments for this operation don't line up with the XDR definition. that includes the case where the request just ends before all the arguments have been read 😭 and it counts when a fixed enumeration (booleans are included here, bestie) has a value in the input stream that isn't valid for that enum. a replier may pre-parse all the operations in a COMPOUND procedure before running any of them, and if it does, it can return RPC-level XDR errors right then ✨
13.1.1.2. NFS4ERR_BAD_COOKIE (Error Code 10003) 🍪
this one is for operations that hand back a set of information indexed by some quantity, either one the client supplied or a cookie the server sent on an earlier call. if that value can't be used for what it was meant for, this error shows up 🧚
13.1.1.3. NFS4ERR_DELAY (Error Code 10008) ⏳
okay so for a bunch of possible reasons, the replier just couldn't get this operation done in a reasonable amount of time. the client should wait a bit and then send the request again, using a new RPC transaction ID 🦋
here are two examples of what could lead to this:
- a server that supports hierarchical storage gets asked to process a file that had been migrated.
- an operation needs a delegation recall to finish first, and waiting for that recall would make it impossible to process this request in a timely way.
13.1.1.4. NFS4ERR_INVAL (Error Code 22) 🙅
the arguments are not valid for some reason, even though they do match the XDR definition for the request. the shape is fine, the content is not 💅
13.1.1.5. NFS4ERR_NOTSUPP (Error Code 10004) 🚫
the operation isn't supported. this happens either because the operation is an OPTIONAL one and this server doesn't support it, or because the operation MUST NOT be implemented in the current minor version. no drama, just not on the menu 🍓
13.1.1.6. NFS4ERR_SERVERFAULT (Error Code 10006) 🔥
something went wrong on the server that doesn't map to any of the specific legal NFSv4 protocol error values. the client should turn this into an appropriate error on its end. UNIX clients may choose to translate it to EIO 🫠
13.1.1.7. NFS4ERR_TOOSMALL (Error Code 10005) 📏
this one is for operations that return a variable amount of data, where the client has set a limit. if the data can't fit inside the limit the client gave, this error comes back 🎀
13.1.2. Filehandle Errors 🦋
these errors are about when the current filehandle, the saved filehandle, or the filehandle passed to PUTFH (the one that's about to become the current filehandle) is invalid somehow. that includes a filehandle that's valid in general but is the wrong object type for the operation you're trying to run 🙈
when an error description says there's a problem with the current or saved filehandle, just know that filehandles only get checked for that condition if they're implicit arguments of the operation in question. the vibe is "we only check what the operation actually touches" 💜
13.1.2.1. NFS4ERR_BADHANDLE (Error Code 10001) 🚩
this gets generated for a filehandle that's illegal for the current server. the current filehandle failed its internal consistency checks. once a filehandle has been accepted as valid (by PUTFH), no later status change can make it generate this error. so once it's in, it's in 🔒
13.1.2.2. NFS4ERR_FHEXPIRED (Error Code 10014) ⌛
the current or saved filehandle that's an argument to this operation is volatile, and it has expired at the server 🫥
13.1.2.3. NFS4ERR_ISDIR (Error Code 21) 📁
the current or saved filehandle points at a directory, but the current operation doesn't allow a directory to be the target 🙅♀️
13.1.2.4. NFS4ERR_MOVED (Error Code 10019) 🚚
the file system that holds the current filehandle object isn't present at this server. it might have been relocated or migrated to another server, or it might never have been here at all. the client can grab the new location by getting the "fs_locations" attribute for the current filehandle. for more on this, see Section 8 ✨
13.1.2.5. NFS4ERR_NOFILEHANDLE (Error Code 10020) 🤷
the logical current or saved filehandle value is needed by the current operation and it isn't set. this can happen when a COMPOUND operation is malformed, like when there's no PUTFH or PUTROOTFH before an operation that needs the current filehandle set 🧩
13.1.2.6. NFS4ERR_NOTDIR (Error Code 20) 📄
the current (or saved) filehandle points at something that isn't a directory, and the operation needs a directory 🙃
13.1.2.7. NFS4ERR_STALE (Error Code 70) 🥀
the current or saved filehandle value that's the argument to this operation is invalid. the file system object it refers to is gone, or access to it has been revoked. it's giving "it's over, sis" 💀
13.1.2.8. NFS4ERR_SYMLINK (Error Code 10029) 🔗
the current filehandle points at a symbolic link, and the current operation doesn't allow a symbolic link as its target 🌸
13.1.3. Compound Structure Errors 🧱
this section is about errors that have to do with the overall structure of a COMPOUND request (and yes, that includes CB_COMPOUND too), instead of any one particular operation 👑
there are a few basic constraints on which operations can show up in a COMPOUND request, and these errors are how the server tells you when you've broken one 🫶
13.1.3.1. NFS_OK (Error Code 0) ✅
NFS_OK means the operation completed successfully, which is to say every one of the constituent operations finished without an error. the dream 🎉
13.1.3.2. NFS4ERR_MINOR_VERS_MISMATCH (Error Code 10021) 🙅♂️
the minor version you asked for isn't one that the current listener supports. this value comes back in the overall status for the COMPOUND procedure, but it isn't tied to any specific operation, because the results have to report a result count of zero 🔢
13.1.3.3. NFS4ERR_OP_ILLEGAL (Error Code 10044) 🚫
the operation code isn't valid for the current COMPOUND procedure. the opcode in the result stream that goes with this error is the ILLEGAL value, even though the value in the request stream might be something different. if an illegal value shows up and the replier pre-parses all the operations in a COMPOUND procedure before running any of them, an RPC-level XDR error may be returned in this case instead 😵
13.1.3.4. NFS4ERR_RESOURCE (Error Code 10018) 🪫
during processing of the COMPOUND procedure, the server may run out of available resources and be unable to keep going through the operations in it. this error gets returned from the server in those resource exhaustion situations tied to processing the COMPOUND procedure 😩
13.1.4. File System Errors 💾
these errors describe situations that happened in the underlying file system implementation, rather than in the protocol or any NFSv4.x feature. the disk is the problem, not the protocol 🧚
13.1.4.1. NFS4ERR_BADTYPE (Error Code 10007) 💅
okay so this one shows up when someone tries to make an object with a type that's just wrong 🙈 Maybe the type is undefined, maybe the server doesn't support it, or maybe it's a type CREATE isn't meant for, like a regular file or a named attribute. Those get made with OPEN instead, bestie 🧚
13.1.4.2. NFS4ERR_DQUOT (Error Code 69) 🍓
this is the "you've hit your limit, sweetie" error 💀 The resource (quota) hard limit has been exceeded, which means the user's resource limit on the server is maxed out.
13.1.4.3. NFS4ERR_EXIST (Error Code 17) 🎀
when you're creating, renaming, or linking and the target name is already taken by a file system object, you get this one. Name's gone, babe 🙅♀️
13.1.4.4. NFS4ERR_FBIG (Error Code 27) 🐘
the file system object is too big!! The operation would have made it grow past the server's limit 😭
13.1.4.5. NFS4ERR_FILE_OPEN (Error Code 10046) 🦋
the operation isn't allowed because something involved is currently open. Servers may, but are not required to, disallow linking to, removing, or renaming open file system objects, so it's kind of a server's call, ngl 💅
13.1.4.6. NFS4ERR_IO (Error Code 5) ⚡
this means an I/O error happened and the file system couldn't recover from it. Major yikes energy 😭
13.1.4.7. NFS4ERR_MLINK (Error Code 31) 🔗
the request would have pushed the number of hard links on an object past the server's limit. Too many links, not enough chill 🙈
13.1.4.8. NFS4ERR_NOENT (Error Code 2) 👻
no such file or directory, fr fr. The name you gave doesn't point to anything that exists 👻
13.1.4.9. NFS4ERR_NOSPC (Error Code 28) 🫠
no space left on the device. The operation would have pushed the server's file system over its limit, so it's a big nope 🫠
13.1.4.10. NFS4ERR_NOTEMPTY (Error Code 66) 🗑️
you tried to remove a directory that still has stuff in it. Empty it out first, bestie 💅
13.1.4.11. NFS4ERR_NXIO (Error Code 6) 🌫️
this one also means an I/O error. There is no such device or address, so it's giving "where did that even go?" ✨
13.1.4.12. NFS4ERR_RESTOREFH (Error Code 10030) 💾
RESTOREFH was called but there's no saved filehandle (the one SAVEFH sets up) for it to work on. You gotta save it first, babe 🫶
13.1.4.13. NFS4ERR_ROFS (Error Code 30) 🔒
read-only file system. A modifying operation got attempted on a file system that's locked for reading, so no edits allowed, periodt 🔒
13.1.4.14. NFS4ERR_XDEV (Error Code 18) 🚧
this is for operations, like linking, that cross a boundary they're not supposed to cross. The boundary might be:
- Between file systems, where the fsids are different.
- Between different named attribute directories, or between a named attribute directory and an ordinary directory.
- Between regions of a file system that the implementation treats as separate (for example, for space accounting purposes), where cross-connection between those regions isn't allowed.
Basically, no crossing the line, bestie 🚧
13.1.5. State Management Errors 🔐
okay these errors are all about the stateid (or one of the stateids) passed to an operation. It could be invalid, or it could be valid but point to revoked locking state. Depending on the operation, a valid stateid might designate opens, byte-range locks, or file delegations. So many moving parts, bestie 🧚
13.1.5.1. NFS4ERR_ADMIN_REVOKED (Error Code 10047) 👑
a stateid designates locking state of any type that got revoked because of administrative interaction. This might happen while the lease is still valid, or because a delegation got revoked for failing to be returned, again while the lease was valid. Rude 💀
13.1.5.2. NFS4ERR_BAD_STATEID (Error Code 10025) ❌
a stateid generated by the current server instance got used, and it either:
- Doesn't designate any locking state (current or superseded) for a current (state-owner, file) pair.
- Designates locking state that got freed after lease expiration but without any lease cancellation, as may happen when handling "courtesy locks".
Not the vibe 🙅♀️
13.1.5.3. NFS4ERR_EXPIRED (Error Code 10011) ⏰
a stateid or clientid designates locking state of any type that got revoked or released because the client's lease was cancelled. That can happen right when the lease expires, or later on when someone requests a conflicting lock. Time's up, sweetie ⏰
13.1.5.4. NFS4ERR_LEASE_MOVED (Error Code 10031) 🚚
a lease being renewed is tied to a file system that has been migrated to a new server. It moved house 🏠✨
13.1.5.5. NFS4ERR_OLD_STATEID (Error Code 10024) 🕰️
a stateid got sent with a seqid value that isn't the most current one. Outdated, babe 🕰️
13.1.5.6. NFS4ERR_STALE_STATEID (Error Code 10023) 🥀
a stateid generated by an earlier server instance got used. It's from a previous era, iykyk 🥀
13.1.6. Security Errors 🛡️
these are all the permission-related errors in NFSv4. Guard up, bestie 🛡️
13.1.6.1. NFS4ERR_ACCESS (Error Code 13) 🚫
this means permission denied. The caller doesn't have the correct permission for the operation. Compare this with NFS4ERR_PERM (Section 13.1.6.2), which only covers owner or privileged user permission failures.
13.1.6.2. NFS4ERR_PERM (Error Code 1) 🙅♀️
this means the requester is not the owner. The operation didn't go through because the caller is neither a privileged user (root) nor the owner of the target. Not your thing, not your call 💅
13.1.6.3. NFS4ERR_WRONGSEC (Error Code 10016) 🔐
the security mechanism the client is using for the operation doesn't match the server's security policy. The client should switch to a different security mechanism and re-send the operation. SECINFO can help figure out which mechanism to use 🫶
13.1.7. Name Errors 🏷️
names in NFSv4 are UTF-8 strings. If a string has length zero, you get NFS4ERR_INVAL. If it isn't valid UTF-8, you also get NFS4ERR_INVAL, but servers may accommodate file systems with different character formats and not return this error. There are also a bunch of other errors for specific name problems, so let's go 🏷️
13.1.7.1. NFS4ERR_BADCHAR (Error Code 10040) 🔤
a UTF-8 string has a character that the server doesn't support in the context where it's being used. Not that character, bestie 🙅♀️
13.1.7.2. NFS4ERR_BADNAME (Error Code 10041) 🙈
a name string in a request was made of valid UTF-8 characters the server supports, but the name itself isn't valid for the current operation. Example: making a file or directory called ".." on a server whose file system uses that name for links to parent directories. Big no 🙈
This error should not be returned due to a normalization issue in a string. When a file system keeps names in a particular normalization form, it's the server's job to do the right normalization instead of rejecting the name 💖
13.1.7.3. NFS4ERR_NAMETOOLONG (Error Code 63) 📏
this gets returned when the filename in an operation goes over the server's implementation limit. Too long, sweetie 📏
13.1.8. Locking Errors 🔒
this section is all about locking errors, both share reservations and byte-range locking. It doesn't cover errors about the process of reclaiming locks. Those are in the next section, so stay tuned ✨
13.1.8.1. NFS4ERR_BAD_RANGE (Error Code 10042) 📐
the range for a LOCK, LOCKT, or LOCKU operation isn't appropriate for the server's allowable range of offsets. For example, this happens when a server that only supports 32-bit ranges gets a range it can't handle. (See Section 16.10.4.) 📐
13.1.8.2. NFS4ERR_BAD_SEQID (Error Code 10026) 🔢
the sequence number (seqid) in a locking request is neither the next expected number nor the last one processed. Count again, babe 🔢
13.1.8.3. NFS4ERR_DEADLOCK (Error Code 10045) 🌀
the server has figured out that a blocking lock request would cause a file locking deadlock. Everybody's stuck 🌀
13.1.8.4. NFS4ERR_DENIED (Error Code 10010) ⏳
an attempt to lock a file got denied. Since it might be temporary, the client is encouraged to re-send the lock request until the lock gets accepted. See Section 9.4 for a discussion of the re-send. Keep trying, bestie 💪
13.1.8.5. NFS4ERR_LOCKED (Error Code 10012) 🔐
a READ or WRITE got attempted on a file where there's a conflict between the I/O and an existing lock:
- There's a share reservation that doesn't match the I/O being done.
- The range being read or written overlaps an existing mandatory byte-range lock.
Clash alert 🚨
13.1.8.6. NFS4ERR_LOCKS_HELD (Error Code 10037) 🔒
an operation got blocked by locks showing up unexpectedly. Surprise, not a fun one 😭
13.1.8.7. NFS4ERR_LOCK_NOTSUPP (Error Code 10043) 🚫
a locking request needed to upgrade or downgrade a lock range the owner already holds, but the server doesn't support atomic upgrade or downgrade of locks. Not supported, babe 🚫
13.1.8.8. NFS4ERR_LOCK_RANGE (Error Code 10028) 🧩
a lock request is working on a range that partially overlaps a lock the current lock-owner already holds, and it doesn't exactly match a single such lock. The server doesn't support this kind of request and doesn't implement POSIX locking semantics [fcntl]. See Sections 16.10.5, 16.11.5, and 16.12.5 for how this applies to LOCK, LOCKT, and LOCKU, respectively. So much overlap, so little time 🧩
13.1.8.9. NFS4ERR_OPENMODE (Error Code 10038) 📖
the client tried a READ, WRITE, LOCK, or other operation that the passed stateid doesn't allow. For example, writing to a file that was only opened for read. Read-only means read-only, sweetie 📖
13.1.8.10. NFS4ERR_SHARE_DENIED (Error Code 10015) 🚪
an attempt to OPEN a file with a share reservation failed because of a share conflict. Door's closed 🚪
13.1.9. Reclaim Errors 🔁
these errors are about the process of reclaiming locks after a server restart. It's a whole comeback arc 🔁
13.1.9.1. NFS4ERR_GRACE (Error Code 10013) ⏳
the server is in its recovery or grace period, which should at least match the server's lease period. During that time, a locking request other than a reclaim can't be granted. Patience, bestie ⏳
13.1.9.2. NFS4ERR_NO_GRACE (Error Code 10033) 🚫
the server can't guarantee it hasn't granted state to another client that might conflict with this client's state. No further reclaims from this client will succeed. Oof 😭
13.1.9.3. NFS4ERR_RECLAIM_BAD (Error Code 10034) 🤔
the server can't guarantee it hasn't granted state to another client that might conflict with the requested state. But this only applies to the state requested in this call, so further reclaims may still succeed.
Unlike NFS4ERR_RECLAIM_CONFLICT, this can happen between correctly functioning clients and servers. The "edge condition" scenarios described in Section 9.6.3.4 leave only the server knowing whether the client's locks are still valid, and NFS4ERR_RECLAIM_BAD is how the server tells the client they aren't 🫶
13.1.9.4. NFS4ERR_RECLAIM_CONFLICT (Error Code 10035) 💥
the client's reclaim conflicts with a lock another client already holds. Unlike NFS4ERR_RECLAIM_BAD, this can only happen if one of the clients misbehaved. Somebody's been naughty 💥
13.1.10. Client Management Errors 🪪
this section covers errors tied to the requests used to create and manage client IDs. Identity matters, bestie 🪪
13.1.10.1. NFS4ERR_CLID_INUSE (Error Code 10017) 🙋♀️
the SETCLIENTID operation found that a clientid is already in use by another client. Taken, sorry 🙋♀️
13.1.10.2. NFS4ERR_STALE_CLIENTID (Error Code 10022) 👻
a client ID the server doesn't recognize got used in a locking or SETCLIENTID_CONFIRM request. Who even are you? 👻
13.1.11. Attribute Handling Errors 🧾
this section is all about errors specific to attribute handling in NFSv4. Let's get into the details 🧾
13.1.11.1. NFS4ERR_ATTRNOTSUPP (Error Code 10032) 🙅♀️
an attribute that got specified isn't supported by the server. This error MUST NOT be returned by the GETATTR operation. Not on GETATTR, babe, it's a whole rule 🙅♀️
13.1.11.2. NFS4ERR_BADOWNER (Error Code 10039) 💅
okay so this error pops up when an owner or owner_group attribute value, or the who field of an ace inside an ACL attribute value, can't be translated into a local representation 🫶
13.1.11.3. NFS4ERR_NOT_SAME (Error Code 10027) 🦋
this one is returned by the VERIFY operation 💖 and it basically means the attributes it compared were NOT the same as what the client asked for in its request 😭
13.1.11.4. NFS4ERR_SAME (Error Code 10009) 🎀
this is the NVERIFY operation's version 🧚 and it's giving the opposite vibe: it means the attributes it compared WERE the same as what the client sent in its request ✨
13.1.12. Miscellaneous Errors 🌈
13.1.12.1. NFS4ERR_CB_PATH_DOWN (Error Code 10048) 🔥
there's a problem reaching the client through the callback path 💀
13.2. Operations and Their Valid Errors 👑
ngl this section is the table bestie, it lists the valid error returns for each protocol operation 🍓 the error code NFS4_OK (which just means no error) isn't listed in there, but it should be understood that every operation except ILLEGAL is allowed to return it 🌸
okay the error lists are about to get LONG 😭🫶
| LOOKUPP | NFS4ERR_ACCESS, NFS4ERR_BADHANDLE, |
| | NFS4ERR_DELAY, NFS4ERR_FHEXPIRED, |
| | NFS4ERR_IO, NFS4ERR_MOVED, NFS4ERR_NOENT, |
| | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTDIR, |
| | NFS4ERR_RESOURCE, NFS4ERR_SERVERFAULT, |
| | NFS4ERR_STALE, NFS4ERR_SYMLINK, |
| | NFS4ERR_WRONGSEC |
| | |
| NVERIFY | NFS4ERR_ACCESS, NFS4ERR_ATTRNOTSUPP, |
| | NFS4ERR_BADCHAR, NFS4ERR_BADHANDLE, |
| | NFS4ERR_BADXDR, NFS4ERR_DELAY, |
| | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE, |
| | NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_MOVED, |
| | NFS4ERR_NOFILEHANDLE, NFS4ERR_SAME, |
| | NFS4ERR_SERVERFAULT, NFS4ERR_STALE |
| | |
| OPEN | NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED, |
| | NFS4ERR_ATTRNOTSUPP, NFS4ERR_BADCHAR, |
| | NFS4ERR_BADHANDLE, NFS4ERR_BADNAME, |
| | NFS4ERR_BADOWNER, NFS4ERR_BAD_SEQID, |
| | NFS4ERR_BAD_STATEID, NFS4ERR_BADXDR, |
| | NFS4ERR_DELAY, NFS4ERR_DQUOT, |
| | NFS4ERR_EXIST, NFS4ERR_EXPIRED, |
| | NFS4ERR_FBIG, NFS4ERR_FHEXPIRED, |
| | NFS4ERR_GRACE, NFS4ERR_INVAL, NFS4ERR_IO, |
| | NFS4ERR_ISDIR, NFS4ERR_MOVED, |
| | NFS4ERR_NAMETOOLONG, NFS4ERR_NOENT, |
| | NFS4ERR_NOFILEHANDLE, NFS4ERR_NO_GRACE, |
| | NFS4ERR_NOSPC, NFS4ERR_NOTDIR, |okay this table is literally a whole wall of error codes and I'm obsessed 💅 every row is an operation and the errors it's allowed to throw back 🦋
| | NFS4ERR_NOTSUPP, NFS4ERR_OLD_STATEID, |
| | NFS4ERR_PERM, NFS4ERR_RECLAIM_BAD, |
| | NFS4ERR_RECLAIM_CONFLICT, NFS4ERR_RESOURCE, |
| | NFS4ERR_ROFS, NFS4ERR_SERVERFAULT, |
| | NFS4ERR_SHARE_DENIED, NFS4ERR_STALE, |
| | NFS4ERR_STALE_CLIENTID, NFS4ERR_SYMLINK, |
| | NFS4ERR_WRONGSEC |
| | |
| OPENATTR | NFS4ERR_ACCESS, NFS4ERR_BADHANDLE, |
| | NFS4ERR_BADXDR, NFS4ERR_DELAY, |
| | NFS4ERR_DQUOT, NFS4ERR_FHEXPIRED, |
| | NFS4ERR_IO, NFS4ERR_MOVED, NFS4ERR_NOENT, |
| | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOSPC, |
| | NFS4ERR_NOTSUPP, NFS4ERR_RESOURCE, |
| | NFS4ERR_ROFS, NFS4ERR_SERVERFAULT, |
| | NFS4ERR_STALE |
| | |
| OPEN_CONFIRM | NFS4ERR_ADMIN_REVOKED, NFS4ERR_BADHANDLE, |
| | NFS4ERR_BAD_SEQID, NFS4ERR_BAD_STATEID, |
| | NFS4ERR_BADXDR, NFS4ERR_EXPIRED, |
| | NFS4ERR_FHEXPIRED, NFS4ERR_INVAL, |
| | NFS4ERR_ISDIR, NFS4ERR_LEASE_MOVED, |
| | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE, |
| | NFS4ERR_OLD_STATEID, NFS4ERR_RESOURCE, |
| | NFS4ERR_SERVERFAULT, NFS4ERR_STALE, |
| | NFS4ERR_STALE_STATEID |
| | |
| OPEN_DOWNGRADE | NFS4ERR_ADMIN_REVOKED, NFS4ERR_BADHANDLE, |
| | NFS4ERR_BAD_SEQID, NFS4ERR_BAD_STATEID, |
| | NFS4ERR_BADXDR, NFS4ERR_DELAY, |
| | NFS4ERR_EXPIRED, NFS4ERR_FHEXPIRED, |
| | NFS4ERR_INVAL, NFS4ERR_LEASE_MOVED, |
| | NFS4ERR_LOCKS_HELD, NFS4ERR_MOVED, |
| | NFS4ERR_NOFILEHANDLE, NFS4ERR_OLD_STATEID, |
| | NFS4ERR_RESOURCE, NFS4ERR_ROFS, |
| | NFS4ERR_SERVERFAULT, NFS4ERR_STALE, |
| | NFS4ERR_STALE_STATEID |
| | |
| PUTFH | NFS4ERR_BADHANDLE, NFS4ERR_BADXDR, |
| | NFS4ERR_DELAY, NFS4ERR_FHEXPIRED, |
| | NFS4ERR_MOVED, NFS4ERR_SERVERFAULT, |
| | NFS4ERR_STALE, NFS4ERR_WRONGSEC |
| | |
| PUTPUBFH | NFS4ERR_DELAY, NFS4ERR_SERVERFAULT, |
| | NFS4ERR_WRONGSEC |
| | |
| PUTROOTFH | NFS4ERR_DELAY, NFS4ERR_SERVERFAULT, |
| | NFS4ERR_WRONGSEC |
| | |
| READ | NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED, |
| | NFS4ERR_BADHANDLE, NFS4ERR_BAD_STATEID, |
| | NFS4ERR_BADXDR, NFS4ERR_DELAY, |
| | NFS4ERR_EXPIRED, NFS4ERR_FHEXPIRED, |
| | NFS4ERR_GRACE, NFS4ERR_INVAL, NFS4ERR_IO, |
| | NFS4ERR_ISDIR, NFS4ERR_LEASE_MOVED, |
| | NFS4ERR_LOCKED, NFS4ERR_MOVED, |
| | NFS4ERR_NOFILEHANDLE, NFS4ERR_OLD_STATEID, |
| | NFS4ERR_OPENMODE, NFS4ERR_RESOURCE, |
| | NFS4ERR_SERVERFAULT, NFS4ERR_STALE, |
| | NFS4ERR_STALE_STATEID, NFS4ERR_SYMLINK |
| | |
| READDIR | NFS4ERR_ACCESS, NFS4ERR_BAD_COOKIE, |
| | NFS4ERR_BADHANDLE, NFS4ERR_BADXDR, |
| | NFS4ERR_DELAY, NFS4ERR_FHEXPIRED, |
| | NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_MOVED, |
| | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTDIR, |
| | NFS4ERR_NOT_SAME, NFS4ERR_RESOURCE, |
| | NFS4ERR_SERVERFAULT, NFS4ERR_STALE, |
| | NFS4ERR_TOOSMALL |
| | |
| READLINK | NFS4ERR_ACCESS, NFS4ERR_BADHANDLE, |
| | NFS4ERR_DELAY, NFS4ERR_FHEXPIRED, |
| | NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_ISDIR, |
| | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE, |
| | NFS4ERR_NOTSUPP, NFS4ERR_RESOURCE, |
| | NFS4ERR_SERVERFAULT, NFS4ERR_STALE |
| | |
| RELEASE_LOCKOWNER | NFS4ERR_BADXDR, NFS4ERR_EXPIRED, |
| | NFS4ERR_LEASE_MOVED, NFS4ERR_LOCKS_HELD, |
| | NFS4ERR_RESOURCE, NFS4ERR_SERVERFAULT, |
| | NFS4ERR_STALE_CLIENTID |
| | |
| REMOVE | NFS4ERR_ACCESS, NFS4ERR_BADCHAR, |
| | NFS4ERR_BADHANDLE, NFS4ERR_BADNAME, |
| | NFS4ERR_BADXDR, NFS4ERR_DELAY, |
| | NFS4ERR_FHEXPIRED, NFS4ERR_FILE_OPEN, |
| | NFS4ERR_GRACE, NFS4ERR_INVAL, NFS4ERR_IO, |
| | NFS4ERR_MOVED, NFS4ERR_NAMETOOLONG, |
| | NFS4ERR_NOENT, NFS4ERR_NOFILEHANDLE, |
| | NFS4ERR_NOTDIR, NFS4ERR_NOTEMPTY, |
| | NFS4ERR_RESOURCE, NFS4ERR_ROFS, |
| | NFS4ERR_SERVERFAULT, NFS4ERR_STALE |
| | |
| RENAME | NFS4ERR_ACCESS, NFS4ERR_BADCHAR, |
| | NFS4ERR_BADHANDLE, NFS4ERR_BADNAME, |
| | NFS4ERR_BADXDR, NFS4ERR_DELAY, |
| | NFS4ERR_DQUOT, NFS4ERR_EXIST, |
| | NFS4ERR_FHEXPIRED, NFS4ERR_FILE_OPEN, |
| | NFS4ERR_GRACE, NFS4ERR_INVAL, NFS4ERR_IO, |
| | NFS4ERR_MOVED, NFS4ERR_NAMETOOLONG, |
| | NFS4ERR_NOENT, NFS4ERR_NOFILEHANDLE, |
| | NFS4ERR_NOSPC, NFS4ERR_NOTDIR, |
| | NFS4ERR_NOTEMPTY, NFS4ERR_RESOURCE, |
| | NFS4ERR_ROFS, NFS4ERR_SERVERFAULT, |
| | NFS4ERR_STALE, NFS4ERR_WRONGSEC, |
| | NFS4ERR_XDEV |
| | |
| RENEW | NFS4ERR_ACCESS, NFS4ERR_BADXDR, |
| | NFS4ERR_CB_PATH_DOWN, NFS4ERR_EXPIRED, |
| | NFS4ERR_LEASE_MOVED, NFS4ERR_RESOURCE, |
| | NFS4ERR_SERVERFAULT, NFS4ERR_STALE_CLIENTID |
| | |
| RESTOREFH | NFS4ERR_BADHANDLE, NFS4ERR_FHEXPIRED, |
| | NFS4ERR_MOVED, NFS4ERR_RESOURCE, |
| | NFS4ERR_RESTOREFH, NFS4ERR_SERVERFAULT, |
| | NFS4ERR_STALE, NFS4ERR_WRONGSEC |
| | |
| SAVEFH | NFS4ERR_BADHANDLE, NFS4ERR_FHEXPIRED, |
| | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE, |
| | NFS4ERR_RESOURCE, NFS4ERR_SERVERFAULT, |
| | NFS4ERR_STALE |
| | |
| SECINFO | NFS4ERR_ACCESS, NFS4ERR_BADCHAR, |
| | NFS4ERR_BADHANDLE, NFS4ERR_BADNAME, |
| | NFS4ERR_BADXDR, NFS4ERR_DELAY, |
| | NFS4ERR_FHEXPIRED, NFS4ERR_INVAL, |
| | NFS4ERR_MOVED, NFS4ERR_NAMETOOLONG, |
| | NFS4ERR_NOENT, NFS4ERR_NOFILEHANDLE, |
| | NFS4ERR_NOTDIR, NFS4ERR_RESOURCE, |
| | NFS4ERR_SERVERFAULT, NFS4ERR_STALE |
| | |
| SETATTR | NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED, |
| | NFS4ERR_ATTRNOTSUPP, NFS4ERR_BADCHAR, |
| | NFS4ERR_BADHANDLE, NFS4ERR_BADOWNER, |
| | NFS4ERR_BAD_STATEID, NFS4ERR_BADXDR, |
| | NFS4ERR_DELAY, NFS4ERR_DQUOT, |
| | NFS4ERR_EXPIRED, NFS4ERR_FBIG, |
| | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE, |
| | NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_ISDIR, |
| | NFS4ERR_LEASE_MOVED, NFS4ERR_LOCKED, |
| | NFS4ERR_MOVED, NFS4ERR_NOFILEHANDLE, |
| | NFS4ERR_NOSPC, NFS4ERR_OLD_STATEID, |
| | NFS4ERR_OPENMODE, NFS4ERR_PERM, |
| | NFS4ERR_RESOURCE, NFS4ERR_ROFS, |
| | NFS4ERR_SERVERFAULT, NFS4ERR_STALE, |
| | NFS4ERR_STALE_STATEID |
| | |
| SETCLIENTID | NFS4ERR_BADXDR, NFS4ERR_CLID_INUSE, |
| | NFS4ERR_DELAY, NFS4ERR_INVAL, |
| | NFS4ERR_RESOURCE, NFS4ERR_SERVERFAULT |
| | |
| SETCLIENTID_CONFIRM | NFS4ERR_BADXDR, NFS4ERR_CLID_INUSE, |
| | NFS4ERR_DELAY, NFS4ERR_RESOURCE, |
| | NFS4ERR_SERVERFAULT, NFS4ERR_STALE_CLIENTID |
| | |
| VERIFY | NFS4ERR_ACCESS, NFS4ERR_ATTRNOTSUPP, |
| | NFS4ERR_BADCHAR, NFS4ERR_BADHANDLE, |
| | NFS4ERR_BADXDR, NFS4ERR_DELAY, |
| | NFS4ERR_FHEXPIRED, NFS4ERR_GRACE, |
| | NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_MOVED, |
| | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOT_SAME, |
| | NFS4ERR_RESOURCE, NFS4ERR_SERVERFAULT, |
| | NFS4ERR_STALE |
| | | | WRITE | NFS4ERR_ACCESS, NFS4ERR_ADMIN_REVOKED, |
| | NFS4ERR_BADHANDLE, NFS4ERR_BAD_STATEID, |
| | NFS4ERR_BADXDR, NFS4ERR_DELAY, |
| | NFS4ERR_DQUOT, NFS4ERR_EXPIRED, |
| | NFS4ERR_FBIG, NFS4ERR_FHEXPIRED, |
| | NFS4ERR_GRACE, NFS4ERR_INVAL, NFS4ERR_IO, |
| | NFS4ERR_ISDIR, NFS4ERR_LEASE_MOVED, |
| | NFS4ERR_LOCKED, NFS4ERR_MOVED, |
| | NFS4ERR_NOFILEHANDLE, NFS4ERR_NOSPC, |
| | NFS4ERR_NXIO, NFS4ERR_OLD_STATEID, |
| | NFS4ERR_OPENMODE, NFS4ERR_RESOURCE, |
| | NFS4ERR_ROFS, NFS4ERR_SERVERFAULT, |
| | NFS4ERR_STALE, NFS4ERR_STALE_STATEID, |
| | NFS4ERR_SYMLINK |
| | |
+---------------------+---------------------------------------------+Table 7: Valid Error Returns for Each Protocol Operation 📋✨
13.3. Callback Operations and Their Valid Errors 📞🦋
okay so this section is basically the cheat sheet for callback operations 💅 it has a table listing which errors each one can hand back. the big thing to know is that NFS4_OK (aka "everything went fine" 🫶) isn't written in the table, but it's totally returnable by every callback operation... except CB_ILLEGAL. that one's the odd one out, bestie 🙈
+-------------+-----------------------------------------------------+
| Callback | Errors |
| Operation | |
+-------------+-----------------------------------------------------+
| CB_GETATTR | NFS4ERR_BADHANDLE, NFS4ERR_BADXDR, NFS4ERR_DELAY, |
| | NFS4ERR_INVAL, NFS4ERR_SERVERFAULT |
| | |
| CB_ILLEGAL | NFS4ERR_BADXDR, NFS4ERR_OP_ILLEGAL |
| | |
| CB_RECALL | NFS4ERR_BADHANDLE, NFS4ERR_BAD_STATEID, |
| | NFS4ERR_BADXDR, NFS4ERR_DELAY, NFS4ERR_SERVERFAULT |
| | |
+-------------+-----------------------------------------------------+Table 8: Valid Error Returns for Each Protocol Callback Operation 🎀
13.4. Errors and the Operations That Use Them 🌸
okay so this is the big cheat sheet 🧚 every error code on the left, and the operations it can come back with on the right. the table keeps going, so grab a snack 🍓
+--------------------------+----------------------------------------+
| Error | Operations |
+--------------------------+----------------------------------------+
| NFS4ERR_ACCESS | ACCESS, COMMIT, CREATE, GETATTR, LINK, |
| | LOCK, LOCKT, LOCKU, LOOKUP, LOOKUPP, |
| | NVERIFY, OPEN, OPENATTR, READ, |
| | READDIR, READLINK, REMOVE, RENAME, |
| | RENEW, SECINFO, SETATTR, VERIFY, WRITE |
| | |
| NFS4ERR_ADMIN_REVOKED | CLOSE, DELEGRETURN, LOCK, LOCKU, OPEN, |
| | OPEN_CONFIRM, OPEN_DOWNGRADE, READ, |
| | SETATTR, WRITE |
| | |
| NFS4ERR_ATTRNOTSUPP | CREATE, NVERIFY, OPEN, SETATTR, VERIFY |
| | |
| NFS4ERR_BADCHAR | CREATE, LINK, LOOKUP, NVERIFY, OPEN, |
| | REMOVE, RENAME, SECINFO, SETATTR, |
| | VERIFY |
| | |
| NFS4ERR_BADHANDLE | ACCESS, CB_GETATTR, CB_RECALL, CLOSE, |
| | COMMIT, CREATE, GETATTR, GETFH, LINK, |
| | LOCK, LOCKT, LOCKU, LOOKUP, LOOKUPP, |
| | NVERIFY, OPEN, OPENATTR, OPEN_CONFIRM, |
| | OPEN_DOWNGRADE, PUTFH, READ, READDIR, |
| | READLINK, REMOVE, RENAME, RESTOREFH, |
| | SAVEFH, SECINFO, SETATTR, VERIFY, |
| | WRITE |
| | |
| NFS4ERR_BADNAME | CREATE, LINK, LOOKUP, OPEN, REMOVE, |
| | RENAME, SECINFO |
| | |
| NFS4ERR_BADOWNER | CREATE, OPEN, SETATTR |
| | |
| NFS4ERR_BADTYPE | CREATE |
| | |
| NFS4ERR_BADXDR | ACCESS, CB_GETATTR, CB_ILLEGAL, |
| | CB_RECALL, CLOSE, COMMIT, CREATE, |
| | DELEGPURGE, DELEGRETURN, GETATTR, |
| | ILLEGAL, LINK, LOCK, LOCKT, LOCKU, |
| | LOOKUP, NVERIFY, OPEN, OPENATTR, |
| | OPEN_CONFIRM, OPEN_DOWNGRADE, PUTFH, |
| | READ, READDIR, RELEASE_LOCKOWNER, |
| | REMOVE, RENAME, RENEW, SECINFO, |
| | SETATTR, SETCLIENTID, |
| | SETCLIENTID_CONFIRM, VERIFY, WRITE |
| | |
| NFS4ERR_BAD_COOKIE | READDIR |
| | |
| NFS4ERR_BAD_RANGE | LOCK, LOCKT, LOCKU |
| | |
| NFS4ERR_BAD_SEQID | CLOSE, LOCK, LOCKU, OPEN, |
| | OPEN_CONFIRM, OPEN_DOWNGRADE |
| | |
| NFS4ERR_BAD_STATEID | CB_RECALL, CLOSE, DELEGRETURN, LOCK, |
| | LOCKU, OPEN, OPEN_CONFIRM, |
| | OPEN_DOWNGRADE, READ, SETATTR, WRITE |
| | |
| NFS4ERR_CB_PATH_DOWN | RENEW |
| | |
| NFS4ERR_CLID_INUSE | SETCLIENTID, SETCLIENTID_CONFIRM |
| | |
| NFS4ERR_DEADLOCK | LOCK |
| | |
| NFS4ERR_DELAY | ACCESS, CB_GETATTR, CB_RECALL, CLOSE, |
| | COMMIT, CREATE, DELEGPURGE, |
| | DELEGRETURN, GETATTR, LINK, LOCK, |
| | LOCKT, LOCKU, LOOKUP, LOOKUPP, |
| | NVERIFY, OPEN, OPENATTR, |
| | OPEN_DOWNGRADE, PUTFH, PUTPUBFH, |
| | PUTROOTFH, READ, READDIR, READLINK, |
| | REMOVE, RENAME, SECINFO, SETATTR, |
| | SETCLIENTID, SETCLIENTID_CONFIRM, |
| | VERIFY, WRITE |
| | |
| NFS4ERR_DENIED | LOCK, LOCKT |
| | |
| NFS4ERR_DQUOT | CREATE, LINK, OPEN, OPENATTR, RENAME, |
| | SETATTR, WRITE |
| | |
| NFS4ERR_EXIST | CREATE, LINK, OPEN, RENAME |
| | |
| NFS4ERR_EXPIRED | CLOSE, DELEGRETURN, LOCK, LOCKT, |
| | LOCKU, OPEN, OPEN_CONFIRM, |
| | OPEN_DOWNGRADE, READ, |
| | RELEASE_LOCKOWNER, RENEW, SETATTR, |
| | WRITE |
| | |
| NFS4ERR_FBIG | OPEN, SETATTR, WRITE |
| | |
| NFS4ERR_FHEXPIRED | ACCESS, CLOSE, COMMIT, CREATE, |
| | GETATTR, GETFH, LINK, LOCK, LOCKT, |
| | LOCKU, LOOKUP, LOOKUPP, NVERIFY, OPEN, |
| | OPENATTR, OPEN_CONFIRM, |
| | OPEN_DOWNGRADE, PUTFH, READ, READDIR, |
| | READLINK, REMOVE, RENAME, RESTOREFH, |
| | SAVEFH, SECINFO, SETATTR, VERIFY, |
| | WRITE |
| | |
| NFS4ERR_FILE_OPEN | LINK, REMOVE, RENAME |
| | |
| NFS4ERR_GRACE | GETATTR, LOCK, LOCKT, LOCKU, NVERIFY, |
| | OPEN, READ, REMOVE, RENAME, SETATTR, |
| | VERIFY, WRITE |
| | |
| NFS4ERR_INVAL | ACCESS, CB_GETATTR, CLOSE, COMMIT, |
| | CREATE, DELEGRETURN, GETATTR, LINK, |
| | LOCK, LOCKT, LOCKU, LOOKUP, NVERIFY, |
| | OPEN, OPEN_CONFIRM, OPEN_DOWNGRADE, |
| | READ, READDIR, READLINK, REMOVE, |
| | RENAME, SECINFO, SETATTR, SETCLIENTID, |
| | VERIFY, WRITE |
| | |
| NFS4ERR_IO | ACCESS, COMMIT, CREATE, GETATTR, LINK, |
| | LOOKUP, LOOKUPP, NVERIFY, OPEN, |
| | OPENATTR, READ, READDIR, READLINK, |
| | REMOVE, RENAME, SETATTR, VERIFY, WRITE |
| | |
| NFS4ERR_ISDIR | CLOSE, COMMIT, LINK, LOCK, LOCKT, |
| | LOCKU, OPEN, OPEN_CONFIRM, READ, |
| | READLINK, SETATTR, WRITE |
| | |
| NFS4ERR_LEASE_MOVED | CLOSE, DELEGPURGE, DELEGRETURN, LOCK, |
| | LOCKT, LOCKU, OPEN_CONFIRM, |
| | OPEN_DOWNGRADE, READ, |
| | RELEASE_LOCKOWNER, RENEW, SETATTR, |
| | WRITE |
| | |
| NFS4ERR_LOCKED | READ, SETATTR, WRITE |
| | |
| NFS4ERR_LOCKS_HELD | CLOSE, OPEN_DOWNGRADE, |
| | RELEASE_LOCKOWNER |
| | |
| NFS4ERR_LOCK_NOTSUPP | LOCK |
| | |
| NFS4ERR_LOCK_RANGE | LOCK, LOCKT, LOCKU |
| | |
| NFS4ERR_MLINK | LINK |
| | |
| NFS4ERR_MOVED | ACCESS, CLOSE, COMMIT, CREATE, |
| | DELEGRETURN, GETATTR, GETFH, LINK, |
| | LOCK, LOCKT, LOCKU, LOOKUP, LOOKUPP, |
| | NVERIFY, OPEN, OPENATTR, OPEN_CONFIRM, |
| | OPEN_DOWNGRADE, PUTFH, READ, READDIR, |
| | READLINK, REMOVE, RENAME, RESTOREFH, |
| | SAVEFH, SECINFO, SETATTR, VERIFY, |
| | WRITE |
| | |
| NFS4ERR_NAMETOOLONG | CREATE, LINK, LOOKUP, OPEN, REMOVE, |
| | RENAME, SECINFO |
| | |
| NFS4ERR_NOENT | LINK, LOOKUP, LOOKUPP, OPEN, OPENATTR, |
| | REMOVE, RENAME, SECINFO |
| | |
| NFS4ERR_NOFILEHANDLE | ACCESS, CLOSE, COMMIT, CREATE, |
| | DELEGRETURN, GETATTR, GETFH, LINK, |
| | LOCK, LOCKT, LOCKU, LOOKUP, LOOKUPP, |
| | NVERIFY, OPEN, OPENATTR, OPEN_CONFIRM, |
| | OPEN_DOWNGRADE, READ, READDIR, |
| | READLINK, REMOVE, RENAME, SAVEFH, |
| | SECINFO, SETATTR, VERIFY, WRITE |
| | |
| NFS4ERR_NOSPC | CREATE, LINK, OPEN, OPENATTR, RENAME, |
| | SETATTR, WRITE |
| | | | NFS4ERR_NOTDIR | CREATE, LINK, LOOKUP, LOOKUPP, OPEN, |
| | READDIR, REMOVE, RENAME, SECINFO |
| | |
| NFS4ERR_NOTEMPTY | REMOVE, RENAME |
| | |
| NFS4ERR_NOTSUPP | DELEGPURGE, DELEGRETURN, LINK, OPEN, |
| | OPENATTR, READLINK |
| | |
| NFS4ERR_NOT_SAME | READDIR, VERIFY |
| | |
| NFS4ERR_NO_GRACE | LOCK, OPEN |
| | |
| NFS4ERR_NXIO | WRITE |
| | |
| NFS4ERR_OLD_STATEID | CLOSE, DELEGRETURN, LOCK, LOCKU, OPEN, |
| | OPEN_CONFIRM, OPEN_DOWNGRADE, READ, |
| | SETATTR, WRITE |
| | |
| NFS4ERR_OPENMODE | LOCK, READ, SETATTR, WRITE |
| | |
| NFS4ERR_OP_ILLEGAL | CB_ILLEGAL, ILLEGAL |
| | |
| NFS4ERR_PERM | CREATE, OPEN, SETATTR |
| | |
| NFS4ERR_RECLAIM_BAD | LOCK, OPEN |
| | |
| NFS4ERR_RECLAIM_CONFLICT | LOCK, OPEN |
| | |
| NFS4ERR_RESOURCE | ACCESS, CLOSE, COMMIT, CREATE, |
| | DELEGPURGE, DELEGRETURN, GETATTR, |
| | GETFH, LINK, LOCK, LOCKT, LOCKU, |
| | LOOKUP, LOOKUPP, OPEN, OPENATTR, |
| | OPEN_CONFIRM, OPEN_DOWNGRADE, READ, |
| | READDIR, READLINK, RELEASE_LOCKOWNER, |
| | REMOVE, RENAME, RENEW, RESTOREFH, |
| | SAVEFH, SECINFO, SETATTR, SETCLIENTID, |
| | SETCLIENTID_CONFIRM, VERIFY, WRITE |
| | |
| NFS4ERR_RESTOREFH | RESTOREFH |
| | |
| NFS4ERR_ROFS | COMMIT, CREATE, LINK, OPEN, OPENATTR, |
| | OPEN_DOWNGRADE, REMOVE, RENAME, |
| | SETATTR, WRITE |
| | |
| NFS4ERR_SAME | NVERIFY |
| | |
| NFS4ERR_SERVERFAULT | ACCESS, CB_GETATTR, CB_RECALL, CLOSE, |
| | COMMIT, CREATE, DELEGPURGE, |
| | DELEGRETURN, GETATTR, GETFH, LINK, |
| | LOCK, LOCKT, LOCKU, LOOKUP, LOOKUPP, |
| | NVERIFY, OPEN, OPENATTR, OPEN_CONFIRM, |
| | OPEN_DOWNGRADE, PUTFH, PUTPUBFH, |
| | PUTROOTFH, READ, READDIR, READLINK, |
| | RELEASE_LOCKOWNER, REMOVE, RENAME, |
| | RENEW, RESTOREFH, SAVEFH, SECINFO, |
| | SETATTR, SETCLIENTID, |
| | SETCLIENTID_CONFIRM, VERIFY, WRITE |
| | |
| NFS4ERR_SHARE_DENIED | OPEN |
| | |
| NFS4ERR_STALE | ACCESS, CLOSE, COMMIT, CREATE, |
| | DELEGRETURN, GETATTR, GETFH, LINK, |
| | LOCK, LOCKT, LOCKU, LOOKUP, LOOKUPP, |
| | NVERIFY, OPEN, OPENATTR, OPEN_CONFIRM, |
| | OPEN_DOWNGRADE, PUTFH, READ, READDIR, |
| | READLINK, REMOVE, RENAME, RESTOREFH, |
| | SAVEFH, SECINFO, SETATTR, VERIFY, |
| | WRITE |
| | |
| NFS4ERR_STALE_CLIENTID | DELEGPURGE, LOCK, LOCKT, OPEN, |
| | RELEASE_LOCKOWNER, RENEW, |
| | SETCLIENTID_CONFIRM |
| | |
| NFS4ERR_STALE_STATEID | CLOSE, DELEGRETURN, LOCK, LOCKU, |
| | OPEN_CONFIRM, OPEN_DOWNGRADE, READ, |
| | SETATTR, WRITE |
| | |
| NFS4ERR_SYMLINK | COMMIT, LOOKUP, LOOKUPP, OPEN, READ, |
| | WRITE |
| | |
| NFS4ERR_TOOSMALL | READDIR |
| | |
| NFS4ERR_WRONGSEC | LINK, LOOKUP, LOOKUPP, OPEN, PUTFH, |
| | PUTPUBFH, PUTROOTFH, RENAME, RESTOREFH |
| | |
| NFS4ERR_XDEV | LINK, RENAME |
| | |
+--------------------------+----------------------------------------+
Table 9: Errors and the Operations That Use Them14. NFSv4 Requests ✨
Okay bestie, so for the NFSv4 RPC program there are only two traditional RPC procedures: NULL and COMPOUND 🌸. Everything else is a set of operations defined in normal XDR/RPC syntax and semantics, but they all live inside COMPOUND 🎁. That means the client has to bundle one or more NFSv4 operations into a single request. Iconic packaging, ngl.
The NFS4_CALLBACK program is how the server signals the client 📞, and it's built pretty much like the NFSv4 program. CB_NULL and CB_COMPOUND work the same way NULL and COMPOUND do in the NFS program, and CB_COMPOUND wraps up the rest of the NFS4_CALLBACK operations 💅. There's no predefined RPC program number for this one, so the client picks one from the "transient" program range 🦋. The client also hands over the program and port numbers for NFS4_CALLBACK during the SETCLIENTID/SETCLIENTID_CONFIRM sequence 🫶. You can change them with another SETCLIENTID/SETCLIENTID_CONFIRM sequence, and you can even do that within a client incarnation without removing the relevant leased client state. Main character energy 👑.
14.1. COMPOUND Procedure 💖
COMPOUND is giving better performance on high-latency networks 🚀. The client skips the cumulative latency of several RPCs by combining multiple dependent operations into one COMPOUND procedure. It also keeps protocols simple, since the client can mash basic procedures together into a single request customized for its own environment 🧚.
CB_COMPOUND follows the exact same features as COMPOUND, as described above ✨.
The basic structure of the COMPOUND procedure looks like this:
+-----+--------------+--------+-----------+-----------+-----------+--
| tag | minorversion | numops | op + args | op + args | op + args |
+-----+--------------+--------+-----------+-----------+-----------+--and the reply's structure looks like this:
+------------+-----+--------+-----------------------+--
|last status | tag | numres | status + op + results |
+------------+-----+--------+-----------------------+--The numops and numres fields, in the diagrams above, are the counts for the counted array encoding that signals how many arguments or results are encoded in the request and response 🔢. Per the XDR encoding, these counts must match exactly the number of operation arguments or results encoded. No wiggle room 🙈.
14.2. Evaluation of a COMPOUND Request 🎀
The server processes the COMPOUND procedure by evaluating each operation inside it, in order 🔥. Each component operation is a 32-bit operation code followed by an argument whose length depends on the type of operation. The results of each operation get encoded in sequence into a reply buffer, and each one is preceded by the opcode and a status code (normally zero). If an operation returns a non-zero status code, that status gets encoded, evaluation of the COMPOUND sequence halts, and the reply is returned. Plot twist: evaluation stops even for "non-error" conditions like NFS4ERR_SAME 💀.
There are no atomicity requirements for the operations inside a COMPOUND procedure 🌈. They might be evaluated at the same time as other COMPOUND requests the server receives.
A COMPOUND is not a transaction, so recovering from a partially completed one is the client's responsibility 😭. These can happen at any point because of errors like NFS4ERR_RESOURCE and NFS4ERR_DELAY, and those errors can show up in an otherwise valid operation string. Also, a server reboot in the middle of processing a COMPOUND can leave the client with the tough job of figuring out how far processing got. So the client should avoid overly complex COMPOUND procedures in case an operation in the procedure fails 🥺.
Each operation assumes a current filehandle and a saved filehandle, which are part of the execution context of the COMPOUND request 🧭. Operations may set, change, or return the current filehandle. The saved filehandle is used for temporary storage of a filehandle value, and as the operands for the RENAME and LINK operations 🦋.
14.3. Synchronous Modifying Operations
okay so NFSv4 operations that modify the file system are synchronous 💅 when an operation finishes successfully on the server, the client can trust that any data tied to the request is now in stable storage 🫶 the one exception is file data in a WRITE operation that uses the UNSTABLE4 option 🙈
that also means any earlier operations in the same COMPOUND request are reflected in stable storage too ✨ this is what lets the client recover from a COMPOUND request that only partly ran because the server failed 🔥 for example, say a COMPOUND request has operations A and B, and the server can't send a reply to the client. depending on how far the server got, both results could be in stable storage, or just operation A could be. the server must not have just the results of operation B in stable storage 💀
14.4. Operation Values
the operations inside a COMPOUND procedure are identified by operation values 🧚 to avoid clashing with the RPC procedure numbers, operations 0 (zero) and 1 are not defined 🌸 operation 2 is also not defined, but it's reserved for future use with minor versioning 🔮
15. NFSv4 Procedures
15.1. Procedure 0: NULL - No Operation
15.1.1. SYNOPSIS
<null>15.1.2. ARGUMENT
void;15.1.3. RESULT
void;15.1.4. DESCRIPTION
this is the standard NULL procedure 🎀 void argument, void response, and it has no functionality attached to it 💖 because of that, people sometimes use it to measure the overhead of handling a service request ✨ so the server should make sure it doesn't do any unnecessary work while servicing this procedure 🙈
15.2. Procedure 1: COMPOUND - COMPOUND Operations
15.2.1. SYNOPSIS
compoundargs -> compoundres15.2.2. ARGUMENT
union nfs_argop4 switch (nfs_opnum4 argop) {
case <OPCODE>: <argument>;
...
};
struct COMPOUND4args {
utf8str_cs tag;
uint32_t minorversion;
nfs_argop4 argarray<>;
};15.2.3. RESULT
union nfs_resop4 switch (nfs_opnum4 resop) {
case <OPCODE>: <argument>;
...
};
struct COMPOUND4res {
nfsstat4 status;
utf8str_cs tag;
nfs_resop4 resarray<>;
};15.2.4. DESCRIPTION
the COMPOUND procedure lets you bundle one or more NFS operations into a single RPC request 🎀 the main NFS RPC program has two main procedures, NULL and COMPOUND, and every other operation rides along inside COMPOUND as a wrapper 🧁
the server reads each operation in turn ✨ if an operation runs and its status is NFS4_OK, the next one in the COMPOUND procedure gets executed 💅 the server keeps going like this until there's nothing left to run, or one of the operations returns a status other than NFS4_OK 🫶
sometimes the server doesn't have enough resources to run some or all of the operations in the COMPOUND sequence 😭 when that happens, the error NFS4ERR_RESOURCE gets returned for the specific operation where the resources ran out 🔥 this assumes every earlier operation in the sequence was evaluated successfully 👑 and the results of all the evaluated operations must be sent back to the client 💖
the server usually picks between two ways to decode the client's request 🧚 the classic way is a one-pass XDR decode, where the whole COMPOUND gets decoded before any operation in it runs 🌈 if an XDR decoding error shows up here, an RPC XDR decode error gets returned ✨ the second way is a first pass that decodes the basic COMPOUND request, then XDR decodes each operation right before the server gets to run it 💜 in that case the server might hit an XDR decode error while decoding an operation, after earlier operations in the COMPOUND have already run 😬 then the server returns NFS4ERR_BADXDR to signal the decode error 🌸
the COMPOUND arguments include a minorversion field 🦋 its initial and default value is 0 (zero) 🎀 future minor versions will use this field so the client can tell the server which minor version it's asking for 🔮 if the server gets a COMPOUND procedure with a minorversion value it doesn't support, the server MUST return an error of NFS4ERR_MINOR_VERS_MISMATCH and a zero-length resultdata array 🚫
the COMPOUND results have a status field inside them ✨ if the results array isn't zero-length, this status has to match the status of the last operation that ran in the COMPOUND procedure 💅 so if an operation hit an error, the status is that same error value, the one being returned for the operation that failed 💀
note that operations 0 (zero), 1 (one), and 2 (two) aren't defined for the COMPOUND procedure 🙈 the server could get a request with an operation below the first legal one (OP_ACCESS) or above the last legal one (OP_RELEASE_LOCKOWNER) 😳 in that case, the server's response encodes the opcode OP_ILLEGAL instead of the illegal opcode from the request 🧚 the status field in the ILLEGAL return results is set to NFS4ERR_OP_ILLEGAL, and the COMPOUND procedure's return results are NFS4ERR_OP_ILLEGAL too 🔥
the "tag" in the request is up to the implementer 🍓 it can summarize what's in the COMPOUND request, which helps packet sniffers and engineers debugging implementations 🕵️ but the "tag" value in the response SHOULD be the same value that was in the request 💖 this applies to the tag field of the CB_COMPOUND procedure too 🫶
15.2.4.1. Current Filehandle
the current filehandle and the saved filehandle get used all over the protocol ✨ most operations implicitly use the current filehandle as an argument, and lots of them set the current filehandle as part of their results 💅 mixing client-specified sequences of operations with the current and saved filehandle arguments and results gives you a lot of protocol flexibility 🦋 the best or easiest example of current filehandle usage looks like this:
PUTFH fh1 {fh1}
LOOKUP "compA" {fh2}
GETATTR {fh2}
LOOKUP "compB" {fh3}
GETATTR {fh3}
LOOKUP "compC" {fh4}
GETATTR {fh4}
GETFHFigure 1: Filehandle Usage Example
in this example, the PUTFH (Section 16.20) operation explicitly sets the current filehandle value 🎀 meanwhile, each LOOKUP operation's result sets the current filehandle to the resulting file system object 🌸 the client can also drop GETATTR operations in that use the current filehandle as an argument ✨
the PUTROOTFH (Section 16.22) and PUTPUBFH (Section 16.21) operations also set the current filehandle 🧚 so the example above could swap "PUTFH fh1" for PUTROOTFH or PUTPUBFH with no filehandle argument and get the same effect 💜 (this assumes "compA" sits directly below the root of the namespace 👑)
next to the current filehandle, there's also a saved filehandle 🦋 the current filehandle gets set as the result of operations like LOOKUP, but the saved filehandle has to be set directly with the SAVEFH operation 💅 SAVEFH copies the current filehandle value into the saved value 🫶 the saved filehandle value gets used together with the current filehandle value for the LINK and RENAME operations 🔥 the RESTOREFH operation copies the saved filehandle value back into the current filehandle value 🌈 so the saved filehandle can act like a kind of "scratch" space for the client's series of operations 🧁
15.2.5. IMPLEMENTATION
an error of any type can show up after only some of the operations have run, so the client has to be ready to recover from any failure 😭 if an NFS4ERR_RESOURCE error came from a complex or lengthy set of operations, there's a good chance the server would be able to evaluate them successfully if there were fewer of them 💡 so the client is in charge of handling that kind of complexity during recovery 💖
a single compound should not contain multiple operations with different values for the clientid field used in OPEN, LOCK, or RENEW 🙅♀️ it can cause confusion, because operations without clientids can interact with operations that have them 🌀 when only one clientid is used, it's clear which client is being referenced ✨ for a specific example of OPEN and GETATTR interacting, check out Section 16.16.6 🔎
16. NFSv4 Operations
16.1. Operation 3: ACCESS - Check Access Rights
16.1.1. SYNOPSIS
(cfh), accessreq -> supported, accessrights16.1.2. ARGUMENT
const ACCESS4_READ = 0x00000001;
const ACCESS4_LOOKUP = 0x00000002;
const ACCESS4_MODIFY = 0x00000004;
const ACCESS4_EXTEND = 0x00000008;
const ACCESS4_DELETE = 0x00000010;
const ACCESS4_EXECUTE = 0x00000020;
struct ACCESS4args {
/* CURRENT_FH: object */
uint32_t access;
};16.1.3. RESULT
struct ACCESS4resok {
uint32_t supported;
uint32_t access;
};
union ACCESS4res switch (nfsstat4 status) {
case NFS4_OK:
ACCESS4resok resok4;
default:
void;
};16.1.4. DESCRIPTION 💅
okay so ACCESS is basically the client asking "am I allowed to do this??" 🦋 The user being checked is whoever is identified by the credentials in the RPC request, and the thing being checked is the file system object the current filehandle points to. The client packs the access rights it wants checked into a bitmask called access, and the server checks those permissions. If you get back NFS4_OK, the response has two bitmasks: supported, which is the set of rights the server can verify reliably, and access, which is the set of rights this user actually has on the filehandle. On success, the current filehandle keeps its value ✨
Note that the supported field only contains as many values as were sent in the arguments 🙈 So if the client sends an ACCESS operation with only ACCESS4_READ set, and the server supports that value, the server returns only ACCESS4_READ, even if it could have reliably checked other values too.
The results of this operation are advisory, bestie 🫶 Getting NFS4_OK and seeing the right bit set does not mean you'll be allowed to do that thing to the object later. Access rights can be revoked by the server at any time, so it's not a guarantee.
The following access permissions may be requested:
ACCESS4_READ: Read data from file or read a directory.ACCESS4_LOOKUP: Look up a name in a directory (no meaning for non-directory objects).ACCESS4_MODIFY: Rewrite existing file data or modify existing directory entries.ACCESS4_EXTEND: Write new data or add directory entries.ACCESS4_DELETE: Delete an existing directory entry.ACCESS4_EXECUTE: Execute file (no meaning for a directory).
On success, the current filehandle retains its value. 🌸
16.1.5. IMPLEMENTATION 🧚
this part is so important, so listen up 👑 In general, it is not enough for the client to try to figure out access permissions by looking at the uid, gid, and mode fields in the file attributes, or by trying to read the contents of the ACL attribute. Why? The server might do uid or gid mapping, or enforce extra access control restrictions. The server might also be in a totally different ID space than the client 💀 In these cases (and maybe others), the client cannot reliably do an access check using only the current file attributes.
In NFSv2, the only reliable way to find out whether an operation was allowed was to just try it and see if it worked or failed 😭 With the ACCESS operation in NFSv4, the client can ask the server whether one or more classes of operations are permitted. ACCESS exists so clients can check before doing a series of operations that might fail on access 🔥 The OPEN operation is also a point where the server can verify access to the file object, and it's how that info gets back to the client. ACCESS is still useful for directory operations, or when the UNIX API "access" is used on the client.
The information the server returns from an ACCESS call is not permanent. It was correct at the exact moment the server did the checks, but not necessarily after that. The server can revoke access permission at any time, so don't treat it like forever 🦋
The client should use the effective credentials of the user to build the authentication information in the ACCESS request that is used to determine access rights. It is the effective user and group credentials that get used in later READ and WRITE operations. ✨
Many implementations don't directly support the ACCESS4_DELETE permission, and this is where it gets a little chaotic 💅 Operating systems like UNIX will ignore the ACCESS4_DELETE bit if it's set on an access request for a non-directory object. In these systems, delete permission on a file is decided by the access permissions on the directory the file lives in, not by the permissions of the file itself. So the mask returned that lists which access rights can be supported will have the ACCESS4_DELETE value set to 0. This tells the client that the server was unable to check that particular access right. The ACCESS4_DELETE bit in the returned access mask will then be ignored by the client. 🎀
16.2. Operation 4: CLOSE - Close File 🌈
16.2.1. SYNOPSIS ⭐
(cfh), seqid, open_stateid -> open_stateid16.2.2. ARGUMENT 🍓
struct CLOSE4args {
/* CURRENT_FH: object */
seqid4 seqid;
stateid4 open_stateid;
};16.2.3. RESULT 🎀
union CLOSE4res switch (nfsstat4 status) {
case NFS4_OK:
stateid4 open_stateid;
default:
void;
};16.2.4. DESCRIPTION 💖
CLOSE is the operation that releases share reservations for the regular or named attribute file specified by the current filehandle 🔑 The share reservations and other state information released at the server because of this CLOSE are only tied to the stateid you supplied. The sequence id keeps everything in the right order, and state tied to other OPENs is not affected at all 🫶
If byte-range locks are held, the client SHOULD release all locks before issuing a CLOSE. The server MAY free all outstanding locks on CLOSE, but some servers may not support closing a file that still has byte-range locks held. The server MUST return failure if any locks would exist after the CLOSE. 😭
On success, the current filehandle retains its value. ✨
16.2.5. IMPLEMENTATION 👑
Even though CLOSE returns a stateid, this stateid is not useful to the client and should be treated as deprecated 💀 CLOSE "shuts down" the state associated with all OPENs for the file by a single open-owner. As noted above, CLOSE will either release all file locking state or return an error. So the stateid returned by CLOSE is not useful for the operations that follow 🙈
16.3. Operation 5: COMMIT - Commit Cached Data 🦋
16.3.1. SYNOPSIS 🧚
(cfh), offset, count -> verifier16.3.2. ARGUMENT 🌸
struct COMMIT4args {
/* CURRENT_FH: file */
offset4 offset;
count4 count;
};16.3.3. RESULT 💅
struct COMMIT4resok {
verifier4 writeverf;
};
union COMMIT4res switch (nfsstat4 status) {
case NFS4_OK:
COMMIT4resok resok4;
default:
void;
};16.3.4. DESCRIPTION 🍓
COMMIT forces or flushes data to stable storage for the file specified by the current filehandle 💾 The data that gets flushed is whatever was previously written with a WRITE operation that had the stable field set to UNSTABLE4.
The offset says where in the file the flush should begin. An offset value of 0 (zero) means flush data starting at the beginning of the file. The count is how many bytes of data to flush. If count is 0 (zero), a flush from the offset all the way to the end of the file happens 🌊
The server returns a write verifier when COMMIT completes successfully. The client uses the write verifier to figure out whether the server has restarted or rebooted between the initial WRITE(s) and the COMMIT. It does this by comparing the write verifier returned from the initial writes with the one returned by COMMIT. The server must vary the value of the write verifier at each server event or instantiation that may lead to a loss of uncommitted data. This most commonly happens when the server reboots, but other events on the server can cause uncommitted data loss too 😱
On success, the current filehandle retains its value. 🌈
16.3.5. IMPLEMENTATION ⭐
COMMIT works a lot like the POSIX fsync() [fsync] system call, which syncs a file's state with the disk (file data and metadata both get flushed to disk or stable storage) 🧠 COMMIT does the same thing for a client: it flushes any unsynchronized data and metadata on the server to the server's disk or stable storage for the specified file. Just like fsync(), there might be some modified data, or there might be none to sync. The data may already have been synced by the server's normal periodic buffer sync activity. COMMIT should return NFS4_OK, unless there has been an unexpected error. 👑
COMMIT is different from fsync() in one way: the client can flush just a range of the file. This is most likely triggered by a buffer-reclamation scheme on the client, before the file has been completely written 🫶
The server side of COMMIT is pretty simple. If the server gets a full file COMMIT request that starts at offset 0 and count 0, it should do the equivalent of fsync()'ing the file. Otherwise, it should arrange for the cached data in the range given by offset and count to be flushed to stable storage. In both cases, any metadata associated with the file must be flushed to stable storage before returning. It's not an error for there to be nothing to flush on the server. This means the data and metadata that needed flushing have already been flushed, or were lost during the last server failure 💀
The client side is a little more complex 💅 There are two reasons a client might want to commit a buffer to stable storage. The first is that the client wants to reuse a buffer. In that case, the offset and count of the buffer are sent to the server in the COMMIT request. The server then flushes any cached data based on the offset and count, and flushes any metadata associated with the file. Then it returns the status of the flush and the write verifier. The other reason is a full file flush, such as might happen at CLOSE. Here the client gathers all of the buffers for this file that contain uncommitted data, does the COMMIT operation with an offset of 0 and count of 0, and then frees all of those buffers. Any other dirty buffers get sent to the server in the normal fashion 🎀
After a buffer is written by the client with the stable parameter set to UNSTABLE4, the buffer must be considered modified by the client until it has been either flushed via a COMMIT operation, or written via a WRITE operation with the stable parameter set to FILE_SYNC4 or DATA_SYNC4. This is done so the buffer can't get freed and reused before the data can be flushed to stable storage on the server 🔒
When a response comes back from either a WRITE or a COMMIT operation and it contains a write verifier that is different from one previously returned by the server, the client will need to retransmit all of the buffers containing uncommitted cached data to the server. How to do this is up to the implementer. If there's only one buffer of interest, it should probably be sent back in a WRITE request with the appropriate stable parameter. If there's more than one buffer, it might be worth retransmitting all of them in WRITE requests with the stable parameter set to UNSTABLE4, and then retransmitting the COMMIT operation to flush all of the data on the server to stable storage. The timing of these retransmissions is left to the implementer 🌸
The above description applies to page-cache-based systems as well as buffer-cache-based systems. In those systems, the virtual memory system will need to be modified instead of the buffer cache. 🧚
16.4. Operation 6: CREATE - Create a Non-regular File Object 🎀
16.4.1. SYNOPSIS
(cfh), name, type, attrs -> (cfh), cinfo, attrset16.4.2. ARGUMENT
union createtype4 switch (nfs_ftype4 type) {
case NF4LNK:
linktext4 linkdata;
case NF4BLK:
case NF4CHR:
specdata4 devdata;
case NF4SOCK:
case NF4FIFO:
case NF4DIR:
void;
default:
void; /* server should return NFS4ERR_BADTYPE */
};
struct CREATE4args {
/* CURRENT_FH: directory for creation */
createtype4 objtype;
component4 objname;
fattr4 createattrs;
};16.4.3. RESULT
struct CREATE4resok {
change_info4 cinfo;
bitmap4 attrset; /* attributes set */
};
union CREATE4res switch (nfsstat4 status) {
case NFS4_OK:
CREATE4resok resok4;
default:
void;
};16.4.4. DESCRIPTION
okay so CREATE is the op for making a non-regular file object inside a directory, and it gets a name for it too 🦋 Regular files? Those belong to OPEN, so don't mix them up bestie 💅
The objname is the name for the new object, and the objtype decides what kind of object it is: a directory, a symlink, etc. 🌸
If there's already an object with the same name in that directory, the server is going to throw NFS4ERR_EXIST at you 😭
For the directory where the new object got created, the server hands back change_info4 info in cinfo. The atomic field of that struct is how the server tells you whether the before and after change attributes were grabbed atomically with respect to creating the object. Main character energy for transparency 👑
A zero-length objname gets you NFS4ERR_INVAL 💀 The objname also has to pass the usual UTF-8, character support, and name checks. Check out Section 12.7 for the full tea ☕
The current filehandle gets replaced by the one for the new object ✨
The createattrs field is the starting set of attributes for the object. That set can include any writable attribute that's valid for the object type. When the op works, the server sends back an attribute mask showing which attributes it actually managed to set 🧚
Now here's the part I'm obsessed with. If createattrs has neither the owner attribute nor an ACL with an ACE for the owner, AND the server's file system both supports and requires an owner attribute (or owner ACE), then the server MUST derive the owner (or the owner ACE). Usually that comes from the principal in the RPC credentials of the call, but the server's operating environment or file system semantics might dictate other ways to derive it 🫶 Same vibe for the group: if createattrs has neither the group attribute nor a group ACE, and the file system supports and requires that notion, the server MUST derive the group attribute (or the corresponding owner ACE) for the file. This could come from the RPC credentials (like the group principal if the credentials include it, as with AUTH_SYS), from the group identifier tied to the principal in the credentials (POSIX systems have a user database [getpwnam] that maps every user identifier to a group identifier), inherited from the directory the object is created in, or whatever else the server's operating environment or file system semantics dictate. This applies to the OPEN operation too 🌈
Plot twist though 🙈 The client could try to put an owner attribute, group attribute, or ACL in createattrs that the principal in the RPC credentials doesn't have permission to create files for. If that happens, the error is NFS4ERR_PERM. This applies to the OPEN operation too 💜
16.4.5. IMPLEMENTATION
If the client wants to set attribute values after the create, it can add a SETATTR operation to the COMPOUND request so the right attributes get set 🍓
16.5. Operation 7: DELEGPURGE - Purge Delegations Awaiting Recovery 🌸
16.5.1. SYNOPSIS
clientid ->16.5.2. ARGUMENT
struct DELEGPURGE4args {
clientid4 clientid;
};16.5.3. RESULT
struct DELEGPURGE4res {
nfsstat4 status;
};16.5.4. DESCRIPTION
DELEGPURGE purges all of the delegations awaiting recovery for a given client ✨ It's super useful for clients that don't commit delegation information to stable storage. It tells the server that conflicting requests don't need to be held up while it waits around for delegation info to be recovered 🏃♀️💨
This op exists to support clients that record delegation info in stable storage on the client side. In that case, DELEGPURGE should be issued right after doing delegation recovery (using CLAIM_DELEGATE_PREV) on all delegations the client knows about. Doing that tells the server that no more delegations for the client are going to be recovered, so it can free up resources and stop delaying other clients whose requests conflict with the unrecovered delegations. All clients SHOULD use DELEGPURGE as part of recovery once it's known that no further CLAIM_DELEGATE_PREV recovery will happen. That includes clients that don't record delegation info in stable storage, who would then do a DELEGPURGE right after SETCLIENTID_CONFIRM 🫶
The set of delegations known to the server and the set known to the client can be different. Here's why:
- A client may fail after making a request that resulted in a delegation, but before it received the results and committed them to its stable storage 😬
- A client may fail after deleting its indication that a delegation exists, but before the delegation return is fully processed by the server 💀
- If the server and the client restart, the server may have limited persistent recording of delegations to a subset of the ones in existence 🥺
- A client may have only persistently recorded info about a subset of delegations 🙃
The server MAY support DELEGPURGE, but its support or non-support should match that of CLAIM_DELEGATE_PREV:
- A server may support both DELEGPURGE and CLAIM_DELEGATE_PREV.
- A server may support neither DELEGPURGE nor CLAIM_DELEGATE_PREV.
This is how a client starting up can figure out whether the server is ready to support persistent storage of delegation info. From there it can decide if it's allowed to use write-back caching to local persistent storage, relying on CLAIM_DELEGATE_PREV recovery to flush that changed data safely to the server if the client restarts 🦋
16.6. Operation 8: DELEGRETURN - Return Delegation 🦋
16.6.1. SYNOPSIS
(cfh), stateid ->16.6.2. ARGUMENT
struct DELEGRETURN4args {
/* CURRENT_FH: delegated file */
stateid4 deleg_stateid;
};16.6.3. RESULT
struct DELEGRETURN4res {
nfsstat4 status;
};16.6.4. DESCRIPTION
DELEGRETURN returns the delegation represented by the current filehandle and stateid 💖
Delegations can get returned when they're recalled, or voluntarily (i.e., before the server has recalled them). Either way, the client must properly propagate state changed under the context of the delegation to the server before returning the delegation 🫶
16.7. Operation 9: GETATTR - Get Attributes 🔥
16.7.1. SYNOPSIS
(cfh), attrbits -> attrbits, attrvals16.7.2. ARGUMENT
struct GETATTR4args {
/* CURRENT_FH: directory or file */
bitmap4 attr_request;
};16.7.3. RESULT
struct GETATTR4resok {
fattr4 obj_attributes;
};
union GETATTR4res switch (nfsstat4 status) {
case NFS4_OK:
GETATTR4resok resok4;
default:
void;
};16.7.4. DESCRIPTION
GETATTR is how you grab attributes for the file system object sitting at the current filehandle 🧚 The client sets a bit in the bitmap argument for each attribute value it'd like back. The server returns an attribute bitmap showing which values it was able to return, followed by the attribute values, ordered lowest attribute number first 📋
The server MUST return a value for each attribute the client requests, as long as the server supports that attribute. If the server doesn't support an attribute, or can't approximate a useful value, then it MUST NOT return the attribute value and MUST NOT set the attribute bit in the result bitmap. The server MUST return an error if it supports an attribute on the target but can't obtain its value. In that case, no attribute values get returned 😭
File systems that are absent should be treated as having support for a very small set of attributes, as described in Section 8.3.1 -- even if, back when the file system was present, more attributes were supported 👻
All servers MUST support the REQUIRED attributes, as specified in Section 5, for all file systems, except absent file systems 👑
On success, the current filehandle keeps its value ✨
16.7.5. IMPLEMENTATION
Okay this one is a whole drama 🎭 Suppose there's an OPEN_DELEGATE_WRITE delegation held by another client for the file in question, and size and/or change are among the attributes being interrogated. The server has two choices. First, it can grab the actual current values of these attributes from the client holding the delegation by using the CB_GETATTR callback. Second, the server (especially when the delegated client is unresponsive) can recall the delegation in question. The GETATTR MUST NOT proceed until one of the following happens:
- The requested attribute values are returned in the response to
CB_GETATTR. - The OPEN_DELEGATE_WRITE delegation is returned.
- The OPEN_DELEGATE_WRITE delegation is revoked.
Unless one of the above happens super quickly, one or more NFS4ERR_DELAY errors will be returned while a delegation is outstanding 😩
16.8. Operation 10: GETFH - Get Current Filehandle 🍓
16.8.1. SYNOPSIS
(cfh) -> filehandle16.8.2. ARGUMENT
/* CURRENT_FH: */
void;16.8.3. RESULT
struct GETFH4resok {
nfs_fh4 object;
};
union GETFH4res switch (nfsstat4 status) {
case NFS4_OK:
GETFH4resok resok4;
default:
void;
};16.8.4. DESCRIPTION
This op returns the current filehandle value 🌈
On success, the current filehandle keeps its value ✨
16.8.5. IMPLEMENTATION
Ops that change the current filehandle, like LOOKUP or CREATE, don't automatically hand back the new filehandle as a result. So say a client needs to look up a directory entry and get its filehandle. Then it needs this request 🧠
PUTFH (directory filehandle)
LOOKUP (entry name)
GETFH16.9. Operation 11: LINK - Create Link to a File 🔗
16.9.1. SYNOPSIS
(sfh), (cfh), newname -> (cfh), cinfo16.9.2. ARGUMENT
struct LINK4args {
/* SAVED_FH: source object */
/* CURRENT_FH: target directory */
component4 newname;
};16.9.3. RESULT
struct LINK4resok {
change_info4 cinfo;
};
union LINK4res switch (nfsstat4 status) {
case NFS4_OK:
LINK4resok resok4;
default:
void;
};16.9.4. DESCRIPTION ✨
okay so LINK makes an extra newname for the file that the saved filehandle points to (you set that with SAVEFH) 🔗. The new name goes in the directory the current filehandle points to. The existing file and the target directory have to live in the same file system on the server 🗂️. If it works, the current filehandle is still the target directory 💖. If something already exists in the target directory with that same name as newname, the server must return NFS4ERR_EXIST 🙅♀️.
For the target directory, the server gives back change_info4 info in cinfo 📦. The atomic field of that change_info4 struct tells you whether the before and after change attributes were grabbed atomically with respect to the link creation 🧐.
If newname has a length of 0 (zero), or if newname doesn't follow the UTF-8 definition, you get NFS4ERR_INVAL 💀.
16.9.5. IMPLEMENTATION 🔥
bestie, any change to a property of a "hard" linked file shows up in every one of its linked files 🪞. When you make a link to a file, its attributes should show a numlinks value that's one bigger than it was before the LINK operation 🔢.
Now the "same file system" part. It means the fsid fields in the objects' attributes are the same. If they live on different file systems, you get NFS4ERR_XDEV 🚫. Some servers may also throw this error when a file system has an internal partition that LINK would break 🧱.
On some servers, "." and ".." aren't allowed as newname. If you use them, you get NFS4ERR_BADNAME 🙈.
Named attribute directories are a whole thing 👑. When the current filehandle is a named attribute directory and the object you're linking (the saved filehandle) is not a named attribute for the same object, the error NFS4ERR_XDEV MUST be returned. And when the saved filehandle is a named attribute but the current filehandle isn't the right named attribute directory, NFS4ERR_XDEV MUST also be returned 😭.
When the current filehandle is a named attribute directory and the thing you're linking (the saved filehandle) is a named attribute inside that directory, the server MAY return NFS4ERR_NOTSUPP 🤷♀️.
If newname is already linked to the file behind the saved filehandle, the server will return NFS4ERR_EXIST 🙃.
And note: symbolic links are made with the CREATE operation, not LINK 🧚.
16.10. Operation 12: LOCK - Create Lock 🔐
16.10.1. SYNOPSIS 🌸
(cfh) locktype, reclaim, offset, length, locker -> stateid16.10.2. ARGUMENT 🎀
enum nfs_lock_type4 {
READ_LT = 1,
WRITE_LT = 2,
READW_LT = 3, /* blocking read */
WRITEW_LT = 4 /* blocking write */
};
/*
* For LOCK, transition from open_owner to new lock_owner
*/
struct open_to_lock_owner4 {
seqid4 open_seqid;
stateid4 open_stateid;
seqid4 lock_seqid;
lock_owner4 lock_owner;
};
/*
* For LOCK, existing lock_owner continues to request file locks
*/
struct exist_lock_owner4 {
stateid4 lock_stateid;
seqid4 lock_seqid;
};
union locker4 switch (bool new_lock_owner) {
case TRUE:
open_to_lock_owner4 open_owner;
case FALSE:
exist_lock_owner4 lock_owner;
};
/*
* LOCK/LOCKT/LOCKU: Record lock management
*/
struct LOCK4args {
/* CURRENT_FH: file */
nfs_lock_type4 locktype;
bool reclaim;
offset4 offset;
length4 length;
locker4 locker;
};16.10.3. RESULT 💅
struct LOCK4denied {
offset4 offset;
length4 length;
nfs_lock_type4 locktype;
lock_owner4 owner;
};
struct LOCK4resok {
stateid4 lock_stateid;
};
union LOCK4res switch (nfsstat4 status) {
case NFS4_OK:
LOCK4resok resok4;
case NFS4ERR_DENIED:
LOCK4denied denied;
default:
void;
};16.10.4. DESCRIPTION 🦋
LOCK asks for a byte-range lock on the range set by the offset and length parameters 🔒. You also pick the lock type, which has to be one of the nfs_lock_type4s. If this is a reclaim request, the reclaim parameter is TRUE 🙌.
You can lock bytes even if they aren't currently allocated to the file 🫶. If you want to lock from a specific offset all the way to end-of-file (no matter how long the file actually is), set the length field with all bits set to 1 (one) ✨. If the length is zero, or if it isn't all bits set to one, and the length added to the offset goes past the maximum 64-bit unsigned integer value, you get NFS4ERR_INVAL 💀.
32-bit servers are servers that support locking for byte offsets that fit within 32 bits (so, less than or equal to NFS4_UINT32_MAX) 🧮. If the client's range overlaps one or more bytes beyond offset NFS4_UINT32_MAX but doesn't end at offset NFS4_UINT64_MAX, then that 32-bit server MUST return NFS4ERR_BAD_RANGE 🚫.
If the lock gets denied, the owner, offset, and length of the conflicting lock come back to you 👀.
On success, the current filehandle keeps its value 💖.
16.10.5. IMPLEMENTATION 🧠
If the server can't figure out the exact offset and length of the conflicting lock, it should return the same offset and length you sent in the arguments in the denied results. Section 9 has the full scoop on this and the other file locking operations 📖.
LOCK operations go through permission checks and checks against the access type of the file 🛡️. But the exact rights and modes for each kind of lock reflect how the server-exported file system works, and the protocol doesn't spell them out. For example, Windows 2000 lets you take a write lock on a file that's open for READ, but a POSIX-compliant system doesn't 🪟.
Here's the tricky bit 🥴. Say the client asks for a lock on a range the lock-owner already locked (with the same or a different lock type), or a sub-region of that range, or a region that covers multiple locks the lock-owner already got, in whole or in part. If the server doesn't support that kind of locking (so, no POSIX locking semantics), the server returns NFS4ERR_LOCK_RANGE 🙅♀️. In that case the client can return an error, or it can emulate what it needs. To do that, it uses only LOCK for ranges that don't include any bytes already locked by that lock-owner, plus LOCKU of locks that lock-owner holds (with an exactly matching range and type) 🧩. Same deal when the client wants to upgrade (read lock to write lock) or downgrade (write lock to read lock) an existing record lock and the server doesn't support that. Then the server returns NFS4ERR_LOCK_NOTSUPP 😬. These workarounds might not perfectly match the real semantics when other clients send conflicting lock requests 🫠.
When a client holds an OPEN_DELEGATE_WRITE delegation, it's guaranteed that no other clients have opens 💜. So there can't be conflicting LOCK operations from those clients. That means the client can handle locking requests locally, without sending LOCK operations to the server 🏠. But if it does that, it has to be ready to update the lock status on the server by sending the right LOCK and LOCKU operations before it returns the delegation 🎁.
When one or more clients hold OPEN_DELEGATE_READ delegations, any LOCK operation on a server that implements mandatory locking semantics MUST result in the recall of all those delegations 📣. The LOCK operation might not get granted until every one of those delegations is returned or revoked ⏳. Unless that happens super fast, you'll get one or more NFS4ERR_DELAY errors for requests made while the delegation is still out 🥲.
The locker argument says which lock-owner goes with the LOCK request 🪪. The locker4 structure is a switched union that tells you whether the client has already made byte-range locking state for the current open file and lock-owner. There are multiple cases, depending on whether locking state already exists for the current open file and lock-owner, and whether the boolean new_lock_owner is set 🌈. In every case there's a lock_seqid, whether the lock-owner is named explicitly or implicitly. That seqid checks sequencing and replay issues for the lock-owner. If the server doesn't know the given lock-owner, the seqid sets up the starting sequence value for the new lock-owner 🌱.
- In the case where the state already exists and the boolean is false, the only thing in the argument besides lock_seqid is a stateid that stands for the set of locks tied to that open file and lock-owner 🎯.
- In the case where the state already exists and the boolean is true, the server rejects the request with
NFS4ERR_BAD_SEQID🚫. The one exception is a retransmission of an earlier request where the boolean was true. In that case lock_seqid matches the original request, and the response reflects the final case below ✨. - In the case where no byte-range locking state exists yet and the boolean is true, the argument holds an open_to_lock_owner structure. It specifies the stateid of the open file and the lock-owner to use for the lock. Even though the open-owner isn't given explicitly, the open_seqid tied to it gets used to check open-owner sequencing issues. This case is how you go from using the open_stateid to using a lock stateid 🔁.
16.11. Operation 13: LOCKT - Test for Lock 🕵️♀️
16.11.1. SYNOPSIS 🌷
(cfh) locktype, offset, length, owner -> {void, NFS4ERR_DENIED ->
owner}16.11.2. ARGUMENT 🍓
struct LOCKT4args {
/* CURRENT_FH: file */
nfs_lock_type4 locktype;
offset4 offset;
length4 length;
lock_owner4 owner;
};16.11.3. RESULT 🎀
union LOCKT4res switch (nfsstat4 status) {
case NFS4ERR_DENIED:
LOCK4denied denied;
case NFS4_OK:
void;
default:
void;
};16.11.4. DESCRIPTION 💜
LOCKT is the "is anyone blocking me?" check 🔍. It tests the lock you describe in the arguments. If a conflicting lock exists, you get back the owner, offset, length, and type of that conflicting lock. If no lock is held, you just get NFS4_OK, nothing else 🫧. READ_LT and READW_LT are handled the same way, so the conflicting lock test ignores blocking vs. non-blocking. Same goes for WRITE_LT and WRITEW_LT 💅.
The ranges work just like they do for LOCK 📏. NFS4ERR_INVAL and NFS4ERR_BAD_RANGE come back under the same circumstances as they do for LOCK ⚠️.
On success, the current filehandle keeps its value 🦋.
16.11.5. IMPLEMENTATION 💅
okay so if the server can't figure out the exact offset and length of the lock that's causing the clash, it should hand back the same offset and length you sent in the arguments, in the denied results 🫶 Section 9 is the deep dive on file locking if you wanna nerd out 🤓
LOCKT uses a lock_owner4 instead of the stateid4 that LOCK uses to identify the owner ✨ The reason is that the client doesn't have to open the file just to check whether a lock exists, so there might not be a stateid around 🙈
The conflict test SHOULD exclude locks for the current lock-owner 💖 Since those locks never get examined, overlapping ranges might not change what LOCKT returns. If the server does look at locks that match the lock-owner for range checking, NFS4ERR_LOCK_RANGE may come back. And if it returns NFS4_OK, clients may still do a LOCK and get NFS4ERR_LOCK_RANGE on that LOCK request, because the server gets to be flexible 💅
When a client holds an OPEN_DELEGATE_WRITE delegation, it may choose (see Section 16.10.5) to handle LOCK requests locally 🦋 If it does, LOCKT requests get handled locally too 🌸
16.12. Operation 14: LOCKU - Unlock File 🔓
16.12.1. SYNOPSIS
(cfh) type, seqid, stateid, offset, length -> stateid16.12.2. ARGUMENT
struct LOCKU4args {
/* CURRENT_FH: file */
nfs_lock_type4 locktype;
seqid4 seqid;
stateid4 lock_stateid;
offset4 offset;
length4 length;
};16.12.3. RESULT
union LOCKU4res switch (nfsstat4 status) {
case NFS4_OK:
stateid4 lock_stateid;
default:
void;
};16.12.4. DESCRIPTION
LOCKU is the operation that unlocks the byte-range lock you point it at 🌈 The client can set the locktype field to any value that's legal for the nfs_lock_type4 enumerated type, and the server MUST accept any legal value for locktype 🫶 Honestly, whatever legal value you pick has zero effect on whether LOCKU succeeds or fails 💅
The ranges work the same way they do for LOCK, so NFS4ERR_INVAL and NFS4ERR_BAD_RANGE get returned under the same conditions as LOCK ✨
If it succeeds, the current filehandle keeps its value 💜
16.12.5. IMPLEMENTATION
If the area you want to unlock doesn't line up exactly with a lock the lock-owner actually holds, the server may return NFS4ERR_LOCK_RANGE 😭 That covers four situations: (1) the area isn't locked at all, (2) the area is a sub-range of what's locked, (3) it overlaps the locked area without matching it exactly, or (4) the area includes multiple locks held by the lock-owner. POSIX locking [fcntl] semantics allow all of these, so it's not a glitch, it's just how it works 🤷♀️ If you still want these operations supported, a client that gets this error should simulate them: use LOCKU on ranges that match locks it really holds, then maybe follow up with LOCK requests for the sub-ranges that didn't get unlocked 🔥
When a client holds an OPEN_DELEGATE_WRITE delegation, it may choose (see Section 16.10.5) to handle LOCK requests locally 🦋 Same deal here: LOCKU requests get handled locally too 🌸
16.13. Operation 15: LOOKUP - Look Up Filename 🔍
16.13.1. SYNOPSIS
(cfh), component -> (cfh)16.13.2. ARGUMENT
struct LOOKUP4args {
/* CURRENT_FH: directory */
component4 objname;
};16.13.3. RESULT
struct LOOKUP4res {
/* CURRENT_FH: object */
nfsstat4 status;
};16.13.4. DESCRIPTION
LOOKUP is how you find a file system object inside the directory the current filehandle points to 🧚 It checks the component, and if the object exists, the current filehandle gets swapped for the component's filehandle ✨
If the component can't be evaluated, either because it doesn't exist or because the client isn't allowed to evaluate it, you get an error and the current filehandle stays exactly where it was 🙈
If the component has zero length, you get NFS4ERR_INVAL 💀 The component also has to pass the normal UTF-8, character support, and name checks. See Section 12.7 for the full tea ☕
16.13.5. IMPLEMENTATION
If you want a multi-component lookup, you can build a COMPOUND request like this one and grab each filehandle along the way 🎀
PUTFH (directory filehandle)
LOOKUP "pub"
GETFH
LOOKUP "foo"
GETFH
LOOKUP "bar"
GETFHNFSv4 servers are a whole mood here 💅 Unlike earlier NFS versions, they let LOOKUP requests cross mount points on the server. Clients can spot a crossing by comparing the fsid attribute of the directory with the fsid attribute of the directory being looked up. If the fsids differ, the new directory is a server mount point. UNIX clients that notice a crossing need to mount the server's file system, because that's what keeps the file object identity checks UNIX clients rely on working 🫶
Servers that limit NFS access to "shares" or "exported" file systems should set up a pseudo-file system where the exported ones can be stitched together, so clients can browse the server's namespace 🌸 A client's view of a pseudo-file system is limited to paths that lead to exported file systems, so it's giving tidy 🧺
Note: Earlier versions of the protocol gave "." and ".." special meanings. NFSv4 gives those names no special meaning at all 🙅♀️ If you want a parent directory, you have to use the LOOKUPP operator 💜
Also, this operation does not follow symbolic links 🔗 The client is responsible for all filename parsing, including names that get changed by symbolic links it runs into during the lookup 🧠
If the current filehandle is a symbolic link instead of a directory, the error is NFS4ERR_SYMLINK 🙈 For every other non-directory file type, you get NFS4ERR_NOTDIR instead 💅
16.14. Operation 16: LOOKUPP - Look Up Parent Directory ⬆️
16.14.1. SYNOPSIS
(cfh) -> (cfh)16.14.2. ARGUMENT
/* CURRENT_FH: object */
void;16.14.3. RESULT
struct LOOKUPP4res {
/* CURRENT_FH: directory */
nfsstat4 status;
};16.14.4. DESCRIPTION
The current filehandle is assumed to point at a regular directory or a named attribute directory 🗂️ LOOKUPP sets the current filehandle to the filehandle of the parent directory. If there's no parent directory, you get an NFS4ERR_NOENT error, which means the server will return NFS4ERR_NOENT when the current filehandle is at the root or top of the server's file tree 🌳 Not it, bestie 😭
16.14.5. IMPLEMENTATION
Just like LOOKUP, LOOKUPP will also cross mount points 🦋
If the current filehandle isn't a directory or a named attribute directory, you get NFS4ERR_NOTDIR 💀
If the current filehandle is a named attribute directory linked to a file system object through OPENATTR (so, not a subdirectory of a named attribute directory), LOOKUPP SHOULD return the filehandle of that associated file system object 🎀
16.15. Operation 17: NVERIFY - Verify Difference in Attributes 🧾
16.15.1. SYNOPSIS
(cfh), fattr -> -16.15.2. ARGUMENT
struct NVERIFY4args {
/* CURRENT_FH: object */
fattr4 obj_attributes;
};16.15.3. RESULT
struct NVERIFY4res {
nfsstat4 status;
};16.15.4. DESCRIPTION
NVERIFY is used to prefix a sequence of operations that only run if one or more attributes have changed on some file system object 🔥 If all the attributes match, the error NFS4ERR_SAME must be returned 🙃
If it succeeds, the current filehandle keeps its value 💖
16.15.5. IMPLEMENTATION
This operation is a super useful cache validation operator 🧚 If the object's attributes have changed, the operations that follow can grab fresh data for it. For instance, to check whether a file has changed and pull new data if it has:
PUTFH (public)
LOOKUP "foobar"
NVERIFY attrbits attrs
READ 0 32767If you specify a RECOMMENDED attribute in NVERIFY and the server doesn't support that attribute for the file system object, the client gets NFS4ERR_ATTRNOTSUPP 😭
When you specify rdattr_error or any write-only attribute (e.g., time_modify_set), the client gets NFS4ERR_INVAL 💀
16.16. Operation 18: OPEN - Open a Regular File 🚪
16.16.1. SYNOPSIS
(cfh), seqid, share_access, share_deny, owner, openhow, claim ->
(cfh), stateid, cinfo, rflags, attrset, delegation16.16.2. ARGUMENT 💅
okay so this is the OPEN args bestie 🎀 there's a bunch of enums and unions here, but the big picture is that the client asks to open a file, can ask to create it, and can ask for a delegation 🦋
/*
* Various definitions for OPEN
*/
enum createmode4 {
UNCHECKED4 = 0,
GUARDED4 = 1,
EXCLUSIVE4 = 2
};
union createhow4 switch (createmode4 mode) {
case UNCHECKED4:
case GUARDED4:
fattr4 createattrs;
case EXCLUSIVE4:
verifier4 createverf;
};
enum opentype4 {
OPEN4_NOCREATE = 0,
OPEN4_CREATE = 1
};
union openflag4 switch (opentype4 opentype) {
case OPEN4_CREATE:
createhow4 how;
default:
void;
};
/* Next definitions used for OPEN delegation */
enum limit_by4 {
NFS_LIMIT_SIZE = 1,
NFS_LIMIT_BLOCKS = 2
/* others as needed */
};
struct nfs_modified_limit4 {
uint32_t num_blocks;
uint32_t bytes_per_block;
};
union nfs_space_limit4 switch (limit_by4 limitby) {
/* limit specified as file size */
case NFS_LIMIT_SIZE:
uint64_t filesize;
/* limit specified by number of blocks */
case NFS_LIMIT_BLOCKS:
nfs_modified_limit4 mod_blocks;
};
enum open_delegation_type4 {
OPEN_DELEGATE_NONE = 0,
OPEN_DELEGATE_READ = 1,
OPEN_DELEGATE_WRITE = 2
};
enum open_claim_type4 {
CLAIM_NULL = 0,
CLAIM_PREVIOUS = 1,
CLAIM_DELEGATE_CUR = 2,
CLAIM_DELEGATE_PREV = 3
};
struct open_claim_delegate_cur4 {
stateid4 delegate_stateid;
component4 file;
};
union open_claim4 switch (open_claim_type4 claim) {
/*
* No special rights to file.
* Ordinary OPEN of the specified file.
*/
case CLAIM_NULL:
/* CURRENT_FH: directory */
component4 file;
/*
* Right to the file established by an
* open previous to server reboot. File
* identified by filehandle obtained at
* that time rather than by name.
*/
case CLAIM_PREVIOUS:
/* CURRENT_FH: file being reclaimed */
open_delegation_type4 delegate_type;
/*
* Right to file based on a delegation
* granted by the server. File is
* specified by name.
*/
case CLAIM_DELEGATE_CUR:
/* CURRENT_FH: directory */
open_claim_delegate_cur4 delegate_cur_info;
/*
* Right to file based on a delegation
* granted to a previous boot instance
* of the client. File is specified by name.
*/
case CLAIM_DELEGATE_PREV:
/* CURRENT_FH: directory */
component4 file_delegate_prev;
};
/*
* OPEN: Open a file, potentially receiving an open delegation
*/
struct OPEN4args {
seqid4 seqid;
uint32_t share_access;
uint32_t share_deny;
open_owner4 owner;
openflag4 openhow;
open_claim4 claim;
};16.16.3. RESULT 🌸
this is what the server sends back, and it's giving full glow-up 🍓 the delegation stuff lives in here too, and there are two result flags you should know about ✨
struct open_read_delegation4 {
stateid4 stateid; /* Stateid for delegation */
bool recall; /* Pre-recalled flag for
delegations obtained
by reclaim (CLAIM_PREVIOUS) */
nfsace4 permissions; /* Defines users who don't
need an ACCESS call to
open for read */
};
struct open_write_delegation4 {
stateid4 stateid; /* Stateid for delegation */
bool recall; /* Pre-recalled flag for
delegations obtained
by reclaim
(CLAIM_PREVIOUS) */
nfs_space_limit4
space_limit; /* Defines condition that
the client must check to
determine whether the
file needs to be flushed
to the server on close */
nfsace4 permissions; /* Defines users who don't
need an ACCESS call as
part of a delegated
open */
};
union open_delegation4 switch
(open_delegation_type4 delegation_type) {
case OPEN_DELEGATE_NONE:
void;
case OPEN_DELEGATE_READ:
open_read_delegation4 read;
case OPEN_DELEGATE_WRITE:
open_write_delegation4 write;
};
/*
* Result flags
*/
/* Client must confirm open */
const OPEN4_RESULT_CONFIRM = 0x00000002;
/* Type of file locking behavior at the server */
const OPEN4_RESULT_LOCKTYPE_POSIX = 0x00000004;
struct OPEN4resok {
stateid4 stateid; /* Stateid for open */
change_info4 cinfo; /* Directory change info */
uint32_t rflags; /* Result flags */
bitmap4 attrset; /* attribute set for create */
open_delegation4 delegation; /* Info on any open
delegation */
};
union OPEN4res switch (nfsstat4 status) {
case NFS4_OK:
/* CURRENT_FH: opened file */
OPEN4resok resok4;
default:
void;
};16.16.4. Warning to Client Implementers ⚠️
bestie, this one is IMPORTANT 🙈 read it twice, no cap
OPEN is kinda like LOOKUP because it hands the client a filehandle to use 🧚 but unlike LOOKUP, OPEN also creates server state on that filehandle 👀 normally, the only way to let that state go is a CLOSE operation. CLOSE figures out which file to close by looking at the current filehandle, so the client MUST follow every OPEN operation with a GETFH operation in the same COMPOUND procedure. that way the client gets the filehandle it needs so CLOSE can work properly 💜
waiting for the lease on the file to expire is NOT enough, lowkey 💀 the server can keep that state around forever as long as no other client tries to make a conflicting access to the same file. so don't just vibe and wait it out 🫶
16.16.5. DESCRIPTION 🎀
Okay bestie, OPEN is the operation that creates and/or opens a regular file in a directory using the name you give it ✨. If the file doesn't exist on the server and you want it made, the openhow parameter tells the server how to do the creating. You get to pick one of three creation methods: UNCHECKED4, GUARDED4, or EXCLUSIVE4 💅.
If the current filehandle is a named attribute directory, OPEN goes ahead and creates or opens a named attribute file 🧚. Heads up though: exclusive create of a named attribute is not supported 🙈. So if the createmode is EXCLUSIVE4 and you're in a named attribute directory, the server gives you EINVAL 💀.
UNCHECKED4 is the chill option. It means the file gets created if there isn't one with that name, and running into an existing regular file with that name is not an error 🌸. For this one, createattrs sets the starting attributes for the file, and the set can include any writable attribute that's valid for regular files. When an UNCHECKED4 create finds a file already there, the createattrs attributes are not used, except that if a size of zero is specified, the existing file gets truncated ✂️.
GUARDED4 is the more careful sibling. The server checks for a duplicate by name before doing the create. If a duplicate exists, you get NFS4ERR_EXIST as the status. If the object doesn't exist, the request is handled just like UNCHECKED4 🦋. For both UNCHECKED4 and GUARDED4, when the operation works, the server sends back an attribute mask showing which attributes were actually set on the object.
EXCLUSIVE4 means the server follows exclusive creation semantics and uses the verifier to make sure the target is created exclusively 🔒. The server should check for a duplicate by name. If the object doesn't exist, the server creates it and stores the verifier with the object. If the object does exist and the stored verifier matches the verifier the client sent, the server uses the existing object as the newly created one. If the stored verifier doesn't match, you get NFS4ERR_EXIST back. No attributes may be provided in this case, since the server might use an attribute of the target object to store the verifier. If the server does use an attribute to store the exclusive create verifier, it signals which attribute it used by setting the matching bit in the attribute mask returned in the results ✨.
For the target directory, the server returns change_info4 information in cinfo. The atomic field of the change_info4 struct tells you whether the before and after change attributes were obtained atomically with respect to the link creation 🍓.
Once the creation succeeds, the current filehandle gets replaced by the one for the new object 🎯.
OPEN also brings the Windows share reservation vibes 🪟 through the share_access and share_deny fields of the OPEN arguments. The client specifies the share_access and share_deny modes it needs at OPEN. For clients that don't directly support SHAREs (think UNIX 🐧), the expected deny value is DENY_NONE. If an existing share reservation conflicts with the OPEN request, the server returns NFS4ERR_SHARE_DENIED. For a complete SHARE request, the client must provide values for the owner and seqid fields of the OPEN argument. For more tea on share semantics, see Section 9.9.
Now, the claim field is the part where a client that's recovering from a server failure says, "hey, this request is meant to reclaim state I already had" 🔁. The claim field specifies the file to be opened and the state information the client claims to possess. There are four basic claim types that cover the different situations an OPEN can be in 👑:
CLAIM_NULL: This is a brand new OPEN request, and there's no previous state associated with the file for the client 🌱.CLAIM_PREVIOUS: The client is claiming basic OPEN state for a file it held before a server reboot. This is generally used when a server is returning persistent filehandles, because the client might not have the filename to reclaim the OPEN 🫶.CLAIM_DELEGATE_CUR: The client is claiming a delegation for OPEN that the server granted. This is usually done as part of recalling a delegation 📞.CLAIM_DELEGATE_PREV: The client is claiming a delegation that was granted to a previous client instance. This claim type is for use after aSETCLIENTID_CONFIRMand before the matchingDELEGPURGE, in two situations: after a client reboot, and after a lease expiration that resulted in loss of all lock state. The server MAY supportCLAIM_DELEGATE_PREV. If it does support it,SETCLIENTID_CONFIRMMUST NOT remove the client's delegation state, and the server MUST support theDELEGPURGEoperation 🧠.
The following errors apply to the CLAIM_DELEGATE_PREV claim type 🚨:
NFS4ERR_NOTSUPPis returned if the server doesn't support this claim type.NFS4ERR_INVALis returned if the reclaim is done at an inappropriate time, e.g., afterDELEGPURGEhas been done.NFS4ERR_BAD_RECLAIMis returned if the other error conditions don't apply and the server has no record of the delegation whose reclaim is being attempted.
For OPEN requests whose claim type is anything other than CLAIM_PREVIOUS (so, requests that aren't reclaiming opens after a server reboot) and that reach the server during its grace or lease expiration period, the server returns NFS4ERR_GRACE ⏳.
For any OPEN request, the server may return an open delegation 🎁. That lets further opens and closes get handled locally on the client, as described in Section 10.4. Note that delegation is up to the server to decide, so the client should never assume a delegation will or won't be granted in a particular instance. It should always be ready for either case 🙃. The partial exception is the reclaim (CLAIM_PREVIOUS) case, where a delegation type is claimed. Here, a delegation will always be granted, although the server may specify an immediate recall in the delegation structure 🔙.
The rflags returned by a successful OPEN let the server hand back info about how the open file should be handled 🎀.
OPEN4_RESULT_CONFIRM means the client MUST execute an OPEN_CONFIRM operation before it uses the open file 🛑. OPEN4_RESULT_LOCKTYPE_POSIX means the server's file locking behavior supports the complete set of POSIX locking techniques [fcntl]. From that, the client can choose to manage its file locking state so it handles any mismatch in how file locking is managed 🔐.
If the component is of zero length, NFS4ERR_INVAL gets returned 🙅. The component is also subject to the normal UTF-8, character support, and name checks. See Section 12.7 for more on that.
When an OPEN is done and the specified open-owner already has the resulting filehandle open, the result is to "OR" together the new share and deny status with the existing status. Only a single CLOSE needs to be done, even though multiple OPENs were completed. Checking of share reservations for the new OPEN proceeds normally, with no exception for the existing OPEN held by the same owner. The stateid returned has an "other" field that matches the previous open, while the seqid field is incremented to reflect the changed status due to the new open (Section 9.1.4) 💜.
If the underlying file system at the server is only accessible in read-only mode and the OPEN request specified OPEN4_SHARE_ACCESS_WRITE or OPEN4_SHARE_ACCESS_BOTH, the server returns NFS4ERR_ROFS to say it's a read-only file system 📖.
Just like with the CREATE operation, the server MUST derive the owner, owner ACE, group, or group ACE if any of those four attributes are required and supported by the server's file system 🧑💻. For an OPEN with the EXCLUSIVE4 createmode, the server has no choice in the matter, since those OPEN calls don't include the createattrs field. On the flip side, if createattrs is specified and includes an owner or group (or corresponding ACEs) that the principal in the RPC's credentials isn't authorized to create files for, then the server may return NFS4ERR_PERM 🚫.
Last but not least, in the case where an OPEN specifies a size of zero (so, truncation) and the file has named attributes, the named attributes are left as is. They are not removed 🌈.
16.16.6. IMPLEMENTATION ✨
okay bestie, OPEN has this whole EXCLUSIVE4 create thing 🔒 It works like the one in NFSv3 [RFC1813], and it gives you reliable exclusive creation, no cap. When how is EXCLUSIVE4, the client brings a verifier that can reasonably be expected to be unique 🦋 A mix of a client identifier (maybe the client network address) and a unique number the client makes (maybe the RPC transaction identifier) could be a great pick.
If the object doesn't exist yet, the server creates it and stores the verifier in stable storage 💾 If the file system has no way to store arbitrary file attributes, the server may use one or more elements of the object metadata to hold the verifier instead. The verifier must be stored in stable storage so a retransmitted request doesn't fail by mistake 😭 Exclusive create is only used when exclusive semantics are critical to the app, so it doesn't lean on the duplicate request cache alone. That cache lives in volatile memory, it doesn't survive a crash, and it may get flushed during a long network partition, which opens up failure windows 🙈 On UNIX local file systems, the verifier usually lives in the object's metadata (its timestamps). That's why an exclusive object create may not include initial attributes, because the server would have nowhere to put the verifier 💅
If the server can't support these exclusive create semantics, maybe because it can't commit the verifier to stable storage, it should fail the OPEN request with the error NFS4ERR_NOTSUPP.
During an exclusive CREATE, if the object already exists, the server rebuilds the object's verifier and compares it with the one in the request. If they match, the server treats the request as a success 🎀 The request is presumed to be a duplicate of an earlier, successful request whose reply got lost and that the server's duplicate request cache didn't catch. If the verifiers don't match, the request gets rejected with NFS4ERR_EXIST.
Once the client has finished a successful exclusive create, it must issue a SETATTR to set the correct object attributes. Until then, it should not rely on any of the object attributes, since the server may be overloading the metadata to store the verifier 👀 The subsequent SETATTR must not occur in the same COMPOUND request as the OPEN. Keeping them apart guarantees that exclusive create keeps working properly even if the request gets retransmitted 🫶
GUARDED4 doesn't give you "exactly-once" semantics, so be careful 💀 If a reply is lost and the server doesn't notice the retransmission, the operation can fail with NFS4ERR_EXIST even though the create actually worked. A client would use this behavior when the app didn't ask for an exclusive create but did ask for the file to be truncated when it's opened. If the client times out and retransmits the create request, it can use GUARDED4 to stop a sequence like create, write, create (retransmitted) from happening 🧚
For share reservations (see Section 9.9), the client must specify a value for share_access that is one of OPEN4_SHARE_ACCESS_READ, OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH. For share_deny, the client must specify one of OPEN4_SHARE_DENY_NONE, OPEN4_SHARE_DENY_READ, OPEN4_SHARE_DENY_WRITE, or OPEN4_SHARE_DENY_BOTH. If the client fails to do this, the server must return NFS4ERR_INVAL 🙅♀️
Based on the share_access value (OPEN4_SHARE_ACCESS_READ, OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH), the client should check that the requester has the proper access rights for the operation. This usually means applying the ACL access rules to the file for the current requester. But just like with the ACCESS operation, the client should not second-guess the server's decisions, because access rights can change and may be controlled by server administrators outside the ACL framework 🔐 If the requester isn't authorized to READ or WRITE (depending on the share_access value), the server must return NFS4ERR_ACCESS. Also, NFSv4 doesn't require READs and WRITEs on an open file to use the same credentials as the OPEN itself, so the server still must do proper access checking on the READs and WRITEs themselves 👑
If the component given to OPEN resolves to something other than a regular file (or a named attribute), an error gets sent back to the client. If it's a directory, NFS4ERR_ISDIR is returned. Otherwise, NFS4ERR_SYMLINK is returned. NFS4ERR_SYMLINK comes back for both symlinks and special files of other types. NFS4ERR_INVAL would be the wrong call here, since the client's arguments were correct, and the client can't necessarily know when it sends the OPEN that the component will resolve to a non-regular file 🦋
If the current filehandle isn't a directory, the error NFS4ERR_NOTDIR will be returned.
If a COMPOUND contains an OPEN that sets up an OPEN_DELEGATE_WRITE delegation, then later GETATTRs normally cause a CB_GETATTR to be sent to the client holding the delegation. But when the OPEN and GETATTR are part of the same COMPOUND, the server SHOULD understand that both operations are for the same client ID and avoid querying the client, since that client won't be able to respond 🙈 This OPEN and GETATTR pair SHOULD be understood as retrieving the size and change attributes at the time of OPEN. Also, as explained in Section 15.2.5, the client should not build a COMPOUND that mixes operations for different client IDs 💖
16.17. Operation 19: OPENATTR - Open Named Attribute Directory 🌈
16.17.1. SYNOPSIS
(cfh) createdir -> (cfh)16.17.2. ARGUMENT
struct OPENATTR4args {
/* CURRENT_FH: object */
bool createdir;
};16.17.3. RESULT
struct OPENATTR4res {
/* CURRENT_FH: named attr directory */
nfsstat4 status;
};16.17.4. DESCRIPTION
The OPENATTR operation is used to get the filehandle of the named attribute directory linked to the current filehandle 📂 The result of OPENATTR is a filehandle to an object of type NF4ATTRDIR. From that filehandle, READDIR and LOOKUP operations can be used to grab filehandles for the various named attributes attached to the original file system object. Filehandles returned inside the named attribute directory will have a type of NF4NAMEDATTR.
The createdir argument lets the client say whether a named attribute directory should be created as a result of the OPENATTR operation. Some clients may use OPENATTR with a value of FALSE for createdir to check whether any named attributes exist for the object. If none exist, then NFS4ERR_NOENT will be returned. If createdir is TRUE and no named attribute directory exists, one gets created ✨ Creating a named attribute directory assumes the server has implemented named attribute support in this way, and this definition doesn't require the server to do so.
16.17.5. IMPLEMENTATION
If the server doesn't support named attributes for the current filehandle, an error of NFS4ERR_NOTSUPP will be returned to the client 🙃
16.18. Operation 20: OPEN_CONFIRM - Confirm Open 🎀
16.18.1. SYNOPSIS
(cfh), seqid, stateid -> stateid16.18.2. ARGUMENT
struct OPEN_CONFIRM4args {
/* CURRENT_FH: opened file */
stateid4 open_stateid;
seqid4 seqid;
};16.18.3. RESULT
struct OPEN_CONFIRM4resok {
stateid4 open_stateid;
};
union OPEN_CONFIRM4res switch (nfsstat4 status) {
case NFS4_OK:
OPEN_CONFIRM4resok resok4;
default:
void;
};16.18.4. DESCRIPTION
This operation is used to confirm the sequence id usage the first time an open-owner gets used by a client 🔢 The stateid returned from the OPEN operation is the argument for this operation, along with the next sequence id for the open-owner. The sequence id passed to OPEN_CONFIRM must be 1 (one) greater than the seqid passed to the OPEN operation (Section 9.1.4). If the server gets an unexpected sequence id compared to the original OPEN, then the server assumes the client won't confirm the original OPEN, and all state tied to that original OPEN gets released by the server 💀
On success, the current filehandle keeps its value. 🌸
16.18.5. IMPLEMENTATION ✨
Okay so a single client can make tons of open_owner4 structures for one client ID 💅. Over time it either throws them away or just stops using them for ages 🙈. That's the reason NFSv4 doesn't have an explicit operation to exit an open_owner4, because it would be totally pointless in that situation. So to avoid unbounded memory use, the server needs a plan for getting rid of open_owner4s that have no open state on any files and haven't been used recently 🧹. How long it waits before cleaning up is up to the implementer. It should definitely be no less than the lease time plus any grace period the server wants to add on top of a lease time ⏳. The OPEN_CONFIRM operation is what lets the server safely dispose of unused open_owner4 data structures 🫶.
Now, if a client sends an OPEN and the server has no record of that open_owner4, the server has to make sure it's a brand-new OPEN and not a replay or a retransmission 👀.
Servers MUST NOT require confirmation on OPENs that grant delegations or are doing reclaim operations. See Section 9.1.11 for details 🔥. This is super easy to handle: the server just notes whether it has ever disposed of an open_owner4 for the given client ID ✨. If the server doesn't support delegation, it could keep a single bit that says whether any open_owner4 (for any client) has been disposed of 🦋.
The server has to hold unconfirmed OPEN state until one of three things happens 💖.
First, the client sends an OPEN_CONFIRM request with the right sequence id and stateid within the lease period 🎀. Then the OPEN state on the server becomes confirmed, and the open_owner4 is fully established 👑.
Second, the client sends another OPEN request with a sequence id that's wrong for that open_owner4 (out of sequence) 😭. In this case, the server assumes the second OPEN is the valid one and the first one was a replay 🙃. It cancels the OPEN state from the first OPEN request, sets up unconfirmed OPEN state for the second one, and replies to the second OPEN saying an OPEN_CONFIRM is needed 🌸. Then the whole process repeats itself 🔁. There is a potential denial-of-service attack on the client here, but it gets mitigated if the client and server both require a security flavor based on Kerberos V5 or some other flavor that uses cryptography 🔐.
What if the server is in the unconfirmed OPEN state for a given open_owner4, and it gets an operation on that open_owner4 that has a stateid but isn't OPEN? Or it's an OPEN_CONFIRM but with the wrong stateid? 🤔 Then the server returns NFS4ERR_BAD_STATEID, even if the seqid is correct 💀. That's because the server treats it as a replay: if there's no established OPEN state, there's no way a LOCK operation, for example, could be valid 🙅♀️.
Third, neither of the first two things happens for that open_owner4 within the lease period ⏰. In this case, the OPEN state gets canceled and the open_owner4 can be disposed of 🗑️.
16.19. Operation 21: OPEN_DOWNGRADE - Reduce Open File Access 🦋
16.19.1. SYNOPSIS
(cfh), stateid, seqid, access, deny -> stateid16.19.2. ARGUMENT
struct OPEN_DOWNGRADE4args {
/* CURRENT_FH: opened file */
stateid4 open_stateid;
seqid4 seqid;
uint32_t share_access;
uint32_t share_deny;
};16.19.3. RESULT
struct OPEN_DOWNGRADE4resok {
stateid4 open_stateid;
};
union OPEN_DOWNGRADE4res switch (nfsstat4 status) {
case NFS4_OK:
OPEN_DOWNGRADE4resok resok4;
default:
void;
};16.19.4. DESCRIPTION
Okay so this operation is how you tweak the share_access and share_deny bits for a given open 🎀. You need it when one open-owner opens the same file several times with different share_access and share_deny flags 🌈. Then, when one of those opens gets closed, it might change which flags still apply, so bits tied to opens that aren't in effect anymore can get removed 🧹.
The share_access and share_deny bits you specify here replace the current ones for the open file 💅. The bits you specify MUST be exactly equal to the union of the share_access and share_deny bits from some subset of the OPENs in effect for the current open-owner on the current file. Hmm, wait, let me say it properly: they must be exactly equal to that union, no more, no less ✨. If that rule isn't followed, the error NFS4ERR_INVAL should be returned 🚫. And since share_access and share_deny bits are only ever subsets of what's already granted, this request can't get denied because of conflicting share reservations 🙌.
Hmm, one more thing 👀: the OPEN_DOWNGRADE might turn a file into not-open-for-write while a write byte-range lock is still held. So the server may have to reject the OPEN_DOWNGRADE with an NFS4ERR_LOCKS_HELD 🔒.
On success, the current filehandle keeps its value 💖.
16.20. Operation 22: PUTFH - Set Current Filehandle 🌸
16.20.1. SYNOPSIS
filehandle -> (cfh)16.20.2. ARGUMENT
struct PUTFH4args {
nfs_fh4 object;
};16.20.3. RESULT
struct PUTFH4res {
/* CURRENT_FH: */
nfsstat4 status;
};16.20.4. DESCRIPTION
PUTFH swaps out the current filehandle for the filehandle you pass in as the argument 🧚.
If the security mechanism the requester is using doesn't meet the requirements of the filehandle given to this operation, the server MUST return NFS4ERR_WRONGSEC 🚨.
For more on the current filehandle, check out Section 15.2.4.1 📖.
16.20.5. IMPLEMENTATION
PUTFH is commonly the first operator in an NFS request, used to set the context for the operations that come after it 🎀.
16.21. Operation 23: PUTPUBFH - Set Public Filehandle 🌈
16.21.1. SYNOPSIS
- -> (cfh)16.21.2. ARGUMENT
void;16.21.3. RESULT
struct PUTPUBFH4res {
/* CURRENT_FH: public fh */
nfsstat4 status;
};16.21.4. DESCRIPTION
PUTPUBFH replaces the current filehandle with the one that represents the server's public filehandle for its namespace 🦋. This one might not be the same as the root filehandle, since the root might be tied to some other directory on the server 🌳.
The public filehandle idea came from [RFC2054], [RFC2055], and [RFC2224] 📚. The goal for NFSv4 is that the public filehandle (the thing PUTPUBFH gives you) gets used to stay compatible with the WebNFS server of NFSv2 and NFSv3 🫶.
The public filehandle and the root filehandle (the one PUTROOTFH gives you) should be equivalent 💅. But if they aren't equivalent, the public filehandle MUST be a descendant of the root filehandle 🌲.
16.21.5. IMPLEMENTATION
PUTPUBFH is used as the first operator in an NFS request to set the context for the operations that follow 🎀.
With the NFSv2 and NFSv3 public filehandle, the client could say whether the pathname in a LOOKUP should be read as an absolute path from the server's root, or as a path relative to the public filehandle 🧭. [RFC2224] has more on that 📖. In NFSv4, you can't specify that directly in LOOKUP 😭, because the component separators you'd need to tell absolute from relative aren't allowed in NFSv4 🚫. So the client has to build its request so that either PUTROOTFH or PUTPUBFH is used, to signal absolute or relative evaluation of an NFS URL 💖.
Also, there are warnings in [RFC2224] about absolute evaluation, and about the restrictions the server might put on it based on how much of its namespace it has made available ⚠️. Those same warnings apply to NFSv4 🙈. So because of server implementation details, an NFSv3 absolute public filehandle lookup might behave differently from an NFSv4 absolute resolution 🤷♀️.
There's a form of security negotiation described in [RFC2755] that uses the public filehandle to employ the Simple and Protected GSS-API Negotiation Mechanism (SNEGO) [RFC4178] 🔐. That method isn't available in NFSv4, since filehandles don't carry special meaning there and so don't offer the same framework NFSv2 and NFSv3 do 🙅♀️. So clients should use the security negotiation mechanisms described in this RFC instead ✨.
16.22. Operation 24: PUTROOTFH - Set Root Filehandle 👑
16.22.1. SYNOPSIS
- -> (cfh)16.22.2. ARGUMENT
void;16.22.3. RESULT
struct PUTROOTFH4res {
/* CURRENT_FH: root fh */
nfsstat4 status;
};16.22.4. DESCRIPTION
PUTROOTFH replaces the current filehandle with the one that represents the root of the server's namespace 🌳. From that filehandle, a LOOKUP operation can find any other filehandle on the server 🔍. This one might be different from the public filehandle, which could be tied to some other directory on the server 💫.
For more on the current filehandle, check out Section 15.2.4.1 📖.
16.22.5. IMPLEMENTATION
PUTROOTFH is commonly the first operator in an NFS request, used to set the context for the operations after it 🎀.
16.23. Operation 25: READ - Read from File 📖
16.23.1. SYNOPSIS
(cfh), stateid, offset, count -> eof, data16.23.2. ARGUMENT
struct READ4args {
/* CURRENT_FH: file */
stateid4 stateid;
offset4 offset;
count4 count;
};16.23.3. RESULT
struct READ4resok {
bool eof;
opaque data<>;
};
union READ4res switch (nfsstat4 status) {
case NFS4_OK:
READ4resok resok4;
default:
void;
};16.23.4. DESCRIPTION 💅
okay so READ is literally the operation that pulls data out of the regular file that the current filehandle points to 📖✨
the client tells the server two things: an offset (where to start) and a count (how many bytes it wants). offset 0 means "start at the very beginning of the file" 🌸 and if the offset is greater than or equal to the size of the file, the server replies with NFS4_OK, a data length of 0, and eof set to TRUE 🙈
READ is still subject to access permissions checking, so no skipping that step bestie 🔐
if the client asks for a count of 0, the READ still succeeds and gives back 0 bytes of data (again, permissions still apply 💅). the server may choose to send back fewer bytes than the client asked for, and the client needs to check for this and handle it properly 🧚
the stateid on a READ is the value that came back from an earlier byte-range lock or share reservation request, or it's the stateid tied to a delegation 🦋 the server uses it to double-check that the share reservation and any byte-range locks are still valid, and it also uses it to refresh the client's lease timeouts ⏰
now the eof flag: it comes back as TRUE if the READ ended right at end-of-file (formally, in a correctly formed READ request, if offset + count is equal to the size of the file) or if the READ went past the end of the file (if offset + count is greater than the size of the file). otherwise it's FALSE 🌈 and a successful READ of an empty file will always give you eof as TRUE, no exceptions 💜
if the current filehandle isn't a regular file, the client gets an error back 🙅♀️ if the current filehandle is a directory, you get NFS4ERR_ISDIR, and for anything else you get NFS4ERR_INVAL 💀
for a READ using the special anonymous stateid, the server MAY allow the READ to be serviced subject to mandatory file locks or the current share_deny modes for the file 🔒 and for a READ using the special READ bypass stateid, the server MAY allow READ operations to bypass locking checks at the server 🦋
the MAY bits here are the server's call, so the client can't assume either behaviour 👀
on success, the current filehandle keeps its value ✨
16.23.5. IMPLEMENTATION 🫶
if the server sends back a "short read" (less data than requested and eof set to FALSE), the client should send another READ to grab the rest 🏃♀️ there are a few reasons a server might return less data than asked for. the file may have been truncated by another client or maybe by the server itself, which changes the file size from what the requesting client thinks it is, and that shrinks the amount of data available to the client 😭 the server might also reduce its transfer size and return a short read on purpose, or server resource exhaustion could cause one too 💅
now the locking drama 🔥: if mandatory byte-range locking is in effect for the file, and the byte range you want to read is WRITE_LT locked by an owner that isn't associated with the stateid, the server will return the NFS4ERR_LOCKED error 🙈 the client should try to get the right READ_LT through the LOCK operation before trying the READ again, and when the READ finishes, the client should release that byte-range lock with LOCKU 🔐✨
and then there's delegations 👑: if another client has an OPEN_DELEGATE_WRITE delegation on the file being read, that delegation must be recalled, and the operation can't go ahead until the delegation is returned or revoked 🥺 unless it happens super quickly, one or more NFS4ERR_DELAY errors will be returned to requests made while the delegation is still outstanding ⏳ normally a READ won't cause a recall, since the recall would have been triggered by an earlier OPEN. BUT, since a READ can be done with a special stateid, the server needs to check for this case even when the client should have done an OPEN first 🧐💖
16.24. Operation 26: READDIR - Read Directory 📂
16.24.1. SYNOPSIS 🌷
(cfh), cookie, cookieverf, dircount, maxcount, attr_request ->
cookieverf { cookie, name, attrs }16.24.2. ARGUMENT 🍓
struct READDIR4args {
/* CURRENT_FH: directory */
nfs_cookie4 cookie;
verifier4 cookieverf;
count4 dircount;
count4 maxcount;
bitmap4 attr_request;
};16.24.3. RESULT 🎀
struct entry4 {
nfs_cookie4 cookie;
component4 name;
fattr4 attrs;
entry4 *nextentry;
};
struct dirlist4 {
entry4 *entries;
bool eof;
};
struct READDIR4resok {
verifier4 cookieverf;
dirlist4 reply;
};
union READDIR4res switch (nfsstat4 status) {
case NFS4_OK:
READDIR4resok resok4;
default:
void;
};16.24.4. DESCRIPTION 💜
READDIR is the op that grabs a variable number of entries from a file system directory 📂 for each entry, it returns whichever attributes the client asked for, plus the info the client needs to ask for more entries in a later READDIR ✨
the arguments include a cookie that says where the READDIR should start in the directory 🔖 a cookie of 0 means "start at the very beginning" 🌸 for later requests, the client passes along a cookie that the server gave back in an earlier READDIR 🦋
the cookieverf should be set to 0 when the cookie is 0 (that's the first directory read) 🙌 on later requests, it should be a cookieverf the server returned. the cookieverf must match the one returned by the READDIR that handed out the cookie 💅 and if the server decides the cookieverf isn't valid for the directory anymore, it must return NFS4ERR_NOT_SAME 😳
the dircount part of the arguments is a hint for the max number of bytes of directory info that should come back 📏 that length covers the names of the entries and their cookie values, and it's measured as the XDR encoding of that data (not the length in the server's native format) 🧚
maxcount is the max number of bytes for the whole result 📦 that includes everything inside READDIR4resok, and it counts the XDR overhead too. the server may send back less data than that. if the server can't fit even a single directory entry within maxcount, the client gets NFS4ERR_TOOSMALL 😭
last but not least, attr_request is the list of attributes to return for each directory entry 📋
on a successful return, the response has a list of directory entries 🎉 each one has the entry's name, a cookie for that entry, and the attributes that were requested. the "eof" flag is TRUE if there are no more entries in the directory 🏁
the cookie is only meaningful to the server and works as a "bookmark" for the directory entry 🔖 the client uses it for later READDIR operations so it can keep reading the directory. it's kind of like a READ offset, but the client shouldn't interpret it like one 🙅♀️ the server SHOULD try to accept cookie values it gave out with READDIR responses, even if the directory changed between the READDIR calls, but it MAY return NFS4ERR_NOT_VALID if that isn't possible, like if the server rebooted in the meantime 🔄
sometimes the server runs into an error while getting the attributes for one directory entry. instead of failing the whole READDIR, it can return the attribute fattr4_rdattr_error 🛟 that way it can tell the client about the failure without failing the entire operation, which is great if the problem is just a temporary glitch ✨ but the client has to request fattr4_rdattr_error for this to work. if the client doesn't ask for it, the server has no choice but to fail the entire READDIR operation 😬
for some file systems, the directory entries "." and ".." have special meaning, and for others they don't mean anything 🤷♀️ if the server supports these special entries, they should not be returned to the client as part of the READDIR response 🙈 to help some client environments, the cookie values 0, 1, and 2 are reserved. the UNIX client uses these values when it combines the server's response with its local representation, so the application gets a fully formed UNIX directory presentation 🐧💖
so for READDIR arguments, cookie values of 1 and 2 SHOULD NOT be used, and for READDIR results, cookie values of 0, 1, and 2 MUST NOT be returned 🚫
on success, the current filehandle keeps its value 🌈
16.24.5. IMPLEMENTATION ✨
okay so bestie, servers' directory setups can be WILDLY different 😭 and client APIs don't always fit neatly onto NFS. That's why the dircount and maxcount fields exist: they let the client drop hints so the server knows how to keep the reply in check. If the client is going all in on attribute collection during a READDIR, the server gets a sense of how to limit the encoded response. dircount is a hint about how many entries to return, based only on the names of the directory entries. Since it's just a hint, the dircount value can be zero, and then the server is free to ignore it and go by the maxcount value instead 🧚
Next up, cookies. The client has no way to say "I'm done with this cookie, I'll never use it again," so server implementations should avoid schemes that allocate memory for each returned cookie. The fix is pretty chill: base the cookie values on something like the offset within the directory where the scan is to be resumed 💅
Cookies made this way should stay valid even if the directory gets modified. If a server invalidated a cookie just because of a directory change, READDIRs of large directories might literally never finish 💀
If a directory is deleted after the client has done one or more READDIR operations on it, the cookies become invalid. But the server doesn't need to stress, because the directory filehandle used before would already be stale, and that's what gets reported on later READDIRs. The server wouldn't need to check the cookie verifier in this case 🫶
BUT, certain reorganization operations on a directory (including directory compaction) may invalidate READDIR cookies that were already handed out. When that happens, the server should modify the cookie verifier so cookies that would no longer be valid can't be used. Main character energy for the verifier, honestly 👑
The cookieverf can be used by the server to help manage cookie values that might go stale. It should be a rare occurrence that a server can't keep reading a directory with the provided cookie/cookieverf pair. The server should make every effort to avoid this condition, since the application at the client may not be able to handle this type of failure properly 😭
The cookieverf also protects the client from using READDIR cookie values that may be stale. For example, if the file system has been migrated, the new server may or may not be able to use the same cookie values the previous server used. With the client providing the cookieverf, the server can give the right response. This prevents the nightmare case where the server accepts a cookie, but the directory has changed and the response is invalid from the client's point of view for its previous READDIR 🙈
Last one: since some servers won't return "." and ".." entries the way previous versions of the NFS protocol did, a client that needs those entries in READDIR responses has to fabricate them 🍓
16.25. Operation 27: READLINK - Read Symbolic Link 🦋
16.25.1. SYNOPSIS
(cfh) -> linktext16.25.2. ARGUMENT
/* CURRENT_FH: symlink */
void;16.25.3. RESULT
struct READLINK4resok {
linktext4 link;
};
union READLINK4res switch (nfsstat4 status) {
case NFS4_OK:
READLINK4resok resok4;
default:
void;
};16.25.4. DESCRIPTION
READLINK reads the data associated with a symbolic link. The data is a UTF-8 string that is opaque to the server. That means, whether it was created by an NFS client or created locally on the server, the data in a symbolic link isn't interpreted when it's created. It's just stored 🌸
On success, the current filehandle keeps its value.
16.25.5. IMPLEMENTATION
A symbolic link is nominally a pointer to another file. The server doesn't necessarily interpret the data; it's just stored in the file. A client implementation can store a pathname in a symbolic link that isn't meaningful to the server's operating system. A READLINK operation returns the data to the client so the client can interpret it. If different implementations want to share access to symbolic links, then they must agree on how the data in the symbolic link is interpreted 🤝
The READLINK operation is only allowed on objects of type NF4LNK. The server should return the error NFS4ERR_INVAL if the object is not of type NF4LNK. ⚠️
16.26. Operation 28: REMOVE - Remove File System Object 🗑️
16.26.1. SYNOPSIS
(cfh), filename -> change_info16.26.2. ARGUMENT
struct REMOVE4args {
/* CURRENT_FH: directory */
component4 target;
};16.26.3. RESULT
struct REMOVE4resok {
change_info4 cinfo;
};
union REMOVE4res switch (nfsstat4 status) {
case NFS4_OK:
REMOVE4resok resok4;
default:
void;
};16.26.4. DESCRIPTION
The REMOVE operation removes (deletes) a directory entry named by filename from the directory that the current filehandle points to. If that entry was the last reference to the corresponding file system object, the object may be destroyed 💥
For the directory where the filename was removed, the server returns change_info4 information in cinfo. With the atomic field of the change_info4 struct, the server indicates whether the before and after change attributes were obtained atomically with respect to the removal.
If the target is of zero length, NFS4ERR_INVAL will be returned. The target is also subject to the normal UTF-8, character support, and name checks. See Section 12.7 for further discussion.
On success, the current filehandle keeps its value.
16.26.5. IMPLEMENTATION
NFSv3 needed a different operator for directory removal (RMDIR) than for non-directory removal (REMOVE). That let clients skip checking the file type when handed a non-directory delete system call (like unlink() [unlink] in POSIX) to remove a directory, and the converse too (like a rmdir() on a non-directory), because they knew the server would check the file type. NFSv4 REMOVE can delete any directory entry, no matter its file type 🙌 The implementer of an NFSv4 client's entry points from the unlink() and rmdir() system calls should first check the file type against the types that system call is allowed to remove before issuing a REMOVE. Alternatively, the implementer can produce a COMPOUND call that includes a LOOKUP/VERIFY sequence to verify the file type before a REMOVE operation in the same COMPOUND call.
The concept of last reference is server specific. However, if the numlinks field in the previous attributes of the object had the value 1, the client should not rely on referring to the object via a filehandle. Likewise, the client should not rely on the resources (disk space, directory entry, and so on) formerly associated with the object becoming immediately available. So if a client needs to keep accessing a file after using REMOVE on it, the client should make sure the file will still be accessible. The usual mechanism is to RENAME the file from its old name to a new hidden name 🫣
If the server finds that the file is still open when the REMOVE arrives:
- The server SHOULD NOT delete the file's directory entry if the file was opened with OPEN4_SHARE_DENY_WRITE or OPEN4_SHARE_DENY_BOTH.
- If the file was not opened with OPEN4_SHARE_DENY_WRITE or OPEN4_SHARE_DENY_BOTH, the server SHOULD delete the file's directory entry. However, until the last CLOSE of the file, the server MAY continue to allow access to the file via its filehandle.
16.27. Operation 29: RENAME - Rename Directory Entry 💌
16.27.1. SYNOPSIS
(sfh), oldname, (cfh), newname -> source_cinfo, target_cinfo16.27.2. ARGUMENT
struct RENAME4args {
/* SAVED_FH: source directory */
component4 oldname;
/* CURRENT_FH: target directory */
component4 newname;
};16.27.3. RESULT
struct RENAME4resok {
change_info4 source_cinfo;
change_info4 target_cinfo;
};
union RENAME4res switch (nfsstat4 status) {
case NFS4_OK:
RENAME4resok resok4;
default:
void;
};16.27.4. DESCRIPTION
The RENAME operation renames the object identified by oldname in the source directory (the one the saved filehandle points to, as set by the SAVEFH operation) to newname in the target directory (the one the current filehandle points to). The operation is required to be atomic to the client. Source and target directories must reside on the same file system on the server. On success, the current filehandle will continue to be the target directory 📂
If the target directory already has an entry with the name newname, the source object must be compatible with the target. That means either both are non-directories, or both are directories and the target must be empty. If they're compatible, the existing target is removed before the rename happens (see Section 16.26 for client and server actions whenever a target is removed). If they're not compatible, or if the target is a directory but isn't empty, the server returns the error NFS4ERR_EXIST 🚫
If oldname and newname both refer to the same file (they might be hard links of each other), then RENAME should perform no action and return success ✅
For both directories involved in the RENAME, the server returns change_info4 information. With the atomic field of the change_info4 struct, the server indicates whether the before and after change attributes were obtained atomically with respect to the rename.
If oldname refers to a named attribute, and the saved and current filehandles refer to the named attribute directories of different file system objects, the server returns NFS4ERR_XDEV, just as if the saved and current filehandles represented directories on different file systems 🌈
If oldname or newname is of zero length, NFS4ERR_INVAL will be returned. oldname and newname are also subject to the normal UTF-8, character support, and name checks. See Section 12.7 for further discussion.
16.27.5. IMPLEMENTATION
okay bestie, RENAME has to be atomic to the client 🦋 no half-done renames, periodt ✨ the bit about source and target directories needing to live on the same file system on the server just means the fsid fields in the directories' attributes have to match 🎀 if they're on different file systems, the server hands back NFS4ERR_XDEV and that's a whole no 💀
so the filehandle might or might not expire during a RENAME, and it depends on the fh_expire_type attribute for the object 🌸 servers are still strongly encouraged to try to keep filehandles from expiring like this, because it's the iconic move 👑
some servers are picky about names: "." and ".." are illegal as either oldname or newname and will get you NFS4ERR_BADNAME 🙈 and on lots of servers, they also check whether oldname or newname is just an alias for the source directory. if that's the case, the server returns NFS4ERR_INVAL 💅
last thing: if either the source or target filehandle isn't a directory, the server returns NFS4ERR_NOTDIR 🧚
16.28. Operation 30: RENEW - Renew a Lease
16.28.1. SYNOPSIS
clientid -> ()16.28.2. ARGUMENT
struct RENEW4args {
clientid4 clientid;
};16.28.3. RESULT
struct RENEW4res {
nfsstat4 status;
};16.28.4. DESCRIPTION
RENEW is how the client keeps its leases alive at a server 🌈 when the server processes a RENEW request, it renews all the leases tied to that client. and those leases are figured out by the clientid you gave through the SETCLIENTID operation 💜
16.28.5. IMPLEMENTATION
okay this part is kind of dramatic 😭 if the client holds delegations, it needs RENEW to find out when the server has decided the callback path is down 💀 once the server has made that call, only RENEW will renew the lease on delegations. and if the server thinks the callback path is down, it returns NFS4ERR_CB_PATH_DOWN.
but here's the twist: even though it returns NFS4ERR_CB_PATH_DOWN, the server MUST still renew the lease on the byte-range locks and share reservations the client set up on the server 🔥 and if the server can't renew the lock and share reservation lease for some reason, it MUST return an error other than NFS4ERR_CB_PATH_DOWN, even if the callback path is also down 😭 if the server could return either NFS4ERR_CB_PATH_DOWN or NFS4ERR_LEASE_MOVED, then NFS4ERR_LEASE_MOVED MUST be handled first 👑
the client that issues RENEW MUST pick the principal, RPC security flavor, and (if it applies) the GSS-API mechanism and service using one of these two algorithms:
- the client uses the same principal, RPC security flavor, and -- if the flavor was RPCSEC_GSS -- the same mechanism and service that were used when the client ID was established via SETCLIENTID_CONFIRM.
- the client uses any principal, RPC security flavor, mechanism, and service combination that currently has an OPEN file on the server. that means the same principal had a successful OPEN operation, the file is still open by that principal, and the flavor, mechanism, and service of RENEW match the previous OPEN.
the server MUST reject any RENEW that doesn't use one of those algorithms, and it does that with NFS4ERR_ACCESS 🙅♀️
16.29. Operation 31: RESTOREFH - Restore Saved Filehandle
16.29.1. SYNOPSIS
(sfh) -> (cfh)16.29.2. ARGUMENT
/* SAVED_FH: */
void;16.29.3. RESULT
struct RESTOREFH4res {
/* CURRENT_FH: value of saved fh */
nfsstat4 status;
};16.29.4. DESCRIPTION
RESTOREFH sets the current filehandle back to whatever's in the saved filehandle 🦋 and if there's no saved filehandle, it returns NFS4ERR_RESTOREFH 🙈
16.29.5. IMPLEMENTATION
so operations like OPEN and LOOKUP use the current filehandle to represent a directory and then swap it for a brand new filehandle 💅 if you saved the previous one with SAVEFH, you can restore it as the current filehandle with RESTOREFH. this is commonly used to grab post-operation attributes for the directory 🎀 for example:
PUTFH (directory filehandle)
SAVEFH
GETATTR attrbits (pre-op dir attrs)
CREATE optbits "foo" attrs
GETATTR attrbits (file attributes)
RESTOREFH
GETATTR attrbits (post-op dir attrs)16.30. Operation 32: SAVEFH - Save Current Filehandle
16.30.1. SYNOPSIS
(cfh) -> (sfh)16.30.2. ARGUMENT
/* CURRENT_FH: */
void;16.30.3. RESULT
struct SAVEFH4res {
/* SAVED_FH: value of current fh */
nfsstat4 status;
};16.30.4. DESCRIPTION
SAVEFH saves the current filehandle 💾 if there was already a saved one, it's gone for good and no longer accessible 😭 you can bring the saved one back as the current filehandle with the RESTOREFH operator 🧚
on success, the current filehandle keeps its value ✨
16.30.5. IMPLEMENTATION
16.31. Operation 33: SECINFO - Obtain Available Security
16.31.1. SYNOPSIS
(cfh), name -> { secinfo }16.31.2. ARGUMENT
struct SECINFO4args {
/* CURRENT_FH: directory */
component4 name;
};16.31.3. RESULT
/*
* From RFC 2203
*/
enum rpc_gss_svc_t {
RPC_GSS_SVC_NONE = 1,
RPC_GSS_SVC_INTEGRITY = 2,
RPC_GSS_SVC_PRIVACY = 3
};
struct rpcsec_gss_info {
sec_oid4 oid;
qop4 qop;
rpc_gss_svc_t service;
};
/* RPCSEC_GSS has a value of '6'. See RFC 2203 */
union secinfo4 switch (uint32_t flavor) {
case RPCSEC_GSS:
rpcsec_gss_info flavor_info;
default:
void;
};
typedef secinfo4 SECINFO4resok<>;
union SECINFO4res switch (nfsstat4 status) {
case NFS4_OK:
SECINFO4resok resok4;
default:
void;
};16.31.4. DESCRIPTION
SECINFO is how the client finds out which RPC authentication flavors are valid for a specific directory filehandle and filename pair 🔍 it should use the same access methodology as LOOKUP when it checks the name. so if the requester doesn't have the access needed to do a LOOKUP for that name, then SECINFO must act the same way and return NFS4ERR_ACCESS 🙅♀️
the result is an array of the security mechanisms the server has available, in the server's order of preference, with the most preferred one first 👑 the client can pick whatever mechanism it wants and supports, or it can go down the server's preference list and take the first one it supports 💅 the array entries are secinfo4 structures. the flavor field will hold AUTH_NONE, AUTH_SYS (as defined in [RFC5531]), or RPCSEC_GSS (as defined in [RFC2203]).
for AUTH_NONE and AUTH_SYS, no extra security info comes back. for RPCSEC_GSS, you get a security triple with the mechanism object id (as defined in [RFC2743]), the quality of protection (as defined in [RFC2743]), and the service type (as defined in [RFC2203]) 🌈 SECINFO can return multiple entries with flavor equal to RPCSEC_GSS, each with different security triple values.
on success, the current filehandle keeps its value 🎀
if the name has a length of 0 (zero), or if the name doesn't follow the UTF-8 definition, the server returns NFS4ERR_INVAL 🙈
16.31.5. IMPLEMENTATION
the NFS client is expected to use SECINFO when another NFS operation returns NFS4ERR_WRONGSEC 🚨 that error means the server's security policy is different from what the client is currently using. at that point, the client is expected to get a list of possible security flavors and pick the one that best fits its policies 💖
so the server's security policies decide when a client request gets NFS4ERR_WRONGSEC. the operations that can get that error are LINK, LOOKUP, LOOKUPP, OPEN, PUTFH, PUTPUBFH, PUTROOTFH, RENAME, RESTOREFH, and (indirectly) READDIR 🧚 LINK and RENAME only get this error if the security used for the operation doesn't work for the saved filehandle. except for READDIR, these operations are the moment the client can put a filehandle into the current filehandle at the server. the filehandle is either handed over by the client (PUTFH, PUTPUBFH, PUTROOTFH) or created by a name-to-filehandle translation (LOOKUP and OPEN). RESTOREFH is different because its filehandle came from an earlier SAVEFH. even if that filehandle already passed the server's security check, the server checks it again on RESTOREFH to make sure the security policy hasn't changed 🦋
if the client wants to deal with a NFS4ERR_WRONGSEC error, here's what happens:
- for LOOKUP and OPEN, the client uses SECINFO with the same current filehandle and name that were in the original LOOKUP or OPEN, so it can list the available security triples.
- for LINK, PUTFH, RENAME, and RESTOREFH, the client uses SECINFO and gives the parent directory filehandle plus the object name that matches the filehandle originally given by PUTFH or RESTOREFH, or -- for LINK and RENAME -- by the SAVEFH.
- for LOOKUPP, PUTROOTFH, and PUTPUBFH, the client can't use SECINFO, since SECINFO needs a current filehandle and none exists for these three operations. so the client must go through the security triples it has available and try PUTROOTFH or PUTPUBFH again. in the unfortunate event that none of the MANDATORY security triples are supported by both the client and the server, the client SHOULD try others that support integrity 😭 if that doesn't work, the client can try AUTH_NONE, but since those forms lack integrity checks, this puts the client at risk. even so, the server SHOULD let the client use whatever security form it requests and the server supports, because the risks are on the client 🫶
READDIR won't directly return NFS4ERR_WRONGSEC. but if the READDIR request asked for attributes, the READDIR request's security triple might not match a directory entry's. if that happens and the client asked for the rdattr_error attribute, the server returns NFS4ERR_WRONGSEC in rdattr_error for that entry 💅
also, a server MAY use the AUTH_NONE flavor to signal that the client can try authentication flavors that aren't explicitly listed in the SECINFO results 👑 instead of a listed flavor, the client might then choose an unlisted RPCSEC_GSS mechanism rather than AUTH_NONE, maybe to meet an application's need for data integrity or privacy. but when the client picks an unlisted flavor, it SHOULD always be ready to fall back to AUTH_NONE or another listed flavor if it fails 🙈 the client can't assume identity mapping is supported, so it should be ready for its identity to get squashed 💀
see Section 19 for the recommendations on which security flavors should be used with SECINFO 🎀
16.32. Operation 34: SETATTR - Set Attributes
16.32.1. SYNOPSIS ✨
(cfh), stateid, attrmask, attr_vals -> attrsset16.32.2. ARGUMENT 🎀
struct SETATTR4args {
/* CURRENT_FH: target object */
stateid4 stateid;
fattr4 obj_attributes;
};16.32.3. RESULT 🧚
struct SETATTR4res {
nfsstat4 status;
bitmap4 attrsset;
};16.32.4. DESCRIPTION 💅
okay so SETATTR is the op that changes one or more attributes of a file system object 🗂️. the new attributes get specified with a bitmap, and then the attribute values that follow the bitmap come in bit order. so the bitmap is basically the guest list 📋 lol
the stateid argument is there to give byte-range locking context for SETATTR requests that set the size attribute 🔒. setting size modifies the file's data, so it has the same locking requirements as a corresponding WRITE, no cap. any SETATTR that sets size is incompatible with a share reservation that specifies OPEN4_SHARE_DENY_WRITE 🙅♀️. the area between the old end-of-file and the new end-of-file counts as modified, just like it would if that area had been the target of a WRITE. this matters for checking conflicts with byte-range locks, in cases where the server is implementing mandatory byte-range locking behavior. A valid stateid SHOULD always be specified. When the file size attribute is not set, the special anonymous stateid MAY be passed ✨
on success or failure, the server returns the attrsset bitmask to show what (if any) attributes were successfully set 💖. the attrsset in the response is a subset of the bitmap4 that's part of the obj_attributes in the argument. and on success, the current filehandle keeps its value 🫶
16.32.5. IMPLEMENTATION 🔥
if the request asks to set the owner attribute, the server SHOULD let the operation succeed when the current owner of the object matches the value in the request 👑. some servers might be built to prohibit setting the owner attribute unless the requester has the privilege to do so. if the server is lenient in this one case of matching owner values, the client implementation can be simpler when it creates an object (for example, an exclusive create via OPEN) and then does a SETATTR 🧚
the file size attribute is how you request changes to a file's size 📏. a value of zero truncates the file. a value less than the current size discards data from the new size to the end of the file. and a size greater than the current size adds logically zeroed data bytes to the end of the file. servers are free to implement this with holes or with actual zero data bytes. clients should not make any assumptions about how a server implements this feature, beyond the fact that the bytes returned will be zeroed. Servers MUST support extending the file size via SETATTR 💅
SETATTR is not guaranteed atomic 🙈. a failed SETATTR may partially change a file's attributes, which is exactly why the reply always includes the status and the list of attributes that were set.
if the object whose attributes are being changed has a file delegation held by a client other than the one doing the SETATTR, the delegation(s) must be recalled. the operation cannot proceed to actually change an attribute until each such delegation is returned or revoked 😭. in every case where delegations get recalled, the server is likely to return one or more NFS4ERR_DELAY errors while the delegation(s) remain outstanding. it might not do that if the delegations come back quickly, though.
changing the size of a file with SETATTR indirectly changes the time_modify and change attributes. a client must account for this, since size changes can result in data deletion 💀
the attributes time_access_set and time_modify_set are write-only attributes, built as a switched union so the client can direct the server in setting the time values ⏰. if the switched union specifies SET_TO_CLIENT_TIME4, the client has provided an nfstime4 to be used for the operation. if the switch union does not specify SET_TO_CLIENT_TIME4, the server uses its current time for the SETATTR operation.
if server and client times differ, programs that compare client times to file times can break. a time maintenance protocol should be used to limit client/server time skew 🚨
using a COMPOUND that contains a VERIFY operation, specifying only the change attribute and immediately followed by a SETATTR, gives a client a way to make a request that emulates the functionality of the SETATTR guard mechanism of NFSv3 🛡️. the guard mechanism exists to avoid changing file attributes based on stale information. so delays between checking the guard condition and setting the attributes have the potential to compromise that function, and so would the matching delay in the NFSv4 emulation. therefore, NFSv4 servers should take care to avoid such delays, to the degree possible, when executing such a request 🫠
if the server does not support an attribute as requested by the client, the server should return NFS4ERR_ATTRNOTSUPP 🙅♀️
a mask of the attributes actually set is returned by SETATTR in all cases 📝. That mask MUST NOT include attribute bits that the client did not request to be set. If the attribute masks in the request and reply are equal, the status field in the reply MUST be NFS4_OK 🎉
16.33. Operation 35: SETCLIENTID - Negotiate Client ID 🦋
16.33.1. SYNOPSIS 🌸
client, callback, callback_ident -> clientid, setclientid_confirm16.33.2. ARGUMENT 💜
struct SETCLIENTID4args {
nfs_client_id4 client;
cb_client4 callback;
uint32_t callback_ident;
};16.33.3. RESULT 🍓
struct SETCLIENTID4resok {
clientid4 clientid;
verifier4 setclientid_confirm;
};
union SETCLIENTID4res switch (nfsstat4 status) {
case NFS4_OK:
SETCLIENTID4resok resok4;
case NFS4ERR_CLID_INUSE:
clientaddr4 client_using;
default:
void;
};16.33.4. DESCRIPTION 🌈
the client uses SETCLIENTID to tell the server it intends to use a particular client identifier, callback, and callback_ident 📣. this is for later requests that create lock, share reservation, and delegation state on the server. when it succeeds, the server hands back a shorthand client ID. if you confirm that ID in a separate step, it gets used in later file locking and file open requests ✨
confirmation of the client ID must be done via SETCLIENTID_CONFIRM. that's how you return the client ID and the setclientid_confirm values, as verifiers, to the server 🔐. two verifiers are needed because it's possible to use SETCLIENTID and SETCLIENTID_CONFIRM to modify the callback and callback_ident info without changing the shorthand client ID. in that case, setclientid_confirm is effectively the only verifier 🙈
the callback info given in this operation gets used if the client is handed an open delegation at some point later 👑. so the client must correctly reflect the program and port numbers for the callback program at the time SETCLIENTID is used 💖
the server uses the callback_ident value on the callback 📞. the client can lean on callback_ident so it doesn't need more than one callback RPC program number, and it can still tell which server is initiating the callback 🫶
16.33.5. IMPLEMENTATION ✨
okay so before we get into how to implement SETCLIENTID 💅, let's set up some notation so we can keep track of everything 🎀. Let:
xbe the value of theclient.idsubfield of theSETCLIENTID4argsstructure.vbe the value of theclient.verifiersubfield of theSETCLIENTID4argsstructure.cbe the value of the client ID field returned in theSETCLIENTID4resokstructure.krepresent the value combination of thecallbackandcallback_identfields of theSETCLIENTID4argsstructure.sbe thesetclientid_confirmvalue returned in theSETCLIENTID4resokstructure.
{ v, x, c, k, s } is a quintuple for a client record 🦋. A client record is confirmed if there has been a SETCLIENTID_CONFIRM operation to confirm it. Otherwise, it is unconfirmed. An unconfirmed record gets established by a SETCLIENTID call.
Since SETCLIENTID is a non-idempotent operation 🙈, let's assume the server is implementing the duplicate request cache (DRC).
When the server gets a SETCLIENTID { v, x, k } request, it handles it like this 👑:
- It first looks up the request in the DRC. If there is a hit, it returns the result cached in the DRC. The server does NOT remove client state (locks, shares, delegations), nor does it modify any recorded callback and
callback_identinformation for client{ x }.
For any DRC miss, the server takes the client ID string x and searches for client records for x that the server may have recorded from previous SETCLIENTID calls. For any confirmed record with the same id string x, if the recorded principal does not match that of the SETCLIENTID call, then the server returns an NFS4ERR_CLID_INUSE error.
For brevity of discussion, the rest of the processing assumes there was a DRC miss, and that wherever the server has previously recorded a confirmed record for client x, the principal check above has successfully passed.
- The server checks whether it has recorded a confirmed record for
{ v, x, c, l, s }, wherelmay or may not equalk. If so, and since the id verifiervof the request matches the one that is confirmed and recorded, the server treats this as a probable callback information update 🌸. It records an unconfirmed{ v, x, c, k, t }and leaves the confirmed{ v, x, c, l, s }in place, such thatt != s. It does not matter whetherkequalslor not. Any pre-existing unconfirmed{ v, x, c, *, * }is removed.
The server returns { c, t }. It is indeed returning the old clientid4 value c, because the client apparently only wants to update callback value k to value l. It's possible this request is one from the Byzantine router that has stale callback information, but this is not a problem. The callback information update is only confirmed if it's followed up by a SETCLIENTID_CONFIRM { c, t }.
The server awaits confirmation of k via SETCLIENTID_CONFIRM { c, t }.
The server does NOT remove client (lock/share/delegation) state for x.
- The server has previously recorded a confirmed
{ u, x, c, l, s }record such thatv != u, wherelmay or may not equalk, and has not recorded any unconfirmed{ *, x, *, *, * }record forx. The server records an unconfirmed{ v, x, d, k, t }(whered != candt != s).
The server returns { d, t }.
The server awaits confirmation of { d, k } via SETCLIENTID_CONFIRM { d, t }.
The server does NOT remove client (lock/share/delegation) state for x.
- The server has previously recorded a confirmed
{ u, x, c, l, s }record such thatv != u, wherelmay or may not equalk, and it recorded an unconfirmed{ w, x, d, m, t }record such thatc != dandt != s. Heremmay or may not equalk,mmay or may not equall, andkmay or may not equall. Whetherw == vorw != vmakes no difference. The server simply removes the unconfirmed{ w, x, d, m, t }record and replaces it with an unconfirmed{ v, x, e, k, r }record, such thate != d,e != c,r != t, andr != s.
The server returns { e, r }.
The server awaits confirmation of { e, k } via SETCLIENTID_CONFIRM { e, r }.
The server does NOT remove client (lock/share/delegation) state for x.
- The server has no confirmed
{ *, x, *, *, * }forx. It may or may not have recorded an unconfirmed{ u, x, c, l, s }, wherelmay or may not equalk, andumay or may not equalv. Any unconfirmed record{ u, x, c, l, * }, regardless of whetheru == vorl == k, is replaced with an unconfirmed record{ v, x, d, k, t }, whered != candt != s.
The server returns { d, t }.
The server awaits confirmation of { d, k } via SETCLIENTID_CONFIRM { d, t }. The server does NOT remove client (lock/share/delegation) state for x.
The server generates the clientid and setclientid_confirm values, and it must take care to ensure that these values are extremely unlikely to ever be regenerated 🥺✨.
16.34. Operation 36: SETCLIENTID_CONFIRM - Confirm Client ID 💜
16.34.1. SYNOPSIS
clientid, setclientid_confirm -> -16.34.2. ARGUMENT
struct SETCLIENTID_CONFIRM4args {
clientid4 clientid;
verifier4 setclientid_confirm;
};16.34.3. RESULT
struct SETCLIENTID_CONFIRM4res {
nfsstat4 status;
};16.34.4. DESCRIPTION
okay so this operation is how the client confirms the results from a previous call to SETCLIENTID 🫶. The client hands over the server-supplied client ID (the one from the SETCLIENTID response), and the server replies with a simple status of success or failure 💖.
16.34.5. IMPLEMENTATION ✨
okay so the client has to use SETCLIENTID_CONFIRM to confirm two totally different situations 👀
- 💖 Case one: the client is using a brand new shorthand client identifier (the one the server handed back in the SETCLIENTID response), a new callback value (from the SETCLIENTID arguments), and a new
callback_identvalue (also from the SETCLIENTID arguments). When the client does this, SETCLIENTID_CONFIRM also confirms that any of the client's old relevant leased state is gone 🧹. That leased state includes byte-range locks, share reservations, and, where the server does not support theCLAIM_DELEGATE_PREVclaim type, delegations. If the server does supportCLAIM_DELEGATE_PREV, then SETCLIENTID_CONFIRM MUST NOT remove delegations for this client. In that case the relevant leased state is just byte-range locks and share reservations 🦋 - 🌸 Case two: the client is reusing an old, previously confirmed shorthand client identifier, plus a new callback value and a new
callback_identvalue. Here SETCLIENTID_CONFIRM MUST NOT result in the removal of any previous leased state (locks, share reservations, and delegations). Nothing gets deleted, bestie 🫶
We're using the same notation and definitions for v, x, c, k, s, and the unconfirmed and confirmed client records that got introduced in the SETCLIENTID description, so no new vocab needed 🧚 The arguments to SETCLIENTID_CONFIRM are written as { c, s }, where c is a clientid4 and s is a verifier4 that matches the setclientid_confirm field 💅
Just like SETCLIENTID, SETCLIENTID_CONFIRM is a non-idempotent operation, and we're assuming the server is implementing the duplicate request cache (DRC) 🔥
When the server gets a SETCLIENTID_CONFIRM { c, s } request, here's how it handles it:
- 🍓 It checks the DRC first. If there's a hit, it returns the cached DRC result. The server doesn't remove any relevant leased client state, and it doesn't touch any recorded callback or
callback_identinfo for client{ x }(the one represented by the shorthand value c).
If it's a DRC miss, the server looks for client records that match the shorthand value c. The possible cases go like this:
- ✨ The server has an unconfirmed
{ v, x, c, k, s }record AND a confirmed{ v, x, c, l, t }record, withs != t. If the principals of those records don't match the principal of the SETCLIENTID_CONFIRM, the server returnsNFS4ERR_CLID_INUSE. No relevant leased state gets removed, and no callback orcallback_identinfo for client{ x }changes. Otherwise, the confirmed{ v, x, c, l, t }record gets removed, the unconfirmed{ v, x, c, k, s }gets marked as confirmed, and that changes the recorded and confirmed callback andcallback_identinfo for client{ x }🎀
The server does not remove any relevant leased client state, and it returns NFS4_OK 💜
- 🌈 The server has NOT recorded an unconfirmed
{ v, x, c, *, * }but HAS recorded a confirmed{ v, x, c, *, s }. If the principals of the record and the SETCLIENTID_CONFIRM don't match, the server returnsNFS4ERR_CLID_INUSE. It doesn't remove any relevant leased state, and it doesn't change the recorded callback andcallback_identvalues for client{ x }🙈
If the principals DO match, what probably happened is the client never got the response to its SETCLIENTID_CONFIRM, and the DRC entry got purged. Whatever the story is, the principals match and { c, s } matches a confirmed record, so the server leaves client x's relevant leased state intact, leaves its callback and callback_ident values alone, and returns NFS4_OK ✅
- 🦄 The server has NOT recorded a confirmed
{ *, *, c, *, * }but HAS recorded an unconfirmed{ *, x, c, k, s }. Even if this is a retry from the client, the client's first SETCLIENTID_CONFIRM attempt never reached the server. Retry or not, the server can't tell, so it treats the request as a first try 🧠 If the principal of the unconfirmed{ *, x, c, k, s }record doesn't match the principal of the SETCLIENTID_CONFIRM request, the server returnsNFS4ERR_CLID_INUSEand doesn't remove any relevant leased state.
Otherwise, the server records a confirmed { *, x, c, k, s }. If there's ALSO a confirmed { *, x, d, *, t }, the server MUST remove client x's relevant leased client state and overwrite the callback state with k. The confirmed { *, x, d, *, t } record gets removed too 🧹
The server returns NFS4_OK 👑
- 💀 The server has no record of a confirmed or unconfirmed
{ *, *, c, *, s }. The server returnsNFS4ERR_STALE_CLIENTID. It doesn't remove any relevant leased state, and it doesn't modify any recorded callback orcallback_identinfo for any client 😭
The server needs to cache unconfirmed { v, x, c, k, s } client records and wait some time for them to get confirmed ⏳ As you can see from the record-processing discussions for SETCLIENTID and SETCLIENTID_CONFIRM, there are situations where the server can't deterministically remove unconfirmed client records. To avoid running out of resources, the server is not required to hold unconfirmed records forever. One strategy is to set a limit on how many unconfirmed client records it'll keep, and when that limit would be exceeded, remove the oldest one. Another strategy is to remove an unconfirmed record once some amount of time has passed. The amount of time is fairly arbitrary, but it's definitely no higher than the server's lease time period ⏰
Here's why that matters: leases need to get renewed before they expire, and that renewal has to come from an operation sent by the client. If the client can't issue a SETCLIENTID_CONFIRM after a SETCLIENTID before a period of time equal to a lease expiration time, then the client is unlikely to be able to keep its state on the server during steady-state operation 😬
If the client sends a SETCLIENTID_CONFIRM for an unconfirmed record the server has already deleted, the client getsNFS4ERR_STALE_CLIENTIDback. If that happens, the client should start over and send SETCLIENTID to re-establish an unconfirmed client record and get back an unconfirmed client ID andsetclientid_confirmverifier. Then the client should send the SETCLIENTID_CONFIRM to confirm the client ID 🔁
SETCLIENTID_CONFIRM does not establish or renew a lease 🙅♀️ However, if SETCLIENTID_CONFIRM removes relevant leased client state, and that state does NOT include existing delegations, the server MUST allow the client a period of time no less than the value of the lease_time attribute to reclaim its delegations (via the CLAIM_DELEGATE_PREV claim type of the OPEN operation) before it removes the unreclaimed delegations. That's a whole grace period, so the client gets a real chance to claw things back 🦋
16.35. Operation 37: VERIFY - Verify Same Attributes ✨
16.35.1. SYNOPSIS
(cfh), fattr -> -16.35.2. ARGUMENT
struct VERIFY4args {
/* CURRENT_FH: object */
fattr4 obj_attributes;
};16.35.3. RESULT
struct VERIFY4res {
nfsstat4 status;
};16.35.4. DESCRIPTION
VERIFY is the operation where the client double-checks that attributes still have the values it's assuming before it moves on to the next operations in the COMPOUND request 🔍 If any of the attributes don't match, the error NFS4ERR_NOT_SAME must be returned 🙃 The current filehandle keeps its value after a successful completion of the operation 💅
16.35.5. IMPLEMENTATION
One possible use of VERIFY is this COMPOUND sequence. Here the client is trying to verify that the file being removed is the one it expects to remove. This can help prevent the accidental deletion of a file 🗑️🚫
PUTFH (directory filehandle)
LOOKUP (filename)
VERIFY (filehandle == fh)
PUTFH (directory filehandle)
REMOVE (filename)This sequence doesn't stop a second client from removing and creating a new file in the middle of all this, but it DOES help avoid the unintended result 🦋
If a RECOMMENDED attribute is specified in the VERIFY operation and the server doesn't support that attribute for the file system object, the error NFS4ERR_ATTRNOTSUPP gets returned to the client 🙈
When the attribute rdattr_error or any write-only attribute (e.g., time_modify_set) is specified, the error NFS4ERR_INVAL gets returned to the client 💀
16.36. Operation 38: WRITE - Write to File 🌸
16.36.1. SYNOPSIS
(cfh), stateid, offset, stable, data -> count, committed, writeverf16.36.2. ARGUMENT
enum stable_how4 {
UNSTABLE4 = 0,
DATA_SYNC4 = 1,
FILE_SYNC4 = 2
};
struct WRITE4args {
/* CURRENT_FH: file */
stateid4 stateid;
offset4 offset;
stable_how4 stable;
opaque data<>;
};16.36.3. RESULT
struct WRITE4resok {
count4 count;
stable_how4 committed;
verifier4 writeverf;
};
union WRITE4res switch (nfsstat4 status) {
case NFS4_OK:
WRITE4resok resok4;
default:
void;
};16.36.4. DESCRIPTION 💖
okay so WRITE is how you put data into a regular file 📝 The file it targets is whatever the current filehandle is, bestie. The offset tells the server where the data should go, and an offset of 0 (zero) means "start at the very beginning of the file" 🌸 The count is tucked inside the opaque data parameter, and it's the number of bytes that are supposed to get written. If count is 0 (zero), the WRITE will succeed and return a count of 0 (zero), subject to permissions checking 🙈 And the server is allowed to write fewer bytes than you asked for, so don't be shook 💅
Now for the main character moment: the client picks the stable parameter, and that decides how the server handles your data 💾 If stable is FILE_SYNC4, the server must commit the data written plus all file system metadata to stable storage before returning results. This is the NFSv2 protocol semantics energy 🧚 Anything else is a protocol violation 🚨 If stable is DATA_SYNC4, the server must commit all of the data to stable storage and enough of the metadata to retrieve the data before returning. The server implementer is free to treat DATA_SYNC4 the same way as FILE_SYNC4, just with a possible performance drop 🐢 If stable is UNSTABLE4, the server is free to commit any part of the data and the metadata to stable storage, including all or none of it, before it replies to the client 🎲 There is no guarantee whether or when uncommitted data gets committed later, so it's giving "no promises" 💀 The only guarantees the server makes are that it won't destroy any data without changing the value of verf, and that it won't commit the data and metadata at a level less than what the client asked for 🫶
The stateid for a WRITE request is a value you got back from a previous byte-range lock or share reservation request, or it's the stateid tied to a delegation 🔐 The server uses it to check that the associated share reservation and any byte-range locks are still valid, and to update lease timeouts for the client ⏰
When it works, here's what comes back ✨ The count result is the number of bytes of data written to the file, and the server may write fewer bytes than requested. If so, the actual number of bytes written, starting at location, offset, is returned 🦋
The server also tells you its level of commitment for the data and metadata through committed 💅 If the server committed all data and metadata to stable storage, committed should be set to FILE_SYNC4. If the commitment was at least as strong as DATA_SYNC4, then committed should be set to DATA_SYNC4. Otherwise, committed must be returned as UNSTABLE4 💜 If stable was FILE4_SYNC, then committed must also be FILE_SYNC4, and anything else is a protocol violation 🚨 If stable was DATA_SYNC4, then committed may be FILE_SYNC4 or DATA_SYNC4, and anything else is a protocol violation 🚨 If stable was UNSTABLE4, then committed may be FILE_SYNC4, DATA_SYNC4, or UNSTABLE4 🌈
The last bit of the result is the write verifier 🍓 It's a cookie the client uses to figure out whether the server has changed instance (boot) state between a call to WRITE and a later call to either WRITE or COMMIT. This cookie must be consistent during a single instance of the NFSv4 protocol service, and it must be unique between instances of the NFSv4 protocol server, since uncommitted data may be lost in between 🥺
If a client writes with the stable argument set to UNSTABLE4 and the reply comes back with a committed of DATA_SYNC4 or UNSTABLE4, the client will follow up at some point with a COMMIT operation to sync outstanding asynchronous data and metadata with the server's stable storage, barring client error ✨ It's possible that because of a client crash or some other error, a subsequent COMMIT never gets received by the server 😭
For a WRITE using the special anonymous stateid, the server MAY allow the WRITE to be serviced subject to mandatory file locks or the current share deny modes for the file 🔒 For a WRITE using the special READ bypass stateid, the server MUST NOT allow the WRITE operation to bypass locking checks at the server, and the WRITE is treated exactly the same as if the anonymous stateid were used 🚫
On success, the current filehandle keeps its value 💖
16.36.5. IMPLEMENTATION 🛠️
It's possible for the server to write fewer bytes than the client asked for 🤷♀️ In that case, the server should not return an error unless no data was written at all. If the server writes fewer bytes than specified, the client should issue another WRITE to write the rest 🔁
It's assumed that writing data to a file will update the file's time_modify attribute 🕰️ But time_modify should not be changed unless the file's contents actually change, so a WRITE request with count set to 0 should not cause time_modify to be updated 🙅♀️
Okay, stable storage has been a historically messy topic, so these are the expected properties that can help sort out design issues in the implementation 🧠 Stable storage is persistent storage that survives:
- Repeated power failures.
- Hardware failures (of any board, power supply, etc.).
- Repeated software crashes, including reboot cycle.
This definition doesn't cover failure of the stable storage module itself 🤐
The verifier lets a client spot different instances of an NFSv4 protocol server, which is important because cached, uncommitted data may be lost across them 👀 Most of the time, the verifier lets the client detect server reboots 🔄 That info is required so the client can safely figure out whether the server could have lost cached data. If the server fails unexpectedly and the client has uncommitted data from earlier WRITE requests (done with stable set to UNSTABLE4 and committed returned as UNSTABLE4 as well), the data may not have been flushed to stable storage 😰 The burden of recovery is on the client, which will need to retransmit the data to the server 📦
One suggested way to use the verifier is the time the server was booted, or the time it was last started (if restarting the server without a reboot results in lost buffers) ⏱️
The committed field in the results lets the client do more effective caching 🧁 If the server commits all WRITE requests to stable storage, then it should return committed set to FILE_SYNC4, no matter what the stable field said in the arguments. A server with an NVRAM accelerator might choose this policy ⚡ The client can use it to make its cache work better by tossing cached data that's already been committed on the server 🗑️
Some implementations return NFS4ERR_NOSPC instead of NFS4ERR_DQUOT when a user's quota is exceeded 💸 If the current filehandle is a directory, the server returns NFS4ERR_ISDIR. If the current filehandle is neither a regular file nor a directory, the server returns NFS4ERR_INVAL 🙃
Now the locking drama 🎭 If mandatory file locking is on for the file, and a matching record of the data to be written is read or write locked by an owner that isn't associated with the stateid, the server returns NFS4ERR_LOCKED. When that happens, the client must check whether the owner tied to the stateid used with the WRITE operation has a conflicting read lock that overlaps the region to be written 🔍 If the stateid's owner has no conflicting read lock, the client should try to get the right write byte-range lock via the LOCK operation before retrying the WRITE 🔐 Once the WRITE completes, the client should release that byte-range lock via LOCKU 🔓
If the stateid's owner did have a conflicting read lock, the client has no choice but to return an error to the application that tried the WRITE 😤 That's because, with that read lock in play, the server either (1) tried to temporarily upgrade this read lock to a write lock, or (2) has no upgrade capability. If the server tried to upgrade the read lock and failed, it's pointless for the client to retry the upgrade via the LOCK operation, since another client might also be trying to upgrade 🙄 If two clients get blocked trying to upgrade the same lock, they deadlock 💀 And if the server has no upgrade capability, trying a LOCK operation to upgrade is pointless too 🚫
16.37. Operation 39: RELEASE_LOCKOWNER - Release Lock-Owner State 🧹
16.37.1. SYNOPSIS
lock-owner -> ()16.37.2. ARGUMENT
struct RELEASE_LOCKOWNER4args {
lock_owner4 lock_owner;
};16.37.3. RESULT
struct RELEASE_LOCKOWNER4res {
nfsstat4 status;
};16.37.4. DESCRIPTION
This operation tells the server that the lock_owner isn't in use by the client anymore, and that future client requests won't reference this lock_owner 👋 That lets the server release cached state related to the specified lock_owner ✨ If file locks tied to the lock_owner are still held at the server, the error NFS4ERR_LOCKS_HELD gets returned and no further action is taken 🙅♀️
16.37.5. IMPLEMENTATION
The client can use this operation to ease the amount of server state being held 🧘♀️ When a RELEASE_LOCKOWNER is done, the info that can be released includes the specified lock-owner string, the seqid associated with the lock-owner, any saved reply for the lock-owner, and any lock stateids associated with that lock-owner 🗂️
Depending on how applications behave at the client, it might be important to use this operation, because the server has certain obligations about holding a reference to lock-owner-associated state for as long as an associated file is open 📂 So if the client knows for certain that the lock_owner won't be used to reference existing lock stateids associated with the lock-owner, or to create new ones, it should use RELEASE_LOCKOWNER 🧹
16.38. Operation 10044: ILLEGAL - Illegal Operation 🚫
16.38.1. SYNOPSIS
<null> -> ()16.38.2. ARGUMENT
void;16.38.3. RESULT
struct ILLEGAL4res {
nfsstat4 status;
};16.38.4. DESCRIPTION
This operation is a placeholder for encoding a result when the client sends an operation code within COMPOUND that isn't supported 🧩 See Section 15.2.4 for more details 📖
The status field of ILLEGAL4res MUST be set to NFS4ERR_OP_ILLEGAL 💅
16.38.5. IMPLEMENTATION
A client will probably never send an operation with code OP_ILLEGAL 🙈 But if it does, the response will be ILLEGAL4res, just like it would be with any other invalid operation code 🤓 Note that if the server gets an illegal operation code that isn't OP_ILLEGAL, and the server checks for legal operation codes during the XDR decode phase, then the ILLEGAL4res would not be returned 👀
17. NFSv4 Callback Procedures ✨
okay so the callback procedures are all defined in the sections below 💖 quick heads up: when we say "client" and "server" we mean NFS clients and servers, even though for one callback RPC the roles are totally flipped 🙃
17.1. Procedure 0: CB_NULL - No Operation
17.1.1. SYNOPSIS
<null>17.1.2. ARGUMENT
void;17.1.3. RESULT
void;17.1.4. DESCRIPTION
this is the classic NULL procedure: nothing in, nothing out 🫶 it doesn't do anything on its own, but the server uses CB_NULL to check that a working path exists for RPCs going from server back to client 🛤️
17.2. Procedure 1: CB_COMPOUND - COMPOUND Operations
17.2.1. SYNOPSIS
compoundargs -> compoundres17.2.2. ARGUMENT
enum nfs_cb_opnum4 {
OP_CB_GETATTR = 3,
OP_CB_RECALL = 4,
OP_CB_ILLEGAL = 10044
};
union nfs_cb_argop4 switch (unsigned argop) {
case OP_CB_GETATTR:
CB_GETATTR4args opcbgetattr;
case OP_CB_RECALL:
CB_RECALL4args opcbrecall;
case OP_CB_ILLEGAL: void;
};
struct CB_COMPOUND4args {
utf8str_cs tag;
uint32_t minorversion;
uint32_t callback_ident;
nfs_cb_argop4 argarray<>;
};17.2.3. RESULT
union nfs_cb_resop4 switch (unsigned resop) {
case OP_CB_GETATTR: CB_GETATTR4res opcbgetattr;
case OP_CB_RECALL: CB_RECALL4res opcbrecall;
case OP_CB_ILLEGAL: CB_ILLEGAL4res opcbillegal;
};
struct CB_COMPOUND4res {
nfsstat4 status;
utf8str_cs tag;
nfs_cb_resop4 resarray<>;
};17.2.4. DESCRIPTION
CB_COMPOUND is the bestie that bundles one or more callback procedures into a single RPC request 🎀 the main callback RPC program has two main procedures, CB_NULL and CB_COMPOUND, and every other operation just gets wrapped inside CB_COMPOUND 🎁
if the client is running low on resources and can't run some or all of the operations in a CB_COMPOUND, it returns NFS4ERR_RESOURCE for the operation where the exhaustion happened 😭 this only applies when everything before it in the sequence went through fine ✅
the status field in the CB_COMPOUND results has to match the status of the last operation that actually ran. so if something errored, the status is that same error value, the one returned for the operation that failed 💀
for the tag field, go check Section 15.2 🔍
the value of callback_ident is supplied by the client during SETCLIENTID. the server must use that client-supplied callback_ident during CB_COMPOUND so the client can properly identify the server 🪪
illegal operation codes get handled the exact same way they do for COMPOUND 🙈
17.2.5. IMPLEMENTATION
CB_COMPOUND bundles individual operations into one RPC request. the client works through each operation in turn. if an operation runs on the client and its status is NFS4_OK, the next operation in the CB_COMPOUND gets executed. this keeps going until there's nothing left to run or an operation returns a status other than NFS4_OK 🏁
18. NFSv4 Callback Operations 🦋
18.1. Operation 3: CB_GETATTR - Get Attributes
18.1.1. SYNOPSIS
fh, attr_request -> attrmask, attr_vals18.1.2. ARGUMENT
struct CB_GETATTR4args {
nfs_fh4 fh;
bitmap4 attr_request;
};18.1.3. RESULT
struct CB_GETATTR4resok {
fattr4 obj_attributes;
};
union CB_GETATTR4res switch (nfsstat4 status) {
case NFS4_OK:
CB_GETATTR4resok resok4;
default:
void;
};18.1.4. DESCRIPTION
CB_GETATTR is how the server asks for the current modified state of a file that's been delegated with OPEN_DELEGATE_WRITE 📂 only the size and change attributes are guaranteed to be serviced by the client 💅 for the full scoop on how the client and server work together with CB_GETATTR, check out Section 10.4.3 🧠
if the filehandle given isn't one the client holds an OPEN_DELEGATE_WRITE delegation for, you get an NFS4ERR_BADHANDLE error 🚫
18.1.5. IMPLEMENTATION
the client only returns attrmask bits and their attribute values for the change attribute, plus attributes it may change (time_modify and size) ✨
18.2. Operation 4: CB_RECALL - Recall an Open Delegation
18.2.1. SYNOPSIS
stateid, truncate, fh -> ()18.2.2. ARGUMENT
struct CB_RECALL4args {
stateid4 stateid;
bool truncate;
nfs_fh4 fh;
};18.2.3. RESULT
struct CB_RECALL4res {
nfsstat4 status;
};18.2.4. DESCRIPTION
CB_RECALL starts the process of recalling an open delegation so it gets handed back to the server 🔄
the truncate flag is an optimization for a file that's about to be truncated to zero. when it's set, the client doesn't have to push modified data for the file back to the server, since that data is irrelevant 🗑️
if the handle given isn't one the client holds an open delegation for, NFS4ERR_BADHANDLE gets returned 🙅♀️
if the stateid given doesn't correspond to an open delegation for the file in the filehandle, NFS4ERR_BAD_STATEID gets returned 😬
18.2.5. IMPLEMENTATION
the client should reply to the callback immediately 🏃♀️ replying doesn't complete the recall though, except when an error was returned. the recall isn't finished until the delegation is returned using DELEGRETURN 🎬
18.3. Operation 10044: CB_ILLEGAL - Illegal Callback Operation
18.3.1. SYNOPSIS
<null> -> ()18.3.2. ARGUMENT
void;18.3.3. RESULT
/*
* CB_ILLEGAL: Response for illegal operation numbers
*/
struct CB_ILLEGAL4res {
nfsstat4 status;
};18.3.4. DESCRIPTION
this one's a placeholder 🧩 it's for encoding a result when the client sends an operation code within COMPOUND that isn't supported. see Section 15.2.4 for more details 📖
the status field of CB_ILLEGAL4res MUST be set to NFS4ERR_OP_ILLEGAL 💀
18.3.5. IMPLEMENTATION
a server will probably never send an operation with code OP_CB_ILLEGAL 🤷♀️ but if it does, the response will be CB_ILLEGAL4res, just like it would be for any other invalid operation code.
⚠️ heads up: if the client gets an illegal operation code that is not OP_ILLEGAL, and it checks for legal operation codes during the XDR decode phase, then CB_ILLEGAL4res would not be returned. no cap, that one's sneaky 🙈
19. Security Considerations 🔒✨
okay bestie, let's talk security 🫶 back in the old NFS days the model was basically "the client IS the machine" (or at least its source IP address) 💀. The server trusted the NFS client to authenticate the end-user properly, and it only shared files with specific clients, identified by their source IP address. In that world the AUTH_SYS RPC security flavor just told the server who the end-user was, and when responses came back, the client checked that they came from the same IP address and port number the request went to. It's super easy to build and simple to deploy and use... but honestly? not a safe model at all 😭.
So NFSv4 says implementations have to support a security model with end-to-end authentication, where an end-user on a client mutually authenticates to a principal on an NFS server 🔐. The crypto schemes used have to avoid exposing passwords or keys in the clear on the network. Integrity and privacy of NFS requests and responses also deserve real attention. End-to-end mutual authentication, integrity, and privacy are all covered in Section 3 🦋.
When an NFSv4 security model is in use and a security principal or an NFSv4 name in user@dns_domain form needs to be translated to or from a local representation (see Section 5.9), that translation SHOULD be done in a secure way that keeps its integrity intact 🫧. If you're talking to a name service like the Lightweight Directory Access Protocol (LDAP) ([RFC4511]), that means using a security service that handles authentication and data integrity. Kerberos and Transport Layer Security (TLS) ([RFC5246]) are examples of those services 💜.
Plot twist 🙈: being REQUIRED to implement does not mean REQUIRED to use. AUTH_SYS can be used by NFSv4 clients and servers 🍓. But AUTH_SYS is merely an OPTIONAL security flavor in NFSv4, so interoperability via AUTH_SYS is not assured 💅.
For reasons like less admin overhead, better performance, and lower CPU usage, folks using NFSv4 might choose security mechanisms that skip integrity protection on each remote procedure call and response. That leaves the customer open to an attacker sitting between the NFS client and server who modifies the RPC request and/or the response 😬. Implementations are free to offer those weaker options, but there are two operations in particular that warrant overriding user choices ⚠️.
The first is SECINFO 🎀. It is recommended that the client issue the SECINFO call protected with a security flavor that has integrity protection, such as RPCSEC_GSS with a security triple that uses either rpc_gss_svc_integrity or rpc_gss_svc_privacy (rpc_gss_svc_privacy includes integrity protection). Without integrity protection wrapping SECINFO and therefore its results, an attacker in the middle could modify those results so the client picks a weaker algorithm from the set the server allows, which leaves the client and/or server open to further attacks 💀.
The second operation that SHOULD use integrity protection is any GETATTR for the fs_locations attribute 🚨. The attack has two steps. First, the attacker modifies the unprotected results of some operation to return NFS4ERR_MOVED. Second, when the client follows up with a GETATTR for fs_locations, the attacker changes the results to trick the client into migrating its traffic to a server the attacker controls. Yikes 😭.
Because SETCLIENTID/SETCLIENTID_CONFIRM are responsible for releasing client state, it is imperative that the principal used for these operations is checked against, and matches, the previous use of these operations 🔑. See Section 9.1.1 for more on this.
Unicode in UTF-8 is used for file component names (both directory and file components), as well as the owner and owner_group attributes 🌈. Other character sets may also be allowed for file component names. String processing (like Unicode normalization) raises security concerns for string comparison, so check out Sections 5.9 and 12 for more, and see [RFC6943] for related identifier comparison security considerations. File component names count as identifiers under the identifier comparison discussion in [RFC6943], because they're used to identify the objects that ACLs get applied to. See Section 6 for that part 👑.
20. IANA Considerations 🧚
This section uses terms that are defined in [RFC5226] 🌸.
20.1. Named Attribute Definitions
IANA has created a registry called the "NFSv4 Named Attribute Definitions Registry" for [RFC3530] and [RFC5661]. This section doesn't introduce any new changes, but it does recap the intent 📝.
The NFSv4 protocol lets a file be associated with zero or more named attributes. The namespace identifiers for these attributes are defined as string names. The protocol doesn't define how the namespace for these file attributes gets assigned. The IANA registry promotes interoperability wherever common interests exist 🤝. Application developers are totally allowed to define and use attributes as they need, but they're encouraged to register them with IANA 💖.
Registered named attributes are presumed to apply to all minor versions of NFSv4, including ones defined after the registration. If a named attribute should be limited with regard to the minor versions where it isn't meant to be used, the registry entry will clearly state those limits 🦋.
The registry is maintained using the Specification Required policy as defined in Section 4.1 of [RFC5226] 📋.
In theory, under the NFSv4 specification, the name of a named attribute can be up to 2^32 - 1 bytes long. In practice, though, NFSv4 clients and servers won't be able to handle a string that long 😅. IANA should reject any assignment request with a named attribute longer than 128 UTF-8 characters. To give the IESG flexibility in setting up the bases of assignment for Experimental Use and Standards Action, the prefixes EXPE and STDS are Reserved. The zero-length named attribute name is Reserved too 🚫.
The prefix PRIV is allocated for Private Use 🔐. A site that wants to use unregistered named attributes without risking a clash with an assignment in IANA's registry should put the PRIV prefix on all of its named attributes 🫶.
Some NFSv4 clients and servers have case-insensitive semantics, so the fifteen additional lowercase and mixed-case permutations of each of EXPE, PRIV, and STDS are Reserved (for example, expe, expE, exPe, and so on are Reserved) 💅. Similarly, IANA must not allow two assignments that would conflict if both named attributes were converted to a common case 🙅♀️.
The registry of named attributes is a list of assignments, and each assignment has three fields 🎀:
- A US-ASCII string name that is the actual name of the attribute. This name must be unique. The string name can be 1 to 128 UTF-8 characters long.
- A reference to the specification of the named attribute. The reference can take up to 256 bytes (or more, if IANA permits).
- The point of contact of the registrant. The point of contact can take up to 256 bytes (or more, if IANA permits).
20.1.1. Initial Registry
There is no initial registry 🌙.
20.1.2. Updating Registrations
The registrant is always permitted to update the point of contact field 🧡. Making any other change will require Expert Review or IESG Approval 🔒.
20.2. Updates to Existing IANA Registries
Because this document obsoletes RFC 3530, IANA has also done a couple of updates ✨:
- replaced all references to RFC 3530 in the Network Identifier (
r_netid) registry with references to this document. - replaced the reference to RFC 3530 in the
nfsregistration of the GSSAPI/Kerberos/SASL Service names registry with a reference to this document.
21. References 📚
21.1. Normative References 💅
okay bestie, these are the docs this spec literally can't function without 🫶 read them before you touch anything ✨
- [RFC20] Cerf, V., "ASCII format for network interchange", STD 80, RFC 20, October 1969,
<http://www.rfc-editor.org/info/rfc20>. - [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>. - [RFC2203] Eisler, M., Chiu, A., and L. Ling, "RPCSEC_GSS Protocol Specification", RFC 2203, September 1997,
<http://www.rfc-editor.org/info/rfc2203>. - [RFC2743] Linn, J., "Generic Security Service Application Program Interface Version 2, Update 1", RFC 2743, January 2000,
<http://www.rfc-editor.org/info/rfc2743>. - [RFC3490] Faltstrom, P., Hoffman, P., and A. Costello, "Internationalizing Domain Names in Applications (IDNA)", RFC 3490, March 2003,
<http://www.rfc-editor.org/info/rfc3490>. - [RFC3492] Costello, A., "Punycode: A Bootstring encoding of Unicode for Internationalized Domain Names in Applications (IDNA)", RFC 3492, March 2003,
<http://www.rfc-editor.org/info/rfc3492>. - [RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, November 2003,
<http://www.rfc-editor.org/info/rfc3629>. - [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 5226, May 2008,
<http://www.rfc-editor.org/info/rfc5226>. - [RFC5403] Eisler, M., "RPCSEC_GSS Version 2", RFC 5403, February 2009,
<http://www.rfc-editor.org/info/rfc5403>. - [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol Specification Version 2", RFC 5531, May 2009,
<http://www.rfc-editor.org/info/rfc5531>. - [RFC5665] Eisler, M., "IANA Considerations for Remote Procedure Call (RPC) Network Identifiers and Universal Address Formats", RFC 5665, January 2010,
<http://www.rfc-editor.org/info/rfc5665>. - [RFC5890] Klensin, J., "Internationalized Domain Names for Applications (IDNA): Definitions and Document Framework", RFC 5890, August 2010,
<http://www.rfc-editor.org/info/rfc5890>. - [RFC5891] Klensin, J., "Internationalized Domain Names in Applications (IDNA): Protocol", RFC 5891, August 2010,
<http://www.rfc-editor.org/info/rfc5891>. - [RFC6649] Hornquist Astrand, L. and T. Yu, "Deprecate DES, RC4-HMAC-EXP, and Other Weak Cryptographic Algorithms in Kerberos", BCP 179, RFC 6649, July 2012,
<http://www.rfc-editor.org/info/rfc6649>. - [RFC7531] Haynes, T., Ed., and D. Noveck, Ed., "Network File System (NFS) Version 4 External Data Representation Standard (XDR) Description", RFC 7531, March 2015,
<http://www.rfc-editor.org/info/rfc7531>. - [SPECIALCASING] The Unicode Consortium, "SpecialCasing-7.0.0.txt", Unicode Character Database, March 2014,
<http://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt>. - [UNICODE] The Unicode Consortium, "The Unicode Standard, Version 7.0.0", (Mountain View, CA: The Unicode Consortium, 2014 ISBN 978-1-936213-09-2), June 2014,
<http://www.unicode.org/versions/latest/>. - [openg_symlink] The Open Group, "Section 3.372 of Chapter 3 of Base Definitions of The Open Group Base Specifications Issue 7", IEEE Std 1003.1, 2013 Edition (HTML Version), ISBN 1937218287, April 2013,
<http://www.opengroup.org/>.
21.2. Informative References 🌈
these ones are the background reading 📚 super helpful for the vibes and history, but the spec doesn't depend on them the way it does on 21.1 🫶
- [Chet] Juszczak, C., "Improving the Performance and Correctness of an NFS Server", USENIX Conference Proceedings, June 1990.
- [Floyd] Floyd, S. and V. Jacobson, "The Synchronization of Periodic Routing Messages", IEEE/ACM Transactions on Networking 2(2), pp. 122-136, April 1994.
- [IESG_ERRATA] IESG, "IESG Processing of RFC Errata for the IETF Stream", July 2008.
- [MS-SMB] Microsoft Corporation, "Server Message Block (SMB) Protocol Specification", MS-SMB 43.0, May 2014.
- [P1003.1e] Institute of Electrical and Electronics Engineers, Inc., "IEEE Draft P1003.1e", 1997.
- [RFC1094] Nowicki, B., "NFS: Network File System Protocol specification", RFC 1094, March 1989,
<http://www.rfc-editor.org/info/rfc1094>. - [RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS Version 3 Protocol Specification", RFC 1813, June 1995,
<http://www.rfc-editor.org/info/rfc1813>. - [RFC1833] Srinivasan, R., "Binding Protocols for ONC RPC Version 2", RFC 1833, August 1995,
<http://www.rfc-editor.org/info/rfc1833>. - [RFC2054] Callaghan, B., "WebNFS Client Specification", RFC 2054, October 1996,
<http://www.rfc-editor.org/info/rfc2054>. - [RFC2055] Callaghan, B., "WebNFS Server Specification", RFC 2055, October 1996,
<http://www.rfc-editor.org/info/rfc2055>. - [RFC2224] Callaghan, B., "NFS URL Scheme", RFC 2224, October 1997,
<http://www.rfc-editor.org/info/rfc2224>. - [RFC2623] Eisler, M., "NFS Version 2 and Version 3 Security Issues and the NFS Protocol's Use of RPCSEC_GSS and Kerberos V5", RFC 2623, June 1999,
<http://www.rfc-editor.org/info/rfc2623>. - [RFC2624] Shepler, S., "NFS Version 4 Design Considerations", RFC 2624, June 1999,
<http://www.rfc-editor.org/info/rfc2624>. - [RFC2755] Chiu, A., Eisler, M., and B. Callaghan, "Security Negotiation for WebNFS", RFC 2755, January 2000,
<http://www.rfc-editor.org/info/rfc2755>. - [RFC3010] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame, C., Eisler, M., and D. Noveck, "NFS version 4 Protocol", RFC 3010, December 2000,
<http://www.rfc-editor.org/info/rfc3010>. - [RFC3232] Reynolds, J., Ed., "Assigned Numbers: RFC 1700 is Replaced by an On-line Database", RFC 3232, January 2002,
<http://www.rfc-editor.org/info/rfc3232>. - [RFC3530] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame, C., Eisler, M., and D. Noveck, "Network File System (NFS) version 4 Protocol", RFC 3530, April 2003,
<http://www.rfc-editor.org/info/rfc3530>. - [RFC4121] Zhu, L., Jaganathan, K., and S. Hartman, "The Kerberos Version 5 Generic Security Service Application Program Interface (GSS-API) Mechanism: Version 2", RFC 4121, July 2005,
<http://www.rfc-editor.org/info/rfc4121>. - [RFC4178] Zhu, L., Leach, P., Jaganathan, K., and W. Ingersoll, "The Simple and Protected Generic Security Service Application Program Interface (GSS-API) Negotiation Mechanism", RFC 4178, October 2005,
<http://www.rfc-editor.org/info/rfc4178>. - [RFC4506] Eisler, M., Ed., "XDR: External Data Representation Standard", STD 67, RFC 4506, May 2006,
<http://www.rfc-editor.org/info/rfc4506>. - [RFC4511] Sermersheim, J., Ed., "Lightweight Directory Access Protocol (LDAP): The Protocol", RFC 4511, June 2006,
<http://www.rfc-editor.org/info/rfc4511>. - [RFC5246] Dierks, T. and E. Rescorla, "The Transport Layer Security (TLS) Protocol Version 1.2", RFC 5246, August 2008,
<http://www.rfc-editor.org/info/rfc5246>. - [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "Network File System (NFS) Version 4 Minor Version 1 Protocol", RFC 5661, January 2010,
<http://www.rfc-editor.org/info/rfc5661>. - [RFC6365] Hoffman, P. and J. Klensin, "Terminology Used in Internationalization in the IETF", BCP 166, RFC 6365, September 2011,
<http://www.rfc-editor.org/info/rfc6365>. - [RFC6943] Thaler, D., Ed., "Issues in Identifier Comparison for Security Purposes", RFC 6943, May 2013,
<http://www.rfc-editor.org/info/rfc6943>. - [fcntl] The Open Group, "Section 'fcntl()' of System Interfaces of The Open Group Base Specifications Issue 7", IEEE Std 1003.1, 2013 Edition (HTML Version), ISBN 1937218287, April 2013,
<http://www.opengroup.org/>. - [fsync] The Open Group, "Section 'fsync()' of System Interfaces of The Open Group Base Specifications Issue 7", IEEE Std 1003.1, 2013 Edition (HTML Version), ISBN 1937218287, April 2013,
<http://www.opengroup.org/>. - [getpwnam] The Open Group, "Section 'getpwnam()' of System Interfaces of The Open Group Base Specifications Issue 7", IEEE Std 1003.1, 2013 Edition (HTML Version), ISBN 1937218287, April 2013,
<http://www.opengroup.org/>. - [read_api] The Open Group, "Section 'read()' of System Interfaces of The Open Group Base Specifications Issue 7", IEEE Std 1003.1, 2013 Edition (HTML Version), ISBN 1937218287, April 2013,
<http://www.opengroup.org/>. - [readdir_api] The Open Group, "Section 'readdir()' of System Interfaces of The Open Group Base Specifications Issue 7", IEEE Std 1003.1, 2013 Edition (HTML Version), ISBN 1937218287, April 2013,
<http://www.opengroup.org/>. - [stat] The Open Group, "Section 'stat()' of System Interfaces of The Open Group Base Specifications Issue 7", IEEE Std 1003.1, 2013 Edition (HTML Version), ISBN 1937218287, April 2013,
<http://www.opengroup.org/>. - [unlink] The Open Group, "Section 'unlink()' of System Interfaces of The Open Group Base Specifications Issue 7", IEEE Std 1003.1, 2013 Edition (HTML Version), ISBN 1937218287, April 2013,
<http://www.opengroup.org/>. - [write_api] The Open Group, "Section 'write()' of System Interfaces of The Open Group Base Specifications Issue 7", IEEE Std 1003.1, 2013 Edition (HTML Version), ISBN 1937218287, April 2013,
<http://www.opengroup.org/>. - [xnfs] The Open Group, "Protocols for Interworking: XNFS, Version 3W, ISBN 1-85912-184-5", February 1998.
Acknowledgments ✨
okay bestie, we're at the thank-you section 🥹🎀 and honestly it's giving "the squad that made this possible" energy. A bis is certainly built on the shoulders of the first attempt. Spencer Shepler, Brent Callaghan, David Robinson, Robert Thurlow, Carl Beame, Mike Eisler, and David Noveck are responsible for a great deal of the effort in this work. 🫶
Tom Haynes would like to thank NetApp, Inc. for its funding of his time on this project. 💼
Rob Thurlow clarified how a client should contact a new server if a migration has occurred. 🦋
David Black, Nico Williams, Mike Eisler, Trond Myklebust, James Lentini, and Mike Kupfer read many earlier draft versions of Section 12 and contributed numerous useful suggestions, without which the necessary revision of that section for this document would not have been possible. Iconic behaviour from the whole group chat 👑
Peter Staubach read almost all of the earlier draft versions of Section 12, leading to the published result, and his numerous comments were always useful and contributed substantially to improving the quality of the final result. 💅
Peter Saint-Andre was gracious enough to read the most recent draft version of Section 12 and provided some key insight as to the concerns of the Internationalization community. 🌈
James Lentini graciously read the rewrite of Section 8, and his comments were vital in improving the quality of that effort. 🌸
Rob Thurlow, Sorin Faibish, James Lentini, Bruce Fields, and Trond Myklebust were faithful attendants of the biweekly triage meeting and accepted many an action item. 🍓
Bruce Fields was a good sounding board for both the third edge condition and courtesy locks in general. He was also the leading advocate of stamping out backport issues from [RFC5661]. He really did the most, ngl 🙈
Marcel Telka was a champion of straightening out the difference between a lock-owner and an open-owner. He has also been diligent in reviewing the final document. 🧚
Benjamin Kaduk reminded us that DES is dead, and Nico Williams helped us close the lid on the coffin. 💀⚰️
Elwyn Davies provided a very thorough and engaging Gen-ART review; thanks! 💖
Authors' Addresses ✨
Thomas Haynes (editor) Primary Data, Inc. 4300 El Camino Real Ste 100 Los Altos, CA 94022 United States
Phone: +1 408 215 1519 EMail: thomas.haynes@primarydata.com
David Noveck (editor) Dell 300 Innovative Way Nashua, NH 03062 United States
Phone: +1 781 572 8038 EMail: dave_noveck@dell.com
that's the whole thing, bestie 💜 the editors are the ones who kept the whole document glued together, so they get their own little spotlight at the end 👑
📦 where this page comes from
- ietf: rfc7530 2015-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms