RFC 7931

NFSv4.0 Migration: Specification Update

📅 published July 2016, from ietf

Internet Engineering Task Force (IETF) 🎀

Okay so bestie, this is RFC 7931 ✨, titled NFSv4.0 Migration: Specification Update 💅. Here's the title block tea:

Authors: D. Noveck (editor), P. Shivam, C. Lever, and B. Baker 🫶
Affiliations: HPE, ORACLE 🌸
Category: Standards Track 👑
ISSN: 2070-1721 🦋
Date: July 2016 🍓
Updates: RFC 7530 (the NFSv4.0 spec) 💜

Literally the whole point is that this doc updates RFC 7530 🔥, so when you're learning NFSv4.0, you have to read this one alongside it. Main character energy, no cap 🧚

Abstract

okay so the migration feature in NFSv4 lets a single file system's responsibility get handed off from one server to another 🔁 without clients noticing anything's wrong ✨. But real implementers ran into problems with how NFSv4.0 described this, lowkey messy 😭. This doc calls out exactly which problem areas are the issue and gives revised spec text that updates RFC 7530 💅🌈.

Status of This Memo

this is an Internet Standards Track document 👑. It's a product of the IETF 🌸 and represents the consensus of the IETF community 🫶. It got public review and was approved for publication by the IESG 🦋. For more on Internet Standards, check out Section 2 of RFC 7841 📚.

info about the current status, errata, and how to send feedback lives at http://www.rfc-editor.org/info/rfc7931 🔗 (I'm not linking it, just telling you where it is, periodt 💅).

Copyright Notice

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, at http://trustee.ietf.org/license-info, as in effect on the publication date 🎀. Read those carefully, bestie, because they explain your rights and restrictions 🙈. Code components pulled out of this doc must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as the Simplified BSD License describes 💜.

Here's the full table of contents, exactly as the original lays it out 📋:

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
2.  Conventions . . . . . . . . . . . . . . . . . . . . . . . . .   3
3.  Definitions . . . . . . . . . . . . . . . . . . . . . . . . .   3
  3.1.  Terminology . . . . . . . . . . . . . . . . . . . . . . .   3
  3.2.  Data Type Definitions . . . . . . . . . . . . . . . . . .   5
4.  Background  . . . . . . . . . . . . . . . . . . . . . . . . .   5
5.  Client Identity Definition  . . . . . . . . . . . . . . . . .   7
  5.1.  Differences from Replaced Sections  . . . . . . . . . . .   7
  5.2.  Client Identity Data Items  . . . . . . . . . . . . . . .   8
    5.2.1.  Client Identity Structure . . . . . . . . . . . . . .   9
    5.2.2.  Client Identity Shorthand . . . . . . . . . . . . . .  11
  5.3.  Server Release of Client ID . . . . . . . . . . . . . . .  13
  5.4.  Client ID String Approaches . . . . . . . . . . . . . . .  14
  5.5.  Non-uniform Client ID String Approach . . . . . . . . . .  16
  5.6.  Uniform Client ID String Approach . . . . . . . . . . . .  16
  5.7.  Mixing Client ID String Approaches  . . . . . . . . . . .  18
  5.8.  Trunking Determination when Using Uniform Client ID
        Strings . . . . . . . . . . . . . . . . . . . . . . . . .  20
  5.9.  Client ID String Construction Details . . . . . . . . . .  26
6.  Locking and Multi-Server Namespace  . . . . . . . . . . . . .  28
  6.1.  Lock State and File System Transitions  . . . . . . . . .  28
    6.1.1.  Migration and State . . . . . . . . . . . . . . . . .  29
      6.1.1.1.  Migration and Client IDs  . . . . . . . . . . . .  31
      6.1.1.2.  Migration and State Owner Information . . . . . .  32
    6.1.2.  Replication and State . . . . . . . . . . . . . . . .  36
    6.1.3.  Notification of Migrated Lease  . . . . . . . . . . .  36
    6.1.4.  Migration and the lease_time Attribute  . . . . . . .  39
7.  Server Implementation Considerations  . . . . . . . . . . . .  39
  7.1.  Relation of Locking State Transfer to Other Aspects of
        File System Motion  . . . . . . . . . . . . . . . . . . .  39
  7.2.  Preventing Locking State Modification during Transfer . .  41
8.  Additional Changes  . . . . . . . . . . . . . . . . . . . . .  44
  8.1.  Summary of Additional Changes from Previous Documents . .  45
  8.2.  NFS4ERR_CLID_INUSE Definition . . . . . . . . . . . . . .  45
  8.3.  NFS4ERR_DELAY Return from RELEASE_LOCKOWNER . . . . . . .  45
  8.4.  Operation 35: SETCLIENTID -- Negotiate Client ID  . . . .  46
  8.5.  Security Considerations for Inter-server Information
        Transfer  . . . . . . . . . . . . . . . . . . . . . . . .  51
  8.6.  Security Considerations Revision  . . . . . . . . . . . .  51
9.  Security Considerations . . . . . . . . . . . . . . . . . . .  52
10. References  . . . . . . . . . . . . . . . . . . . . . . . . .  52
  10.1.  Normative References . . . . . . . . . . . . . . . . . .  52
  10.2.  Informative References . . . . . . . . . . . . . . . . .  52
Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  53
Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  54

1. Introduction ✨

okay so this Standards Track doc corrects the existing definitive spec of the NFSv4.0 protocol, which is described in [RFC7530] 💅. Because of that, you should totally factor this doc in when you're learning NFSv4.0, especially if you care about stuff related to:

File system migration 🔁, especially when it involves transparent state migration 🦋.
Building and reading the nfs_client_id4 structure 🧚, and especially the rules for the id string inside it, which we'll call a "client ID string" below 🎀.

2. Conventions 🫶

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 💜.

3. Definitions 📚

3.1. Terminology 🌸

these definitions are here so you have the right context 💖. This section is derived from Section 1.5 of [RFC7530], but it's been tweaked for what this doc needs 🦋.

Boot Instance Id: a boot instance id is an identifier, like a boot time, that lets two different instances of the same client be reliably told apart 🔍. It's opaque to the server, and it's often used as the verifier field in the nfs_client_id4 structure, which is how the client identifies itself to the server 🫶.

Client: a client is an entity that accesses the NFS server's resources 💅. It can be an application that has the logic to reach the NFS server directly 🧚. It can also be the traditional operating system client that provides remote file system services for a bunch of applications 🌈.

With respect to byte-range locking, the client is also the entity that keeps a set of locks on behalf of one or more applications 🔒. This client is responsible for crash or failure recovery for the locks it manages 😭.

Note that multiple clients may share the same transport and connection, and multiple clients may exist on the same network node 👯.

Client ID: a client ID is a 64-bit quantity (in the form of a clientid4) used as a unique, shorthand reference to a particular client instance 🎀. That instance is identified by a client-supplied verifier (in the form of a boot instance id) and a client ID string 💜. The server is responsible for supplying the client ID ✨.

File System: a file system is the collection of objects on a server that share the same fsid attribute (see Section 5.8.1.9 of [RFC7530]) 🗂️.

Grace Period: a grace period is an interval of time during which the server will only grant locking requests that reclaim existing locks, not ones that create new locks ⏳. It gives clients a chance to rebuild their locking state after a potentially disruptive event 🦋. The grace period may be general, to help deal with a server reboot, or it may be specific to a file system, to deal with file system migration when transparent state migration isn't provided 🔁.

Lease: a lease is an interval of time that the server defines, during which the client is irrevocably granted a lock 🔐. When a lease period ends, the lock may be revoked if the lease hasn't been extended 😬. The lock MUST be revoked if a conflicting lock has been granted after the lease interval 💀.

All leases granted by a server have the same fixed duration 🕰️. Note that the fixed duration was chosen to save a server from the expense of tracking state about variable-length leases across server failures 🧠.

Lock: the term "lock" refers to record (byte-range) locks as well as share reservations, unless it's specifically stated otherwise 🔒.

Lock-Owner: each byte-range lock is associated with a specific lock-owner and an open-owner 🎟️. The lock-owner consists of a client ID and an opaque owner string 🧩. The client presents this to the server to establish ownership of the byte-range lock as needed 🫶.

Open-Owner: each open file is associated with a specific open-owner, which consists of a client ID and an opaque owner string 📂. The client presents this to the server to establish ownership of the open as needed 🙌.

Server: a server is an entity responsible for coordinating client access to a set of file systems 🖥️.

Stateid: a stateid is a 128-bit quantity returned by a server that uniquely identifies the open and locking states the server provides for a specific open-owner, or a lock-owner/open-owner pair, for a specific file and type of lock 🆔.

Trunking: a situation in which multiple physical addresses are connected to the same logical server 🌐.

Verifier: a verifier is a quantity, in the form of a verifier4, that lets one party in an interaction know about a reinitialization or some other significant change to the state of the other party 🔄. In [RFC7530], this term most often refers to the verifier field of an nfs_client_id4, which holds a boot instance id so the server can tell when a client has rebooted 🔁. When that happens, the server has to get rid of the locking state tied to the previous instance of that client 🧹.

3.2. Data Type Definitions 📋

this section has a table showing where the data types referred to in this document are defined 🗺️:

        +-----------------+--------------------------------+
        | Item            | Section                        |
        +-----------------+--------------------------------+
        | cb_client4      | Section 2.2.11 in [RFC7530]    |
        | clientaddr4     | Section 2.2.10 in [RFC7530]    |
        | clientid4       | Section 2.1 in [RFC7530]       |
        | lock_owner4     | Section 2.2.14 in [RFC7530]    |
        | nfs_client_id4  | Section 5.2.1 (this document)  |
        | open_owner4     | Section 2.2.13 in [RFC7530]    |
        | verifier4       | Section 2.1 in [RFC7530]       |
        +-----------------+--------------------------------+

okay bestie that's part 1 of 13 🎀💖 we're just getting started, so stay tuned for Section 4 (Background) next 🧚✨ periodt 💅

4. Background 🎀

Okay so bestie, implementing transparent state migration exposed a bunch of problems with how [RFC7530] and its predecessors described this feature. The symptoms were:

After migration of a file system, a reboot of the associated client got mishandled, and the state tied to that rebooting client wasn't promptly freed. 💀
A given server could end up with multiple leases sharing the same nfs_client_id4 (made of the id and verifier fields), even though the protocol clearly assumes there's only ever one. 🙈
Clients had to deal with way too much mess, since one client's locking state with a server could get split across multiple leases, each with its own clientid4. 😭

An analysis of these symptoms leads to the conclusion that the existing specifications messed up. 🥲 They assume that locking state, including both state ids and clientid4s, should be transferred as part of transparent state migration. The troubling symptoms come from never describing how migrating state is supposed to fit into the existing client definition structures on the destination server.

The server needs to merge stateids that belong to a common client boot instance, and that's a seriously tricky problem. ✨ The issue is that the usual client practice for presenting unique strings that identify a client makes it basically impossible for the client to tell whether two stateids, originally generated on different servers, belong to the same client. This practice is allowed and endorsed by the existing NFSv4.0 specification [RFC7530].

Here's the plot twist: when people prototyped clients using a different approach, they found that some servers just don't work well with these new clients. 🙃 Because one client pattern got adopted everywhere, some servers can't interoperate with alternate client patterns. So we need to be really careful untangling these compatibility issues. 🧶

This document updates the existing NFSv4.0 specification [RFC7530] as follows:

It makes clear that NFSv4.0 supports multiple approaches to constructing client ID strings, including the ones previously endorsed by NFSV4.0 specifications and the ones being widely deployed right now. 🌈
It explains how clients can effectively use client ID strings that are presented to multiple servers.
It addresses the potential compatibility issues that might come up for clients adopting a previously non-favored client ID string construction approach, including servers that have problems with the new approach.
It gives some guidance on the factors that might govern a client's choice of client ID string construction approach, and it recommends that clients construct client ID strings in a way that supports lease merger if they intend to support transparent state migration. 🦋
It specifies how state is transparently migrated, including how state arriving at a new server as part of migration gets merged into existing leases for clients connected to the target server.
It makes further clarifications and corrections wherever the specification text doesn't properly account for the issues raised by state migration, or where the existing text turned out to be insufficiently clear. This includes a revised definition of the SETCLIENTID operation in Section 8.4, which replaces Section 16.33 in [RFC7530]. 💅

For a more complete explanation of the choices made in addressing these issues, see [INFO-MIGR].

5. Client Identity Definition 🪪

This section replaces Sections 9.1.1 and 9.1.2 in [RFC7530]. The replaced sections are named "Client ID" and "Server Release of Client ID", respectively.

It supersedes the replaced sections. ✨

5.1. Differences from Replaced Sections

This section is way bigger than the sections it replaces, because this area needs a lot more care and a much more detailed description. 🔍 The main changes and additions are:

It fixes inconsistencies about whether the client IP address plays a role in constructing client ID strings (or not).
It clearly addresses the need to keep a non-volatile record, across reboots, of client ID strings and of any changeable values used to construct them. 💾
It gives a more complete description of the circumstances that make a clientid4 invalid, plus the right recovery actions. 🛠️
It presents two approaches to client ID string construction as valid alternatives, named "uniform" and "non-uniform", and gives some implementation guidance to help implementers choose between them. 👑
It adds a discussion of the issues clients face when interacting with servers whose behavior isn't consistent with uniform client ID strings. 🫶
It adds a description of how a client can use server behavior to figure out when multiple server IP addresses correspond to the same server. 🕵️‍♀️

5.2. Client Identity Data Items

The NFSv4 protocol has a bunch of protocol entities for identifying clients and client-based entities for locking purposes:

The nfs_client_id4 structure, which uniquely identifies a specific client boot instance. That identification gets presented to the server through a SETCLIENTID operation. The SETCLIENTID operation is described in Section 8.4, which modifies a description in Section 16.33 of [RFC7530]. 🎀
The clientid4, which the server returns when a SETCLIENTID operation completes successfully. The client uses this id to identify itself in later locking-related operations. A clientid4 is tied to a particular lease, where a client instance holds state on a server instance, and it can become invalid because of a client reboot, a server reboot, or other circumstances. 🔄
Opaque arrays, which are used together with the clientid4 to mark within-client entities (like processes) as the owners of opens (open-owners) and the owners of byte-range locks (lock-owners). 🧚

5.2.1. Client Identity Structure

The whole client identification infrastructure is built on the following data structure, which also shows up in Section 9.1.1 of [RFC7530]:

struct nfs_client_id4 {
        verifier4       verifier;
        opaque          id<NFS4_OPAQUE_LIMIT>;
};

The nfs_client_id4 structure uniquely defines a particular client boot instance, like this:

The id field is a variable-length string that uniquely identifies a specific client. Even though it's described here as a string and is often called a "client string", the protocol actually defines it as opaque data. 🫠 So anyone receiving one of these ids shouldn't assume it'll be UTF-8 encoded. Servers MUST NOT reject an nfs_client_id4 simply because the id string doesn't follow the rules of UTF-8 encoding. 🙅‍♀️

The encoding and decoding processes for this field (for example, using network byte order) need to produce the same internal representation no matter the endianness of the originating and receiving machines. 🌸

The verifier field contains a client boot instance identifier that the server uses to detect client reboots. The server cancels the client's leased state only if the boot instance differs from the one it previously recorded for that client (as identified by the id field). 🧨 That cancellation happens once the server receives confirmation of the new nfs_clientd4 via SETCLIENTID_CONFIRM. The SETCLIENTID_CONFIRM operation is described in Section 16.34 of [RFC7530].

To prevent anyone from maliciously destroying the locking state associated with a client, the server MUST NOT cancel a client's leased state if the principal that established the state for a given id string is not the same as the principal issuing the SETCLIENTID. 🛡️ This one is dead serious, bestie. 🚨

There are several things to think about when the client generates its id string:

The string should be unique, so multiple clients don't present the same string. The fallout from two clients presenting the same string ranges from one client getting an error all the way to one client having its leased state abruptly and unexpectedly canceled. 😱
The string should be picked so that later incarnations (like reboots) of the same client present the same string. The implementer is cautioned against an approach that requires the string to be recorded in a local file, because that rules out using the implementation in an environment with no local disk, where all file access goes through an NFSv4 server. 💀
The string MAY be different for each server network address the client accesses, rather than common to all server network addresses. 🦋

The considerations that might push a client toward using different strings for different server network addresses are explained in Section 5.4.

The algorithm for generating the string should not assume that the client's network addresses will stay the same for any set period of time. Even while the client keeps running in its current incarnation, things can change between client incarnations. 🌬️

If the client ID string changes because of network address changes, successive SETCLIENTID operations from the same client will look like they're coming from different clients. That gets in the way of using the nfs_client_id4 verifier field to cancel state associated with previous boot instances of the same client. 🙃

It's even worse if the client address is the only client-based information in the client ID string. In that case, there's a real risk that after the client gives up the network address, another client using the same algorithm would generate a conflicting id string. That would likely cause an inappropriate loss of locking state. 😭 See Section 5.9 for detailed guidance on client ID string construction. 📖

5.2.2. Client Identity Shorthand ✨

Okay so once a SETCLIENTID and SETCLIENTID_CONFIRM sequence has successfully completed, the client gets to use a shorthand identifier, clientid4, instead of the longer, clunkier nfs_client_id4 structure 💅. This shorthand (a client ID) is assigned by the server, and it should be picked so it won't clash with a client ID the same server handed out before and, as far as is practicable, with ones from other servers too 🫶. This still holds across server restarts or reboots 🔄.

Establishing the client ID from a new incarnation of the client also immediately breaks any leased state that a previous incarnation of the client might have had on the server 💥. The new client doesn't have to sit around waiting for those leases to expire, which is iconic 👑. Breaking that lease state means the server removes all locks, share reservations, and delegation states that weren't requested using the CLAIM_DELEGATE_PREV claim type for a client with the same identity 🧹. For more on delegation state recovery, see Section 10.2.1 of [RFC7530] 📚.

Quick aside 🤭: SETCLIENTID and SETCLIENTID_CONFIRM also have a secondary job. They set up the info the server needs to make callbacks to the client so delegations can work 📞. The client can change that info through SETCLIENTID and SETCLIENTID_CONFIRM within the same incarnation without the client's leased state getting removed 🙌.

Here's the tea 🍵: distinct servers MAY assign clientid4s independently, and they'll generally do exactly that 🌍. So a client has to be ready to handle several instances of the same clientid4 value showing up on different IP addresses, each one meaning a separate entity 🫠. When trunking of server IP addresses isn't a consideration, a client should keep track of <IP-address, clientid4> pairs so every pair stays distinct 🗂️. If trunking is possible, see Section 5.4 for how to handle that 🧩.

Owners of opens and owners of byte-range locks are totally separate entities, even if the same opaque arrays are used to name each one 🙅‍♀️. The protocol tells open-owners (open_owner4 structures) and lock-owners (lock_owner4 structures) apart 💜.

Both kinds of owner are made of a clientid4 plus an opaque owner string 🎀. For each client, the distinct owner values used with it make up the set of known owners of that type for that client 📋.

Each open belongs to one specific open-owner 🔐. Each byte-range lock belongs to a lock-owner and an open-owner, and that open-owner is the one tied to the open file the LOCK operation was done under 🔗.

If a clientid4 gets presented and it isn't valid, the server rejects the request with an error based on why it's invalid 🚫. NFS4ERR_ADMIN_REVOKED shows up when the invalidation came from administrative action 🛑. When the clientid4 is unrecognizable, NFS4ERR_STALE_CLIENTID or NFS4ERR_EXPIRED may be returned 🥲. An unrecognizable clientid4 can happen for a bunch of reasons 🕵️‍♀️:

A server reboot causing loss of the server's knowledge of the client. (Always returns NFS4ERR_STALE_CLIENTID.)
Client error sending an incorrect clientid4, or a valid clientid4 to the wrong server. (Either error may come back.)
Loss of lease state due to lease expiration. (Always returns NFS4ERR_EXPIRED.)
Client or server error causing the server to believe that the client has rebooted (i.e., receiving a SETCLIENTID with an nfs_client_id4 that has a matching id string and a non-matching boot instance id as the verifier). (Either error may come back.)
Migration of all state under the associated lease causes its non-existence to be recognized on the source server. (Always returns NFS4ERR_STALE_CLIENTID.)
Merger of state under the associated lease with another lease under a different client ID causes the clientid4 serving as the source of the merge to cease being recognized on its server. (Always returns NFS4ERR_STALE_CLIENTID.)

So if there's a server reboot, lease state gets lost to expiration, or a clientid4 gets administratively revoked, the client must get a new clientid4 through the SETCLIENTID operation 🔁. After that it moves on to any other recovery the server reboot case needs (see Section 9.6.2 in [RFC7530]) 🛠️. If server or client error makes a clientid4 unusable, using SETCLIENTID to set up a new lease is desirable too ✨.

Plot twist 🎬: when the server stops knowing about a clientid4 because of migration, the recovery procedures are different 🔀. See Section 6.1.1 for the details 📖. And if you're ever unsure which kind of handling applies, here's the trick 🧠: in reboot-like cases, the clientid4 and all its associated stateids stop existing. In migration-related cases, only the clientid4 stops existing, and the stateids are still valid 💫.

The client must also use the SETCLIENTID operation when it gets an NFS4ERR_STALE_STATEID error on a stateid derived from its current clientid4 🚨. That error means something like a server reboot has invalidated the existing clientid4 and its stateids (see Section 9.1.5 in [RFC7530] for details) 😵‍💫.

For the full spec on these ops, check out SETCLIENTID (in Section 8.4) and SETCLIENTID_CONFIRM (in Section 16.34 of [RFC7530]) 📜💖.

5.3. Server Release of Client ID 🗑️

If the server figures out the client holds no state tied to its clientid4, the server may choose to release that clientid4 🫧. It might do this for an inactive client so resources aren't wasted on clients that only pop in now and then 😴. If the client contacts the server after that release, the server must make sure the client gets the right error so it knows to use the SETCLIENTID/SETCLIENTID_CONFIRM sequence to establish a new identity 🪪. Honestly the server has to be super hesitant about releasing a client ID 😬, because the client then has to do the same recovery work it would've done if the server had crashed and restarted 💀. Typically, a server wouldn't release a client ID unless there had been no activity from that client for many minutes ⏰.

If the id string in a SETCLIENTID request is built properly, and the client makes sure to use the same principal for every SETCLIENTID, then, barring an active denial-of-service attack, NFS4ERR_CLID_INUSE should never come back 🙏.

But wait 😭: client bugs, server bugs, or a deliberate change of the principal owner of the id string can cause NFS4ERR_CLID_INUSE in rare cases 💔. One example is a client switching security flavors where the new flavor has no mapping to the previous owner 🔄.

When there's an apparent change of principal, and the server gets a SETCLIENTID for a client ID string whose clientid4 currently has no state, or has state but the lease has expired, the server MUST allow the SETCLIENTID instead of returning NFS4ERR_CLID_INUSE 🫶🌈. Then the server MUST confirm the new client ID if it's followed by the appropriate SETCLIENTID_CONFIRM ✅✨.

5.4. Client ID String Approaches

okay so the nfs_client_id4 string has been a total troublemaker 😭 like, it keeps coming up again and again 🙈 The client gets a choice of:

Presenting the same id string to multiple server addresses. This is referred to as the "uniform client ID string approach" and is discussed in Section 5.6.
Presenting different id strings to multiple server addresses. This is referred to as the "non-uniform client ID string approach" and is discussed in Section 5.5.

ngl, implementation considerations like compatibility with existing servers may make it desirable for a client to use both approaches, based on configuration information such as mount options 🎀 This issue gets its own spotlight in Section 5.7 ✨

Building the client ID string is a hard problem 💀 and it's only hard because of how the NFS protocols have evolved over time. It helps to look at two points in that evolution 🦋

NFSv3 was stateless 🌸 so it had no need to identify the state shared by a particular client-server pair (see [RFC1813]). That means nobody had to figure out whether a set of requests came from the same client, or whether two server IP addresses were connected to the same server. The user typed in the target server IP address when mounting the remote file system into their namespace, so there was no occasion to notice server trunking at all. Within a stateless protocol, the situation was symmetrical: the client had no server identity information, and the server had no client identity information. 🫶
NFSv4.1 is stateful and has full support for client and server identity determination (see [RFC5661]) 👑 This lets the server tell when two requests come from the same client (they're on sessions sharing a clientid4), and it lets the client tell when two server IP addresses are connected to the same server. Section 2.10.5.1 of [RFC5661] explains how the client can assure itself that the connections go to the same logical server 💅

NFSv4.0 is unfortunately stuck halfway between those two 😭 It brought in new requirements, like the need to identify specific clients and client instances, but it didn't address server identity issues. The two client ID string approaches came out of attempts to keep up with those changing requirements as implementation went on. Features that weren't very substantial in early NFSv4.0 implementations became way more substantial as things moved along ⭐

If no one implements the fs_locations-related features (replication, referral, and migration), things look very similar to NFSv3 (see Section 8.1 and the subsections within Section 8.4 of [RFC7530] for discussion of these features). Locking state has been added, but there's no need to worry about accurate client and server identity determination. This is the situation that gave rise to the non-uniform client ID string approach 🍓
With replication and referrals in the mix, the client may have occasion to take advantage of knowing the server trunking information 🌈 Even more important, transparent state migration, which transfers state among servers, causes trouble for the non-uniform client ID string approach. The two different client ID strings sent to different IP addresses may wind up being processed by the same logical server, which just adds confusion 🙈
Client implementations usually provide NFSv4.1 by augmenting their existing NFSv4.0 implementation, rather than building two separate ones 🧚 So the more NFSv4.0 and NFSv4.1 can work alike, the less complex the clients are. This is a key reason why people implementing NFSv4.0 clients might prefer the uniform client string model, even if they've chosen not to provide fs_locations-related features in their NFSv4.0 client 💜

Both approaches have to deal with the asymmetry in client and server identity information. Each one tries to make the client's view and the server's view match up. In the process, each one runs into some mix of inelegant protocol features and/or implementation difficulties 😩 Which one to use is up to the client implementer, and the sections below try to give some useful guidance ✨

5.5. Non-uniform Client ID String Approach

okay the non-uniform client ID string approach is basically an attempt to handle all of this for NFSv4.0 client implementations in as NFSv3-like a way as possible 🎀

For a client using the non-uniform approach, all internal recording of clientid4 values has to include the server IP address, either explicitly or implicitly. That way you always have an <IP-address, clientid4> pair. Two such pairs from different servers are always distinct, even when the clientid4 values happen to be the same, which they may occasionally be 🦋 In this approach, that kind of equality is always treated as simple happenstance 🙈

Making the client ID string different on different server IP addresses means the server has no way to tie together information from the same client when that client accesses multiple server IP addresses 😭 As a result, the server treats a single client as multiple clients, each with a separate lease for each server network address. Since there's no way in the protocol for the client to determine whether two network addresses are connected to the same server, the resulting lack of knowledge is symmetrical. This can lead to simpler client implementations, where there's a single clientid4/lease per server network address 💅

Migration is a whole other story, especially transparent state migration with the non-uniform approach 💀 For example, migrating a lease can result in multiple leases for the same client accessing the same server addresses, which kind of wipes out a lot of the advantages of this approach. So client implementations that support migration with transparent state migration are likely to have trouble using the non-uniform client ID string approach, and they should not do so, except where it's necessary for compatibility with existing server implementations. (For details on arranging the use of multiple client ID string approaches, see Section 5.7 🌸)

5.6. Uniform Client ID String Approach

When the client ID string stays uniform, the server has what it needs to keep a single clientid4/lease for each distinct client 👑 The problem that has to be solved is the lack of explicit server identity information, which NFSv4.1 finally made available 🫶

When the same client ID string goes to multiple IP addresses, the client can figure out whether two IP addresses correspond to a single server by looking at the server's behavior 🔥 This is basically the inverse of the strategy used for the non-uniform approach, where different server IP addresses are told about different clients. That strategy exists just to stop a server from acting in ways that don't match there being one server per IP address, which is in line with the traditions of NFS 🧚 So, to compare:

In the non-uniform approach, servers are told about different clients because, if a server used accurate client identity information, two IP addresses on the same server would behave as if they were talking to the same client. That might be pretty disconcerting to a client that isn't expecting that behavior 🙈
In the uniform approach, servers are told that there's a single client, which is, after all, the truth ✨ Then, when the server uses this information, two IP addresses on the same server will behave as if they're talking to the same client. That difference in behavior lets the client infer the server IP address trunking configuration, even though NFSv4.0 doesn't explicitly provide this information 🦋

The approach given in the section below shows one example of how this might be done 🌈

The uniform client ID string approach means you have to be extra careful about how you define the boot instance id sent as the verifier field in an nfs_client_id4 🎀

In [RFC7530], the client is told to change the verifier field value when a reboot occurs. There's no explicit statement about the converse, though, so any requirement to keep the verifier field constant unless rebooting is only there by implication 🤔
Many existing clients change the boot instance id every time they destroy and recreate the data structure that tracks an <IP-address, clientid4> pair. That might happen if the last mount of a particular server is removed and then a fresh mount is created. Also, this might result in each <IP-address, clientid4> pair having its own boot instance id, independent of the others 😩
With the uniform client ID string approach, an nfs_client_id4 designates a globally known client instance. So the verifier field should change if and only if a new client instance is created, typically as a result of a reboot 💖

Clients using the uniform client ID string approach are therefore well advised to use a verifier established only once for each reboot, typically at reboot time 🫶

Here are the advantages of using the uniform client ID string approach for implementations 🌸

Clients can take advantage of server trunking (and clustering with single-server-equivalent semantics) to increase bandwidth or reliability 🚀
There are advantages in state management. For example, one never has a delegation under one clientid4 revoked because of a reference to the same file from the same client under a different clientid4 👑
The uniform client ID string approach lets the server do any necessary automatic lease merger in connection with transparent state migration, without requiring any client involvement. This consideration carries enough weight that it leads us to recommend the uniform client ID string approach for clients that support transparent state migration 💅

These implementation considerations might cause issues for client implementations 😬

This approach is considerably different from the non-uniform approach, which most client implementations have been following. Until there's substantial implementation experience with this approach, reluctance to embrace something so new is to be expected 💀
Mapping between server network addresses and leases is more complicated, because it's no longer a one-to-one mapping 🙈

There's also a set of privacy concerns that users of the client might have. The uniform client ID string approach would let multiple servers acting in concert determine when multiple requests received at different times come from the same NFSv4.0 client. For example, this might make it possible to determine that multiple distinct user identities are likely to correspond to requests made by the same person, even when those requests go to different servers 🔒

How to balance all of these considerations depends on implementation goals ⭐

5.7. Mixing Client ID String Approaches ✨

okay so, we already talked about this 💅 a client that needs the uniform client ID string approach (say, to support migration) might also have to deal with existing servers whose implementations don't work right with it 😭

Here are some of the server issues that cause this:

Some existing NFSv4.0 server implementations of IP address failover depend on clients using the non-uniform client ID string approach. Specifically, when a server supports both its own IP address and one failed over from a partner server, it may have separate sets of state for the two IP addresses. Those sets are owned by different servers but live on a single one.

In this situation, some servers have relied on clients using the non-uniform client ID string approach to keep those sets of state separate. That approach was suggested but not mandated by [RFC7530]. Servers like this will have problems with clients using the uniform client ID string approach, because those clients will see changes in trunking relationships whenever server failover and giveback happen. 🥲

Some existing servers wrongly return NFS4ERR_CLID_INUSE just because a clientid4 already exists for the same client, one established using a different IP address. This causes trouble for a multihomed client using the uniform client ID string approach.

This behaviour is not correct, but these servers still exist. This specification should give clients guidance on dealing with the situation, and also make the correct behaviour clear. 🎀

To support these kinds of servers, the client can use different client ID string approaches for different mounts. That way it can make sure that:

The uniform client ID string approach is used when accessing servers that may return NFS4ERR_MOVED, and when the client wants to enable transparent state migration. 🦋
The non-uniform client ID string approach is used when accessing servers whose implementations make them incompatible with the uniform client ID string approach. 🌸

The client usually can't easily tell which of the above is true. So, in cases where a client supports using multiple approaches at the same time, implementations are likely to rely on user-specified mount options to pick the right one. Choosing a default for those cases is up to the client implementation. 🧚

If the same server has multiple mounts and both approaches are specified for that server, the client could end up with multiple clientid4s for that one server, one per approach. It would then have to keep those separate. 💜

5.8. Trunking Determination when Using Uniform Client ID Strings 🦋

okay this section is giving "here's one way to figure out trunking" 🤓 It's an example of how a client following the uniform client ID string approach (whether that's for all its mounts or not) could do trunking determination. Clients don't have to follow this procedure, BUT implementers should make sure the issues this procedure handles are all properly addressed. Non-negotiable, bestie. 💅

It's worth clarifying the different purposes of trunking determination, and what they require from server behaviour. Here's what to keep in mind:

The primary purpose of the trunking determination algorithm is to make sure that, if the server treats client requests on two IP addresses as part of the same client, the client won't be surprised and run into disconcerting server behaviour, like the kind mentioned in Section 5.6. That behaviour could happen if the client didn't know that all of its requests on the two IP addresses were being handled as part of a single client talking to a single server. 😳
A second purpose is being able to use knowledge of trunking relationships for better performance, etc. 📈
If a server gave out distinct clientid4s in response to receiving the same nfs_client_id4 on different network addresses, and acted as if these were separate clients, the primary purpose of trunking determination would still be met, as long as the server didn't treat them as part of the same client. In that case, with regard to that client, the server would be acting as if it were two distinct servers. This would interfere with the secondary purpose of trunking determination, but there's nothing the client can do about that. 🥺
Now suppose a server gave such a client two different clientid4s but acted as if they were one. That's the only way the server could behave in a way that would defeat the primary purpose of the trunking determination algorithm. 👀

Servers MUST NOT behave that way. 🚫

For a client using the uniform approach, clientid4 values are treated as important information when figuring out server trunking patterns. 🔍

Two different IP addresses returning the same clientid4 value is a necessary, but not sufficient, condition for them to be considered connected to the same server. So when two different IP addresses return the same clientid4, the client needs to figure out whether the IP addresses are connected to the same server. It can use the procedure below or some other way to do that. For these clients, all internal recording of clientid4 values needs to include, explicitly or implicitly, the server the clientid4 came from, so there's always a (server, clientid4) pair. Two such pairs from different servers are always considered distinct, even when the clientid4 values are the same. Those values may occasionally be the same, after all. 🫶

To make this approach work, the client needs certain information for each nfs_client_id4 used by the uniform approach (only one in general). The client needs to maintain a list of all server IP addresses, together with the associated clientid4 values, SETCLIENTID principals, and authentication flavors. As part of the associated data structures, there should be a way to mark a server IP structure as having the same server as another one, and a way to mark an IP address as currently unresolved. One way to do this is to let each entry point to another, with the pointer value being one of:

A pointer to another entry for an IP address associated with the same server, where that IP address is the first one referenced to access that server.
A pointer to the current entry if there is no earlier IP address associated with the same server, i.e., where the current IP address is the first one referenced to access that server. The text below refers to such an IP address as the lead IP address for a given server. 👑
The value NULL if the address's server identity is currently unresolved. 🌀

To keep the above information current, RENEWs should be done periodically on each server, for the most effective trunking determination. Even if that isn't done, though, the primary purpose of the trunking determination algorithm, preventing confusion due to trunking hidden from the client, will still be achieved. 💖

With this apparatus in place, when a SETCLIENTID is done and a clientid4 is returned, the data structure can be searched for a matching clientid4. If one is found, further processing can determine whether the clientid4 match is accidental, or the result of trunking. 🔎

In this algorithm, the initial SETCLIENTID uses the common nfs_client_id4 and specifies the current target IP address as callback.cb_location within the callback parameters. The clientid4 and SETCLIENTID verifier returned by this operation are called XC and XV, respectively. ✨

These callback parameters are provisional. They reflect the client's preferences in case the IP address isn't trunked with other IP addresses. The algorithm is built so that only the appropriate callback parameters, the ones reflecting observed trunking patterns, actually get confirmed. 🎯

Note that if the client has done previous SETCLIENTIDs to any IP addresses using more than one principal or authentication flavor, it's possible to get NFS4ERR_CLID_INUSE. That's because it isn't yet known which of the connections to existing IP addresses might be trunked with the current one. If the SETCLIENTID fails with NFS4ERR_CLID_INUSE, the client must try every other combination of principals and authentication flavors currently in use. Eventually one will be correct, and won't return NFS4ERR_CLID_INUSE. 🔁

Note that no SETCLIENTID_CONFIRM has been done at this point. The SETCLIENTID just done either established a new clientid4 on a previously unknown server, or changed the callback parameters on a clientid4 tied to a server already known. It's undesirable to confirm something that shouldn't happen, so what comes next depends on information about existing clientid4s. 🙈

If no matching clientid4 is found, the IP address X and clientid4 XC are added to the list and considered as having no existing known IP addresses trunked with them. The IP address is marked as a lead IP address for a new server. A SETCLIENTID_CONFIRM is done using XC and XV. 🌱
If a matching clientid4 is found that is marked unresolved, processing on the new IP address is suspended. To keep processing simple, there can only be one unresolved IP address for any given clientid4. ⏸️
If one or more matching clientid4s are found, none of which are marked unresolved, the new IP address X is entered and marked unresolved. A SETCLIENTID_CONFIRM is done to X using XC and XV. 🧩

When an unresolved IP address exists because of the last of the three cases above, further processing is required. The steps below are applied to each of the lead IP addresses with a matching clientid4, and after that the address will have been resolved. It may have been determined to be part of an already known server, as a new IP address added to an existing set of IP addresses for that server. Otherwise, it will be recognized as a new server. At the point this determination is made, the unresolved indication is cleared and any suspended SETCLIENTID processing is restarted. 🥳

For each lead IP address IPn with a clientid4 matching XC, the following steps are done. The Remote Procedure Call (RPC) to do a SETCLIENTID could take a while, so it's desirable for the client to do these operations in parallel. Because the clientid4 is a 64-bit value, the number of such IP addresses that would need testing is expected to be quite small, even when the client is talking to many NFSv4.0 servers. So while parallel processing is desirable, it isn't necessary. 🙌

If the principal for IPn doesn't match the one for X, the IP address is skipped, since IPn and X can't possibly be trunked in these circumstances. If the principal does match but the authentication flavor doesn't, the authentication flavor already used should be used for address X as well. This avoids any chance that NFS4ERR_CLID_INUSE will be returned for the SETCLIENTID and SETCLIENTID_CONFIRM done below, as long as the server(s) at IP addresses IPn and X are correctly implemented. 🛡️

Okay so the trunking detective work is still going, bestie 🕵️💅

A SETCLIENTID is done to update the callback parameters, since X might now be marked as associated with the server whose lead IP address is IPn. The client picks the exact cb_client4 and callback_ident values based on how it wants callbacks handled for the shared clientid4, in case X and IPn turn out to be trunked together 🎀. When a SETCLIENTID is done on IP address IPn, the server sends back a setclientid_confirm value (a verifier4), and we're calling that one SCn 🔖.
The setclientid_confirm value SCn gets saved so it can be used later to confirm the SETCLIENTID done to IPn 💾.

Once all the SCn values are collected, each one gets tested as the verifier in a SETCLIENTID_CONFIRM operation sent to the original IP address X, the one whose trunking relationships we're figuring out. These RPC operations can run in parallel 🏃‍♀️💨

There are a few things to note here, and they're lowkey important 👀

The SETCLIENTID operations done on the various IPn addresses will never be confirmed by SETCLIENTID_CONFIRM operations sent to those IPn addresses. If those callback updates are going to be confirmed at all, it'll happen through SETCLIENTID_CONFIRM operations aimed at the original address X. That can only work if SCn was generated by an IPn that's trunked with X. If it works, the SETCLIENTID gets confirmed and we can infer that the trunking relationship exists 🦋
The number of successful SETCLIENTID_CONFIRM operations should never be more than one. If X accepts both SCn and SCm, that would mean both IPn and IPm are trunked with X, which is only possible if IPn and IPm are trunked together. Those two addresses were already recognized as not trunked together, so this should be impossible, as long as the servers involved are implemented correctly 🙅‍♀️

What happens next depends on whether each of the various SETCLIENTD_CONFIRM operations from the step above succeeded or failed 🤞

If the setclientid_confirm value generated by a particular IPn is accepted on X, then X and IPn are recognized as connected to the same server, and the entry for X gets marked as associated with IPn 🫶
If none of the confirm operations are accepted, then X is recognized as a distinct server. Its callback parameters stay as the ones set up by the original SETCLIENTID 🌸

Either way, the entry is considered resolved, and processing can restart for the IP addresses whose clientid4 matched XC but whose resolution had been put on hold 🔄✨

The whole procedure has to be done so that multiple SETCLIENTIDs sent to different server IP addresses, and returning the same clientid4, can't "race" each other in a way that leaves no clear answer about whether they belong to the same server 🏁 There are several valid ways to serialize this, and implementers can pick one based on the tradeoff between performance and complexity. They're listed in order of increasing parallelism 📈

An NFSv4.0 client might serialize all instances of SETCLIENTID/SETCLIENTID_CONFIRM processing, either directly or by serializing mount operations that use NFSv4.0. That prevents the races mentioned above, but this much serialization can cause performance issues when there's a high volume of mount operations 🐢
Another option is to serialize the period that starts when the clientid4 received from the server gets processed and ends when all trunking determination for that server is complete. This prevents the races above without adding delay, except when trunking determination is common 🫠
You could also skip most of that serialization by letting trunking determination for distinct clientid4 values happen in parallel, with serialization of trunking determination happening independently for each distinct clientid4 value 🚀

The procedure above doesn't explicitly mention that a server reboot can happen at any time, so let's cover that 🔁 The client should make sure the following steps are taken:

When a SETCLIENTID_CONFIRM is rejected by a given IPn, the client should keep in mind that the rejection might be due to XC being invalid, rather than XV. To handle that, the client can do a RENEW specifying XC directed to the IP address X. If that succeeds, the rejection should be acted on normally, because either XV is invalid on IPn, or XC has become invalid on IPn while still being valid on X, which shows that IPn and X are not trunked. If XC is not valid on X, then the trunking detection process should be restarted once a new client ID is established on X 😵‍💫
If a reboot is detected on any server-lead IP, the set of IP addresses associated with the server should not change. State should be re-established for the lease as a whole, using all available connected server IP addresses. It's prudent to verify connectivity by doing a RENEW using the new clientid4 on each such server address before using it, however 🛡️

Another situation the procedure doesn't cover explicitly is a SETCLIENTID done to one of the IPn addresses taking so long that the operation has to time out, so it doesn't hold up the MOUNT operation for too long ⏳ One simple option is to just fail the MOUNT at that point. Since the average number of IP addresses that might need testing is pretty small, this won't greatly increase the chance of MOUNT failure 🤷‍♀️ Other possible approaches are:

If IPn has enough state in existence, the existing stateids and sequence values might be validated by using them on IP address X. If that succeeds, X and IPn should be considered trunked together 🧩
- What counts as "sufficient" state here is an implementation decision. It depends on how willing the implementer is to fail the MOUNT in an uncertain case, and on how strong the state verification procedure is 💭
If IPn has no locking state in existence, X could be recorded as a lead IP address on a provisional basis, subject to trunking being tested again once IPn starts responding. To avoid confusing IPn and X, and to avoid needing to merge distinct state corpora for X and IPn later, this retest of trunking should happen after RENEWs on IPn get responses, and before any new state is established for either IPn as a separate server or for IPn considered as a server address trunked with X 🧐
The client's locking-related code could be made more tolerant of results that would otherwise look anomalous because of an unrecognized trunking relationship. The client could treat behaviour that's explainable by a previously unknown trunking relationship as the cue to consider the addresses trunked 🫣

This choice is super complicated, and few implementations will probably use it. When the set of locking state on IPn is small (e.g., a single stateid) but not empty, most client implementations will likely either fail the MOUNT or use a stricter verification procedure that builds on the existing stateid on IPn to generate further state as raw material for the trunking verification process 🧱

5.9. Client ID String Construction Details 🎀

This section gives more detailed guidance on building client ID strings 📝 The guidance covers cases where either the uniform or the non-uniform approach to the client ID string is used.

Heads up, many of the items suggested for inclusion may change over time 🙈 To keep the client ID string valid when that happens, the client SHOULD either:

Use a saved copy of such a value rather than the changeable value itself, or
Save the constructed client ID string rather than building it again at SETCLIENTID time from unchangeable parameters and saved copies of changeable data items 💜

A file isn't always a good place to store this info, because diskless clients exist. In those situations, use whatever facilities the client has for storing configuration information, such as boot arguments 💾

Given the considerations listed in Section 5.2.1, an id string should be built on a basis that includes:

An identifier uniquely associated with the node the client is running on 🖥️
For a user-level NFSv4.0 client, extra information to tell it apart from a kernel-based client and from other user-level clients on the same node, such as a universally unique identifier (UUID) 🪪
Where the non-uniform approach is used, the IP address of the server 🌐
Additional information that tends to be unique, such as one or more of:
- The timestamp of when the NFSv4 software was first installed on the client. This is subject to the caution mentioned earlier about storing info in a file, because the file might only be accessible over NFSv4 ⏰
- A true random number, generally established once and saved 🎲

For the identifier tied to the node the client runs on, these are the likely candidates 👀

The client machine's serial number 🏷️
The client's IP address. Note that this SHOULD be treated as a changeable value 🔀
A Media Access Control (MAC) address. This should also be considered a changeable value, because configuration changes can happen 🔧

Privacy concerns can come up if some of the items above (e.g., machine serial number and MAC address) get used. When it's necessary to use such items to ensure uniqueness, applying a one-way hash function is desirable. When the non-uniform approach is used, that hash function should be applied to all of the components chosen, as a unit, rather than to individual elements on their own 🔐🙈

6. Locking and Multi-Server Namespace ✨

okay bestie, this section is a replacement for Section 9.14 of [RFC7530], "Migration, Replication, and State" 💅

The replacement lives in Section 6.1 and it supersedes the replaced section, so that's the one to read 🦋

Here's the tea on what changed ☕

Added text for the stateid conflict on migration case 😭
When leases get moved because of a file system migration, they're now merged with the leases on the destination server that are connected to the same client 🫶
Added text for when a clientid4 gets changed during state transfer because it conflicts with an existing clientid4 🔥
Added a section on how the info tied to open-owners and lock-owners gets managed during migration 🧚
The description of how NFS4ERR_LEASE_MOVED is handled got rewritten so it's way clearer now 💜

6.1. Lock State and File System Transitions 🌸

A file system can move to a different server in a few situations:

Responsibility for the file system gets handed to a new server via migration 🚚
A client picks an alternate server (like when the current one stops responding) because the file system is being replicated 🔁

In those cases, the shared state between the client and server (locks, leases, stateids, and client IDs) gets handled in the way described below. Heads up, migration and replication handle it differently 👀

If a server replica or a server immigrating a file system agrees to, or is expected to, accept opaque values from the client that came from another server, then it's a wise implementation practice for the servers to encode those "opaque" values in network byte order (i.e., in a big-endian format) 🧩 That way, replicas and immigrating servers can parse things like stateids, directory cookies, filehandles, etc., even if their native byte order is different from the other servers working together on the replication and migration of the file system ✨

6.1.1. Migration and State 🦋

In the case of migration, the servers involved should transfer all server state associated with the migrating file system from the source to the destination server. If state is transferred, this MUST be done in a way that is transparent to the client 🫶 This makes the client's transition way smoother when a file system migration happens. If the servers succeed in transferring all state, the client keeps using the stateids the original server assigned. So the new server MUST recognize those stateids as valid and treat them as representing the same locks they did on the source server 💅

In this context, the phrase "the same locks" means that:

They're associated with the same file.
They're the same types of locks: opens, delegations, advisory byte-range locks, or mandatory byte-range locks.
They have the same lock particulars, like access modes, deny modes, and byte ranges.
They're associated with the same owner string(s).

If transferring stateids from server to server would cause a conflict with an existing stateid on the destination server for the existing client, transparent state migration MUST NOT happen for that client 😭 Servers that take part in transparent state migration should coordinate their stateid assignment policies to make this situation unlikely or impossible 🤝 How that gets done, like all of the inter-server interactions for migration, is not specified by the NFS version 4.0 protocol (neither in [RFC7530] nor this update).

A client can figure out what happened to its migrated state by using a stateid tied to that state on the new server 🔍

If the stateid isn't valid and an error NFS4ERR_BAD_STATEID comes back, then either transparent state migration didn't happen, or the state got purged because of a mismatch in the verifier (i.e., the boot instance id) 💀
If the stateid is valid, transparent state migration has happened 🎉

Since responsibility for an entire file system transfers with a migration event, there's no chance of conflicts showing up on the destination server because of the locks being moved 🙌

The servers may choose not to transfer state information upon migration. This choice is discouraged, though, except where specific issues like stateid conflicts make it necessary 🙈 When a server implements migration and doesn't transfer state information, it MUST provide a file-system-specific grace period so clients can reclaim locks associated with files in the migrated file system ⏳ Without that, clients would have to re-obtain their locks, with no assurance that a conflicting lock wasn't granted after the file system was migrated and before the lock was re-obtained 😱

In the case of migration without state transfer, when the client presents state information from the original server (for example, in a RENEW operation or a READ operation of zero length), the client must be prepared to receive either NFS4ERR_STALE_CLIENTID or NFS4ERR_BAD_STATEID from the new server 🫣 The client should then recover its state information the same way it normally would in response to a server failure. The new server must take care to allow for the recovery of state information the same way it would in the event of a server restart 🔧

When state hasn't been transferred, shown by a return of NFS4ERR_BAD_STATEID, the client may try to reclaim locks so it can take advantage of cases where the destination server has set up a file-system-specific grace period to support the migration 🌈

6.1.1.1. Migration and Client IDs 🧚

The handling of clientid4 values is similar to how stateids are handled. There are some differences, though, because a clientid4 is an object that spans multiple file systems, while a stateid is limited to a single file system 🧠

The clientid4 and nfs_client_id4 information (id string and boot instance id) gets transferred along with the rest of the state information, and the destination server should use that info to decide how to handle the clientid4 🔑 The destination server might make state stored under an existing lease available under the clientid4 used on the source server, but the client should not assume that's always the case. Specifically:

If there's an existing lease with an nfs_client_id4 that matches a migrated lease (same id string and verifier), the server SHOULD merge the two, making the union of the sets of stateids available under the clientid4 for the existing lease 🫶 As part of the lease merger, the lease's expiration time reflects the renewal done within either of the ancestor leases, so it reflects the latest of the renewals ⏰
If there's an existing lease with an nfs_client_id4 that only partially matches a migrated lease (same id string but a different (boot) verifier), the server MUST eliminate one of the two, which could possibly invalidate one of the ancestor clientid4s 💅 Since boot instance ids aren't ordered, the later lease renewal time wins 🏆
If the destination server already has the transferred clientid4 in use for something else, it's free to substitute a different clientid4 and associate that with the transferred nfs_client_id4 🔄

When leases aren't merged, the transfer of state should result in creating a confirmed client record with empty callback information, but matching the {v, x, c}, with v and x derived from the transferred client information and c chosen by the destination server. For a description of this notation, see Section 8.4.5.

In these cases, the client SHOULD re-establish new callback information with the new server as soon as possible, following the sequences described in the sections "Operation 35: SETCLIENTID -- Negotiate Client ID" and "Operation 36: SETCLIENTID_CONFIRM -- Confirm Client ID" 📞 This makes sure server operations don't get delayed because delegations can't be recalled, and it prevents unwanted revocation of existing delegations 🛡️ The client can find the new clientid4 (the value c) in the response to SETCLIENTID 🎀

The client can use its own info about leases with the destination server to check whether lease merger should have happened. When there's any ambiguity, the client MAY use the above procedure to set the proper callback information and, as part of the process, find out the correct value of its clientid4 with respect to the server in question 🦋💜

6.1.1.2. Migration and State Owner Information ✨

Okay so, besides stateids (and the locks they represent) plus client identity info, servers also need to move over info about where open-owners and lock-owners are at right now 💅 That info includes:

The sequence number of the last operation tied to the owner 🔢
Enough detail about the results of that last operation so reissued operations can get a correct response 🫶

When an open-owner or lock-owner has only been used with one particular file system, the server SHOULD transfer this info together with the lock state 🎀 The owner stops existing on the source server and gets rebuilt on the destination server 🦋 This happens with clients that were built to isolate each owner to one file system, but it may happen with other clients too 🌸

Note that if servers do this for every owner whose state is limited to the file system being migrated, it won't cause trouble for clients that don't keep owners isolated 💖 As long as the client notices the transferred state is gone, the protocol lets that owner disappear. The client may just have to deal with an owner confirmation request it wouldn't have seen without the migration 😭

Now, when migration happens and the source server finds an owner whose state includes the migrated file system and other file systems too, it can't transfer that owner state 🙈 The existing owner state stays put, but owner status gets propagated like this:

When the current seqid for an owner represents an operation tied to the file system being migrated, owner status SHOULD be propagated to the destination file system 💜
When the current seqid for an owner does not represent an operation tied to the file system being migrated, owner status MAY be propagated to the destination file system 🌈
When the owner has never been used for an operation involving the migrated file system, the owner information SHOULD NOT be propagated to the destination file system 🚫

Note that a server can follow all of the conditions above without the overhead of tracking which file systems each owner has been associated with 🧚 Picture this: the source server has decided to keep lock-related state for a file system fixed, getting ready to propagate it to the destination file system. If a client is free to create new locks tied to existing owners on other file systems, the owner information may get propagated anyway. That's because, when the client recognizes the migration, the owner's last operation might not be tied to the migrating file system at all 🤷‍♀️

When a source server propagates owner-related state for owners that span multiple file systems, it propagates the owner sequence value to the destination server while keeping it on the source server, for as long as state tied to the owner still exists 🔥 Then source and destination start with the same owner sequence value and update it independently as the client sends owner-related requests to each server 👯‍♀️ For a while, each server will have an owner sequence value identical to the one transferred during migration. During those times, if a server gets a request with a matching owner sequence value, it MUST NOT respond with the stored response if the associated file system is not part of the set of file systems that server handles when the reissued request arrives 🙅‍♀️

One case can get a bit more complicated 👀 When multiple file systems get migrated, one at a time, to a specific destination server, an owner might be migrated to a destination server where it already exists. Now you've got to figure out how the resident owner information and the newly migrated information get reconciled 🧩

If file system migration hits a situation where owner information needs to be merged, it MAY decline to transfer that state, even if it handles other cases where locks for a given owner are spread across multiple file systems 🫣

To understand the situations that come up when owner information needs merging, check out this scenario:

There's client C and two servers, X and Y. There are two clientid4s designating C, called CX and CY 👭
Initially, server X supports file systems F1, F2, F3, and F4. These get migrated, one at a time, to server Y 🚚
While the migrations are happening, the client makes locking requests for file systems F1 through F4 on behalf of owner O (either a lock-owner or an open-owner). Each request goes to X or Y depending on which server is supporting the relevant file system at the moment the request is made 🗺️
Once the first migration event happens, client C will maintain two instances for owner O, one for each server 👀
It's always possible that C sends a request to server X about owner O, and before getting a response, it finds the target file system has moved to Y and has to reissue the request to server Y 😵‍💫
At the same time, C may send a request to server Y about owner O, and that one could run into a lost-response situation too 💀

Because of merger situations like this, the server needs to support retransmission of owner-sequenced requests that don't follow the usual model. In that usual model, replies are only retransmitted for a request whose sequence value exactly matches the last one sent. Here, there can be two requests that each had the last sequence when they were issued. After migration and owner merger, one of them is no longer the last by sequence 🙃

When servers do support merging owner information on the destination server, these rules apply:

When an owner sequence value gets propagated to a destination server where it already exists, the resulting sequence value is the greater of the one on the destination server and the one being propagated as part of migration 📈
If an owner sequence value on a server represents a request applying to a file system currently on that server, it must not be made invalid just because owner information propagation changed that sequence value during file system migration. Instead, it's kept until it can be deduced that the client has received the reply 🔒

Because of how these rules work, there are three ways a server can end up with more reply data than usual. Normally there's just one request's data per owner, the one with the last sequence received, where the next sequence to be used is one beyond it ✨

When the owner sequence value for a migrating file system is greater than the corresponding value on the destination server, the last request for the owner in effect at the destination server needs to be kept, even though it's no longer one less than the next sequence to be received 💅
When the owner sequence value for a migrating file system is less than the corresponding value on the destination server, the sequence number for the last request for the owner in effect on the migrating file system needs to be kept, even though it's no longer one less than the next sequence to be received 🫶
When the owner sequence value for a migrating file system is equal to the corresponding value on the destination server, you have two different "last" requests that both have to be kept. The next sequence value to be used is one beyond the sequence value those two requests share 👯

Here are some guidelines on when servers can drop that extra reply data, the data created as part of owner information migration 🗑️

The server SHOULD NOT drop this information just because it gets a new sequence value for the owner, since that request may have been issued before the client knew about the migration 🛑
The server SHOULD drop this information if it gets a new sequence value for the owner and the request relates to the same file system 🧹
The server SHOULD drop the part of this information that relates to non-migrated file systems if it gets a new sequence value for the owner and the request relates to a non-migrated file system 🪄
The server MAY drop this information when it gets a new sequence value for the owner after a considerable period of time (more than one or two lease periods) following the migration 🕰️

6.1.2. Replication and State 🦋

Okay, this one's different because switch-over in the case of replication isn't under server control 🙃 So leases, stateids, and client IDs do NOT carry validity across a transition from one server to another. The client must re-establish its locks on the new server 🔁 This is kinda like re-establishing locks with reclaim-type requests after a server reboot 💜 The difference is that the server has no way to tell reclaim requests apart from requests getting new locks, or to hold off on the latter 😬 So a client re-establishing a lock on the new server (with a LOCK or OPEN request) may get that request denied because of a conflicting lock 🚫 Replication is meant for read-only use of file systems, so these denials shouldn't cause huge problems in practice 🌸 And when an attempt to re-establish a lock on a new server is denied, the client should treat the situation as if its original lock had been revoked 🫠

6.1.3. Notification of Migrated Lease 🦋

okay so picture this: a file system gets migrated to another server 🚚 while a client that holds state for it is just chilling and not sending any requests there. no cap, the migration gets reported to the client during lease renewal. lease renewal can happen two ways. it can be explicit, with a RENEW operation, or implicit, when the client does a lease-renewing operation on another file system on that same server 💅

the client needs to find out about lease relocation before those leases expire ⏰, so it can schedule renewal for any leases that moved to the new server. it's also the same deal when migration happens without transparent state migration: the client has to hear about it soon enough to reclaim its lock within the destination server's grace period 🔒. so every operation that implicitly renews leases for a client (like OPEN, CLOSE, READ, WRITE, RENEW, LOCK, and others) returns the error NFS4ERR_LEASE_MOVED if responsibility for any of the leases being renewed has been transferred to a new server 🙈. btw, if the transfer leaves some state for that lease on the source server, the lease still gets renewed just like it would in the NFS4ERR_OK case, even though the error comes back anyway ✨.

responsibility gets transferred when the server receives a GETATTR(fs_locations) from the client for each file system that has a lease moved to a new server. usually this happens right after the client gets an NFS4ERR_MOVED for an access to that file system. the server is not required to verify that this actually happens before it stops returning NFS4ERR_LEASE_MOVED, though 💀. by convention, compounds containing GETATTR(fs_locations) SHOULD include an appended RENEW operation, so the server can tell which client is asking for the info 🫶.

heads up: NFS4ERR_LEASE_MOVED is only required when responsibility for at least one stateid has been affected. for a null lease, where the only associated state is a clientid4, an NFS4ERR_LEASE_MOVED error SHOULD NOT be generated. 🙅‍♀️

when a client that supports file system migration gets NFS4ERR_LEASE_MOVED, it MUST perform the necessary GETATTR operation for each migrated file system that has state. this gives the server proof that the client knows about the migration 🧾. once the client has done this for all migrated file systems it holds state on, the server MUST resume normal handling of stateful requests from that client 👑.

if there are tons of file systems, there's a way to do this efficiently 🧚. it splits the job into two phases: first you find the migrated file systems, then you do the necessary GETATTRs.

phase one is finding the migrated file systems. the client builds and sends one or more COMPOUND requests, each made of a set of PUTFH/GETFH pairs, with each pair using a filehandle from one of the file systems in question. all these COMPOUND requests can run in parallel 🏃‍♀️. if a request succeeds, that means none of the file systems it checked have migrated. if it terminates with NFS4ERR_MOVED, the file system that got the error has migrated, while the ones checked before it in the same COMPOUND have not. the ones checked after the error are in an uncertain state, so the client can check them again by restarting from right after the point where NFS4ERR_MOVED came back, or by sending a new set of COMPOUND requests for the file systems that are still uncertain 🌸.

phase two is giving the server evidence. the client just needs to show it knows the status of the file systems it found in phase one, by checking the fs_locations attribute for a filehandle in each migrated file system 💖. the client builds and sends one or more COMPOUND requests, each made of a set of PUTFH operations, with each one followed by a GETATTR of the fs_locations attribute. a RENEW is needed so these operations get linked to the lease that returned NFS4ERR_LEASE_MOVED 🎀. once the client has done this for all migrated file systems it holds state on, the server will resume normal handling of stateful requests from that client 🌈.

now, legacy clients don't handle NFS4ERR_LEASE_MOVED correctly, so the server SHOULD time out after waiting at least two lease periods. when that happens, it resumes normal handling of stateful requests from all clients 🕰️. if a client tries to access the migrated files after that, the server MUST reply with NFS4ERR_MOVED 🚫. at this point the client would probably find its lease expired, but a server can use "courtesy" locks (as described in Section 9.6.3.1 of [RFC7530]) to make that less of a problem 🛟.

when the client gets NFS4ERR_MOVED, it can follow the usual process to get the destination server info through the fs_locations attribute, then renew those leases on the new server 🗺️. if state wasn't transferred transparently to the new server, the client will get either NFS4ERR_STALE_CLIENTID or NFS4ERR_STALE_STATEID from it, same as described above. then the client can recover its state information just like it does after a server failure 🔁.

getting a migration notice isn't the only reason a client might want fs_locations info 🧭. if a server stops responding, for example, the client can use cached fs_locations data to find an alternate server hosting the same file system data. a client may also ask a server for fs_locations data every so often, just to keep its cache fresh 🍓.

since a GETATTR(fs_locations) operation is also used to refresh cached data, a server could mistake that request for a sign the client recognized an NFS4ERR_LEASE_MOVED condition 🙈. so a compound that isn't meant to signal that a client recognized a migrated lease SHOULD be prefixed with a guard operation. that guard fails with NFS4ERR_MOVED if the filehandle being queried is no longer on the server. the guard can be as simple as a GETFH operation 🛡️.

it's unlikely, but the target of such a compound could get migrated after the guard runs on the server and before the GETATTR(fs_locations) operation is reached 😱. when a client sends a GETATTR(fs_locations) as part of a compound not meant to signal recognition of a migrated lease, it SHOULD be prepared to process the fs_locations data in the reply, which shows that the current location of the file system is gone 🏃‍♀️💨.

6.1.4. Migration and the lease_time Attribute 🕒

so the client can manage its leases properly during a migration, the destination server has to set up the right values for the lease_time attribute 💅.

when state is transferred transparently, that state should include the correct value of the lease_time attribute 📦. the lease_time attribute on the destination server must never be less than the one on the source, because that would make leases granted by the source expire early 🚨. in a transparent state transfer, the client isn't obligated to re-fetch the lease_time attribute, and it can keep using the value it got earlier from the source server 🫶.

if lease merger happens as part of the state transfer, the lease_time attribute of the destination lease stays in effect. the client can just renew that lease with its existing lease_time attribute 🔄. the state in the source lease gets renewed at the time of transfer so it can't expire, as long as the destination lease is renewed properly 🔒✨.

if state wasn't transferred transparently (meaning the client needs to reclaim or re-obtain its locks), the client should fetch the value of lease_time from the new, destination server and use it for later locking requests 📝. the server, though, must respect a grace period at least as long as the lease_time on the source server. that gives clients plenty of time to reclaim their locks before any conflicting non-reclaimed locks get granted 🙌. how the new server gets the lease_time value from the old server is up to the server implementations. the NFS version 4.0 protocol doesn't specify it 🤷‍♀️.

7. Server Implementation Considerations 🛠️

this section has suggestions to help server implementers deal with the issues involved in transparently transferring file-system-related data between servers 🧠. servers aren't obliged to follow these suggestions, but they should be sure their approach handles all the potential problems covered below 💜.

7.1. Relation of Locking State Transfer to Other Aspects of File System Motion

okay so, state transfer is usually part of a bigger move where a whole file system gets shipped from one server to another 📦✨ The details change depending on the implementation, but the way persistent file data and metadata move is basically going to fit one of these vibes 💅

ok so in some implementations, the on-disk contents of a file system can change hands because the storage devices are physically reachable from multiple servers, and the right and responsibility to handle that file system get handed over from server to server 🔌🫶

the locking state transfers on its own here, like it's described in Section 7.2 🔒 the physical access move happens after the locking state is transferred and before anyone touches the file system again 🙅‍♀️ if that handoff isn't instant, there's a gap where everything on the file system gets held off, either because the operations themselves just return NFS4ERR_DELAY, or, if that's not allowed, by using the techniques in Section 7.2 below ⏳

in other implementations, the file system data and metadata have to be copied from the old server to the new one 📋 because there's usually a LOT of data, it's not really practical to freeze the file system during the copy 😭 so normal access keeps going, including modifying operations, while the transfer is happening 🏃‍♀️💨

eventually the copying finishes, and now there are two valid copies of the file system, one on the source server and one on the destination server 👯‍♀️ servers can keep that going by making sure every change to the file system data gets applied on both the source and the destination 🪞

the locking state transfer can start before we get to that point, but servers usually wait until the two copies are in sync before they begin transferring locking state 🤝 once the locking state transfer is done (more on that in the next section 👀), clients can get notified about the migration and start accessing the file system on the destination server 🎉

ngl this one is a fun twist: the file system data and metadata still have to be copied from server to server, but it's a variant of the pattern above 🔁 when a single file system moves between a small set of servers, it might end up back on a server where an earlier copy of that same file system already lived 🏠 in that situation, it's often more efficient to update that earlier instance with the changes made while the active file system was somewhere else, instead of copying everything from scratch 🧠

so the copying gets replaced by a process that validates each visible file system object, copies over the new ones, and updates the ones that changed since the file system was last on the destination server 🔍 this is usually shorter than a full copy, but it's still long enough that holding off access during the update isn't practical either 🙈

once the updating finishes, there are again two valid copies of the file system, one on the source and one on the destination server 💖 servers can keep that state of affairs going the same way as in the previous case ✨ and the locking state transfer, once it's complete, lets clients be notified of the migration event and access the destination file system on the destination server 🦋

7.2. Preventing Locking State Modification during Transfer ✨

Okay so, when a server is moving locking state over to a destination server, there are times when the source server has to stop anything that would change the state being moved 🫣. Say the locking state gets sent at time T. If something changes on the source after that but before the file system transfer takes effect, the destination ends up with different state than the source. And that's not what the client expects to see 💀

In general, a server can freeze some server-maintained data by returning NFS4ERR_DELAY on any operation that tries to change it. For NFSv4.0 locking state, though, there are two specific problems that can get in the way 🙈

Returning NFS4ERR_DELAY doesn't actually stop state from changing, because owner-based sequence values still move forward even when NFS4ERR_DELAY comes back. For example, OPEN and LOCK still change state (in the form of owner seqid values) even when they return NFS4ERR_DELAY.
Some operations that change locking state aren't allowed to return NFS4ERR_DELAY at all: OPEN_CONFIRM, RELEASE_LOCKOWNER, and RENEW.

The first problem, and most of the second, can be fixed by returning NFS4ERR_DELAY on any operation that sets a filehandle inside the target as one of the request's filehandles (either the current or the saved one) 🧚 That means returning NFS4ERR_DELAY in these cases:

On a PUTFH that specifies a filehandle within the target file system.
On a LOOKUP or LOOKUPP that crosses into the target file system.

Once you do that, OPEN_CONFIRM is handled, and only RELEASE_LOCKOWNER and RENEW are left to deal with 🎀

Also, if the server makes sure that no request ever has a filehandle inside the target file system as its current or saved filehandle, then no special handling of SAVEFH or RESTOREFH is needed. Those two can't return NFS4ERR_DELAY, but that doesn't matter here, since neither one can set the current filehandle to something in the target file system. Not a problem at all 💅

If the server wants to set that situation up, it may have to keep a close eye on long-running requests that started before the state migration. Part of solving this means telling apart two different points in time when the handling for the target file system changes. Let's call them:

A time T, after which the operations mentioned above return NFS4ERR_DELAY.
A later time T', at which the server can treat the file system's locking state as fixed, so it can be sent to the destination server.

To pick T', the server has to make sure that requests started before T can't change the target file system's locking state. That works because every request started after T gets NFS4ERR_DELAY when it tries to set a filehandle inside the target file system. A couple of ways to do this:

Keep track of the earliest-started request that's still running (for example, by keeping a list of active requests ordered by start time). Requests that started before T and are still in progress might affect the locking state. Once the start time of the earliest-started active request is later than T, that request's start time can be picked as T'. Any request still running after T' started after T, so it wouldn't have been allowed to change locking state for the migrating file system. It would have gotten NFS4ERR_DELAY if it had tried 🦋
Keep a count of requests that started before time T and have a filehandle inside the target file system as either the current or saved filehandle. The server can then define T' as the first time after T when that count is zero 🌸

There are two operations that changes locking state which the approaches above can't handle, since they aren't tied to a specific file system and don't use a current filehandle as an implicit parameter 😭

RENEW can be handled by applying the renewal to the state of non-transitioning file systems. The effect of the renewal on the transitioning file system can be ignored, as long as the servers make sure the lease on the destination server expires no earlier than the latest renewal done on the source server. This is easy to get right: just set the lease expiration on the destination server to the time the state transfer completed plus the lease period ✨
RELEASE_LOCKOWNER can be handled by passing the lock-owner deletion along to the destination server (for example, with an RPC). That propagation can happen as part of the operation, or the deletion can be recorded locally and the deletions sent to the destination server in a batch later. Either way, the actual deletions on the destination server have to wait until all the other state information has been transferred 🫶

There's also another option: handle RELEASE_LOCKOWNER by returning NFS4ERR_DELAY. To avoid compatibility issues with clients that aren't ready to receive NFS4ERR_DELAY from RELEASE_LOCKOWNER, you have to be careful here. (See Section 8.3 for details.)

The approach above, where NFS4ERR_DELAY is returned mainly based on the use of current and saved filehandles in the file system, blocks all references to the transitioning file system. It doesn't just block the operations that change locking state on that file system. Because of that, servers may want to limit how long this broad approach is used, by splitting the issue into layers 🌈

During the preparatory phase, operations that change, create, or destroy locks, or that modify the valid set of stateids, return NFS4ERR_DELAY. During this phase, owner-associated seqids may change, and the identity of the file system tied to the last request for a given owner may change too. Also, RELEASE_LOCKOWNER operations may be processed without returning NFS4ERR_DELAY, as long as the lock-owner deletion is recorded locally to send later.
During the restrictive phase, operations that change locking state for the file system in transition are blocked by returning NFS4ERR_DELAY on any attempt to make a filehandle inside that file system the current or saved filehandle for a request. RELEASE_LOCKOWNER operations may return NFS4ERR_DELAY, but if they are processed, the lock-owner deletion needs to be communicated to the destination server right away.

Here's a possible sequence, bestie 📝

The server enters the preparatory phase for the transitioning file system.
Locking state, including stateids, locks, and owner strings, is transferred to the destination server. The seqids associated with owners are either not transferred or transferred provisionally, subject to later change.
Once the above has been transferred, the server may enter the restrictive phase for the file system.
At this point, the updated seqid values may be sent to the destination server.

Pending owner deletions (from RELEASE_LOCKOWNER operations) can be reported at the same time.

Once it's known that all of this information has reached the destination server, and there are no pending RELEASE_LOCKOWNER notifications outstanding, the source server may treat the file system transition as having happened. From then on it returns NFS4ERR_MOVED whenever someone tries to access that file system 👑

8. Additional Changes 💖

This section has a bunch of items that relate to the changes in the section above, but for one reason or another they live in other parts of the specification that need updating 🧚

8.1. Summary of Additional Changes from Previous Documents 📋

Here's a quick rundown of all the remaining changes that aren't in the two main sections 💅

A new definition of the error NFS4ERR_CLID_INUSE, which appears in Section 8.2. This replaces the definition in Section 13.1.10.1 in [RFC7530].
A revision of the error definitions section to allow RELEASE_LOCKOWNER to return NFS4ERR_DELAY, with appropriate constraints so it stays interoperable with clients that don't expect this error. These changes are discussed in Section 8.2, and they modify the error tables in Sections 13.2 and 13.4 in [RFC7530].
A revised description of SETCLIENTID, which appears in Section 8.4. This brings the description in line with the rest of the specification regarding NFS4ERR_CLID_INUSE. The revised description replaces the one in Section 16.33 of [RFC7530].
Some security-related changes appear in Sections 8.5 and 8.6. The Security Considerations section of this document (Section 9) describes the effect on the corresponding section (Section 19) in [RFC7530] 🔐

8.2. NFS4ERR_CLID_INUSE Definition 🚨

The definition of this error now reads as follows:

The SETCLIENTID operation has found that the id string within the specified nfs_client_id4 was previously presented with a different principal, and that client instance currently holds an active lease. A server MAY return this error if the same principal is used, but a change in authentication flavor gives good reason to reject the new SETCLIENTID operation as not bona fide. 💀✨

8.3. NFS4ERR_DELAY Return from RELEASE_LOCKOWNER ✨

okay so the existing error tables are to be considered modified to allow NFS4ERR_DELAY to be returned by RELEASE_LOCKOWNER 🙈 but bestie, the scope here is super limited. this is NOT giving "it's fine everywhere now" energy 💀

servers may not return this error to clients that aren't ready for file system migration 🦋 those clients might be following the error specs in [RFC7530], so they won't be expecting NFS4ERR_DELAY back from RELEASE_LOCKOWNER at all. so don't surprise them 😭

this constraint is attached to the new error return, like a note sitting right next to the newly allowed error code 📌:

to freeze server state for a file system that's being migrated 🧊, a server MAY return NFS4ERR_DELAY in response to a RELEASE_LOCKOWNER that will affect locking state being propagated to a destination server. the source server MUST NOT do so unless it is likely that it will later return NFS4ERR_MOVED for that file system 🚫

in the context of lock-owner release, the set of file systems where freezing server state can result in NFS4ERR_DELAY has to include the file system that the operation tied to the current lock-owner seqid ran on 🗂️

and this set can also include other file systems where an operation tied to an earlier seqid for the current lock-owner was performed 🔁 since servers will have to deal with an owner getting used back-to-back across multiple file systems.

so if a client is ready to get NFS4ERR_MOVED after creating state tied to a file system, it also has to be ready to get NFS4ERR_DELAY back from RELEASE_LOCKOWNER if it used that owner with a file on that file system 💖

8.4. Operation 35: SETCLIENTID -- Negotiate Client ID 👑

8.4.1. SYNOPSIS

     client, callback, callback_ident -> clientid, setclientid_confirm

8.4.2. ARGUMENT

   struct SETCLIENTID4args {
           nfs_client_id4  client;
           cb_client4      callback;
           uint32_t        callback_ident;
   };

8.4.3. RESULT

   struct SETCLIENTID4resok {
           clientid4       clientid;
           verifier4       setclientid_confirm;
   };

   union SETCLIENTID4res switch (nfsstat4 status) {
    case NFS4_OK:
            SETCLIENTID4resok      resok4;
    case NFS4ERR_CLID_INUSE:
            clientaddr4    client_using;
   default:
            void;
   };

8.4.4. DESCRIPTION

the client uses SETCLIENTID to tell the server it wants to use a particular client identifier, callback, and callback_ident for later requests that create lock, share reservation, and delegation state on the server 🔐. if that works out, the server hands back a shorthand client ID, and once it's confirmed in a separate step, that shorthand is what shows up in later file locking and file open requests 🧾.

confirmation has to happen via SETCLIENTID_CONFIRM, which sends the clientid and setclientid_confirm values back to the server as verifiers ✅. why two verifiers? bc SETCLIENTID and SETCLIENTID_CONFIRM can be used to change the callback and callback_ident info without changing the shorthand client ID 🙃. in that case, setclientid_confirm is basically the only verifier doing the job.

the callback info given in this operation gets used if the client is handed an open delegation later on, so the client has to correctly reflect the program and port numbers for the callback program at the time SETCLIENTID is used 📞

callback_ident is used by the server on the callback 🪄. the client can lean on callback_ident to avoid needing more than one callback RPC program number, and still figure out which server is starting the callback 🧚

8.4.5. IMPLEMENTATION

to specify the implementation of SETCLIENTID, the following notations are used.

Let:

x be the value of the client.id subfield of the SETCLIENTID4args structure.
v be the value of the client.verifier subfield of the SETCLIENTID4args structure.
c be the value of the client ID field returned in the SETCLIENTID4resok structure.
k represent the value combination of the callback and callback_ident fields of the SETCLIENTID4args structure.
s be the setclientid_confirm value returned in the SETCLIENTID4resok structure.
{ v, x, c, k, s } be a quintuple for a client record. A client record is confirmed if there has been a SETCLIENTID_CONFIRM operation to confirm it. Otherwise, it is unconfirmed. An unconfirmed record is established by a SETCLIENTID call.

8.4.5.1. IMPLEMENTATION (Preparatory Phase)

since SETCLIENTID is a non-idempotent operation, our treatment assumes use of a duplicate request cache (DRC). for the DRC deets, see Section 9.1.7 of [RFC7530].

when the server gets a SETCLIENTID { v, x, k } request, it first runs a bunch of preliminary checks, listed below, before moving on to the main part of SETCLIENTID processing 🔍

It first looks up the request in the DRC. If there is a hit, it returns the result cached in the DRC. The server does NOT remove client state (locks, shares, delegations) nor does it modify any recorded callback and callback_ident information for client { x }. The server now proceeds to the main part of SETCLIENTID.
Otherwise (i.e., in the case of any DRC miss), the server takes the client ID string x and searches for confirmed client records for x that the server may have recorded from previous SETCLIENTID calls. If there are no such records, or if all such records have a recorded principal that matches that of the current request's principal, then the preparatory phase proceeds as follows.
- If there is a confirmed client record with a matching client ID string and a non-matching principal, the server checks the current state of the associated lease. If there is no associated state for the lease, or the lease has expired, the server proceeds to the main part of SETCLIENTID.
- Otherwise, the server is being asked to do a SETCLIENTID for a client by a non-matching principal while there is active state. In this case, the server rejects the SETCLIENTID request, returning an NFS4ERR_CLID_INUSE error, since use of a single client with multiple principals is not allowed. Note that even though the previously used clientaddr4 is returned with this error, the use of the same id string with multiple clientaddr4s is not prohibited, while its use with multiple principals is prohibited.

8.4.5.2. IMPLEMENTATION (Main Phase)

if the SETCLIENTID has not been dealt with by DRC processing, and has not been rejected with an NFS4ERR_CLID_INUSE error, then the main part of SETCLIENTID processing proceeds, as described below.

The server checks if it has recorded a confirmed record for { v, x, c, l, s }, where l may or may not equal k. If so, and since the id verifier v of the request matches that which is confirmed and recorded, the server treats this as a probable callback information update and records an unconfirmed { v, x, c, k, t } and leaves the confirmed { v, x, c, l, s } in place, such that t != s. It does not matter if k equals l or not. Any pre-existing unconfirmed { v, x, c, *, * } is removed.

The server returns { c, t }. It is indeed returning the old clientid4 value c, because the client apparently only wants to update callback value k to value l. It's possible this request is one from the Byzantine router that has stale callback information, but this is not a problem. The callback information update is only confirmed if followed up by a SETCLIENTID_CONFIRM { c, t }.

The server awaits confirmation of k via SETCLIENTID_CONFIRM { c, t }.

The server does NOT remove client (lock/share/delegation) state for x.

The server has previously recorded a confirmed { u, x, c, l, s } record such that v != u, l may or may not equal k, and has not recorded any unconfirmed { *, x, *, *, * } record for x. The server records an unconfirmed { v, x, d, k, t } (d != c, t != s).

The server returns { d, t }.

The server awaits confirmation of { d, k } via SETCLIENTID_CONFIRM { d, t }.

The server does NOT remove client (lock/share/delegation) state for x.

The server has previously recorded a confirmed { u, x, c, l, s } record such that v != u, l may or may not equal k, and recorded an unconfirmed { w, x, d, m, t } record such that c != d, t != s, m may or may not equal k, m may or may not equal l, and k may or may not equal l. Whether w == v or w != v makes no difference. The server simply removes the unconfirmed { w, x, d, m, t } record and replaces it with an unconfirmed { v, x, e, k, r } record, such that e != d, e != c, r != t, r != s.

The server returns { e, r }.

The server awaits confirmation of { e, k } via SETCLIENTID_CONFIRM { e, r }.

The server does NOT remove client (lock/share/delegation) state for x.

The server has no confirmed { *, x, *, *, * } for x. It may or may not have recorded an unconfirmed { u, x, c, l, s }, where l may or may not equal k, and u may or may not equal v. Any unconfirmed record { u, x, c, l, * }, regardless whether u == v or l == k, is replaced with an unconfirmed record { v, x, d, k, t } where d != c, t != s.

The server returns { d, t }.

The server awaits confirmation of { d, k } via SETCLIENTID_CONFIRM { d, t }. The server does NOT remove client (lock/share/delegation) state for x.

the server generates the clientid and setclientid_confirm values and has to take care to make sure these values are extremely unlikely to ever get regenerated 🎲✨

8.5. Security Considerations for Inter-server Information Transfer ✨

okay bestie, NFSv4.0 doesn't say how the source and destination servers talk to each other 🫣 but the security stuff still matters for that chat, so here's the tea 🍵

Keep the chat private 🔐: the source and destination servers need to talk securely. That means protecting against someone sneakily changing the data while it's in transit, using either a private network or a security mechanism that ensures integrity. In lots of cases privacy is required too, so if there's no private network, you need a strengthened security mechanism to cover it. 🛡️
Trust is everything 🤝: for the file system migration function to work, the source and destination servers have to trust each other. The details of that trust depend on the specifics of the inter-server transfer protocol, and that protocol is outside the scope of this spec, so it's not our problem here 💅
Extra receipts, but no cheating 🧾: the source server can tell the destination server some security-related info so it can check a client's identity more rigorously. For example, the destination server might reject a SETCLIENTID done with a different principal, or with a different IP address, than the client used before on the source server. BUT the destination server MUST NOT use that info to let the client do anything that wouldn't be allowed otherwise. no exceptions, periodt 🚫

8.6. Security Considerations Revision 🎀

The penultimate paragraph of Section 19 of [RFC7530] should be revised to read as follows. This is the new wording, so read it carefully:

Because the operations SETCLIENTID/SETCLIENTID_CONFIRM are responsible for the release of client state, it is imperative that the principal used for these operations be checked against and match the previous use of these operations. In addition, use of integrity protection is desirable on the SETCLIENTID operation, to prevent an attack whereby a change in the boot instance id (verifier) forces an undesired loss of client state. See Section 5 for further discussion.

9. Security Considerations 🔒

The security considerations of [RFC7530] remain appropriate, with two changes: the modification to the penultimate paragraph specified in Section 8.6 of this document, and the addition of the material in Section 8.5. So go read both of those, bestie 🥹

10. References 📚

10.1. Normative References 💜

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, http://www.rfc-editor.org/info/rfc2119.
[RFC7530] Haynes, T., Ed. and D. Noveck, Ed., "Network File System (NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530, March 2015, http://www.rfc-editor.org/info/rfc7530.

10.2. Informative References 🌸

[INFO-MIGR] Noveck, D., Ed., Shivam, P., Lever, C., and B. Baker, "NFSv4 migration: Implementation experience and spec issues to resolve", Work in Progress, draft-ietf-nfsv4-migration-issues-09, February 2016.
[RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS Version 3 Protocol Specification", RFC 1813, DOI 10.17487/RFC1813, June 1995, http://www.rfc-editor.org/info/rfc1813.
[RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "Network File System (NFS) Version 4 Minor Version 1 Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010, http://www.rfc-editor.org/info/rfc5661.

Acknowledgements 🫶

The editor and authors of this document gratefully acknowledge the contributions of Trond Myklebust of Primary Data and Robert Thurlow of Oracle. We also thank Tom Haynes of Primary Data and Spencer Shepler of Microsoft for their guidance and suggestions. 💖

Special thanks go to members of the Oracle Solaris NFS team, especially Rick Mesta and James Wahlig, for their work implementing an NFSv4.0 migration prototype and identifying many of the issues addressed here. They're literally the MVPs 👑

Authors' Addresses 🦋

David Noveck (editor):

David Noveck (editor)
Hewlett Packard Enterprise
165 Dascomb Road
Andover, MA  01810
United States of America

Phone: +1 978 474 2011
Email: davenoveck@gmail.com

Piyush Shivam:

Piyush Shivam
Oracle Corporation
5300 Riata Park Ct.
Austin, TX  78727
United States of America

Phone: +1 512 401 1019
Email: piyush.shivam@oracle.com

Charles Lever:

Charles Lever
Oracle Corporation
1015 Granger Avenue
Ann Arbor, MI  48104
United States of America

Phone: +1 734 274 2396
Email: chuck.lever@oracle.com

Bill Baker:

Bill Baker
Oracle Corporation
5300 Riata Park Ct.
Austin, TX  78727
United States of America

Phone: +1 512 401 1081
Email: bill.baker@oracle.com

📦 where this page comes from

ietf: rfc7931 2016-07
💖 girlified by claude-haiku-5-5 on 2026-10-18: a language model's rewrite of the original, under its terms

RFC 7931

Internet Engineering Task Force (IETF) 🎀

Abstract

Status of This Memo

Copyright Notice

Table of Contents

1. Introduction ✨

2. Conventions 🫶

3. Definitions 📚

3.1. Terminology 🌸

3.2. Data Type Definitions 📋

4. Background 🎀

5. Client Identity Definition 🪪

5.1. Differences from Replaced Sections

5.2. Client Identity Data Items

5.2.1. Client Identity Structure

5.2.2. Client Identity Shorthand ✨

5.3. Server Release of Client ID 🗑️

5.4. Client ID String Approaches

5.5. Non-uniform Client ID String Approach

5.6. Uniform Client ID String Approach

5.7. Mixing Client ID String Approaches ✨

5.8. Trunking Determination when Using Uniform Client ID Strings 🦋

5.9. Client ID String Construction Details 🎀

6. Locking and Multi-Server Namespace ✨

6.1. Lock State and File System Transitions 🌸

6.1.1. Migration and State 🦋

6.1.1.1. Migration and Client IDs 🧚

6.1.1.2. Migration and State Owner Information ✨

6.1.2. Replication and State 🦋

6.1.3. Notification of Migrated Lease 🦋

6.1.4. Migration and the lease_time Attribute 🕒

7. Server Implementation Considerations 🛠️

7.1. Relation of Locking State Transfer to Other Aspects of File System Motion

7.2. Preventing Locking State Modification during Transfer ✨

8. Additional Changes 💖

8.1. Summary of Additional Changes from Previous Documents 📋

8.2. NFS4ERR_CLID_INUSE Definition 🚨

8.3. NFS4ERR_DELAY Return from RELEASE_LOCKOWNER ✨

8.4. Operation 35: SETCLIENTID -- Negotiate Client ID 👑

8.4.1. SYNOPSIS

8.4.2. ARGUMENT

8.4.3. RESULT

8.4.4. DESCRIPTION

8.4.5. IMPLEMENTATION

8.4.5.1. IMPLEMENTATION (Preparatory Phase)

8.4.5.2. IMPLEMENTATION (Main Phase)

8.5. Security Considerations for Inter-server Information Transfer ✨

8.6. Security Considerations Revision 🎀

9. Security Considerations 🔒

10. References 📚

10.1. Normative References 💜

10.2. Informative References 🌸

Acknowledgements 🫶

Authors' Addresses 🦋