RFC 8435
Parallel NFS (pNFS) Flexible File Layout
Parallel NFS (pNFS) Flexible File Layout 🦋
RFC 8435 💖 Authors: B. Halevy and T. Haynes, both from Hammerspace ✨ Category: Standards Track 👑 ISSN: 2070-1721 Date: August 2018 🌸
Abstract ✨
okay so pNFS lets you split a file up into two pieces: the metadata (which lives on a metadata server) and the data (which lives on a storage device) 🧚 this doc defines the flexible file layout type, which is an extension to pNFS that lets storage devices do only a limited amount of talking with the metadata server and use protocols that already exist 💅 it also adds client-side mirroring so files can get replicated 🫶
Status of This Memo 📜
This is an Internet Standards Track document. ✨
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841. 🎀
Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc8435. 🔥
Copyright Notice ⭐
Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. 🌈
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. 💜
Table of Contents 🍓
1. Introduction ....................................................3
1.1. Definitions ................................................4
1.2. Requirements Language ......................................6
2. Coupling of Storage Devices .....................................6
2.1. LAYOUTCOMMIT ...............................................7
2.2. Fencing Clients from the Storage Device ....................7
2.2.1. Implementation Notes for Synthetic uids/gids ........8
2.2.2. Example of Using Synthetic uids/gids ................9
2.3. State and Locking Models ..................................10
2.3.1. Loosely Coupled Locking Model ......................11
2.3.2. Tightly Coupled Locking Model ......................12
3. XDR Description of the Flexible File Layout Type ...............13
3.1. Code Components Licensing Notice ..........................14
4. Device Addressing and Discovery ................................16
4.1. ff_device_addr4 ...........................................16
4.2. Storage Device Multipathing ...............................17
5. Flexible File Layout Type ......................................18
5.1. ff_layout4 ................................................19
5.1.1. Error Codes from LAYOUTGET .........................23
5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS ...23
5.2. LAYOUTCOMMIT ..............................................24
5.3. Interactions between Devices and Layouts ..................24
5.4. Handling Version Errors ...................................24
6. Striping via Sparse Mapping ....................................25
7. Recovering from Client I/O Errors ..............................25
8. Mirroring ......................................................26
8.1. Selecting a Mirror ........................................26
8.2. Writing to Mirrors ........................................27
8.2.1. Single Storage Device Updates Mirrors ..............27
8.2.2. Client Updates All Mirrors .........................27
8.2.3. Handling Write Errors ..............................28
8.2.4. Handling Write COMMITs .............................28
8.3. Metadata Server Resilvering of the File ...................29
9. Flexible File Layout Type Return ...............................29
9.1. I/O Error Reporting .......................................30
9.1.1. ff_ioerr4 ..........................................30
9.2. Layout Usage Statistics ...................................31
9.2.1. ff_io_latency4 .....................................31
9.2.2. ff_layoutupdate4 ...................................32
9.2.3. ff_iostats4 ........................................33
9.3. ff_layoutreturn4 ..........................................34
10. Flexible File Layout Type LAYOUTERROR .........................35
11. Flexible File Layout Type LAYOUTSTATS .........................35
12. Flexible File Layout Type Creation Hint .......................35
12.1. ff_layouthint4 ...........................................35
13. Recalling a Layout ............................................36
13.1. CB_RECALL_ANY ............................................36
14. Client Fencing ................................................37
15. Security Considerations .......................................37
15.1. RPCSEC_GSS and Security Services .........................39
15.1.1. Loosely Coupled ...................................39
15.1.2. Tightly Coupled ...................................39
16. IANA Considerations ...........................................39
17. References ....................................................40
17.1. Normative References .....................................40
17.2. Informative References ...................................41
Acknowledgments ...................................................42
Authors' Addresses ................................................421. Introduction 🧚
okay so in pNFS, the metadata server hands back layout type structures that describe where the file data actually lives 💖 there are different layout types for different storage systems and for different ways of arranging data on storage devices ✨ this document defines the flexible file layout type, and it's built for file-based data servers that you reach using the NFS protocols: NFSv3 [RFC1813], NFSv4.0 [RFC7530], NFSv4.1 [RFC5661], and NFSv4.2 [RFC7862] 🎀
so the files layout type gives you a global state model, and to get something equivalent, a back-end control protocol might get set up between the metadata server and NFSv4.1+ storage devices 🌈 an implementation can either make up its own proprietary mechanism, or it could define a control protocol in a Standards Track document 💅 the requirements for a control protocol are spelled out in [RFC5661] and clarified in [RFC8434] 🫶
the control protocol in this document is based on NFS 🦋 it does not let stateids get passed between the metadata server and the storage devices, and that's the iconic part 😭 instead, the storage devices get set up so the metadata server has full access rights to the data file system, and then the metadata server uses synthetic ids to control which clients can access individual files 🔥
now, traditional mirroring: the server is the one responsible for replicating, validating, and repairing copies of the data file 🍓 with client-side mirroring, it's a whole different vibe 👑 the metadata server provides a layout that shows the client which mirrors are available, the client picks a mirror to read from, and the client makes sure all writes go to all mirrors 💜 the client only considers a write to have succeeded if every single mirror was updated successfully ⭐ if something goes wrong, the client can use the LAYOUTERROR operation to tell the metadata server, and then the metadata server is the one responsible for repairing the mirrored copies of the file 💀
1.1. Definitions 💅
Okay bestie, here's the glossary 📖 Every term is exact, so you can actually rely on these.
- control communication requirements: the spec for all the info on layouts, stateids, file metadata, and file data that has to get communicated between the metadata server and the storage devices 🫶 Each layout type gets its own separate set of these requirements.
- control protocol: the specific mechanism an implementation of a layout type uses to meet the control communication requirement for that layout type ✨ It doesn't have to be a protocol in the usual sense. Sometimes the same protocol can be both the control protocol and the storage protocol 💜
- client-side mirroring: the setup where the client, not the server, is the one in charge of updating all the mirrored copies of a layout segment 🪞
- (file) data: the part of the file system object that holds the stuff you read or write. It's the contents of the object, not its attributes 📄
- data server (DS): a pNFS server that serves up the file's data when the file system object gets accessed over a file-based protocol 🌸
- fencing: the process where the metadata server stops the storage devices from processing I/O from one specific client to one specific file 🚫
- file layout type: a layout type where the storage devices get accessed over NFS (see Section 13 of [RFC5661]) 🎀
- gid: the group id, a number that tells you which group a file belongs to 🧚
- layout: the info a client uses to reach file data on a storage device 🗺️ It includes the protocol (layout type) and which storage devices to use.
- layout iomode: a grant of either read-only or read/write I/O to the client 🔐
- layout segment: a sub-division of a layout. It can be split by layout iomode (see Sections 3.3.20 and 12.2.9 of [RFC5661]), by a striping pattern (see Section 13.3 of [RFC5661]), or by a requested byte range 🍰
- layout stateid: a 128-bit quantity a server hands back that uniquely identifies the layout state it's providing for one specific layout (describing a layout type and file) (see Section 12.5.2 of [RFC5661]). Section 12.5.3 of [RFC5661] also covers how layout stateids get handled differently from other stateid types 🪪
- layout type: a spec for both the storage protocol used to access the data and the aggregation scheme used to lay out the file data across the underlying storage devices 🧩
- loose coupling: when the control protocol is a storage protocol 🧵
- (file) metadata: the part of the file system object with all the descriptive info about the file, as opposed to the file data itself. Think time of last modification, access time, EOF position, etc. 🗂️
- metadata server (MDS): the pNFS server that provides metadata info for a file system object. It's also in charge of generating, recalling, and revoking layouts for file system objects, doing directory operations, and doing I/O on regular files when clients send those requests to the metadata server itself 👑
- mirror: a copy of a layout segment. If one copy of the mirror gets updated, then all the copies have to get updated too 🪞💕
- recalling a layout: a graceful recall, done via a callback, of a specific layout from the metadata server back to the client 📞 "Graceful" here means the client gets the chance to flush any
WRITEs and so on before it hands the layout back to the metadata server 🌷 - revoking a layout: the metadata server invalidating a specific layout 🚨 Once that happens, the metadata server won't accept any reference to the revoked layout as valid, and a storage device won't accept any client access based on that layout 💀
- resilvering: rebuilding a mirrored copy of a layout segment from a known good copy of that segment 🛠️ It can also be done to make a brand new mirrored copy of the layout segment.
- rsize: the data transfer buffer size used for
READs 📦 - stateid: a 128-bit quantity a server returns that uniquely identifies the set of locking-related state it's providing 🔑 Stateids can designate state tied to open files, byte-range locks, delegations, or layouts.
- storage device: the target clients can direct I/O requests to when they hold an appropriate layout 🎯 See Section 2.1 of [RFC8434] for more on how a data server and a storage device are different.
- storage protocol: the protocol clients use to do I/O operations against the storage device 📡 Each layout type specifies the set of storage protocols it allows.
- tight coupling: an arrangement where the control protocol is one built specifically for control communication 🤝 It might be a proprietary protocol adapted to one particular metadata server, or a protocol based on a Standards Track document.
- uid: the user id, a number that tells you which user owns a file 🙋♀️
- wsize: the data transfer buffer size used for
WRITEs 📬
1.2. Requirements Language 🎀
Okay, so the big BCP 14 key words in this doc are "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL". They get read as described in BCP 14 [RFC2119] [RFC8174], but only when, and only when, they show up in all capitals just like this 🔥
2. Coupling of Storage Devices 🦋
A server implementation can go with either a loosely coupled model or a tightly coupled model between the metadata server and the storage devices. [RFC8434] lays out the general problems pNFS implementations deal with, and this doc breaks down how the new flexible file layout type handles them 💅 To build the tightly coupled model, you have to define a control protocol. Since the flexible file layout doesn't put any special requirements on the client, the control protocol will need to provide:
(1) management of both security and LAYOUTCOMMITs and
(2) a global stateid model and management of these stateids.
When you go with the loosely coupled model, the only control protocol is a version of NFS. That means no ability to provide a global stateid model, and no way to stop clients from using layouts inappropriately 😭 So to make client use work in that setup, this doc specifies how security, state, and locking are supposed to be managed 🧠
2.1. LAYOUTCOMMIT 📝
No matter which coupling model you pick, the metadata server has the responsibility, when it gets a LAYOUTCOMMIT (see Section 18.42 of [RFC5661]), to make sure the semantics of pNFS are respected (see Section 3.1 of [RFC8434]) ✨ That includes requiring that data written to a data storage device is stable before the LAYOUTCOMMIT happens.
It's the client's job to make sure the data file is stable before the metadata server starts asking the storage devices about changes to the file 🫶 If any WRITE to a storage device didn't come back with stable_how equal to FILE_SYNC, then a LAYOUTCOMMIT to the metadata server MUST be preceded by a COMMIT to the storage devices that were written to 🚨 And heads up: if the client hasn't done a COMMIT to the storage device, the LAYOUTCOMMIT might not be synced to the last WRITE operation to that storage device 👀
2.2. Fencing Clients from the Storage Device 🔒
With loosely coupled storage devices, the metadata server uses synthetic uids (user ids) and gids (group ids) for the data file. The uid owner of the data file gets read/write access, and the gid owner gets read-only access 💜 As part of the layout (see ffds_user and ffds_group in Section 5.1), the client is handed the user and group to put in the Remote Procedure Call (RPC) [RFC5531] credentials it needs to access the data file. Fencing a client off works by the metadata server changing the synthetic uid and/or gid owners of the data file on the storage device, which implicitly revokes the outstanding RPC credentials 🌈 A client showing the wrong credential for the access it wants will get an NFS4ERR_ACCESS error 🙅♀️
In this loosely coupled model, the metadata server can't fence off a single client. It's forced to fence off all of them 😩 But here's the plot twist: as the other clients react to the fencing, returning their layouts and trying to get new ones, the metadata server can hand out a new uid and gid so access works again 🦋
It is RECOMMENDED to set up common access control methods at the storage device file system so that only the metadata server root (super user) gets access to the storage device, and to make the owner of all directories holding data files the root user 🛡️ This is a practical model for enforcing access control and fencing off cooperative clients, but it can't protect against malicious clients. So it gives you a level of security equivalent to AUTH_SYS 💀 It is RECOMMENDED that the communication between the metadata server and the storage device be secure from eavesdroppers and man-in-the-middle protocol tampering 🕵️♀️ That security could be physical (for example, the servers are co-located in a physically secure area), encrypted communications, or some other technique 🔐
With tightly coupled storage devices, the metadata server sets the user and group owners, mode bits, and Access Control List (ACL) of the data file to match the metadata file 🪞 The client also has to authenticate with the storage device and go through the same authorization process it would go through via the metadata server 💅 With tight coupling, fencing is the responsibility of the control protocol and isn't described in detail in this doc. However, implementations of the tightly coupled locking model (see Section 2.3) will need a way to stop certain clients from accessing specific files, by invalidating the corresponding stateids on the storage device 🚫 In that case, the client gets an error of NFS4ERR_BAD_STATEID 😭
The client doesn't need to know which model is used between the metadata server and the storage device. It just needs to react consistently to any errors it runs into when talking to the storage device 🧠 It should both return the layout and the error to the metadata server, and ask for a new layout 💖 At that point, the metadata server can either hand out a new layout, hand out no layout (forcing the I/O to go through it), or deny the client any further access to the file 👑
2.2.1. Implementation Notes for Synthetic uids/gids 🎀
okay so the way synthetic uids and gids get picked for fencing on loosely coupled storage devices is strictly an implementation thing 💅 nobody's handing you a rulebook for it. an admin could restrict a range of those ids to the Lightweight Directory Access Protocol (LDAP) uid field [RFC4519] 🧚 and might also be able to pick an id that would never be used to grant access 🙈. then, when the metadata server gets a request to access a file, a SETATTR gets sent to the storage device to set the owner and group of the data file. the user and group might be picked round-robin from the range of available ids 🔁
those ids get sent back to the client as ffds_user and ffds_group, and the client presents them as the RPC credentials to the storage device 🫶. when the client is done with the file and the metadata server knows no other client is accessing it, the metadata server can reset the owner and group to lock the data file back down 🔒
when the metadata server wants to fence off a client, it changes the synthetic uid and/or gid to the restricted ids. note that using a restricted id guarantees a change of owner and at least one id that never gets allowed access 👑
Under an AUTH_SYS security model, synthetic uids and gids of 0 SHOULD be avoided. these typically either grant super access to files on a storage device or get mapped to an anonymous id. in the first case, even if the data file is fenced, the client might still be able to access the file 😬. in the second case, multiple ids might be mapped to the anonymous ids 🌀
2.2.2. Example of Using Synthetic uids/gids 🦋
the user loghyr creates a file "ompha.c" on the metadata server, which then creates a matching data file on the storage device 📁
the metadata server entry may look like:
-rw-r--r-- 1 loghyr staff 1697 Dec 4 11:31 ompha.con the storage device, the file may be assigned some unpredictable synthetic uid/gid to deny access 🙅♀️:
-rw-r----- 1 19452 28418 1697 Dec 4 11:31 data_ompha.cwhen the file is opened on a client and accessed, the user tries to get a layout for the data file. since the layout knows nothing about the user (and doesn't care), it doesn't matter whether loghyr or garbo opens the file 🤷♀️. the client has to present a uid of 19452 to get write permission. if it presents any other value for the uid, then it must give a gid of 28418 to get read access 🔑
if the metadata server decides to fence the file, it should change the uid and/or gid so these values neither match earlier values for that file nor match a predictable change based on an earlier fencing 🚧:
-rw-r----- 1 19453 28419 1697 Dec 4 11:31 data_ompha.cthe set of synthetic gids on the storage device should be picked so there's no mapping in any of the name services the storage device uses, i.e., each group should have no members 🧹
if the layout segment has an iomode of LAYOUTIOMODE4_READ, then the metadata server should return a synthetic uid that is not set on the storage device. only the synthetic gid would be valid 🫧
so the client is totally solely responsible for enforcing file permissions in a loosely coupled model 💅. to let loghyr write, it sends an RPC to the storage device with a credential of 1066:1067. to let garbo read, it sends an RPC to the storage device with a credential of 1067:1067. the value of the uid doesn't matter as long as it isn't the synthetic uid granted when getting the layout ✨
ngl pushing permission checks onto the client might seem like it weakens security 🥺, but the client may already be responsible for enforcing permissions before modifications get sent to a server. with cached writes, the client is always responsible for tracking who is modifying a file and making sure it doesn't coalesce requests from multiple users into one request 🫶
2.3. State and Locking Models 🌸
an implementation can always be deployed as a loosely coupled model. there is, however, no way for a storage device to indicate over an NFS protocol that it can definitively take part in a tightly coupled model 🙃:
- storage devices implementing the NFSv3 and NFSv4.0 protocols are always treated as loosely coupled.
- NFSv4.1+ storage devices that do not return the
EXCHGID4_FLAG_USE_PNFS_DSflag set toEXCHANGE_IDare indicating that they are to be treated as loosely coupled. from the locking viewpoint, they're treated the same way as NFSv4.0 storage devices. - NFSv4.1+ storage devices that do identify themselves with the
EXCHGID4_FLAG_USE_PNFS_DSflag set toEXCHANGE_IDcan potentially be tightly coupled. they'd use a back-end control protocol to implement the global stateid model as described in [RFC5661] 🧠
a storage device would have to be either discovered or advertised over the control protocol to enable a tightly coupled model 📣
2.3.1. Loosely Coupled Locking Model 🧚♀️
when locking-related operations get requested, they're mostly handled by the metadata server, which generates the right stateids. when an NFSv4 version is used as the data access protocol, the metadata server may make stateid-related requests of the storage devices. however, it is not required to do so, and the resulting stateids are known only to the metadata server and the storage device 🤫
given this basic structure, locking-related operations are handled as follows 📋:
- OPENs are dealt with by the metadata server. stateids are selected by the metadata server and associated with the client ID describing the client's connection to the metadata server. the metadata server may need to interact with the storage device to locate the file to be opened, but no locking-related functionality need be used on the storage device.
OPEN_DOWNGRADE and CLOSE only require local execution on the metadata server.
- advisory byte-range locks can be implemented locally on the metadata server. as with OPENs, the stateids associated with byte-range locks are assigned by the metadata server and only used on the metadata server 🔐.
- delegations are assigned by the metadata server, which initiates recalls when conflicting OPENs are processed. no storage device involvement is required 🙌.
TEST_STATEIDandFREE_STATEIDare processed locally on the metadata server, without storage device involvement.
all I/O operations to the storage device are done using the anonymous stateid. so the storage device has no info about the openowner and lockowner responsible for a particular I/O operation. as a result 💀:
- mandatory byte-range locking cannot be supported, because the storage device has no way of telling I/O done for the lock owner apart from I/O done by others.
- enforcement of share reservations is the responsibility of the client. even though I/O is done using the anonymous stateid, the client must ensure that it has a valid stateid associated with the openowner.
if a stateid gets revoked, the metadata server is responsible for preventing client access, since it has no way of being sure the client knows that the stateid in question has been revoked 🚨
as the client never receives a stateid generated by a storage device, there's no client lease on the storage device and no prospect of lease expiration, even when access is via NFSv4 protocols 🫧. clients will have leases on the metadata server. when dealing with lease expiration, the metadata server may need to use fencing to stop revoked stateids from being relied on by a client that doesn't realize they've been revoked 🛑
2.3.2. Tightly Coupled Locking Model
Okay bestie, tightly coupled locking is giving main character energy 💅✨ When locking stuff gets requested, the metadata server is the one doing the heavy lifting and generating the stateids 🎀 Those stateids then have to get passed along to the storage device using control protocol facilities. This doc doesn't dig into those details, so no spoilers 🫶
With that setup in mind, here's how each locking operation gets handled, no cap:
OPENs are handled primarily on the metadata server 😭 The metadata server picks the stateids and ties them to the client ID that describes the client's connection to the metadata server. It also has to talk to the storage device so the storage device can find the file being opened, and so the storage device knows which client and openowner a metadata-server-chosen stateid belongs to.
OPEN_DOWNGRADE and CLOSE start out on the metadata server, but the state change has to be propagated to the storage device too 🦋
- Advisory byte-range locks can be done locally on the metadata server 🌸 Just like with
OPENs, the metadata server assigns the stateids for these locks and they get used right there on the metadata server. Since I/O operations are allowed to present lock stateids, the metadata server needs to be able to tell the storage device which metadata-server-chosen stateid goes with which open stateid. - Mandatory byte-range locks can work when both the metadata server and the storage devices have the right support 🔥 Same as advisory locks, the metadata server assigns these and uses them on its own end. To get mandatory lock enforcement happening on the storage device, the metadata server has to let the storage device know which client, openowner, and lock (that's the lockowner, byte-range, and lock-type) a metadata-server-chosen stateid represents. Since I/O operations can present lock stateids, this info needs to go to all storage devices that I/O might be directed to, not just the ones holding the locked region 👑
- Delegations are assigned by the metadata server, which starts recalls when conflicting
OPENs come in 🍓 Since I/O operations are allowed to present delegation stateids, the metadata server needs to be able to (1) tell the storage device which filehandle and delegation type a metadata-server-chosen stateid goes with, and (2) break that association. TEST_STATEIDgets processed locally on the metadata server, with zero storage device involvement 🧚FREE_STATEIDis processed on the metadata server, but the metadata server needs to be able to pass the request along to the matching storage devices 💜
Here's the tea ☕: since the client will hold and use stateids that are valid on the storage device, there's a client lease on the storage device too, and yes, it can expire 😬 The best move for the storage device is to keep these locks as a courtesy 🫶 But if it doesn't, control protocol facilities need to give a way to sync lock state between the metadata server and the storage device.
Clients also have leases on the metadata server that can expire 💀 When a lease expires, the metadata server would be expected to use control protocol facilities that let it invalidate revoked stateids on the storage device. If the client isn't responsive, the metadata server may need to use fencing so that revoked stateids can't get acted upon by the storage device 🙈
3. XDR Description of the Flexible File Layout Type 🧚
This document contains the External Data Representation (XDR) [RFC4506] description of the flexible file layout type. The XDR description is embedded in this document in a way that makes it super easy to pull out into a ready-to-compile form. You can feed this document into the following shell script to produce the machine-readable XDR description of the flexible file layout type:
#!/bin/sh
grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??'So, if the above script is stored in a file called "extract.sh" and this document is in a file called "spec.txt", then you can do:
sh extract.sh < spec.txt > flex_files_prot.xWhat the script does is remove leading white space from each line, plus a sentinel sequence of "///". ✨
The embedded XDR file header comes next. Subsequent XDR descriptions with the sentinel sequence are embedded throughout the document 💖
Quick heads up: the XDR code in this document depends on types from the NFSv4.1 nfs4_prot.x file [RFC5662]. That includes nfs types that end with a 4, like offset4, length4, etc., as well as more generic types like uint32_t and uint64_t 🌈
3.1. Code Components Licensing Notice
Both the XDR description and the scripts used for extracting the XDR description are Code Components as described in Section 4 of "Trust Legal Provisions (TLP)" [LEGAL]. These Code Components are licensed according to the terms of that document 📜
/// /*
/// * Copyright (c) 2018 IETF Trust and the persons identified
/// * as authors of the code. All rights reserved.
/// *
/// * Redistribution and use in source and binary forms, with
/// * or without modification, are permitted provided that the
/// * following conditions are met:
/// *
/// * - Redistributions of source code must retain the above
/// * copyright notice, this list of conditions and the
/// * following disclaimer.
/// *
/// * - Redistributions in binary form must reproduce the above
/// * copyright notice, this list of conditions and the
/// * following disclaimer in the documentation and/or other
/// * materials provided with the distribution.
/// *
/// * - Neither the name of Internet Society, IETF or IETF
/// * Trust, nor the names of specific contributors, may be
/// * used to endorse or promote products derived from this
/// * software without specific prior written permission.
/// *
/// * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS
/// * AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED
/// * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
/// * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
/// * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
/// * EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
/// * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
/// * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
/// * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
/// * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
/// * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
/// * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
/// * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
/// * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
/// * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
/// *
/// * This code was derived from RFC 8435.
/// * Please reproduce this note if possible.
/// */
///
/// /*
/// * flex_files_prot.x
/// */
///
/// /*
/// * The following include statements are for example only.
/// * The actual XDR definition files are generated separately
/// * and independently and are likely to have a different name.
/// * %#include <nfsv42.x>
/// * %#include <rpc_prot.x>
/// */
///4. Device Addressing and Discovery ✨
Data operations to a storage device require the client to know the network address of the storage device 🗺️ The NFSv4.1+ GETDEVICEINFO operation (Section 18.40 of [RFC5661]) is what the client uses to retrieve that info.
4.1. ff_device_addr4
The ff_device_addr4 data structure is returned by the server as the layout-type-specific opaque field da_addr_body in the device_addr4 structure by a successful GETDEVICEINFO operation 🎀
struct ff_device_versions4 {
uint32_t ffdv_version;
uint32_t ffdv_minorversion;
uint32_t ffdv_rsize;
uint32_t ffdv_wsize;
bool ffdv_tightly_coupled;
};
struct ff_device_addr4 {
multipath_list4 ffda_netaddrs;
ff_device_versions4 ffda_versions<>;
};The ffda_netaddrs field is what locates the storage device 📍 The server MUST set it to a list holding one or more of the device network addresses.
The ffda_versions array lets the metadata server offer the client some choices when it comes to NFS version, minor version, and coupling strength 💅 The ffdv_version and ffdv_minorversion fields represent the NFS protocol to be used to access the storage device. This layout specification defines the semantics for ffdv_versions 3 and 4. If ffdv_version equals 3, then the server MUST set ffdv_minorversion to 0 and ffdv_tightly_coupled to false. The client MUST then access the storage device using the NFSv3 protocol [RFC1813]. If ffdv_version equals 4, then the server MUST set ffdv_minorversion to one of the NFSv4 minor version numbers, and the client MUST access the storage device using NFSv4 with the specified minor version 🦋
Heads up, though: the client might figure out that it can't use any of the configured combinations of ffdv_version, ffdv_minorversion, and ffdv_tightly_coupled once it gets the device list from the metadata server. There's no way to tell the metadata server which device it's version incompatible with, though 😩 But if the client waits until it gets the layout from the metadata server, it can clearly identify the storage device in question at that point (see Section 5.4) 🙈
The ffdv_rsize and ffdv_wsize fields communicate the maximum rsize and wsize supported by the storage device 🍓 Since the storage device can have a different rsize or wsize than the metadata server, these two fields let the metadata server pass that info along on behalf of the storage device.
ffdv_tightly_coupled tells the client whether or not the metadata server is tightly coupled with the storage devices 🌸 Even if the data protocol is at least NFSv4.1, there can still be loose coupling in effect. If ffdv_tightly_coupled is not set, then the client MUST commit writes to the storage devices for the file before sending a LAYOUTCOMMIT to the metadata server. In other words, the writes MUST be committed by the client to stable storage, either by issuing WRITEs with stable_how == FILE_SYNC or by issuing a COMMIT after WRITEs with stable_how != FILE_SYNC (see Section 3.3.7 of [RFC1813]) 🫶
4.2. Storage Device Multipathing 🦋
okay so the flexible file layout type supports multipathing to multiple storage device addresses 💅 bestie, this is giving redundancy AND bandwidth. storage-device-level multipathing is used for bandwidth scaling via trunking, and it also keeps things up when a storage device goes down 😭 the big win is that the client can switch to another storage device address, which might belong to a different storage device exporting the same data stripe unit, without having to bug the metadata server for a new layout 🫶
to make this work, ffda_netaddrs holds an array of one or more storage device network addresses ✨ that array (data type multipath_list4) is a list of storage devices, each identified by a network address, and the same storage device is allowed to show up in the list more than once 🌸
the client is free to use any of the network addresses as a destination for storage device requests 💖 now, if some addresses are less desirable paths to the data than others, the metadata server SHOULD NOT include those addresses in ffda_netaddrs 🙈 if less desirable addresses exist purely to provide failover, the RECOMMENDED move is to offer them in a replacement device-ID-to-device-address mapping or a replacement device ID 🧚
here's the failover flow, iykyk 🎀 when a client finds no response from the storage device on any address in ffda_netaddrs, it SHOULD send a GETDEVICEINFO to try to replace the existing device-ID-to-device-address mappings 💀 and if the metadata server notices that every network path in ffda_netaddrs is unavailable, it SHOULD send a CB_NOTIFY_DEVICEID (but only if the client has said it wants device ID notifications for changed device IDs) so the mappings can point at the available addresses 👑
last but not least, if the device ID itself is going to be replaced, the metadata server SHOULD recall all layouts with that device ID 😭 that forces the client to get new layouts and new device ID mappings through LAYOUTGET and GETDEVICEINFO, and honestly that's a whole glow-up 🌈
generally, if two network addresses show up in ffda_netaddrs, they will designate the same storage device ✨ when the storage device is accessed over NFSv4.1 or a higher minor version, those two addresses can support client ID or session trunking (session trunking is RECOMMENDED) as defined in [RFC5661] 🦋 the two storage device addresses will share the same server owner or the same major ID of the server owner 💜 it's not always necessary for the two addresses to designate the same storage device when trunking is in use 🙈 for example, the data could be read-only and made up of exact replicas 🍓
5. Flexible File Layout Type 🌈
okay bestie, the original layouttype4 introduced in [RFC5662] is modified to look like this 💅
enum layouttype4 {
LAYOUT4_NFSV4_1_FILES = 1,
LAYOUT4_OSD2_OBJECTS = 2,
LAYOUT4_BLOCK_VOLUME = 3,
LAYOUT4_FLEX_FILES = 4
};
struct layout_content4 {
layouttype4 loc_type;
opaque loc_body<>;
};
struct layout4 {
offset4 lo_offset;
length4 lo_length;
layoutiomode4 lo_iomode;
layout_content4 lo_content;
};this document defines the structures associated with the layouttype4 value LAYOUT4_FLEX_FILES ✨ [RFC5661] specifies the loc_body structure as an XDR type "opaque" 🫶 that opaque layout is uninterpreted by the generic pNFS client layers, but it is interpreted by the flexible file layout type implementation 🔥 this section defines the structure of this otherwise opaque value, ff_layout4 👑
5.1. ff_layout4 ✨
okay so ff_layout4 is the struct that describes a layout for the chunk of the data file covered by the current layout segment 💅 It's either one single instance or a whole set of mirrored copies of that chunk, and when mirroring is in effect it protects your data from getting lost across layout segments 🫶
/// const FF_FLAGS_NO_LAYOUTCOMMIT = 0x00000001;
/// const FF_FLAGS_NO_IO_THRU_MDS = 0x00000002;
/// const FF_FLAGS_NO_READ_IO = 0x00000004;
/// const FF_FLAGS_WRITE_ONE_MIRROR = 0x00000008;
/// typedef uint32_t ff_flags4;
///
/// struct ff_data_server4 {
/// deviceid4 ffds_deviceid;
/// uint32_t ffds_efficiency;
/// stateid4 ffds_stateid;
/// nfs_fh4 ffds_fh_vers<>;
/// fattr4_owner ffds_user;
/// fattr4_owner_group ffds_group;
/// };
///
/// struct ff_mirror4 {
/// ff_data_server4 ffm_data_servers<>;
/// };
///
/// struct ff_layout4 {
/// length4 ffl_stripe_unit;
/// ff_mirror4 ffl_mirrors<>;
/// ff_flags4 ffl_flags;
/// uint32_t ffl_stats_collect_hint;
/// };Now, the XDR doesn't spell this out, but every layout4 element in the logr_layout array of LAYOUTGET4res (see Section 18.43.2 of [RFC5661]) describes a layout segment 🧚 So each ff_layout4 also describes a layout segment. A file can be stitched together from more than one layout segment, and each one MAY have different striping parameters 🌈
The ffl_stripe_unit field is the stripe unit size in use for the current layout segment 📏 The number of stripes is whatever the number of elements in ffm_data_servers is, inside each mirror. If there's only one stripe, then ffl_stripe_unit MUST default to zero. The only supported mapping scheme is sparse, and we get into the details in Section 6. Heads up, there's an assumption baked in here: the stripe unit size and the number of stripes are the same across all mirrors 👀
The ffl_mirrors field is the array of mirrored storage devices that provide the storage for the current stripe, see Figure 1 🦋
The ffl_stats_collect_hint field gives the client a hint about how often the server wants it to report LAYOUTSTATS for a file. The time is in seconds ⏱️
+-----------+
| |
| |
| File |
| |
| |
+-----+-----+
|
+------------+------------+
| |
+----+-----+ +-----+----+
| Mirror 1 | | Mirror 2 |
+----+-----+ +-----+----+
| |
+-----------+ +-----------+
|+-----------+ |+-----------+
||+-----------+ ||+-----------+
+|| Storage | +|| Storage |
+| Devices | +| Devices |
+-----------+ +-----------+
Figure 1The ffs_mirrors field is an array of state information for each mirrored copy of the current layout segment. Each element is described by a ff_mirror4 type 🪞
ffds_deviceid gives you the deviceid of the storage device that holds the data file 💾
ffds_fh_vers is an array of filehandles for the data file, matching the NFS versions available on the given storage device. There MUST be exactly as many elements in ffds_fh_vers as there are in ffda_versions. Each element lines up with one particular combination of ffdv_version, ffdv_minorversion, and ffdv_tightly_coupled provided for the device. The array is what lets servers keep different filehandles for different combos of version, minor version, and coupling strength, which is a whole mood 🎀 For how to handle versioning issues between the client and the storage devices, check Section 5.4.
For tight coupling, ffds_stateid gives the stateid the client uses to access the file 🔑 For loose coupling and an NFSv4 storage device, the client has to use an anonymous stateid to do I/O on the storage device. With no control protocol, the metadata server stateid can't be used to provide a global stateid model, so the server MUST set ffds_stateid to be the anonymous stateid 🙈
This spec of ffds_stateid limits both models for NFSv4.x storage protocols:
- loosely coupled model: the stateid has to be an anonymous stateid
- tightly coupled model: the stateid has to be a global stateid
Here's some drama 🍿 A bunch of issues come from the fact that ffds_stateid is a single item while ffds_fh_vers is an array. It's possible for each open file on the storage device to need its own open stateid. But there are already established loosely coupled implementations of this version of the protocol, so these potential issues haven't been addressed here. If it ever becomes important to give the same underlying file multiple stateids, future layout types could be defined to handle it 💜
For loosely coupled storage devices, ffds_user and ffds_group give the synthetic user and group to be used in the RPC credentials the client presents to the storage device when accessing the data files 🧸 For tightly coupled storage devices, the user and group on the storage device are the same as on the metadata server. So if ffdv_tightly_coupled (see Section 4.1) is set, the client MUST ignore both ffds_user and ffds_group 🚫
The allowed values for ffds_user and ffds_group are the ones specified as owner and owner_group in Section 5.9 of [RFC5661], respectively. For NFSv3 compatibility, user and group strings made of decimal numeric values with no leading zeros can get a special interpretation, but only if clients and servers choose to support it 🔢 The receiver may treat such a string as representing the same user that an NFSv3 uid or gid with that numeric value would represent. Also, if you're using Kerberos for security, these values are expected to look like a name@domain string 🪪
ffds_efficiency is the metadata server's evaluation of how effective each mirror is. This is per layout and not per device, since the metric can change based on perceived load, availability to the metadata server, and so on. Higher values mean higher perceived utility 📈 Section 8.1 covers how the client can pick the best mirror to access.
ffl_flags is a bitmap that lets the metadata server tell the client about particular conditions that can result from more or less tight coupling of the storage devices 🧃
FF_FLAGS_NO_LAYOUTCOMMIT: can be set to indicate that the client is not required to send LAYOUTCOMMIT to the metadata server.FF_FLAGS_NO_IO_THRU_MDS: can be set to indicate that the client should not send I/O operations to the metadata server. That means that even if the client could tell there was a network disconnect to a storage device, it should not try to proxy the I/O through the metadata server 🚧FF_FLAGS_NO_READ_IO: can be set to indicate that the client should not send READ requests with layouts of iomodeLAYOUTIOMODE4_RW. Instead, it should request a layout of iomodeLAYOUTIOMODE4_READfrom the metadata server 📖FF_FLAGS_WRITE_ONE_MIRROR: can be set to indicate that the client only needs to update one of the mirrors (see Section 8.2) 🪞
5.1.1. Error Codes from LAYOUTGET
Okay, so [RFC5661] doesn't really tell the client what to do when a LAYOUTGET comes back with NFS4ERR_LAYOUTTRYLATER, NFS4ERR_LAYOUTUNAVAILABLE, or NFS4ERR_DELAY 😭 So here's how it works in this document:
- NFS4ERR_LAYOUTUNAVAILABLE: there's no layout available and the I/O is going to go to the metadata server. It's possible to have had a layout before a recall and not have one after, so don't be shook 💀
- NFS4ERR_LAYOUTTRYLATER: there's some issue stopping the layout from being granted. If the client already has an appropriate layout, it should continue with I/O to the storage devices ✨
- NFS4ERR_DELAY: there's some issue stopping the layout from being granted. If the client already has an appropriate layout, it should not continue with I/O to the storage devices 🛑
Note the difference between the last two! TRYLATER keeps the I/O going, DELAY pauses it. Don't mix them up, bestie 🫶
5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS
Even if the metadata server gives you the FF_FLAGS_NO_IO_THRU_MDS flag, the client can still do I/O to the metadata server 🙃 The flag is just a hint. It tells the client that the metadata server prefers to split the metadata I/O from the data I/O, most likely for performance reasons 🏎️
5.2. LAYOUTCOMMIT 💌
The flexible file layout doesn't use lou_body inside the loca_layoutupdate argument to LAYOUTCOMMIT. If lou_type is LAYOUT4_FLEX_FILES, the lou_body field MUST have a zero length (see Section 18.42.1 of [RFC5661]) 🫧
5.3. Interactions between Devices and Layouts 🦋
In [RFC5661], the file layout type is set up so that the relationship between multipathing and filehandles can result in 0, 1, or N filehandles (see Section 13.3). The reasons for that include clustered servers that share the same filehandle, or allowing multiple read-only copies of the file on the same storage device 🧩 In the flexible file layout type, there's still an array of filehandles, but they're independent of whatever multipathing is being used. If the metadata server wants to provide multiple read-only copies of the same file on the same storage device, then it should provide multiple mirrored instances, each with a different ff_device_addr4 🪞 The client can then tell that, since each of the ffds_fh_vers is different, there are multiple copies of the file available for the current layout segment. Main character energy for sure 👑
5.4. Handling Version Errors 🎀
okay so when the metadata server hands you the ffda_versions array inside ff_device_addr4 (see Section 4.1), you can totally figure out whether you can access a storage device with any of the ffdv_version, ffdv_minorversion, and ffdv_tightly_coupled combos it lists 💅. but here's the tea 🫖: GETDEVICEINFO error reporting is limited (see Section 18.40 in [RFC5661]), so you can't say which specific device you can't talk to over one of the given ffdv_version and ffdv_minorversion combos 😭. the good news is that by putting ff_ioerr4 (see Section 9.1.1) inside either the LAYOUTRETURN (see Section 18.44 of [RFC5661]) or the LAYOUTERROR (see Section 15.6 of [RFC7862] and Section 10 of this document), you can isolate the problematic storage device ✨.
the error code to return for LAYOUTRETURN and/or LAYOUTERROR is NFS4ERR_MINOR_VERS_MISMATCH. it doesn't matter whether the mismatch is a major version (like you can use NFSv3 but not NFSv4) or a minor version (like you can use NFSv4.1 but not NFSv4.2). the error just means that for every combo you were given for ffdv_version and ffdv_minorversion, you can't communicate with that storage device 🙈. you can retry the GETDEVICEINFO to see if the metadata server can come up with a different combo, or you can fall back to doing the I/O through the metadata server 🧚.
6. Striping via Sparse Mapping ✨
while other layout types support both dense and sparse mapping of logical offsets to physical offsets within a file (see, for example, Section 13.4 of [RFC5661]), the flexible file layout type only supports a sparse mapping 🌸.
with sparse mappings, the logical offset within a file (L) is also the physical offset on the storage device. as detailed in Section 13.4.4 of [RFC5661], this creates holes across each storage device that does not contain the current stripe index 🕳️.
L: logical offset within the file
W: stripe width
W = number of elements in ffm_data_servers
S: number of bytes in a stripe
S = W * ffl_stripe_unit
N: stripe number
N = L / S7. Recovering from Client I/O Errors 🩹
the pNFS client may run into errors when it's directly accessing the storage devices. however, recovering from I/O errors is the metadata server's job 👑. when the LAYOUT4_FLEX_FILES layout type is used, the client MUST report the I/O errors to the server at LAYOUTRETURN time, using the ff_ioerr4 structure (see Section 9.1.1) 📮.
the metadata server analyzes the error and figures out what recovery operations it needs, like recovering media failures or reconstructing missing data files 🔍.
the metadata server MUST recall any outstanding layouts so it gets exclusive write access to the stripes being recovered, and so other clients don't hit the same error condition 🚫. in these cases, the server MUST complete recovery before handing out any new layouts to the affected byte ranges.
the client can drop any unwritten data and pass a corresponding error up to the application that started the I/O, but it should really try again first 💀.
although the client implementation has the option to propagate a corresponding error to the application that initiated the I/O operation and drop any unwritten data, the client should attempt to retry the original I/O operation, either by requesting a new layout or by sending the I/O via regular NFSv4.1+ READ or WRITE operations to the metadata server. the client SHOULD attempt to retrieve a new layout and retry the I/O operation using the storage device first, and only retry the I/O operation via the metadata server if the error persists 🫶.
8. Mirroring 🦋
the flexible file layout type has a simple model for mirroring the file data constrained by a layout segment 💜. there's no assumption that each copy of the mirror is stored identically on the storage devices. for example, one device might use compression or deduplication on the data. however, the over-the-wire transfer of the file contents MUST appear identical. heads up, this is a constraint of the selected XDR representation, where each mirrored copy of the layout segment has the same striping pattern (see Figure 1) 🧩.
the metadata server is responsible for deciding how many mirrored copies there are and where each mirror lives. the client may give a hint about how many copies it wants (see Section 12), but the metadata server can ignore that hint. either way, the client has no say over which storage device it gets (which also means the coupling and/or protocol levels used to access the layout segments) or the location of that storage device 🙅♀️.
updating mirrored layout segments is done via client-side mirroring. so the client is responsible for making sure modifications are made on all copies of the layout segments it's told about via the layout 🪞. if a layout segment is being resilvered to a storage device, that mirrored copy won't be in the layout. thus, the metadata server MUST update that copy until the client is presented with it in a layout. if the FF_FLAGS_WRITE_ONE_MIRROR is set in ffl_flags, the client need only update one of the mirrors (see Section 8.2). if the client is writing to the layout segments via the metadata server, then the metadata server MUST update all copies of the mirror. as seen in Section 8.3, during the resilvering, the layout is recalled, and the client has to make modifications via the metadata server 🫠.
8.1. Selecting a Mirror 🎯
when the metadata server grants a layout to a client, it MAY let the client know how fast it expects each mirror to be once the request arrives at the storage devices, via the ffds_efficiency member. the algorithms for calculating that value are left up to metadata server implementations, but factors that could feed into the calculation include storage device speed, physical memory available to the device, operating system version, current load, etc. 🌈
however, what should not be involved in that calculation is a perceived network distance between the client and the storage device. the client is better positioned to make that call, based on past interaction with the storage device over the different available network interfaces between the two. that's because the metadata server might not know about a transient outage between the client and storage device, since it has no presence on the given subnet 🙈.
so, as such, it's the client that decides which mirror to access for reading the file 👑. the requirements for writing to mirrored layout segments are presented below.
8.2. Writing to Mirrors ✍️
8.2.1. Single Storage Device Updates Mirrors
if the FF_FLAGS_WRITE_ONE_MIRROR flag in ffl_flags is set, the client only needs to update one of the copies of the layout segment. for this case, the storage device MUST ensure that all copies of the mirror are updated when any one of the mirrors is updated. if the storage device gets an error when updating one of the mirrors, then it MUST inform the client that the original WRITE had an error. the client then MUST inform the metadata server (see Section 8.2.3). the client's responsibility with respect to COMMIT is explained in Section 8.2.4. the client may choose any one of the mirrors and may use ffds_efficiency as described in Section 8.1 when making this choice 💖.
8.2.2. Client Updates All Mirrors
if the FF_FLAGS_WRITE_ONE_MIRROR flag in ffl_flags is not set, the client is responsible for updating all mirrored copies of the layout segments that it's given in the layout. a single failed update is enough to fail the entire operation 😭. if all but one copy is updated successfully and the last one gives an error, then the client needs to inform the metadata server about the error. the client can use either LAYOUTRETURN or LAYOUTERROR to tell the metadata server that the update failed to that storage device. if the client is updating the mirrors serially, then it SHOULD stop at the first error encountered and report that to the metadata server. if the client is updating the mirrors in parallel, then it SHOULD wait until all storage devices respond, so it can report all errors encountered during the update 🍓.
8.2.3. Handling Write Errors
when the client reports a write error to the metadata server, the metadata server is responsible for deciding whether to remove the errant mirror from the layout, whether the mirror has recovered from some transient error, and so on 🧠. when the client tries to get a new layout, the metadata server tells it the decision through the contents of the layout. the client MUST NOT assume that the contents of the previous layout will match those of the new one. if it has updates that weren't committed to all mirrors, then it MUST resend those updates to all mirrors 🔁.
there's no provision in the protocol for the metadata server to directly determine whether the client has or hasn't recovered from an error. for example, if a storage device was network partitioned from the client and the client reported the error to the metadata server, then the network partition would be repaired, and all of the copies would be successfully updated. there's no mechanism for the client to report that fact, so the metadata server is forced to repair the file across the mirror 🥺.
if the client supports NFSv4.2, it can use LAYOUTERROR and LAYOUTRETURN to give the metadata server hints about the recovery efforts 💅. a LAYOUTERROR on a file is for a non-fatal error. a subsequent LAYOUTRETURN without a ff_ioerr4 means the client successfully replayed the I/O to all mirrors 🎉. any LAYOUTRETURN with a ff_ioerr4 is an error that the metadata server needs to repair. the client MUST be prepared for the LAYOUTERROR to trigger a CB_LAYOUTRECALL if the metadata server determines it needs to start repairing the file 🫣.
8.2.4. Handling Write COMMITs
when stable writes are done to the metadata server or to a single replica (if allowed by the use of FF_FLAGS_WRITE_ONE_MIRROR), it's the responsibility of the receiving node to propagate the written data stably before replying to the client 🔒.
in the corresponding cases where unstable writes are done, the receiving node doesn't have that obligation, although it may choose to asynchronously propagate the updates. however, once a COMMIT is replied to, all replicas must reflect the writes that have been done, and this data must have been committed to stable storage on all replicas 🌟.
to avoid situations where stale data gets read from replicas that haven't had writes propagated 🙃:
- a client that has outstanding unstable writes made to a single node (metadata server or storage device) MUST do all reads from that same node.
- when writes are flushed to the server (for example, to implement close-to-open semantics), a COMMIT must be done by the client to ensure that up-to-date written data will be available irrespective of the particular replica read 🧚♀️.
8.3. Metadata Server Resilvering of the File ✨
okay so the metadata server is allowed to make a brand new mirror of the layout segments whenever it wants 💅 maybe to resilver a copy on a storage device that was down for servicing, or to get a copy onto storage with different performance vibes, etc. 🌸 but here's the tea: the client won't know that new mirror exists, and the metadata server won't know about the updates the client is making to the layout segments 😭 so the metadata server MUST recall the writable layout segment(s) that it is resilvering. 🙈 if the client sends a LAYOUTGET for a writable layout segment that's currently being resilvered, the metadata server can deny that request with an NFS4ERR_LAYOUTUNAVAILABLE 💀 and then the client has to do its I/O through the metadata server instead.
9. Flexible File Layout Type Return 🎀
layoutreturn_file4 is what the LAYOUTRETURN operation uses to send layout-type-specific info to the server. it's defined in Section 18.44.1 of [RFC5661] like this 🦋
/* Constants used for LAYOUTRETURN and CB_LAYOUTRECALL */
const LAYOUT4_RET_REC_FILE = 1;
const LAYOUT4_RET_REC_FSID = 2;
const LAYOUT4_RET_REC_ALL = 3;
enum layoutreturn_type4 {
LAYOUTRETURN4_FILE = LAYOUT4_RET_REC_FILE,
LAYOUTRETURN4_FSID = LAYOUT4_RET_REC_FSID,
LAYOUTRETURN4_ALL = LAYOUT4_RET_REC_ALL
};
struct layoutreturn_file4 {
offset4 lrf_offset;
length4 lrf_length;
stateid4 lrf_stateid;
/* layouttype4 specific data */
opaque lrf_body<>;
};
union layoutreturn4 switch(layoutreturn_type4 lr_returntype) {
case LAYOUTRETURN4_FILE:
layoutreturn_file4 lr_layout;
default:
void;
};
struct LAYOUTRETURN4args {
/* CURRENT_FH: file */
bool lora_reclaim;
layouttype4 lora_layout_type;
layoutiomode4 lora_iomode;
layoutreturn4 lora_layoutreturn;
};okay this is the important bit 🧠 if lora_layout_type is LAYOUT4_FLEX_FILES and lr_returntype is LAYOUTRETURN4_FILE, then the lrf_body opaque value is defined by ff_layoutreturn4 (see Section 9.3). this lets the client report I/O error info or layout usage statistics back to the metadata server, as described below. ngl the data structures are built on NFSv4.2 concepts, but the effective discriminated union (lora_layout_type combined with ff_layoutreturn4) means an NFSv4.1 metadata server can use the data too 🙌
9.1. I/O Error Reporting 🚨
9.1.1. ff_ioerr4
/// struct ff_ioerr4 {
/// offset4 ffie_offset;
/// length4 ffie_length;
/// stateid4 ffie_stateid;
/// device_error4 ffie_errors<>;
/// };
///recall that [RFC7862] defines device_error4 as:
struct device_error4 {
deviceid4 de_deviceid;
nfsstat4 de_status;
nfs_opnum4 de_opnum;
};the ff_ioerr4 structure is used to return error indications for data files that ran into errors during data transfers. these are hints to the metadata server that something's up with that file 👀 for each error, ffie_errors.de_deviceid, ffie_offset, and ffie_length tell you the storage device and the byte range within the file where the error happened, and ffie_errors tells you the operation and type of error. the use of device_error4 is described in Section 15.6 of [RFC7862].
even though the storage device might be accessed via NFSv3 and report NFSv3 errors to the client, the client is responsible for mapping those to proper NFSv4 status codes as de_status. likewise, the NFSv3 operations need to be mapped to their NFSv4 equivalents 🔁
9.2. Layout Usage Statistics 📊
9.2.1. ff_io_latency4
/// struct ff_io_latency4 {
/// uint64_t ffil_ops_requested;
/// uint64_t ffil_bytes_requested;
/// uint64_t ffil_ops_completed;
/// uint64_t ffil_bytes_completed;
/// uint64_t ffil_bytes_not_delivered;
/// nfstime4 ffil_total_busy_time;
/// nfstime4 ffil_aggregate_completion_time;
/// };
///both operation counts and bytes transferred live in the ff_io_latency4. as you'll see in ff_layoutupdate4 (see Section 9.2.2), READ and WRITE operations get aggregated separately. READ operations feed ffl_read in ff_io_latency4. both WRITE and COMMIT operations feed ffl_write. the "requested" counters track what the client is trying to do, and the "completed" counters track what actually got done 💅 there's no requirement that the client only report completed results that have matching requested results from the same reporting period, so don't panic if they don't line up 😌
ffil_bytes_not_delivered tracks the total number of bytes that were requested but not fulfilled because of error conditions 💔 ffil_total_busy_time is the total time spent with outstanding RPC calls ⏳ and ffil_aggregate_completion_time is the sum of all round-trip times for completed RPC calls 🧮
in Section 3.3.1 of [RFC5661], nfstime4 is defined as the number of seconds and nanoseconds since midnight (or zero hour) January 1, 1970 Coordinated Universal Time (UTC). but here it's used in ff_io_latency4 to store the time since the start of the first I/O from the client after it got the layout. so these are supposed to be decoded as a duration, not as a date and time ⏱️ bestie, don't mix them up 🙃
note that LAYOUTSTATS are cumulative, meaning they aren't reset each time the operation is sent 📈 if two LAYOUTSTATS operations for the same file and layout stateid come from the same NFS client and get processed at the same time by the metadata server, then the one with the larger values has the most recent time series data 🏆
9.2.2. ff_layoutupdate4
/// struct ff_layoutupdate4 {
/// netaddr4 ffl_addr;
/// nfs_fh4 ffl_fhandle;
/// ff_io_latency4 ffl_read;
/// ff_io_latency4 ffl_write;
/// nfstime4 ffl_duration;
/// bool ffl_local;
/// };
///ffl_addr tells you which network address the client is connected to on the storage device 🌐 when multipathing is in play, ffl_fhandle shows which read-only copy got selected 🪞 ffl_read and ffl_write carry the latencies for READ and WRITE operations respectively. ffl_duration shows the time period over which the stats were collected ⌛ if ffl_local is true, the I/O was served by the client's cache 🔥 this flag lets the client tell the metadata server about "hot" access to a file it wouldn't normally be allowed to report on. main character behavior, honestly 👑
9.2.3. ff_iostats4
/// struct ff_iostats4 {
/// offset4 ffis_offset;
/// length4 ffis_length;
/// stateid4 ffis_stateid;
/// io_info4 ffis_read;
/// io_info4 ffis_write;
/// deviceid4 ffis_deviceid;
/// ff_layoutupdate4 ffis_layoutupdate;
/// };
///[RFC7862] defines io_info4 as:
struct io_info4 {
uint64_t ii_count;
uint64_t ii_bytes;
};with pNFS, data transfers happen directly between the pNFS client and the storage devices. that means the metadata server has no direct knowledge of the I/O operations being done, so it can't come up with its own statistical info about client I/O to optimize where data gets stored 🫠 ff_iostats4 MAY be used by the client to report I/O statistics back to the metadata server when it returns the layout 💌
since it isn't feasible for the client to report every single I/O that used the layout, the client MAY identify "hot" byte ranges to report I/O statistics for 🔍 how "hot" gets defined or configured, and how big the reported byte range is, are out of the scope of this document 🤷♀️ for client implementers, it's suggested to provide reasonable default values and an optional run-time management interface to control these parameters 🛠️ for example, a client could set the default byte-range resolution to 1 MB in size, with reporting thresholds of 1 MB/second or 10 I/O operations per second 📏
for each byte range, ffis_offset and ffis_length represent the starting offset of the range and the range length in bytes. ffis_read.ii_count, ffis_read.ii_bytes, ffis_write.ii_count, and ffis_write.ii_bytes represent the number of contiguous READ and WRITE I/Os and the matching aggregate number of bytes transferred within the reported byte range 🧁
the combo of ffis_deviceid and ffl_addr uniquely identifies both the storage path and the network route to it 🛣️ and last but not least, ffl_fhandle lets the metadata server tell apart multiple read-only copies of the file on the same storage device 👯
9.3. ff_layoutreturn4
/// struct ff_layoutreturn4 {
/// ff_ioerr4 fflr_ioerr_report<>;
/// ff_iostats4 fflr_iostats_report<>;
/// };
///when data file I/O operations fail, fflr_ioerr_report<> is used to report those errors to the metadata server as an array of ff_ioerr4 elements 📮 each element represents an error that happened on the data file identified by ffie_errors.de_deviceid. if there are no errors to report, the size of the fflr_ioerr_report<> array is set to zero 0️⃣ the client MAY also use fflr_iostats_report<> to report a list of I/O statistics as an array of ff_iostats4 elements. each element represents stats for a particular byte range. byte ranges aren't guaranteed to be disjoint, so they MAY repeat or intersect 🔀
10. Flexible File Layout Type LAYOUTERROR 🙅♀️
if the client is using NFSv4.2 to talk to the metadata server, then instead of waiting for a LAYOUTRETURN to send error info (see Section 9.1), it MAY use LAYOUTERROR (see Section 15.6 of [RFC7862]) to communicate that info 📨 for the flexible file layout type, this means LAYOUTERROR4args is treated the same as ff_ioerr4. same energy, different delivery 💅
11. Flexible File Layout Type LAYOUTSTATS 📈
if the client is using NFSv4.2 to talk to the metadata server, then instead of waiting for a LAYOUTRETURN to send I/O statistics (see Section 9.2), it MAY use LAYOUTSTATS (see Section 15.7 of [RFC7862]) to communicate that info 🚀 for the flexible file layout type, this means LAYOUTSTATS4args.lsa_layoutupdate is overloaded with the same contents as ffis_layoutupdate ✨
12. Flexible File Layout Type Creation Hint 🧚
the layouthint4 type is defined in [RFC5661] like this:
struct layouthint4 {
layouttype4 loh_type;
opaque loh_body<>;
};the layouthint4 structure is how the client passes a hint about the type of layout it'd like created for a particular file 💜 if loh_type is LAYOUT4_FLEX_FILES, then the loh_body opaque value is defined by the ff_layouthint4 type 🫶
12.1. ff_layouthint4
okay so this is the hint struct 🎀 it's a union called ff_mirrors_hint, and it's giving very "maybe, maybe not" energy:
/// union ff_mirrors_hint switch (bool ffmc_valid) {
/// case TRUE:
/// uint32_t ffmc_mirrors;
/// case FALSE:
/// void;
/// };
///
/// struct ff_layouthint4 {
/// ff_mirrors_hint fflh_mirrors_hint;
/// };
///This type is how the client gives hints about the data map it would like ✨ and every parameter is optional, so the client can just fill in the one it cares about 💅
13. Recalling a Layout ✨
While Section 12.5.5 of [RFC5661] covers reasons for recalling a layout that have nothing to do with layout type, the flexible file layout type metadata server should recall outstanding layouts in these situations:
- When the file's security policy changes, i.e., ACLs or permission mode bits are set.
- When the file's layout changes, rendering outstanding layouts invalid.
- When existing layouts are inconsistent with the need to enforce locking constraints.
- When existing layouts are inconsistent with the requirements regarding resilvering as described in Section 8.3.
13.1. CB_RECALL_ANY
The metadata server can use the CB_RECALL_ANY callback operation to tell the client to give back some or all of its layouts 🦋 Section 22.3 of [RFC5661] defines the allowed types of the "NFSv4 Recallable Object Types Registry".
/// const RCA4_TYPE_MASK_FF_LAYOUT_MIN = 16;
/// const RCA4_TYPE_MASK_FF_LAYOUT_MAX = 17;
///
struct CB_RECALL_ANY4args {
uint32_t craa_layouts_to_keep;
bitmap4 craa_type_mask;
};Typically, CB_RECALL_ANY is used to recall client state when the server needs to reclaim resources 🧚 The craa_type_mask bitmap says which types of resources get recalled, and craa_layouts_to_keep says how many of the recalled flexible file layouts the client is allowed to keep. The mask flags for the flexible file layout type look like this:
/// enum ff_cb_recall_any_mask {
/// PNFS_FF_RCA4_TYPE_MASK_READ = 16,
/// PNFS_FF_RCA4_TYPE_MASK_RW = 17
/// };
///These flags represent the iomode of the recalled layouts 💜 In response, the client SHOULD return the layouts of the recalled iomode that it needs the least, keeping at most craa_layouts_to_keep flexible file layouts.
The PNFS_FF_RCA4_TYPE_MASK_READ flag tells the client to return layouts of iomode LAYOUTIOMODE4_READ. Similarly, the PNFS_FF_RCA4_TYPE_MASK_RW flag tells the client to return layouts of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client is told to return layouts of either iomode 🌈
14. Client Fencing 🔥
When clients go quiet and their lease has expired, or when they fail to return recalled layouts within a lease period, the server MAY revoke client layouts and reassign those resources to other clients (see Section 12.5.5 of [RFC5661]). To avoid data corruption, the metadata server MUST fence off the revoked clients from the respective data files as described in Section 2.2. no skipping this one, bestie 🫶
15. Security Considerations 🔐
Okay, this section is serious but also kind of iconic 😭 The combination of components in a pNFS system is required to preserve the security properties of NFSv4.1+ with respect to an entity accessing data via a client. pNFS splits the NFSv4.1+ file system protocol into two parts: the control protocol and the data protocol. Since the control protocol in this document is NFS, its security properties are equivalent to whatever version of NFS is being used. The flexible file layout then divides the data protocol into metadata and data paths. The security properties of the metadata path are equivalent to those of NFSv4.1x (see Sections 1.7.1 and 2.2.1 of [RFC5661]). The security properties of the data path are equivalent to those of the version of NFS used to access the storage device, with the provision that the metadata server is responsible for authenticating client access to the data file. The metadata server gives the client appropriate credentials to reach data files on the storage device, and it is also responsible for revoking that access when needed.
The metadata server enforces the file access control policy at LAYOUTGET time. The client should use RPC authorization credentials when getting the layout for the requested iomode (((LAYOUTIOMODE4_READ or LAYOUTIOMODE4_RW).), and the server verifies the permissions and ACL for those credentials, possibly returning NFS4ERR_ACCESS if the client isn't allowed the requested iomode. If the LAYOUTGET operation succeeds, the client receives a set of credentials as part of the layout, letting it do I/O on the specified data files that match the requested iomode. When the client does I/O on behalf of its local users, it MUST authenticate and authorize that user by issuing the respective OPEN and ACCESS calls to the metadata server, similar to how NFSv4 data delegations work.
The combination of filehandle, synthetic uid, and gid in the layout is how the metadata server enforces access control to the data server 👑 The client only has access to filehandles of file objects, not directory objects. So, given a filehandle in a layout, it isn't possible to guess the parent directory filehandle. And since the data file permissions only allow the given synthetic uid read/write permission and the given synthetic gid read permission, knowing the synthetic ids of one file doesn't necessarily give access to any other data file on the storage device.
The metadata server can also deny access at any time by fencing the data file, which means changing the synthetic ids. That forces the client to return its current layout and get a new one if it still wants to do I/O to the data file ✨
If access is allowed, the client uses the matching (read-only or read/write) credentials to do I/O operations at the data file's storage devices. When the metadata server gets a request to change a file's permissions or ACL, it SHOULD recall all layouts for that file, and then it MUST fence off any clients still holding outstanding layouts for the respective files. It does this by implicitly invalidating the previously distributed credential on all data files that make up the file in question. It is REQUIRED that this happens before the server commits to the new permissions and/or ACL. By requesting new layouts, clients reauthorize access against the modified access control metadata. Recalling the layouts in this case is meant to stop clients from getting an error on I/Os done after they were fenced off 💀
15.1. RPCSEC_GSS and Security Services
Because of the special use of principals within the loosely coupled model, the issues are different depending on the coupling model 🙈
15.1.1. Loosely Coupled
RPCSEC_GSS version 3 (RPCSEC_GSSv3) [RFC7861] has facilities that would let it authorize the client to the storage device on behalf of the metadata server. Doing that would require that the metadata server, storage device, and client each implement RPCSEC_GSSv3, using an RPC-application-defined structured privilege assertion in the way described in Section 4.9.1 of [RFC7862]. The specifics needed to do this aren't described in this document. That's mainly because any such specification would need a ton of implementation work across a wide range of storage devices, which would be unlikely to produce a widely usable specification for a long time 😩
So, the layout type described in this document won't support using RPCSEC_GSS together with the loosely coupled model. However, future layout types could be specified that would allow such support, either through RPCSEC_GSSv3 or in other ways ✨
15.1.2. Tightly Coupled
With tight coupling, the principal used to access the metadata file is exactly the same as the one used to access the data file. The storage device can use the control protocol to validate any RPC credentials. Because of that, there are no security issues related to using RPCSEC_GSS with a tightly coupled system. For example, if Kerberos V5 Generic Security Service Application Program Interface (GSS-API) [RFC4121] is used as the security mechanism, then the storage device could use a control protocol to validate the RPC credentials to the metadata server 🎀
16. IANA Considerations 🌸
[RFC5661] introduced the "pNFS Layout Types Registry". New layout type numbers in this registry need to be assigned by IANA. This document defines the protocol associated with an existing layout type number: LAYOUT4_FLEX_FILES. See Table 1.
+--------------------+------------+----------+-----+----------------+
| Layout Type Name | Value | RFC | How | Minor Versions |
+--------------------+------------+----------+-----+----------------+
| LAYOUT4_FLEX_FILES | 0x00000004 | RFC 8435 | L | 1 |
+--------------------+------------+----------+-----+----------------+Table 1: Layout Type Assignments
[RFC5661] also introduced the "NFSv4 Recallable Object Types Registry". This document defines new recallable objects for RCA4_TYPE_MASK_FF_LAYOUT_MIN and RCA4_TYPE_MASK_FF_LAYOUT_MAX (see Table 2) 🦋
+------------------------------+-------+--------+-----+-------------+
| Recallable Object Type Name | Value | RFC | How | Minor |
| | | | | Versions |
+------------------------------+-------+--------+-----+-------------+
| RCA4_TYPE_MASK_FF_LAYOUT_MIN | 16 | RFC | L | 1 |
| | | 8435 | | |
| RCA4_TYPE_MASK_FF_LAYOUT_MAX | 17 | RFC | L | 1 |
| | | 8435 | | |
+------------------------------+-------+--------+-----+-------------+Table 2: Recallable Object Type Assignments
17. References 📚
(the reference list picks up in the next part 🎀)
17.1. Normative References 📚
[LEGAL] IETF Trust, "Trust Legal Provisions (TLP)",
<https://trustee.ietf.org/trust-legal-provisions.html>.
[RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS
Version 3 Protocol Specification", RFC 1813,
DOI 10.17487/RFC1813, June 1995,
<https://www.rfc-editor.org/info/rfc1813>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC4121] Zhu, L., Jaganathan, K., and S. Hartman, "The Kerberos
Version 5 Generic Security Service Application Program
Interface (GSS-API) Mechanism: Version 2", RFC 4121,
DOI 10.17487/RFC4121, July 2005,
<https://www.rfc-editor.org/info/rfc4121>.
[RFC4506] Eisler, M., Ed., "XDR: External Data Representation
Standard", STD 67, RFC 4506, DOI 10.17487/RFC4506, May
2006, <https://www.rfc-editor.org/info/rfc4506>.
[RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol
Specification Version 2", RFC 5531, DOI 10.17487/RFC5531,
May 2009, <https://www.rfc-editor.org/info/rfc5531>.
[RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed.,
"Network File System (NFS) Version 4 Minor Version 1
Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010,
<https://www.rfc-editor.org/info/rfc5661>.
[RFC5662] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed.,
"Network File System (NFS) Version 4 Minor Version 1
External Data Representation Standard (XDR) Description",
RFC 5662, DOI 10.17487/RFC5662, January 2010,
<https://www.rfc-editor.org/info/rfc5662>.
[RFC7530] Haynes, T., Ed. and D. Noveck, Ed., "Network File System
(NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530,
March 2015, <https://www.rfc-editor.org/info/rfc7530>.
[RFC7861] Adamson, A. and N. Williams, "Remote Procedure Call (RPC)
Security Version 3", RFC 7861, DOI 10.17487/RFC7861,
November 2016, <https://www.rfc-editor.org/info/rfc7861>.
[RFC7862] Haynes, T., "Network File System (NFS) Version 4 Minor
Version 2 Protocol", RFC 7862, DOI 10.17487/RFC7862,
November 2016, <https://www.rfc-editor.org/info/rfc7862>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8434] Haynes, T., "Requirements for Parallel NFS (pNFS) Layout
Types", RFC 8434, DOI 10.17487/RFC8434, August 2018,
<https://www.rfc-editor.org/info/rfc8434>.17.2. Informative References 🌸
[RFC4519] Sciberras, A., Ed., "Lightweight Directory Access Protocol
(LDAP): Schema for User Applications", RFC 4519,
DOI 10.17487/RFC4519, June 2006,
<https://www.rfc-editor.org/info/rfc4519>.Acknowledgments 🫶
okay bestie, time to give the people who made this doc happen their flowers 💐
- Early draft versions got miscellaneous comments from Matt W. Benjamin, Adam Emerson, J. Bruce Fields, and Lev Solomonov.
- The final draft versions got miscellaneous comments from Anand Ganesh, Robert Wipfel, Gobikrishnan Sundharraj, Trond Myklebust, Rick Macklem, and Jim Sermersheim.
- Idan Kedar caught a nasty bug in the interaction of client-side mirroring and the minor versioning of devices. 🐛
- Dave Noveck gave comprehensive reviews during the working group last calls, and he also rewrote Section 2.3. He's literally the MVP 👑
- Olga Kornievskaia made a convincing case against using a credential versus a principal in the fencing approach. Andy Adamson and Benjamin Kaduk helped sharpen the focus. ✨
- Benjamin Kaduk and Olga Kornievskaia helped provide concrete scenarios for loosely coupled security mechanisms. In the end, Olga proved that as defined, the loosely coupled model would not work with
RPCSEC_GSS. Iconic behavior, honestly 💀 - Tigran Mkrtchyan provided the use case for not allowing the client to proxy the I/O through the data server. 🦋
- Rick Macklem provided the use case for only writing to a single mirror. 🧚
Authors' Addresses 💜
Benny Halevy
Email: bhalevy@gmail.com
Thomas Haynes
Hammerspace
4300 El Camino Real Ste 105
Los Altos, CA 94022
United States of America
Email: loghyr@gmail.com📦 where this page comes from
- ietf: rfc8435 2018-08
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms