RFC 5663
Parallel NFS (pNFS) Block/Volume Layout
Parallel NFS (pNFS) Block/Volume Layout 🎀
Okay bestie, let's get into RFC 5663 💖. It's an Internet Standards Track document (Category: Standards Track) from the IETF, with ISSN 2070-1721, dated January 2010. It was written by D. Black and S. Fridella from EMC Corporation, and J. Glasgow from Google. The title block doesn't list anything it obsoletes or updates, so it's a fresh one 🦋.
Abstract ✨
So here's the tea 🫖: Parallel NFS (pNFS) extends Network File Sharing version 4 (NFSv4) so clients can read and write file data straight from the storage the NFSv4 server uses. Skipping the server for data access can boost both performance and parallelism 🚀. The catch is that clients need extra functionality to do it, and some of that depends on what kind of storage is involved. The main pNFS operations document covers the storage-agnostic extensions to NFS, and this document covers the add-ons for block- and volume-based storage, mostly the data structures 🧚.
Status of This Memo 💅
This is an Internet Standards Track document.
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community, got public review, and was approved for publication by the Internet Engineering Steering Group (IESG). For more on Internet Standards, check Section 2 of RFC 5741.
For the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc5663 🔥.
Copyright Notice 🌈
Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please read these carefully, because they spell out your rights and restrictions. Code Components pulled out of this document must include the Simplified BSD License text described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as the Simplified BSD License describes.
Table of Contents 📚
1. Introduction ....................................................4
1.1. Conventions Used in This Document ..........................4
1.2. General Definitions ........................................5
1.3. Code Components Licensing Notice ...........................5
1.4. XDR Description ............................................5
2. Block Layout Description ........................................7
2.1. Background and Architecture ................................7
2.2. GETDEVICELIST and GETDEVICEINFO ............................9
2.2.1. Volume Identification ...............................9
2.2.2. Volume Topology ....................................10
2.2.3. GETDEVICELIST and GETDEVICEINFO deviceid4 ..........12
2.3. Data Structures: Extents and Extent Lists .................12
2.3.1. Layout Requests and Extent Lists ...................15
2.3.2. Layout Commits .....................................16
2.3.3. Layout Returns .....................................16
2.3.4. Client Copy-on-Write Processing ....................17
2.3.5. Extents are Permissions ............................18
2.3.6. End-of-file Processing .............................20
2.3.7. Layout Hints .......................................20
2.3.8. Client Fencing .....................................21
2.4. Crash Recovery Issues .....................................23
2.5. Recalling Resources: CB_RECALL_ANY ........................23
2.6. Transient and Permanent Errors ............................24
3. Security Considerations ........................................24
4. Conclusions ....................................................26
5. IANA Considerations ............................................26
6. Acknowledgments ................................................26
7. References .....................................................27
7.1. Normative References ......................................27
7.2. Informative References ....................................271. Introduction ✨
Here's the big picture 🖼️. Figure 1 shows the overall architecture of a Parallel NFS (pNFS) system:
+-----------+
|+-----------+ +-----------+
||+-----------+ | |
||| | NFSv4.1 + pNFS | |
+|| Clients |<------------------------------>| Server |
+| | | |
+-----------+ | |
||| +-----------+
||| |
||| |
||| Storage +-----------+ |
||| Protocol |+-----------+ |
||+----------------||+-----------+ Control |
|+-----------------||| | Protocol|
+------------------+|| Storage |------------+
+| Systems |
+-----------+
Figure 1: pNFS ArchitectureThe overall idea is that pNFS-enhanced clients get enough info from the server to reach the underlying storage (on the storage systems) directly 💁. For more, see the pNFS portion of [NFSv4.1]. This document is all about pNFS clients talking to storage systems over block- and volume-based storage protocols, like the Small Computer System Interface (SCSI) protocol family. That family includes parallel SCSI, Fibre Channel Protocol (FCP) for Fibre Channel, Internet SCSI (iSCSI), Serial Attached SCSI (SAS), and Fibre Channel over Ethernet (FCoE) 🔌. This class of storage is called block/volume storage. The Server to Storage System protocol, called the "Control Protocol", isn't something this document needs to make interoperable, but when clients use block/volume protocols, it'll usually be a block/volume protocol too.
1.1. Conventions Used in This Document 📝
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].
1.2. General Definitions 🫧
Here are some definitions so you have the context you need 🧠.
Byte
This document defines a byte as an octet, which is a datum exactly 8 bits long.
Client
The "client" is the entity that accesses the NFS server's resources. It might be an application with the logic to reach the NFS server directly. It could also be the traditional operating system client that gives remote file system services to a bunch of applications.
Server
The "server" is the entity in charge of coordinating client access to a set of file systems, and it's identified by a server owner 👑.
1.3. Code Components Licensing Notice ⚖️
The external data representation (XDR) description and the scripts for extracting it are Code Components, as described in Section 4 of "Legal Provisions Relating to IETF Documents" [LEGAL]. These Code Components are licensed under the terms of Section 4 of "Legal Provisions Relating to IETF Documents".
1.4. XDR Description 🧩
This document has the XDR ([XDR]) description of the NFSv4.1 block layout protocol. The XDR description is embedded in a way that makes it easy to pull out into a ready-to-compile form. You can feed this document into the following shell script to get the machine-readable XDR description of the NFSv4.1 block layout:
#!/bin/sh
grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??'So if you save that script as extract.sh and this document as spec.txt, you just run:
sh extract.sh < spec.txt > nfs4_block_layout_spec.x
What the script does is strip leading white space and a sentinel sequence of /// from each matching line 🪄.
The embedded XDR file header comes first, and the other pieces are embedded throughout the document:
/// /*
/// * This code was derived from RFC 5663.
/// * Please reproduce this note if possible.
/// */
/// /*
/// * Copyright (c) 2010 IETF Trust and the persons identified
/// * as the document authors. All rights reserved.
/// *
/// * Redistribution and use in source and binary forms, with
/// * or without modification, are permitted provided that the
/// * following conditions are met:
/// *
/// * - Redistributions of source code must retain the above
/// * copyright notice, this list of conditions and the
/// * following disclaimer.
/// *
/// * - Redistributions in binary form must reproduce the above
/// * copyright notice, this list of conditions and the
/// * following disclaimer in the documentation and/or other
/// * materials provided with the distribution.
/// *
/// * - Neither the name of Internet Society, IETF or IETF
/// * Trust, nor the names of specific contributors, may be
/// * used to endorse or promote products derived from this
/// * software without specific prior written permission.
/// *
/// * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS
/// * AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED
/// * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
/// * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
/// * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
/// * EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
/// * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
/// * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
/// * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
/// * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
/// * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
/// * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
/// * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
/// * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
/// * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
/// */
///
/// /*
/// * nfs4_block_layout_prot.x
/// */
///
/// %#include "nfsv41.h"
///The XDR code in this document depends on types from the nfsv41.x file 🎀. That includes the NFS types that end in a 4, like offset4 and length4, plus more generic types like uint32_t and uint64_t.
2. Block Layout Description ✨
2.1. Background and Architecture 🧚
Okay so the core idea behind block/volume storage is a storage volume that's just a sequential series of fixed-size blocks. Think of it as a logical disk 💾. The storage system can back it with a physical disk, a portion of a physical disk, or something more complex (e.g., concatenation, striping, RAID, and combinations thereof) involving multiple physical disks or portions thereof. It's giving layers 🎀
A pNFS layout for this block/volume class of storage is responsible for mapping from an NFS file (or portion of a file) to the blocks of storage volumes that contain the file. The blocks are described as extents with 64-bit offsets and lengths, using the existing NFSv4 offset4 and length4 types. Clients must be able to do I/O to the block extents without affecting extra areas of storage (especially important for writes), so extents MUST be aligned to 512-byte boundaries, and writable extents MUST be aligned to the block size the NFSv4 server uses to manage the actual file system (4 kilobytes and 8 kilobytes are common block sizes) ✨. That block size is available as the NFSv4.1 layout_blksize attribute. [NFSv4.1]. Readable extents SHOULD be aligned to the block size used by the NFSv4 server. But to support legacy file systems with fragments, alignment to 512-byte boundaries is acceptable 🫶
The pNFS operation for requesting a layout (LAYOUTGET) includes the layoutiomode4 loga_iomode argument, which says whether the layout is for read-only or read-write use 💜. A read-only layout may contain holes that are read as zero, whereas a read-write layout will contain allocated but un-initialized storage in those holes (read as zero, and can be written by the client). This document also supports client participation in copy-on-write (e.g., for file systems with snapshots) by giving both read-only and un-initialized storage for the same range in a layout. Reads start on the read-only storage and writes go to the un-initialized storage. After the first write initializes that storage, all reads go to the now-initialized writable storage, and the corresponding read-only storage is no longer used. Iconic plot twist 😭
The block/volume layout solution expands the security responsibilities of pNFS clients, and there are a bunch of environments where the mandatory to implement security properties for NFS can't be satisfied. Here are the extra security responsibilities the client picks up, and the full discussion is in Section 3, "Security Considerations" 🔥
- Typically, storage area network (SAN) disk arrays and SAN protocols provide access control mechanisms (e.g., Logical Unit Number (LUN) mapping and/or masking), which operate at the granularity of individual hosts, not individual blocks. So block-based protection has to come from the client software 🙈
- Similarly, SAN disk arrays and SAN protocols typically can't validate NFS locks that apply to file regions. For instance, if a file is covered by a mandatory read-only lock, the server can make sure only readable layouts for that file get granted to pNFS clients. But it's up to each pNFS client to make sure the readable layout is only used to service read requests, and doesn't allow writes to the existing parts of the file 💀
Since block/volume storage systems generally can't enforce this kind of file-based security, pNFS block/volume storage layouts SHOULD NOT be used in environments where pNFS clients can't be trusted to enforce those policies. Not the vibe 🚫
2.2. GETDEVICELIST and GETDEVICEINFO 🦋
2.2.1. Volume Identification 🔍
Storage systems like storage arrays can have multiple physical network ports that don't need to be connected to a common network. So a pNFS client can end up with simultaneous multipath access to the same storage volumes through different ports on different networks 🌈
The networks might not even be the same technology. For example, you can reach the same volume over both iSCSI and Fibre Channel, which means network addresses are hard to use for volume identification 😩. So this pNFS block layout identifies storage volumes by content, for example by providing a way to match (unique portions of) labels used by volume managers. Volume identification happens by matching one or more opaque byte sequences to specific parts of the stored data. Any block pNFS system using this layout MUST support a means of content-based unique volume identification that can be employed via the data structure given here 🧚
/// struct pnfs_block_sig_component4 { /* disk signature component */
/// int64_t bsc_sig_offset; /* byte offset of component
/// on volume*/
/// opaque bsc_contents<>; /* contents of this component
/// of the signature */
/// };
///Be careful here, bestie ⚠️ The opaque bsc_contents field in the pnfs_block_sig_component4 structure MUST NOT be interpreted as a zero-terminated string, because it can contain embedded zero-valued bytes. There are no restrictions on alignment (neither bsc_sig_offset nor the length has to be a multiple of 4, so no stress there 💖). The bsc_sig_offset is a signed quantity. When it's positive, it's a byte offset from the start of the volume. When it's negative, it's a byte offset from the end of the volume.
Negative offsets are allowed to simplify the client implementation on systems where the device label sits at a fixed offset from the end of the volume. But if the server uses negative offsets to describe the signature, then the client and server MUST NOT see different volume sizes. Negative offsets SHOULD NOT be used in systems that resize volumes dynamically, unless care is taken to make sure the device label is always present at the offset from the end of the volume as the clients see it. Dynamic resizing with negative offsets is a whole mess, so be warned 🙈
A signature is an array of up to "PNFS_BLOCK_MAX_SIG_COMP" (defined below) signature components. The client MUST NOT assume that all signature components are co-located within a single sector on a block device. Don't assume the components are all snuggled together, babe 🫣
The pNFS client block layout driver uses this volume identification to map pnfs_block_volume_type4 PNFS_BLOCK_VOLUME_SIMPLE deviceid4s to its local view of a LUN 👑
2.2.2. Volume Topology 🌸
The pNFS block server volume topology is expressed as an arbitrary combination of base volume types, listed in the following data structures. The individual components of the topology sit in an array, and components can refer to other components by using array indices. It's a whole family tree of volumes 🌳
/// enum pnfs_block_volume_type4 {
/// PNFS_BLOCK_VOLUME_SIMPLE = 0, /* volume maps to a single
/// LU */
/// PNFS_BLOCK_VOLUME_SLICE = 1, /* volume is a slice of
/// another volume */
/// PNFS_BLOCK_VOLUME_CONCAT = 2, /* volume is a
/// concatenation of
/// multiple volumes */
/// PNFS_BLOCK_VOLUME_STRIPE = 3 /* volume is striped across
/// multiple volumes */
/// };
///
/// const PNFS_BLOCK_MAX_SIG_COMP = 16;/* maximum components per
/// signature */
/// struct pnfs_block_simple_volume_info4 {
/// pnfs_block_sig_component4 bsv_ds<PNFS_BLOCK_MAX_SIG_COMP>;
/// /* disk signature */
/// };
///
///
/// struct pnfs_block_slice_volume_info4 {
/// offset4 bsv_start; /* offset of the start of the
/// slice in bytes */
/// length4 bsv_length; /* length of slice in bytes */
/// uint32_t bsv_volume; /* array index of sliced
/// volume */
/// };
///
/// struct pnfs_block_concat_volume_info4 {
/// uint32_t bcv_volumes<>; /* array indices of volumes
/// which are concatenated */
/// };
///
/// struct pnfs_block_stripe_volume_info4 {
/// length4 bsv_stripe_unit; /* size of stripe in bytes */
/// uint32_t bsv_volumes<>; /* array indices of volumes
/// which are striped across --
/// MUST be same size */
/// };
///
/// union pnfs_block_volume4 switch (pnfs_block_volume_type4 type) {
/// case PNFS_BLOCK_VOLUME_SIMPLE:
/// pnfs_block_simple_volume_info4 bv_simple_info;
/// case PNFS_BLOCK_VOLUME_SLICE:
/// pnfs_block_slice_volume_info4 bv_slice_info;
/// case PNFS_BLOCK_VOLUME_CONCAT:
/// pnfs_block_concat_volume_info4 bv_concat_info;
/// case PNFS_BLOCK_VOLUME_STRIPE:
/// pnfs_block_stripe_volume_info4 bv_stripe_info;
/// };
///
/// /* block layout specific type for da_addr_body */
/// struct pnfs_block_deviceaddr4 {
/// pnfs_block_volume4 bda_volumes<>; /* array of volumes */
/// };
///The pnfs_block_deviceaddr4 data structure is what lets you encode arbitrarily complex nested volume structures. The aggregations allowed are stripes, concatenations, and slices. Here's the tea ☕ The volume topology expressed in pnfs_block_deviceaddr4 always resolves to a set of pnfs_block_volume_type4 PNFS_BLOCK_VOLUME_SIMPLE volumes. The array of volumes is ordered so the root of the volume hierarchy is the last element of the array. Concat, slice, and stripe volumes MUST refer to volumes defined by lower-indexed elements of the array, so no backwards references, periodt 💅
The pnfs_block_device_addr4 data structure is returned by the server as the storage-protocol-specific opaque field da_addr_body in the device_addr4 structure, from a successful GETDEVICEINFO operation [NFSv4.1].
As mentioned above, all device_addr4 structures eventually resolve to a set of volumes of type PNFS_BLOCK_VOLUME_SIMPLE. Each of these volumes is uniquely identified by a set of signature components. Complicated volume hierarchies can be made of dozens of volumes, each with several signature components, so the device address might need several kilobytes 🍓 The client SHOULD be prepared to allocate a large buffer to hold the result. If the server returns NFS4ERR_TOOSMALL, the client SHOULD allocate a buffer of at least gdir_mincount_bytes to hold the expected result, and then retry the GETDEVICEINFO request. Big buffer energy, no notes 🎀
2.2.3. GETDEVICELIST and GETDEVICEINFO deviceid4 🎀
okay so when the server answers a GETDEVICELIST request, it typically will hand back just one deviceid4 in the gdlr_deviceid_list array 🫶 why? Because that one deviceid4, when you pass it to GETDEVICEINFO, gives you a device_addr4 that encodes the ENTIRE volume hierarchy. one id, whole picture, iconic ✨
copy-on-write file systems are the plot twist 😭 there the gdlr_deviceid_list array may contain two deviceid4s: one for the read-only volume hierarchy and one for the writable one. there's no required ordering between the readable and writable IDs in the array, because each volume is uniquely identified by its deviceid4, and layouts refer to them by that deviceid4. so the order doesn't matter, the ID does 💅
another time you'll get multiple device items is when the file handle is the root of a namespace that spans several physical file systems on the server, each with its own volume hierarchy 🌈 in that case, the server implementation may return either a list of the device IDs used by each of those physical file systems, or it may return an empty list. choice is yours, boo 🙈
every deviceid4 that comes back from a successful GETDEVICELIST operation is a shorthand ID that points at the whole volume topology 🦋 those device IDs, plus the device IDs that show up in extents of a LAYOUTGET operation, can all be fed into GETDEVICEINFO. decoding the pnfs_block_deviceaddr4 gives you a flat ordering of data blocks mapped to PNFS_BLOCK_VOLUME_SIMPLE volumes. and once you combine that with the mapping to a client LUN from Section 2.2.1 "Volume Identification", a logical volume offset can be mapped to a block on a pNFS client LUN [NFSv4.1]. the map is complete, bestie 🗺️✨
2.3. Data Structures: Extents and Extent Lists 💅
okay this part is where the layout actually gets built 🧱 a pNFS block layout is a list of extents inside a flat array of data blocks in a logical volume. if you want the details of the volume topology, you use the GETDEVICEINFO operation (see the discussion of volume identification in Section 2.2 above). the block layout describes the individual block extents on the volume that make up the file, and the offsets and lengths in an extent are measured in bytes 📏
here's the state enum and the structs, verbatim because they are not messing around 🔥
/// enum pnfs_block_extent_state4 {
/// PNFS_BLOCK_READ_WRITE_DATA = 0,/* the data located by this
/// extent is valid
/// for reading and writing. */
/// PNFS_BLOCK_READ_DATA = 1, /* the data located by this
/// extent is valid for reading
/// only; it may not be
/// written. */
/// PNFS_BLOCK_INVALID_DATA = 2, /* the location is valid; the
/// data is invalid. It is a
/// newly (pre-) allocated
/// extent. There is physical
/// space on the volume. */
/// PNFS_BLOCK_NONE_DATA = 3 /* the location is invalid.
/// It is a hole in the file.
/// There is no physical space
/// on the volume. */
/// };
///
/// struct pnfs_block_extent4 {
/// deviceid4 bex_vol_id; /* id of logical volume on
/// which extent of file is
/// stored. */
/// offset4 bex_file_offset; /* the starting byte offset in
/// the file */
/// length4 bex_length; /* the size in bytes of the
/// extent */
/// offset4 bex_storage_offset; /* the starting byte offset
/// in the volume */
/// pnfs_block_extent_state4 bex_state;
/// /* the state of this extent */
/// };
///
/// /* block layout specific type for loc_body */
/// struct pnfs_block_layout4 {
/// pnfs_block_extent4 blo_extents<>;
/// /* extents which make up this
/// layout. */
/// };
///so the block layout is a list of extents that map the logical regions of the file to physical spots on a volume 🗂️ the bex_storage_offset field in each extent tells you where you are on the logical volume named by that extent's bex_vol_id. and the bex_vol_id is itself shorthand for the whole topology of the logical volume the file lives on 🌸 the client is responsible for translating that logical offset into an offset on the right underlying SAN logical unit. most of the time, every extent in a layout lives on the same volume and has the same bex_vol_id. the exception is copy-on-write file systems, where the PNFS_BLOCK_READ_DATA extents can have a different bex_vol_id from the writable extents 👀
each extent maps a logical region of the file onto a chunk of the specified logical volume. the bex_file_offset, bex_length, and bex_state fields are valid for every extent the server returns. the bex_storage_offset field, though, means different things depending on bex_state, so here's the rundown (in increasing order):
PNFS_BLOCK_READ_WRITE_DATAmeansbex_storage_offsetis valid, and points to valid/initialized data that can be read and written. ✍️PNFS_BLOCK_READ_DATAmeansbex_storage_offsetis valid and points to valid/ initialized data that can only be read. Write operations are prohibited, and the client may need to request a read-write layout. 🚫✏️PNFS_BLOCK_INVALID_DATAmeansbex_storage_offsetis valid, but points to invalid un-initialized data. this data must not be physically read from the disk until it has been initialized. A read request for aPNFS_BLOCK_INVALID_DATAextent must fill the user buffer with zeros, unless the extent is covered by aPNFS_BLOCK_READ_DATAextent of a copy-on-write file system. Write requests must write whole server-sized blocks to the disk, and bytes not initialized by the user must be set to zero. any write to storage in aPNFS_BLOCK_INVALID_DATAextent changes the written portion of the extent toPNFS_BLOCK_READ_WRITE_DATA, and the pNFS client is responsible for reporting that change via LAYOUTCOMMIT. 🧊➡️🔥PNFS_BLOCK_NONE_DATAmeansbex_storage_offsetis not valid, and this extent may not be used to satisfy write requests. Read requests may be satisfied by zero-filling, same as forPNFS_BLOCK_INVALID_DATA.PNFS_BLOCK_NONE_DATAextents may be returned by requests for readable extents, but they are never returned if the request was for a writable extent. 🕳️
an extent list holds all the relevant extents in increasing order of each extent's bex_file_offset. if there's a tie, it's broken by increasing order of the extent state (bex_state) 🧮
2.3.1. Layout Requests and Extent Lists 🦋
every layout request specifies at least three parameters: file offset, desired size, and minimum size 📋 if the status of a request says success, the extent list that comes back has to meet these criteria:
- A request for a readable (but not writable) layout returns only
PNFS_BLOCK_READ_DATAorPNFS_BLOCK_NONE_DATAextents (but notPNFS_BLOCK_INVALID_DATAorPNFS_BLOCK_READ_WRITE_DATAextents). - A request for a writable layout returns
PNFS_BLOCK_READ_WRITE_DATAorPNFS_BLOCK_INVALID_DATAextents (but notPNFS_BLOCK_NONE_DATAextents). It may also returnPNFS_BLOCK_READ_DATAextents only when the offset ranges in those extents are also covered byPNFS_BLOCK_INVALID_DATAextents, so writes are allowed. 🔐 - The first extent in the list MUST contain the requested starting offset. 🎯
- The total size of extents within the requested range MUST cover at least the minimum size. There is one exception: the total size MAY be smaller if only readable extents were requested and EOF is encountered. 🏁
- Extents in the extent list MUST be logically contiguous for a read-only layout. For a read-write layout, the set of writable extents (i.e., excluding
PNFS_BLOCK_READ_DATAextents) MUST be logically contiguous. EveryPNFS_BLOCK_READ_DATAextent in a read-write layout MUST be covered by one or morePNFS_BLOCK_INVALID_DATAextents. This overlap ofPNFS_BLOCK_READ_DATAandPNFS_BLOCK_INVALID_DATAextents is the only permitted extent overlap. 🧩 - Extents MUST be ordered in the list by starting offset, with
PNFS_BLOCK_READ_DATAextents precedingPNFS_BLOCK_INVALID_DATAextents in the case of equalbex_file_offsets. 📚
if the minimum requested size, loga_minlength, is zero, that's the server's cue that the client wants any layout at offset loga_offset or less that the metadata server has "readily available". "Readily" is subjective and depends on the layout type and the pNFS server implementation 🤷♀️ for block layout servers, "readily available" SHOULD be interpreted so that readable layouts are always available, even if some extents are in the PNFS_BLOCK_NONE_DATA state. when processing requests for writable layouts, a layout is readily available if extents can be returned in the PNFS_BLOCK_READ_WRITE_DATA state 💖
2.3.2. Layout Commits 🌸
the layout commit structure, same deal, copied exactly 🫶
/// /* block layout specific type for lou_body */
/// struct pnfs_block_layoutupdate4 {
/// pnfs_block_extent4 blu_commit_list<>;
/// /* list of extents which
/// * now contain valid data.
/// */
/// };
///the pnfs_block_layoutupdate4 structure is what the client uses as the block-protocol specific argument in a LAYOUTCOMMIT operation ✨ its blu_commit_list field is an extent list covering regions of the file layout that used to be in the PNFS_BLOCK_INVALID_DATA state, but that the client has since written, so they should now count as PNFS_BLOCK_READ_WRITE_DATA. the bex_state field of each extent in the blu_commit_list MUST be set to PNFS_BLOCK_READ_WRITE_DATA. the extents in the commit list MUST be disjoint and MUST be sorted by bex_file_offset. the bex_storage_offset field is unused 🙅♀️
heads up, implementors: a server may be unable to commit regions at a granularity smaller than a file-system block (typically 4 KB or 8 KB) 🚨 as noted above, the block size the server uses is available as an NFSv4 attribute, and any extents included in the blu_commit_list MUST be aligned to this granularity and have a size that is a multiple of this granularity. if the client believes its actions have moved the end-of-file into the middle of a block being committed, the client MUST write zeroes from the end-of-file to the end of that block before committing the block. if it doesn't, junk (un-initialized data) may show up in that area if the file is later extended by moving the end-of-file 😱
2.3.3. Layout Returns 🎀
okay so the LAYOUTRETURN operation doesn't need any block layout specific data at all 💅 when it's a LAYOUTRETURN4_FILE_return type, the layoutreturn_file4 data structure tells the server which region of the file layout the client no longer needs 🦋 The opaque lrf_body field of the layoutreturn_file4 data structure MUST have length zero. A LAYOUTRETURN is basically the client explicitly giving resources back, usually so the server doesn't have to send a CB_LAYOUTRECALL later 🙈 The client can return disjoint regions of the file by stacking multiple LAYOUTRETURN operations inside a single COMPOUND operation ✨
the block/volume layout supports unilateral layout revocation, and this is where it gets intense 😭 when the server revokes a layout on its own, usually because the client's lease time expired, a delegation got recalled, or the client took forever to return a layout, it really matters for correctness that any in-flight I/Os the client sent before the revoke get rejected at the storage 🔥 for the block/volume protocol, that's possible by fencing a client with an expired layout timer away from the physical storage. BUT the granularity of this can only be at the host/logical-unit level 💀 so if one of a client's layouts gets unilaterally revoked, it effectively makes all of that client's layouts useless for files sitting on the storage units that make up the logical volume. This may even make the client's layouts for files in other file systems useless too 😩
2.3.4. Client Copy-on-Write Processing 🌸
copy-on-write is the mechanism used to support file and/or file system snapshots 📸 when you write to unaligned regions, or to regions smaller than a file system block, the writer has to copy the parts of the original file data to a new location on disk. This can be done on the client OR the server, and the paragraphs below spill how a pNFS block layout client handles a file that needs copy-on-write semantics 🧚
the way the PNFS_BLOCK_READ_WRITE_DATA and PNFS_BLOCK_READ_DATA extent types are distinguished, plus the allowed overlap of PNFS_BLOCK_READ_DATA extents with PNFS_BLOCK_INVALID_DATA extents, is what lets pNFS clients do copy-on-write processing themselves 🙌 in classic NFS, the server would do this. But since pNFS lets clients do direct block access, it's useful for clients to participate in copy-on-write operations. All block/volume pNFS clients MUST support this copy-on-write processing 👑
when a client wants to write data covered by a PNFS_BLOCK_READ_DATA extent, it MUST have requested a writable layout from the server. That layout will contain PNFS_BLOCK_INVALID_DATA extents covering all the data ranges of that layout's PNFS_BLOCK_READ_DATA extents 🫶 more precisely, for any bex_file_offset range covered by one or more PNFS_BLOCK_READ_DATA extents in a writable layout, the server MUST include one or more PNFS_BLOCK_INVALID_DATA extents in the layout that cover that same bex_file_offset range. When writing to that area, the client MUST effectively copy the data from the PNFS_BLOCK_READ_DATA extent for any partial blocks of that bex_file_offset and range, merge in the changes being written, and write the result to the PNFS_BLOCK_INVALID_DATA extent for the blocks of that bex_file_offset and range. So if entire blocks are getting overwritten, the matching PNFS_BLOCK_READ_DATA blocks don't need to be fetched, but any partial-block writes must be merged with data fetched via PNFS_BLOCK_READ_DATA extents before the result gets stored via PNFS_BLOCK_INVALID_DATA extents 🍓 for this discussion, "entire blocks" and "partial blocks" refer to the server's file-system block size.
storing data in a PNFS_BLOCK_INVALID_DATA extent converts the written portion into a PNFS_BLOCK_READ_WRITE_DATA extent. All subsequent reads MUST be performed from this extent, and the corresponding portion of the PNFS_BLOCK_READ_DATA extent MUST NOT be used after storing data in a PNFS_BLOCK_INVALID_DATA extent 🔥 if a client only writes part of an extent, the extent may be split at block aligned boundaries ✂️
when a client wants to write to a PNFS_BLOCK_INVALID_DATA extent that isn't covered by a PNFS_BLOCK_READ_DATA extent, it MUST treat that write exactly like a write to a file that isn't involved with copy-on-write semantics 🧁 so the data must be written in at least block-sized increments, aligned to multiples of block-sized offsets, and unwritten portions of blocks must be zero filled.
in the LAYOUTCOMMIT operation that normally sends updated layout information back to the server, for writable data, some PNFS_BLOCK_INVALID_DATA extents may get committed as PNFS_BLOCK_READ_WRITE_DATA extents. That signals the storage at the matching bex_storage_offset values has been stored into and is now valid data to be read 💖 PNFS_BLOCK_READ_DATA extents are NOT committed to the server, though. For extents the client gets via LAYOUTGET as PNFS_BLOCK_INVALID_DATA and returns via LAYOUTCOMMIT as PNFS_BLOCK_READ_WRITE_DATA, the server understands that the PNFS_BLOCK_READ_DATA mapping for that extent is no longer valid or necessary for that file ✨
⚠️ the NOT above is just me being dramatic. The original says "are not committed to the server", so it's not a requirement keyword and it stays lowercase in the spec itself.
2.3.5. Extents are Permissions 🦋
layout extents returned to pNFS clients grant permission to read or write 💅 PNFS_BLOCK_READ_DATA and PNFS_BLOCK_NONE_DATA are read-only (PNFS_BLOCK_NONE_DATA reads as zeroes), while PNFS_BLOCK_READ_WRITE_DATA and PNFS_BLOCK_INVALID_DATA are read/write (PNFS_BLOCK_INVALID_DATA reads as zeros, and any write converts it to PNFS_BLOCK_READ_WRITE_DATA). This is the ONLY way a client can get permission to do direct I/O to storage devices, so a pNFS client MUST NOT perform direct I/O operations that aren't permitted by an extent it holds 🙅♀️ when clients follow this rule, the pNFS server stays in control of potentially conflicting storage device operations, so it can figure out what conflicts and how to avoid them by granting and recalling extents to and from clients 🧠
block/volume class storage devices aren't required to perform read and write operations atomically 😬 overlapping concurrent reads and writes to the same data can make the read return a mix of before-write and after-write data. Overlapping writes can be even worse, since the result could be a mixture of data from both writes, and data corruption can happen if the underlying storage is striped and the operations finish in different orders on different stripes 💀 when multiple clients want to access the same data, a pNFS server can avoid these conflicts by using a concurrency control policy of single writer XOR multiple readers. This policy MUST be implemented when storage devices don't provide atomicity for concurrent read/write and write/write operations to the same data 🫶
if a client makes a layout request that conflicts with an existing layout delegation, the request gets rejected with the error NFS4ERR_LAYOUTTRYLATER. The client is then expected to retry the request after a short interval. During that interval, the server SHOULD recall the conflicting portion of the layout delegation from the client that currently holds it. This reject-and-retry approach does NOT prevent client starvation when there's contention for the layout of a particular file, so a pNFS server SHOULD implement a mechanism to prevent starvation 🥺 one possibility is that the server keeps a queue of rejected layout requests. Each new layout request can be checked to see if it conflicts with a previously rejected request, and if so, the newer request can be rejected. Once the original requesting client retries its request, its entry in the rejected request queue can be cleared, or the entry can be removed when it reaches a certain age ⏳
NFSv4 supports mandatory locks and share reservations. These are mechanisms clients can use to restrict which I/O operations are permissible to other clients 🔒 since all I/O operations ultimately arrive at the NFSv4 server for processing, the server is in a position to enforce these restrictions. But with pNFS layouts, I/Os get issued from the clients holding the layouts directly to the storage devices hosting the data. Those devices have no knowledge of files, mandatory locks, or share reservations, and they aren't in a position to enforce any of that 🙈 for this reason, the NFSv4 server MUST NOT grant layouts that conflict with mandatory locks or share reservations. Further, if a conflicting mandatory lock request or a conflicting open request arrives at the server, the server MUST recall the part of the layout in conflict with the request before granting the request 👑
2.3.6. End-of-file Processing 🌈
the end-of-file location can change in two ways: implicitly, as the result of a WRITE or LAYOUTCOMMIT beyond the current end-of-file, or explicitly, as the result of a SETATTR request 📏 typically, when an NFSv4 client truncates a file via the SETATTR call, the server frees any disk blocks belonging to the file that sit beyond the new end-of-file byte, and it MUST write zeros to the portion of the new end-of-file block beyond the new end-of-file byte. These actions make any pNFS layouts that refer to the freed or written blocks semantically invalid 🚫 so the server MUST recall from clients the portions of any pNFS layouts that refer to blocks that will be freed or written by the server before it processes the truncate request. These recalls may take time to finish. As explained in [NFSv4.1], if the server can't respond to the client SETATTR request in a reasonable amount of time, it SHOULD reply to the client with the error NFS4ERR_DELAY 🕰️
blocks in the PNFS_BLOCK_INVALID_DATA state that lie beyond the new end-of-file block are a special case 🧐 the server has reserved these blocks for use by a pNFS client with a writable layout for the file, but the client hasn't committed them yet, so they aren't part of the file mapping on disk. The server MAY free these blocks while processing the SETATTR request. If it does, the server MUST recall any layouts from pNFS clients that refer to those blocks before processing the truncate. If the server does NOT free the PNFS_BLOCK_INVALID_DATA blocks while processing the SETATTR request, it need not recall layouts that refer only to the PNFS_BLOCK_INVALID_DATA blocks 🌸
when a file gets extended implicitly by a WRITE or LAYOUTCOMMIT beyond the current end-of-file, or extended explicitly by a SETATTR request, the server need not recall any portions of any pNFS layouts 🎉
2.3.7. Layout Hints
okay so the SETATTR operation has this layout hint attribute [NFSv4.1] 🎀 when the client sets a layout hint (data type layouthint4) with a layout type of LAYOUT4_BLOCK_VOLUME (that's the loh_type field), the loh_body field holds a value of data type pnfs_block_layouthint4. 🧚
/// /* block layout specific type for loh_body */
/// struct pnfs_block_layouthint4 {
/// uint64_t blh_maximum_io_time; /* maximum i/o time in seconds
/// */
/// };
///the block layout client uses this hint to tell the server the longest time an I/O could take to run on the client 💅 and it's a must, no skipping: clients using block layouts MUST set the layout hint attribute before they use LAYOUTGET operations 🫶
2.3.8. Client Fencing
the pNFS block protocol has to handle the messy moments 😭 like a system failure (usually a network connectivity issue) where the server has to unilaterally revoke extents from one client so it can hand them to a different client. the pNFS server implementation MUST make sure that once resources move to another client, the original owner doesn't use them anymore 🔥 and this has to hold up against any possible combination of partitions and delays between all the participants in the protocol (server, storage and client). there are two ways to guarantee that isolation, and we'll get into both below 👑
first up, the block client can be fenced away from the block storage with LUN masking or mapping, done at the storage systems or the storage area network, to disable the isolated client's access 🙈 this needs the server to have access to a management interface for the storage system, plus the authorization to do LUN masking and management operations. for example, the Storage Management Initiative Specification (SMI-S) [SMIS] gives a way to discover and mask LUNs, including a way to link clients with the World Wide Names or Initiator names that need masking 🌸
if the storage doesn't support LUN masking, the server has to lean on the clients to run a timed-lease I/O fencing mechanism 🦋 since clients can't tell whether the server is using LUN masking, the client MUST implement timed-lease fencing in all cases. ngl this part has two timers. the first one, "lease_time", is how long a lease lasts, as set by the server's lease_time attribute (see [NFSv4.1]). the second, "blh_maximum_io_time", is the longest a client I/O to the storage system can take to either complete or fail. this is often 30 seconds or 60 seconds, but it can be longer in some environments ⭐ and if the maximum client I/O time can't be bounded, the client MUST use a value of all 1s as the blh_maximum_io_time 💀
after a new client ID is set up, the client MUST use SETATTR with a layout hint of type LAYOUT4_BLOCK_VOLUME to tell the server its maximum I/O time, and it has to do this before it issues the first LAYOUTGET operation 🎀 the maximum I/O time hint is technically a per-file attribute, but it's really a per-client thing 💜 so the server MUST keep the last maximum I/O time hint sent separately for each client. every time the maximum I/O time changes, the server MUST apply it to all of the files that client has a layout for ✨ if the client doesn't specify this attribute on a file where a block layout gets requested, the server SHOULD use the most recent value that same client sent for any file. and if that client never sent a value for this attribute, the server SHOULD reject the layout request with the error NFS4ERR_LAYOUTUNAVAILABLE 🙃 the client SHOULD NOT send a SETATTR of the layout hint with every LAYOUTGET, bestie, that's just extra noise 🧚
a server that implements fencing via LUN masking SHOULD accept any maximum I/O time value from a client 🌈 a server that doesn't implement fencing may return the error NFS4ERR_INVAL to the SETATTR operation, and if it does that, it SHOULD return NFS4ERR_INVAL when a client sends an unbounded maximum I/O time (all 1s), or when the maximum I/O time is way bigger than what other block-layout clients using pNFS are sending 😭
when a client gets NFS4ERR_INVAL back from SETATTR for a layout hint, it MUST NOT use the LAYOUTGET operation 💅 and after the server answers SETATTR for the layout hint with NFS4ERR_INVAL, the server MUST return NFS4ERR_LAYOUTUNAVAILABLE to every later LAYOUTGET from that client. so basically, by returning either NFS4ERR_INVAL or NFS4_OK, the server decides whether a client with a large (or unbounded) maximum I/O time gets to use pNFS at all 🫶
now here's how the lease time and maximum I/O time values work together for the client and the server 🔥
when a client gets layout info from a LAYOUTGET operation, those layouts are valid for at most "lease_time" seconds from the moment the server granted them ⏳ a layout gets renewed by any successful SEQUENCE operation, or whenever a new stateid is created or updated (see the section "Lease Renewal" of [NFSv4.1]). if the layout lease doesn't get renewed before it expires, the client MUST stop using the layout once "lease_time" seconds have passed since it sent either the original LAYOUTGET command or the last operation that renewed the lease ☠️ in other words, the client can't issue any I/O to blocks covered by an expired layout 🙅♀️ and with big communication delays between client and server, the lease could even run out before the server's response shows up at the client. in that situation the client MUST NOT use the expired layouts, and it SHOULD go back to regular NFSv41 READ and WRITE operations 🦋 on top of that, the client has to be set up so that I/O operations finish within "blh_maximum_io_time", even when multipath drivers keep retrying I/Os over multiple paths 🍓
as spelled out in the "Dealing with Lease Expiration on the Client" section of [NFSv4.1], if any SEQUENCE operation succeeds but sr_status_flag has SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED, SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, or SEQ4_STATUS_ADMIN_STATE_REVOKED set, the client MUST immediately stop using all layouts and all device ID to device address mappings tied to that server 🚫
if there's no known two-way communication between client and server on the fore channel, the server needs to wait at least "lease_time" plus "blh_maximum_io_time" before it hands layouts from the original client to any other client ⏰ the server, like the client, has to play it safe and start the lease expiration timer from when it got the operation that last renewed the lease 🫶
2.4. Crash Recovery Issues
the big rule in crash recovery is that both the client and the server need to know when the other one has failed 💀 on top of that, a client has to see a consistent view of data across server restarts 🥺 all of these requirements, plus the full crash recovery discussion, live in the "Crash Recovery" section of the NFSv41 specification [NFSv4.1]. this document only adds extra crash recovery stuff that's specific to the block/volume layout 🧚
so, when the server crashes while the client holds a writable layout, the client has written data to blocks covered by that layout, and those blocks are still in the PNFS_BLOCK_INVALID_DATA state, the client has two ways to recover 🎀 if the client still has the data that was written to those blocks in its cache, it can just re-write the data via NFSv4 once the server is back online 💖 but if the data is no longer in the cache, the client MUST NOT try to source the data from the data servers. instead, it should try to commit those blocks to the server during the server's recovery grace period, by sending a LAYOUTCOMMIT with the "loca_reclaim" flag set to true 🫶 this whole process is described in detail in Section 18.42.4 of [NFSv4.1] 📖
2.5. Recalling Resources: CB_RECALL_ANY
the server might decide it can't hold all the state for layouts without running out of resources 😰 in that case, it's totally free to recall individual layouts using CB_LAYOUTRECALL to lighten the load, or it can ask the client to return any layout at all 🧹
the NFSv4.1 spec [NFSv4.1] defines the following types:
const RCA4_TYPE_MASK_BLK_LAYOUT = 4;
struct CB_RECALL_ANY4args {
uint32_t craa_objects_to_keep;
bitmap4 craa_type_mask;
};when the server sends a CB_RECALL_ANY request to a client with the RCA4_TYPE_MASK_BLK_LAYOUT bit set in craa_type_mask, the client should immediately respond with NFS4_OK, and then asynchronously return complete file layouts until the number of files with layouts cached on the client is less than craa_object_to_keep 🌸 it's giving tidy housekeeping 🧽
2.6. Transient and Permanent Errors
the server can answer LAYOUTGET with a bunch of different error statuses 🙈 some of these errors just mean "not right now" (transient), and others mean something more stubborn that probably won't get fixed soon (permanent) 🌧️
the transient errors, NFS4ERR_RECALLCONFLICT and NFS4ERR_TRYLATER, mean the server can't grant the layout to the client right away 🦋 with NFS4ERR_RECALLCONFLICT, it's because the server has just sent a CB_LAYOUTRECALL to the requesting client. with NFS4ERR_TRYLATER, the server can't grant the request, possibly because other clients have sharing conflicts with it 🙃 either way, a reasonable move for the client is to wait a few milliseconds and retry the request. the client SHOULD track how many retries it has done, and if it's not making forward progress, the client SHOULD send the READ or WRITE operation directly to the server 🫶
the error NFS4ERR_LAYOUTUNAVAILABLE can come back from the server if layouts aren't supported for the requested file or for the file system that contains it 💔 the server can also return this error if it's in the progress of migrating the file off secondary storage, or for any other reason that leaves it unable to supply the layout ⏳ when a client gets NFS4ERR_LAYOUTUNAVAILABLE, it SHOULD send future READ and WRITE requests directly to the server 🎀 a client isn't expected to cache the file's layoutunavailable state forever, particularly if the file gets closed, so eventually the client MAY reissue a LAYOUTGET operation ✨
3. Security Considerations 🔐
okay bestie, this section is the one where the SAN storage gets real 😭🫶. Usually, SAN disk arrays and SAN protocols have access control (like LUN mapping and/or masking) that works per individual host 🧚. That lets the server "fence" individual client machines away from certain physical disks, meaning it can stop a client from reading or writing to them 🚧. Finer-grained access control just isn't generally available, so certain security responsibilities get handed off to pNFS clients for block/volume layouts 💅. Block/volume storage systems mostly control access at the volume level, so pNFS clients have to be trusted to only do the accesses their layout extents allow (and not touch storage for files they don't hold a layout extent for) 🎀. In general, the server can't stop a client holding a layout for a file from reaching parts of the physical disk that the layout doesn't cover 🙈. The server also can't stop a client from touching blocks covered by a layout it already returned ⚠️. So this block-level protection has to be provided by the client software, ngl 💀.
Here's the alternate route: the storage devices can export virtualized block addresses that actually reflect which files the blocks belong to 🦋. Those virtual addresses get sent to pNFS clients through layouts, so the storage device can do its own access checks while still mapping them to physical block addresses ✨. In environments where client-side protection from access to storage outside the authorized layout extents isn't enough, pNFS block/volume storage layouts SHOULD NOT be used, unless the storage device can actually do those checks (through virtualized block addresses or some other means) 🚫💖. The environment where client-side protection can be enough looks like co-located clients, servers, and storage systems in a data center, with a physically isolated SAN run by a single system administrator or a small group of them 🏠🌸.
This also affects some NFSv4 stuff outside of pNFS 👀. Like, if a file is covered by a mandatory read-only lock, the server can make sure only readable layouts for that file get granted to pNFS clients 🔒. But it's up to each pNFS client to make sure a readable layout is only used to service read requests, and not to let writes hit the existing parts of the file 📝. Similarly, block/volume storage devices can't validate NFS Access Control Lists (ACLs) or file open modes, so the client has to enforce those policies before sending a READ or WRITE request to the storage device 🙅♀️. Block/volume storage systems usually can't enforce file-based security at all, so in environments where pNFS clients can't be trusted to enforce those policies, pNFS block/volume storage layouts SHOULD NOT be used 😤💅.
Access to block/volume storage sits at a lower layer of the I/O stack than NFSv4, so NFSv4 security isn't directly applicable to protocols that touch that storage directly 🧱. Depending on the protocol, some NFSv4 security mechanisms (like encryption or cryptographic integrity) may not be available, or they might come through different means 🔐. At one extreme, pNFS with block/volume storage can be used with storage access protocols (like parallel SCSI) that give you basically no security functionality 💀. At the other extreme, pNFS can be used with storage protocols like iSCSI that can provide serious security functionality 💪. Whoever administers and deploys pNFS with a block/volume storage access protocol is responsible for making sure that protocol gets appropriate protection (physical security is a common way to do this for protocols that aren't based on IP) 🛡️. If the security requirements for the storage protocol can't be met, pNFS block/volume storage layouts SHOULD NOT be used 🙏.
When security is available for a storage protocol, it's usually at a different granularity and uses a different notion of identity than NFSv4 🪪. (NFSv4 controls user access to files, and iSCSI controls initiator access to volumes, so they're basically speaking different languages 🗣️.) The pNFS client is responsible for enforcing correct correspondences between these security layers 🫶. Same as the first paragraph of this section: if client-side protection from access to storage outside the layout isn't sufficient for the security requirements, pNFS block/volume storage layouts SHOULD NOT be used 🚫✨.
4. Conclusions 🎀
This document specifies the block/volume layout type for pNFS and its associated functionality. That's the whole wrap-up, and it's giving main character energy 👑.
5. IANA Considerations 📋
There are no IANA considerations in this document 🙌. All pNFS IANA Considerations are covered in [NFSv4.1] 💜, so go check there if you're looking for registry stuff.
6. Acknowledgments 🫶
This document draws heavily on the authors' familiarity with the mapping functionality and protocol in EMC's Multi-Path File System (MPFS) (previously named HighRoad) system [MPFS] 🌈. The protocol MPFS uses is called FMP (File Mapping Protocol). It's an add-on protocol that runs alongside file system protocols like NFSv3 to provide pNFS-like functionality for block/volume storage 🧚. While it drew on FMP, the data structures and functional considerations in this document differ in significant ways, based on lessons learned and the chance to use NFSv4 features like COMPOUND operations 🔥. The design that lets pNFS clients take part in copy-on-write is based on text and ideas contributed by Craig Everhart 💖.
Andy Adamson, Ben Campbell, Richard Chandler, Benny Halevy, Fredric Isaman, and Mario Wurzl all helped review versions of this specification 🥰. Iconic team behavior, honestly 💅.
7. References 📚
7.1. Normative References
[LEGAL] IETF Trust, "Legal Provisions Relating to IETF Documents",
http://trustee.ietf.org/docs/IETF-Trust-License-Policy.pdf,
November 2008.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[NFSv4.1] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed.,
"Network File System (NFS) Version 4 Minor Version 1
Protocol", RFC 5661, January 2010.
[XDR] Eisler, M., Ed., "XDR: External Data Representation
Standard", STD 67, RFC 4506, May 2006.7.2. Informative References
[MPFS] EMC Corporation, "EMC Celerra Multi-Path File System
(MPFS)", EMC Data Sheet,
http://www.emc.com/collateral/software/data-sheet/
h2006-celerra-mpfs-mpfsi.pdf.
[SMIS] SNIA, "Storage Management Initiative Specification (SMI-S)
v1.4", http://www.snia.org/tech_activities/standards/
curr_standards/smi/SMI-S_Technical_Position_v1.4.0r4.zip.Authors' Addresses 🦋
David L. Black
EMC Corporation
176 South Street
Hopkinton, MA 01748
Phone: +1 (508) 293-7953
EMail: black_david@emc.com
Stephen Fridella
Nasuni Inc
313 Speen St
Natick MA 01760
EMail: stevef@nasuni.com
Jason Glasgow
Google
5 Cambridge Center
Cambridge, MA 02142
Phone: +1 (617) 575 1599
EMail: jglasgow@aya.yale.edu📦 where this page comes from
- ietf: rfc5663 2010-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms