RFC 5664

Object-Based Parallel NFS (pNFS) Operations

📅 published January 2010, from ietf

RFC 5664: Object-Based Parallel NFS (pNFS) Operations 💖

  • RFC number: 5664
  • Category: Standards Track, which means it's an Internet Standards Track document 🎀
  • Date: January 2010
  • Authors: B. Halevy, B. Welch, and J. Zelenka, all from Panasas 🧚
  • ISSN: 2070-1721
  • Obsoletes and updates: the header doesn't list any, so this RFC doesn't obsolete or update anything else 🫶
  • Published by: the Internet Engineering Task Force (IETF) 🌈
part 1 of 7, more coming bestie, don't go anywhere 💅

Abstract ✨

okay so pNFS (Parallel NFS) is an extension of Network File System version 4 (NFSv4) that lets clients talk straight to the storage where the file data actually lives, instead of going through the NFSv4 server for every byte 😭 bypassing the server can seriously boost performance and parallelism, but clients then need extra functionality to access the data, and some of that depends on the class of storage, a.k.a. the Layout Type 🔥 the main pNFS operations and data types in NFSv4 Minor version 1 are layout-type-independent, and layout-type-specific info gets carried in opaque data structures whose internal structure is defined by each particular layout type spec. this document defines the NFSv4.1 Object-Based pNFS Layout Type as a companion to the main NFSv4 Minor version 1 spec 💜

Status of This Memo 🌸

this is an Internet Standards Track document 👑

it's a product of the IETF, which means it represents the consensus of the IETF community. it got public review and was approved for publication by the Internet Engineering Steering Group (IESG). more info on Internet Standards is in Section 2 of RFC 5741 🦋

for the current status, any errata, and how to send feedback, check http://www.rfc-editor.org/info/rfc5664 🍓

Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. 💅

this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. please actually read those, they spell out your rights and restrictions for this doc 🙈 also, Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, per the Simplified BSD License ⭐

Table of Contents 🧚

   1. Introduction ....................................................3
      1.1. Requirements Language ......................................4
   2. XDR Description of the Objects-Based Layout Protocol ............4
      2.1. Code Components Licensing Notice ...........................4
   3. Basic Data Type Definitions .....................................6
      3.1. pnfs_osd_objid4 ............................................6
      3.2. pnfs_osd_version4 ..........................................6
      3.3. pnfs_osd_object_cred4 ......................................7
      3.4. pnfs_osd_raid_algorithm4 ...................................8
   4. Object Storage Device Addressing and Discovery ..................8
      4.1. pnfs_osd_targetid_type4 ...................................10
      4.2. pnfs_osd_deviceaddr4 ......................................10
           4.2.1. SCSI Target Identifier .............................11
           4.2.2. Device Network Address .............................11
   5. Object-Based Layout ............................................12
      5.1. pnfs_osd_data_map4 ........................................13
      5.2. pnfs_osd_layout4 ..........................................14
      5.3. Data Mapping Schemes ......................................14
           5.3.1. Simple Striping ....................................15
           5.3.2. Nested Striping ....................................16
           5.3.3. Mirroring ..........................................17
      5.4. RAID Algorithms ...........................................18
           5.4.1. PNFS_OSD_RAID_0 ....................................18
           5.4.2. PNFS_OSD_RAID_4 ....................................18
           5.4.3. PNFS_OSD_RAID_5 ....................................18
           5.4.4. PNFS_OSD_RAID_PQ ...................................19
           5.4.5. RAID Usage and Implementation Notes ................19
   6. Object-Based Layout Update .....................................20
      6.1. pnfs_osd_deltaspaceused4 ..................................20
      6.2. pnfs_osd_layoutupdate4 ....................................21
   7. Recovering from Client I/O Errors ..............................21
   8. Object-Based Layout Return .....................................22
      8.1. pnfs_osd_errno4 ...........................................23
      8.2. pnfs_osd_ioerr4 ...........................................24
      8.3. pnfs_osd_layoutreturn4 ....................................24
   9. Object-Based Creation Layout Hint ..............................25
      9.1. pnfs_osd_layouthint4 ......................................25
   10. Layout Segments ...............................................26
      10.1. CB_LAYOUTRECALL and LAYOUTRETURN .........................27
      10.2. LAYOUTCOMMIT .............................................27
   11. Recalling Layouts .............................................27
      11.1. CB_RECALL_ANY ............................................28
   12. Client Fencing ................................................29
   13. Security Considerations .......................................29
      13.1. OSD Security Data Types ..................................30
      13.2. The OSD Security Protocol ................................30
      13.3. Protocol Privacy Requirements ............................32
      13.4. Revoking Capabilities ....................................32
   14. IANA Considerations ...........................................33
   15. References ....................................................33
      15.1. Normative References .....................................33
      15.2. Informative References ...................................34
   Appendix A.  Acknowledgments ......................................35

1. Introduction ✨

in pNFS, the file server hands back typed layout structures that describe where the file data is located 🦋 different storage systems and different ways of arranging data on storage devices get different layouts. this document covers the layouts used with object-based storage devices (OSDs), which are accessed according to the OSD storage protocol standard (ANSI INCITS 400-2004 [1]) 🍓

an "object" is a container for data and attributes, and files are stored in one or more objects 💜 the OSD protocol has several operations on objects, including READ, WRITE, FLUSH, GET ATTRIBUTES, SET ATTRIBUTES, CREATE, and DELETE 😭 but with the object-based layout, the client only uses READ, WRITE, GET ATTRIBUTES, and FLUSH. the other commands are only used by the pNFS server 🙈

an object-based layout for pNFS includes object identifiers, capabilities that let clients READ or WRITE those objects, and various parameters that control how file data gets striped across the component objects 🌈 the OSD protocol has a capability-based security scheme that lets the pNFS server control which operations and which objects clients can use 🔥 that scheme is described in more detail in the "Security Considerations" section (Section 13) 🛡️

1.1. Requirements Language 🎀

the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [2] 💅

2. XDR Description of the Objects-Based Layout Protocol 🧚

this document contains the external data representation (XDR [3]) description of the NFSv4.1 objects layout protocol ✨ the XDR description is embedded in the document so it's easy for you to pull out into a ready-to-compile form. you can feed this document into the following shell script to get the machine-readable XDR description of the NFSv4.1 objects layout protocol:

   #!/bin/sh
   grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??'

so if you save the script above as a file called "extract.sh" and this document is in a file called "spec.txt", you run:

   sh extract.sh < spec.txt > pnfs_osd_prot.x

what the script does is strip leading white space from each line, plus a sentinel sequence of "///" 💖

the embedded XDR file header comes next. more XDR descriptions, marked with the sentinel sequence, are embedded throughout the document 🌸

heads up: the XDR code in this document depends on types from the NFSv4.1 nfs4_prot.x file ([4]). that includes the nfs types that end in a 4, like offset4 and length4, as well as more generic types like uint32_t and uint64_t 🫶

2.1. Code Components Licensing Notice 🦋

the XDR description (the lines starting with "///") and the scripts for extracting it are Code Components, as described in Section 4 of "Legal Provisions Relating to IETF Documents" [5]. they're licensed under the terms of Section 4 of "Legal Provisions Relating to IETF Documents" 🌈

   /// /*
   ///  * Copyright (c) 2010 IETF Trust and the persons identified
   ///  * as authors of the code.  All rights reserved.
   ///  *
   ///  * Redistribution and use in source and binary forms, with
   ///  * or without modification, are permitted provided that the
   ///  * following conditions are met:
   ///  *
   ///  * o Redistributions of source code must retain the above
   ///  *   copyright notice, this list of conditions and the
   ///  *   following disclaimer.
   ///  *
   ///  * o Redistributions in binary form must reproduce the above
   ///  *   copyright notice, this list of conditions and the
   ///  *   following disclaimer in the documentation and/or other
   ///  *   materials provided with the distribution.
   ///  *
   ///  * o Neither the name of Internet Society, IETF or IETF
   ///  *   Trust, nor the names of specific contributors, may be
   ///  *   used to endorse or promote products derived from this
   ///  *   software without specific prior written permission.
   ///  *
   ///  *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS
   ///  *   AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED
   ///  *   WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
   ///  *   IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
   ///  *   FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO
   ///  *   EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
   ///  *   LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
   ///  *   EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
   ///  *   NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
   ///  *   SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
   ///  *   INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
   ///  *   LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
   ///  *   OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
   ///  *   IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
   ///  *   ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   ///  *
   ///  * This code was derived from RFC 5664.
   ///  * Please reproduce this note if possible.
   ///  */
   ///
   /// /*
   ///  * pnfs_osd_prot.x
   ///  */
   ///
   /// %#include <nfs4_prot.x>
   ///
that's the end of part 1! 🎀 part 2 is next, and it's where the Basic Data Type Definitions start 💜

3. Basic Data Type Definitions ✨

okay so this section is where we set up the basic building blocks 🧱 the Object-Based Layout protocol uses, plus a few constants. let's go!! 🎀

3.1. pnfs_osd_objid4 🔎

okay so an object is identified by a number, kinda like an inode number 🦋 and the object storage model has a two-level scheme where objects inside an object storage device get grouped into partitions 🗂️

/// struct pnfs_osd_objid4 {
///     deviceid4       oid_device_id;
///     uint64_t        oid_partition_id;
///     uint64_t        oid_object_id;
/// };
///

the pnfs_osd_objid4 type is used to identify an object within a partition on a specified object storage device 💅 oid_device_id picks the object storage device out of the set of available storage devices. the device is identified with the deviceid4 type, which is an index into addressing information about that device returned by the GETDEVICELIST and GETDEVICEINFO operations. the deviceid4 data type is defined in NFSv4.1 [6] 📚 within an OSD, a partition is identified with a 64-bit number, oid_partition_id. within a partition, an object is identified with a 64-bit number, oid_object_id. creating and managing partitions is outside the scope of this document 🙅‍♀️ it's a facility the object-based storage file system provides, so that's their job, not ours 💖

3.2. pnfs_osd_version4 🌸

/// enum pnfs_osd_version4 {
///     PNFS_OSD_MISSING    = 0,
///     PNFS_OSD_VERSION_1  = 1,
///     PNFS_OSD_VERSION_2  = 2
/// };
///

okay so pnfs_osd_version4 tells you the OSD protocol version, or whether an object is missing (aka unavailable) 🫣 some of the object-based layout-supported RAID algorithms encode redundant information and can make up for missing components, but the data placement algorithm needs to know which parts are missing 🧩

at this point the OSD standard is at version 1.0, and we're anticipating a version 2.0 of the standard (SNIA T10/1729-D [14]) ✨ the second generation OSD protocol has extra proposed features for more robust error recovery, snapshots, and byte-range capabilities 🔥 so the OSD version gets called out explicitly in the information returned in the layout. (you could also figure it out by looking inside the capability type at the format field, which is the first byte. the format value is 0x1 for an OSD v1 capability. however, it seems most robust to call out the version explicitly. 🙈)

3.3. pnfs_osd_object_cred4 🔐

/// enum pnfs_osd_cap_key_sec4 {
///     PNFS_OSD_CAP_KEY_SEC_NONE = 0,
///     PNFS_OSD_CAP_KEY_SEC_SSV  = 1
/// };
///
/// struct pnfs_osd_object_cred4 {
///     pnfs_osd_objid4         oc_object_id;
///     pnfs_osd_version4       oc_osd_version;
///     pnfs_osd_cap_key_sec4   oc_cap_key_sec;
///     opaque                  oc_capability_key<>;
///     opaque                  oc_capability<>;
/// };
///

the pnfs_osd_object_cred4 structure is how each component making up the file gets identified 🧚 oc_object_id identifies the component object, oc_osd_version represents the OSD protocol version (or whether that component is unavailable), and oc_capability plus oc_capability_key, along with the oda_systemid from the pnfs_osd_deviceaddr4, give you the OSD security credentials needed to access that object 💜 oc_cap_key_sec shows the method used to secure the oc_capability_key (see Section 13.1 for more details) 👑

to meet the OSD security requirements, the capability key SHOULD be transferred securely so nobody can eavesdrop on it (see Section 13) 🕵️‍♀️ so a client SHOULD either issue the LAYOUTGET or GETDEVICEINFO operations via RPCSEC_GSS with the privacy service, or previously establish a secret state verifier (SSV) for the sessions via the NFSv4.1 SET_SSV operation. the pnfs_osd_cap_key_sec4 type is used to identify the method the server uses to secure the capability key 🎀

  • PNFS_OSD_CAP_KEY_SEC_NONE means the oc_capability_key is not encrypted 🙅‍♀️ in that case the client SHOULD issue the LAYOUTGET or GETDEVICEINFO operations with RPCSEC_GSS with the privacy service, or the NFSv4.1 transport should be secured by using methods that are external to NFSv4.1, like the use of IPsec [15] for transporting the NFSV4.1 protocol.
  • PNFS_OSD_CAP_KEY_SEC_SSV means the oc_capability_key contents are encrypted using the SSV GSS context and the capability key as inputs to the GSS_Wrap() function (see GSS-API [7]) with the conf_req_flag set to TRUE 🔒 the client MUST use the secret SSV key as part of the client's GSS context to decrypt the capability key, using the value of the oc_capability_key field as the input_message to the GSS_unwrap() function. also, to stop the SSV key from getting eavesdropped, the client SHOULD issue SET_SSV via RPCSEC_GSS with the privacy service. 🫶

the method that actually gets picked depends on whether the client established an SSV key with the server, and whether it issued the operation with the RPCSEC_GSS privacy method 💅 naturally, if the client did not establish an SSV key via SET_SSV, the server MUST use the PNFS_OSD_CAP_KEY_SEC_NONE method. otherwise, if the operation was not issued with the RPCSEC_GSS privacy method, the server SHOULD secure the oc_capability_key with the PNFS_OSD_CAP_KEY_SEC_SSV method. the server MAY also use the PNFS_OSD_CAP_KEY_SEC_SSV method when the operation was issued with the RPCSEC_GSS privacy method 🌈

3.4. pnfs_osd_raid_algorithm4 🧱

/// enum pnfs_osd_raid_algorithm4 {
///     PNFS_OSD_RAID_0     = 1,
///     PNFS_OSD_RAID_4     = 2,
///     PNFS_OSD_RAID_5     = 3,
///     PNFS_OSD_RAID_PQ    = 4     /* Reed-Solomon P+Q */
/// };
///

pnfs_osd_raid_algorithm4 represents the data redundancy algorithm used to protect the file's contents 🛡️ see Section 5.4 for more details ✨

4. Object Storage Device Addressing and Discovery 🗺️

okay so data operations to an OSD need the client to know the "address" of each OSD's root object 📍 the root object is synonymous with the Small Computer System Interface (SCSI) logical unit. the client tells its SCSI protocol stack which logical units to use with a representation that's local to the client. since those representations are local, GETDEVICEINFO has to return information the client can use to pick the correct local representation 🎯

in the block world, a set offset (logical block number or track/sector) holds a disk label, and that label identifies the disk uniquely 💿 an OSD is different because it has a standard set of attributes on its root object 🧠 for device identification, the OSD System ID (root information attribute number 3) and the OSD Name (root information attribute number 9) get used as the label. they show up in the pnfs_osd_deviceaddr4 type below under the oda_systemid and oda_osdname fields 🏷️

sometimes SCSI target discovery has to be driven by info in the GETDEVICEINFO response 🔍 one example is Internet SCSI (iSCSI) targets that the client doesn't know about until a layout has been requested. the info given as the oda_targetid, oda_targetaddr, and oda_lun fields in the pnfs_osd_deviceaddr4 type described below (see Section 4.2) lets the client probe a specific device given its network address and, optionally, its iSCSI Name (see iSCSI [8]). or, when the device network address is omitted, it lets the client discover the object storage device using the provided device name or SCSI Device Identifier (see SPC-3 [9.) 🕵️‍♀️✨

the oda_systemid is implicitly used by the client: the client signs each request with the request integrity check value, using the object credential signing key. this protects the client from accidentally accessing a device if the device address mapping was changed (or revoked) 🛡️ the server computes the capability key using its own view of the systemid tied to the respective deviceid in the credential. if the client's view of the deviceid mapping is stale, the client will use the wrong systemid (which must be system-wide unique), and the I/O request to the OSD will fail the integrity check verification 😭

to recover from this, the client should report the error and return the layout using LAYOUTRETURN, and invalidate all the device address mappings associated with this layout 🔄 then, if it wants, the client can ask for a new layout using LAYOUTGET and resolve the referenced deviceids using GETDEVICEINFO or GETDEVICELIST 🎀

the server MUST provide the oda_systemid and SHOULD also provide the oda_osdname. when the OSD name is present, the client SHOULD get the root information attributes whenever it sets up communication with the OSD, and verify that the OSD name it got from the OSD matches the one sent by the metadata server 🫡 to do that, the client uses the root_obj_cred credentials 💜

4.1. pnfs_osd_targetid_type4 🎯

the following enum specifies the way a SCSI target can be specified 💅 the target can be given as a SCSI Name, or as a SCSI Device Identifier.

/// enum pnfs_osd_targetid_type4 {
///     OBJ_TARGET_ANON             = 1,
///     OBJ_TARGET_SCSI_NAME        = 2,
///     OBJ_TARGET_SCSI_DEVICE_ID   = 3
/// };
///

4.2. pnfs_osd_deviceaddr4 🦋

here's the spec for an object device address 🧚

/// union pnfs_osd_targetid4 switch (pnfs_osd_targetid_type4 oti_type) {
///     case OBJ_TARGET_SCSI_NAME:
///         string              oti_scsi_name<>;
///
///     case OBJ_TARGET_SCSI_DEVICE_ID:
///         opaque              oti_scsi_device_id<>;
///
///     default:
///         void;
/// };
///
/// union pnfs_osd_targetaddr4 switch (bool ota_available) {
///     case TRUE:
///         netaddr4            ota_netaddr;
///     case FALSE:
///         void;
/// };
///
/// struct pnfs_osd_deviceaddr4 {
///     pnfs_osd_targetid4      oda_targetid;
///     pnfs_osd_targetaddr4    oda_targetaddr;
///     opaque                  oda_lun[8];
///     opaque                  oda_systemid<>;
///     pnfs_osd_object_cred4   oda_root_obj_cred;
///     opaque                  oda_osdname<>;
/// };
///

4.2.1. SCSI Target Identifier ✨

okay so when oda_targetid is set to OBJ_TARGET_SCSI_NAME, the oti_scsi_name string MUST be formatted as an "iSCSI Name" like iSCSI [8] and [10] describe 🎀. bestie, the actual string format isn't this document's job 🙈. parsing is based on the prefix, so you look for "iqn.", "eui.", or "naa." 💅. and more formats MAY be specified later, following the iSCSI Names properties 🦋

the iSCSI Name lets you name the target device with an IQN (iSCSI Qualified Name) or an EUI (Extended Unique Identifier) [11] string 💖. those are usually for iSCSI or Secure Routing Protocol (SRP) [16] devices 🫶. the NAA (Network Address Authority) string format (see [10]) gives you globally unique IDs from Fibre Channel Framing and Signaling (FC-FS) [17] 🔥. those are usually for Fibre Channel or SAS [18] (Serial Attached SCSI) devices. the main-character case is devices that are dual-attached over Fibre Channel or SAS and over iSCSI 👑✨

when oda_targetid is OBJ_TARGET_SCSI_DEVICE_ID, the oti_scsi_device_id opaque field MUST be a SCSI Device Identifier from SPC-3 [9] VPD Page 83h (Section 7.6.3. "Device Identification VPD Page") 🧚. if that Device Identifier matches the OSD System ID from oda_systemid, the server SHOULD send a zero-length oti_scsi_device_id opaque value 🌸. same as with oti_scsi_name, the contents of that opaque value are outside this document's scope, and more formats MAY be specified later according to SPC-3 🌈

OBJ_TARGET_ANON is a pnfs_osd_targetid_type4 that MAY be used when you don't want to identify a target at all 🙈. then the only things used to find the device are the OSD System ID and, optionally, the network address you were given 💜

4.2.2. Device Network Address 🦋

the optional oda_targetaddr field MAY be given by the server as a hint to speed up device discovery, over the iSCSI transport protocol for example 💖. the address uses the netaddr4 type, which is a TCP/IP endpoint (as specified in NFSv4.1 [6]) 🫶. if it's there, the client SHOULD use it to probe for the SCSI device at that address 👑. the client MAY still use other discovery stuff like Internet Storage Name Service (iSNS) [12] to find the device by oda_targetid 🔥. and honestly, that external name service SHOULD be used when devices might be connected through multiple connections and/or multiple storage fabrics (like Fibre Channel and iSCSI) 😭✨

the oda_lun field is the OSD 64-bit Logical Unit Number, formatted per SAM-3 [13] 🧚. the client uses that LUN to talk to a specific OSD Logical Unit 💅. how it gets used is spelled out in detail by the SCSI transport protocol, like iSCSI [8] 🌈

5. Object-Based Layout 🎀

okay the layout4 type comes from NFSv4.1 [6] and it looks like this 🦋

enum layouttype4 {
    LAYOUT4_NFSV4_1_FILES   = 1,
    LAYOUT4_OSD2_OBJECTS    = 2,
    LAYOUT4_BLOCK_VOLUME    = 3
};

struct layout_content4 {
    layouttype4             loc_type;
    opaque                  loc_body<>;
};

struct layout4 {
    offset4                 lo_offset;
    length4                 lo_length;
    layoutiomode4           lo_iomode;
    layout_content4         lo_content;
};

this doc defines the structure that goes with the layouttype4 value LAYOUT4_OSD2_OBJECTS 💖. NFSv4.1 [6] says loc_body is just an XDR type "opaque" 🫶. generic pNFS client layers don't interpret that opaque layout at all, but the object storage layout driver absolutely has to 🔥. this section explains what's inside that opaque value, which is pnfs_osd_layout4 👑✨

5.1. pnfs_osd_data_map4 💅

/// struct pnfs_osd_data_map4 {
///     uint32_t                    odm_num_comps;
///     length4                     odm_stripe_unit;
///     uint32_t                    odm_group_width;
///     uint32_t                    odm_group_depth;
///     uint32_t                    odm_mirror_cnt;
///     pnfs_osd_raid_algorithm4    odm_raid_algorithm;
/// };
///

pnfs_osd_data_map4 is the struct that sets up the algorithm mapping a file's contents across the component objects 🧚. it's not just one simple striping scheme where losing one component means losing data 😭. the map parameters can do mirroring and fancier schemes that protect against losing a component object 🌸

odm_num_comps is how many component objects the file is striped over 🌈. the server MAY grow the file by adding more components to the stripe while clients still hold valid layouts, until the file hits its final stripe width 💖. in that case, the file length MUST be limited to the number of bytes in one full stripe 🫶

"odm_stripe_unit" is how many bytes go on one component before moving to the next component in the list 👑. a full stripe is odm_stripe_unit times the number of components 🦋. in some RAID setups a stripe also has redundant info (parity) so the system can recover when a component object gets lost or damaged 🔥

"odm_group_width" and "odm_group_depth" let you do a nested striping pattern (see Section 5.3.2 for details) 💅. if there's no nesting, odm_group_width and odm_group_depth MUST be zero 🙈. the size of the components array MUST be a multiple of odm_group_width ✨

"odm_mirror_cnt" is used to replicate a file by replicating its component objects 🧚. if there's no mirroring, odm_mirror_cnt MUST be 0 🌈. if odm_mirror_cnt is greater than zero, then the size of the component array MUST be a multiple of (odm_mirror_cnt+1) 💜

see Section 5.3 for more details 🎀

5.2. pnfs_osd_layout4 🦋

/// struct pnfs_osd_layout4 {
///     pnfs_osd_data_map4      olo_map;
///     uint32_t                olo_comps_index;
///     pnfs_osd_object_cred4   olo_components<>;
/// };
///

pnfs_osd_layout4 describes a layout over a set of component objects 💖. "olo_components" is an array of object identifiers plus the security credentials that grant access to each object 🫶. the way the data is organized is set by pnfs_osd_data_map4, which says how the file's data gets mapped onto the component objects (aka the striping pattern) 👑. the placement algorithm assumes every component object shows up exactly once in the components array 🔥. so component objects MUST appear only once in the olo_components array 😭. the components array can cover all the objects that make up the file, in which case "olo_comps_index" is zero and the number of entries in olo_components equals olo_map.odm_num_comps ✨

the server MAY return fewer components than odm_num_comps, as long as the ones it returns are enough to reach any byte in the layout's data range (like a sub-stripe of "odm_group_width" components) 🧚. in that case, olo_comps_index tells you where the returned components array sits inside the full array of components for the file 🌸

heads up, the layout depends on the file size, which the client learns from the generic return parameters of LAYOUTGET by sending GETATTR commands to the metadata server 🌈. the client uses the file size to decide whether to fill holes with zeros or return a short read 💅. and striping patterns can make some component objects shorter than others, because a hole happens to line up with the last part of that component object 🙈

5.3. Data Mapping Schemes 💜

this section goes into the different data mapping schemes in detail 🎀. the object layout always uses a "dense" layout like NFSv4.1 [6] describes 🦋. so the second stripe unit of the file starts at offset 0 of the second component, not at offset stripe_unit bytes 💖. once a full stripe has been written, the next stripe unit gets appended to the first component object in the list, with no holes in the component objects 🫶

5.3.1. Simple Striping ✨

this is how a logical offset in a file (L) maps to a component object C and an object-specific offset O 🧚

L = logical offset into the file
W = total number of components
S = W * stripe_unit
N = L / S
C = (L-(N*S)) / stripe_unit
O = (N*stripe_unit)+(L%stripe_unit)

S is the number of bytes in a full stripe, and N is the stripe number 🔥. C is an index into the array of components, so it picks out one particular object storage device 👑. N and C both count from zero 🌈. O is the offset within the object that matches the file offset 💅. heads up, this computation can't handle the same object showing up more than once in olo_components 🙈

here's an example: an object striped over four devices, <D0 D1 D2 D3> 💖. the stripe_unit is 4096 bytes, so the stripe width S is 4 * 4096 = 16384 🦋

Offset 0:
  N = 0 / 16384 = 0
  C = 0-0/4096 = 0 (D0)
  O = 0*4096 + (0%4096) = 0

Offset 4096:
  N = 4096 / 16384 = 0
  C = (4096-(0*16384)) / 4096 = 1 (D1)
  O = (0*4096)+(4096%4096) = 0

Offset 9000:
  N = 9000 / 16384 = 0
  C = (9000-(0*16384)) / 4096 = 2 (D2)
  O = (0*4096)+(9000%4096) = 808

Offset 132000:
  N = 132000 / 16384 = 8
  C = (132000-(8*16384)) / 4096 = 0 (D0)
  O = (8*4096) + (132000%4096) = 33696

and that's the math, bestie 🫶💜 the numbers all check out!

5.3.2. Nested Striping 🧩

okay so nested striping is giving two-level vibes 💅 The odm_group_width and odm_group_depth parameters let you set up a nested striping pattern 🦋 odm_group_width is the width of a data stripe, and odm_group_depth is how many stripes get written before you move on to the next group of component objects in the file's list 🌸 Below is the math that maps a file offset to a component object and an offset inside that object. It takes the logical offset L and works out the component index C and the offset relative O within that component object ✨

L = logical offset into the file
W = total number of components
S = stripe_unit * group_depth * W
T = stripe_unit * group_depth * group_width
U = stripe_unit * group_width
M = L / S
G = (L - (M * S)) / T
H = (L - (M * S)) % T
N = H / U
C = (H - (N * U)) / stripe_unit + G * group_width
O = L % stripe_unit + N * stripe_unit + M * group_depth * stripe_unit

In these equations, S is the number of bytes striped across all component objects before the pattern repeats 🔁 T is the number of bytes striped within one group of component objects before it moves on to the next group 💜 U is the number of bytes in a stripe within a group. M is the "major" stripe number (across all the components), and N is the "minor" stripe number (across the group) 👑 G counts the groups from the start of the major stripe, and H is the byte offset within the group 🫶

Let's do an example, because it's so much easier to get once you see numbers 🍓 Picture an object striped over 100 devices, with a group_width of 10, a group_depth of 50, and a stripe_unit of 1 MB. In this setup, 500 MB gets written to the first 10 components, and then 5000 MB gets written before the pattern wraps back around to the first component in the array 😭

Offset 0:
  W = 100
  S = 1 MB * 50 * 100 = 5000 MB
  T = 1 MB * 50 * 10 = 500 MB
  U = 1 MB * 10 = 10 MB
  M = 0 / 5000 MB = 0
  G = (0 - (0 * 5000 MB)) / 500 MB = 0
  H = (0 - (0 * 5000 MB)) % 500 MB = 0
  N = 0 / 10 MB = 0
  C = (0 - (0 * 10 MB)) / 1 MB + 0 * 10 = 0
  O = 0 % 1 MB + 0 * 1 MB + 0 * 50 * 1 MB = 0

Offset 27 MB:
  M = 27 MB / 5000 MB = 0
  G = (27 MB - (0 * 5000 MB)) / 500 MB = 0
  H = (27 MB - (0 * 5000 MB)) % 500 MB = 27 MB
  N = 27 MB / 10 MB = 2
  C = (27 MB - (2 * 10 MB)) / 1 MB + 0 * 10 = 7
  O = 27 MB % 1 MB + 2 * 1 MB + 0 * 50 * 1 MB = 2 MB

Offset 7232 MB:
  M = 7232 MB / 5000 MB = 1
  G = (7232 MB - (1 * 5000 MB)) / 500 MB = 4
  H = (7232 MB - (1 * 5000 MB)) % 500 MB = 232 MB
  N = 232 MB / 10 MB = 23
  C = (232 MB - (23 * 10 MB)) / 1 MB + 4 * 10 = 42
  O = 7232 MB % 1 MB + 23 * 1 MB + 1 * 50 * 1 MB = 73 MB

5.3.3. Mirroring 🪞

The odm_mirror_cnt is how you replicate a file, by replicating its component objects 👯‍♀️ If there's no mirroring, then odm_mirror_cnt MUST be 0 🙅‍♀️ If odm_mirror_cnt is greater than zero, then the size of the olo_components array MUST be a multiple of (odm_mirror_cnt+1). So for a classic mirror on two objects, odm_mirror_cnt is one ✨ Mirroring can be defined over any RAID algorithm and any striping pattern (simple or nested), so it's super flexible 🌈 If odm_group_width is also non-zero, then the size of the olo_components array MUST be a multiple of odm_group_width * (odm_mirror_cnt+1). The replicas sit next to each other in the olo_components array, and the value C from the equations above is not a direct index into that array 🙈 Instead, you use the following equations to find the replica component index RCi, where i goes from 0 to odm_mirror_cnt.

C = component index for striping or two-level striping
i ranges from 0 to odm_mirror_cnt, inclusive
RCi = C * (odm_mirror_cnt+1) + i

5.4. RAID Algorithms 🛡️

pnfs_osd_raid_algorithm4 determines the algorithm and the placement of redundant data 💖 This section defines the different redundancy algorithms 🎀 Quick note: in this document, the term "RAID" (Redundant Array of Independent Disks) is used to mean an array of component objects that store data for one file 🧚 The objects live on independent object-based storage devices, and file data is encoded and striped across the array using algorithms that were developed for block-based RAID systems 🌸

5.4.1. PNFS_OSD_RAID_0 🫧

PNFS_OSD_RAID_0 means there's no parity data, so every byte in the component objects is a data byte, located by the equations above for C and O 💅 If a component object is marked as PNFS_OSD_MISSING, the pNFS client MUST either return an I/O error when that component is read, or it can retry the READ against the pNFS server instead 😭

5.4.2. PNFS_OSD_RAID_4 🧱

PNFS_OSD_RAID_4 means the last component object (or the last one in each group, if odm_group_width is greater than zero) holds parity information, computed with an XOR operation over the rest of the stripe 🔥 If a component object is unavailable, the client can read the other stripe units in the damaged stripe and rebuild the missing stripe unit by XORing them together 🧩 Or the client can replay the READ against the pNFS server, which presumably does the reconstructed read on the client's behalf 🫶

When parity is present in the file, there's an extra step to map the file offset L to an offset that accounts for the embedded parity, called L' 🎀 You compute L' first, then plug L' into the equations above for C and O.

L = file offset, not accounting for parity
P = number of parity devices in each stripe
W = group_width, if not zero, else size of olo_components array
N = L / (W-P * stripe_unit)
L' = N * (W * stripe_unit) +
     (L % (W-P * stripe_unit))

5.4.3. PNFS_OSD_RAID_5 🌀

PNFS_OSD_RAID_5 means the position of the parity data gets rotated on each stripe, or on each group (if odm_group_width is greater than zero) 🦋 In the first stripe, the last component holds the parity. In the second stripe, the next-to-last component holds it, and so on 💜 Rotating everything like this means I/O gets evenly spread across the objects as the file is read sequentially, which is honestly iconic 👑 The rotated parity layout is shown below, with the numbers standing for the stripe units 👇

0 1 2 P
4 5 P 3
8 P 6 7
P 9 a b

To compute the component object C, you first compute the offset that accounts for parity, L', and use that to work out C. Then you rotate C to get C'. Last step: increase C' by one if the parity information comes at or before C' within that stripe ✨ The equations below show this by computing I, which is the index of the component holding parity for a given stripe 🧚

L = file offset, not accounting for parity
W = odm_group_width, if not zero, else size of olo_components array
N = L / (W-1 * stripe_unit)
(Compute L' as describe above)
(Compute C based on L' as described above)
C' = (C - (N%W)) % W
I = W - (N%W) - 1
if (C' <= I) {
  C'++
}

5.4.4. PNFS_OSD_RAID_PQ 💎

PNFS_OSD_RAID_PQ is a double-parity scheme that uses the Reed-Solomon P+Q encoding scheme [19] 🌟 In this layout, the last two component objects hold the P and Q data, respectively. P is parity computed with XOR, and Q is a more complex equation that isn't described here 🙈 You can use the equations above for embedded parity to map a file offset to the correct component object, by setting the number of parity components to 2 instead of 1 for RAID4 or RAID5 🔁 Clients may simply choose to read data through the metadata server if two components are missing or damaged 🫶

5.4.5. RAID Usage and Implementation Notes 📝

RAID layouts with redundant data in their stripes need extra serialization of updates to work correctly 🔒 Otherwise, if two clients write to the same logical range of an object at the same time, the result could include different data in the same ranges of mirrored tuples, or corrupt the parity information 😱 It's the metadata server's job to enforce serialization requirements like this one 👑 For example, the metadata server can do it by not granting overlapping write layouts within mirrored objects 💅

6. Object-Based Layout Update 📦

layoutupdate4 is used in the LAYOUTCOMMIT operation to convey updates to the layout, along with additional information, to the metadata server ✨ It is defined in the NFSv4.1 [6] as follows:

struct layoutupdate4 {
    layouttype4             lou_type;
    opaque                  lou_body<>;
};

The layoutupdate4 type is an opaque value at the generic pNFS client level 🫧 If the lou_type layout type is LAYOUT4_OSD2_OBJECTS, then the lou_body opaque value is defined by the pnfs_osd_layoutupdate4 type 🎀

Object-Based pNFS clients are not allowed to modify the layout 🙅‍♀️ So the information passed in pnfs_osd_layoutupdate4 is only used to update the file's attributes. On top of the generic information the client can pass to the metadata server in LAYOUTCOMMIT (like the highest offset it wrote to, and the last time it modified the file), the client MAY use pnfs_osd_layoutupdate4 to convey the capacity consumed (or released) by writes using the layout, and to indicate that I/O errors were encountered by those writes 💖

6.1. pnfs_osd_deltaspaceused4 📊

/// union pnfs_osd_deltaspaceused4 switch (bool dsu_valid) {
///     case TRUE:
///         int64_t     dsu_delta;
///     case FALSE:
///         void;
/// };
///

pnfs_osd_deltaspaceused4 is used to convey space utilization information at the time of LAYOUTCOMMIT 📈 For the file system to keep its capacity-used information right, it needs to track how much capacity the client's WRITE operations consumed. In this protocol, the OSD returns the capacity consumed by a write ((*)), which can differ from the number of bytes written because of internal overhead like block-level allocation and indirect blocks 🧠 The client then reflects this back to the pNFS server so it can track quota accurately 🍓 The pNFS server can choose to trust this information from the clients, which means it can skip querying the OSDs at LAYOUTCOMMIT time 🙌 If the client can't get this information from the OSD, it simply returns an invalid olu_delta_space_used 🥺

6.2. pnfs_osd_layoutupdate4 💌

/// struct pnfs_osd_layoutupdate4 {
///     pnfs_osd_deltaspaceused4    olu_delta_space_used;
///     bool                        olu_ioerr_flag;
/// };
///

olu_delta_space_used is used to send capacity usage information back to the metadata server 💜

The olu_ioerr_flag is used when I/O errors were encountered while writing the file 🚨 The client MUST report those errors using the pnfs_osd_ioerr4 structure (see Section 8.1) at LAYOUTRETURN time 📮

If the client updated the file successfully before hitting the I/O errors, it MAY use LAYOUTCOMMIT to update the metadata server as described above ✨ Typically, in the error-free case, the server MAY turn around and update the file's attributes on the storage devices 🌈 But if I/O errors were encountered, the server better not try to write the new attributes on the storage devices until it gets the I/O error report 😭 So the client MUST set the olu_ioerr_flag to true 🫶 Note that in this case, the client SHOULD send both the LAYOUTCOMMIT and LAYOUTRETURN operations in the same COMPOUND RPC 💅

7. Recovering from Client I/O Errors ✨

okay so the pNFS client can run into errors when it talks straight to the object storage devices 😭 but the metadata server is the one in charge of fixing them 💅 when the LAYOUT4_OSD2_OBJECTS layout type is used, the client MUST report those I/O errors back to the server at LAYOUTRETURN time, using the pnfs_osd_ioerr4 structure (see Section 8.2) 🫶

the metadata server looks at the error and figures out what recovery is needed. that could mean fixing parity inconsistencies, recovering from media failures, or rebuilding missing objects 🔧

the server SHOULD recall any outstanding layouts so it gets exclusive write access to the stripes being recovered, and so other clients don't run into the same error 🙈 and in these cases the server MUST complete recovery before it hands out any new layouts for the affected byte ranges. no shortcuts, bestie 👑

it MAY be acceptable for the client to pass a matching error up to the application that started the I/O and throw away any unwritten data 🥲 but the client SHOULD try to retry the original I/O. it can ask for a new layout with LAYOUTGET and retry over that, or the client MAY just retry the I/O with regular NFS READ or WRITE operations through the metadata server 🦋 the client SHOULD try to grab a new layout and retry with OSD commands first, and only if the error keeps happening should it retry through the metadata server ✨

8. Object-Based Layout Return

layoutreturn_file4 is used in the LAYOUTRETURN operation to carry layout-type specific info to the server 🎀 it's defined in the NFSv4.1 [6] like this:

struct layoutreturn_file4 {
        offset4         lrf_offset;
        length4         lrf_length;
        stateid4        lrf_stateid;
        /* layouttype4 specific data */
        opaque          lrf_body<>;
};

union layoutreturn4 switch(layoutreturn_type4 lr_returntype) {
        case LAYOUTRETURN4_FILE:
                layoutreturn_file4      lr_layout;
        default:
                void;
};

struct LAYOUTRETURN4args {
        /* CURRENT_FH: file */
        bool                    lora_reclaim;
        layoutreturn_stateid    lora_recallstateid;
        layouttype4             lora_layout_type;
        layoutiomode4           lora_iomode;
        layoutreturn4           lora_layoutreturn;
};

if lora_layout_type is LAYOUT4_OSD2_OBJECTS, then the lrf_body opaque value is defined by the pnfs_osd_layoutreturn4 type 🌸

that type lets the client report I/O error info back to the metadata server, and it's defined below ✨

8.1. pnfs_osd_errno4

/// enum pnfs_osd_errno4 {
///     PNFS_OSD_ERR_EIO            = 1,
///     PNFS_OSD_ERR_NOT_FOUND      = 2,
///     PNFS_OSD_ERR_NO_SPACE       = 3,
///     PNFS_OSD_ERR_BAD_CRED       = 4,
///     PNFS_OSD_ERR_NO_ACCESS      = 5,
///     PNFS_OSD_ERR_UNREACHABLE    = 6,
///     PNFS_OSD_ERR_RESOURCE       = 7
/// };
///

pnfs_osd_errno4 is how error types get represented when read/write errors are reported to the metadata server 💜 the error codes are just hints that can help the server figure out the exact reason and fix it 🕵️‍♀️

  • PNFS_OSD_ERR_EIO means the operation failed because the object storage device had a failure while trying to access the object 💀 the most common cause is media errors, but other internal errors can cause it too. the metadata server should go take a closer look at the broken object, which is why it should be the default error code 🧐
  • PNFS_OSD_ERR_NOT_FOUND means the object ID points to an object that doesn't exist on the object storage device 🫥
  • PNFS_OSD_ERR_NO_SPACE means the object storage device ran out of free capacity during the operation 📦
  • PNFS_OSD_ERR_BAD_CRED means the security parameters aren't valid 🔐 the main reason is that the capability has expired, or the access policy tag (a.k.a., capability version number) has been changed to revoke capabilities. the client will need to return the layout and get a new one with fresh capabilities 🔄
  • PNFS_OSD_ERR_NO_ACCESS means the capability doesn't allow the requested operation 🚫 this shouldn't happen in normal operation, because the metadata server should hand out correct capabilities, or none at all 🙅‍♀️
  • PNFS_OSD_ERR_UNREACHABLE means the client didn't finish the I/O at the object storage device because of a communication failure 📡 it's undetermined whether or not the OSD actually ran the I/O 🌫️
  • PNFS_OSD_ERR_RESOURCE means the client never issued the I/O because of a local problem on the initiator (that's the client) side, like running out of memory 🧠 the client MUST guarantee that the OSD command was never dispatched to the OSD. that's a hard rule, no cap 🛑

8.2. pnfs_osd_ioerr4

/// struct pnfs_osd_ioerr4 {
///     pnfs_osd_objid4     oer_component;
///     length4             oer_comp_offset;
///     length4             oer_comp_length;
///     bool                oer_iswrite;
///     pnfs_osd_errno4     oer_errno;
/// };
///

pnfs_osd_ioerr4 is used to return error indications for objects that ran into errors during data transfers 🎈 these are just hints that something's wrong with that object. for each error, oer_component, oer_comp_offset, and oer_comp_length describe the object and the byte range inside the component object where the error happened 📍 oer_iswrite is set to true if the failed OSD operation was data modifying, and oer_errno is the type of error 🎯

component byte ranges in this optional structure are used for recovering the object, and the client MUST set them to cover all failed I/O operations to the component 🫶

8.3. pnfs_osd_layoutreturn4

/// struct pnfs_osd_layoutreturn4 {
///     pnfs_osd_ioerr4             olr_ioerr_report<>;
/// };
///

when OSD I/O operations fail, olr_ioerr_report<> is how you report those errors to the metadata server 📝 it's an array of pnfs_osd_ioerr4 elements, and each element is one error on the object named by oer_component 🧩 if there are no errors to report, the size of the olr_ioerr_report<> array is set to zero 🌈

9. Object-Based Creation Layout Hint

the layouthint4 type is defined in the NFSv4.1 [6] like this:

struct layouthint4 {
    layouttype4           loh_type;
    opaque                loh_body<>;
};

the client uses the layouthint4 structure to pass a hint about what type of layout it'd like created for a particular file ✨ if loh_type is LAYOUT4_OSD2_OBJECTS, then the loh_body opaque value is defined by the pnfs_osd_layouthint4 type 💖

9.1. pnfs_osd_layouthint4

/// union pnfs_osd_max_comps_hint4 switch (bool omx_valid) {
///     case TRUE:
///         uint32_t            omx_max_comps;
///     case FALSE:
///         void;
/// };
///
/// union pnfs_osd_stripe_unit_hint4 switch (bool osu_valid) {
///     case TRUE:
///         length4             osu_stripe_unit;
///     case FALSE:
///         void;
/// };
///
/// union pnfs_osd_group_width_hint4 switch (bool ogw_valid) {
///     case TRUE:
///         uint32_t            ogw_group_width;
///     case FALSE:
///         void;
/// };
///
/// union pnfs_osd_group_depth_hint4 switch (bool ogd_valid) {
///     case TRUE:
///         uint32_t            ogd_group_depth;
///     case FALSE:

///         void;
/// };
///
/// union pnfs_osd_mirror_cnt_hint4 switch (bool omc_valid) {
///     case TRUE:
///         uint32_t            omc_mirror_cnt;
///     case FALSE:
///         void;
/// };
///
/// union pnfs_osd_raid_algorithm_hint4 switch (bool ora_valid) {
///     case TRUE:
///         pnfs_osd_raid_algorithm4    ora_raid_algorithm;
///     case FALSE:
///         void;
/// };
///
/// struct pnfs_osd_layouthint4 {
///     pnfs_osd_max_comps_hint4        olh_max_comps_hint;
///     pnfs_osd_stripe_unit_hint4      olh_stripe_unit_hint;
///     pnfs_osd_group_width_hint4      olh_group_width_hint;
///     pnfs_osd_group_depth_hint4      olh_group_depth_hint;
///     pnfs_osd_mirror_cnt_hint4       olh_mirror_cnt_hint;
///     pnfs_osd_raid_algorithm_hint4   olh_raid_algorithm_hint;
/// };
///

this type conveys hints for the data map you'd like 🗺️ every parameter is optional, so the client can give values for just the ones it cares about. for example, it can give a hint for the number of mirrored components, no matter which RAID algorithm got picked for the file 🪞 the server should make an attempt to honor the hints, but it can ignore any or all of them whenever it wants, and that won't make the respective CREATE operation fail 🥰

the olh_max_comps_hint can be used to limit the total number of component objects that make up the file 🧚 all the other hints map directly to the different fields of pnfs_osd_data_map4 💜

10. Layout Segments

the pnfs layout operations work on logical byte ranges 🦋 the protocol doesn't require any relationship between the byte ranges used in LAYOUTGET to get layouts and the ones used in CB_LAYOUTRECALL, LAYOUTCOMMIT, or LAYOUTRETURN. BUT, using OSD byte-range capabilities puts limits on these operations, since the capabilities tied to layout segments can't be merged or split 🙈 the following guidelines should be followed for object-based layouts to work properly 🫶

10.1. CB_LAYOUTRECALL and LAYOUTRETURN

in general, the object-based layout driver should keep track of each layout segment it got, and record the segment's iomode, offset, and length 📒 the server should let the client get multiple overlapping layout segments, but it's free to recall the layout to prevent overlap 🔁

when CB_LAYOUTRECALL comes in, the client should return all layout segments that match the given iomode and overlap the recalled range 📬 when it returns those layouts for this byte range with LAYOUTRETURN, the client MUST NOT return a sub-range of a layout segment it has. each LAYOUTRETURN sent MUST completely cover at least one outstanding layout segment 🧩

the server, in turn, should release any segment that exactly matches the clientid, iomode, and byte range given in LAYOUTRETURN 🎀 if there's no exact match, the server should release all layout segments that match the clientid and iomode and are fully contained in the returned byte range. if none are found, and the byte range is a subset of an outstanding layout segment for the same clientid and iomode, then the client can be considered malfunctioning, and the server SHOULD recall all layouts from this client to reset its state 🚨 if this keeps happening, the server SHOULD deny all LAYOUTGETs from this client 🙅‍♀️

10.2. LAYOUTCOMMIT

LAYOUTCOMMIT is only used by object-based pNFS to send modified attributes hints and/or to report the presence of I/O errors to the metadata server (MDS) 💅 that's why the offset and length in LAYOUTCOMMIT4args are reserved for future use, and should be set to 0 🌸

11. Recalling Layouts ✨

Okay so, the object-based metadata server should recall outstanding layouts in these cases, bestie 💅

  • When the file's security policy changes, i.e., Access Control Lists (ACLs) or permission mode bits are set. 🔐
  • When the file's aggregation map changes, rendering outstanding layouts invalid. 🗺️
  • When there are sharing conflicts. For example, the server will issue stripe-aligned layout segments for RAID-5 objects. To prevent corruption of the file's parity, multiple clients must not hold valid write layouts for the same stripes. An outstanding READ/WRITE (RW) layout should be recalled when a conflicting LAYOUTGET is received from a different client for LAYOUTIOMODE4_RW and for a byte range overlapping with the outstanding layout segment. 🦋

11.1. CB_RECALL_ANY

The metadata server can use the CB_RECALL_ANY callback operation to notify the client to return some or all of its layouts. 📣 The NFSv4.1 [6] defines the following types:

const RCA4_TYPE_MASK_OBJ_LAYOUT_MIN     = 8;
const RCA4_TYPE_MASK_OBJ_LAYOUT_MAX     = 9;

struct  CB_RECALL_ANY4args      {
    uint32_t        craa_objects_to_keep;
    bitmap4         craa_type_mask;
};

Typically, CB_RECALL_ANY gets used to recall client state when the server needs to reclaim resources, so it's giving clean-up energy 🧹. The craa_type_mask bitmap specifies the type of resources that are recalled, and the craa_objects_to_keep value specifies how many of the recalled objects the client is allowed to keep. The object-based layout type mask flags are defined as follows. They represent the iomode of the recalled layouts. In response, the client SHOULD return layouts of the recalled iomode that it needs the least, keeping at most craa_objects_to_keep object-based layouts. 🎀

/// enum pnfs_osd_cb_recall_any_mask {
///     PNFS_OSD_RCA4_TYPE_MASK_READ = 8,
///     PNFS_OSD_RCA4_TYPE_MASK_RW   = 9
/// };
///

The PNFS_OSD_RCA4_TYPE_MASK_READ flag notifies the client to return layouts of iomode LAYOUTIOMODE4_READ. Similarly, the PNFS_OSD_RCA4_TYPE_MASK_RW flag notifies the client to return layouts of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client is notified to return layouts of either iomode. 🌈

12. Client Fencing 🚧

In cases where clients are uncommunicative and their lease has expired, or when clients fail to return recalled layouts within a lease period at the least (see "Recalling a Layout"[6]), the server MAY revoke client layouts and/or device address mappings and reassign these resources to other clients. Ngl, that's a lot of drama 😭. To avoid data corruption, the metadata server MUST fence off the revoked clients from the respective objects as described in Section 13.4. 🔥

13. Security Considerations 🔐💀

Okay, this section is the main character of the whole doc, so pay attention bestie 👑. The pNFS extension partitions the NFSv4 file system protocol into two parts, the control path and the data path (storage protocol). The control path contains all the new operations described by this extension; all existing NFSv4 security mechanisms and features apply to the control path. The combination of components in a pNFS system is required to preserve the security properties of NFSv4 with respect to an entity accessing data via a client, including security countermeasures to defend against threats that NFSv4 provides defenses for in environments where these threats are considered significant. 🛡️

The metadata server enforces the file access-control policy at LAYOUTGET time. The client should use suitable authorization credentials for getting the layout for the requested iomode (READ or RW), and the server verifies the permissions and ACL for these credentials, possibly returning NFS4ERR_ACCESS if the client is not allowed the requested iomode. If the LAYOUTGET operation succeeds, the client receives, as part of the layout, a set of object capabilities allowing it I/O access to the specified objects corresponding to the requested iomode. When the client acts on I/O operations on behalf of its local users, it MUST authenticate and authorize the user by issuing respective OPEN and ACCESS calls to the metadata server, similar to having NFSv4 data delegations. If access is allowed, the client uses the corresponding (READ or RW) capabilities to perform the I/O operations at the object storage devices. When the metadata server receives a request to change a file's permissions or ACL, it SHOULD recall all layouts for that file, and it MUST change the capability version attribute on all objects comprising the file to implicitly invalidate any outstanding capabilities before committing to the new permissions and ACL. Doing this will ensure that clients re-authorize their layouts according to the modified permissions and ACL by requesting new layouts. Recalling the layouts in this case is courtesy of the server, intended to prevent clients from getting an error on I/Os done after the capability version changed. 🙏

The object storage protocol MUST implement the security aspects described in version 1 of the T10 OSD protocol definition [1]. The standard defines four security methods: NOSEC, CAPKEY, CMDRSP, and ALLDATA. To provide minimum level of security allowing verification and enforcement of the server access control policy using the layout security credentials, the NOSEC security method MUST NOT be used for any I/O operation. 🚫 The remainder of this section gives an overview of the security mechanism described in that standard. The goal is to give the reader a basic understanding of the object security model. Any discrepancies between this text and the actual standard are obviously to be resolved in favor of the OSD standard. 📚

13.1. OSD Security Data Types

There are three main data types associated with object security: a capability, a credential, and security parameters. The capability is a set of fields that specifies an object and what operations can be performed on it. A credential is a signed capability. Only a security manager that knows the secret device keys can correctly sign a capability to form a valid credential. In pNFS, the file server acts as the security manager and returns signed capabilities (i.e., credentials) to the pNFS client. The security parameters are values computed by the issuer of OSD commands (i.e., the client) that prove they hold valid credentials. The client uses the credential as a signing key to sign the requests it makes to OSD, and puts the resulting signatures into the security_parameters field of the OSD command. The object storage device uses the secret keys it shares with the security manager to validate the signature values in the security parameters. 🔑

The security types are opaque to the generic layers of the pNFS client. The credential contents are defined as opaque within the pnfs_osd_object_cred4 type. Instead of repeating the definitions here, the reader is referred to Section 4.9.2.2 of the OSD standard. 📖

13.2. The OSD Security Protocol

The object storage protocol relies on a cryptographically secure capability to control accesses at the object storage devices. Capabilities are generated by the metadata server, returned to the client, and used by the client as described below to authenticate their requests to the object-based storage device. Capabilities therefore achieve the required access and open mode checking. They allow the file server to define and check a policy (e.g., open mode) and the OSD to enforce that policy without knowing the details (e.g., user IDs and ACLs). 🧠

Since capabilities are tied to layouts, and since they are used to enforce access control, when the file ACL or mode changes, the outstanding capabilities MUST be revoked to enforce the new access permissions. The server SHOULD recall layouts to allow clients to gracefully return their capabilities before the access permissions change. 🫶

Each capability is specific to a particular object, an operation on that object, a byte range within the object (in OSDv2), and has an explicit expiration time. The capabilities are signed with a secret key that is shared by the object storage devices and the metadata managers. Clients do not have device keys, so they are unable to forge the signatures in the security parameters. The combination of a capability, the OSD System ID, and a signature is called a "credential" in the OSD specification. 🎟️

The details of the security and privacy model for object storage are defined in the T10 OSD standard. The following sketch of the algorithm should help the reader understand the basic model. 🧚

LAYOUTGET returns a CapKey and a Cap, which, together with the OSD System ID, are also called a credential. It is a capability and a signature over that capability and the SystemID. The OSD Standard refers to the CapKey as the "Credential integrity check value" and to the ReqMAC as the "Request integrity check value".

CapKey = MAC<SecretKey>(Cap, SystemID)
Credential = {Cap, SystemID, CapKey}

The client uses CapKey to sign all the requests it issues for that object using the respective Cap. In other words, the Cap appears in the request to the storage device, and that request is signed with the CapKey as follows:

ReqMAC = MAC<CapKey>(Req, ReqNonce)
Request = {Cap, Req, ReqNonce, ReqMAC}

The following is sent to the OSD: {Cap, Req, ReqNonce, ReqMAC}. The OSD uses the SecretKey it shares with the metadata server to compare the ReqMAC the client sent with a locally computed value:

LocalCapKey = MAC<SecretKey>(Cap, SystemID)
LocalReqMAC = MAC<LocalCapKey>(Req, ReqNonce)

and if they match, the OSD assumes that the capabilities came from an authentic metadata server and allows access to the object, as allowed by the Cap. ✅

13.3. Protocol Privacy Requirements

Note that if the server LAYOUTGET reply, holding CapKey and Cap, is snooped by another client, it can be used to generate valid OSD requests (within the Cap access restrictions). 👀 Yikes, that's a whole vibe check 😬

To provide the required privacy requirements for the capability key returned by LAYOUTGET, the GSS-API [7] framework can be used, e.g., by using the RPCSEC_GSS privacy method to send the LAYOUTGET operation or by using the SSV key to encrypt the oc_capability_key using the GSS_Wrap() function. Two general ways to provide privacy in the absence of GSS-API that are independent of NFSv4 are either an isolated network such as a VLAN or a secure channel provided by IPsec [15]. 🌸

13.4. Revoking Capabilities 🔐

okay bestie, so at any time the metadata server can invalidate every outstanding capability on an object just by changing its POLICY ACCESS TAG attribute 💅 The tag's value is part of a capability, so it has to match the state of the object attribute. If they don't match, the OSD rejects the access with the sense key set to ILLEGAL REQUEST and an additional sense code set to INVALID FIELD IN CDB 🙅‍♀️ When a client tries a capability and gets rejected like that, it should issue a LAYOUTCOMMIT for the object and specify PNFS_OSD_BAD_CRED in the olr_ioerr_report parameter. Then the client may elect to send a compound LAYOUTRETURN/LAYOUTGET (or LAYOUTCOMMIT/LAYOUTRETURN/LAYOUTGET) to try to grab a refreshed set of capabilities 🔄✨

The metadata server may change the access policy tag on an object whenever it wants, for any reason 🌸 (just know there's likely a performance penalty, especially if the object has outstanding layouts 😭). The metadata server MUST revoke outstanding capabilities when any one of these happens:

  • the permissions on the object change,
  • a conflicting mandatory byte-range lock is granted, or
  • a layout is revoked and reassigned to another client.

A pNFS client will typically hold one layout for each byte range, for either READ or READ/WRITE 🦋 The metadata server checks the client's credentials at LAYOUTGET time, and it's the client's job to enforce access control among multiple users accessing the same file. It's neither required nor expected that the pNFS client gets a separate layout for every user sharing an object, so no need to be extra about it 🙈 The client SHOULD use OPEN and ACCESS calls to check user permissions when it does I/O, so the server's access control policies get enforced correctly. The result of the ACCESS operation can be cached while the client holds a valid layout, because the server is expected to recall layouts when the file's access permissions or ACL change 💜

14. IANA Considerations 🦋

As described in NFSv4.1 [6], IANA has assigned new layout type numbers. This document defines the protocol for the existing layout type number, LAYOUT4_OSD2_OBJECTS, so it requires no further actions for IANA ✨ nothing to do here, bestie 🫶

15. References 📚

15.1. Normative References

  • [1] Weber, R., "Information Technology - SCSI Object-Based Storage Device Commands (OSD)", ANSI INCITS 400-2004, December 2004.
  • [2] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [3] Eisler, M., "XDR: External Data Representation Standard", STD 67, RFC 4506, May 2006.
  • [4] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "Network File System (NFS) Version 4 Minor Version 1 External Data Representation Standard (XDR) Description", RFC 5662, January 2010.
  • [5] IETF Trust, "Legal Provisions Relating to IETF Documents", November 2008, <http://trustee.ietf.org/docs/IETF-Trust-License-Policy.pdf>.
  • [6] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "Network File System (NFS) Version 4 Minor Version 1 Protocol", RFC 5661, January 2010.
  • [7] Linn, J., "Generic Security Service Application Program Interface Version 2, Update 1", RFC 2743, January 2000.
  • [8] Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and E. Zeidner, "Internet Small Computer Systems Interface (iSCSI)", RFC 3720, April 2004.
  • [9] Weber, R., "SCSI Primary Commands - 3 (SPC-3)", ANSI INCITS 408-2005, October 2005.
  • [10] Krueger, M., Chadalapaka, M., and R. Elliott, "T11 Network Address Authority (NAA) Naming Format for iSCSI Node Names", RFC 3980, February 2005.
  • [11] IEEE, "Guidelines for 64-bit Global Identifier (EUI-64) Registration Authority", <http://standards.ieee.org/regauth/oui/tutorials/EUI64.html>.
  • [12] Tseng, J., Gibbons, K., Travostino, F., Du Laney, C., and J. Souza, "Internet Storage Name Service (iSNS)", RFC 4171, September 2005.
  • [13] Weber, R., "SCSI Architecture Model - 3 (SAM-3)", ANSI INCITS 402-2005, February 2005.

15.2. Informative References

  • [14] Weber, R., "SCSI Object-Based Storage Device Commands -2 (OSD-2)", January 2009, <http://www.t10.org/cgi-bin/ac.pl?t=f&f=osd2r05a.pdf>.
  • [15] Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, December 2005.
  • [16] T10 1415-D, "SCSI RDMA Protocol (SRP)", ANSI INCITS 365-2002, December 2002.
  • [17] T11 1619-D, "Fibre Channel Framing and Signaling - 2 (FC-FS-2)", ANSI INCITS 424-2007, February 2007.
  • [18] T10 1601-D, "Serial Attached SCSI - 1.1 (SAS-1.1)", ANSI INCITS 417-2006, June 2006.
  • [19] MacWilliams, F. and N. Sloane, "The Theory of Error-Correcting Codes, Part I", 1977.

Appendix A. Acknowledgments 🌈

Big thank-you 🫶 to Todd Pisek, who co-edited the initial versions of this document. Daniel E. Messinger, Pete Wyckoff, Mike Eisler, Sean P. Turner, Brian E. Carpenter, Jari Arkko, David Black, and Jason Glasgow reviewed and commented on it. Icons, all of them 👑

Authors' Addresses 💌

Benny Halevy
Panasas, Inc.
1501 Reedsdale St. Suite 400
Pittsburgh, PA  15233
USA

Phone: +1-412-323-3500
EMail: bhalevy@panasas.com
URI:   http://www.panasas.com/


Brent Welch
Panasas, Inc.
6520 Kaiser Drive
Fremont, CA  95444
USA

Phone: +1-510-608-7770
EMail: welch@panasas.com
URI:   http://www.panasas.com/


Jim Zelenka
Panasas, Inc.
1501 Reedsdale St. Suite 400
Pittsburgh, PA  15233
USA

Phone: +1-412-323-3500
EMail: jimz@panasas.com
URI:   http://www.panasas.com/
📦 where this page comes from