RFC 9561

Using the Parallel NFS (pNFS) SCSI Layout to Access Non-Volatile Memory Express (NVMe) Storage Devices

📅 published April 2024, from ietf

RFC 9561: Using the Parallel NFS (pNFS) SCSI Layout to Access Non-Volatile Memory Express (NVMe) Storage Devices 💖

okay bestie, let's do the title block tea ☕✨ this is an IETF Request for Comments, published in April 2024, and it's a Standards Track doc (ISSN 2070-1721) 🎀. The main author/editor is C. Hellwig, Ed., and the co-authors are C. Lever from Oracle, S. Faibish from Opendrives.com, and D. Black from Dell Technologies 🫶. The header doesn't list anything this RFC obsoletes or updates, so no drama there 💅.

Abstract ✨

okay so this doc explains how to use the pNFS Small Computer System Interface (SCSI) Layout Type to reach storage devices that speak the Non-Volatile Memory Express (NVMe) protocol family 🦋.

Status of This Memo 📜

this one is an Internet Standards Track document 🔥.

it's a product of the Internet Engineering Task Force (IETF), so it represents the consensus of the IETF community ✨. it got public review and was approved for publication by the Internet Engineering Steering Group (IESG). if you want more on Internet Standards, check Section 2 of RFC 7841 🧚.

info about the current status, errata, and how to send feedback lives at https://www.rfc-editor.org/info/rfc9561 💜.

Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved.

this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they were on the publication date 📚. please actually read those, bestie, because they explain your rights and restrictions. 🙈 also, any Code Components pulled out of this doc must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as the Revised BSD License describes 🎀.

Table of Contents 🍓

   1.  Introduction
     1.1.  Requirements Language
     1.2.  General Definitions
     1.3.  Numerical Conventions
   2.  SCSI Layout Mapping to NVMe
     2.1.  Volume Identification
     2.2.  Client Fencing
       2.2.1.  PRs - Key Registration
       2.2.2.  PRs - MDS Registration and Reservation
       2.2.3.  Fencing Action
       2.2.4.  Client Recovery after a Fence Action
     2.3.  Volatile Write Caches
   3.  Security Considerations
   4.  IANA Considerations
   5.  References
     5.1.  Normative References
     5.2.  Informative References
   Acknowledgements
   Authors' Addresses

1. Introduction ✨

okay so NFSv4.1 [RFC8881] has this pNFS feature that lets reads and writes happen by means other than sending read and write operations to the server 🧚. the server, acting as the metadata server, handles file and directory metadata, while separate means handle the actual reads and writes 💅.

those other means are defined by individual mapping types, and they usually come with their own specs 🎀.

the pNFS SCSI layout [RFC8154] is a layout type that lets NFS clients do I/O straight to block storage devices, bypassing the Metadata Server (MDS) 🦋. it's specified using concepts from the SCSI protocol family for the data path to those storage devices ✨.

NVM Express (NVMe), aka the Non-Volatile Memory Host Controller Interface Specification, is a set of specs for talking to storage devices over a bunch of protocols, like PCI Express (PCIe), Fibre Channel (FC), TCP/IP, or Remote Direct Memory Access (RDMA) networking 🌸. NVMe is the main protocol for PCIe Solid State Disks (SSDs) right now, and it's increasingly replacing SCSI-based protocols like iSCSI for remote storage access 🔥.

this doc defines how NVMe Namespaces using the NVM Command Set [NVME-NVM], exported by NVMe Controllers that implement the NVMe Base specification [NVME-BASE], can be used as storage devices with the SCSI Layout Type 💖. the definition doesn't depend on the transport the NVMe Controller uses, so it works with controllers over PCIe, RDMA, TCP, and FC alike 🌈.

this doc does not amend the existing SCSI layout document 🙈. instead, it explains how NVMe Namespaces fit into the SCSI Layout by mapping the SCSI constructs from the SCSI layout doc onto their NVMe equivalents 🫶.

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 🎀

1.2. General Definitions

these definitions are here to give you context 💅.

  • Client: the "client" is the entity that accesses the NFS server's resources 🦋. it may be an application with the logic to reach the NFS server directly, or it may be part of the operating system that provides remote file system services to a set of applications ✨.
  • Metadata Server (MDS): the Metadata Server (MDS) is the entity in charge of coordinating client access to a set of file systems, and it's identified by a server owner 🧚.

1.3. Numerical Conventions

numbers defined in the SCSI specs (like [SPC5]) and the NVMe specs (like [NVME-BASE]) follow the same conventions as those specs 🔢. a 'b' suffix means binary (base 2), so 110b = 6 decimal 💜. an 'h' suffix means hexadecimal (base 16), so 1ch = 28 decimal 🌸.

2. SCSI Layout Mapping to NVMe 🦋

the SCSI layout definition [RFC8154] only refers to a few SCSI-specific concepts directly 💅. this doc gives you a mapping from those SCSI concepts to NVM Express concepts, for when you use the pNFS SCSI layout with NVMe namespaces 🎀.

2.1. Volume Identification

the pNFS SCSI layout uses the Device Identification Vital Product Data (VPD) page (page code 83h) from [SPC5] to identify the devices a layout uses 🔥. implementations that use NVMe namespaces as storage devices map NVMe namespace identifiers onto a subset of the identifiers that the Device Identification VPD page supports for SCSI logical units ✨.

to be used as storage devices for the pNFS SCSI layout, NVMe namespaces MUST support either the IEEE Extended Unique Identifier (EUI64) or the Namespace Globally Unique Identifier (NGUID) value. that value is reported in a Namespace Identification Descriptor, the I/O Command Set Independent Identify Namespace data structure, and the Identify Namespace data structure, NVM Command Set [NVME-BASE]. if NGUID is available, use it, because it's the larger identifier 💖.

      |  Note: The PS_DESIGNATOR_T10 and PS_DESIGNATOR_NAME have no
      |  equivalent in NVMe and cannot be used to identify NVMe storage
      |  devices.

the pnfs_scsi_base_volume_info4 structure for an NVMe namespace SHALL be built like this 🧚:

  1. the sbv_code_set field SHALL be set to PS_CODE_SET_BINARY.
  2. the pnfs_scsi_designator_type field SHALL be set to PS_DESIGNATOR_EUI64.
  3. the sbv_designator field SHALL contain either the NGUID or the EUI64 identifier for the namespace. if both NGUID and EUI64 identifiers are available, then the NGUID identifier SHOULD be used, because it's the larger identifier 🌈.

RFC 8154 [RFC8154] specifies the sbv_designator field as an XDR variable length opaque<> (see Section 4.10 of RFC 4506 [RFC4506]) 🤓. the length of that XDR opaque<> value (part of its XDR representation) tells you which NVMe identifier is present 🔍. that length MUST be 16 octets for an NVMe NGUID identifier and MUST be 8 octets for an NVMe EUI64 identifier 🫶. all other lengths MUST NOT be used with an NVMe namespace 🙅‍♀️.

2.2. Client Fencing

the SCSI layout uses Persistent Reservations (PRs) to do client fencing 🔒. for this to work, both the MDS and the Clients have to register a key with the storage device, and the MDS has to create a reservation on the storage device 💅.

the following subsections give a full mapping of the required PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT SCSI commands [SPC5] to the NVMe commands that MUST be used when you use NVMe namespaces as storage devices for the pNFS SCSI layout 🎀.

2.2.1. PRs - Key Registration

on NVMe namespaces, reservation keys get registered with the Reservation Register command (see Section 7.3 of [NVME-BASE]). set the Reservation Register Action (RREGA) field to 000b (aka Register Reservation Key) and put the reservation key in the New Reservation Key (NRKEY) field 🦋.

reservation keys get unregistered with the Reservation Register command too, but this time RREGA is set to 001b (aka Unregister Reservation Key), and the reservation key goes in the Current Reservation Key (CRKEY) field ✨.

here's the big difference between SCSI Persistent Reservations and NVMe Reservations 👀: NVMe reservation keys always apply to all controllers a host uses (as shown by the NVMe Host Identifier). that's analogous to setting the ALL_TG_PT bit when you register a SCSI Reservation Key, and NVMe Reservations always support it 💖. SCSI support for ALL_TG_PT is inconsistent and can't be relied on, so this is a glow-up 🌸. registering a reservation key with a namespace creates an association between a host and a namespace 🧚. a host that's a registrant of a namespace may use any controller it's associated with (that is, one with the same Host Identifier, see Section 5.27.1.25 of [NVME-BASE]) to access that namespace as a registrant 🫶.

2.2.2. PRs - MDS Registration and Reservation

before returning a PNFS_SCSI_VOLUME_BASE volume to the client, the MDS needs to prep the volume for fencing using PRs 🔥. this happens by registering the reservation generated for the MDS with the device (see Section 2.2.1), followed by a Reservation Acquire command (see Section 7.2 of [NVME-BASE]). set the Reservation Acquire Action (RACQA) field to 000b (aka Acquire) and the Reservation Type (RTYPE) field to 4h (aka Exclusive Access - Registrants Only Reservation) 👑.

2.2.3. Fencing Action

if a client stops responding, the MDS fences it by running a Reservation Acquire command (see Section 7.2 of [NVME-BASE]) 🚫. the Reservation Acquire Action (RACQA) field gets set to 001b (aka Preempt) or 010b (aka Preempt and Abort) 💅. the Current Reservation Key (CRKEY) field gets set to the server's reservation key, and the Preempt Reservation Key (PRKEY) field gets set to the reservation key of the non-responding client 🎀. the Reservation Type (RTYPE) field gets set to 4h (aka Exclusive Access - Registrants Only Reservation) ✨.

the client can tell I/O errors caused by fencing apart from other errors by checking for the Reservation Conflict NVMe status code 🦋.

2.2.4. Client Recovery after a Fence Action 💅

okay so if an NVMe command the client sends to the storage device comes back with a non-retryable error (check the DNR bit in Figure 92 of [NVME-BASE] 🙈), the client MUST commit all layouts that use that storage device through the MDS, return every outstanding layout for the device, forget the device ID, and unregister the reservation key. main character recovery arc 🎀

2.3. Volatile Write Caches 🧊

for NVMe controllers, a volatile write cache counts as enabled when two things are true. First, bit 0 of the Volatile Write Cache (VWC) field in the Identify Controller data structure, I/O Command Set Independent (see Figure 275 in [NVME-BASE]), is set. Second, the Volatile Write Cache Enable (WCE) bit (i.e., bit 00) in the Volatile Write Cache Feature (Feature Identifier 06h, see Section 5.27.1.4 of [NVME-BASE]) is set. ✨

now the important part 🔥 if a volatile write cache is enabled on an NVMe namespace that's being used as a storage device for the pNFS SCSI layout, the pNFS server (MDS) MUST use the NVMe Flush command to flush that cache to stable storage before the LAYOUTCOMMIT operation returns. It does this by issuing the Flush command (see Section 7.1 of [NVME-BASE]). The NVMe Flush command is the equivalent of the SCSI SYNCHRONIZE CACHE commands, so it's basically the same energy, just in NVMe form 💜

3. Security Considerations 🔒

okay this section is serious, so read it with your whole chest 😭

NFSv4 clients talk to NFSv4 metadata servers over the NFSv4 protocol. The security considerations in [RFC8881] apply to what the client does with the metadata server. BUT, NFSv4 clients and servers touch NVMe storage devices at a lower layer than NFSv4 does. That means NFSv4 and RPC security don't directly cover the I/Os going to data servers over NVMe. If you want the security story for direct block storage access from NFS clients, check Section 2.4.6 (Extents Are Permissions) and Section 4 (Security Considerations) of [RFC8154]. 🧚

pNFS with an NVMe layout can run over NVMe transports (for example NVMe over PCIe [NVME-PCIE]) that give you essentially no extra security functionality. 💀 Or, pNFS can be used with storage protocols like NVMe over TCP [NVME-TCP], which can provide significant transport layer security. much better 🫶

it's the job of whoever administers and deploys pNFS with an NVMe layout to make sure the right protection is in place for that protocol. The right level depends on the deployment environment and on how sensitive the data and storage devices are. When you're using IP-based storage protocols like NVMe over TCP, data confidentiality and integrity SHOULD be provided for traffic between pNFS clients and NVMe storage devices, using a secure communication protocol such as Transport Layer Security (TLS) [RFC8446]. For NVMe over TCP, TLS SHOULD be used as described in [NVME-TCP] to protect traffic between pNFS clients and NVMe namespaces used as storage devices. 🔐

a secure communication protocol might not be needed for pNFS with NVMe layouts in environments where physical and/or logical security measures (e.g., air gaps, isolated VLANs) give effective access control that matches the sensitivity and value of the storage devices and data involved. For example, public website contents are probably way less sensitive than a database holding personal identifying information, passwords, and other authentication credentials 🙈

physical security is the usual approach for protocols that aren't based on IP, so keep that in mind when you plan your setup 🔒

if the storage protocol's security requirements can't be met, then pNFS with an NVMe layout SHOULD NOT be deployed. 🚫

when security is available for the data server storage protocol, it usually works at a different granularity and with a different idea of identity than NFSv4. For instance, NFSv4 controls user access to files, while NVMe controls initiator access to volumes. Just like pNFS with the block layout type [RFC5663], the pNFS client is responsible for making sure the correct mappings exist between these security layers. If your environment needs client-side protection from access to storage outside of the layout to be stronger than that, then pNFS with a SCSI layout on a NVMe namespace SHOULD NOT be deployed. 🙅‍♀️

like other block-oriented pNFS layout types, the metadata server can fence off a client's access to the data on an NVMe namespace used as a storage device. If a metadata server revokes a layout, the client's access MUST be terminated at the storage devices via fencing, as specified in Section 2.2. After that, the client still gets a chance to acquire a new layout. sis, we love a comeback arc 🦋

4. IANA Considerations 📋

no IANA actions here! this document has none. 🌸

5. References 📚

5.1. Normative References

5.2. Informative References

Acknowledgements 🫶

Carsten Bormann converted an earlier RFCXML v2 source for this document to a markdown source format. 🍓

David Noveck provided ample feedback to various drafts of this document. he's the real MVP 👑

Authors' Addresses 🎀

  • Christoph Hellwig (editor)
    • Email: hch@lst.de
  • Charles Lever
    • Oracle Corporation
    • United States of America
    • Email: chuck.lever@oracle.com
  • Sorin Faibish
    • Opendrives.com
    • 11 Selwyn Road
    • Newton, MA 02461
    • United States of America
    • Phone: +1 617-510-0422
    • Email: s.faibish@opendrives.com
  • David L. Black
    • Dell Technologies
    • 176 South Street
    • Hopkinton, MA 01748
    • United States of America
    • Email: david.black@dell.com
📦 where this page comes from