RFC 3783

Small Computer Systems Interface (SCSI) Command Ordering Considerations with iSCSI

📅 published May 2004, from ietf

RFC 3783: Small Computer Systems Interface (SCSI) Command Ordering Considerations with iSCSI ✨

okay bestie, this one's the Informational RFC (💅 not a standard, just really useful info) about how SCSI commands stay in order when they're sent over iSCSI. It's written by M. Chadalapaka and R. Elliott from Hewlett-Packard Co., and it dropped in May 2004 🎀. The header doesn't list any obsoletes or updates, so it's standing on its own 🦋.

Status of this Memo 🌸

this memo is just here to give the Internet community some info 📖. it doesn't set any Internet standard, and you can share it as much as you want (distribution is unlimited) 💖.

Copyright (C) The Internet Society (2004). All Rights Reserved. the Internet Society owns the copyright, so no sneaky copying without permission 🙈.

Abstract 🧚

okay so iSCSI is a SCSI transport protocol built to run on top of TCP 💜. its session is basically the same as the classic SCSI "I_T nexus," which is the logical relationship between an Initiator and a Target (I and T) that you need in order to talk through the SCSI family of protocols. the iSCSI session gives you ordered command delivery from the SCSI initiator to the SCSI target, and this doc explains why the session got designed the way it is today. it also connects the SCSI command ordering features from the T10 specs to iSCSI concepts, and then gives system designers guidance on how to build real command ordering solutions on top of iSCSI 👑✨.

Table of Contents 📋

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  2
   2.  Definitions and Acronyms . . . . . . . . . . . . . . . . . . .  3
       2.1.  Definitions. . . . . . . . . . . . . . . . . . . . . . .  3
       2.2.  Acronyms . . . . . . . . . . . . . . . . . . . . . . . .  4
   3.  Overview of the iSCSI Protocol . . . . . . . . . . . . . . . .  4
       3.1.  Protocol Mapping Description . . . . . . . . . . . . . .  4
       3.2.  The I_T Nexus Model. . . . . . . . . . . . . . . . . . .  5
       3.3.  Ordered Command Delivery . . . . . . . . . . . . . . . .  6
             3.3.1.  Questions. . . . . . . . . . . . . . . . . . . .  6
             3.3.2.  The Session Guarantee. . . . . . . . . . . . . .  6
             3.3.3.  Ordering Onus. . . . . . . . . . . . . . . . . .  7
             3.3.4.  Design Intent. . . . . . . . . . . . . . . . . .  7

   4.  The Command Ordering Scenario. . . . . . . . . . . . . . . . .  8
       4.1.  SCSI Layer . . . . . . . . . . . . . . . . . . . . . . .  8
             4.1.1.  Command Reference Number (CRN) . . . . . . . . .  8
             4.1.2.  Task Attributes. . . . . . . . . . . . . . . . .  8
             4.1.3.  Auto Contingent Allegiance (ACA) . . . . . . . .  8
             4.1.4.  UA Interlock . . . . . . . . . . . . . . . . . .  9
       4.2.  iSCSI Layer. . . . . . . . . . . . . . . . . . . . . . .  9
   5.  Connection Failure Considerations. . . . . . . . . . . . . . .  9
   6.  Command Ordering System Considerations . . . . . . . . . . . . 10
   7.  Reservation Considerations . . . . . . . . . . . . . . . . . . 11
   8.  Security Considerations. . . . . . . . . . . . . . . . . . . . 12
   9.  References and Bibliography. . . . . . . . . . . . . . . . . . 12
       9.1.  Normative References.. . . . . . . . . . . . . . . . . . 12
       9.2.  Informative References . . . . . . . . . . . . . . . . . 12
   10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 12
   11. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 13
   12. Full Copyright Statement . . . . . . . . . . . . . . . . . . . 14

1. Introduction ✨

okay so iSCSI is a SCSI transport protocol ([iSCSI]) made to let SCSI application protocols run over TCP/IP networks, which can even include the Internet 🌈. the Internet is huge, so iSCSI unlocks some seriously exciting new SCSI applications 🤩. some new areas where iSCSI is a total game changer are:

  • a) Larger (diameter) Storage Area Networks (SANs) than had been possible until now
  • b) Asynchronous remote mirroring
  • c) Remote tape vaulting

each of these takes advantage of the basically unlimited geographic distance iSCSI makes possible between a SCSI initiator and a SCSI target 🗺️. and because the delays get really long in these setups, the initiator has a super strong reason to stream SCSI commands back-to-back instead of waiting for the SCSI status of each previous command 🏃‍♀️💨. command streaming is mainly used by two kinds of apps. one kind doesn't really care whether commands run in order, but the other kind does rely on ordered execution (there's an application-level dependency on the ordering among SCSI commands) 🙈. so cases b) and c) from the list clearly need ordered command execution. a mirroring app does not want writes to get committed out of order on the remote SCSI target, because that would mess up the transactional integrity of the data on that target 😭. basically, SCSI command streaming plus a guarantee of ordered command execution on the SCSI target is extremely valuable for a critical class of apps running over long-latency networks 💖.

this doc looks at the different protocol considerations you'd run into when designing storage solutions that use SCSI command ordering 🧠. it also analyzes and explains the design intent of [iSCSI] when it comes to command ordering 💅.

2. Definitions and Acronyms 📚

2.1. Definitions 🎀

  • I_T nexus: [SAM2] defines the I_T nexus as a relationship between a SCSI initiator port and a SCSI target port. [iSCSI] defines an iSCSI session as the iSCSI representation of an I_T nexus. in the iSCSI context, the I_T nexus (aka the iSCSI session) is a relationship between an iSCSI initiator's end of the session (SCSI Initiator Port) and the iSCSI target's Portal Group (SCSI Target Port) 🦋.
  • PDU (Protocol Data Unit): an iSCSI initiator and iSCSI target talk to each other using iSCSI protocol messages. these messages are called "iSCSI protocol data units" (iSCSI PDUs) 💌.
  • SCSI device: a SCSI device is an entity that has one or more SCSI ports hooked up to a service delivery subsystem, and it supports SCSI application protocols 🔌. in the iSCSI context, the SCSI Device is the part inside an iSCSI Node that handles the SCSI functionality. the SCSI Device Name is defined to be the iSCSI Name of the node 🏷️.
  • Session: a group of logically related iSCSI connections that link an initiator with a target form a session (equivalent to a SCSI I-T nexus) 🌸. the number of iSCSI connections in a session can change over time. the multiple connections at the iSCSI level are completely hidden from the SCSI layer, since each SCSI port in an I_T nexus only sees one peer SCSI port across all of a session's connections 🫶.

2.2. Acronyms 🍓

   Acronym                      Definition
   --------------------------------------------------------------
   ACA                          Auto Contingent Allegiance
   ASC                          Additional Sense Code
   ASCQ                         Additional Sense Code Qualifier
   CRN                          Command Reference Number
   IETF                         Internet Engineering Task Force
   ISID                         Initiator Session Identifier
   ITT                          Initiator Task Tag
   LU                           Logical Unit
   LUN                          Logical Unit Number
   NIC                          Network Interface Card
   PDU                          Protocol Data Unit
   TMF                          Task Management Function
   TSIH                         Target Session Identifying Handle
   SAN                          Storage Area Network
   SCSI                         Small Computer Systems Interface
   TCP                          Transmission Control Protocol
   UA                           Unit Attention
   WG                           Working Group

3. Overview of the iSCSI Protocol 🌈

3.1. Protocol Mapping Description 🧚‍♀️

okay so the iSCSI protocol is basically a mapping of the SCSI remote procedure invocation model (see [SAM2]) over the TCP protocol 🔥.

SCSI's idea of a task maps straight onto an iSCSI task 💜. every iSCSI task is uniquely identified within that I_T nexus by a 32-bit unique identifier called the Initiator Task Tag (ITT) 🏷️. the ITT is both an iSCSI identifier for the task and a classic SCSI task tag, so it does double duty ✨👑.

SCSI commands going from the initiator to the target ride along in iSCSI requests called SCSI Command PDUs 🚀. the SCSI status coming back to the initiator is carried in iSCSI responses called SCSI Response PDUs 📬. SCSI Data-Out traffic from the initiator to the target goes in SCSI Data-Out PDUs, and SCSI Data-in going back to the initiator goes in SCSI Data-in PDUs 💅🫶.

3.2. The I_T Nexus Model ✨

okay so bestie, in iSCSI the SCSI I_T nexus maps straight onto the iSCSI session 🧚 A session is an iSCSI protocol abstraction that can span one or more TCP connections, and together they give you one logical flow of two-way communication on the I_T nexus. That flow can use as many as 2^16 TCP connections 🦋 All of that connection juggling stays inside the iSCSI layer, so the SCSI layer just sees a single I_T nexus, even when the session has multiple connections. 💅

A session between two iSCSI nodes is identified by the session identifier (SSID), and each connection inside that session gets its own connection identifier (CID) 🎀 The SSID has two parts, the Initiator Session Identifier (ISID) and the Target Session Identifying Handler (TSIH), and each one identifies a different end of the same session. Iconic. 👑

There are four key functional facets that let iSCSI present this single logical flow to the SCSI layer, even when one session is spread over several connections. Let's go through them 💜

  • a) Ordered command delivery: A sequence of SCSI commands that gets striped across all the connections in the session is "reordered" by the target iSCSI layer into an identical sequence, based on a Command Sequence Number (CmdSN) that's unique across the session. The point is to get bandwidth aggregation from multiple TCP connections while still making it look to the target SCSI layer like every command travelled in one flow. ✨
  • b) Connection allegiance: All the PDU exchanges for a SCSI Command, up to and including the SCSI Response PDU for that Command, are required to flow on the same iSCSI connection at any given time. This is also meant to hide the multi-connection nature of a session, because the SCSI layer on either side never sees the PDU contents out of order. Like, status cannot bypass read data for an initiator. Not on my watch 😭
  • c) Task set management function handling: [iSCSI] specifies an ordered sequence of steps for the iSCSI layer on the SCSI target when it handles the two SCSI task management functions (TMFs) that manage SCSI task sets. Those two are ABORT TASK SET, which aborts all active tasks in a session, and CLEAR TASK SET, which clears the tasks in the task set. The goal of those steps is to guarantee that the initiator gets the SCSI Response PDUs of all unaffected tasks before the TMF Response itself arrives, no matter how many connections the iSCSI session has. Again, this keeps that single-flow vibe for the SCSI layer. 🌸
  • d) Immediate task management function handling: Even when a TMF request is marked as "immediate" (i.e. it only has a position in the command stream, but doesn't consume a CmdSN), [iSCSI] defines semantics that require the target iSCSI layer to make sure the TMF request is executed as if the commands and the TMF request were all flowing on a single logical channel. This makes sure the TMF request acts on the tasks it was actually meant to manage. Literally so important 🔥

The next sections dig into the "Ordered command delivery" facet in more detail, since command ordering is the whole focus of this document 🍓

3.3. Ordered Command Delivery

3.3.1. Questions

okay so a couple of important questions about iSCSI command ordering came up early on when the protocol was being designed 🧠 They were:

  • a) What command ordering behavior should be required of iSCSI implementations when there are transport errors, like errors that corrupt the data in a way the TCP checksum doesn't catch (e.g., two offsetting bit flips in the same bit position), but that the iSCSI CRC digest does catch? 🙈
  • b) Should [iSCSI] require both initiators and targets to use ordered command delivery? 🤔

These answers are critical to understanding the ordering behavior iSCSI requires, so the next sub-sections break them down properly 💖

3.3.2. The Session Guarantee

The final call on question a) in section 3.3.1 was reflected in [RFC3347], which says "iSCSI MUST specify strictly ordered delivery of SCSI commands over an iSCSI session between an initiator/target pair, even in the presence of transport errors." 🫶 Put another way, an iSCSI digest failure or an iSCSI connection termination must not cause the iSCSI layer on a target to execute commands in an order that's different from the one the initiator intended (as indicated by the CmdSN order). This was a super helpful design choice, because it means storage systems and solutions can always assume command ordering is a service characteristic of an iSCSI substrate. Main character energy for sure 👑

Heads up: by saying an iSCSI session always guarantees command ordering, [iSCSI] was basically hinting that the main reason for the multi-connection session abstraction was to allow ordered bandwidth aggregation for an I_T nexus. If that cross-connection ordering that [iSCSI] mandates turns out to be expensive in a particular deployment model, a serious consideration should be given to deploying multiple single-connection sessions instead. 🚨

3.3.3. Ordering Onus

The final resolution of b) in section 3.3.1 by the iSCSI protocol designers was in favor of not always requiring initiators to use command ordering 💅 This is reflected in dropping the mandatory ACA usage requirement on initiators, and in allowing an ABORT TASK TMF to plug a command hole and so on. These are conscious choices an initiator makes in favor of not using ordered command delivery. The net result, if you read [iSCSI] carefully, is that the onus of ensuring ordered command delivery is always on the iSCSI targets, while initiators may or may not use command ordering. 🦋

iSCSI targets are the servers in the client-server model, so they don't really try to figure out whether a client (initiator) plans to take advantage of the command ordering service. They just always provide the guaranteed delivery service 🌈 The reasoning is that there are inherent SCSI and application-level dependencies, which are beyond the scope of [iSCSI], that you'd need to mandate or even detect in order to build a command ordered solution. So [iSCSI] doesn't try 🙈

3.3.4. Design Intent

okay, to sum up the design intent of [iSCSI] 🎀

The service delivery subsystem (see [SAM2]) abstraction provided by an iSCSI session is guaranteed to have the intrinsic property of ordered delivery of commands to the target SCSI layer under all conditions. So the guarantee of ordered command delivery covers the entire I_T nexus, spanning all the LUs that the nexus is authorized to access. Whether this property gets used is up to the initiator's discretion. ✨

4. The Command Ordering Scenario 🧚

A storage systems designer working with SCSI and iSCSI has to consider the following protocol features in the SCSI and iSCSI layers. Each one has a role to play in realizing the command ordering goal 💜

4.1. SCSI Layer

The SCSI application layer has several tools for enforcing ordering 🔥

4.1.1. Command Reference Number (CRN)

CRN is an ordered sequence number which, when enabled for a device server, increments by one for each I_T_L nexus (see [SAM2]). The one notable drawback with CRN is that there's no SCSI-generic way (such as through mode pages) to enable or disable the CRN feature 🙃 [SAM2] also leaves the usage semantics of CRN for SCSI transport protocols, like iSCSI, for those protocols to specify. [iSCSI] chose not to support the CRN feature, for various reasons. 💀

4.1.2. Task Attributes

[SAM2] defines four task attributes: SIMPLE, ORDERED, HEAD OF QUEUE, and ACA 🌸 Each task to an LU may be assigned an attribute, and [SAM2] defines the ordering constraints that each attribute conveys to the device server servicing the task. The big takeaway is that judicious use of ORDERED and SIMPLE attributes on a stream of pipelined commands can convey the precise execution schema the initiator wants, as long as the commands are received in the same order on the target 🦋

4.1.3. Auto Contingent Allegiance (ACA)

ACA is an LU-level condition that gets triggered when a command (with the NACA bit set to 1) completes with CHECK CONDITION 😭 Once ACA is triggered, it blocks all commands other than those carrying the ACA attribute from executing until the CLEAR ACA task management function is executed, and it also blocks all the other tasks already in the task set. See [SAM2] for the detailed semantics of ACA. Since ACA is closely tied to the notion of a task set, you'd ideally select the scope of the task set (by setting the TST bit to 1 in the control mode page of the LU) to be per-initiator. That way, command failures in one I_T_L nexus won't impact other I_T_L nexuses through ACA 💅

4.1.4. UA Interlock

When UA interlock is enabled, the logical unit doesn't clear any standard Unit Attention condition reported with autosense. On top of that, it establishes a Unit Attention condition when a task is terminated with one of BUSY, TASK SET FULL, or RESERVATION CONFLICT statuses 🎀 This so-called "interlocked UA" only gets cleared when the device server executes an explicit REQUEST SENSE ([SPC3]) command from the same initiator. Functionally, the scope of UA interlock is a little different from ACA's, because it enforces ordering behavior for completion statuses other than CHECK CONDITION. Otherwise it has basically the same design intent as ACA. ACA is a bit more sophisticated, though, because it lets special "cleanup" tasks (ones with the ACA attribute) execute while ACA is active 🌈 One of the main reasons UA interlock came into being was that SCSI designers wanted a command ordering feature without the side effects of using the TST bit in the control mode page 🫶

4.2. iSCSI Layer

As noted in section 3.2 and section 3.3, the iSCSI protocol enforces and guarantees ordered command delivery per iSCSI session using the CmdSN, and this is an attribute of the SCSI transport layer 🧚 Any command ordering solution that wants ordering from the initiator SCSI layer to the target SCSI layer is only practically valuable when the SCSI transport layer guarantees that ordering. In other words, SCSI application layer features like ACA are built on the premise of an ordered SCSI transport. So iSCSI's command ordering is the last piece of the puzzle for solutions that rely on ordered command execution, because it provides the crucial guarantee that all the commands handed to the initiator iSCSI layer will be transported and handed to the target SCSI layer in the same order 👑

5. Connection Failure Considerations ✨

okay so [iSCSI] says that when an iSCSI connection fails, the active tasks on that connection have to be terminated if they aren't recovered within a certain negotiated time limit 💔. when an iSCSI target terminates some of those tasks because of connection drama, there's a risk that the SCSI layer just moves on to the next waiting tasks and runs them out of order, and the initiator's SCSI layer has no idea it happened 😭. to stop that, [iSCSI] also says:

a) the tasks terminated because of the connection failure must be internally terminated by the iSCSI target "as if" due to a CHECK CONDITION. ngl this status is never sent back to the initiator, but the "as if" still matters a lot 🫶. if the initiator is using ACA as its command ordering mechanism, a SCSI-level ACA gets triggered by this mandatory CHECK CONDITION, so the danger from earlier gets handled 🔥.

b) after the tasks are terminated because of the connection failure, the iSCSI target must report a Unit Attention condition on the next command processed on any connection, for each affected I_T_L nexus of that session. this is required because if the initiator uses UA interlock as its command ordering mechanism, a SCSI-level UA triggers a UA-interlock. so again, the danger is handled 💅. iSCSI targets must report this UA with the status of CHECK CONDITION and the ASC/ASCQ value of 47h/7Fh ("SOME COMMANDS CLEARED BY ISCSI PROTOCOL EVENT") 🌸.

6. Command Ordering System Considerations 💖

in general, command ordering gets enforced automatically if targets and initiators follow the iSCSI specification 🎀. still, there are some extra implementation considerations that iSCSI initiators and targets should totally take note of 👑.

a) even when all of the iSCSI and SCSI command ordering considerations earlier in this document have been applied, it's beneficial for iSCSI initiators to proactively avoid scenarios that would otherwise lead to out-of-order command execution 🧚. the reason is that SCSI command ordering features like UA interlock are likely to cost more performance when they get triggered 😩. [iSCSI] gives enough guidance on how to implement this proactive detection of PDU ordering errors 🦋.

b) the whole idea of command streaming assumes the target supports command queueing 💜. an iSCSI target that wants to support command ordering solutions should make sure the SCSI layer on the target supports command queuing 🌈. the remote backup (tape vaulting) applications that iSCSI enables make a super compelling case that tape devices should give very serious consideration to supporting command queuing, at least when they're used together with iSCSI 📼✨.

c) an iSCSI target that wants to support high-performance command ordering solutions, the kind that involve specifying a description of execution schema, should make sure the SCSI layer on the target actually supports the ORDERED and SIMPLE task attributes 🍓.

d) there's some consideration of expanding the scope of UA interlock to cover CHECK CONDITION status, which would make it the only required command ordering functionality for implementations building command ordering solutions 🤔. until T10 resolves this, the currently defined semantics of UA interlock and ACA mean that iSCSI targets wanting to support command ordering solutions should implement both features 💅🔥.

7. Reservation Considerations 🔐

[iSCSI] describes a "principle of conservative reuse" that encourages iSCSI initiators to reuse the same ISIDs (see section 3.2) across various SCSI target ports, so they present the same SCSI initiator port name to those target ports 🫶. this is honestly a crucial implementation consideration that must be complied with 😭💖. [SPC3] requires SCSI targets to associate persistent reservations and the related registrations with SCSI initiator port names whenever the SCSI transport protocol requires them 👑. since [iSCSI] requires mandatory SCSI initiator port names based on ISIDs, iSCSI targets are required to work off the SCSI initiator port names, and therefore indirectly off the ISIDs, when enforcing persistent reservations ✨.

this has the following implications for implementations:

a) if a persistent reservation/registration is meant to be used across multiple SCSI ports of a SCSI device, the initiator iSCSI implementation must use the same ISID across the associated iSCSI sessions connecting to different iSCSI target portal groups of the SCSI device 🧚‍♀️.

b) if a persistent reservation/registration is meant to be used across the power loss of a SCSI target, the initiator iSCSI implementation must use the same ISIDs as before when re-establishing the associated iSCSI sessions after a subsequent reboot, so it can rely on the persist through power loss capability 🌸⚡.

8. Security Considerations 🛡️

for security considerations when using the iSCSI protocol, go check the Security Considerations section in [iSCSI] 🙈. this document doesn't introduce any additional security considerations beyond the ones already discussed in [iSCSI] 💜.

9. References 📚

9.1. Normative References

[iSCSI]   Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M. and
          E. Zeidner, "Internet Small Computer Systems Inferface
          (iSCSI)", RFC 3720, May 2004.

[SAM2]    ANSI INCITS.366:2003 SCSI Architecture Model - 2 (SAM-2).

9.2. Informative References

[RFC793]  Postel, J., "Transmission Control Protocol", STD 7, RFC
          793, September 1981.

[RFC2119] Bradner, S., "Key Words for use in RFCs to Indicate
          Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC3347] Krueger, M. and R. Haagens, "iSCSI Requirements and Design
          Considerations", RFC 3347, July 2002.

[SPC3]    INCITS T10/1416-D, SCSI Primary Commands-3 (SPC-3).

10. Acknowledgments 🫶

we're so grateful to the IPS working group, whose work defined the iSCSI protocol 🎀. thanks also to David Black (EMC) for encouraging the publication of this document 💖. special thanks to Randy Haagens (HP) for his insights on command ordering 🔥. thanks are also due to Elizabeth Rodriguez for carefully reviewing this document 🌷.

11. Authors' Addresses 💌

Mallikarjun Chadalapaka
Hewlett-Packard Company
8000 Foothills Blvd.
Roseville, CA 95747-5668, USA

Phone: +1.916.785.5621
EMail: cbm@rose.hp.com


Rob Elliott
Hewlett-Packard Company
MC140801
PO Box 692000
Houston, TX 77269-2000  USA

Phone: +1.281.518.5037
EMail: elliott@hp.com

Copyright (C) The Internet Society (2004). this document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors keep all their rights 📜.

this document and the information in it are provided on an "AS IS" basis, and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT USING THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS, OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE 💀.

Intellectual Property

the IETF takes no position on the validity or scope of any Intellectual Property Rights or other rights that might be claimed to apply to implementing or using the technology described in this document, or on whether any license under such rights might or might not be available. it also doesn't claim it has made any independent effort to identify such rights 🧐. information on the procedures for rights in RFC documents can be found in BCP 78 and BCP 79 📖.

copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses to be made available, or the result of an attempt to obtain a general license or permission for implementers or users of this specification to use such proprietary rights can be obtained from the IETF online IPR repository at http://www.ietf.org/ipr 🔗.

the IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology required to implement this standard. please send that information to the IETF at ietf-ipr@ietf.org 💌.

Acknowledgement 🌈

funding for the RFC Editor function is currently provided by the Internet Society 💜.

📦 where this page comes from