RFC 5047
DA: Datamover Architecture for the Internet Small Computer System Interface (iSCSI)
RFC 5047 💖 DA: Datamover Architecture for the Internet Small Computer System Interface (iSCSI) ✨
- Series: Network Working Group 🌸
- Category: Informational 📚
- Date: October 2007 📅
- Authors: M. Chadalapaka (HP), J. Hufferd (Brocade Inc.), J. Satran (IBM), H. Shah (Broadcom Corporation) 👯
- Obsoletes / Updates: Nothing listed in the header, so it doesn't obsolete or update any other RFC 🫶
Status of This Memo 📜
Okay so this memo is giving "for information only" 💅 It provides info for the Internet community and does not specify an Internet standard of any kind. Distribution is unlimited, so share it with everyone 🌈
Abstract 🦋
The Internet Small Computer System Interface (iSCSI) is a SCSI transport protocol that maps the SCSI family of application protocols onto TCP/IP. Main character energy 👑 Datamover Architecture for iSCSI (DA) defines an abstract model where moving data between iSCSI end nodes is logically separated from the rest of the iSCSI protocol. The point is to let iSCSI adapt to new IP transports without drama 🧚
DA only defines the architectural functions needed by the class of Datamover protocols. It does not define any specific Datamover protocol 🙈 Each Datamover protocol gets its own separate document. It provides a reliable transport for all iSCSI PDUs, but it actually moves the data for certain iSCSI PDUs without involving the remote iSCSI layer itself. Iconic ✨
This document starts by introducing a few new abstractions and then defines a layered architecture for iSCSI and Datamover protocols. After that, it models the interactions inside an iSCSI end node between the iSCSI layer and the Datamover layer, which happen so remote data movement can occur transparently within an IP fabric 💜 The goal is to help map iSCSI onto generic Remote Direct Memory Access (RDMA)-capable IP fabrics in the future, comprising TCP, the Stream Control Transmission Protocol (SCTP), and possibly other underlying network transport layers such as InfiniBand 🔥
Table of Contents 🗂️
1. Motivation ......................................................4
1.1. Intent .....................................................4
1.2. Interpretation of Requirements .............................5
2. Definitions and Acronyms ........................................5
2.1. Definitions ................................................5
2.2. Acronyms ...................................................6
3. Architectural Layering of iSCSI and Datamover Layers ............7
4. Design Overview .................................................9
5. Architectural Concepts .........................................10
5.1. iSCSI PDU Types ...........................................10
5.1.1. iSCSI Data-Type PDUs ...............................10
5.1.2. iSCSI Control-Type PDUs ............................11
5.2. Data_Descriptor ...........................................11
5.3. Connection_Handle .........................................11
5.4. Operational Primitive .....................................12
5.5. Transport Connection ......................................13
6. Datamover Layer and Datamover Protocol .........................13
7. Functional Overview ............................................14
7.1. Startup ...................................................14
7.2. Full Feature Phase ........................................15
7.3. Wrap-up ...................................................15
8. Operational Primitives Provided by the Datamover Layer .........16
8.1. Send_Control ..............................................16
8.2. Put_Data ..................................................17
8.3. Get_Data ..................................................17
8.4. Allocate_Connection_Resources .............................18
8.5. Deallocate_Connection_Resources ...........................19
8.6. Enable_Datamover ..........................................19
8.7. Connection_Terminate ......................................20
8.8. Notice_Key_Values .........................................20
8.9. Deallocate_Task_Resources .................................20
9. Operational Primitives Provided by the iSCSI Layer .............21
9.1. Control_Notify ............................................21
9.2. Connection_Terminate_Notify ...............................22
9.3. Data_Completion_Notify ....................................22
9.4. Data_ACK_Notify ...........................................23
10. Datamover Interface (DI) ......................................23
10.1. Overview .................................................23
10.2. Interactions for Handling Asynchronous Notifications .....24
10.2.1. Connection Termination ............................24
10.2.2. Data Transfer Completion ..........................24
10.2.3. Data Acknowledgement ..............................25
10.3. Interactions for Sending an iSCSI PDU ....................25
10.3.1. SCSI Command ......................................26
10.3.2. SCSI Response .....................................26
10.3.3. Task Management Function Request ..................26
10.3.4. Task Management Function Response .................27
10.3.5. SCSI Data-Out and SCSI Data-In ....................27
10.3.6. Ready To Transfer (R2T) ...........................28
10.3.7. Asynchronous Message ..............................28
10.3.8. Text Request ......................................28
10.3.9. Text Response .....................................28
10.3.10. Login Request ....................................29
10.3.11. Login Response ...................................29
10.3.12. Logout Command ...................................29
10.3.13. Logout Response ..................................30
10.3.14. SNACK Request ....................................30
10.3.15. Reject ...........................................30
10.3.16. NOP-Out ..........................................30
10.3.17. NOP-In ...........................................30
10.4. Interactions for Receiving an iSCSI PDU ..................31
10.4.1. General Control-Type PDU Notification .............31
10.4.2. SCSI Data Transfer PDUs ...........................31
10.4.3. Login Request .....................................32
10.4.4. Login Response ....................................32
11. Security Considerations .......................................33
11.1. Architectural Considerations .............................33
11.2. Wire Protocol Considerations .............................33
12. References ....................................................34
12.1. Normative References .....................................34
12.2. Informative References ...................................34
Appendix A. Design Considerations and Examples ....................35
A.1. Design Considerations for a Datamover Protocol ............35
A.2. Examples of Datamover Interactions ........................35
Acknowledgements ..................................................44Table of Figures 🎀
Figure 1. Datamover Architecture Diagram, with the RDMAP Example ...8
Figure 2. A Successful iSCSI Login on Initiator ...................37
Figure 3. A Successful iSCSI Login on Target ......................37
Figure 4. A Failed iSCSI Login on Initiator .......................38
Figure 5. A Failed iSCSI Login on Target ..........................38
Figure 6. iSCSI Does Not Enable the Datamover .....................39
Figure 7. A Normal iSCSI Connection Termination ...................40
Figure 8. An Abnormal iSCSI Connection Termination ................40
Figure 9. A SCSI Write Data Transfer ..............................41
Figure 10. A SCSI Read Data Transfer ..............................42
Figure 11. A SCSI Read Data Acknowledgement .......................43
Figure 12. Task Resource Cleanup on Abort .........................441. Motivation ✨
1.1. Intent 🍓
Okay so there are relatively new standard protocols that enable Remote Direct Memory Access (RDMA) and Remote Direct Data Placement (RDDP) technologies to work over IP fabrics 🌐 The big win is that they let one end node place data straight into the final intended buffer on the remote end node. That skips the receive-path data copy that would otherwise move the data to its final location, and honestly that's so iconic 💅 Skipping the copy means less unnecessary memory bandwidth gets eaten up, the reassembly buffer size requirements drop substantially, and CPU cycles that would've gone into copying stay free 🧠✨
The iSCSI specification [RFC3720] defines a very detailed data transfer model. It uses SCSI Data-In PDUs, SCSI Data-Out PDUs, and R2T PDUs, plus the SCSI Command and SCSI Response PDUs that create and conclude the task context for the data transfer 📦 In the traditional iSCSI model, the iSCSI protocol layer runs the show: it paces the data transfer and does the transfer itself 😩 An alternative would be for iSCSI to hand off a large part of that data transfer role to a separate protocol layer that's built only to move data. That layer might be helped along by a data movement and placement technology such as RDMA 🚀
If iSCSI operated in these RDMA environments, it would be shielded from the low-level data transfer mechanics, but it would only be privy to the conclusion of the requested data transfer 🙈 So compared to today's iSCSI end nodes, there's an effective "off-loading" of the work the iSCSI protocol layer is expected to do. Lowkey genius 🧚 For RDMA environments, it's highly desirable to have a standard architecture that separates the data movement part of the iSCSI protocol definition from the rest of iSCSI functionality. This architecture precisely defines what a Datamover layer is and also describes the model of interactions between the iSCSI layer and the Datamover layer (Section 6) 💜
To meet that need, this document presents a Datamover Architecture for iSCSI (DA). It summarizes a reasonable model for interactions between the iSCSI layer and the Datamover layer for each of the iSCSI PDUs defined in [RFC3720] 🫶 DA is motivated by the arrival of RDMA over TCP/IP technology, but the architecture is not dependent on RDMA in its design. It's meant to be a generic architectural framework that allows different types of Datamovers based on different types of RDMA and transport protocols 🌈 Adopting this model will help iSCSI proliferate into more environments, which is giving big "expansion era" energy 🔥
1.2. Interpretation of Requirements 💅
Okay so this part is basically the vibe check for the whole architecture 🫶 DA builds an abstract functional interface model between iSCSI and the Datamover protocol layers, using some architectural abstractions. It's giving intentional design, and there are two big reasons for it:
- a) Give Datamover protocol designers guidance on where the line between iSCSI and the Datamover protocols sits. This is super critical 🎀 because a big chunk of the [RFC3720] protocol definition stays unchanged under DA, and iSCSI stuff like tasks and ITTs gets reused by the Datamover protocol 🧚
- b) Help existing iSCSI implementations adapt to DA quickly, mostly by turning those architectural abstractions into implementation things like functions, APIs, and modules 🍓
BUT here's the tea 🍵: DA does not try to impose implementation specifics. When a DA concept (like Operational Primitive) is described as mandatory ("MUST") or recommended ("SHOULD") for a layer (iSCSI or Datamover) in this document, the intent is that an implementation respectively MUST or SHOULD produce the same protocol action as the model describes. So using [RFC2119] terms here doesn't mean names, modules, or API arguments have to match. Only functional compliance is sought ✨
2. Definitions and Acronyms ✨
2.1. Definitions
I/O Buffer - a buffer used in a SCSI Read or Write operation so that SCSI data can be sent from or received by the buffer.
Datamover protocol - a Datamover protocol is a data transfer wire protocol for iSCSI that meets the requirements stated in Section 6 🔥
Datamover layer - a Datamover layer is a protocol layer within an end node that implements the Datamover protocol.
Datamover-assisted - an iSCSI connection is said to be "Datamover-assisted" when a Datamover layer is enabled for moving control and data information on that iSCSI connection 💜
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 🫶
2.2. Acronyms
Here's the whole acronym lineup, no cap 🌸
Acronym Definition
-------------------------------------------------------------
DA Datamover Architecture for iSCSI
DDP Direct Data Placement Protocol
DI Datamover Interface
IANA Internet Assigned Numbers Authority
IETF Internet Engineering Task Force
I/O Input - Output
IP Internet Protocol
iSCSI Internet SCSI
iSER iSCSI Extensions for RDMA
ITT Initiator Task Tag
LO Leading Only
MPA Marker PDU Aligned Framing for TCP
PDU Protocol Data Unit
RDDP Remote Direct Data Placement
RDMA Remote Direct Memory Access
R2T Ready To Transfer
R2TSN Ready To Transfer Sequence Number
RDMA Remote Direct Memory Access
RDMAP Remote Direct Memory Access Protocol
RFC Request For Comments
SAM SCSI Architecture Model
SCSI Small Computer Systems Interface
SN Sequence Number
SNACK Selective Negative Acknowledgment - also
Sequence Number Acknowledgement for Data
TCP Transmission Control Protocol
TTT Target Transfer Tag3. Architectural Layering of iSCSI and Datamover Layers ✨
Figure 1 shows an example of how the iSCSI and Datamover layers are layered, together with a TCP/IP implementation of the RDMAP/DDP ([DDP]) layers in an iSCSI end node. Heads up: the RDMAP/DDP/MPA and TCP layers are only shown as an example. In reality, DA is completely oblivious to the protocol layers below the Datamover layer 🙈 The RDMAP/DDP/MPA stack gives you a generic transport service with direct data placement, and you don't need to tailor that stack to the specific ULP to get those benefits 💅
Initiator stack Target stack
+----------------+ SCSI application +----------------+
| SCSI Layer | protocols | SCSI Layer |
+----------------+ +----------------+
^ ^
| |
v v
+----------------+ iSCSI protocol +----------------+
| iSCSI Layer | (excluding data | iSCSI Layer |
+----------------+ movement) +----------------+
^ ^
-- ---+-- ---- DI (Datamover Interface)--- ----+--- ----
v v
+----------------+ a Datamover +----------------+
| Datamover Layer| protocol | Datamover Layer|
+----------------+ +----------------+
^ ^
+-------+----------+ +---------+-----------+
| v | | v |
|+---------------+ | | +-----------------+ |
|| RDMAP/DDP/MPA | | RDMAP/DDP/MPA | | RDMAP/DDP/MPA | |
|| Layers | | protocols | | Layers | |
|+---------------+ | | +-----------------+ |
| ^ | | ^ |
| | network | | | network |
| | transport| | | transport |
| v | | v |
|+---------------+ | | +----------------+ |
|| TCP Layer | | TCP protocol | | TCP Layer | |
|+---------------+ | | +----------------+ |
| ^ | | ^ |
+-------+----------+ +---------+-----------+
+------------------------------------------+
Figure 1. Datamover Architecture Diagram,
with the RDMAP ExampleThe scope of this document is limited to:
- Defining the notion of a Datamover layer and a Datamover protocol (Section 6) 🎀
- Defining the functionality distribution between the iSCSI layer and the Datamover layer, along with the communication model between the two (Operational Primitives) 🦋
- Modeling the interactions between the blocks labeled as "iSCSI Layer" and "Datamover Layer" in Figure 1 -- i.e., defining the interface labeled "DI" in the figure -- for each defined iSCSI PDU, based on the Operational Primitives 👑
4. Design Overview 🧚
This document talks about and defines a model for how the iSCSI layer interacts with a "Datamover layer" (see Section 6) running inside an iSCSI end node. That end node is presumably talking to one or more iSCSI end nodes with similar layering. The model for handling different iSCSI operations is called the "Datamover Interface" (DI, Section 10), and the whole architecture is called the "Datamover Architecture for iSCSI" (DA). The architecture will probably have implications for Datamover wire protocols, since DA puts certain requirements and functionality expectations on the Datamover layer. However, this document doesn't define any new wire protocol for the Datamover layer, and it doesn't define any changes to the iSCSI wire protocol to use the Datamover layer either. The scope is strictly limited to the architectural framework and the minimum interactions that happen inside an iSCSI end node to use the Datamover layer 🫶
Here's how the DA design ideas break down:
- DA defines an abstract functional interface model of the iSCSI layer's interactions with a Datamover layer below. In other words, DA models the interactions between the logical "bottom" interface of iSCSI and the logical "top" interface of a Datamover 🍓
- DA guides the wire protocol for a Datamover layer by defining the iSCSI knowledge that the Datamover layer may use in its protocol definition. For example, this document completely limits the notion of "iSCSI session" to the iSCSI layer 💅
- DA is designed so the Datamover layer can be implemented either in hardware or in software 🔥
- DA is not a wire protocol spec. It's an architecture that also models the interactions between the iSCSI and Datamover layers running inside an iSCSI end node 💖
- DA is designed to model iSCSI-Datamover interactions independently of the specifics of either a particular iSCSI revision or an instantiation of a Datamover layer 🌈
- DA introduces and relies on a defined set of Operational Primitives (you could think of them as entry point definitions in implementation terms) provided by each layer to the other, to carry out the request-response interactions ✨
- DA is intended to allow Datamover protocol definitions with minimal changes to existing iSCSI implementations 🎀
- DA is designed to let the iSCSI layer completely rely on the Datamover layer for all data transport needs 👑
- DA models the architecturally required minimal interactions between an operational iSCSI layer and a Datamover layer to realize iSCSI-transparent data movement. A typical implementation may have several other interactions to bootstrap a Datamover layer (or an iSCSI layer) into operation, but those are outside the scope of this document 🦋
So in summary, DA is architected to support many different Datamover protocols running under the iSCSI layer. One such example of a Datamover protocol is iSER [iSER] 🧚
5. Architectural Concepts 🌸
5.1. iSCSI PDU Types 💜
This section defines the iSCSI PDU classification terminology, as defined and used in this document. Out of the set of legal iSCSI PDUs defined in [RFC3720], the iSCSI layer doesn't request a SCSI Data-Out PDU carrying solicited data for transmission across the Datamover Interface under this architecture (we'll see more in Section 5.1.1). So the SCSI Data-Out PDU carrying solicited data is left out of the iSCSI PDU classification introduced in this section (for SCSI Data-Out PDUs carrying unsolicited data, see Section 5.1.2). Every other legal iSCSI PDU that may be exchanged across the Datamover Interface falls into one of two classes:
- iSCSI data-type PDUs 🎀
- iSCSI control-type PDUs 🫶
5.1.1. iSCSI Data-Type PDUs 🎀
Okay so an iSCSI data-type PDU is an iSCSI PDU that makes data move between the peer iSCSI nodes on a Full Feature Phase iSCSI connection, and the remote iSCSI layer doesn't see the transfer happen 🙈 When the sender iSCSI layer asks for one to be sent, the data transfer happens without the remote iSCSI layer getting involved, which means the PDU itself never gets handed to the remote iSCSI layer as-is 💅 The set of iSCSI data-type PDUs is just these two:
- SCSI Data-In PDU
- R2T PDU
Now here's the part I'm obsessed with 😭 In an iSCSI end node that's split into an iSCSI layer and a Datamover layer (like this document describes), the solicitation for Data-Out, which is the R2T PDU, never gets delivered to the initiator iSCSI layer, because that's what makes it a data-type PDU. The data transfer happens through the mechanisms the Datamover layer knows about instead, like RDMA Read 🦋 So basically, a SCSI Data-Out PDU for solicited data is never requested for transmission across the Datamover Interface at the initiator. Iconic restraint, honestly ✨
5.1.2. iSCSI Control-Type PDUs 🌸
Any iSCSI PDU that isn't an iSCSI data-type PDU, and also isn't a solicited SCSI Data-Out PDU, counts as an iSCSI control-type PDU 🧚 Quick flag though: SCSI Data-Out PDUs for unsolicited Data are defined as iSCSI control-type PDUs, so don't mix those two up bestie 🙈
5.2. Data_Descriptor 🍓
A Data_Descriptor is an information element that describes an iSCSI/SCSI data buffer. The iSCSI layer gives it to its local Datamover layer, or the Datamover layer gives it to its local iSCSI layer, so both sides can identify the data tied to the requested or completed operation 💜
In implementation terms, a Data_Descriptor may be a scatter-gather list describing a local buffer. Its exact structure depends on the constraints the operating environment puts on the local iSCSI node 🌈
5.3. Connection_Handle 🔥
A Connection_Handle is an information element that identifies the particular iSCSI connection that an inbound or outbound iSCSI PDU is meant for. It's unique for a given pair of an iSCSI layer instance and a Datamover layer instance, and the Connection_Handle qualifier shows up in every invocation of any Operational Primitive for connection identification 👑
Here's the twist 🙈 the Connection_Handle is conceptually different from the Connection Identifier (CID) defined by the iSCSI specification. The CID is unique within an iSCSI session, but the Connection_Handle's uniqueness covers the entire iSCSI layer instance coupled with the Datamover layer instance, across possibly multiple iSCSI sessions 🦋
In implementation terms, a Connection_Handle could be an opaque identifier that the iSCSI layer and the Datamover layer swap at connection login time. You could also think of it as similar in scope of uniqueness to a socket identifier ✨ How the handle gets exchanged between the two layers is implementation-specific, so that's up to the builders 💅
5.4. Operational Primitive 💖
An Operational Primitive, in this document, is an abstract functional interface procedure. It either asks another layer to do a specific action on the requestor's behalf, or it tells the other layer about some event 🧚 The Datamover Interface between an iSCSI layer instance and a Datamover layer instance inside an iSCSI end node uses a set of Operational Primitives to define the functional interface between those two layers. Note that not every invocation of an Operational Primitive has to get a response from the layer it was requested of 🫶 This document describes the types of Operational Primitives that the iSCSI protocol layer, as defined in [RFC3720], implicitly needs and provides, and it also describes what those Primitives mean.
Heads up that ownership of buffers and data structures is likely to get passed back and forth between the iSCSI layer and its local Datamover layer when these Operational Primitives are invoked 🌸 How buffers get allocated and released is implementation-specific, so it's outside this document's scope.
Each Operational Primitive invocation needs a certain "information context" (for example, a Connection_Handle) to do the action being asked for 🎀 This document lists that required information context as "qualifiers" on each invocation, written in the style of function call arguments. That notation doesn't imply any specific implementation. The "qualifiers" of any Operational Primitive invocation in this document represent the mandatory information context that the invocation MUST consider when performing the action. The qualifiers are required, but how you actually realize them (passed in synchronously with the invocation, pulled from task context, pulled from shared memory, etc.) is totally up to the implementations 🌈
When an Operational Primitive implementation is described as mandatory ("MUST") or recommended ("SHOULD") for a layer (iSCSI or Datamover) in this document, the intent is that the implementation MUST or SHOULD, respectively, produce the same protocol action the model describes 👑
5.5. Transport Connection 🦋
"Transport Connection" is used in this document as a generic term for the end-to-end logical connection defined by the underlying reliable transport protocol. For this document, every instance of Transport Connection refers to a TCP connection 💜
6. Datamover Layer and Datamover Protocol ✨
This section introduces the ideas of a "Datamover layer" and a "Datamover protocol" as this document means them, and it defines the requirements on a Datamover protocol 🧚
A Datamover layer is the implementation component that makes a Datamover protocol's functionality happen in an iSCSI-capable end node, so it can talk to other iSCSI end nodes with similar capabilities 🫶 More specifically, a "Datamover layer" MUST provide the following functionality, and the "Datamover protocol" MUST consist of the wire protocol needed to realize it:
- guarantee that all the necessary data transfers take place when the local iSCSI layer requests transmitting a command (in order to complete a SCSI command, for an initiator), or sending/receiving an iSCSI data sequence (in order to complete part of a SCSI command for a target).
- transport an iSCSI control-type PDU as-is to the peer Datamover layer when requested to do so by the local iSCSI layer.
- provide notification and delivery to the iSCSI layer upon arrival of an iSCSI control-type PDU.
- provide an initiator-to-target data acknowledgement of SCSI read data back to the target iSCSI layer, when requested.
- provide an asynchronous notification upon completion of a requested data transfer operation that moved data without involving the iSCSI layer.
- place the SCSI data into the I/O buffers or pick up the SCSI data for transmission out of the data buffers that the iSCSI layer had requested to be used for a SCSI I/O.
- provide an error-free (i.e., must have at least the same level of assurance of data integrity as the CRC32C iSCSI data digest), reliable, in-order delivery transport mechanism over IP networks in performing the data transfer, and asynchronously notify the iSCSI layer upon iSCSI connection termination.
Note that this architecture expects each compliant Datamover protocol to define the precise means of satisfying the requirements in this section 💅
To meet the functional requirements listed here, certain Datamover protocols may need pre-posted buffers from the local iSCSI protocol layer, through mechanisms outside this document's scope 😭 In some implementations, if those buffers are missing, the connection could fail. Datamover protocols may also realize these functional requirements in ways that aren't explicitly listed in this document 🌸
7. Functional Overview 🌈
This section gives an overview of the functional interactions between the iSCSI layer and the Datamover layer, as this Architecture intends them 👑
7.1. Startup 🦋
The iSCSI Login Phase on an iSCSI connection happens as defined in [RFC3720] 💜 The Architecture assumes that at the end of the Login Phase, both the initiator and the target, if they've decided to, transition the connection to being Datamover-assisted. How an iSCSI initiator and an iSCSI target agree to make the connection Datamover-assisted is defined by the Datamover protocol 🎀 The only architectural requirement is that all iSCSI interactions in the iSCSI Full Feature Phase MUST be Datamover-assisted, subject to that prior agreement. That means the Datamover protocol sits in the iSCSI-to-iSCSI communication path below the iSCSI layer on either side, as shown in Figure 1. DA defines the Enable_Datamover Operational Primitive (Section 8.6) to bring about this transition to a Datamover-assisted connection ✨
The Architecture also assumes the Datamover layer may need a certain number of opaque local resources to make a connection Datamover-assisted 🧚 So DA defines the Allocate_Connection_Resources Operational Primitive (Section 8.4) to model this interaction. It's meant to be invoked on each side once the two sides have decided (as noted above) to make the connection Datamover-assisted. The expected sequence of Primitive invocations is shown in Figures 2 and 3 in Section 13.2 🫶 Figures 4, 5, and 6 show how the Primitives can be used to handle various legal login outcomes 🔥
7.2. Full Feature Phase 💖
All iSCSI peer communication in the Full Feature Phase goes through the Datamover layers if the iSCSI connection is Datamover-assisted 🌸 The Architecture assumes a Datamover layer may need a certain number of opaque local resources for each new iSCSI task. In normal execution, these task-level resources in the Datamover layer are assumed to be allocated transparently when each task starts and deallocated when each task concludes, as appropriate 🎀 But in exception scenarios, the ones that don't produce a SCSI Response for each task (like the ABORT TASK operation), the Architecture assumes the Datamover layer needs to be told about each individual task termination so it can manage its task-level resources 😭 DA defines the Deallocate_Task_Resources Operational Primitive (Section 8.9) to model this task-resource management. When specifying the ITT qualifier for Deallocate_Task_Resources, the Architecture further assumes that the Datamover layer tracks its opaque task-level local resources by the iSCSI ITT 🦋
DA also defines the Send_Control (Section 8.1), Put_Data (Section 8.2), Get_Data (Section 8.3), Data_Completion_Notify (Section 9.3), Data_ACK_Notify (Section 9.4), and Control_Notify (Section 9.1) Operational Primitives to model the various Full Feature Phase interactions 👑
Figures 9, 10, and 11 in Section 13.2 show some Full Feature Phase interactions: a SCSI Write task, a SCSI Read task, and a SCSI Read Data acknowledgement, respectively 💜 Figure 12 in Section 13.2 shows how an ABORT TASK operation can be modeled so the Datamover layer gets deterministic resource cleanup 🌈
7.3. Wrap-up
Okay so once an iSCSI connection becomes Datamover-assisted, it stays that way 💅 until the end of the Full Feature Phase, which is basically when the connection terminates 🦋. The Architecture assumes that when a connection gets normally logged out, the Datamover layer needs to be notified so it can free up its connection-level opaque resources (see Section 7.1) 🫶. That's why DA defines a Connection_Terminate Operational Primitive (Section 8.7) to model this moment 🎀.
The Architecture also assumes that sometimes a connection dies without the iSCSI layer being involved at all (for example, a TCP RST 💀). In that case the Datamover layer is capable of locally cleaning up its task-level and connection-level resources before it tells the iSCSI layer what happened ✨. DA defines the Connection_Terminate_Notify Operational Primitive (Section 9.2) for exactly this interaction 🌸.
Figures 7 and 8 in Section 13.2 show the iSCSI and Datamover layers chatting in both the normal and the unexpected termination scenarios 📸.
8. Operational Primitives Provided by the Datamover Layer
Even though the iSCSI specification itself doesn't have a notion of Operational Primitives 🤓, any iSCSI layer that implements the iSCSI specification functionally needs the following Operational Primitives from its Datamover layer 🧚. So any Datamover protocol that's compliant with this architecture MUST implement the Operational Primitives described in this section 👑. The iSCSI layer invokes these as appropriate 💖. Unless otherwise stated, all of them may be used on both the initiator side and the target side 🙌. In general programming terminology, this set is construed as "down calls" 📞.
Send_ControlPut_DataGet_DataAllocate_Connection_ResourcesDeallocate_Connection_ResourcesEnable_DatamoverConnection_TerminateNotice_Key_ValuesDeallocate_Task_Resources
8.1. Send_Control
Input qualifiers: Connection_Handle, iSCSI PDU-specific qualifiers
Return Results: Not specified.
Here the iSCSI layer asks its local Datamover layer to transmit an iSCSI control-type PDU to the peer iSCSI layer on the remote iSCSI node 📬. The Datamover layer does the requested operation and may add its own protocol headers while it's at it 🎁. The iSCSI layer MUST NOT invoke Send_Control on an iSCSI connection that isn't Datamover-assisted yet 🚫.
An initiator iSCSI layer requesting the transfer of a SCSI Command PDU, or a target iSCSI layer requesting the transfer of a SCSI response PDU, are both examples of invoking Send_Control 🙌. As Section 10.3.1 shows later on, the iSCSI PDU-specific qualifiers in this example are: BHS and AHS, DataDescriptorOut, DataDescriptorIn, ImmediateDataSize, and UnsolicitedDataSize 🧾.
8.2. Put_Data
Input qualifiers: Connection_Handle, contents of a SCSI Data-In PDU header, Data_Descriptor, Notify_Enable
Return Results: Not specified.
Here the iSCSI layer asks its local Datamover layer to transmit the data identified by the Data_Descriptor for the SCSI Data-In PDU to the peer iSCSI layer on the remote iSCSI node 🚚. The Datamover layer handles it with its own protocol means, totally transparent to the remote iSCSI layer 🥷. The iSCSI layer MUST NOT invoke Put_Data on an iSCSI connection that isn't Datamover-assisted yet 🚫.
The Notify_Enable qualifier tells the local Datamover layer whether to generate (or not generate) the eventual local completion notification to the iSCSI layer for this Put_Data invocation 🔔. For the detailed semantics, see Section 9.3 📖.
A Put_Data Primitive may only be invoked by an iSCSI layer on the target, to its local Datamover layer 🎯.
A target iSCSI layer requesting the transfer of an iSCSI read data sequence (also known as a read burst) is an example of invoking Put_Data 🍓.
8.3. Get_Data
Input qualifiers: Connection_Handle, contents of an R2T PDU, Data_Descriptor, Notify_Enable
Return Results: Not specified.
Here the iSCSI layer asks its local Datamover layer to retrieve certain data identified by the R2T PDU from the peer iSCSI layer on the remote iSCSI node, and drop it into the buffer identified by the Data_Descriptor 🪣. The Datamover layer does this with its own protocol means, completely transparent to the remote iSCSI layer 🫥. The iSCSI layer MUST NOT invoke Get_Data on an iSCSI connection that isn't Datamover-assisted yet 🚫.
The Notify_Enable qualifier asks the local Datamover layer to generate (or not generate) the eventual local completion notification to the iSCSI layer for this Get_Data invocation 🔔. For the detailed semantics, see Section 9.3 📖.
A Get_Data Primitive may only be invoked by an iSCSI layer on the target, to its local Datamover layer 🎯.
A target iSCSI layer requesting the transfer of an iSCSI write data sequence (also known as a write burst) is an example of invoking Get_Data 💫.
8.4. Allocate_Connection_Resources
Input qualifiers: Connection_Handle[, Resource_Descriptor ]
Return Results: Status.
By invoking this Operational Primitive, the iSCSI layer asks its local Datamover layer to do all the Datamover-specific resource allocations needed for the Full Feature Phase of an iSCSI connection 🗂️. The Connection_Handle says which connection the iSCSI layer wants resources for 🪪. This allocation is a step toward eventually turning the connection into a Datamover-assisted iSCSI connection 🦋. Note that the Datamover layer does not allocate any Datamover-specific task-level resources when this Primitive is invoked 🙅♀️.
The iSCSI layer can also, optionally, tell the Datamover layer about its implementation-specific resource requirements for the iSCSI connection by passing an input qualifier called Resource_Descriptor 🧩. The exact structure of a Resource_Descriptor is implementation-dependent, so it's structurally opaque to DA 🫣.
A return result of Status=success means the Allocate_Connection_Resources invocation for that Connection_Handle worked ✅. If an Allocate_Connection_Resources invocation is made for a Connection_Handle where an earlier invocation already succeeded, the return Status MUST be success, and the Datamover layer ignores the request 🔁. A return result of Status=failure means the invocation for that Connection_Handle failed ❌. There MUST NOT be more than one Allocate_Connection_Resources Primitive invocation outstanding for a given Connection_Handle at any time ☝️.
The iSCSI layer has to invoke Allocate_Connection_Resources before it invokes Enable_Datamover 🫡.
8.5. Deallocate_Connection_Resources
Input qualifiers: Connection_Handle
Return Results: Not specified.
By invoking this Operational Primitive, the iSCSI layer asks its local Datamover layer to deallocate all the Datamover-specific resources that might have been allocated earlier for the Transport Connection identified by the Connection_Handle 🧹. The iSCSI layer may invoke this when those Datamover-specific resources aren't needed anymore, like after a Login failure on the corresponding iSCSI connection 🙃.
8.6. Enable_Datamover
Input qualifiers: Connection_Handle, Transport_Connection_Descriptor [, Final_Login_Response_PDU]
Return Results: Not specified.
By invoking this Operational Primitive, the iSCSI layer asks its local Datamover layer to assist all further iSCSI exchanges on the iSCSI connection (in other words, to make the connection Datamover-assisted) identified by the Connection_Handle, which had its Datamover-specific resources allocated earlier 🚀. The iSCSI layer MUST NOT invoke Enable_Datamover for an iSCSI connection unless there's a corresponding prior resource allocation 🛑.
The Final_Login_Response_PDU input qualifier only applies to a target 🎯. It holds the final Login Response that wraps up the iSCSI Login Phase, and it has to be sent as a byte stream the way the initiator iSCSI layer expects 📦. When this qualifier is used, the target-Datamover layer MUST transmit this final Login Response before Datamover assistance is enabled for the Transport Connection 🔐.
The iSCSI layer tells the Datamover layer which specific Transport Connection goes with the Connection_Handle by specifying the Transport_Connection_Descriptor 🔗. The exact structure of this Descriptor is implementation-dependent 🤷♀️.
8.7. Connection_Terminate
Input qualifiers: Connection_Handle
Return Results: Not specified.
By invoking this Operational Primitive, the iSCSI layer asks its local Datamover layer to terminate the Transport Connection and deallocate all the connection and task resources associated with the Connection_Handle 🧨. When this invocation returns to the iSCSI layer, the iSCSI layer may assume full ownership of all the iSCSI-level resources (for example, I/O Buffers) associated with the connection 🤝. This Operational Primitive may only be invoked with a valid Connection_Handle, and the Transport Connection associated with that Connection_Handle must already be Datamover-assisted 💅.
8.8. Notice_Key_Values
Input qualifiers: Connection_Handle, Number of keys, a list of Key-Value pairs.
Return Results: Not specified.
By invoking this Operational Primitive, the iSCSI layer asks its local Datamover layer to take note of the negotiated values of the listed keys for the Transport Connection 📝. This Operational Primitive may only be invoked with a valid Connection_Handle, and the Key-Value pairs MUST be the current values that the iSCSI peers successfully agreed upon for the connection 🤙. The Datamover layer may use the values of the keys to aid the Datamover operation as it deems appropriate 🧠. Which keys get passed as input qualifiers, and the point(s) in time this Operational Primitive is invoked, are implementation-dependent 🌈.
8.9. Deallocate_Task_Resources 🧹
Input qualifiers: Connection_Handle, ITT
Return Results: Not specified.
okay so this one is the cleanup crew 🧹✨ When the iSCSI layer needs its local Datamover layer to free up every Datamover-specific resource that was set aside for a task (the one identified by the ITT qualifier), it calls this primitive. It's for exception processing, when one or more active tasks have to be killed off without a SCSI Response PDU wrapping them up. The rule is that this MUST be invoked for each active task terminated without a SCSI Response PDU 💅 And the other side of it is just as strict: this Primitive MUST NOT be invoked by the iSCSI layer when a SCSI Response PDU normally concludes a task 🙅♀️ Why? Because when a SCSI Response PDU closes out a task (even if the SCSI Status isn't a success), the Datamover layer is assumed to have already cleaned up all of that task's Datamover-specific resources on its own. Main character energy for the Datamover, honestly 👑 If you want the backstory on when these resources are assumed to get allocated, check out Section 7.2, because it covers the architectural assumptions about task-level Datamover resource management.
9. Operational Primitives Provided by the iSCSI Layer 🎀
Now we flip the script 🔄 The iSCSI specification itself doesn't really talk about Operational Primitives, but any iSCSI layer that implements it would still have to provide the following Operational Primitives up to its local Datamover layer. So the bottom line is that any iSCSI protocol implementation compliant with this architecture MUST implement the Operational Primitives described in this section 🚨 These are called by the Datamover layer whenever it's needed and only when the iSCSI connection is Datamover-assisted. Unless it says otherwise, all of these can be used on both the initiator side and the target side 🤝 In normal programming language, this whole set is basically "up calls" 📞
- Control_Notify
- Connection_Terminate_Notify
- Data_Completion_Notify
- Data_ACK_Notify
9.1. Control_Notify 📬
Input qualifiers: Connection_Handle, an iSCSI control-type PDU.
Return Results: Not specified.
This is how the Datamover layer tells its local iSCSI layer that an iSCSI control-type PDU has landed from the peer Datamover layer on the remote iSCSI node 💌 The iSCSI layer then handles that control-type PDU the way [RFC3720] defines.
A good example: a target iSCSI layer getting told that a SCSI command has arrived. That's a Control_Notify moment 🎯
Heads up, implementations are allowed to describe the "iSCSI control-type PDU" qualifier in this notification using a Data_Descriptor (Section 5.2), and it doesn't have to be one contiguous buffer. Flexible, bestie 🧩
9.2. Connection_Terminate_Notify 💔
Input qualifiers: Connection_Handle
Return Results: Not specified.
This one is the "the connection just died on me" alert 😭 The Datamover layer uses it to tell its local iSCSI layer about an unsolicited termination or failure of the iSCSI connection that goes with a given Connection_Handle. The iSCSI layer MUST consider the Connection_Handle to be invalid upon being so notified 🙈 Then the iSCSI layer handles the termination the way [RFC3720] describes. And here's the order of operations that really matters: the Datamover layer MUST deallocate the connection and task resources associated with the terminated connection before notifying the iSCSI layer of the termination via this Operational Primitive 🧹 Clean up first, then gossip 💅
A target iSCSI layer gets this when the Datamover layer notices the underlying Transport Connection has been torn down ungracefully. For example, this could be triggered by a TCP RESET when the Transport Connection is TCP 💥
9.3. Data_Completion_Notify 🎉
Input qualifiers: Connection_Handle, ITT, SN
Return Results: Not specified.
This primitive is the "your data is done!" ping 🥳 The Datamover layer uses it to tell its local iSCSI layer that it has finished either retrieving data or sending it, based on an earlier iSCSI data-type PDU, to or from the peer Datamover layer on the remote iSCSI node. The iSCSI layer handles the operation as [RFC3720] says.
Here's the tea ☕ The SN qualifier can be either the DataSN tied to a SCSI Data-In PDU or the R2TSN tied to an R2T PDU, depending on which SCSI operation it is. A TTT (see [RFC3720]) could have been used on targets instead of an SN, but the SN won, and for two good reasons. First, a) it's generic and works for initiators and targets, and for Data-In and Data-Out alike 🙌 Second, b) having both SN and TTT as qualifiers would make the Datamover layer work way harder, because it'd have to track extra state for every single completion notification 😩 The catch is that iSCSI target implementations will have to adapt and use the ITT-SN tuple to match solicited data to the right task, instead of the ITT-TTT tuple they'd normally use. Plan accordingly, targets 🎯
Now the rules. If Notify_Enable is set in either a Put_Data or a Get_Data invocation, the Datamover layer MUST invoke the Data_Completion_Notify Operational Primitive upon completing that requested data transfer ✅ And if Notify_Enable was cleared in either one, the Datamover layer MUST NOT invoke the Data_Completion_Notify Operational Primitive upon completing that requested data transfer 🙅♀️
Basically, a Data_Completion_Notify tells the iSCSI layer that a Put_Data or Get_Data just finished, respectively. As noted back in Sections 8.2 and 8.3, specific Datamover protocol definitions can restrict how Put_Data and Get_Data get used, and that automatically limits how Data_Completion_Notify gets used too 🧐
Classic example: a target iSCSI layer being told that a write data sequence has been retrieved 📦
9.4. Data_ACK_Notify 🫧
Input qualifiers: Connection_Handle, ITT, DataSN
Return Results: Not specified.
A target Datamover layer uses this one to tell its local iSCSI layer that a data acknowledgement it asked for has arrived from the peer Datamover layer on the remote (initiator) iSCSI node 📩 The iSCSI layer handles that acknowledgement notification the way [RFC3720] defines.
The only example given is a target iSCSI layer finding out that a data acknowledgement came back for a particular SCSI Read data PDU. That's the one and only Data_ACK_Notify moment in this doc, so it's iconic in its own way 🦋
10. Datamover Interface (DI) 🌸
10.1. Overview 👀
This section describes the back-and-forth between the iSCSI and Datamover layers when the iSCSI connection is Datamover-assisted. That way the iSCSI layer can do a few things:
- send iSCSI data-type PDUs and exchange iSCSI control-type PDUs, and
- handle asynchronous notifications, like a finished data sequence transfer or a dead connection.
This chapter leans on the idea of Operational Primitives (Section 5.4) to define DI, so you'll want that concept locked in before diving deeper ✨
10.2. Interactions for Handling Asynchronous Notifications 🔔
10.2.1. Connection Termination 💀
As mentioned in Section 9.2, the Datamover layer tells the iSCSI layer about a failed or terminated connection through the Connection_Terminate_Notify Operational Primitive. The iSCSI layer MUST consider the connection unusable upon the invocation of this Primitive and handle the connection termination as specified in [RFC3720] 🙅♀️ Dead is dead, no cap.
10.2.2. Data Transfer Completion 🏁
As mentioned in Section 9.3, the Datamover layer tells the iSCSI layer that a data transfer operation is done through the Data_Completion_Notify Operational Primitive. The iSCSI layer processes that transfer completion as [RFC3720] specifies 📦
10.2.2.1. Completion of a Requested SCSI Data Transfer 📬
To tell the iSCSI layer that a requested iSCSI data-type PDU transfer is complete, the Datamover layer uses the Data_Completion_Notify Operational Primitive with these input qualifiers:
- a) Connection_Handle.
- b) ITT: Initiator Task Tag semantics as defined in [RFC3720].
- c) SN: DataSN for a SCSI Data-in/Data-out PDU, and R2TSN for an iSCSI R2T PDU. The semantics for both types of sequence numbers are as defined in [RFC3720].
The reasoning behind choosing SN is explained in Section 9.3 🧠
Here's the key rule: every invocation of the Data_Completion_Notify Operational Primitive MUST be preceded by an invocation of the Put_Data or Get_Data Operational Primitive with the Notify_Enable qualifier set by the iSCSI layer at an earlier point in time 🌈 No surprise notifications, the Datamover always comes with a receipt 🧾
10.2.3. Data Acknowledgement 🫶
[RFC3720] lets iSCSI targets optionally ask the initiator for a data acknowledgement on one or more Data-In PDUs, by setting the A-bit on a Data-In PDU. The Data_ACK_Notify Operational Primitive, with the following input qualifiers, is how the target Datamover layer tells the local iSCSI layer that a previously requested read data acknowledgement has arrived. This primitive only applies to iSCSI targets, so initiators can skip this one 🎯
- a) Connection_Handle.
- b) ITT: Initiator Task Tag semantics as defined in [RFC3720].
- c) DataSN: of the next SCSI Data-In PDU, which immediately follows the SCSI Data-In PDU with the A-bit set to which this notification corresponds, with semantics as defined in [RFC3720].
Every invocation of the Data_ACK_Notify Operational Primitive MUST be preceded by an invocation of the Put_Data Operational Primitive by the iSCSI target layer with the A-bit set to 1 at an earlier point in time ⏰ So the ack always has a matching request behind it 💜
10.3. Interactions for Sending an iSCSI PDU 📤
This section covers the interactions for sending each of the iSCSI PDUs defined in [RFC3720]. Each of these interactions is assumed to be qualified by a Connection_Handle (see Section 5.3), so the Datamover layer knows which Transport Connection to route it to. The qualifying Connection_Handle isn't explicitly listed in the sections that follow 🗺️
Quick note: the input qualifiers listed here are the semantically required set the Datamover layer needs to consider when implementing the Primitive in each interaction (see Section 5.4 for more on that). Implementations are allowed to deduce those qualifiers in ways that are optimized for their own specifics. Two examples:
- For SCSI command (Section 10.3.1), deducing the ImmediateDataSize input qualifier from the DataSegmentLength field of the SCSI Command PDU.
- For SCSI Data-Out (Section 10.3.5.1), deducing the DataDescriptorOut input qualifier from the associated SCSI command invocation qualifiers (assuming such state is maintained) in conjunction with BHS fields of the SCSI Data-Out PDU.
10.3.1. SCSI Command ✨
Okay bestie, this is the Send_Control Operational Primitive doing its main job 💅. It's how you ask for a SCSI Command PDU to get sent out. Here are the input qualifiers it takes:
- a) BHS and AHS, if any, of the SCSI Command PDU as defined in [RFC3720].
- b)
DataDescriptorOut: defines the I/O Buffer meant for Data-Out for the entire command, in the case of a write or bidirectional command. - c)
DataDescriptorIn: defines the I/O Buffer meant for Data-In for the entire command, in the case of a read or bidirectional command. - d)
ImmediateDataSize: defines the number of octets of immediate unsolicited data for a write/bidirectional command. - e)
UnsolicitedDataSize: defines the number of octets of immediate and non-immediate unsolicited data for a write/bidirectional command.
10.3.2. SCSI Response 🎀
Same primitive, Send_Control, but now we're requesting the transmission of a SCSI Response PDU. It only needs two things:
- a) BHS of the SCSI Response PDU as defined in [RFC3720].
- b)
DataDescriptorStatus: defines the iSCSI buffer that contains the sense and response information for the command.
10.3.3. Task Management Function Request 🦋
Still Send_Control, this time asking for a Task Management Function Request PDU to be sent. Heads up, the two data descriptors below are only valid for one specific function:
- a) BHS of the Task Management Function Request PDU as defined in [RFC3720].
- b)
DataDescriptorOut: defines the I/O Buffer meant for Data-Out for the entire command, in the case of a write or bidirectional command. (Only valid ifFunction="TASK REASSIGN"- [RFC3720].) - c)
DataDescriptorIn: defines the I/O Buffer meant for Data-In for the entire command, in the case of a read or bidirectional command. (Only valid ifFunction="TASK REASSIGN"- [RFC3720].)
10.3.4. Task Management Function Response 🌸
Send_Control again, this time to request the transmission of a Task Management Function Response PDU. Just one qualifier here, and it's the BHS:
- a) BHS of the Task Management Function Response PDU as defined in [RFC3720].
10.3.5. SCSI Data-Out and SCSI Data-In 🌈
10.3.5.1. SCSI Data-Out 💜
This one's Send_Control too, and it's the initiator iSCSI layer asking for a SCSI Data-Out PDU carrying the non-immediate unsolicited data to be sent:
- a) BHS of the SCSI Data-Out PDU as defined in [RFC3720].
- b)
DataDescriptorOut: defines the I/O Buffer with the Data-Out to be carried in the iSCSI data segment of the PDU.
10.3.5.2. SCSI Data-In 🍓
Plot twist, for Data-In we switch to the Put_Data Operational Primitive. The target iSCSI layer uses it to request the transmission of the data carried by a SCSI Data-In PDU:
- a) BHS of the SCSI Data-In PDU as defined in [RFC3720].
- b)
DataDescriptorIn: defines the I/O Buffer with the Data-In being requested for transmission.
10.3.6. Ready To Transfer (R2T) 🧚
Now the target iSCSI layer uses the Get_Data Operational Primitive to request retrieval of the data described by the semantic content of an R2T PDU. Two inputs:
- a) BHS of the Ready To Transfer PDU as defined in [RFC3720].
- b)
DataDescriptorOut: defines the I/O Buffer for the Data-Out being requested for retrieval.
10.3.7. Asynchronous Message ⭐
Send_Control is back to request the transmission of an Asynchronous Message PDU. The descriptor here holds the sense and iSCSI Event info:
- a) BHS of the Asynchronous Message PDU as defined in [RFC3720].
- b)
DataDescriptorSense: defines an iSCSI buffer that contains the sense and iSCSI Event information.
10.3.8. Text Request 🫶
Yep, Send_Control, now for a Text Request PDU:
- a) BHS of the Text Request PDU as defined in [RFC3720].
- b)
DataDescriptorTextOut: defines the iSCSI Text Request buffer.
10.3.9. Text Response 🔥
Same deal, Send_Control, but for a Text Response PDU:
- a) BHS of the Text Response PDU as defined in [RFC3720].
- b)
DataDescriptorTextIn: defines the iSCSI Text Response buffer.
10.3.10. Login Request 🙈
Send_Control requests the transmission of a Login Request PDU:
- a) BHS of the Login Request PDU as defined in [RFC3720].
- b)
DataDescriptorLoginRequest: defines the iSCSI Login Request buffer.
Note that specific Datamover protocols may choose to disallow the standard DA Primitives from being used for the iSCSI Login Phase. When used with those Datamover protocols, trying to send a Login Request through a Send_Control invocation is clearly an error scenario, because the Login Request PDU is being sent while the connection is in the iSCSI Full Feature Phase. Specifying what the implementation does in this case is outside the scope of this document -- [RFC3720] already defines the error handling for this error scenario.
10.3.11. Login Response 💀
Send_Control requests the transmission of a Login Response PDU:
- a) BHS of the Login Response PDU as defined in [RFC3720].
- b)
DataDescriptorLoginResponse: defines the iSCSI Login Response buffer.
Note that specific Datamover protocols may choose to disallow the standard DA Primitives from being used for the iSCSI Login Phase. When used with those Datamover protocols, trying to send a Login Response through a Send_Control invocation is clearly an error scenario, because the Login Response PDU is being sent while in the iSCSI Full Feature Phase. Specifying what the implementation does in this case is outside the scope of this document -- [RFC3720] already defines the error handling for this error scenario.
10.3.12. Logout Command 👑
Send_Control requests the transmission of a Logout Command PDU. Just the BHS:
- a) BHS of the Logout Command PDU as defined in [RFC3720].
10.3.13. Logout Response 🌷
Send_Control again, for a Logout Response PDU. Just the BHS:
- a) BHS of the Logout Response PDU as defined in [RFC3720].
10.3.14. SNACK Request 🍭
Send_Control requests the transmission of a SNACK Request PDU. Just the BHS:
- a) BHS of the SNACK Request PDU as defined in [RFC3720].
10.3.15. Reject 🙅♀️
Send_Control requests the transmission of a Reject PDU:
- a) BHS of the Reject PDU as defined in [RFC3720].
- b)
DataDescriptorReject: defines the iSCSI Reject buffer.
10.3.16. NOP-Out 🏓
Send_Control requests the transmission of a NOP-Out PDU:
- a) BHS of the NOP-Out PDU as defined in [RFC3720].
- b)
DataDescriptorNOPOut: defines the iSCSI Ping data buffer.
10.3.17. NOP-In 🏓
Send_Control requests the transmission of a NOP-In PDU:
- a) BHS of the NOP-In PDU as defined in [RFC3720].
- b)
DataDescriptorNOPIn: defines the iSCSI Return Ping data buffer.
10.4. Interactions for Receiving an iSCSI PDU 📬
Okay, flipping sides now. The only PDUs an iSCSI layer sitting on a Datamover layer ever receives are the iSCSI control-type PDUs. The Datamover layer hands them over as they arrive, and tags each one with the Connection_Handle (see Section 5.3) saying which iSCSI connection it's meant for. After that, processing the iSCSI control-type PDUs goes exactly as defined in [RFC3720].
10.4.1. General Control-Type PDU Notification 🔔
This sub-section covers the general mechanics that apply to several control-type PDUs. The sub-sections after it note extra considerations for the control-type PDUs that aren't covered here.
The Control_Notify Operational Primitive is what notifies the iSCSI layer that one of these control-type PDUs has arrived: SCSI Command, SCSI Response, Task Management Function Request, Task Management Function Response, Asynchronous Message, Text Request, Text Response, Logout Command, Logout Response, SNACK, Reject, NOP-Out, NOP-In.
10.4.2. SCSI Data Transfer PDUs 🚚
10.4.2.1. SCSI Data-Out 💌
Control_Notify tells the iSCSI layer that a SCSI Data-Out PDU carrying the non-immediate unsolicited data has arrived. Now, the solicited SCSI Data-Out arriving on the target does not trigger a Control_Notify to the iSCSI layer, because the initiator iSCSI layer didn't send the solicited SCSI Data-Out as control-type PDUs. Sneaky, right? 🙈
10.4.2.2. SCSI Data-In 🧃
The Datamover layer does not notify the iSCSI layer when SCSI Data-In arrives at the initiator, because SCSI Data-In is an iSCSI data-type PDU (see section 5.1). The initiator's iSCSI layer can still infer that the SCSI Data-In arrived, though, once it gets a later Control_Notify about the SCSI Response PDU.
Now, this document doesn't contemplate a Data-In PDU ever being received at the initiator iSCSI layer 🤨. Specific Datamover protocols may define how to handle an unexpected inbound SCSI Data-In PDU that ends up with the initiator iSCSI layer receiving it. This document leaves the details of that error scenario to the specific Datamover protocols, so each one can define its own error handling for its Datamover environment.
10.4.2.3. Ready To Transfer (R2T) 🎈
R2T PDUs are iSCSI data-type PDUs (see Section 5.1) that never get handed to the initiator iSCSI layer as-is. So the Datamover layer doesn't notify the iSCSI layer when an R2T PDU arrives. When an iSCSI node sends an R2T PDU to its local Datamover layer, the local and remote Datamover layers quietly take care of the data transfer that the R2T asked for ✨.
Same plot twist as before: this document doesn't contemplate an R2T PDU ever being received at the initiator iSCSI layer. Specific Datamover protocols may define how to handle an unexpected inbound R2T PDU that ends up with the initiator iSCSI layer receiving it. This document leaves that error scenario to the specific Datamover protocols, so each one can define the right error handling for its Datamover environment 💖.
10.4.3. Login Request
okay so the Control_Notify Operational Primitive is how the target iSCSI layer finds out a Login Request PDU has arrived 📬✨ bestie, lowkey plot twist: specific Datamover protocols are allowed to say "nope, no standard DA Primitives for the iSCSI Login Phase" 🙅♀️. When one of those is in use, a Login Request that needs a Control_Notify call is clearly an error scenario 💀, because the Login Request PDU is arriving in the iSCSI Full Feature Phase, which is not where it belongs. What the implementation should do about it is outside the scope of this doc 🙈. [RFC3720] already defines the error handling for this exact situation, so that's where to look 💖.
10.4.4. Login Response
the Control_Notify Operational Primitive is used to tell the initiator iSCSI layer that a Login Response PDU has arrived 🦋. Same tea as before: specific Datamover protocols may choose to disallow the standard DA Primitives for the iSCSI Login Phase 🌸. If you're using one of those, a Login Response that needs a Control_Notify call is clearly an error scenario 😭, since the Login Response PDU is showing up in the iSCSI Full Feature Phase. Figuring out the resulting implementation behavior isn't this document's job, and [RFC3720] already covers the error handling for this scenario 🫶.
11. Security Considerations ✨
11.1. Architectural Considerations
okay security time, and this part is SERIOUS 🔥 DA lets compliant iSCSI implementations split control and data apart in the way they talk to their Datamover protocols 🧚. BUT, and this is a big but 💅, that separation does NOT mean control traffic and data traffic get their own transport medium. The basic iSCSI architecture for tasks and how PDUs relate to tasks stays exactly the same 👑. [RFC3720] defines several MUST requirements on ordering relationships across control and data for a given task, plus a mandatory deterministic task allegiance model. DA doesn't change any of that (DA has a normative reference to [RFC3720]), and it doesn't allow any extra flexibility in compliance in this area 🙅♀️.
To sum it up, sending bulk data transfers (the ones prompted by Put_Data and Get_Data Primitive invocations) over a different transport medium would be just as ill-advised as sending only the Data-Out/Data-In PDUs on a different TCP connection in RFC 3720-based iSCSI implementations 😭. So, all the iSCSI-related security text in [RFC3723] is directly applicable to a DA-enabled iSCSI implementation 🔐.
Another area with security implications is the Datamover connection resource management model that DA defines 🌈, especially the Allocate_Connection_Resources Primitive. If that model gets realized inadvertently, an iSCSI implementation could be left exposed to denial-of-service attacks 💀. As Figures 2 and 3 in Section 13.2 illustrate, the most effective countermeasure is to do the Datamover resource allocation once the iSCSI layer is far enough along in the iSCSI Login Phase that it's reasonably certain the peer side is not an attacker 🛡️. In particular, if the Login Phase includes a SecurityNegotiation stage, an iSCSI end node MUST defer the Datamover connection resource allocation (i.e., invoking the Allocate_Connection_Resources Primitive) to the LoginOperationalNegotiation stage [RFC3720], so the resource allocation happens post-authentication. This considerably minimizes the potential for a denial-of-service attack 🫶.
11.2. Wire Protocol Considerations
because the DA architecture itself doesn't define any new wire protocol or propose modifications to the existing protocols, there are no additional wire protocol security considerations in employing DA itself 🦋. HOWEVER (yes, this is loud on purpose 📣), a DA-compliant iSCSI implementation MUST comply with all the iSCSI-related requirements stipulated in [RFC3723] and [RFC3720]. Also note that when realizing DA, each Datamover protocol must define and elaborate, as appropriate, on any additional security considerations that come from using that Datamover protocol 💜.
All Datamover protocol designers are strongly recommended to refer to [RDDPSEC] for the types of security issues to consider 🙈. [RDDPSEC] elaborates on the security considerations for an RDDP-based Datamover [iSER], and it's representative of the kind of analysis needed for resource exhaustion and the application of countermeasures for any Datamover protocol, so it's a must-read 📚✨.
12. References 🎀
12.1. Normative References
[RFC3720] Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and E. Zeidner, "Internet Small Computer Systems Interface (iSCSI)", RFC 3720, April 2004.
[RFC3723] Aboba, B., Tseng, J., Walker, J., Rangan, V., and F. Travostino, "Securing Block Storage Protocols over IP", RFC 3723, April 2004.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
12.2. Informative References
[DDP] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct Data Placement over Reliable Transports", RFC 5041, October 2007.
[iSER] Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah, H., and P. Thaler, "Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)", RFC 5046, October 2007.
[RDDPSEC] Pinkerton, J. and E. Deleganes, "Direct Data Placement Protocol (DDP) / Remote Direct Memory Access Protocol (RDMAP) Security", RFC 5042, October 2007.
Appendix A. Design Considerations and Examples 🧚♀️
A.1. Design Considerations for a Datamover Protocol
this section is the tea on the specific considerations for RDMA-based and RDDP-based Datamover protocols ☕💖
a) heads up, the modeling of interactions for SCSI Data-Out (Section 10.3.5.1) is only used for unsolicited data transfer 📦.
b) the modeling of interactions for SNACK (Sections 10.3.14 and 10.4.1) is not expected to be used, because one of the design requirements on the Datamover is that it "guarantees an error-free, reliable, in-order transport mechanism" (Section 6) 🙌. The interactions for sending and receiving a SNACK are still modeled in this document, though, because the receiving iSCSI layer can deterministically deal with an inadvertent SNACK 🤓. This also shows the DA designers' intent that DI is not meant to filter certain types of PDUs 💅.
c) the onus is on a reliable Datamover (per the requirements in Section 6) to realize end-to-end data acknowledgements via Datamover-specific means 🫶. Given that, even use of data-ACK-type SNACKs is unnecessary 🙃. So an initiator may never request sending a SNACK Request in this model, assuming the proactive (timeout-driven) SNACK functionality is turned off in the legacy iSCSI code 💀.
d) the current DA model for bootstrapping a Connection_Handle into service (i.e., associating a new iSCSI connection with a Connection_Handle) clearly implies that the iSCSI connection must already be in Full Feature Phase when the Datamover layer comes into the stack ✨. This further implies that the iSCSI Login Phase must be carried out in the traditional "Byte streaming mode" with no assistance or involvement from the Datamover layer 🦋.
A.2. Examples of Datamover Interactions
the figures in this section show some examples of how Operational Primitives get used in interactions between the iSCSI layer and the Datamover layer 🎨. Here are the abbreviations used in this section, so you can decode the diagrams like a pro 🧠:
Avail - Available
Abted - Aborted
Buf - I/O Buffer
Cmd - Command
Compl - Complete
Conn - Connection
Ctrl_Ntfy - Control_Notify
Dal_Tk_Res - Deallocate_Task_Resources
Data_Cmp_Nfy - Data_Completion_Notify
Data_ACK_Nfy - Data_ACK_Notify
DM - Datamover
Imm - Immediate
Snd_Ctrl - Send_Control
Msg - Message
Resp - Response
Sol - Solicited
TMF Req - Task Management Function Request
TMF Res - Task Management Function Response
Trans - Transfer
Unsol - Unsolicited | | Allocate_Connection_Resources | D | ^
| |------------------------------->| a | |
| | Connection resources are | t | |
| i | successfully allocated | a | | iSCSI
| S | | m | | Login
| C | | o | | Phase
| S | | v | |
| I | | e | |
| | | r | | Login Phase
| L | Final Login Response (success) v succeeds
| a |<----------------------------------------^
| y | | L | | iSCSI
| e | Enable_Datamover | a | | Full
| r |------------------------------->| y | | Feature
| | Datamover is enabled | e | | Phase
| | | r | |
| | Full Feature Phase | | |
| | control and data Transfer | | vFigure 2. A Successful iSCSI Login on Initiator ✨
| | Notice_Key_Values | | |
| |------------------------------->| | |
| | Datamover layer is notified | | |
| | of the negotiated key values | | |
| | | | |
| | Allocate_Connection_Resources | | |
| |------------------------------->| D | |
| | Connection resources are | a | |
| i | successfully allocated | t | | iSCSI
| S | | a | | Login
| C | | m |Final | Phase
| S | | o |Login |
| I |Enable_Datamover(Login Response)| v |Resp |
| |------------------------------->| e |---->vLogin Phase
| L | Datamover is enabled | r | ^ succeeds
| a | | | |
| y | | L | | iSCSI
| e | | a | | Full
| r | | y | | Feature
| | | e | | Phase
| | Full Feature Phase | r | |
| | control and data Transfer | | |
| | | | vFigure 3. A Successful iSCSI Login on Target 🎀
Okay bestie, the figures keep coming and they are dramatic 💅🦋
| | Allocate_Connection_Resources | D | ^
| |------------------------------->| a | |
| | Connection resources are | t | |
| i | successfully allocated | a | | iSCSI
| S | | m | | Login
| C | | o | | Phase
| S | | v | |
| I | | e | |
| | | r | | Login
| | | | | Phase
| L | Final Login Response (failure) v fails
| a |<------------------------------------------
| y | | L |
| e | Deallocate_Connection_Resources| a |
| r |------------------------------->| y |
| | Datamover-specific | e |
| | connection resources freed | r |
| | | |
| |
| | Connection terminated by standard means
| |--------------------------------------------->Figure 4. A Failed iSCSI Login on Initiator
Okay so this one is the initiator side of a failed login. Connection resources get allocated, the final login response comes back as a failure, and then everything gets cleaned up. Not the vibe, but very clear 🥲
| | Allocate_Connection_Resources | D | ^
| |------------------------------->| a | |
| | Connection resources are | t | |
| i | successfully allocated | a | | iSCSI
| S | | m | | Login
| C | | o | | Phase
| S | | v | |
| I | | e | |
| | | r | | Login
| | | | | Phase
| L | Final Login Response (failure) v fails
| a |---------------------------------------------->
| y | | L |
| e | Deallocate_Connection_Resources| a |
| r |------------------------------->| y |
| | Datamover-specific | e |
| | connection resources freed | r |
| | | |
| |
| | Connection terminated by standard means
| |-------------------------------------------->Figure 5. A Failed iSCSI Login on Target
Same drama, but this time it's the target 🎀 The arrows point the other way, which is giving mirror energy ✨
| | Allocate_Connection_Resources | D | ^
| |------------------------------->| a | |
| | Connection resources are | t | |
| i | successfully allocated | a | | iSCSI
| S | | m | | Login
| C | | o | | Phase
| S | | v | |
| I | | e | |
| | | r | |
| L | Login non-Final Request/Response |
| a |<-----------------------------------------|
| y | iSCSI layer decides not to | L | |
| e | enable Datamover for this | a | |
| r | connection | y | |
| | | e | |
| | Deallocate_Connection_Resources| r | |
| |------------------------------->| | |
| | All Datamover-specific | | |
| | resources deallocated | | |
| | | | | Login
| | | | | Phase
| | | continues
| | Regular Login negotiation continues |
| |<---------------------------------------->|
| | .
| | .
| | .Figure 6. iSCSI Does Not Enable the Datamover
Here the iSCSI layer just says "nah, not this connection" 🙈 Everything Datamover-specific gets deallocated and regular login negotiation carries on like nothing happened.
| | | | ^
| | Full Feature Phase Control & | | |
| | Data Transfer Using DM | D | | iSCSI
| | | a | | Full Feature
| i | | t | | Phase
| S | | a | | (DM Enabled)
| C | | m | |
| S | Successful iSCSI Logout | o | |
| I | Connection_Terminate | e | v
| |------------------------------->| r |
| L | Connection is terminated | |
| a | Datamover-specific resources | L | Transport
| y | deallocated, both connection | a | Connection
| e | level & task level | y | is terminated
| r | | e |
| | | r |
| | | |
| | | |Figure 7. A Normal iSCSI Connection Termination
The happy ending 🌸 A successful iSCSI logout kicks off Connection_Terminate, and the Datamover-specific resources get cleaned up at both the connection and task level. Iconic exit ✨
| | | | ^
| | Full Feature Phase Control & | D | | iSCSI
| | Data Transfer Using DM | a | | Full Feature
| i | | t | | Phase
| S | | a | | (DM Enabled)
| C | | m | |
| S | | o | v
| I | Datamover-specific resources | v |<--Transport
| | deallocated, both connection | e | Connection
| L | level & task level | r | Terminated (e.g.
| a | | | unexpected
| y | | L | FIN/RESET)
| e | Connection_Terminate_Notify | a |
| r |<-------------------------------| y |
| | | e |
| | | r |
| | | |Figure 8. An Abnormal iSCSI Connection Termination
And now the plot twist 😭 The transport connection gets terminated unexpectedly, for example by a FIN or RESET, and the Datamover still tells everyone through Connection_Terminate_Notify. Not ideal, but the cleanup still happens, periodt 💀
<-----Initiator-----> <-------Target------->
| | | | DM Msg holding | | | |
SCSI | | | | SCSI Cmd PDU & | | | |SCSI
Cmd | | Snd_Ctrl | |Unsol Imm Data | |Ctrl_Notify | |Cmd
---->| |--------->| |--------------->| |----------->| |--->
| | | | | | | |
| | | | DM Msg holding | | | |
| | Snd_Ctrl | |SCSI Dataout PDU| |Ctrl_Notify | |
| |--------->| |--------------->| |----------->| |
| | . | | . | | . | |Unsol
| | . | D| . | D| . | |Data
| | . | a| DM Msg holding | a| . | |Trans
| i| Snd_Ctrl | t|SCSI Dataout PDU| t|Ctrl_Notify | i|
| S|--------->| a|--------------->| a|----------->| S|
| C| | m| | m| | C|Buf
| S| | o| | o| | S|Avail
| I| | v| | v| Get_Data | I|(R2T)
| | | e|----------------| e|<-----------| |<----
| L| | r||Solicited Data | r| | L| .
| a| | || Transfer | | | a| .
| y| | L|--------------->| L| . | y|Buf
| e| | a| . | a| . | e|Avail
| r| | y| . | y| Get_Data | r|(R2T)
| | | e|----------------| e|<-----------| |<----
| | | r||Solicited Data | r| | |
| | | || Transfer | | | |
| | | |--------------->| |Data_Cmp_Nfy| |Data
| | | | | |----------->| |Trans
| | | | | | | |Compl
| | | | DM Msg holding | | | |
SCSI | | | |SCSI Resp PDU & | | | |SCSI
Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp
<----| |<---------| |<---------------| |<-----------| |<----
| | | | | | | |
Figure 9. A SCSI Write Data TransferFigure 9. A SCSI Write Data Transfer
Okay this is a whole production 🎬 The initiator sends a SCSI command, then SCSI Dataout PDUs go out in solicited chunks each time a buffer is available (R2T), and the whole thing wraps up with a SCSI response and sense data. Main character energy for real 👑
<-----Initiator-----> <-------Target------->
| | | | | | | |
SCSI | | | | DM Msg holding | | | |SCSI
Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd
---->| |--------->| |--------------->| |----------->| |--->
| | | | | | | |
| | | D| SCSI Read | D| | |Buf
| | | a| Data Transfer | a| Put_Data | |Avail
| i| | t|<---------------| t|<-----------| i|<----
| S| | a| . | a| . | S| .
| C| | m| . | m| . | C| .
| S| | o| . | o| . | S| .
| I| | v| SCSI Read | v| . | I|Buf
| | | e| Data Transfer | e| Put_Data | |Avail
| L| | r|<---------------| r|<-----------| L|<----
| a| | | | | | a|
| y| | L| | L| | y|
| e| | a| | a|Data_Cmp_Nfy| e|Data
| r| | y| | y|----------->| r|Trans
| | | e| | e| | |Compl
| | | r| DM Msg holding | r| | |
SCSI | | | |SCSI Resp PDU & | | | |SCSI
Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp
<----| |<---------| |<---------------| |<-----------| |<----
| | | | | | | |
Figure 10. A SCSI Read Data Transfer
<-----Initiator-----> <-------Target------->Figure 10. A SCSI Read Data Transfer
And the mirror image 🦋 This time the target pushes SCSI Read data back with Put_Data each time a buffer frees up, and then the SCSI response and sense data close it out. Read it like a plot twist and you're golden ✨
Okay bestie, the last part is basically all diagrams and then the wrap-up stuff, so grab a snack 🍓 Here are the final two sequence diagrams from the examples section, and they show what a SCSI read acknowledgement and an abort cleanup look like on the wire 🦋
| | | | | | | |
SCSI | | | | DM Msg holding | | | |SCSI
Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd
---->| |--------->| |--------------->| |----------->| |---->
| | | | | | | |
| | | D| SCSI Read | D| Put_Data | |Buf
| | | a| Data Transfer | a|Data_in.A=1 | |Avail
| i| | t|<---------------| t|<-----------| i|<----
| S| | a| . | a| . | S| .
| C| | m| . | m|Data_ACK_Nfy| C| .
| S| | o| | o|----------->| S| .
| I| | v| | v| . | I|
| | | e| | e| . | |
| L| | r| | r| | L|
| a| | | | | | a|
| y| | L| | L| | y|
| e| | a| | a| | e|Data
| r| | y| | y| | r|Trans
| | | e| | e| | |Compl
| | | r| DM Msg holding | r| | |
SCSI | | | |SCSI Resp PDU & | | | |SCSI
Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp
<----| |<---------| |<---------------| |<-----------| |<----
| | | | | | | |
Figure 11. A SCSI Read Data Acknowledgement
<-----Initiator-----> <-------Target------->
| | | | | | | |
SCSI | | | | DM Msg holding | | | |SCSI
Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd
---->| |--------->| |--------------->| |----------->| |---->
| | | | | | | |
| | | D| SCSI Read | D| | |Buf
| | | a| Data Transfer | a| Put_Data | |Avail
| i| | t|<---------------| t|<-----------| i|<----
| S| | a| . | a| . | S| .
Abort| C| | m| DM Msg holding | m| . | C|Abort
Task | S| Snd_Ctrl | o| Abort TMF Req | o|Ctrl_Notify | S|Task
---->| I|--------->| v|--------------->| v|----------->| I|---->
| | | e| . | e| . | |
Abort| L| | r| DM Msg holding| r| | L| .
Done | a|Ctrl_Ntfy | | Abort TMF Res| | Snd_Ctrl | |Abted
<----| y|<---------| L|<---------------| L|<-----------| y|<----
| e| | a| | a| | e|
| r| | y| | y| | r|
| | | e| | e| | |
| | | r| | r| | |
| | | | | | | |
| |Dal_Tk_Res| | | |Dal_Tk_Res | |
| |--------->| | | |<-----------| |
| | | | | | | |
Figure 12. Task Resource Cleanup on AbortAcknowledgements 🫶
The IP Storage (IPS) Working Group in the Transport Area of IETF is the crew that defined the iSCSI protocol, plus a whole bunch of other IP Storage protocols 💜 The authors are so grateful to the entire working group, because their work let this document build on the ideas and details of iSCSI. Iconic teamwork 👑
Next up, the people who reviewed and helped make this document better. The authors are grateful for every single one of them ✨
John Carrier
Adaptec, Inc.
691 S. Milpitas Blvd., Milpitas, CA 95035, USA
Phone: +1 (360) 378-8526
EMail: john_carrier@adaptec.com
Hari Ghadia
Adaptec, Inc.
691 S. Milpitas Blvd., Milpitas, CA 95035, USA
Phone: +1 (408) 957-5608
EMail: hari_ghadia@adaptec.com
Hari Mudaliar
Adaptec, Inc.
691 S. Milpitas Blvd., Milpitas, CA 95035, USA
Phone: +1 (408) 957-6012
EMail: hari_mudaliar@adaptec.com
Patricia Thaler
Agilent Technologies, Inc.
1101 Creekside Ridge Drive, #100, M/S-RG10,
Roseville, CA 95678, USA
Phone: +1-916-788-5662
EMail: pat_thaler@agilent.com
Uri Elzur
Broadcom Corporation
16215 Alton Parkway, Irvine, CA 92619-7013, USA
Phone: +1 (949) 585-6432
EMail: Uri@Broadcom.com
Mike Penna
Broadcom Corporation
16215 Alton Parkway,Irvine, CA 92619-7013, USA
Phone: +1 (949) 926-7149
EMail: MPenna@Broadcom.com
David Black
EMC Corporation
176 South St., Hopkinton, MA 01748, USA
Phone: +1 (508) 293-7953
EMail: black_david@emc.com
Ted Compton
EMC Corporation
Research Triangle Park, NC 27709, USA
Phone: +1-919-248-6075
EMail: compton_ted@emc.com
Dwight Barron
Hewlett-Packard Company
20555 SH 249, Houston, TX 77070-2698, USA
Phone: +1 (281) 514-2769
EMail: Dwight.Barron@Hp.com
Paul R. Culley
Hewlett-Packard Company
20555 SH 249, Houston, TX 77070-2698, USA
Phone: +1 (281) 514-5543
EMail: paul.culley@hp.com
Dave Garcia
Hewlett-Packard Company
19333 Vallco Parkway, Cupertino, CA 95014, USA
Phone: +1 (408) 285-6116
EMail: dave.garcia@hp.com
Randy Haagens
Hewlett-Packard Company
8000 Foothills Blvd, MS 5668, Roseville CA, USA
Phone: +1-916-785-4578
EMail: randy_haagens@hp.com
Jeff Hilland
Hewlett-Packard Company
20555 SH 249, Houston, TX 77070-2698, USA
Phone: +1 (281) 514-9489
EMail: jeff.hilland@hp.com
Mike Krause
Hewlett-Packard Company, 43LN
19410 Homestead Road, Cupertino, CA 95014, USA
Phone: +1 (408) 447-3191
EMail: krause@cup.hp.com
Jim Wendt
Hewlett-Packard Company
8000 Foothills Blvd, MS 5668, Roseville CA, USA
Phone: +1-916-785-5198
EMail: jim_wendt@hp.com
Mike Ko
IBM
650 Harry Rd, San Jose, CA 95120, USA
Phone: +1 (408) 927-2085
EMail: mako@us.ibm.com
Renato Recio
IBM Corporation
11501 Burnett Road, Austin, TX 78758, USA
Phone: +1 (512) 838-1365
EMail: recio@us.ibm.com
Howard C. Herbert
Intel Corporation
MS CH7-404,5000 West Chandler Blvd., Chandler, AZ 85226, USA
Phone: +1 (480) 554-3116
EMail: howard.c.herbert@intel.com
Dave Minturn
Intel Corporation
MS JF1-210, 5200 North East Elam Young Parkway
Hillsboro, OR 97124, USA
Phone: +1 (503) 712-4106
EMail: dave.b.minturn@intel.com
James Pinkerton
Microsoft Corporation
One Microsoft Way, Redmond, WA 98052, USA
Phone: +1 (425) 705-5442
EMail: jpink@microsoft.com
Tom Talpey
Network Appliance
375 Totten Pond Road, Waltham, MA 02451, USA
Phone: +1 (781) 768-5329
EMail: thomas.talpey@netapp.comAuthors' Addresses 🎀
Meet the people who wrote this one 🧚
Mallikarjun Chadalapaka
Hewlett-Packard Company
8000 Foothills Blvd.
Roseville, CA 95747-5668, USA
Phone: +1-916-785-5621
EMail: cbm@rose.hp.com
John L. Hufferd
Brocade, Inc.
1745 Technology Drive
San Jose, CA 95110, USA
Phone: +1-408-333-5244
EMail: jhufferd@brocade.com
Julian Satran
IBM, Haifa Research Lab
Haifa University Campus - Mount Carmel
Haifa 31905, Israel
Phone +972-4-829-6264
EMail: Julian_Satran@il.ibm.com
Hemal Shah
Broadcom Corporation
5300 California Avenue
Irvine, California 92617, USA
Phone: +1-949-926-6941
EMail: hemal@broadcom.comComments may be sent to Mallikarjun Chadalapaka 💌
Full Copyright Statement 📜
Copyright (C) The IETF Trust (2007).
This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.
This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property 💅
The IETF takes no position on the validity or scope of any Intellectual Property Rights or other rights that might be claimed to apply to the technology described in this document, or on whether any license under those rights might be available. It also doesn't claim to have made any independent effort to find such rights. You can find the procedures for rights in RFC documents in BCP 78 and BCP 79 🌈
Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses to be made available, or the results of an attempt to obtain a general license or permission for implementers or users to use these proprietary rights can be found in the IETF on-line IPR repository at http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology needed to implement this standard. Please send that information to the IETF at ietf-ipr@ietf.org. Thanks for looking out, babe 🫶
📦 where this page comes from
- ietf: rfc5047 2007-10
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms