RFC 5046
Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)
RFC 5046: Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA) 🎀
Okay so bestie, this is RFC 5046, a Standards Track doc from the Network Working Group, dated October 2007 ✨. The header lists M. Ko (IBM Corporation), M. Chadalapaka (Hewlett-Packard Company), J. Hufferd (Brocade, Inc.), U. Elzur, H. Shah, and P. Thaler (Broadcom Corporation) as authors 🫶. The title block doesn't list anything this RFC obsoletes or updates, so there's nothing to chase there 💅.
Status of This Memo 🦋
This document specifies an Internet standards track protocol for the Internet community, and it's asking for discussion and suggestions for improvements 🧚. For the standardization state and status of this protocol, check the current edition of the "Internet Official Protocol Standards" (STD 1) 📚. Distribution of this memo is unlimited 🌈.
Abstract 🌸
Okay, the big idea: iSCSI (the protocol that lets computers talk to storage over a network) gets RDMA data transfer by layering iSCSI on top of an RDMA-Capable Protocol, like the iWARP protocol suite 🔥. RDMA-Capable Protocols give you RDMA Read and Write services, which means data can go straight into SCSI I/O Buffers without intermediate data copies 😭. That's giving zero-copy energy 💜. This document describes the extensions to iSCSI that support those RDMA services, as provided by an RDMA-Capable Protocol such as iWARP 👑.
Table of Contents 🍓
1. Introduction ....................................................5
1.1. Motivation .................................................5
1.2. Architectural Goals ........................................6
1.3. Protocol Overview ..........................................7
1.4. RDMA Services and iSER .....................................8
1.4.1. STag ................................................8
1.4.2. Send ................................................9
1.4.3. RDMA Write ..........................................9
1.4.4. RDMA Read ...........................................9
1.5. SCSI Read Overview ........................................10
1.6. SCSI Write Overview .......................................10
1.7. iSCSI/iSER Layering .......................................10
2. Definitions and Acronyms .......................................11
2.1. Definitions ...............................................11
2.2. Acronyms ..................................................17
2.3. Conventions ...............................................19
3. Upper Layer Interface Requirements .............................19
3.1. Operational Primitives Offered by iSER ....................20
3.1.1. Send_Control .......................................20
3.1.2. Put_Data ...........................................20
3.1.3. Get_Data ...........................................21
3.1.4. Allocate_Connection_Resources ......................21
3.1.5. Deallocate_Connection_Resources ....................22
3.1.6. Enable_Datamover ...................................22
3.1.7. Connection_Terminate ...............................22
3.1.8. Notice_Key_Values ..................................23
3.1.9. Deallocate_Task_Resources ..........................23
3.2. Operational Primitives Used by iSER .......................23
3.2.1. Control_Notify .....................................24
3.2.2. Data_Completion_Notify .............................24
3.2.3. Data_ACK_Notify ....................................24
3.2.4. Connection_Terminate_Notify ........................25
3.3. iSCSI Protocol Usage Requirements .........................25
4. Lower Layer Interface Requirements .............................26
4.1. Interactions with the RCaP Layer ..........................26
4.2. Interactions with the Transport Layer .....................27
5. Connection Setup and Termination ...............................27
5.1. iSCSI/iSER Connection Setup ...............................27
5.1.1. Initiator Behavior .................................29
5.1.2. Target Behavior ....................................30
5.1.3. iSER Hello Exchange ................................32
5.2. iSCSI/iSER Connection Termination .........................33
5.2.1. Normal Connection Termination at the Initiator .....33
5.2.2. Normal Connection Termination at the Target ........34
5.2.3. Termination without Logout Request/Response PDUs ...34
6. Login/Text Operational Keys ....................................35
6.1. HeaderDigest and DataDigest ...............................35
6.2. MaxRecvDataSegmentLength ..................................36
6.3. RDMAExtensions ............................................36
6.4. TargetRecvDataSegmentLength ...............................37
6.5. InitiatorRecvDataSegmentLength ............................38
6.6. OFMarker and IFMarker .....................................38
6.7. MaxOutstandingUnexpectedPDUs ..............................38
7. iSCSI PDU Considerations .......................................39
7.1. iSCSI Data-Type PDU .......................................39
7.2. iSCSI Control-Type PDU ....................................40
7.3. iSCSI PDUs ................................................40
7.3.1. SCSI Command .......................................40
7.3.2. SCSI Response ......................................42
7.3.3. Task Management Function Request/Response ..........44
7.3.4. SCSI Data-Out ......................................45
7.3.5. SCSI Data-In .......................................46
7.3.6. Ready to Transfer (R2T) ............................48
7.3.7. Asynchronous Message ...............................50
7.3.8. Text Request and Text Response .....................50
7.3.9. Login Request and Login Response ...................50
7.3.10. Logout Request and Logout Response ................51
7.3.11. SNACK Request .....................................51
7.3.12. Reject ............................................51
7.3.13. NOP-Out and NOP-In ................................51
8. Flow Control and STag Management ...............................52
8.1. Flow Control for RDMA Send Message Types ..................52
8.1.1. Flow Control for Control-Type PDUs from the
Initiator ..........................................52
8.1.2. Flow Control for Control-Type PDUs from the
Target .............................................55
8.2. Flow Control for RDMA Read Resources ......................56
8.3. STag Management ...........................................56
8.3.1. Allocation of STags ................................57
8.3.2. Invalidation of STags ..............................57
9. iSER Control and Data Transfer .................................58
9.1. iSER Header Format ........................................58
9.2. iSER Header Format for the iSCSI Control-Type PDU .........59
9.3. iSER Header Format for the iSER Hello Message .............60
9.4. iSER Header Format for the iSER HelloReply Message ........61
9.5. SCSI Data Transfer Operations .............................62
9.5.1. SCSI Write Operation ...............................62
9.5.2. SCSI Read Operation ................................63
9.5.3. Bidirectional Operation ............................64
10. iSER Error Handling and Recovery ..............................64
10.1. Error Handling ...........................................64
10.1.1. Errors in the Transport Layer .....................64
10.1.2. Errors in the RCaP Layer ..........................65
10.1.3. Errors in the iSER Layer ..........................66
10.1.4. Errors in the iSCSI Layer .........................67
10.2. Error Recovery ...........................................69
10.2.1. PDU Recovery ......................................69
10.2.2. Connection Recovery ...............................70
11. Security Considerations .......................................71
12. References ....................................................71
12.1. Normative References .....................................71
12.2. Informative References ...................................72
Appendix A. iWARP Message Format for iSER .........................73
A.1. iWARP Message Format for iSER Hello Message ...............73
A.2. iWARP Message Format for iSER HelloReply Message ..........74
A.3. iWARP Message Format for SCSI Read Command PDU ............75
A.4. iWARP Message Format for SCSI Read Data ...................76
A.5. iWARP Message Format for SCSI Write Command PDU ...........77
A.6. iWARP Message Format for RDMA Read Request ................78
A.7. iWARP Message Format for Solicited SCSI Write Data ........79
A.8. iWARP Message Format for SCSI Response PDU ................80
Appendix B. Architectural Discussion of iSER over InfiniBand ......81
B.1. The Host Side of the iSCSI and iSER Connections
in InfiniBand .............................................81
B.2. The Storage Side of the iSCSI and iSER Mixed
Network Environment .......................................82
B.3. Discovery Processes for an InfiniBand Host ................82
B.4. IBTA Connection Specifications ............................83
Acknowledgments ...................................................83Table of Figures 🧚
Figure 1. Example of iSCSI/iSER Layering in Full Feature Phase ....11
Figure 2. iSER Header Format ......................................58
Figure 3. iSER Header Format for iSCSI Control-Type PDU ...........59
Figure 4. iSER Header Format for iSER Hello Message ...............60
Figure 5. iSER Header Format for iSER HelloReply Message ..........61
Figure 6. SendSE Message containing an iSER Hello Message .........72
Figure 7. SendSE Message containing an iSER HelloReply Message ....74
Figure 8. SendSE Message containing a SCSI Read Command PDU .......75
Figure 9. RDMA Write Message containing SCSI Read Data ............76
Figure 10. SendSE Message containing a SCSI Write Command PDU .....77
Figure 11. RDMA Read Request Message ..............................78
Figure 12. RDMA Read Response Message containing SCSI Write Data ..79
Figure 13. SendInvSE Message containing SCSI Response PDU .........80
Figure 14. iSCSI and iSER on IB ...................................81
Figure 15. Storage Controller with TCP, iWARP, and IB Connections .821. Introduction ✨
To be continued in part 2, bestie 🫶💖
1.1. Motivation 💖
Okay so, iSCSI [RFC3720] is basically the SCSI Architecture Model (see [SAM2]) getting to ride over TCP 🦋. SCSI commands travel inside iSCSI requests, and SCSI responses and status come back inside iSCSI responses. Other iSCSI exchanges and SCSI data also get shipped around in iSCSI Protocol Data Units (PDUs) ✨.
Here's the tea 🫖: in the Traditional iSCSI model, out-of-order TCP segments have to be stored and put back together before the iSCSI layer on an end node can drop the data into the iSCSI buffers. That reassembly is needed because not every TCP segment will carry an iSCSI header to say where it goes, and TCP has no built-in way to flag Upper Level Protocol (ULP) message boundaries to help place out-of-order segments. At high network speeds, this TCP reassembly is seriously counterproductive 😭. The reasons: memory bandwidth gets wasted on data copying, you need reassembly memory, CPU cycles get wasted on data copying, and there's the general store-and-forward latency from an application's point of view.
[RFC3720] clocked TCP reassembly as a serious issue, so it introduced a "sync and steering layer" that is optional to implement and use. One specific sync and steering mechanism, called "markers", was defined in [RFC3720]. It gives an application-level way of framing iSCSI PDUs inside the TCP data stream, even when the TCP segments aren't reassembled into order yet 🧚.
With those techniques from [RFC3720], a Network Interface Controller customized for iSCSI (SNIC) could offload the TCP/IP processing and support direct data placement. But most iSCSI implementations don't support iSCSI "markers," so SNIC marker-based direct data placement is unusable in practice 💀.
The iWARP protocol stack provides direct data placement that is usable in practice 🙌. There's also interest in using iSCSI with other Remote Direct Memory Access (RDMA) protocol stacks that support direct data placement, like the one InfiniBand provides. The generic term RDMA-Capable Protocol (RCaP) refers to the RDMA functionality those stacks provide.
Now, with RDMA-Capable Controllers available in a host system (which doesn't have SNICs), it's only right for iSCSI to exploit the direct data placement function of the RDMA-Capable Controller, just like other applications do 👑.
iSCSI Extensions for RDMA (iSER) is designed precisely to take advantage of generic RDMA technologies. iSER's goal is to let iSCSI use direct data placement and RDMA capabilities through a generic RDMA-Capable Controller. In summary, the iSCSI/iSER protocol stack is built to scale to high speeds by relying on a generic data placement process and RDMA technologies and products, which enable direct data placement of both in-order and out-of-order data. Iconic, honestly 🔥.
This document describes iSER as a protocol extension to iSCSI, both for convenience of description and because it's true in a very strict protocol sense. However, note that iSER is in reality extending the connectivity of the iSCSI protocol defined in [RFC3720], and the name iSER reflects this reality 🎀.
When the iSCSI protocol as defined in [RFC3720] (i.e., without the iSER enhancements) is what the rest of the document means, the term "Traditional iSCSI" is used to make the intention clear 💅.
1.2. Architectural Goals 🎯
This section sums up the architectural goals that guided the design of iSER, bestie.
- Provide an RDMA data transfer model for iSCSI that enables direct in-order or out-of-order data placement of SCSI data into pre-allocated SCSI buffers while maintaining in-order data delivery.
- Not require any major changes to the SCSI Architecture Model [SAM2] and SCSI command set standards.
- Utilize the existing iSCSI infrastructure (sometimes called the "iSCSI ecosystem"), including but not limited to MIB, bootstrapping, negotiation, naming and discovery, and security.
- Require a session to operate in the Traditional iSCSI data transfer mode if iSER is not supported by either the initiator or the target. In other words, don't require iSCSI Full Feature Phase interoperability between an end node in Traditional iSCSI mode and an end node in iSER-assisted mode 🙈.
- Allow initiator and target implementations to use generic RDMA-Capable Controllers, such as RDMA-enabled Network Interface Controllers (RNICs), or to implement iSCSI and iSER in software. Don't require iSCSI- or iSER-specific assists in the RCaP implementation or the RDMA-Capable Controller 🧁.
- Require full and only generic RCaP functionality at both the initiator and the target 🤝.
- Implement a lightweight Datamover protocol for iSCSI with minimal state maintenance 🦋.
1.3. Protocol Overview 📋
Consistent with the architectural goals stated in Section 2.2, the iSER protocol does not require changes in the iSCSI ecosystem or any related SCSI specifications. The iSER protocol defines how iSCSI PDUs map to RCaP Messages, in such a way that it is entirely feasible to build iSCSI/iSER implementations on generic RDMA-Capable Controllers. The iSER protocol layer requires minimal state maintenance to assist an iSCSI Full Feature Phase connection, and it is oblivious to the notion of an iSCSI session. The crucial protocol aspects of iSER can be summarized like this ✨:
- iSER-assisted mode is negotiated during the iSCSI login for each session, and an entire iSCSI session can only operate in one mode. In other words, a connection in a session cannot operate in iSER-assisted mode if a different connection of the same session is already in Full Feature Phase in the Traditional iSCSI mode 🚫.
- Once in iSER-assisted mode, all iSCSI interactions on that connection use RCaP Messages.
- A Send Message Type is used to carry an iSCSI control-type PDU preceded by an iSER header. See Section 7.2 for more details on iSCSI control-type PDUs.
- RDMA Write, RDMA Read Request, and RDMA Read Response Messages are used to carry control and all data information associated with the iSCSI data-type PDUs. See Section 7.1 for more details on iSCSI data-type PDUs.
- The target drives all data transfer (with the exception of iSCSI unsolicited data) for SCSI writes and SCSI reads, by issuing RDMA Read Requests and RDMA Writes, respectively 🎬.
- RCaP is responsible for ensuring data integrity. (For example, iWARP includes a CRC-enhanced framing layer called Marker PDU Aligned Framing for TCP (MPA) on top of TCP; and for InfiniBand, the CRCs are included in the Reliable Connection mode.) For this reason, iSCSI header and data digests are negotiated to "None" for iSCSI/iSER sessions 🛡️.
- The iSCSI error recovery hierarchy defined in [RFC3720] is fully supported by iSER. (However, see Section 7.3.11 on the handling of SNACK Request PDUs.) 🧯
- iSER requires no changes to iSCSI authentication, security, and text mode negotiation mechanisms 🔐.
Note that Traditional iSCSI implementations may have to be adapted to employ iSER. The adaptation, when required, is likely to be centered around the upper layer interface requirements of iSER (Section 3). Just a heads up 🫶.
1.4. RDMA Services and iSER 🧩
iSER is designed to work with software and/or hardware protocol stacks that provide the protocol services defined in RCaP documents such as [RDMAP], [IB], etc. The following subsections describe the key protocol elements of RCaP services that iSER relies on.
1.4.1. STag 🏷️
A Steering Tag (STag) is the identifier of an I/O Buffer, unique to an RDMA-Capable Controller, that the iSER layer Advertises to the remote iSCSI/iSER node in order to complete a SCSI I/O.
In iSER, Advertisement is the act of the initiator informing the target that an I/O Buffer is available at the initiator for RDMA Read or RDMA Write access by the target. The initiator Advertises the I/O Buffer by including the STag in the header of an iSER Message containing the SCSI Command PDU sent to the target. The base Tagged Offset is not explicitly specified, but the target must always assume it as zero. The buffer length is as specified in the SCSI Command PDU 📏.
The iSER layer at the initiator Advertises the STag for the I/O Buffer of each SCSI I/O to the iSER layer at the target, in the iSER header of the Send with Solicited Event (SendSE) Message containing the SCSI Command PDU. The exception is when the I/O can be completely satisfied by unsolicited data alone 🙃.
The iSER layer at the target provides the STag for the I/O Buffer that is the Data Sink of an RDMA Read Operation (Section 2.4.4) to the RCaP layer on the initiator node. In other words, this is completely transparent to the iSER layer at the initiator 🪄.
The iSER protocol is defined so that the Advertised STag is automatically invalidated upon a normal completion of the associated task. This automatic invalidation is realized via the Send with Solicited Event and Invalidate (SendInvSE) Message carrying the SCSI Response PDU. There are two exceptions to this automatic invalidation: bidirectional commands, and abnormal completion of a command. In these cases, the iSER layer at the initiator is required to explicitly invalidate the STag, in addition to sanity checking the automatic invalidation even when that does happen 🧹.
1.4.2. Send 📨
Send is the RDMA Operation that is not addressed to an Advertised buffer by the sending side, and thus uses Untagged buffers on the receiving side.
The iSER layer at the initiator uses the Send Operation to transmit any iSCSI control-type PDU to the target. For example, the initiator uses Send Operations to transfer iSER Messages containing SCSI Command PDUs to the iSER layer at the target.
The iSER layer at the target uses the Send Operation to transmit any iSCSI control-type PDU to the initiator. For example, the target uses Send Operations to transfer iSER Messages containing SCSI Response PDUs to the iSER layer at the initiator 💌.
1.4.3. RDMA Write ✍️
RDMA Write is the RDMA Operation used to place data into an Advertised buffer on the receiving side. The sending side addresses the Message using an STag and a Tagged Offset that are valid on the Data Sink.
The iSER layer at the target uses the RDMA Write Operation to transfer the contents of a local I/O Buffer to an Advertised I/O Buffer at the initiator. The iSER layer at the target uses the RDMA Write to transfer whole or part of the data required to complete a SCSI read command 📤.
The iSER layer at the initiator does not employ RDMA Writes 🙅.
1.4.4. RDMA Read 📖
RDMA Read is the RDMA Operation used to retrieve data from an Advertised buffer on a remote node. The sending side of the RDMA Read Request addresses the Message using an STag and a Tagged Offset that are valid on the Data Source, and also provides a valid local STag and Tagged Offset that identify the Data Sink.
The iSER layer at the target uses the RDMA Read Operation to transfer the contents of an Advertised I/O Buffer at the initiator to a local I/O Buffer at the target. The iSER layer at the target uses the RDMA Read to fetch whole or part of the data required to complete a SCSI write command 📥.
The iSER layer at the initiator does not employ RDMA Reads 💅.
1.5. SCSI Read Overview ✨
okay so let's walk through a SCSI Read 🦋 The iSER layer at the initiator gets the SCSI Command PDU from the iSCSI layer. Then it generates an STag for the I/O Buffer of the SCSI Read and advertises that buffer by putting the STag inside the iSER header for the PDU. The iSER Message gets sent to the target using a SendSE Message, and it's giving main character energy 💅
Next, the iSER layer at the target uses one or more RDMA Writes to move the data needed to finish the SCSI Read. So the data just gets written straight into the buffer 🔥
Last step: the iSER layer at the target uses a SendInvSE Message to send the SCSI Response PDU back to the iSER layer at the initiator. The initiator's iSER layer then tells the iSCSI layer that the SCSI Response PDU is ready 🎀
1.6. SCSI Write Overview 🌸
same vibe but for writes, bestie 💜 The iSER layer at the initiator gets the SCSI Command PDU from the iSCSI layer. If solicited data transfer is involved, the initiator's iSER layer generates an STag for the I/O Buffer of the SCSI Write and advertises it by putting the STag in the iSER header for the PDU. The iSER Message goes to the target in a SendSE Message 📨
The iSER layer at the initiator may optionally send one or more non-immediate unsolicited data PDUs to the target, using Send Message Types. Totally up to the initiator 🙈
If solicited data transfer is involved, the iSER layer at the target uses one or more RDMA Reads to pull in the data needed to complete the SCSI Write. Reads from the target's side, iconic 👑
Then, same as before: the iSER layer at the target uses a SendInvSE Message to send the SCSI Response PDU back to the initiator's iSER layer, and the initiator's iSER layer tells the iSCSI layer the SCSI Response PDU is available 🫶
1.7. iSCSI/iSER Layering 🧚
ok so iSCSI Extensions for RDMA (iSER) sits between the iSCSI layer and the RCaP layer 🌈 Heads up though, the RCaP layer might be made of more than one distinct protocol layer, depending on which RCaP you're using. Figure 1 shows an example of how SCSI, iSCSI, iSER, and the different RCaP layers relate to each other. For TCP, the RCaP is iWARP. For InfiniBand, the RCaP is the Reliable Connected Transport Service. And the iSCSI layer described here supports the RDMA Extensions as used in iSER, so it's all connected 🦋
+-------------------------------------+
| SCSI |
+-------------------------------------+
| iSCSI |
DI ------> +-------------------------------------+
| iSER |
+---------+--------------+------------+
| RDMAP | | |
+---------+ InfiniBand | |
| DDP | Reliable | Other |
+---------+ Connected | RDMA- |
| MPA | Transport | Capable |
+---------+ Service | Protocol |
| TCP | | |
+---------+--------------+------------+
| | InfiniBand | Other |
| IP | Network | Network |
| | Layer | Layer |
+---------+--------------+------------+Figure 1. Example of iSCSI/iSER Layering in Full Feature Phase 🌸
2. Definitions and Acronyms 💖
2.1. Definitions 📚
Advertisement (Advertised, Advertise, Advertisements, Advertises) - The act of telling a remote iSER layer that a local node's buffer is available to it. A Node makes a buffer available for incoming RDMA Read Request Message or incoming RDMA Write Message access by telling the remote iSER layer the Tagged Buffer identifiers (STag, TO, and buffer length). Heads up: this Advertisement of Tagged Buffer information is the job of the iSER layer on either end, and the RDMA-Capable Protocol doesn't define it. A typical method is for the iSER layer to embed the Tagged Buffer's STag, TO, and buffer length in a Send Message headed to the remote iSER layer 💌
Completion (Completed, Complete, Completes) - Completion is the process where the RDMA-Capable Protocol layer tells the iSER layer that a particular RDMA Operation has done everything it was supposed to do ✅
Connection - A connection is a logical circuit between the initiator and the target, for example a TCP connection. Communication between the initiator and the target happens over one or more connections. The connections carry control messages, SCSI commands, parameters, and data inside iSCSI Protocol Data Units (iSCSI PDUs) 🔗
Connection Handle - An information element that identifies the particular iSCSI connection and is unique for a given iSCSI-iSER pair. Every invocation of an Operational Primitive is qualified with the Connection Handle 🎀
Data Sink - The peer that receives a data payload. The Data Sink can be required to both send and receive RCaP Messages to transfer a data payload 📥
Data Source - The peer that sends a data payload. The Data Source can be required to both send and receive RCaP Messages to transfer a data payload 📤
Datamover Interface (DI) - The interface between the iSCSI layer and the Datamover layer, as described in [DA] 🧩
Datamover Layer - A layer that sits directly below the iSCSI layer and above the underlying transport layers. This layer exposes and uses a set of transport independent Operational Primitives for communication between the iSCSI layer and itself. Working together with the transport layers, the Datamover layer moves the control and data information on the iSCSI connection. In this specification, the iSER layer is the Datamover layer 🚚
Datamover Protocol - A Datamover protocol is the wire-protocol defined to carry out the Datamover layer's functionality. In this specification, the iSER protocol is the Datamover protocol 📡
Event - An indication that the RDMA-Capable Protocol layer gives the iSER layer to signal a Completion or some other condition that needs immediate attention 🚨
Inbound RDMA Read Queue Depth (IRD) - The maximum number of incoming outstanding RDMA Read Requests that the RDMA-Capable Controller can handle on a particular RCaP Stream at the Data Source. Some RDMA-Capable Protocol layers call this by a different name. For example, for InfiniBand, the equivalent of IRD is the Responder Resources 🔢
Invalidate STag - A mechanism used to stop the Remote Peer from reusing a previously explicitly Advertised STag, until the iSER layer at the local node makes it available again through a later explicit Advertisement 🚫
I/O Buffer - A buffer used in a SCSI Read or Write operation, so SCSI data can be sent from it or received into it 🗂️
iSCSI - The iSCSI protocol, as defined in [RFC3720], is a mapping of the SCSI Architecture Model of SAM-2 over TCP 🌐
iSCSI control-type PDU - Any iSCSI PDU that is neither an iSCSI data-type PDU nor a SCSI Data-out PDU carrying solicited data is defined as an iSCSI control-type PDU. Specifically, SCSI Data-out PDUs for unsolicited data are defined as iSCSI control-type PDUs 🎛️
iSCSI data-type PDU - An iSCSI data-type PDU is an iSCSI PDU that causes data transfer to take place between the peer iSCSI nodes on a Full Feature Phase iSCSI connection, and the remote iSCSI layer doesn't see it. The data transfer happens transparently to the remote iSCSI layer, which means the PDU itself is not delivered as-is to the remote iSCSI layer. The following iSCSI PDUs make up the set of iSCSI data-type PDUs: SCSI Data-In PDU and R2T PDU 🌊
iSCSI Layer - A layer in the protocol stack implementation within an end node that implements the iSCSI protocol and interfaces with the iSER layer via the Datamover Interface 🥰
iSCSI PDU (iSCSI Protocol Data Unit) - The iSCSI layer at the initiator and the iSCSI layer at the target divide their communications into messages. The term "iSCSI protocol data unit" (iSCSI PDU) is used for these messages 📦
iSCSI/iSER Connection - An iSER-assisted iSCSI connection 🔌
iSCSI/iSER Session - An iSER-assisted iSCSI session 🗓️
iSCSI-iSER Pair - The iSCSI layer and the underlying iSER layer, together 👯
iSER - iSCSI Extensions for RDMA, the protocol defined in this document 💜
iSER-assisted - A term generally used to describe iSCSI operating when the iSER functionality is also enabled below the iSCSI layer for the specific iSCSI/iSER connection in question 🪄
iSER-IRD - This variable represents the maximum number of incoming outstanding RDMA Read Requests that the iSER layer at the initiator declares on a particular RCaP Stream 📣
iSER-ORD - This variable represents the maximum number of outstanding RDMA Read Requests that the iSER layer can initiate on a particular RCaP Stream. This variable is maintained only by the iSER layer at the target 🎯
iSER Layer - The layer that implements the iSCSI Extensions for RDMA (iSER) protocol 🌟
iWARP - A suite of wire protocols made up of [RDMAP], [DDP], and [MPA] when layered above [TCP]. [RDMAP] and [DDP] may be layered above SCTP or other transport protocols 🕸️
Local Mapping - A task state record maintained by the iSER layer that associates the Initiator Task Tag with the local STag(s). The specifics of the record structure are implementation dependent 🗺️
Local Peer - The implementation of the RDMA-Capable Protocol on the local end of the connection. This term is used to refer to the local entity when describing protocol exchanges or other interactions between two Nodes 🏠
Node - A computing device attached to one or more links of a network. In this context, a Node doesn't refer to a specific application or protocol instance running on the computer. A Node may consist of one or more RDMA-Capable Controllers installed in a host computer 🖥️
Operational Primitive - An Operational Primitive is an abstract functional interface procedure that asks another layer to do a specific action on the requestor's behalf, or tells the other layer about some event. The Datamover Interface between an iSCSI layer and a Datamover layer within an iSCSI end node uses a set of Operational Primitives to define the functional interface between the two layers. Note that not every invocation of an Operational Primitive may get a response from the requested layer. A full discussion of the Operational Primitive types and request-response semantics available to iSCSI and iSER can be found in [DA] 🧠
Outbound RDMA Read Queue Depth (ORD) - The maximum number of outstanding RDMA Read Requests that the RDMA-Capable Controller can initiate on a particular RCaP Stream at the Data Sink. For some RDMA-Capable Protocol layers, this term may be known by a different name. For example, for InfiniBand, the equivalent of ORD is the Initiator Depth 📤
Phase-Collapse - Refers to the optimization in iSCSI where the SCSI status is transferred along with the final SCSI Data-in PDU from a target. See Section 3.2 in [RFC3720] 🌼
RCaP Message 💌 - One or more packets of the network layer that make up a single RDMA Operation, or a part of an RDMA Read Operation, of the RDMA-Capable Protocol. For iWARP, an RCaP Message is known as an RDMAP Message.
RCaP Stream 🦋 - A single bidirectional association between the peer RDMA-Capable Protocol layers on two Nodes, running over a single transport-level stream. For iWARP, an RCaP Stream is known as an RDMAP Stream, and the association is created when the connection moves into iSER-assisted mode after a successful Login Phase where iSER support gets negotiated. ✨
RDMA-Capable Protocol (RCaP) 👑 - The protocol or protocol suite that gives you reliable RDMA transport functionality, like iWARP, InfiniBand, etc.
RDMA-Capable Controller 🔥 - A network I/O adapter or embedded controller with RDMA functionality. For iWARP this could be an RNIC, and for InfiniBand it could be an HCA (Host Channel Adapter) or a TCA (Target Channel Adapter).
RDMA-enabled Network Interface Controller (RNIC) 🌸 - A network I/O adapter or embedded controller with iWARP functionality.
RDMA Operation 🧚 - A sequence of RCaP Messages, control Messages included, that moves data from a Data Source to a Data Sink. These RDMA Operations are defined: RDMA Write Operation, RDMA Read Operation, Send Operation, Send with Invalidate Operation, Send with Solicited Event Operation, Send with Solicited Event and Invalidate Operation, and Terminate Operation.
RDMA Protocol (RDMAP) 💜 - A wire protocol that supports RDMA Operations to transfer ULP data between a Local Peer and the Remote Peer, as described in [RDMAP].
RDMA Read Operation 🍓 - An RDMA Operation the Data Sink uses to pull the contents of a Data Source buffer from the Remote Peer into a Data Sink buffer at the Local Peer. An RDMA Read Operation is made of a single RDMA Read Request Message and a single RDMA Read Response Message.
RDMA Read Request - An RCaP Message the Data Sink uses to ask the Data Source to transfer the contents of a buffer. The RDMA Read Request Message describes both the Data Source and the Data Sink buffers.
RDMA Read Response - An RCaP Message the Data Source uses to send a buffer's contents to the Data Sink, replying to an RDMA Read Request. The RDMA Read Response Message only describes the Data Sink buffer.
RDMA Write Operation 🦋 - An RDMA Operation the Data Source uses to transfer the contents of a Data Source buffer from the Local Peer into a Data Sink buffer at the Remote Peer. The RDMA Write Message only describes the Data Sink buffer.
Remote Direct Memory Access (RDMA) ⭐ - A way of accessing memory on a remote system where the local system specifies the remote location of the data to be transferred. Using an RDMA-Capable Controller in the remote system lets that access happen without interrupting the processing of the CPU(s) on the system.
Remote Mapping 🎀 - A task state record kept by the iSER layer that associates the Initiator Task Tag with the Advertised STag(s). The specifics of the record structure are implementation dependent.
Remote Peer 🫶 - The implementation of the RDMA-Capable Protocol on the opposite end of the connection. It's used to refer to the remote entity when describing protocol exchanges or other interactions between two Nodes.
SCSI Layer 💅 - This layer builds and receives SCSI CDBs (Command Descriptor Blocks), and sends and receives them, along with the remaining command execute [SAM2] parameters, to and from the iSCSI layer.
Send 🌈 - An RDMA Operation that transfers the contents of a Buffer from the Local Peer to a Buffer at the Remote Peer.
Send Message Type - A Send Message, Send with Invalidate Message, Send with Solicited Event Message, or Send with Solicited Event and Invalidate Message.
SendInvSE Message - A Send with Solicited Event and Invalidate Message.
SendSE Message - A Send with Solicited Event Message.
Sequence Number (SN) - DataSN for a SCSI Data-in PDU and R2TSN for an R2T PDU. The semantics for both types of sequence numbers are as defined in [RFC3720].
Session, iSCSI Session 👑 - The group of connections that link an initiator SCSI port with a target SCSI port form an iSCSI session (equivalent to a SCSI I-T nexus). Connections can be added to and removed from a session even while the I-T nexus is intact. Across all connections within a session, an initiator sees one and the same target.
Solicited Event (SE) 🔥 - A facility by which an RDMA Operation sender can cause an Event to be generated at the recipient, if the recipient is configured to generate such an Event, when a Send with Solicited Event or Send with Solicited Event and Invalidate Message is received.
Steering Tag (STag) 🦋 - An identifier of a Tagged Buffer on a Node (Local or Remote), as defined in [RDMAP] and [DDP]. For other RDMA-Capable Protocols, the Steering Tag may go by different names, but it will be referred to here as STags. For example, in InfiniBand a Remote STag is known as an R-Key and a local STag is known as an L-Key, and both count as STags.
Tagged Buffer 🌸 - A buffer that is explicitly Advertised to the iSER layer at the remote node through the exchange of an STag, Tagged Offset, and length.
Tagged Offset (TO) - The offset within a Tagged Buffer.
Traditional iSCSI 🍓 - The iSCSI protocol as defined in [RFC3720], meaning without the iSER enhancements.
Untagged Buffer 🙈 - A buffer that is not explicitly Advertised to the iSER layer at the remote node.
2.2. Acronyms
Acronym Definition
--------------------------------------------------------------
AHS Additional Header Segment
BHS Basic Header Segment
CO Connection Only
CRC Cyclic Redundancy Check
DDP Direct Data Placement Protocol
DI Datamover Interface
HCA Host Channel Adapter
IANA Internet Assigned Numbers Authority
IB InfiniBand
IETF Internet Engineering Task Force
I/O Input - Output
IO Initialize Only
IP Internet Protocol
IPoIB IP over InfiniBand
IPsec Internet Protocol Security
iSER iSCSI Extensions for RDMA
ITT Initiator Task Tag
LO Leading Only
MPA Marker PDU Aligned Framing for TCP
NOP No Operation
NSG Next Stage (during the iSCSI Login Phase)
OS Operating System
PDU Protocol Data Unit
R2T Ready To Transfer
R2TSN Ready To Transfer Sequence Number
RDMA Remote Direct Memory Access
RDMAP Remote Direct Memory Access Protocol
RFC Request For Comments
RNIC RDMA-enabled Network Interface Controller
SAM2 SCSI Architecture Model - 2
SCSI Small Computer Systems Interface
SNACK Selective Negative Acknowledgment - also
Sequence Number Acknowledgement for data
STag Steering Tag
SW Session Wide
TCA Target Channel Adapter
TCP Transmission Control Protocol
TMF Task Management Function
TTT Target Transfer Tag
TO Tagged Offset
ULP Upper Level Protocol2.3. Conventions
Heads up, bestie: the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 🧠
3. Upper Layer Interface Requirements
Okay, this section is where we get into the upper layer interface requirements, shown as an abstract model of the required interactions between the iSCSI layer and the iSER layer. ✨ The abstract model comes from the architectural model in [DA], and [DA] also gives a functional overview of how the iSCSI layer and the Datamover layer are meant to interact under the Datamover Architecture.
The interface requirements are specified using Operational Primitives. An Operational Primitive is an abstract functional interface procedure between the iSCSI layer and the iSER layer. It either asks one layer to do something on behalf of the other, or tells the other layer that something happened. Whenever an Operational Primitive is invoked, the Connection_Handle qualifier identifies a particular iSCSI connection. For some Operational Primitives, a Data_Descriptor identifies the iSCSI/SCSI data buffer tied to the requested or completed operation.
The abstract model and the Operational Primitives in this section make the iSER protocol easier to describe. Later in the iSER specification, the compliance statements about these Operational Primitives only apply to the required interactions between the iSCSI layer and the iSER layer. Those later compliance statements only require functional equivalence between implementations, and they don't put any requirements on how the interface between the iSCSI layer and the iSER layer is actually implemented. 💅
Each Operational Primitive is invoked with a set of qualifiers that describe the information context for the action being requested. The qualifiers are required, but how they're passed (synchronously with the invocation, retrieved from task context, retrieved from shared memory, and so on) is implementation dependent. 🌈
3.1. Operational Primitives Offered by iSER
The iSER protocol layer MUST support the following Operational Primitives to be used by the iSCSI protocol layer. 🫶
3.1.1. Send_Control
Input qualifiers: Connection_Handle, BHS and AHS (if any) of the
iSCSI PDU, PDU-specific qualifiers
Return results: Not specifiedThe iSCSI layers at both the initiator and the target use this to request the outbound transfer of an iSCSI control-type PDU (see Section 7.2). Qualifiers that only apply to a particular control-type PDU are called PDU-specific qualifiers, e.g., ImmediateDataSize for a SCSI write command. For details on PDU-specific qualifiers, see Section 7.3. The iSCSI layer can only invoke the Send_Control Operational Primitive when the connection is in iSER-assisted mode. 🦋
3.1.2. Put_Data
Input qualifiers: Connection_Handle, content of a SCSI Data-in
PDU header, Data_Descriptor, Notify_Enable
Return results: Not specifiedThe iSCSI layer at the target uses this to request the outbound transfer of data for a SCSI Data-in PDU, taking the data from the buffer identified by the Data_Descriptor qualifier. The iSCSI layer can only invoke the Put_Data Operational Primitive when the connection is in iSER-assisted mode. 🍓
The Notify_Enable qualifier tells the iSER layer whether or not it should generate an eventual local completion notification to the iSCSI layer. See Section 3.2.2 on Data_Completion_Notify for details. 💖
3.1.3. Get_Data 💖
Okay so Get_Data is how the iSCSI layer at the target says "pls go grab the solicited data this R2T PDU asked for and drop it into the buffer I named" 🎀
Input qualifiers: Connection_Handle, content of an R2T PDU,
Data_Descriptor, Notify_Enable
Return results: Not specifiedThe iSCSI layer can only call Get_Data when the connection is in iSER-assisted mode, so no skipping ahead, bestie 🙈 The buffer it writes into is the one identified by the Data_Descriptor qualifier.
The Notify_Enable qualifier tells the iSER layer whether it should send the eventual local completion notification back up to iSCSI. Check out Section 3.2.2 on Data_Completion_Notify for the full tea ☕
3.1.4. Allocate_Connection_Resources ✨
Input qualifiers: Connection_Handle, Resource_Descriptor
(optional)
Return results: StatusThis one is used by the iSCSI layers at both the initiator and the target to ask for every connection resource needed to support RCaP on an operational iSCSI/iSER connection 🧚 The iSCSI layer may (optionally!) use the Resource_Descriptor qualifier to spell out the implementation-specific resource requirements for that connection.
The result comes back as either Status=success, meaning the invocation worked, or Status=failure, meaning it didn't 💅 If you call it with a Connection_Handle that already succeeded earlier, the iSER layer just ignores the request and gives you Status=success again. Iconic efficiency 👑
One more rule: only one Allocate_Connection_Resources invocation can be outstanding for a given Connection_Handle at any time. No double-booking, periodt 💜
3.1.5. Deallocate_Connection_Resources 🦋
Input qualifiers: Connection_Handle
Return results: Not specifiedBoth the initiator and the target use this to ask for all the connection resources that were allocated earlier by a successful Allocate_Connection_Resources invocation to be deallocated. Clean up after yourself, bestie 🧹✨
3.1.6. Enable_Datamover 🌸
Input qualifiers: Connection_Handle,
Transport_Connection_Descriptor, Final Login_Response_PDU
(optional)
Return results: Not specifiedThe initiator and the target both use this to request that a particular iSCSI connection switch over to iSER-assisted mode 🦋 The Transport_Connection_Descriptor qualifier is how you identify the exact connection tied to the Connection_Handle.
Same as before, the iSCSI layer can only call Enable_Datamover when there's a matching resource allocation already done. No allocation, no glow-up 🙅♀️
The Final_Login_Response_PDU qualifier only applies to the target. It holds the final Login Response PDU that wraps up the iSCSI Login Phase. If the transport is TCP, that final Login Response PDU must be sent as a byte stream, the way the iSCSI layer at the initiator expects it.
When this qualifier is used, the iSER layer at the target MUST transmit this final Login Response PDU before transitioning to iSER-assisted mode. This ordering is non-negotiable, bestie 🚨
3.1.7. Connection_Terminate 💀
Input qualifiers: Connection_Handle
Return results: Not specifiedThe initiator and the target use this to request that a specified iSCSI/iSER connection gets terminated and that all its associated connection and task resources get freed 🫠 Once this returns to the iSCSI layer, the iSCSI layer may assume full ownership of all iSCSI-level resources tied to the connection, for example I/O Buffers. Mine now, thank you 💅
3.1.8. Notice_Key_Values 🍓
Input qualifiers: Connection_Handle, number of keys, list of
Key-Value pairs
Return results: Not specifiedThe initiator and the target use this to ask the iSER layer to take note of the Key-Value pairs that the iSCSI peers negotiated for the connection. The iSER layer is basically the friend who remembers everything 🧠💜
3.1.9. Deallocate_Task_Resources 🎀
Input qualifiers: Connection_Handle, ITT
Return results: Not specifiedBoth the initiator and the target use this to request the deallocation of all RCaP-specific resources the iSER layer allocated for the task identified by the ITT qualifier 🌈
Here's the vibe: the iSER layer might need a certain number of RCaP-specific resources tied to the ITT for every new iSCSI task. In the normal flow, those task-level resources are allocated automatically when a task starts and deallocated automatically when it finishes, so you don't have to think about it 🧚
Exception scenarios are where this primitive comes in. If a task doesn't end with a SCSI Response PDU, the iSER layer needs to be told about each individual task termination so it can manage its task-level resources properly. That's the whole job of this primitive. It isn't needed when a SCSI Response PDU normally wraps up the task.
Note that RCaP-specific task resources get deallocated by the iSER layer when a SCSI Response PDU normally concludes a task, even if the SCSI status was not success. Even failed tasks get tidied up, bestie 🧹
3.2. Operational Primitives Used by iSER 🫶
The iSER layer MUST use the following Operational Primitives offered by the iSCSI protocol layer when the connection is in iSER-assisted mode. These are the iSCSI-side tools iSER leans on, no cap 💜
3.2.1. Control_Notify 🦋
Input qualifiers: Connection_Handle, an iSCSI control-type PDU
Return results: Not specifiedThe iSER layers at the initiator and the target use this to tell the iSCSI layer that an inbound iSCSI control-type PDU is available 📬 A PDU counts as "available" to the iSCSI layer once the iSER layer has notified it that the inbound PDU was received, along with an implementation-specific indication of where that PDU is sitting.
3.2.2. Data_Completion_Notify 🌸
Input qualifiers: Connection_Handle, ITT, SN
Return results: Not specifiedThis is used by the iSER layer to tell the iSCSI layer that an outbound data transfer finished. It only happens if the Put_Data invocation (see Section 3.1.2) was qualified with Notify_Enable set. Here, SN refers to the DataSN associated with the SCSI Data-in PDU.
The same primitive also covers the other direction 🔄 The iSER layer uses it to say that an inbound data transfer finished, but only if the Get_Data invocation (see Section 3.1.3) was qualified with Notify_Enable set. In that case, SN refers to the R2TSN associated with the R2T PDU.
3.2.3. Data_ACK_Notify 💌
Input qualifier: Connection_Handle, ITT, DataSN
Return results: Not specifiedThe iSER layer at the target uses this to tell the iSCSI layer that a data acknowledgement arrived. That acknowledgement is defined in [RFC3720], and the iSCSI layer asked for it earlier through a Put_Data invocation where the A-bit in the SCSI Data-in PDU was set to 1 🎀 See Section 7.3.5 for more.
DataSN refers to the expected DataSN of the next SCSI Data-in PDU. That's the one that comes right after the SCSI Data-in PDU with the A-bit set that this notification is about. The semantics are the ones defined in [RFC3720], so go check there too, bestie 📖
3.2.4. Connection_Terminate_Notify 💔
Input qualifiers: Connection_Handle
Return results: Not specifiedThe iSER layers at the initiator and the target use this to tell the iSCSI layer about an unsolicited termination or failure of an iSCSI/iSER connection 😭 The iSER layer MUST deallocate the connection and task resources associated with the terminated connection before invoking this primitive. Order matters!
Also, this primitive does not get invoked when the local iSCSI layer asked for the termination itself. It's only for surprise endings 🙈
3.3. iSCSI Protocol Usage Requirements 🧚
To run in iSER-assisted mode, the iSCSI layers at both the initiator and the target MUST negotiate the RDMAExtensions key (see Section 6.3) to "Yes" on the leading connection. If RDMAExtensions doesn't negotiate to "Yes", then iSER-assisted mode MUST NOT be used 🚫
Things get a bit tricky here. If RDMAExtensions is negotiated to "Yes" but the Allocate_Connection_Resources invocation to the iSER layer fails, the iSCSI layer MUST fail the iSCSI Login process or terminate the connection, whichever fits the situation. See Section 10.1.3.1 for the details 😬
If RDMAExtensions is negotiated to "Yes", the iSCSI layer MUST also satisfy the following protocol usage requirements from the iSER protocol 💅
- The iSCSI layer at the initiator MUST set ExpDataSN to 0 in Task Management Function Requests for Task Allegiance Reassignment for read/bidirectional commands, so as to cause the target to send all unacknowledged read data 🔁
- The iSCSI layer at the target MUST always return the SCSI status in a separate SCSI Response PDU for read commands. In other words, there MUST NOT be a "phase collapse" in concluding a SCSI read command. No squishing the steps together, bestie 🙅♀️
- The iSCSI layers at both the initiator and the target MUST support the keys as defined in Section 6 on Login/Text Operational Keys. If used as specified, these keys MUST NOT be answered with NotUnderstood, and the semantics as defined MUST be followed for each iSER-assisted connection 🔑
- The iSCSI layer at the initiator MUST NOT issue SNACKs for PDUs. Hard pass on SNACKs for the initiator, iykyk 🙅♀️💜
4. Lower Layer Interface Requirements ✨
4.1. Interactions with the RCaP Layer 🦋
okay so the iSER layer sits on top of an RCaP layer (peep Figure 1 🌸), and there are some key features we just assume any RCaP layer already has 💅
- The RCaP layer supports all the basic RDMA operations: RDMA Write Operation, RDMA Read Operation, Send Operation, Send with Invalidate Operation, Send with Solicited Event Operation, Send with Solicited Event and Invalidate Operation, and Terminate Operation. 💖
- It gives you reliable, in-order message delivery AND direct data placement. no cap 🔥
- When the iSER layer starts an RDMA Read Operation right after an RDMA Write Operation on one RCaP Stream, the RDMA Read Response Message processing on the remote node only kicks off after the preceding RDMA Write Message payload has been placed in that node's memory. ✨
- On the Data Source side, RCaP wraps one iSER Message into one RCaP Message. On the Data Sink side, it unwraps (decapsulates) that iSER Message before handing it to the iSER layer 🎀
- When iSER gives RCaP the STag to be remotely invalidated for a SendInvSE Message, RCaP uses that STag as the one that gets invalidated in the SendInvSE Message. 🧚
- RCaP uses the STag and Tagged Offset that iSER provides for the RDMA Write and RDMA Read Request Messages. 🍓
- When RCaP delivers the content of an RDMA Send Message Type up to iSER, it also tells iSER the length of that RDMA Send message. So iSER doesn't have to carry a length field in its header. iconic 👑
- When RCaP delivers a SendSE or SendInvSE Message to iSER, it notifies iSER using the mechanism provided on that interface. 🦋
- When RCaP delivers a SendInvSE Message to iSER, it passes along the value of the STag that was invalidated. 💜
- RCaP propagates all status and error indications up to iSER. 🫶
- For a byte stream transport like TCP, the RCaP implementation supports enabling RDMA mode after connection establishment and after the exchange of Login parameters in byte stream mode. For a message-delivery transport like [IB], the RCaP implementation supports the iSCSI layer using that messaging capability directly for the Login Phase after connection establishment, before iSER-assisted mode gets enabled. ✨
- Whenever the iSER layer terminates the RCaP Stream, the RCaP layer terminates the associated connection. the way that's a whole cascade 💀
4.2. Interactions with the Transport Layer 🌈
so iSER doesn't set up the transport layer connection itself (we're talking TCP, or [IB]) 🙈 During connection setup, the iSCSI layer is the one in charge. If the login succeeds, the iSCSI layer invokes the Enable_Datamover Operational Primitive to ask the iSER layer to switch that iSCSI connection into iSER-assisted mode. See Section 5.1 on iSCSI/iSER connection setup. After that switch, the RCaP layer and the underlying transport layer are responsible for keeping the connection alive and reporting any connection failures back up to the iSER layer. 🫶
5. Connection Setup and Termination 🎀
5.1. iSCSI/iSER Connection Setup 🧚
during connection setup, the iSCSI layer at the initiator is responsible for establishing a connection with the target. Once it's up, the iSCSI layers at the initiator and target enter the Login Phase using the same rules as in [RFC3720]. The connection transitions to iSER-assisted mode when it moves into the iSCSI Full Feature Phase, which happens after a successful login negotiation where iSER-assisted mode is negotiated AND the connection resources needed to support RCaP have been allocated at both the initiator and the target. The same connection MUST be used for both the iSCSI Login Phase and the later iSER-assisted Full Feature Phase. bestie, this is non-negotiable 🔒
iSER-assisted mode MUST be enabled only if it's negotiated on the leading connection during the LoginOperationalNegotiation stage of the iSCSI Login Phase. It gets negotiated using RDMAExtensions=<boolean-value>. Both the initiator and the target MUST exchange the RDMAExtensions key with the value set to "Yes" to turn it on. If both of them fail to negotiate RDMAExtensions to "Yes", then the connection MUST continue with the login semantics defined in [RFC3720]. If RDMAExtensions doesn't get negotiated to Yes, then for some RCaP implementations (like [IB]) the connection may need to be re-established in TCP capable mode. (For InfiniBand, that means an [IPoIB] type connection 🦋)
iSER-assisted mode is defined for a Normal session only, and the RDMAExtensions key MUST NOT be negotiated for a Discovery session. Discovery sessions are always conducted over the transport layer as described in [RFC3720]. not me making the rules look so clean 🙈
An iSER enabled node is not required to initiate the RDMAExtensions key exchange if its preference is for the Traditional iSCSI mode. But if it does offer the RDMAExtensions key, it MUST be sent in the first available Login Response or Login Request PDU in the LoginOperationalNegotiation stage. This is because the value of some login parameters might depend on whether iSER-assisted mode is enabled. ✨
iSER-assisted mode is a session-wide attribute, so this is the vibe 💅 If both the initiator and the target negotiate RDMAExtensions="Yes" on the leading connection of a session, then all subsequent connections of that same session MUST enable iSER-assisted mode without having to exchange an RDMAExtensions key during the iSCSI Login Phase.
Conversely, if both the initiator and the target fail to negotiate RDMAExtensions to "Yes" on the leading connection, then the RDMAExtensions key MUST NOT be negotiated further on any additional subsequent connection of the session. 💀
When RDMAExtensions is negotiated to "Yes", the HeaderDigest and the DataDigest keys MUST be negotiated to "None" on all iSCSI/iSER connections participating in that iSCSI session. Here's why, bestie: for an iSCSI/iSER connection, RCaP is responsible for providing error detection that's at least as good as a 32-bit CRC for all iSER Messages. Plus, all SCSI Read data gets sent using RDMA Write Messages instead of SCSI Data-in PDUs, and all solicited SCSI write data gets sent using RDMA Read Response Messages instead of SCSI Data-out PDUs. HeaderDigest and DataDigest apply to iSCSI PDUs, so they would not be appropriate for the RDMA Read and RDMA Write operations used with iSER. main character energy 👑
5.1.1. Initiator Behavior 💖
If the outcome of the iSCSI negotiation is to enable iSER-assisted mode, then on the initiator side, before sending the Login Request with the T (Transit) bit set to 1 and the NSG (Next Stage) field set to FullFeaturePhase, the iSCSI layer MUST request that the iSER layer allocate the connection resources needed to support RCaP by invoking the Allocate_Connection_Resources Operational Primitive. Which resources those are is up to the implementation, and that's outside the scope of this spec 🌸 The iSCSI layer may invoke the Notice_Key_Values Operational Primitive before invoking Allocate_Connection_Resources, so the iSER layer takes note of the negotiated values of the iSCSI keys for the connection. Which specific keys get passed as input qualifiers is implementation dependent. They may include, but are not limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc.
To minimize the potential for a denial-of service attack, the iSCSI layer MUST NOT request that the iSER layer allocate the connection resources needed to support RCaP until the iSCSI layer is far enough along in the iSCSI Login Phase that it's reasonably certain the peer side isn't an attacker. 🔒 In particular, if the Login Phase includes a SecurityNegotiation stage, the iSCSI layer MUST defer the connection resource allocation (i.e., invoking the Allocate_Connection_Resources Operational Primitive) to the LoginOperationalNegotiation stage [RFC3720], so the allocation happens after the authentication phase is completed.
Among the connection resources allocated at the initiator is the Inbound RDMA Read Queue Depth (IRD). As Section 9.5.1 describes, R2Ts get transformed by the target into RDMA Read operations. IRD limits the maximum number of simultaneously incoming outstanding RDMA Read Requests per an RCaP Stream, going from the target to the initiator. The required value of IRD is outside the scope of the iSER specification. The iSER layer at the initiator MUST set IRD to 1 or higher if R2Ts are to be used in the connection. However, the iSER layer at the initiator MAY set IRD to 0 based on implementation configuration, which indicates that no R2Ts will be used on that connection. Initially, the iSER-IRD value at the initiator SHOULD be set to the IRD value at the initiator, and it MUST NOT be more than the IRD value. ✨
On the other hand, the Outbound RDMA Read Queue Depth (ORD) MAY be set to 0, since the iSER layer at the initiator doesn't issue RDMA Read Requests to the target. 💜
If the iSER layer at the initiator can't allocate the requested connection resources locally, that results in a login failure. How that gets handled is described in Section 10.1.3.1. 🥲
If the iSER layer at the initiator is successful in allocating the connection resources needed to support RCaP, the following events MUST occur in the specified sequence:
- The iSER layer MUST return a success status to the iSCSI layer in response to the
Allocate_Connection_ResourcesOperational Primitive. - After the target returns the Login Response with the
Tbit set to 1 and theNSGfield set toFullFeaturePhase, and a status class of 0 (Success), the iSCSI layer MUST request that the iSER layer transition to iSER-assisted mode by invoking theEnable_DatamoverOperational Primitive with the following qualifiers. (See Section 10.1.4.6 for the case when the status class is not Success.):- a.
Connection_Handlethat identifies the iSCSI connection. - b.
Transport_Connection_Descriptorthat identifies the specific transport connection associated with theConnection_Handle.
- a.
- If necessary, the iSER layer should enable RCaP and transition the connection to iSER-assisted mode. When the RCaP is iWARP, then this step MUST be done. Not all RCaPs may need it, depending on the RCaP Stream start-up state. 🌈
- The iSER layer MUST send the iSER Hello Message as the first iSER Message. See Section 5.1.3 on iSER Hello Exchange. 🦋
5.1.2. Target Behavior ✨
Okay, now let's talk about the target side 🎯. If the iSCSI negotiation lands on iSER-assisted mode, the target has to get its resources ready before it sends the Login Response with the T (Transit) bit set to 1 and the NSG (Next Stage) field set to FullFeaturePhase 💅. So the iSCSI layer MUST ask the iSER layer to allocate what it needs to support RCaP by invoking the Allocate_Connection_Resources Operational Primitive 🧚. What resources those are is up to the implementation, and this spec doesn't cover them 🌸. The iSCSI layer MAY also call Notice_Key_Values right before that, so the iSER layer knows the negotiated iSCSI key values for the connection 🔑. Which keys get passed is implementation dependent, and it could include things like MaxOutstandingR2T, ErrorRecoveryLevel, and so on 💖.
Here's the security part, and it's serious 🙈. To avoid a denial-of-service attack, the iSCSI layer MUST NOT ask for RCaP connection resources until it's far enough into the iSCSI Login Phase that it's pretty sure the peer isn't an attacker 🛡️. If the Login Phase has a SecurityNegotiation stage, the iSCSI layer MUST wait and do the allocation (invoking Allocate_Connection_Resources) during the LoginOperationalNegotiation stage [RFC3720], so the allocation happens after authentication is done 🔐.
Among the resources the target allocates is the Outbound RDMA Read Queue Depth (ORD) 🏹. As Section 9.5.1 explains, the target turns R2Ts into RDMA Read operations 🔄. The ORD caps how many RDMA Read Requests per RCaP Stream can be outstanding at once, going from the target to the initiator 📊. To start with, the iSER-ORD value at the target SHOULD be set to the ORD value at the target 🌈.
Flip side: the IRD at the target MAY be set to 0, since the iSER layer at the target doesn't expect the initiator to issue RDMA Read Requests 🦋.
If the target can't allocate the requested connection resources locally, that causes a login failure 😭. Section 10.1.3.1 covers how that gets handled.
If the iSER layer at the target manages to allocate what it needs for RCaP, these events MUST happen in exactly this order ⭐:
- The iSER layer MUST return a success status to the iSCSI layer in response to the
Allocate_Connection_ResourcesOperational Primitive. - The iSCSI layer MUST ask the iSER layer to switch to iSER-assisted mode by invoking the
Enable_DatamoverOperational Primitive with these qualifiers:- a.
Connection_Handle, which identifies the iSCSI connection. - b.
Transport_Connection_Descriptor, which identifies the specific transport connection tied to theConnection_Handle. - c. The final transport layer (e.g., TCP) message carrying the Login Response with the T bit set to 1 and the NSG field set to
FullFeaturePhase.
- a.
- The iSER layer MUST send the final Login Response PDU in the native transport mode to wrap up the iSCSI Login Phase. If the underlying transport is TCP, then the iSER layer MUST send the final Login Response PDU in byte stream mode 🌊.
- After sending the final Login Response PDU, the iSER layer should enable RCaP if it's needed and move the connection into iSER-assisted mode. When the RCaP is iWARP, this step MUST be done. Not every RCaP needs it, since that depends on the RCaP Stream start-up state 🤷♀️.
- After receiving the iSER Hello Message from the initiator, the iSER layer MUST respond with the iSER HelloReply Message, which gets sent as the first iSER Message 👋. Check Section 5.1.3 on the iSER Hello Exchange for the full scoop.
Note: In the sequence above, bullets 3 and 4 MUST be carried out atomically for iWARP connections. Skipping that atomicity may cause race conditions 💀.
5.1.3. iSER Hello Exchange 🫶
Once the connection switches into iSER-assisted mode, the very first iSER Message the initiator's iSER layer sends to the target MUST be the iSER Hello Message 🌸. That's how the initiator's iSER layer declares its iSER parameters to the target. Check Section 9.3 on iSER Header Format for the iSER Hello Message.
Then the target's iSER layer MUST reply with the iSER HelloReply Message as the very first iSER Message it sends 💜. That's how the target's iSER layer declares its iSER parameters to the initiator. Check Section 9.4 on iSER Header Format for the iSER HelloReply Message.
In the iSER Hello Message, the initiator's iSER layer declares the iSER-IRD value to the target 🦋.
When the target gets the iSER Hello Message, its iSER layer MUST set the iSER-ORD value to the smaller of its own ORD value and the iSER-IRD value the initiator declared 📉. The target's iSER layer MAY also lower its ORD value to match the iSER-ORD value, if iSER-ORD is smaller, so it can free up resources it isn't using 🧹.
In the iSER HelloReply Message, the target's iSER layer declares the iSER-ORD value to the initiator 📣.
When the initiator gets the iSER HelloReply Message, its iSER layer MAY lower its IRD value to match the iSER-ORD value, if the target's declared iSER-ORD is smaller than the iSER-IRD the initiator declared. That frees up unused resources 🌈.
It's an iSER level negotiation failure if the iSER parameters the initiator declared in the iSER Hello Message aren't acceptable to the target 🚫. That includes these cases:
- The initiator-declared iSER-IRD value is greater than 0 and the target-declared iSER-ORD value is 0.
- The initiator-supported and the target-supported iSER protocol versions do not overlap.
See Section 10.1.3.2 for how that error gets handled 🎀.
5.2. iSCSI/iSER Connection Termination 👋
5.2.1. Normal Connection Termination at the Initiator 🌙
The iSCSI layer at the initiator ends an iSCSI/iSER connection normally by invoking the Send_Control Operational Primitive qualified with the Logout Request PDU 🚪. The iSER layer at the initiator MUST use a SendSE Message to send the Logout Request PDU to the target. Once the initiator's iSER layer gets the SendSE Message carrying the Logout Response PDU from the target, it MUST notify the iSCSI layer by invoking the Control_Notify Operational Primitive qualified with the Logout Response PDU ✅.
After the iSCSI logout is done, the iSCSI layer at the target is in charge of closing the iSCSI/iSER connection, as Section 5.2.2 describes 🧚♀️. After the RCaP layer at the initiator reports the connection closed, the initiator's iSER layer MUST deallocate all connection and task resources (if any) tied to the connection. It also MUST invalidate the Local Mapping(s) (if any) that link the ITT(s) used on that connection to the local STag(s). It does all of that before notifying the iSCSI layer by invoking the Connection_Terminate_Notify Operational Primitive 💖.
5.2.2. Normal Connection Termination at the Target 🎬
When the iSER layer at the target gets the SendSE Message carrying the Logout Request PDU, it MUST notify the iSCSI layer at the target by invoking the Control_Notify Operational Primitive qualified with the Logout Request PDU 🔔. The iSCSI layer finishes the logout by invoking the Send_Control Operational Primitive qualified with the Logout Response PDU ✨. The iSER layer at the target MUST use a SendSE Message to send the Logout Response PDU to the initiator.
After the iSCSI logout is done, the iSCSI layer at the target MUST ask the iSER layer at the target to terminate the RCaP Stream by invoking the Connection_Terminate Operational Primitive 🛑.
As part of termination, the RCaP layer MUST close the connection 🔒. When the RCaP layer tells the iSER layer that the RCaP Stream and its connection are terminated, the iSER layer MUST deallocate all connection and task resources (if any) tied to the connection. It also MUST invalidate the Local and Remote Mapping(s) (if any) that link the ITT(s) used on that connection to the local STag(s) and the Advertised STag(s), respectively 🧹.
5.2.3. Termination without Logout Request/Response PDUs 😬
5.2.3.1. Connection Termination Initiated by the iSCSI Layer 🌪️
The Connection_Terminate Operational Primitive MAY be invoked by the iSCSI layer to ask the iSER layer to terminate the RCaP Stream, even if the Logout Request and Logout Response PDUs were never exchanged between the two iSCSI/iSER nodes 💫. As part of termination, the RCaP layer will close the connection. When the RCaP layer tells the iSER layer that the RCaP Stream and its connection are terminated, the iSER layer MUST do the following 👇.
If the iSCSI layer at the target invoked Connection_Terminate, then the iSER layer at the target MUST deallocate all connection and task resources (if any) tied to the connection. It MUST also invalidate the Local and Remote Mapping(s) (if any) that link the ITT(s) used on the connection to the local STag(s) and the Advertised STag(s), respectively 🎯.
If the iSCSI layer at the initiator invoked Connection_Terminate, then the iSER layer at the initiator MUST deallocate all connection and task resources (if any) tied to the connection. It MUST also invalidate the Local Mapping(s) (if any) that link the ITT(s) used on the connection to the local STag(s) 🫧.
5.2.3.2. Connection Termination Notification to the iSCSI Layer 📣
If the iSCSI/iSER connection ends without the iSCSI layer invoking Connection_Terminate, the iSER layer MUST notify the iSCSI layer that the connection has been terminated by invoking the Connection_Terminate_Notify Operational Primitive 🔔.
Before invoking Connection_Terminate_Notify, the iSER layer at the target MUST deallocate all connection and task resources (if any) tied to the connection. It MUST also invalidate the Local and Remote Mapping(s) (if any) that link the ITT(s) used on the connection to the local STag(s) and the Advertised STag(s), respectively 🧼.
Before invoking Connection_Terminate_Notify, the iSER layer at the initiator MUST deallocate all connection and task resources (if any) tied to the connection. It MUST also invalidate the Local Mapping(s) (if any) that link the ITT(s) used on the connection to the local STag(s) 🫧.
If the remote iSCSI/iSER node started closing the connection (e.g., by sending a TCP FIN or TCP RST), the iSER layer MUST notify the iSCSI layer after the RCaP layer reports the connection closed, by invoking the Connection_Terminate_Notify Operational Primitive 🌷.
Another example of termination without a preceding logout is when the iSCSI layer at the initiator does an implicit logout (connection reinstatement) 🦋.
6. Login/Text Operational Keys ✨
Okay so some iSCSI login/text operational keys have restricted usage in iSER, and there are extra keys that exist purely to make iSER work 🫶 All other keys from [RFC3720] that aren't covered in this section may be used on iSCSI/iSER connections with the same semantics 💅
6.1. HeaderDigest and DataDigest
Irrelevant when: RDMAExtensions=YesNegotiating RDMAExtensions=Yes for a session implies HeaderDigest=None and DataDigest=None for all connections in that session 🔒 It overrides both the default and any explicit setting, so no arguing with it, bestie 🙈
6.2. MaxRecvDataSegmentLength
For an iSCSI connection in a session where RDMAExtensions=Yes was negotiated on the leading connection, you don't need to declare MaxRecvDataSegmentLength during the Login Phase ✨ Instead, the InitiatorRecvDataSegmentLength key (see Section 6.5) and the TargetRecvDataSegmentLength key (see Section 6.4) get negotiated. The local and remote MaxRecvDataSegmentLength values are derived from those two keys, even if MaxRecvDataSegmentLength was declared during the Login Phase 🦋
In the Full Feature Phase, the initiator MUST treat its local MaxRecvDataSegmentLength (the one it would have declared to the target) as having the value of InitiatorRecvDataSegmentLength. It also has to treat the remote MaxRecvDataSegmentLength (the one the target would have declared) as having the value of TargetRecvDataSegmentLength. Same energy on the other side: the target MUST treat its local MaxRecvDataSegmentLength (the one it would have declared to the initiator) as having the value of TargetRecvDataSegmentLength, and the remote MaxRecvDataSegmentLength (the one the initiator would have declared) as having the value of InitiatorRecvDataSegmentLength 💜
The MaxRecvDataSegmentLength key only applies to iSCSI control-type PDUs 🎀
6.3. RDMAExtensions
Use: LO (leading only)
Senders: Initiator and Target
Scope: SW (session-wide)
RDMAExtensions=<boolean-value>
Irrelevant when: SessionType=Discovery
Default is No
Result function is ANDThe initiator and the target use this key to negotiate support for iSER-assisted mode 🔥 To turn iSER-assisted mode on, both the initiator and the target MUST exchange RDMAExtensions=Yes, no exceptions 👑
iSER-assisted mode MUST NOT be used if either the initiator or the target offers RDMAExtensions=No. 💀
An iSER-enabled node doesn't have to initiate the RDMAExtensions key exchange if it would rather operate in Traditional iSCSI mode, totally valid ✨ But if the RDMAExtensions key is going to be negotiated, an initiator MUST offer the key in the first Login Request PDU in the LoginOperationalNegotiation stage of the leading connection. A target MUST offer the key in the first Login Response PDU it's allowed to send for this purpose, meaning the first Login Response PDU issued after the first Login Request PDU with the C bit set to 0, again in the LoginOperationalNegotiation stage of the leading connection 🧚 When the offered key=value pair RDMAExtensions=yes comes in, an initiator MUST respond in the next Login Request PDU it's allowed to send, and a target MUST respond in the next Login Response PDU it's allowed to send 🌸
Negotiating RDMAExtensions first lets a node negotiate the optimal value for other keys, which is lowkey genius 🧠 Certain iSCSI keys such as MaxBurstLength, MaxOutstandingR2T, ErrorRecoveryLevel, InitialR2T, ImmediateData, etc., may be negotiated differently depending on whether the connection is in Traditional iSCSI mode or iSER-assisted mode 🌈
6.4. TargetRecvDataSegmentLength
Use: IO (Initialize only)
Senders: Initiator and Target
Scope: CO (connection-only)
Irrelevant when: RDMAExtensions=No
TargetRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>
Default is 8192 bytes
Result function is minimumThis key only matters for the iSCSI connection of an iSCSI session if RDMAExtensions=Yes was negotiated on the leading connection of that session 🍓 The initiator and target use it to negotiate the maximum size of the data segment an initiator can send to the target in an iSCSI control-type PDU during the Full Feature Phase 🦋
For SCSI Command PDUs and SCSI Data-out PDUs containing non-immediate unsolicited data that the initiator sends, the initiator MUST send all non-Final PDUs with a data segment size of exactly TargetRecvDataSegmentLength, whenever those PDUs make up a data sequence that's larger than TargetRecvDataSegmentLength 🫶
6.5. InitiatorRecvDataSegmentLength
Use: IO (Initialize only)
Senders: Initiator and Target
Scope: CO (connection-only)
Irrelevant when: RDMAExtensions=No
InitiatorRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>
Default is 8192 bytes
Result function is minimumSame vibe as the key above, just flipped 🔄 This one only matters for the iSCSI connection of a session where RDMAExtensions=Yes was negotiated on the leading connection 💖 The initiator and target use it to negotiate the maximum size of the data segment a target can send to the initiator in an iSCSI control-type PDU during the Full Feature Phase ✨
6.6. OFMarker and IFMarker
Irrelevant when: RDMAExtensions=YesNegotiating RDMAExtensions=Yes for a session implies OFMarker=No and IFMarker=No for all connections in that session 🙅♀️ That overrides both the default and any explicit setting, periodt 💅
6.7. MaxOutstandingUnexpectedPDUs
Use: LO (leading only), Declarative
Senders: Initiator and Target
Scope: SW (session-wide)
Irrelevant when: RDMAExtensions=No
MaxOutstandingUnexpectedPDUs=<numerical-value-from-2-to-(2**32-1) | 0>
Default is 0The initiator and the target use this key to declare the maximum number of outstanding "unexpected" iSCSI control-type PDUs they can receive during the Full Feature Phase 🧚 It exists so the receiving side can figure out how much buffer space it needs beyond the normal flow control that iSCSI already has 🎀 An initiator or target should pick a value that wouldn't put an unnecessary constraint on the iSCSI layer under normal circumstances, so don't go too stingy, bestie 🌷 A value of 0 means the declarer has no limit on how many outstanding "unexpected" iSCSI control-type PDUs it can receive 🌈 Check Sections 8.1.1 and 8.1.2 for how this key gets used 📖
Also, iSER Hello and HelloReply Messages are not iSCSI control-type PDUs, so this key doesn't affect them 🙈
7. iSCSI PDU Considerations 🦋
When a connection is in the iSER-assisted mode, two types of message transfers are allowed between the iSCSI layer at the initiator and the iSCSI layer at the target 💜 They're called iSCSI data-type PDUs and iSCSI control-type PDUs, and the next sections break down what each one is 🫶
7.1. iSCSI Data-Type PDU
An iSCSI data-type PDU is an iSCSI PDU that causes data transfer to happen between the peer iSCSI nodes during the full feature phase of an iSCSI/iSER connection, and the remote iSCSI layer doesn't see it happening 🙈 When the iSCSI layer in the sending node requests one of these to be transmitted, the data gets sent without the iSCSI layers at either the sending or the receiving node taking part 🔥 That's because the PDU itself never gets delivered as-is to the iSCSI layer in the receiving node. Instead, the data transfer operations get transformed into the appropriate RDMA operations, which the RDMA-Capable Controller handles 🧠 The set of iSCSI data-type PDUs consists of SCSI Data-in PDUs and R2T PDUs 🍓
If the iSCSI layer's invocation of the Operational Primitive asking the iSER layer to process an iSCSI data-type PDU is qualified with Notify_Enable set, then once the RDMA operation completes, the iSER layer at the target MUST notify the iSCSI layer at the target by invoking the Data_Completion_Notify Operational Primitive qualified with ITT and SN 🔔 There's no data completion notification at the initiator, since the RDMA-Capable Controller handles the RDMA operations completely at the initiator, and the iSER layer at the initiator isn't involved in the data transfer for iSCSI data-type PDUs 🌸
If the invocation is qualified with Notify_Enable cleared, then once the RDMA operation completes, the iSER layer at the target MUST NOT notify the iSCSI layer at the target and MUST NOT invoke the Data_Completion_Notify Operational Primitive 🙅♀️
If an operation associated with an iSCSI data-type PDU fails for any reason, the contents of the Data Sink buffers associated with the operation are considered indeterminate. ⚠️
7.2. iSCSI Control-Type PDU
Any iSCSI PDU that isn't an iSCSI data-type PDU, and also isn't a SCSI Data-out PDU carrying solicited data, is an iSCSI control-type PDU 🧚 The iSCSI layer invokes the Send_Control Operational Primitive to ask the iSER layer to process an iSCSI control-type PDU 🦋 iSCSI control-type PDUs get transferred using Send Message Types of RCaP. Also note that SCSI Data-out PDUs carrying unsolicited data are defined as iSCSI control-type PDUs 👀 See Section 7.3.4 for the treatment of SCSI Data-out PDUs 📖
When the iSER layer receives an iSCSI control-type PDU, it MUST notify the iSCSI layer by invoking the Control_Notify Operational Primitive qualified with the iSCSI control-type PDU 💅
7.3. iSCSI PDUs
This section covers how the iSER layer handles each iSCSI PDU type 🌈 The iSCSI layer asks the iSER layer to process the iSCSI PDU by invoking the appropriate Operational Primitive. A Connection_Handle MUST qualify each of these invocations 🔑 On top of that, the BHS and the optional AHS of the iSCSI PDU, as defined in [RFC3720], MUST qualify each of the invocations 🫶 The qualifying Connection_Handle, the BHS, and the AHS aren't explicitly listed in the subsequent sections, so keep them in mind as you read ✨
7.3.1. SCSI Command 💅
okay so this is the SCSI Command PDU, and it's a control-type PDU 🎀 here's the vibe check:
Type: control-type PDU
PDU-specific qualifiers (for SCSI Write or bidirectional command):
ImmediateDataSize, UnsolicitedDataSize, DataDescriptorOut
PDU-specific qualifiers (for SCSI read or bidirectional command):
DataDescriptorInbestie, the iSER layer at the initiator MUST send the SCSI command in a SendSE Message to the target 🚀
for a SCSI Write or bidirectional command, the iSCSI layer at the initiator MUST invoke the Send_Control Operational Primitive like this 👑
- if there's immediate data for the Write or bidirectional command, the
ImmediateDataSizequalifier MUST be used to say how many bytes of immediate unsolicited data go along with the command 🍓 and theDataDescriptorOutqualifier MUST be used to define the initiator's I/O Buffer with the SCSI Write data. - if there's unsolicited data, the
UnsolicitedDataSizequalifier MUST be used to define how many bytes of immediate and non-immediate unsolicited data the command has ✨ the iSCSI layer will send one or more SCSI Data-out PDUs for the non-immediate unsolicited data. see Section 7.3.4 on SCSI Data-out 🦋 - if there's solicited data for the SCSI write or bidirectional command (aka the Expected Data Transfer Length in the SCSI Command PDU is bigger than the value of
UnsolicitedDataSize), the iSER layer at the initiator MUST do the following 😭- a. it MUST allocate a Write STag for the I/O Buffer defined by the
DataDescriptorOutqualifier. theDataDescriptorOutdescribes the I/O buffer starting with the immediate unsolicited data (if any), then the non-immediate unsolicited data (if any), and then the solicited data. so the BufferOffset for the SCSI Data-out for this command is equal to the TO, and a zero TO for this STag points to the beginning of this I/O Buffer 🌸 - b. it MUST establish a Local Mapping that associates the Initiator Task Tag (ITT) to the Write STag 🫶
- c. it MUST Advertise the Write STag to the target by sending it as the Write STag in the iSER header of the iSER Message (the payload of the SendSE Message of RCaP) that contains the SCSI write or bidirectional command PDU. see Section 9.2 on iSER Header Format for the iSCSI Control-Type PDU 💜
- a. it MUST allocate a Write STag for the I/O Buffer defined by the
for a SCSI read or bidirectional command, the iSCSI layer at the initiator MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorIn, which defines the initiator's I/O Buffer for receiving the SCSI Read data 🧚 then the iSER layer at the initiator MUST do the following:
- a. it MUST allocate a Read STag for the I/O Buffer.
- b. it MUST establish a Local Mapping that associates the Initiator Task Tag (ITT) to the Read STag.
- c. it MUST Advertise the Read STag to the target by sending it as the Read STag in the iSER header of the iSER Message (the payload of the SendSE Message of RCaP) that contains the SCSI read or bidirectional command PDU. see Section 9.2 on iSER Header Format for the iSCSI Control-Type PDU 🔥
if the amount of unsolicited data in a SCSI command goes over TargetRecvDataSegmentLength, then the iSCSI layer at the initiator MUST split the data into multiple iSCSI control-type PDUs. every PDU except the last one has a data segment length of exactly TargetRecvDataSegmentLength, and the last iSCSI control-type PDU carrying the unsolicited data can be up to TargetRecvDataSegmentLength 🌈
when the iSER layer at the target gets the SCSI command, it MUST set up a Remote Mapping that associates the ITT with the Advertised Write STag and the Read STag (if present in the iSER header) 🍓 the Write STag is used by the iSER layer at the target when it handles the data transfer tied to the R2T PDU(s), as described in Section 7.3.6. the Read STag is used when handling the SCSI Data-in PDU(s) from the iSCSI layer at the target, as described in Section 7.3.5 💖
7.3.2. SCSI Response 🦋
okay this one's the SCSI Response PDU, another control-type PDU 💅
Type: control-type PDU
PDU-specific qualifiers: DataDescriptorStatusthe iSCSI layer at the target MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorStatus, which defines the buffer holding the sense and response information 🎀 the iSCSI layer at the target MUST always return the SCSI status for a SCSI command in a separate SCSI Response PDU. "Phase collapse" for transferring SCSI status in a SCSI Data-in PDU MUST NOT be used 🙈 the iSER layer at the target sends the SCSI Response PDU using these rules:
- if the initiator advertised no STags in the iSER Message containing the SCSI command PDU, then the iSER layer at the target MUST send a SendSE Message containing the SCSI Response PDU 🌸
- if the initiator advertised a Read STag in the iSER Message containing the SCSI Command PDU, then the iSER layer at the target MUST send a SendInvSE Message containing the SCSI Response PDU. the header of the SendInvSE Message MUST carry the Read STag to be invalidated at the initiator 🔥
- if the initiator advertised only the Write STag in the iSER Message containing the SCSI Command PDU, then the iSER layer at the target MUST send a SendInvSE Message containing the SCSI Response PDU. the header of the SendInvSE Message MUST carry the Write STag to be invalidated at the initiator 👑
when the iSCSI layer at the target invokes the Send_Control Operational Primitive to send the SCSI Response PDU, the iSER layer at the target MUST invalidate the Remote Mapping that associates the ITT with the Advertised STag(s) before it transfers the SCSI Response PDU to the initiator ✨
when the initiator gets the SendInvSE Message containing the SCSI Response PDU from the target, the RCaP layer at the initiator will invalidate the STag in the header. the iSER layer at the initiator MUST make sure the correct STag gets invalidated 🫶 if the initiator advertised both the Read and the Write STags earlier, then the iSER layer at the initiator MUST explicitly invalidate the Write STag when it receives the SendInvSE Message, because the header of the SendInvSE Message can only carry one STag (in this case, the Read STag) to be invalidated 🧚
the iSER layer at the initiator MUST make sure the STag(s) used in a command are invalidated before it notifies the iSCSI layer at the initiator by invoking the Control_Notify Operational Primitive qualified with the SCSI Response. this blocks any chance of using the STag(s) after the command finishes, which could cause data corruption 😭💀
when the iSER layer at the initiator receives the SendSE or the SendInvSE Message containing the SCSI Response PDU, it SHOULD invalidate the Local Mapping that associates the ITT with the local STag(s). the iSER layer MUST make sure all local STag(s) associated with the ITT are invalidated before it notifies the iSCSI layer of the SCSI Response PDU by invoking the Control_Notify Operational Primitive qualified with the SCSI Response PDU 🌈
7.3.3. Task Management Function Request/Response 🧚
and the last one in this batch is the Task Management Function Request/Response, a control-type PDU too 💜
Type: control-type PDU
PDU-specific qualifiers (for TMF Request): DataDescriptorOut,
DataDescriptorInthe iSER layer MUST use a SendSE Message to send the Task Management Function Request/Response PDU 🍓
for the Task Management Function Request with the TASK REASSIGN function, the iSER layer at the initiator MUST do the following 👑
- it MUST use the ITT, as specified in the Referenced Task Tag from the Task Management Function Request PDU, to find the existing STag(s), if any, in the Local Mapping(s) that associate the ITT with the local STag(s) 🔍
- it MUST invalidate the existing STag(s), if any, and the Local Mapping(s) that associate the ITT with the local STag(s) 💥
- it MUST allocate a Read STag for the I/O Buffer defined by the
DataDescriptorInqualifier, if theSend_ControlOperational Primitive invocation is qualified withDataDescriptorIn🌸 - it MUST allocate a Write STag for the I/O Buffer defined by the
DataDescriptorOutqualifier, if theSend_ControlOperational Primitive invocation is qualified withDataDescriptorOut🎀 - if STags are allocated, it MUST establish new Local Mapping(s) that associate the ITT with the allocated STag(s) 🫶
- it MUST Advertise the STags, if allocated, to the target in the iSER header of the SendSE Message carrying the iSCSI PDU, as described in Section 9.2 ✨
for the Task Management Function Request with the TASK REASSIGN function for a SCSI read or bidirectional command, the iSCSI layer at the initiator MUST set ExpDataSN to 0, since the data transfer and acknowledgements happen transparently to the iSCSI layer at the initiator 🦋 this gives the iSCSI layer at the target the flexibility to request transmission of only the unacknowledged data, as specified in [RFC3720] 🌈
when the iSER layer at the target receives the Task Management Function Request with the TASK REASSIGN function, it MUST do the following 🔥
- it MUST use the ITT, as specified in the Referenced Task Tag from the Task Management Function Request PDU, to find the mappings that associate the ITT with the Advertised STag(s) and the local STag(s), if any 🔍
- it MUST invalidate the local STag(s), if any, associated with the ITT 💅
- it MUST replace the Advertised STag(s) in the Remote Mapping that associates the ITT with the Advertised STag(s) with the Write STag and the Read STag (if present in the iSER header). the Write STag is used when handling the R2T PDU(s) from the iSCSI layer at the target, as described in Section 7.3.6. the Read STag is used when handling the SCSI Data-in PDU(s) from the iSCSI layer at the target, as described in Section 7.3.5 💖
7.3.4. SCSI Data-Out ✨
Type: control-type PDU 💅
PDU-specific qualifiers: DataDescriptorOut
okay bestie, the iSCSI layer at the initiator MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorOut, which defines the initiator's I/O Buffer containing unsolicited SCSI Write data 🫶
If the amount of unsolicited data to be transferred as SCSI Data-out exceeds TargetRecvDataSegmentLength, then the iSCSI layer at the initiator MUST segment the data into multiple iSCSI control-type PDUs, with the DataSegmentLength having the value of TargetRecvDataSegmentLength in all PDUs generated except the last one. The DataSegmentLength of the last iSCSI control-type PDU carrying the unsolicited data can be up to TargetRecvDataSegmentLength. The iSCSI layer at the target MUST perform the reassembly function for the unsolicited data 🧚
For unsolicited data, if the F bit is set to 0 in a SCSI Data-out PDU, the iSER layer at the initiator MUST use a Send Message to send the SCSI Data-out PDU. If the F bit is set to 1, the iSER layer at the initiator MUST use a SendSE Message to send the SCSI Data-out PDU.
Note that for solicited data, the SCSI Data-out PDUs are not used, since R2T PDUs are not delivered to the iSCSI layer at the initiator. Instead, R2T PDUs are transformed by the iSER layer at the target into RDMA Read operations. (See Section 7.3.6.) 🦋
7.3.5. SCSI Data-In 🌸
Type: data-type PDU
PDU-specific qualifiers: DataDescriptorIn
When the iSCSI layer at the target is ready to return the SCSI Read data to the initiator, it MUST invoke the Put_Data Operational Primitive qualified with DataDescriptorIn, which defines the SCSI Data-in buffer. See Section 7.1 on the general requirement on the handling of iSCSI data-type PDUs. SCSI Data-in PDU(s) are used in SCSI Read data transfer as described in Section 9.5.2. 💖
The iSER layer at the target MUST do the following for each invocation of the Put_Data Operational Primitive:
- It MUST use the
ITTin the SCSI Data-in PDU to locate the remote Read STag in the Remote Mapping that associates theITTto Advertised STag(s). The Remote Mapping was established earlier by the iSER layer at the target when the SCSI read command was received from the initiator. - It MUST generate and send an RDMA Write Message containing the read data to the initiator. a. It MUST use the remote Read STag as the Data Sink STag of the RDMA Write Message. b. It MUST use the Buffer Offset from the SCSI Data-in PDU as the Data Sink Tagged Offset of the RDMA Write Message. c. It MUST use
DataSegmentLengthfrom the SCSI Data-in PDU to determine the amount of data to be sent in the RDMA Write Message. - It MUST associate
DataSNandITTfrom the SCSI Data-in PDU with the RDMA Write operation. If thePut_DataOperational Primitive invocation was qualified withNotify_Enableset, then when the iSER layer at the target receives a completion from the RCaP layer for the RDMA Write Message, the iSER layer at the target MUST notify the iSCSI layer by invoking theData_Completion_NotifyOperational Primitive qualified withDataSNandITT. Conversely, if thePut_DataOperational Primitive invocation was qualified withNotify_Enablecleared, then the iSER layer at the target MUST NOT notify the iSCSI layer on completion and MUST NOT invoke theData_Completion_NotifyOperational Primitive.
When the A-bit is set to 1 in the SCSI Data-in PDU, the iSER layer at the target MUST notify the iSCSI layer at the target when the data transfer is complete at the initiator. To perform this additional function, the iSER layer at the target can take advantage of the operational ErrorRecoveryLevel if previously disclosed by the iSCSI layer via an earlier invocation of the Notice_Key_Values Operational Primitive. There are two approaches that can be taken:
- If the iSER layer at the target knows that the operational
ErrorRecoveryLevelis 2, or if the iSER layer at the target does not know the operationalErrorRecoveryLevel, then the iSER layer at the target MUST issue a zero-length RDMA Read Request Message following the RDMA Write Message. When the iSER layer at the target receives a completion for the RDMA Read Request Message from the RCaP layer, implying that the RDMA-Capable Controller at the initiator has completed processing the RDMA Write Message due to the completion ordering semantics of RCaP, the iSER layer at the target MUST notify the iSCSI layer at the target by invoking theData_Ack_NotifyOperational Primitive qualified withITTandDataSN(see Section 3.2.3). - If the iSER layer at the target knows that the operational
ErrorRecoveryLevelis 1, then the iSER layer at the target MUST do one of the following: a. It MUST notify the iSCSI layer at the target by invoking theData_Ack_NotifyOperational Primitive qualified withITTandDataSN(see Section 3.2.3) when it receives the local completion from the RCaP layer for the RDMA Write Message. This is allowed since digest errors do not occur in iSER (see Section 10.1.4.2), and a CRC error will cause the connection to be terminated and the task to be terminated anyway. The local RDMA Write completion from the RCaP layer guarantees that the RCaP layer will not access the I/O Buffer again to transfer the data associated with that RDMA Write operation. b. Alternatively, it MUST use the same procedure for handling the data transfer completion at the initiator as forErrorRecoveryLevel2.
Note that the iSCSI layer at the target cannot set the A-bit to 1 if the ErrorRecoveryLevel=0. 🙈
The SCSI status MUST always be returned in a separate SCSI Response PDU. The S bit in the SCSI Data-in PDU MUST always be set to 0. There MUST NOT be a "phase collapse" in the SCSI Data-in PDU. 🔥
Since the RDMA Write Message only transfers the data portion of the SCSI Data-in PDU but not the control information in the header, such as ExpCmdSN, if timely updates of such information are crucial, the iSCSI layer at the initiator MAY issue NOP-Out PDUs to request that the iSCSI layer at the target respond with the information using NOP-In PDUs. ✨
7.3.6. Ready to Transfer (R2T) 👑
Type: data-type PDU
PDU-specific qualifiers: DataDescriptorOut
In order to send an R2T PDU, the iSCSI layer at the target MUST invoke the Get_Data Operational Primitive qualified with DataDescriptorOut, which defines the I/O Buffer for receiving the SCSI Write data from the initiator. See Section 7.1 on the general requirements on the handling of iSCSI data-type PDUs.
The iSER layer at the target MUST do the following for each invocation of the Get_Data Operational Primitive:
- It MUST ensure a valid local STag for the I/O Buffer and a valid Local Mapping that associates the Initiator Task Tag (
ITT) to the local STag. This may involve allocating a valid local STag and establishing a Local Mapping. - It MUST use the
ITTin the R2T to locate the remote Write STag in the Remote Mapping that associates theITTto Advertised STag(s). The Remote Mapping is established earlier by the iSER layer at the target when the iSER Message containing the Advertised Write STag and the SCSI Command PDU for a SCSI write or bidirectional command is received from the initiator. - If the
iSER-ORDvalue at the target is set to 0, the iSER layer at the target MUST terminate the connection and free up the resources associated with the connection (as described in Section 5.2.3) if it receives the R2T PDU from the iSCSI layer at the target. Upon termination of the connection, the iSER layer at the target MUST notify the iSCSI layer at the target by invoking theConnection_Terminate_NotifyOperational Primitive. - If the
iSER-ORDvalue at the target is set to greater than 0, the iSER layer at the target MUST transform the R2T PDU into an RDMA Read Request Message. While transforming the R2T PDU, the iSER layer at the target MUST ensure that the number of outstanding RDMA Read Request Messages does not exceed theiSER-ORDvalue. To transform the R2T PDU, the iSER layer at the target: a. MUST derive the local STag and local Tagged Offset from theDataDescriptorOutthat qualified theGet_Datainvocation. b. MUST use the local STag as the Data Sink STag of the RDMA Read Request Message. c. MUST use the local Tagged Offset as the Data Sink Tagged Offset of the RDMA Read Request Message. d. MUST use the Desired Data Transfer Length from the R2T PDU as the RDMA Read Message Size of the RDMA Read Request Message. e. MUST use the remote Write STag as the Data Source STag of the RDMA Read Request Message. f. MUST use the Buffer Offset from the R2T PDU as the Data Source Tagged Offset of the RDMA Read Request Message. - It MUST associate
R2TSNandITTfrom the R2T PDU with the RDMA Read operation. If theGet_DataOperational Primitive invocation is qualified withNotify_Enableset, then when the iSER layer at the target receives a completion from the RCaP layer for the RDMA Read operation, the iSER layer at the target MUST notify the iSCSI layer by invoking theData_Completion_NotifyOperational Primitive qualified withR2TSNandITT. Conversely, if theGet_DataOperational Primitive invocation is qualified withNotify_Enablecleared, then the iSER layer at the target MUST NOT notify the iSCSI layer on completion and MUST NOT invoke theData_Completion_NotifyOperational Primitive.
When the RCaP layer at the initiator receives a valid RDMA Read Request Message, it will return an RDMA Read Response Message containing the solicited write data to the target. When the RCaP layer at target receives the RDMA Read Response Message from the initiator, it will place the solicited data in the I/O Buffer referenced by the Data Sink STag in the RDMA Read Response Message. 🌈
Since the RDMA Read Request Message from the target does not transfer the control information in the R2T PDU, such as ExpCmdSN, if timely updates of such information are crucial, the iSCSI layer at the initiator MAY issue NOP-Out PDUs to request that the iSCSI layer at the target respond with the information using NOP-In PDUs. 💜
Similarly, since the RDMA Read Response Message from the initiator only transfers the data but not the control information normally found in the SCSI Data-out PDU, such as ExpStatSN, if timely updates of such information are crucial, the iSCSI layer at the target MAY issue NOP-In PDUs to request that the iSCSI layer at the initiator respond with the information using NOP-Out PDUs. 🫶
7.3.7. Asynchronous Message 🦋
Type: control-type PDU
PDU-specific qualifiers: DataDescriptorSense
The iSCSI layer MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorSense, which defines the buffer containing the sense and iSCSI Event information. The iSER layer MUST use a SendSE Message to send the Asynchronous Message PDU. 💅
7.3.8. Text Request and Text Response ✨
Type: control-type PDU
PDU-specific qualifiers: DataDescriptorTextOut (for Text Request), DataDescriptorIn (for Text Response)
The iSCSI layer MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorTextOut (or DataDescriptorIn), which defines the Text Request (or Text Response) buffer. The iSER layer MUST use SendSE Messages to send the Text Request (or Text Response) PDUs. 💖
7.3.9. Login Request and Login Response 🔐
During the login negotiation, the iSCSI layer talks to the transport layer directly, and the iSER layer is not involved. See Section 5.1 on iSCSI/iSER connection setup. If the underlying transport is TCP, the Login Request PDUs and the Login Response PDUs are exchanged while the connection between the initiator and the target is still in the byte stream mode.
The iSCSI layer MUST not send a Login Request (or a Login Response) PDU during the Full Feature Phase. A Login Request (or a Login Response) PDU, if used, MUST be treated as an iSCSI protocol error. The iSER layer MAY reject such a PDU from the iSCSI layer with an appropriate error code. If a Login Request PDU is received by the iSCSI layer at the target, it MUST respond with a Reject PDU with a reason code of "protocol error". 🚫
7.3.10. Logout Request and Logout Response 👋
Type: control-type PDU
PDU-specific qualifiers: None
The iSER layer MUST use a SendSE Message to send the Logout Request or Logout Response PDU. Sections 5.2.1 and 5.2.2 describe how the Logout Request and the Logout Response are handled at the initiator and the target, and the interactions between the initiator and the target to terminate a connection. Bestie, this is the goodbye choreography. 🫶
7.3.11. SNACK Request 🍓
Since HeaderDigest and DataDigest must be negotiated to "None", there are no digest errors when the connection is in iSER-assisted mode. Also, since RCaP delivers all messages in the order they were sent, there are no sequence errors when the connection is in iSER-assisted mode. Therefore, the iSCSI layer MUST NOT send SNACK Request PDUs. A SNCAK Request PDU, if used, MUST be treated as an iSCSI protocol error. The iSER layer MAY reject such a PDU from the iSCSI layer with an appropriate error code. If a SNACK Request PDU is received by the iSCSI layer at the target, it MUST respond with a Reject PDU with a reason code of "protocol error". No retransmission drama here, iykyk. 🙈
7.3.12. Reject 🚩
Type: control-type PDU
PDU-specific qualifiers: DataDescriptorReject
The iSCSI layer MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorReject, which defines the Reject buffer. The iSER layer MUST use a SendSE Message to send the Reject PDU.
7.3.13. NOP-Out and NOP-In 🏓
Type: control-type PDU
PDU-specific qualifiers: DataDescriptorNOPOut (for NOP-Out), DataDescriptorNOPIn (for NOP-In)
The iSCSI layer MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorNOPOut (or DataDescriptorNOPIn), which defines the Ping (or Return Ping) data buffer. The iSER layer MUST use SendSE Messages to send the NOP-Out (or NOP-In) PDU. Ping pong, but make it protocol. 🧚
8. Flow Control and STag Management 🌈
8.1. Flow Control for RDMA Send Message Types 💌
Send Message Types in RCaP are used by the iSER layer to transfer iSCSI control-type PDUs. Each Send Message Type in RCaP consumes an Untagged Buffer at the Data Sink. However, neither the RCaP layer nor the iSER layer provides an explicit flow control mechanism for the Send Message Types. Therefore, the iSER layer SHOULD provision enough Untagged buffers for handling incoming Send Message Types to prevent buffer exhaustion at the RCaP layer. If buffer exhaustion occurs, it may result in the termination of the connection. Not the drama we want. 😭
An implementation may choose to satisfy the buffer requirement by using a common buffer pool shared across multiple connections, with usage limits on a per-connection basis and usage limits on the buffer pool itself. In such an implementation, exceeding the buffer usage limit for a connection or the buffer pool itself may trigger interventions from the iSER layer to replenish the buffer pool and/or to isolate the connection causing the problem.
iSER also provides the MaxOutstandingUnexpectedPDUs key, which the initiator and the target use to declare the maximum number of outstanding "unexpected" control-type PDUs that they can receive. It is intended to allow the receiving side to determine the amount of buffer resources needed beyond the normal flow control mechanism available in iSCSI.
The buffer resources required at both the initiator and the target as a result of control-type PDUs sent by the initiator are described in Section 8.1.1. The buffer resources required at both the initiator and target as a result of control-type PDUs sent by the target are described in Section 8.1.2.
8.1.1. Flow Control for Control-Type PDUs from the Initiator 📤
The control-type PDUs that an initiator can send to a target can be grouped into the following categories:
- Regulated: Control-type PDUs in this category are regulated by the iSCSI CmdSN window mechanism, and the immediate flag is not set.
- Unregulated but Expected: Control-type PDUs in this category are not regulated by the iSCSI CmdSN window mechanism, but they are expected by the target.
- Unregulated and Unexpected: Control-type PDUs in this category are not regulated by the iSCSI CmdSN window mechanism, and they are "unexpected" by the target. 🌀
8.1.1.1. Control-Type PDUs from the Initiator in the Regulated Category 🚦
Control-type PDUs that the initiator can send in this category are regulated by the iSCSI CmdSN window mechanism, and the immediate flag is not set.
The queuing capacity required of the iSCSI layer at the target is described in Section 3.2.2.1 of [RFC3720]. For each of the control-type PDUs that the initiator can send in this category, the initiator MUST provision for the buffer resources required for the corresponding control-type PDU sent as a response from the target. The following is a list of the PDUs that the initiator can send, along with the PDUs the target sends in response:
- a. When an initiator sends a SCSI Command PDU, it expects a SCSI Response PDU from the target.
- b. When the initiator sends a Task Management Function Request PDU, it expects a Task Management Function Response PDU from the target.
- c. When the initiator sends a Text Request PDU, it expects a Text Response PDU from the target.
- d. When the initiator sends a Logout Request PDU, it expects a Logout Response PDU from the target.
- e. When the initiator sends a NOP-Out PDU as a ping request with
ITT != 0xffffffffandTTT = 0xffffffff, it expects a NOP-In PDU from the target with the sameITTandTTTas in the ping request. 🏓
The response from the target for any of the PDUs listed here may instead take the form of a Reject PDU, sent before the task is active, as described in Section 6.3 of [RFC3720]. 🙅♀️
8.1.1.2. Control-Type PDUs from the Initiator in the Unregulated but Expected Category 🎯
For the control-type PDUs in the Unregulated but Expected category, the amount of buffering resources required at the target can be predetermined. The following PDUs are in this category:
- a. SCSI Data-out PDUs are used by the initiator to send unsolicited data. The amount of buffer resources required by the target can be determined using
FirstBurstLength. Note that SCSI Data-out PDUs are not used for solicited data, since the R2T PDU that is used for solicitation is transformed into RDMA Read operations by the iSER layer at the target. See Section 7.3.4. - b. A NOP-Out PDU with
TTT != 0xffffffffis sent as a ping response by the initiator to the NOP-In PDU sent as a ping request by the target. 🪞
8.1.1.3. Control-Type PDUs from the Initiator in the Unregulated and Unexpected Category ⚡
PDUs in the Unregulated and Unexpected category are PDUs with the immediate flag set. The number of PDUs in this category that an initiator can send is controlled by the value of MaxOutstandingUnexpectedPDUs declared by the target (see Section 6.7). After a PDU in this category is sent by the initiator, it is outstanding until it is retired. At any time, the number of outstanding unexpected PDUs MUST not exceed the value of MaxOutstandingUnexpectedPDUs declared by the target.
The target uses the value of MaxOutstandingUnexpectedPDUs that it declared to determine the amount of buffer resources required for control-type PDUs in this category that an initiator can send. For the initiator, for each of the control-type PDUs that can be sent in this category, the initiator MUST provision for the buffer resources, if required, for the corresponding control-type PDU that can be sent as a response from the target.
An outstanding PDU in this category is retired as follows. If the CmdSN of the PDU sent by the initiator in this category is x, the PDU is outstanding until the initiator sends a non-immediate control-type PDU on the same connection with CmdSN = y (where y is at least x), and the target responds with a control-type PDU on any connection where ExpCmdSN is at least y+1. 🧾
When the number of outstanding unexpected control-type PDUs equals MaxOutstandingUnexpectedPDUs, the iSCSI layer at the initiator MUST NOT generate any unexpected PDUs that it otherwise would have generated, even if they are intended for immediate delivery. Full queue, no cutting in line. That's the whole vibe. 👑
8.1.2. Flow Control for Control-Type PDUs from the Target ✨
Okay bestie, now we flip to the target side 🎀 Control-type PDUs that a target can send and that the initiator is expecting are listed in the Regulated category, so check Section 8.1.1.1 for those 💅
The unexpected ones are different. Their count is controlled by MaxOutstandingUnexpectedPDUs, which the initiator declares (see Section 6.7). Once a target sends one of these, it stays outstanding until it gets retired 🦋 At any time, the number of outstanding unexpected PDUs MUST not exceed the value of MaxOutstandingUnexpectedPDUs that the initiator declared. The initiator uses that declared value to figure out how much buffer space it needs for control-type PDUs in this category that a target can send 🧚 Here's the list of PDUs in this category, plus how each one gets retired:
- a. For an Asynchronous Message PDU with
StatSN= x, the PDU is outstanding until the initiator sends a control-type PDU withExpStatSNset to at least x+1. - b. For a Reject PDU with
StatSN= x that is sent after a task is active, the PDU is outstanding until the initiator sends a control-type PDU withExpStatSNset to at least x+1. - c. For a NOP-In PDU with
ITT=0xffffffffandStatSN= x, the PDU is outstanding until the initiator responds with a control-type PDU on the same connection whereExpStatSNis at least x+1. But if the NOP-In PDU is sent as a ping request withTTT!=0xffffffff, the PDU can also be retired when the initiator sends a NOP-Out PDU with the sameITTandTTTas in the ping request. Note that when a target sends a NOP-In PDU as a ping request, it must provision a buffer for the NOP-Out PDU sent as a ping response from the initiator.
When the number of outstanding unexpected control-type PDUs equals MaxOutstandingUnexpectedPDUs, the iSCSI layer at the target MUST NOT generate any unexpected PDUs that it otherwise would have generated, even if its intent is to indicate an iSCSI error condition (e.g., Asynchronous Message, Reject) 😭 Task timeouts (like the initiator waiting for a command completion) and other connection and session level exceptions will make sure correct operational behavior still happens in these cases, even though that PDU isn't generated 🙈 This rule overrides any other requirements elsewhere that require that a Reject PDU MUST be sent. Yep, it's a whole override moment 👑
(Implementation note: A SCSI task timeout and recovery can be a lengthy process and hence SHOULD be avoided by proper provisioning of resources.)
(Implementation note: To ensure that the initiator has a means to inform the target that outstanding PDUs have been retired, the target should reserve the last unexpected control-type PDU allowable by the value of MaxOutstandingUnexpectedPDUs declared by the initiator for sending a NOP-In ping request with TTT != 0xffffffff to allow the initiator to return the NOP-Out ping response with the current ExpStatSN.)
8.2. Flow Control for RDMA Read Resources 🌸
The total number of RDMA Read operations that can be active at the same time on an iSCSI/iSER connection depends on how many resources were allocated, as declared in the iSER Hello exchange described in Section 5.1.3. Going over the number of RDMA Read operations allowed on a connection will result in the connection being terminated by the RCaP layer 💀 The iSER layer at the target keeps track of the iSER-ORD so it knows the maximum number of RDMA Read Requests the iSER layer can issue on a particular RCaP Stream.
During connection setup (see Section 5.1), the initiator knows iSER-IRD, and the target knows iSER-ORD after the iSER layers at both ends have allocated the connection resources needed to support RCaP. This happens as directed by the Allocate_Connection_Resources Operational Primitive from the iSCSI layer, before the end of the iSCSI Login Phase. In the Full Feature Phase, the first message the initiator sends is the iSER Hello Message (see Section 9.3), which contains the value of iSER-IRD. In response to the iSER Hello Message, the target sends the iSER HelloReply Message (see Section 9.4), which contains the value of iSER-ORD. The iSER layer at both the initiator and the target MAY adjust (lower) the resources associated with iSER-IRD and iSER-ORD respectively, to match the iSER-ORD value declared in the HelloReply Message. The iSER layer at the target MUST flow control the RDMA Read Request Messages so they do not exceed the iSER-ORD value at the target 🦋
8.3. STag Management 🔥
An STag, as defined in [RDMAP], is an identifier of a Tagged Buffer used in an RDMA operation. The allocation and the later invalidation of STags are specified in this document if the STags are exposed on the wire, which happens when they're Advertised in the iSER header or declared in the header of an RCaP Message 👀
8.3.1. Allocation of STags
When the iSCSI layer at the initiator invokes the Send_Control Operational Primitive to ask the iSER layer at the initiator to process a SCSI command, zero, one, or two STags may be allocated by the iSER layer. See Section 7.3.1 for details. How many STags get allocated depends on whether the command is unidirectional or bidirectional, and on whether or not solicited write data transfer is involved 🧚
When the iSCSI layer at the initiator invokes the Send_Control Operational Primitive to ask the iSER layer at the initiator to process a Task Management Function Request with the TASK REASSIGN function, then besides allocating zero, one, or two STags, the iSER layer MUST invalidate the existing STags, if any, associated with the ITT. See Section 7.3.3 for details.
The iSER layer at the target allocates a local Data Sink STag when the iSCSI layer at the target invokes the Get_Data Operational Primitive to ask the iSER layer to process an R2T PDU. See Section 7.3.6 for details 💖
8.3.2. Invalidation of STags
The invalidation of STags at the initiator, when a unidirectional or bidirectional command completes and the associated SCSI Response PDU is sent by the target, is described in Section 7.3.2 ✨
When a unidirectional or bidirectional command concludes without the associated SCSI Response PDU being sent by the target, the iSCSI layer at the initiator MUST request that the iSER layer at the initiator invalidate the STags by invoking the Deallocate_Task_Resources Operational Primitive qualified with ITT. In response, the iSER layer at the initiator MUST locate the STag(s) (if any) in the Local Mapping that associates the ITT to the local STag(s). The iSER layer at the initiator MUST invalidate the STag(s) (if any) and the Local Mapping.
For an RDMA Read operation used to realize a SCSI Write data transfer, the iSER layer at the target SHOULD invalidate the Data Sink STag at the conclusion of the RDMA Read operation that references the Data Sink STag, so the buffer resources can be reused right away 🌈
For an RDMA Write operation used to realize a SCSI Read data transfer, the Data Source STag at the target is not declared to the initiator and is not exposed on the wire. So invalidation of that STag is not specified 🙈
When a unidirectional or bidirectional command concludes without the associated SCSI Response PDU being sent by the target, the iSCSI layer at the target MUST request that the iSER layer at the target invalidate the STags by invoking the Deallocate_Task_Resources Operational Primitive qualified with ITT. In response, the iSER layer at the target MUST locate the local STag(s) (if any) in the Local Mapping that associates the ITT to the local STag(s). The iSER layer at the target MUST invalidate the local STag(s) (if any) and the mapping.
9. iSER Control and Data Transfer 🌈
For iSCSI data-type PDUs (see Section 7.1), the iSER layer uses RDMA Read and RDMA Write operations to transfer the solicited data. For iSCSI control-type PDUs (see Section 7.2), the iSER layer uses Send Message Types of RCaP 💜
9.1. iSER Header Format
An iSER header MUST be present in every Send Message Type of RCaP. The iSER header is located in the first 12 bytes of the message payload of the Send Message Type of RCaP, as shown in Figure 2 🎀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Opcode| Opcode Specific Fields |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Opcode Specific Fields |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Opcode Specific Fields |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2. iSER Header FormatOpcode - Operation Code: 4 bits
The Opcode field identifies the type of iSER Messages 🦋
0001b = iSCSI control-type PDU
0010b = iSER Hello Message
0011b = iSER HelloReply Message
All other opcodes are reserved.9.2. iSER Header Format for the iSCSI Control-Type PDU 🎀
okay so this one is about the iSER header for iSCSI control-type PDUs, and it's giving organized 💅. The iSER layer uses Send Message Types of RCaP to move iSCSI control-type PDUs around (see Section 7.2). Each Send Message Type of RCaP that carries an iSER Message has a message payload with an iSER Header first, then an iSCSI control-type PDU right after it.
The iSER header in a Send Message Type of RCaP carrying an iSCSI control-type PDU MUST have the format as described in Figure 3. 🧚
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |W|R| |
| 0001b |S|S| Reserved |
| |V|V| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Write STag (or N/A) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Read STag (or N/A) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 3. iSER Header Format for iSCSI Control-Type PDU
WSV - Write STag Valid flag: 1 bit
This flag tells you whether the Write STag field in the iSER Header is valid. If it's set to one, the Write STag field is valid. If it's set to zero, the Write STag field MUST be ignored at the receiver. The Write STag Valid flag is set to one when there's solicited data to transfer for a SCSI write or bidirectional command, or when there's non-immediate unsolicited and solicited data to transfer for the referenced task in a Task Management Function Request with the TASK REASSIGN function. 💜
RSV - Read STag Valid flag: 1 bit
This flag tells you whether the Read STag field in the iSER Header is valid. If it's set to one, the Read STag field is valid. If it's set to zero, the Read STag field in this iSER Header MUST be ignored at the receiver. The Read STag Valid flag is set to one for a SCSI read or bidirectional command, or for a Task Management Function Request with the TASK REASSIGN function. 🦋
Write STag - Write Steering Tag: 32 bits
This field holds the Write STag when the Write STag Valid flag is set to one. For a SCSI write or bidirectional command, the Write STag is used to Advertise the initiator's I/O Buffer containing the solicited data. For a Task Management Function Request with the TASK REASSIGN function, the Write STag is used to Advertise the initiator's I/O Buffer containing the non-immediate unsolicited data and solicited data. This Write STag gets used as the Data Source STag in the resulting RDMA Read operation(s). When the Write STag Valid flag is set to zero, this field MUST be set to zero. 🌸
Read STag - Read Steering Tag: 32 bits
This field holds the Read STag when the Read STag Valid flag is set to one. The Read STag is used to Advertise the initiator's Read I/O Buffer of a SCSI read or bidirectional command, or of a Task Management Function Request with the TASK REASSIGN function. This Read STag gets used as the Data Sink STag in the resulting RDMA Write operation(s). When the Read STag Valid flag is zero, this field MUST be set to zero. 🍓
Reserved:
Reserved fields MUST be set to zero on transmit and MUST be ignored on reception. ✨
9.3. iSER Header Format for the iSER Hello Message 👑
An iSER Hello Message MUST only contain the iSER header, and that header MUST have the format as described in Figure 4. The iSER Hello Message is the very first iSER Message sent on the RCaP Stream from the iSER layer at the initiator to the iSER layer at the target. It's giving first impressions 🫶
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | | | | |
| 0010b | Rsvd | MaxVer| MinVer| iSER-IRD |
| | | | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 4. iSER Header Format for iSER Hello Message
MaxVer - Maximum Version: 4 bits
This field specifies the maximum version of the iSER protocol supported. It MUST be set to one to indicate the version of the specification described in this document. 🌈
MinVer - Minimum Version: 4 bits
This field specifies the minimum version of the iSER protocol supported. It MUST be set to one to indicate the version of the specification described in this document. 💖
iSER-IRD: 16 bits
This field holds the value of the iSER-IRD at the initiator. 🍓
Reserved (Rsvd):
Reserved fields MUST be set to zero on transmit, and MUST be ignored on reception. 🫶
9.4. iSER Header Format for the iSER HelloReply Message 🦋
An iSER HelloReply Message MUST only contain the iSER header, which MUST have the format as described in Figure 5. The iSER HelloReply Message is the first iSER Message sent on the RCaP Stream from the iSER layer at the target to the iSER layer at the initiator. It's the reply, so it's basically the other half of the conversation 💅
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |R| | | |
| 0011b |Rsvd |E| MaxVer| CurVer| iSER-ORD |
| | |J| | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 5. iSER Header Format for iSER HelloReply Message
REJ - Reject flag: 1 bit
This flag tells you whether the target is rejecting this connection. If it's set to one, the target is rejecting the connection. 🚫
MaxVer - Maximum Version: 4 bits
This field specifies the maximum version of the iSER protocol supported. It MUST be set to one to indicate the version of the specification described in this document. 🌸
CurVer - Current Version: 4 bits
This field specifies the current version of the iSER protocol supported. It MUST be set to one to indicate the version of the specification described in this document. ✨
iSER-ORD: 16 bits
This field holds the value of the iSER-ORD at the target. 🧚
Reserved (Rsvd):
Reserved fields MUST be set to zero on transmit, and MUST be ignored on reception. 💜
9.5. SCSI Data Transfer Operations 🚚
The iSER layer at the initiator and the iSER layer at the target handle each SCSI Write, SCSI Read, and bidirectional operation as described below. Think of it as the choreography for the whole data dance 💃
9.5.1. SCSI Write Operation ✍️
The iSCSI layer at the initiator MUST invoke the Send_Control Operational Primitive to request that the iSER layer at the initiator send the SCSI write command. The iSER layer at the initiator MUST request that the RCaP layer transmit a SendSE Message with the message payload consisting of the iSER header followed by the SCSI Command PDU and immediate data (if any). If there's solicited data, the iSER layer MUST Advertise the Write STag in the iSER header of the SendSE Message, as described in Section 9.2. Once the SendSE Message arrives, the iSER layer at the target MUST notify the iSCSI layer at the target by invoking the Control_Notify Operational Primitive qualified with the SCSI Command PDU. See Section 7.3.1 for details on handling the SCSI write command. 🎀
For the non-immediate unsolicited data, the iSCSI layer at the initiator MUST invoke a Send_Control Operational Primitive qualified with the SCSI Data-out PDU. Each time a Send or SendSE Message containing the non-immediate unsolicited data comes in, the iSER layer at the target MUST notify the iSCSI layer at the target by invoking the Control_Notify Operational Primitive qualified with the SCSI Data-out PDU. See Section 7.3.4 for details on handling the SCSI Data-out PDU. 🦋
For the solicited data, when the iSCSI layer at the target has an I/O Buffer available, it MUST invoke the Get_Data Operational Primitive qualified with the R2T PDU. See Section 7.3.6 for details on handling the R2T PDU. 🌸
When the data transfer for this SCSI Write operation is complete, the iSCSI layer at the target MUST invoke the Send_Control Operational Primitive once it's ready to send the SCSI Response PDU. When a SendSE or SendInvSE Message containing the SCSI Response PDU arrives, the iSER layer at the initiator MUST notify the iSCSI layer at the initiator by invoking the Control_Notify Operational Primitive qualified with the SCSI Response PDU. See Section 7.3.2 for details on handling the SCSI Response PDU. 🏁
9.5.2. SCSI Read Operation 📖
The iSCSI layer at the initiator MUST invoke the Send_Control Operational Primitive to request that the iSER layer at the initiator send the SCSI read command. The iSER layer at the initiator MUST request that the RCaP layer transmit a SendSE Message with the message payload consisting of the iSER header followed by the SCSI Command PDU. The iSER layer at the initiator MUST Advertise the Read STag in the iSER header of the SendSE Message, as described in Section 9.2. Once the SendSE Message arrives, the iSER layer at the target MUST notify the iSCSI layer at the target by invoking the Control_Notify Operational Primitive qualified with the SCSI Command PDU. See Section 7.3.1 for details on handling the SCSI read command. 🧚
When the requested SCSI data is ready in the I/O Buffer, the iSCSI layer at the target MUST invoke the Put_Data Operational Primitive qualified with the SCSI Data-in PDU. See Section 7.3.5 for details on handling the SCSI Data-in PDU. 💖
When the data transfer for this SCSI Read operation is complete, the iSCSI layer at the target MUST invoke the Send_Control Operational Primitive once it's ready to send the SCSI Response PDU. When the SendInvSE Message containing the SCSI Response PDU arrives, the iSER layer at the initiator MUST notify the iSCSI layer at the initiator by invoking the Control_Notify Operational Primitive qualified with the SCSI Response PDU. See Section 7.3.2 for details on handling the SCSI Response PDU. 💜
9.5.3. Bidirectional Operation 🔄
The initiator and the target handle the SCSI Write and the SCSI Read portions of this bidirectional operation the same way as described in Sections 9.5.1 and 9.5.2, respectively. It's giving two-for-one, bestie 🫶
10. iSER Error Handling and Recovery ✨
RCaP gives the iSER layer reliable in-order delivery 🫶, so the error management for an iSER-assisted connection is kinda different from what you'd deal with on a Traditional iSCSI connection. Okay, let's get into it 💅
10.1. Error Handling 🎀
iSER error handling is sorted loosely by where the error comes from, and there are four buckets 🧚:
- Errors coming from the transport layer (e.g., TCP).
- Errors coming from the RCaP layer.
- Errors coming from the iSER layer.
- Errors coming from the iSCSI layer.
10.1.1. Errors in the Transport Layer 🦋
If the transport layer is TCP, then TCP packets with detected errors just get silently dropped by TCP and retransmitted by TCP. The iSER layer doesn't even notice, which is so iconic 😭 However, connection loss (e.g., link failure) and unexpected termination (e.g., TCP graceful or abnormal close without the iSCSI Logout exchanges) at the transport layer will take the iSCSI/iSER connection down with it, no cap 💀
10.1.1.1. Failure in the Transport Layer before RCaP Mode Is Enabled 🌸
If the connection is lost or terminated before the iSCSI layer invokes the Allocate_Connection_Resources Operational Primitive, the login process is over and nothing else needs to happen 🙈
If the connection is lost or terminated after the iSCSI layer has invoked the Allocate_Connection_Resources Operational Primitive, then the iSCSI layer MUST request that the iSER layer deallocate all connection resources by invoking the Deallocate_Connection_Resources Operational Primitive.
10.1.1.2. Failure in the Transport Layer after RCaP Mode Is Enabled 💜
If the connection is lost or terminated after the iSCSI layer has invoked the Enable_Datamover Operational Primitive, the iSER layer MUST notify the iSCSI layer of the connection loss by invoking the Connection_Terminate_Notify Operational Primitive. Before it invokes the Connection_Terminate_Notify Operational Primitive, the iSER layer MUST perform the actions described in Section 5.2.3.2.
10.1.2. Errors in the RCaP Layer 🍓
The RCaP layer doesn't have error recovery built in, bestie. If it detects errors, it terminates the RCaP Stream and the associated connection 💅
10.1.2.1. Errors Detected in the Local RCaP Layer ⭐
If an error shows up in the local RCaP layer, the RCaP layer MAY send a Terminate Message to the Remote Peer to report the error, if it's possible. (For iWARP, see [RDMAP] for the list of errors where a Terminate Message is sent.) The RCaP layer is responsible for terminating the connection. After the RCaP layer tells the iSER layer that the connection is terminated, the iSER layer MUST notify the iSCSI layer by invoking the Connection_Terminate_Notify Operational Primitive. Before invoking the Connection_Terminate_Notify Operational Primitive, the iSER layer MUST perform the actions described in Section 5.2.3.2.
10.1.2.2. Errors Detected in the RCaP Layer at the Remote Peer 🔥
If an error shows up in the RCaP layer at the Remote Peer, that RCaP layer MAY send a Terminate Message to report it, if it's possible. If it can't send the Terminate Message, the connection is terminated. This gets treated the same as a failure in the transport layer after RDMA is enabled, as described in Section 10.1.1.2.
If an error shows up in the RCaP layer at the Remote Peer and it is able to send a Terminate Message, the RCaP layer at the Remote Peer is responsible for terminating the connection. After the local RCaP layer tells the iSER layer that the connection is terminated, the iSER layer MUST notify the iSCSI layer by invoking the Connection_Terminate_Notify Operational Primitive. Before invoking the Connection_Terminate_Notify Operational Primitive, the iSER layer MUST perform the actions described in Section 5.2.3.2.
10.1.3. Errors in the iSER Layer 🧁
The iSER layer's error handling is described in the following sections, so go read those 🤓
10.1.3.1. Insufficient Connection Resources to Support RCaP at Connection Setup 🌈
After the iSCSI layer at the initiator invokes the Allocate_Connection_Resources Operational Primitive during the iSCSI Login Negotiation Phase, if the iSER layer at the initiator fails to allocate the connection resources needed to support RCaP, it MUST return a status of failure to the iSCSI layer at the initiator. The iSCSI layer at the initiator MUST terminate the connection as described in Section 5.2.3.1.
After the iSCSI layer at the target invokes the Allocate_Connection_Resources Operational Primitive during the iSCSI Login Negotiation Phase, if the iSER layer at the target fails to allocate the connection resources needed to support RCaP, it MUST return a status of failure to the iSCSI layer at the target. The iSCSI layer at the target MUST send a Login Response with a status class of 3 (Target Error), and a status code of "0302" (Out of Resources). The iSCSI layers at the initiator and the target MUST terminate the connection as described in Section 5.2.3.1.
10.1.3.2. iSER Negotiation Failures 🙅♀️
If the RCaP or iSER related parameters the initiator declared in the iSER Hello Message are unacceptable to the iSER layer at the target, the iSER layer at the target MUST set the Reject (REJ) flag, as described in Section 9.4, in the iSER HelloReply Message. The iSER layer MUST set the REJ flag to one in the HelloReply Message in these cases:
- The initiator-declared iSER-IRD value is greater than 0 and the target-declared iSER-ORD value is 0.
- The initiator-supported and the target-supported iSER protocol versions do not overlap.
After it asks the RCaP layer to send the iSER HelloReply Message, the handling of the error is the same as for iSER format errors, as described in Section 10.1.3.3.
10.1.3.3. iSER Format Errors 📝
These count as format errors in an iSER header:
- Illegal contents of any iSER header field
- Inconsistent field contents in an iSER header
- Length error for an iSER Hello or HelloReply Message (see Section 9.3 and 9.4)
When a format error is detected, these events MUST happen in this exact order:
- The iSER layer MUST request that the RCaP layer terminate the RCaP Stream. The RCaP layer MUST terminate the associated connection.
- The iSER layer MUST notify the iSCSI layer of the connection termination by invoking the Connection_Terminate_Notify Operational Primitive. Before invoking the Connection_Terminate_Notify Operational Primitive, the iSER layer MUST perform the actions described in Section 5.2.3.2.
10.1.3.4. iSER Protocol Errors 🚫
The first iSER Message the iSER layer at the initiator sends after switching into iSER-assisted mode MUST be the iSER Hello Message (see Section 9.3). Likewise, the first iSER Message the iSER layer at the target sends after switching into iSER-assisted mode MUST be the iSER HelloReply Message (see Section 9.4). Failing to send the iSER Hello or HelloReply Message, as shown by the wrong Opcode in the iSER header, is a protocol error. This error gets handled the same as iSER format errors, as described in Section 10.1.3.3.
If the sending side of an iSER-enabled connection acts in a way that isn't permitted by the negotiated or declared login/text operational key values described in Section 6, that's a protocol error. The receiving side MAY handle it the same as iSER format errors, as described in Section 10.1.3.3.
10.1.4. Errors in the iSCSI Layer 🌷
The error handling for iSCSI layer errors is described in the following sections. For error recovery, see Section 10.2.
10.1.4.1. iSCSI Format Errors 💌
When an iSCSI format error is detected, the iSCSI layer MUST request that the iSER layer terminate the RCaP Stream by invoking the Connection_Terminate Operational Primitive. For more details on connection termination, see Section 5.2.3.1.
10.1.4.2. iSCSI Digest Errors 🥰
In iSER-assisted mode, the iSCSI layer won't see any digest errors, because both the HeaderDigest and the DataDigest keys are negotiated to "None".
10.1.4.3. iSCSI Sequence Errors ⏩
For Traditional iSCSI, sequence errors happen when PDUs get dropped because of header or data digest errors. Since digests aren't used in iSER-assisted mode, and the RCaP layer delivers all messages in the order they were sent, sequence errors won't occur in iSER-assisted mode.
10.1.4.4. iSCSI Protocol Error 🙃
When the iSCSI layer handles certain protocol errors by dropping the connection, the error handling is the same as for iSCSI format errors, as described in Section 10.1.4.1.
When the iSCSI layer uses the iSCSI Reject PDU and response codes to handle certain other protocol errors, the iSER layer doesn't need any special handling.
10.1.4.5. SCSI Timeouts and Session Errors ⏰
SCSI Timeouts and Session Errors are handled at the iSCSI layer, so the iSER layer doesn't need any special handling for them.
10.1.4.6. iSCSI Negotiation Failures 🧨
For negotiation failures during the Login Phase at the initiator, after the iSCSI layer has invoked the Allocate_Connection_Resources Operational Primitive and before the Enable_Datamover Operational Primitive has been invoked, the iSCSI layer MUST request that the iSER layer deallocate all connection resources by invoking the Deallocate_Connection_Resources Operational Primitive. The iSCSI layer at the initiator MUST terminate the connection.
For negotiation failures during the Login Phase at the target, the iSCSI layer can use a Login Response with a status class other than 0 (success) to end the Login Phase. If the iSCSI layer has invoked the Allocate_Connection_Resources Operational Primitive before the Enable_Datamover Operational Primitive has been invoked, the iSCSI layer at the target MUST request that the iSER layer at the target deallocate all connection resources by invoking the Deallocate_Connection_Resources Operational Primitive. The iSCSI layer at both the initiator and the target MUST terminate the connection.
During the iSCSI Login Phase, if the iSCSI layer at the initiator receives a Login Response from the target with a status class other than 0 (Success) after the iSCSI layer at the initiator has invoked the Allocate_Connection_Resources Operational Primitive, the iSCSI layer MUST request that the iSER layer deallocate all connection resources by invoking the Deallocate_Connection_Resources Operational Primitive. The iSCSI layer MUST terminate the connection in this case.
For negotiation failures during the Full Feature Phase, the error handling is left to the iSCSI layer, and the iSER layer doesn't need any special handling.
10.2. Error Recovery ✨
Okay so, error recovery for iSCSI/iSER is exactly the same as it is for Traditional iSCSI, bestie 🫶 Nothing new to learn! All three ErrorRecoveryLevels from [RFC3720] are supported in iSCSI/iSER, and here's how each one works:
- ErrorRecoveryLevel 0: session recovery is handled by iSCSI, and the iSER layer doesn't need any special handling 💅
- ErrorRecoveryLevel 1: see Section 10.2.1 on PDU Recovery 🔁
- ErrorRecoveryLevel 2: see Section 10.2.2 on Connection Recovery 🦋
Also, the iSCSI layer can invoke the Notice_Key_Values Operational Primitive during connection setup. This tells the iSER layer to take note of the value of the operational ErrorRecoveryLevel, which is all covered in Sections 5.1.1 and 5.1.2 🌸
10.2.1. PDU Recovery 🧚
Remember how Sections 10.1.4.2 and 10.1.4.3 talked about digest and sequence errors? Well, in the iSER-assisted mode they just won't happen 🙈 And if the RCaP layer spots an error, it closes the iSCSI/iSER connection, as described in Section 10.1.2. So PDU recovery isn't useful in the iSER-assisted mode. No cap!
Initiator tip (it's a rule, not a suggestion): the iSCSI layer at the initiator SHOULD disable iSCSI timeout-driven PDU retransmissions. 💖
10.2.2. Connection Recovery 🔥
This part is giving main character energy 👑 The iSCSI layer at the initiator MAY reassign connection allegiance for non-immediate commands that are still in progress and are tied to the failed connection. It does this with a Task Management Function Request using the TASK REASSIGN function. See Section 7.3.3 for more details ✨
Now, if the initiator reassigns a task for a SCSI write command, it MUST qualify the Send_Control Operational Primitive invocation with DataDescriptorOut. That defines the I/O Buffer for both the non-immediate unsolicited data and the solicited data. This is what lets the iSCSI layer at the target use recovery R2Ts to request data that was originally sent as unsolicited and solicited from the initiator. Iconic 😭
Next up, the target side:
- When the iSCSI layer at the target accepts a reassignment request for a SCSI read command, it MUST request that the iSER layer process SCSI Data-in for all unacknowledged data by invoking the
Put_DataOperational Primitive. Check out Section 7.3.5 for how SCSI Data-in is handled 🍓 - When the iSCSI layer at the target accepts a reassignment request for a SCSI write command, it MUST request that the iSER layer process a recovery R2T for any non-immediate unsolicited data and any solicited data sequences that haven't been received, by invoking the
Get_DataOperational Primitive. See Section 7.3.6 for the handling of Ready To Transfer (R2T) 💜
Important, bestie: the iSCSI layer at the target MUST NOT issue recovery R2Ts on an iSCSI/iSER connection for a task whose connection allegiance was never reassigned. 🚫 The iSER layer at the target MAY reject such a recovery R2T that arrives via the Get_Data Operational Primitive invocation from the iSCSI layer at the target, with an appropriate error code 💀
Last but not least, the iSER layer at the target will process the requests invoked by the Put_Data and Get_Data Operational Primitives for a reassigned task in the same way it did for the original commands. Consistent behavior, periodt 💅
11. Security Considerations 🔐
Okay, security time, and this one's serious 😭 When iSER is layered on top of an RCaP layer and provides the RDMA extensions to the iSCSI protocol, the security considerations of iSER are the same as those of the underlying RCaP layer. For iWARP, this is described in [RDMAP] and [RDDPSEC].
Here's the thing though: the iSER-assisted iSCSI protocol is still functionally iSCSI from a security perspective. So all of the iSCSI security requirements described in [RFC3720] and [RFC3723] apply. If the IPsec [IPSEC] mechanism is used, it MUST be established before the connection transitions to the iSER-assisted mode. 🔒
If iSER is layered on top of a non-IP based RCaP layer, then all the security protocol mechanisms that apply to that RCaP layer also apply to an iSCSI/iSER connection. And if iSER is layered on top of a non-IP protocol, the IPsec mechanism specified in [RFC3720] MUST be implemented at any point where the iSER protocol enters the IP network (for example, via gateways). The non-IP protocol SHOULD implement (optional to use) a packet-by-packet security protocol equal in strength to the IPsec mechanism specified by [RFC3720] 🛡️
Denial-of-service alert 🚨 To minimize the potential for a denial-of-service attack, the iSCSI layer MUST NOT request that the iSER layer allocate the connection resources necessary to support RCaP until the iSCSI layer is far enough along in the iSCSI Login Phase that it's reasonably certain the peer side is not an attacker. See Sections 5.1.1 and 5.1.2 for details 🙅♀️
Note that the IPsec requirements for this document are based on the version of IPsec specified in RFC 2401 [IPSEC] and related RFCs, as profiled by RFC 3723 [RFC3723]. That's true even though there's a newer version of IPsec, specified in RFC 4301 [RFC4301] and related RFCs ✨
12. References 📚
12.1. Normative References
[RFC3720]Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and E. Zeidner, "Internet Small Computer Systems Interface (iSCSI)", RFC 3720, April 2004. 🦋[RFC3723]Aboba, B., Tseng, J., Walker, J., Rangan, V., and F. Travostino, "Securing Block Storage Protocols over IP", RFC 3723, April 2004. 🔐[RDMAP]Recio, R., Culley, P., Garcia, D., Hilland, J., and B. Metzler, "A Remote Direct Memory Access Protocol Specification", RFC 5040, October 2007. 🌈[DDP]Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct Data Placement over Reliable Transports", RFC 5041, October 2007. 🍓[IPSEC]Kent, S. and R. Atkinson, "Security Architecture for the Internet Protocol", RFC 2401, November 1998. 🔒[MPA]Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, "Marker PDU Aligned Framing for TCP Specification", RFC 5044, October 2007. 🧚[RDDPSEC]Pinkerton, J. and E. Deleganes, "Direct Data Placement Protocol (DDP) / Remote Direct Memory Access Protocol (RDMAP) Security", RFC 5042, October 2007. 💜[TCP]Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981. 🌸[RFC2119]Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. ⭐
12.2. Informative References
[SAM2]T10/1157D, SCSI Architecture Model - 2 (SAM-2) 💅[DA]Chadalapaka, M., Hufferd, J., Satran, J., and H. Shah, "DA: Datamover Architecture for the Internet Small Computer System Interface (iSCSI)", RFC 5047, October 2007. 🎀[IB]InfiniBand Architecture Specification Volume 1 Release 1.2, October 2004 ✨[IPoIB]Chu, J. and V. Kashyap, "Transmission of IP over InfiniBand (IPoIB)", RFC 4391, April 2006. 🦋[RFC4301]Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, December 2005. 🔐
Appendix A. iWARP Message Format for iSER 🎀
Heads up, this section is for information only and is NOT part of the standard 🙈 It just shows what the iWARP Message format looks like for the various iSER Messages when the transport layer is TCP.
A.1. iWARP Message Format for iSER Hello Message 🧚
The following figure shows an iSER Hello Message wrapped up in an iWARP SendSE Message 🎁
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0010b | Zeros | 0001b | 0001b | iSER-IRD |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 6. SendSE Message Containing an iSER Hello Message
A.2. iWARP Message Format for iSER HelloReply Message 🌈
Here's the iSER HelloReply Message, wrapped in an iWARP SendSE Message. The Reject (REJ) flag is set to 0 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0011b |Zeros|0| 0001b | 0001b | iSER-ORD |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 7. SendSE Message Containing an iSER HelloReply Message
A.3. iWARP Message Format for SCSI Read Command PDU 🎀
okay bestie, this figure shows a SCSI Read Command PDU tucked inside an iSER Message, which is then wrapped up in an iWARP SendSE Message 🎁 for this example, the iSER header has the Write STag Valid flag set to zero and the Read STag Valid flag set to one. The Write STag field is all zeros, and the Read STag field holds a valid Read STag ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0001b |0|1| All zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Read STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SCSI Read Command PDU |
// //
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 8. SendSE Message Containing a SCSI Read Command PDU 💖
A.4. iWARP Message Format for SCSI Read Data 🌸
here's an iWARP RDMA Write Message carrying SCSI Read data in its payload 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Sink STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Sink Tagged Offset |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SCSI Read data |
// //
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 9. RDMA Write Message Containing SCSI Read Data ✨
A.5. iWARP Message Format for SCSI Write Command PDU 🧚
this one is a SCSI Write Command PDU inside an iSER Message, encapsulated in an iWARP SendSE Message 💅 in this example, the Write STag Valid flag is set to one and the Read STag Valid flag is set to zero. The Write STag field has a valid Write STag, and the Read STag field is all zeros since it isn't used ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0001b |1|0| All zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Write STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SCSI Write Command PDU |
// //
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 10. SendSE Message Containing a SCSI Write Command PDU 🔥
A.6. iWARP Message Format for RDMA Read Request 👑
an iSCSI R2T gets transformed into an iWARP RDMA Read Request Message, which is shown in the figure below 🌈
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved (Not Used) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (RDMA Read Request) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (RDMA Read Request) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (RDMA Read Request) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Sink STag (SinkSTag) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Data Sink Tagged Offset (SinkTO) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RDMA Read Message Size (RDMARDSZ) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Source STag (SrcSTag) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Data Source Tagged Offset (SrcTO) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 11. RDMA Read Request Message 🍓
A.7. iWARP Message Format for Solicited SCSI Write Data 💜
this figure shows an iWARP RDMA Read Response Message carrying the solicited SCSI Write data in its payload 🫶
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Sink STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Sink Tagged Offset |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SCSI Write Data |
// //
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 12. RDMA Read Response Message Containing SCSI Write Data 🌸
A.8. iWARP Message Format for SCSI Response PDU 🍓
the last one here is a SCSI Response PDU embedded in an iSER Message, which is encapsulated in an iWARP SendInvSE Message 🎀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Invalidate STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0001b |0|0| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SCSI Response PDU |
// //
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 13. SendInvSE Message Containing SCSI Response PDU 🧚
Appendix B. Architectural Discussion of iSER over InfiniBand 🧚
okay so this section is the big-picture tour of how an InfiniBand network with gateways is put together 🦋 it's informational only, but it's giving insight into how iSER actually gets used in an IB environment 💅
B.1. The Host Side of the iSCSI and iSER Connections in InfiniBand 🌸
Figure 14 lays out the topologies where iSCSI and iSER are able to run on an InfiniBand Network ✨
+---------+ +---------+ +---------+ +---------+ +--- -----+
| Host | | Host | | Host | | Host | | Host |
| | | | | | | | | |
+---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+
|HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA|
+-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+
|----+------|-----+-----|-----+-----|-----+-----|-----+---> To IB
IB| IB | IB | IB | IB | SubNet2 SWTCH
+-v-----------v-----------v-----------v-----------v---------+
| InfiniBand Switch for Subnet1 |
+---+-----+--------+-----+--------+-----+------------v------+
| TCA | | TCA | | TCA | |
+-----+ +-----+ +-----+ | IB
/ IB \ / IB \ / \ +--+--v--+--+
| iSER | | iSER | | IPoIB | | | TCA | |
| Gateway | | Gateway | | Gateway | | +-----+ |
| to | | to | | to | | Storage |
| iSCSI | | iSER | | IP | | Controller|
| TCP | | iWARP | |Ethernet | +-----+-----+
+---v-----| +---v-----| +----v----+
| EN | EN | EN
+--------------+---------------+----> to IP based storage
Ethernet links that carry iSCSI or iWARPFigure 14. iSCSI and iSER on IB
In Figure 14, the Host systems hook up through the InfiniBand Host Channel Adapters (HCAs) to the InfiniBand links 🔥 with some IB switch(es) in the mix, those links reach the InfiniBand Target Channel Adapters (TCAs) sitting in gateways or Storage Controllers. An iSER-capable IB-IP Gateway turns the iSER Messages wrapped in IB protocols into either plain iSCSI or iSER Messages for iWARP 🧚. An [IPoIB] Gateway converts the InfiniBand [IPoIB] protocol to IP protocol, and in the iSCSI case it lets iSCSI run on an IB Network between the Hosts and the [IPoIB] Gateway 🫶
B.2. The Storage Side of the iSCSI and iSER Mixed Network Environment 💜
Figure 15 shows a storage controller with three different portal groups: one that only does iSCSI (TPG-4), one that does iSER/iWARP or iSCSI (TPG-2), and one that does iSER/IB (TPG-1) 🎀
| | |
| | |
+--+--v--+----------+--v--+----------+--v--+--+
| | IB | |iWARP| | EN | |
| | | | TCP | | NIC | |
| |(TCA)| | RNIC| | | |
| +-----| +-----+ +-----+ |
| TPG-1 TPG-2 TPG-4 |
| 9.1.3.3 9.1.2.4 9.1.2.6 |
| |
| Storage Controller |
| |
+---------------------------------------------+Figure 15. Storage Controller with TCP, iWARP, and IB Connections
The usual iSCSI portal group advertising processes (through the Service Location Protocol (SLP), the Internet Storage Name Service (iSNS), or SendTargets) are all available to a Storage Controller 🦋
B.3. Discovery Processes for an InfiniBand Host ✨
An InfiniBand Host system can grab portal group IP addresses from SLP, iSNS, or the SendTargets discovery processes by running TCP/IP over [IPoIB] 🌈 once it has one or more remote portal IP addresses, the Initiator uses the standard IP mechanisms to resolve the address to a local outgoing interface and the destination hardware address (an Ethernet MAC, or the IB GID of the target or of a gateway leading to the target) 💅 if the resolved interface is an [IPoIB] network interface, then the target portal can be reached over an InfiniBand fabric. In that case the Initiator can set up an iSCSI/TCP or iSCSI/iSER session with the Target over that InfiniBand interface, using the Hardware Address (InfiniBand GID) it got through the standard Address Resolution (ARP) processes 🧚
If more than one IP address comes back from the discovery process, the Initiator should pick a Target IP address that's on the same IP subnet as the Initiator, if one exists 👑 this avoids the extra overhead of going through a gateway when a direct path exists.
Also, a user can set up manual static IP route entries if a particular path to the target is preferred 🍓
B.4. IBTA Connection Specifications 🦋
The InfiniBand Trade Association (IBTA) connection specifications are outside the scope of this document, but it's expected that the IBTA has or will define:
- The iSER ServiceID.
- A Means for permitting a Host to establish a connection with a peer InfiniBand end-node, and to fall back to iSCSI/TCP over [IPoIB] if that peer indicates iSER is not supported.
- A Means for permitting the Host to establish connections with IB iSER connections on storage controllers or IB iSER connected Gateways in preference to [IPoIB] connected Gateways/Bridges or connections to Target Storage Controllers that also accept iSCSI via [IPoIB].
- A Means for combining the IB ServiceID for iSER and the IP port number such that the IB Host can use normal IB connection processes, yet ensure that the iSER target peer can actually connect to the required IP port number.
Acknowledgments ✨
This protocol was developed by a design team that, along with the authors, included Dwight Barron (HP), John Carrier (formerly from Adaptec), Ted Compton (EMC), Paul R. Culley (HP), Yaron Haviv (Voltaire), Jeff Hilland (HP), Mike Krause (HP), Alex Nezhinsky (Voltaire), Jim Pinkerton (Microsoft), Renato J. Recio (IBM), Julian Satran (IBM), Tom Talpey (Network Appliance), and Jim Wendt (HP). Special thanks to David Black (EMC) for his extensive review comments 🫶💖
Author's Address 💌
Mallikarjun Chadalapaka
Hewlett-Packard Company
8000 Foothills Blvd.
Roseville, CA 95747-5668, USA
Phone: +1-916-785-5621
EMail: cbm@rose.hp.com
Uri Elzur
Broadcom Corporation
5300 California Avenue
Irvine, CA 92617, USA
Phone: +1-949-926-6432
EMail: Uri@Broadcom.com
John Hufferd
Brocade Communications Systems, Inc.
1745 Technology Drive
San Jose, CA 95110, USA
Phone: +1-408-333-5244
EMail: jhufferd@brocade.com
Mike Ko
IBM Corp.
650 Harry Rd.
San Jose, CA 95120, USA
Phone: +1-408-927-2085
EMail: mako@us.ibm.com
Hemal Shah
Broadcom Corporation
5300 California Avenue
Irvine, CA 92617, USA
Phone: +1-949-926-6941
EMail: hemal@broadcom.com
Patricia Thaler
Broadcom Corporation
5300 California Avenue
Irvine, CA 92617, USA
Phone: +1-916-570-2707
EMail: pthaler@broadcom.comFull Copyright Statement 🎀
Copyright (C) The IETF Trust (2007).
This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth there, the authors keep all their rights 🌸
This document and the info inside it is provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE 💀
Intellectual Property 🌈
The IETF doesn't take a position on whether any Intellectual Property Rights or other rights that someone might claim over the technology in this document are valid or how far they reach, and it doesn't say whether any license under those rights might be available. It also doesn't claim to have made any independent effort to find such rights 🧐 Info about how rights work for RFC documents lives in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses that might be available, and the results of any attempt to get a general license or permission to use those proprietary rights for implementers or users of this specification can be found in the IETF on-line IPR repository at http://www.ietf.org/ipr 🔥
The IETF invites anyone interested to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that might cover technology needed to implement this standard. Send that info to the IETF at ietf-ipr@ietf.org 💖
📦 where this page comes from
- ietf: rfc5046 2007-10
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms