RFC 7145
Internet Small Computer System Interface (iSCSI) Extensions for the Remote Direct Memory Access (RDMA) Specification
Part 1 of 19 🎀
Okay bestie, this is RFC 7145, Internet Small Computer System Interface (iSCSI) Extensions for the Remote Direct Memory Access (RDMA) Specification 💖. It's a Standards Track doc from the IETF, so it's an actual official spec, not a vibe. The authors are M. Ko and A. Nezhinsky, both from Mellanox. It was published in April 2014 with ISSN 2070-1721. It obsoletes RFC 5046 🫶, so if you're still reading 5046, this is the one to switch to.
Abstract ✨
okay so the big idea is that iSCSI Extensions for RDMA lets iSCSI use RDMA for data transfer 🦋. It does this by layering iSCSI on top of an RDMA-Capable Protocol 💅. That protocol gives you RDMA Read and Write services, which means data can go straight into SCSI I/O Buffers with no intermediate copies, and I'm obsessed with that 😭. This doc describes the extensions to iSCSI that support those RDMA services as provided by an RDMA-Capable Protocol 🌸.
This document obsoletes RFC 5046 🍓.
Status of This Memo 🌈
This is an Internet Standards Track document 💜.
It's a product of the Internet Engineering Task Force (IETF) and it represents the consensus of the IETF community 🔥. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG) ✨. If you want more on Internet Standards, check Section 2 of RFC 5741 🧚.
For the current status of this document, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc7145 🌸.
Copyright Notice 💅
Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved 🫶.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document 💖. Please review these documents carefully, since they explain your rights and restrictions with respect to this document 🙈. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Simplified BSD License 🔥.
Table of Contents 🦋
1. Introduction ....................................................5
1.1. Motivation .................................................5
1.2. iSCSI/iSER Layering ........................................6
1.3. Architectural Goals ........................................7
1.4. Protocol Overview ..........................................7
1.5. RDMA Services and iSER .....................................9
1.5.1. STag ................................................9
1.5.2. Send ...............................................10
1.5.3. RDMA Write .........................................11
1.5.4. RDMA Read ..........................................11
1.6. SCSI Read Overview ........................................11
1.7. SCSI Write Overview .......................................12
2. Definitions and Acronyms .......................................12
2.1. Definitions ...............................................12
2.2. Acronyms ..................................................18
2.3. Conventions ...............................................20
3. Upper-Layer Interface Requirements .............................20
3.1. Operational Primitives offered by iSER ....................21
3.1.1. Send_Control .......................................21
3.1.2. Put_Data ...........................................21
3.1.3. Get_Data ...........................................22
3.1.4. Allocate_Connection_Resources ......................22
3.1.5. Deallocate_Connection_Resources ....................23
3.1.6. Enable_Datamover ...................................23
3.1.7. Connection_Terminate ...............................23
3.1.8. Notice_Key_Values ..................................24
3.1.9. Deallocate_Task_Resources ..........................24
3.2. Operational Primitives Used by iSER .......................24
3.2.1. Control_Notify .....................................25
3.2.2. Data_Completion_Notify .............................25
3.2.3. Data_ACK_Notify ....................................25
3.2.4. Connection_Terminate_Notify ........................26
3.3. iSCSI Protocol Usage Requirements .........................26
4. Lower-Layer Interface Requirements .............................27
4.1. Interactions with the RCaP Layer ..........................27
4.2. Interactions with the Transport Layer .....................28
5. Connection Setup and Termination ...............................28
5.1. iSCSI/iSER Connection Setup ...............................28
5.1.1. Initiator Behavior .................................30
5.1.2. Target Behavior ....................................31
5.1.3. iSER Hello Exchange ................................33
5.2. iSCSI/iSER Connection Termination .........................36
5.2.1. Normal Connection Termination at the Initiator .....36
5.2.2. Normal Connection Termination at the Target ........36
5.2.3. Termination without Logout Request/Response PDUs ...37
6. Login/Text Operational Keys ....................................38
6.1. HeaderDigest and DataDigest ...............................38
6.2. MaxRecvDataSegmentLength ..................................38
6.3. RDMAExtensions ............................................39
6.4. TargetRecvDataSegmentLength ...............................40
6.5. InitiatorRecvDataSegmentLength ............................41
6.6. OFMarker and IFMarker .....................................41
6.7. MaxOutstandingUnexpectedPDUs ..............................41
6.8. MaxAHSLength ..............................................42
6.9. TaggedBufferForSolicitedDataOnly ..........................43
6.10. iSERHelloRequired ........................................43
7. iSCSI PDU Considerations .......................................44
7.1. iSCSI Data-Type PDU .......................................44
7.2. iSCSI Control-Type PDU ....................................45
7.3. iSCSI PDUs ................................................45
7.3.1. SCSI Command .......................................45
7.3.2. SCSI Response ......................................47
7.3.3. Task Management Function Request/Response ..........49
7.3.4. SCSI Data-out ......................................50
7.3.5. SCSI Data-in .......................................51
7.3.6. Ready To Transfer (R2T) ............................53
7.3.7. Asynchronous Message ...............................55
7.3.8. Text Request and Text Response .....................55
7.3.9. Login Request and Login Response ...................55
7.3.10. Logout Request and Logout Response ................56
7.3.11. SNACK Request .....................................56
7.3.12. Reject ............................................56
7.3.13. NOP-Out and NOP-In ................................57
8. Flow Control and STag Management ...............................57
8.1. Flow Control for RDMA Send Messages .......................57
8.1.1. Flow Control for Control-Type PDUs from the
Initiator ..........................................58
8.1.2. Flow Control for Control-Type PDUs from the
Target .............................................60
8.2. Flow Control for RDMA Read Resources ......................61
8.3. STag Management ...........................................62
8.3.1. Allocation of STags ................................62
8.3.2. Invalidation of STags ..............................62
9. iSER Control and Data Transfer .................................64
9.1. iSER Header Format ........................................64
9.2. iSER Header Format for iSCSI Control-Type PDU .............65
9.3. iSER Header Format for iSER Hello Message .................67
9.4. iSER Header Format for iSER HelloReply Message ............68
9.5. SCSI Data Transfer Operations .............................69
9.5.1. SCSI Write Operation ...............................69
9.5.2. SCSI Read Operation ................................70
9.5.3. Bidirectional Operation ............................70
10. iSER Error Handling and Recovery ..............................71
10.1. Error Handling ...........................................71
10.1.1. Errors in the Transport Layer .....................71
10.1.2. Errors in the RCaP Layer ..........................72
10.1.3. Errors in the iSER Layer ..........................73
10.1.4. Errors in the iSCSI Layer .........................75
10.2. Error Recovery ...........................................76
10.2.1. PDU Recovery ......................................77
10.2.2. Connection Recovery ...............................77
11. Security Considerations .......................................78
12. IANA Considerations ...........................................79
13. References ....................................................79
13.1. Normative References .....................................79
13.2. Informative References ...................................80
Appendix A. Summary of Changes from RFC 5046 ......................81
Appendix B. Message Format for iSER ...............................83
B.1. iWARP Message Format for iSER Hello Message ..................83
B.2. iWARP Message Format for iSER HelloReply Message .............84
B.3. iSER Header Format for SCSI Read Command PDU .................85
B.4. iSER Header Format for SCSI Write Command PDU ................86
B.5. iSER Header Format for SCSI Response PDU .....................87
Appendix C. Architectural discussion of iSER over InfiniBand ......88
C.1. Host Side of iSCSI and iSER Connections in InfiniBand ........88
C.2. Storage Side of iSCSI and iSER Mixed Network Environment .....89
C.3. Discovery Processes for an InfiniBand Host ...................89
C.4. IBTA Connection Specifications ...............................90
Appendix D. Acknowledgments .......................................90Table of Figures 🧚
Figure 1. Example of iSCSI/iSER Layering in Full Feature Phase .....6
Figure 2. iSER Header Format ......................................64
Figure 3. iSER Header Format for iSCSI Control-Type PDU ...........65
Figure 4. iSER Header Format for iSER Hello Message ...............67
Figure 5. iSER Header Format for iSER HelloReply Message ..........68
Figure 6. SendSE Message Containing an iSER Hello Message .........83
Figure 7. SendSE Message Containing an iSER HelloReply Message ....84
Figure 8. iSER Header Format for SCSI Read Command PDU ............85
Figure 9. iSER Header Format for SCSI Write Command PDU ...........86
Figure 10. iSER Header Format for SCSI Response PDU ...............87
Figure 11. iSCSI and iSER on IB ...................................88
Figure 12. Storage Controller with TCP, iWARP, and IB Connections .891. Introduction ✨
okay the actual Introduction text is coming up in part 2, so stay tuned bestie 🫶💖
1.1. Motivation 💖
Okay so, the iSCSI protocol ([iSCSI]) is basically a mapping of the SCSI Architecture Model (see [SAM5] and [iSCSI-SAM]) over TCP 🌸. SCSI commands ride inside iSCSI requests, SCSI responses and status come back in iSCSI responses, and other iSCSI protocol exchanges plus SCSI Data also travel in iSCSI PDUs ✨.
Now here's the tea. In the Traditional iSCSI model, out-of-order TCP segments have to be stored and reassembled before the iSCSI layer on an end node can place the data into iSCSI buffers 😭. Reassembly is needed because not every TCP segment is likely to carry an iSCSI header that enables placement, and TCP itself has no built-in way of signaling ULP (Upper Level Protocol) message boundaries to help place out-of-order segments. At high network speeds, that TCP reassembly is quite counterproductive, and it's giving inefficiency 💀. The reasons are: wasted memory bandwidth in data copying, the need for reassembly memory, wasted CPU cycles in data copying, and the general store-and-forward latency from an application perspective.
The generic term RDMA-Capable Protocol (RCaP) is used to refer to protocol stacks that provide Remote Direct Memory Access (RDMA) functionality, like iWARP and InfiniBand 🦋.
With RDMA-Capable Controllers now available inside host systems, it makes total sense for iSCSI to exploit the direct data placement function of those controllers, just like other applications do 🫶.
iSCSI Extensions for RDMA (iSER) is designed precisely to take advantage of generic RDMA technologies 👑. iSER's goal is to let iSCSI use direct data placement and RDMA capabilities through a generic RDMA-Capable Controller. In summary, the iSCSI/iSER protocol stack is built to scale to high speeds by relying on a generic data placement process and on RDMA technologies and products that enable direct data placement of both in-order and out-of-order data 🔥.
This document describes iSER as a protocol extension to iSCSI, both for ease of description and because it's true in a very strict protocol sense 🎀. That said, iSER is really extending the connectivity of the iSCSI protocol defined in [iSCSI], and the name "iSER" reflects this reality 🌈.
When the rest of the document means the iSCSI protocol as defined in [iSCSI] (so, without the iSER enhancements), it uses the term "Traditional iSCSI" to keep things crystal clear 💅.
This document obsoletes RFC 5046. See Appendix A for the list of changes from RFC 5046. 🙈
1.2. iSCSI/iSER Layering 🧚
iSCSI Extensions for RDMA (iSER) sits between the iSCSI layer and the RCaP layer 🌸.
+--------------------------------------------------------+
| SCSI |
+--------------------------------------------------------+
| iSCSI |
DI -> +--------------------------------------------------------+
| iSER |
+-------+--------------------------+---------------------+
| RDMAP | | |
+-------+ InfiniBand | |
| DDP | Reliable | Other |
+-------+ Connected | RDMA |
| MPA | Transport | Capable |
+-------+ Service | Protocol |
| TCP | | |
+-------+--------------------------+---------------------+
| IP | InfiniBand Network Layer | Other Network Layer |
+-------+--------------------------+---------------------+
Figure 1: Example of iSCSI/iSER Layering in Full Feature PhaseFigure 1 shows how SCSI, iSCSI, iSER, and the different RCaP layers relate to each other 🦋. For TCP, the RCaP is iWARP. For InfiniBand, the RCaP is the Reliable Connected Transport Service. Also note that the iSCSI layer, as described here, supports the RDMA Extensions as used in iSER ✨.
1.3. Architectural Goals 🎀
This section sums up the architectural goals that guided the design of iSER 💖.
- Provide an RDMA data transfer model for iSCSI that enables direct in-order or out-of-order data placement of SCSI data into pre-allocated SCSI buffers while maintaining in-order data delivery. 🚀
- Do not require any major changes to the SCSI Architecture Model [SAM5] and SCSI command set standards. 💅
- Utilize the existing iSCSI infrastructure (sometimes referred to as "iSCSI ecosystem") including but not limited to MIB, bootstrapping, negotiation, naming and discovery, and security. 🫶
- Enable a session to operate in the Traditional iSCSI data transfer mode if iSER is not supported by either the initiator or the target. (Do not require iSCSI Full Feature Phase interoperability between an end node operating in Traditional iSCSI mode and an end node operating in iSER-assisted mode.) 🌈
- Allow initiator and target implementations to utilize generic RDMA-Capable Controllers such as RNICs, or to implement iSCSI and iSER in software. (Do not require iSCSI- or iSER-specific assists in the RCaP implementation or RDMA-Capable Controller.) 👑
- Implement a lightweight Datamover protocol for iSCSI with minimal state maintenance. 🧚
1.4. Protocol Overview 🌸
Staying consistent with the architectural goals from Section 1.3, the iSER protocol does not require changes in the iSCSI ecosystem or any related SCSI specifications 🙌. It defines the mapping of iSCSI PDUs to RCaP Messages in a way that makes it entirely feasible to build iSCSI/iSER implementations on generic RDMA-Capable Controllers 🔥. The iSER protocol layer keeps state maintenance to a minimum while helping a connection during the iSCSI Full Feature Phase, and it's totally oblivious to the notion of an iSCSI session 🙈. The crucial aspects of iSER can be summed up like this:
- iSER-assisted mode is negotiated during the iSCSI login in the leading connection for each session, and an entire iSCSI session can only operate in one mode. (That means a connection in a session cannot operate in iSER-assisted mode if a different connection of the same session is already in Full Feature Phase in the Traditional iSCSI mode.) 💜
- Once in iSER-assisted mode, all iSCSI interactions on that connection use RCaP Messages. ✨
- A Send Message is used for carrying an iSCSI control-type PDU preceded by an iSER header. See Section 7.2 for more details on iSCSI control-type PDUs. 📬
- RDMA Write, RDMA Read Request, and RDMA Read Response Messages are used for carrying control and all data information associated with the iSCSI data-type PDUs (i.e., SCSI Data-In PDUs and R2T PDUs). iSER does not use SCSI Data-Out PDUs for solicited data, and SCSI Data-Out PDUs for unsolicited data are not treated as iSCSI data-type PDUs by iSER because RDMA is not used. See Section 7.1 for more details on iSCSI data-type PDUs. 🦋
- The target drives all data transfer (with the exception of iSCSI unsolicited data) for SCSI writes and SCSI reads, by issuing RDMA Read Requests and RDMA Writes, respectively. 🎯
- RCaP is responsible for ensuring data integrity. (For example, iWARP includes a CRC-enhanced framing layer called MPA on top of TCP; and for InfiniBand, the CRCs are included in the Reliable Connection mode.) For this reason, iSCSI header and data digests are negotiated to "None" for iSCSI/iSER sessions. 🛡️
- The iSCSI error recovery hierarchy defined in [iSCSI] is fully supported by iSER. (However, see Section 7.3.11 on the handling of SNACK Request PDUs.) 🧯
- iSER requires no changes to iSCSI security and text mode negotiation mechanisms. 🔐
Note that Traditional iSCSI implementations may have to be adapted to employ iSER 💅. That adaptation, when required, is likely to center on the upper-layer interface requirements of iSER (Section 3) 🫶.
1.5. RDMA Services and iSER 🌈
iSER is designed to work with software and/or hardware protocol stacks that provide the protocol services defined in RCaP documents such as [RDMAP], [IB], etc. 🧚 The following subsections describe the key protocol elements of RCaP services that iSER relies on.
1.5.1. STag 👑
An STag is the identifier of an I/O Buffer, unique to an RDMA-Capable Controller, that the iSER layer Advertises to the remote iSCSI/iSER node in order to complete a SCSI I/O 🎀.
In iSER, Advertisement is the act of the initiator informing the target that an I/O Buffer is available at the initiator for RDMA Read or RDMA Write access by the target 💖. The initiator Advertises the I/O Buffer by including the STag and the Base Offset in the header of an iSER Message that carries the SCSI Command PDU to the target. The buffer length is as specified in the SCSI Command PDU 🌸.
The iSER layer at the initiator Advertises the STag and the Base Offset for the I/O Buffer of each SCSI I/O to the iSER layer at the target, in the iSER header of a Send Message containing the SCSI Command PDU, unless the I/O can be completely satisfied by unsolicited data alone 📬. The SendSE Message should be used if supported by the RCaP layer (e.g., iWARP) 🦋.
The iSER layer at the target provides the STag for the I/O Buffer that is the Data Sink of an RDMA Read Operation (Section 1.5.4) to the RCaP layer on the initiator node. In other words, this is completely transparent to the iSER layer at the initiator ✨.
The iSER layer at the initiator SHOULD invalidate the Advertised STag upon a normal completion of the associated task 🧹. The Send with Invalidate Message, if supported by the RCaP layer (e.g., iWARP), can be used for automatic invalidation when it is used to carry the SCSI Response PDU 🙌. There are two exceptions to this automatic invalidation: bidirectional commands and abnormal completion of a command. In these two cases, the iSER layer at the initiator SHOULD explicitly invalidate the STag 🙈. That iSER layer MUST check that STag invalidation has occurred whenever receipt of a Send with Invalidate message is the expected means of causing an STag to be invalidated, and it MUST perform the STag invalidation if the STag has not already been invalidated (e.g., because a Send Message was used instead of Send with Invalidate) 🚨.
If the Advertised STag is not invalidated as recommended in the foregoing paragraph (e.g., in order to cache the STag for future reuse), the I/O Buffer remains exposed to the network for access by the RCaP. Such an I/O Buffer is capable of being read or written by the RCaP outside the scope of the iSCSI operation for which it was originally established. This fact has both robustness and security considerations 😭. The robustness considerations are that the system containing the iSER initiator may react poorly to an unexpected modification of its memory. For the security considerations, see Section 11. 💀
1.5.2. Send ✨
Okay so Send is the RDMA Operation that's not addressed to an Advertised buffer 📦, and it uses Untagged buffers as the message gets received 📬. Main character energy, honestly 💅
The iSER layer at the initiator uses the Send Operation to transmit any iSCSI control-type PDU to the target 🎀 For example, the initiator uses Send Operations to deliver iSER Messages carrying SCSI Command PDUs to the iSER layer at the target.
The iSER layer at the target uses the Send Operation to transmit any iSCSI control-type PDU to the initiator 🌸 For example, the target uses Send Operations to deliver iSER Messages carrying SCSI Response PDUs to the iSER layer at the initiator.
For interop 🤝, iSER implementations SHOULD accept and correctly process SendSE and SendInvSE messages. BUT, SendSE and SendInvSE are basically treated as optimizations or enhancements on top of the basic Send Message, and whether they're supported can vary by RCaP protocol and by specific implementation 🤷♀️ In general, these messages SHOULD NOT be used, unless the RCaP requires support for them in all implementations. If these messages are used, the implementation SHOULD be capable of falling back to Send so it can work with a receiver that doesn't support them 🔄 Trying to use these messages with a peer that doesn't support them may result in a fatal error that closes the RCaP connection 💀 For example, these messages SHOULD NOT be used with the InfiniBand RCaP, because InfiniBand doesn't require support for them in all cases. New iSER implementations SHOULD use Send (and not SendSE or SendInvSE) unless there are compelling reasons to do otherwise ✨ Similarly, iSER implementations SHOULD NOT rely on events triggered by SendSE and SendInvSE, since these messages may not be used 🙈
1.5.3. RDMA Write 🦋
RDMA Write is the RDMA Operation used to place data into an Advertised buffer at the Data Sink 📥 The Data Source addresses the Message using an STag and a Tagged Offset that are valid on the Data Sink, so it knows exactly where to drop things 🎯
The iSER layer at the target uses the RDMA Write Operation to move the contents of a local I/O Buffer into an Advertised I/O Buffer at the initiator 🚚 It uses RDMA Write to transfer all or part of the data needed to complete a SCSI Read command 🍓
The iSER layer at the initiator does not employ RDMA Writes 🙅♀️
1.5.4. RDMA Read 🧚
RDMA Read is the RDMA Operation used to pull data out of an Advertised buffer at the Data Source 📤 The sender of the RDMA Read Request addresses the Message using an STag and a Tagged Offset that are valid on the Data Source, plus a valid local STag and Tagged Offset that identify the Data Sink 🫶
The iSER layer at the target uses the RDMA Read Operation to move the contents of an Advertised I/O Buffer at the initiator into a local I/O Buffer at the target 📦 It uses RDMA Read to fetch whole or part of the data needed to complete a SCSI Write Command 💜
The iSER layer at the initiator does not employ RDMA Reads 🌈
1.6. SCSI Read Overview 📖
The iSER layer at the initiator gets the SCSI Command PDU from the iSCSI layer 📬 Then it generates an STag for the I/O Buffer of the SCSI Read and Advertises the buffer by including the STag and the Base Offset as part of the iSER header for the PDU 🎀 The iSER Message goes to the target using a Send Message. The SendSE Message should be used if the RCaP layer supports it (e.g., iWARP) 💅
The iSER layer at the target uses one or more RDMA Writes to transfer the data needed to complete the SCSI Read 🚚
The iSER layer at the target uses a Send Message to send the SCSI Response PDU back to the iSER layer at the initiator 💌 The iSER layer at the initiator then invalidates the STag and tells the iSCSI layer that the SCSI Response PDU is ready 🔥 The Send with Invalidate Message, if the RCaP layer supports it (e.g., iWARP), can be used for automatic invalidation of the STag ✨
1.7. SCSI Write Overview ✍️
The iSER layer at the initiator gets the SCSI Command PDU from the iSCSI layer 📬 If solicited data transfer is involved, the iSER layer at the initiator generates an STag for the I/O Buffer of the SCSI Write and Advertises the buffer by including the STag and the Base Offset as part of the iSER header for the PDU 🎀 The iSER Message goes to the target using a Send Message. The SendSE Message should be used if the RCaP layer supports it (e.g., iWARP) 💅
The iSER layer at the initiator may optionally send one or more non-immediate unsolicited data PDUs to the target using Send Messages 🌸
If solicited data transfer is involved, the iSER layer at the target uses one or more RDMA Reads to transfer the data needed to complete the SCSI Write 💜
The iSER layer at the target uses a Send Message to send the SCSI Response PDU back to the iSER layer at the initiator 💌 The iSER layer at the initiator then invalidates the STag and tells the iSCSI layer that the SCSI Response PDU is ready 🔥 The Send with Invalidate Message, if the RCaP layer supports it (e.g., iWARP), can be used for automatic invalidation of the STag ✨
2. Definitions and Acronyms 📚
2.1. Definitions 💖
Advertisement (Advertised, Advertise, Advertisements, Advertises) -- The act of informing a remote iSER (iSCSI Extensions for RDMA) layer that a local node's buffer is available to it. A node makes a buffer available for incoming RDMA Read Request Message or incoming RDMA Write Message access by informing the remote iSER layer of the Tagged Buffer identifiers (STag, Base Offset, and buffer length). Note that this Advertisement of Tagged Buffer information is the responsibility of the iSER layer on either end and is not defined by the RDMA-Capable Protocol. A typical method would be for the iSER layer to embed the Tagged Buffer's STag, Base Offset, and buffer length in a message destined for the remote iSER layer 🦋
Base Offset - A value that, when added to the Buffer Offset, forms the Tagged Offset 🧮
Completion (Completed, Complete, Completes) - Completion is the process by which the RDMA-Capable Protocol layer tells the iSER layer that a particular RDMA Operation has performed all functions specified for that RDMA Operation ✅
Connection - A connection is a logical bidirectional communication channel between the initiator and the target, e.g., a TCP connection. Communication between the initiator and the target happens over one or more connections. The connections carry control messages, SCSI commands, parameters, and data within iSCSI Protocol Data Units (iSCSI PDUs) 🔗
Connection Handle - An information element that identifies the particular iSCSI connection and is unique for a given iSCSI layer and the underlying iSER layer. Every invocation of an Operational Primitive is qualified with the Connection Handle 🏷️
Data Sink - The peer receiving a data payload. Note that the Data Sink can be required to both send and receive RCaP (RDMA-Capable Protocol) Messages to transfer a data payload 📥
Data Source - The peer sending a data payload. Note that the Data Source can be required to both send and receive RCaP Messages to transfer a data payload 📤
Datamover Interface (DI) - The interface between the iSCSI layer and the Datamover Layer as described in [DA] 🔌
Datamover Layer - A layer that sits directly below the iSCSI layer and above the underlying transport layers. This layer exposes and uses a set of transport-independent Operational Primitives for communication between the iSCSI layer and itself. The Datamover layer, working together with the transport layers, moves the control and data information on the iSCSI connection. In this specification, the iSER layer is the Datamover layer 🚚
Datamover Protocol - A Datamover protocol is the wire protocol defined to realize the Datamover-layer functionality. In this specification, the iSER protocol is the Datamover protocol 📡
Inbound RDMA Read Queue Depth (IRD) - The maximum number of incoming outstanding RDMA Read Requests that the RDMA-Capable Controller can handle on a particular RCaP Stream at the Data Source. For some RDMA-Capable Protocol layers, the term "IRD" may be known by a different name. For example, for InfiniBand, the equivalent of IRD is the Responder Resources 🧾
I/O Buffer - A buffer used in a SCSI Read or Write operation, so SCSI data may be sent from or received into that buffer 💾
iSCSI - The iSCSI protocol as defined in [iSCSI] is a mapping of the SCSI Architecture Model of SAM-5 over TCP 🌐
iSCSI control-type PDU - Any iSCSI PDU that is not an iSCSI data-type PDU and is also not a SCSI Data-Out PDU carrying solicited data is defined as an iSCSI control-type PDU. Specifically, it is to be noted that SCSI Data-Out PDUs for unsolicited data are defined as iSCSI control-type PDUs 🎛️
iSCSI data-type PDU - An iSCSI data-type PDU is defined as an iSCSI PDU that causes data transfer via RDMA operations at the iSER layer, transparent to the remote iSCSI layer, to take place between the peer iSCSI nodes on a Full Feature Phase iSCSI connection. An iSCSI data-type PDU, when requested for transmission by the sender iSCSI layer, results in the associated data transfer without the participation of the remote iSCSI layer, i.e., the PDU itself is not delivered as-is to the remote iSCSI layer. The following iSCSI PDUs make up the set of iSCSI data-type PDUs -- SCSI Data-In PDU and R2T PDU 🧠
iSCSI Layer - A layer in the protocol stack implementation within an end node that implements the iSCSI protocol and interfaces with the iSER layer via the Datamover Interface 🧱
iSCSI PDU (iSCSI Protocol Data Unit) - The iSCSI layer at the initiator and the iSCSI layer at the target divide their communications into messages. The term "iSCSI Protocol Data Unit" (iSCSI PDU) is used for these messages 📨
iSCSI/iSER Connection - An iSER-assisted iSCSI connection. An iSCSI connection that is not iSER assisted always maps onto a TCP connection at the transport level. But an iSER-assisted iSCSI connection may not have an underlying TCP connection. For some RCaP implementations (e.g., iWARP), an iSER-assisted iSCSI connection has an underlying TCP connection. For other RCaP implementations (e.g., InfiniBand), there is no underlying TCP connection. (In the specific example of InfiniBand [IB], an iSER-assisted iSCSI connection is directly mapped onto the InfiniBand Reliable Connection-based (RC) channel 🔍)
iSCSI/iSER Session - An iSER-assisted iSCSI session. All connections of an iSCSI/iSER session are iSCSI/iSER connections 🗂️
iSER - iSCSI Extensions for RDMA, the protocol defined in this document 💜
iSER-assisted - A term generally used to describe the operation of iSCSI when the iSER functionality is also enabled below the iSCSI layer for the specific iSCSI/iSER connection in question 🌟
iSER-IRD - This variable represents the maximum number of incoming outstanding RDMA Read Requests that the iSER layer at the initiator grants on a particular RCaP Stream 🎀
- iSER-ORD 💖 This variable is the maximum number of outstanding RDMA Read Requests that the iSER layer can start on a particular RCaP Stream. Only the iSER layer at the target maintains it.
- iSER Layer 🎀 The layer that implements the iSCSI Extensions for RDMA (iSER) protocol.
- iWARP 🦋 A suite of wire protocols made up of [RDMAP], [DDP], and [MPA] when layered above [TCP]. [RDMAP] and [DDP] may also be layered above SCTP or other transport protocols.
- Local Mapping 🧚 A task state record kept by the iSER layer that links the Initiator Task Tag to the Local STag(s). The exact structure of the record is implementation dependent.
- Local Peer 🌸 The implementation of the RDMA-Capable Protocol on the local end of the connection. It's used to refer to the local entity when describing protocol exchanges or other interactions between two nodes.
- Node 💅 A computing device attached to one or more links of a network. In this context, a node doesn't mean a specific application or protocol running on the computer. A node may be made up of one or more RDMA-Capable Controllers installed in a host computer.
- Operational Primitive ✨ An Operational Primitive is an abstract functional interface procedure. It asks another layer to do a specific action on the requestor's behalf, or it tells the other layer that some event happened. The Datamover Interface between an iSCSI layer and a Datamover layer inside an iSCSI end node uses a set of Operational Primitives to define how the two layers talk to each other. Not every invocation of an Operational Primitive will get a response from the layer it was requested of. The full story on the Operational Primitive types and their request-response semantics for iSCSI and iSER is in [DA] 🔥.
- Outbound RDMA Read Queue Depth (ORD) 👑 The maximum number of outstanding RDMA Read Requests that the RDMA-Capable Controller can start on a particular RCaP Stream at the Data Sink. Some RDMA-Capable Protocol layers call ORD by a different name. For example, for InfiniBand, the equivalent of ORD is the Initiator Depth.
- Phase Collapse 🌈 This is the iSCSI optimization where the SCSI status is sent along with the final SCSI Data-In PDU from a target. See Section 4.2 in [iSCSI].
- RCaP Message 🍓 One or more network layer packets that make up a single RDMA operation, or a part of an RDMA Read Operation of the RDMA-Capable Protocol. For iWARP, an RCaP Message is called an RDMAP Message.
- RCaP Stream 🦋 A single bidirectional association between the peer RDMA-Capable Protocol layers on two nodes, running over a single transport-level stream. For iWARP, an RCaP Stream is called an RDMAP Stream, and the association gets created after a successful Login Phase where iSER support is negotiated.
- RDMA-Capable Protocol (RCaP) ⭐ The protocol or protocol suite that gives you reliable RDMA transport functionality, for example iWARP, InfiniBand, and so on.
- RDMA-Capable Controller 🔥 A network I/O adapter or embedded controller with RDMA functionality. For iWARP, this could be an RNIC. For InfiniBand, this could be an HCA (Host Channel Adapter) or a TCA (Target Channel Adapter).
- RDMA-enabled Network Interface Controller (RNIC) 💜 A network I/O adapter or embedded controller with iWARP functionality.
- RDMA Operation 🫶 A sequence of RCaP Messages, control messages included, that transfers data from a Data Source to a Data Sink. The defined RDMA Operations are the RDMA Write Operation, the RDMA Read Operation, and the Send Operation.
- RDMA Protocol (RDMAP) 🧚 A wire protocol that supports RDMA Operations to move ULP data between a Local Peer and the Remote Peer, as described in [RDMAP].
- RDMA Read Operation 🦋 An RDMA Operation that the Data Sink uses to pull the contents of a Data Source buffer from the Remote Peer into a Data Sink buffer at the Local Peer. An RDMA Read Operation is made of one RDMA Read Request Message and one RDMA Read Response Message.
- RDMA Read Request 🌸 An RCaP Message the Data Sink uses to ask the Data Source to transfer the contents of a buffer. The RDMA Read Request Message describes both the Data Source buffer and the Data Sink buffer.
- RDMA Read Response 💖 An RCaP Message the Data Source uses to send the contents of a buffer to the Data Sink, in response to an RDMA Read Request. The RDMA Read Response Message only describes the Data Sink buffer.
- RDMA Write Operation 🎀 An RDMA Operation the Data Source uses to push the contents of a Data Source buffer from the Local Peer into a Data Sink buffer at the Remote Peer. The RDMA Write Message only describes the Data Sink buffer.
- Remote Direct Memory Access (RDMA) 💅 A way of accessing memory on a remote system where the local system specifies where the data to be moved lives on the remote side. Putting an RDMA-Capable Controller in the remote system lets that access happen without interrupting the CPU(s) on that system.
- Remote Mapping 🍓 A task state record kept by the iSER layer that links the Initiator Task Tag to the Advertised STag(s) and the Base Offset(s). The exact structure of the record is implementation dependent.
- Remote Peer ✨ The implementation of the RDMA-Capable Protocol on the opposite end of the connection. It's used to refer to the remote entity when describing protocol exchanges or other interactions between two nodes.
- SCSI Layer 👑 This layer builds and receives SCSI CDBs (Command Descriptor Blocks). It sends and receives them, along with the rest of the command execute [SAM5] parameters, to and from the iSCSI layer.
- Send 🌈 An RDMA Operation that transfers the contents of a buffer from the Local Peer into an untagged buffer at the Remote Peer.
- SendInvSE Message 🔥 A Send with Solicited Event and Invalidate Message.
- SendSE Message 💜 A Send with Solicited Event Message.
- Sequence Number (SN) 🫶
DataSNfor a SCSI Data-In PDU andR2TSNfor an R2T PDU. The semantics of both types of sequence numbers are defined in [iSCSI]. - Session, iSCSI Session 🧚 The group of connections that link an initiator SCSI port with a target SCSI port form an iSCSI session. It's equivalent to a SCSI Initiator-Target (I-T) nexus. Connections can be added to and removed from a session even while the I-T nexus stays intact. Across all connections in a session, an initiator sees one and the same target.
- Steering Tag (STag) 🌸 An identifier of a Tagged Buffer on a node, Local or Remote, as defined in [RDMAP] and [DDP]. Other RDMA-Capable Protocols may call STags by different names, but here they'll be referred to as STags. For example, for InfiniBand, a Remote STag is called an R-Key and a Local STag is called an L-Key, and both count as STags.
- Tagged Buffer 💖 A buffer that gets explicitly Advertised to the iSER layer at the remote node through an exchange of an STag, a Base Offset, and a length.
- Tagged Offset 🎀 The offset within a Tagged Buffer.
- Traditional iSCSI 🍓 The iSCSI protocol as defined in [iSCSI], meaning without the iSER enhancements.
- Untagged Buffer 💅 A buffer that is not explicitly Advertised to the iSER layer at the remote node.
2.2. Acronyms ✨
Acronym Definition
--------------------------------------------------------------
AHS Additional Header Segment
BHS Basic Header Segment
CO Connection Only
CRC Cyclic Redundancy Check
DDP Direct Data Placement Protocol
DI Datamover Interface
HCA Host Channel Adapter
IANA Internet Assigned Numbers Authority
IB InfiniBand
IETF Internet Engineering Task Force
I/O Input - Output
IO Initialize Only
IP Internet Protocol
IPoIB IP over InfiniBand
IPsec Internet Protocol Security
iSER iSCSI Extensions for RDMA
ITT Initiator Task Tag
LO Leading Only
MPA Marker PDU Aligned Framing for TCP
NOP No Operation
NSG Next Stage (during the iSCSI Login Phase)
PDU Protocol Data Unit
R2T Ready To Transfer
R2TSN Ready To Transfer Sequence Number
RCaP RDMA-Capable Protocol
RDMA Remote Direct Memory Access
RDMAP Remote Direct Memory Access Protocol
RFC Request For Comments
RNIC RDMA-enabled Network Interface Controller
SAM5 SCSI Architecture Model - 5
SCSI Small Computer System Interface
SNACK Selective Negative Acknowledgment - also
Sequence Number Acknowledgement for data
STag Steering Tag
SW Session Wide
TCA Target Channel Adapter
TCP Transmission Control Protocol
TMF Task Management Function
TTT Target Transfer Tag
ULP Upper Level Protocol2.3. Conventions 💅
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 🫶.
3. Upper-Layer Interface Requirements 🎀
okay so this section is where we get into how the iSCSI layer and the iSER layer actually talk to each other 💬 it's an abstract model of the interactions they need, and it's built on the architectural model in [DA]. [DA] also gives a functional overview of the iSCSI and Datamover layer interactions the way the Datamover Architecture intends them ✨
The interface requirements are written as Operational Primitives. An Operational Primitive is an abstract functional interface procedure between iSCSI and iSER. It either asks one layer to do something specific for the other layer, or tells the other layer that something happened 🔔 Whenever one gets invoked, the Connection_Handle qualifier tells you which iSCSI connection it's about 🔗 For some of them, a Data_Descriptor points to the iSCSI/SCSI data buffer tied to the requested or finished operation.
This abstract model and the primitives in this section are there to make the iSER protocol easier to describe. In the rest of the iSER spec, the compliance statements about using these primitives only apply to the required interactions between iSCSI and iSER 💅 Those later compliance statements only ask for functional equivalence between implementations. They don't put any requirements on how the interface itself gets implemented, which I think is so iconic of them 🙈
Every primitive comes with a set of qualifiers that describe the context for whatever action it's asking for. The qualifiers are required, but how you actually provide them is up to the implementation. You could pass them synchronously when you invoke the primitive, grab them from task context, read them from shared memory, or do something else entirely 🧚
3.1. Operational Primitives offered by iSER
The iSER protocol layer MUST support the following Operational Primitives so the iSCSI protocol layer can use them 🛠️
3.1.1. Send_Control
Input qualifiers: Connection_Handle, BHS and AHS (if any) of the
iSCSI PDU, PDU-specific qualifiers
Return results: Not specifiedThis one is used by iSCSI layers at both the initiator and the target to ask for the outbound transfer of an iSCSI control-type PDU (see Section 7.2) 📤 Qualifiers that only matter for one particular control-type PDU are called PDU-specific qualifiers. ImmediateDataSize for a SCSI Write command is a good example 🍓 For more on those, check Section 7.3. The iSCSI layer can only invoke Send_Control when the connection is in iSER-assisted mode.
3.1.2. Put_Data
Input qualifiers: Connection_Handle, content of a SCSI Data-In
PDU header, Data_Descriptor, Notify_Enable
Return results: Not specifiedThe target's iSCSI layer uses this to ask for the outbound transfer of SCSI Data-In PDU data from the buffer named by Data_Descriptor 📦 It can only be invoked when the connection is in iSER-assisted mode.
Notify_Enable tells the iSER layer whether it should send the iSCSI layer a local completion notification at some point. More on that in Section 3.2.2 on Data_Completion_Notify 🔔
3.1.3. Get_Data
Input qualifiers: Connection_Handle, content of an R2T PDU,
Data_Descriptor, Notify_Enable
Return results: Not specifiedHere the target's iSCSI layer asks for the inbound transfer of solicited data, the data an R2T PDU requested, into the buffer named by Data_Descriptor 📥 Again, it can only be invoked in iSER-assisted mode.
Notify_Enable works the same way as before. It tells the iSER layer whether it should generate the eventual local completion notification for the iSCSI layer. See Section 3.2.2 on Data_Completion_Notify 💜
3.1.4. Allocate_Connection_Resources
Input qualifiers: Connection_Handle, Resource_Descriptor
(optional)
Return results: StatusThis is how the iSCSI layers at both ends ask for every connection resource needed to support RCaP on an operational iSCSI/iSER connection 🧱 The iSCSI layer can optionally describe its implementation-specific resource needs with the Resource_Descriptor qualifier.
Status=success means the invocation worked, and Status=failure means it didn't 😭 If the invocation is for a Connection_Handle where an earlier invocation already succeeded, the iSER layer ignores the request and returns Status=success. Only one Allocate_Connection_Resources invocation can be outstanding for a given Connection_Handle at any moment ⏳
3.1.5. Deallocate_Connection_Resources
Input qualifiers: Connection_Handle
Return results: Not specifiedBoth iSCSI layers use this to ask for all the connection resources that were allocated earlier through a successful Allocate_Connection_Resources invocation 🧹
3.1.6. Enable_Datamover
Input qualifiers: Connection_Handle,
Transport_Connection_Descriptor, Final Login_Response_PDU
(optional)
Return results: Not specifiedBoth iSCSI layers use this to request that iSER-assisted mode be used for the connection 🚀 Transport_Connection_Descriptor identifies the exact connection tied to the Connection_Handle. The iSCSI layer can only invoke Enable_Datamover when there was a matching resource allocation before it.
Final_Login_Response_PDU only applies to a target. It holds the final Login Response PDU, the one that wraps up the iSCSI Login Phase 🎬
3.1.7. Connection_Terminate
Input qualifiers: Connection_Handle
Return results: Not specifiedThis is how both iSCSI layers ask for a specific iSCSI/iSER connection to be terminated, with all its connection and task resources freed 🧨 Once the invocation returns to the iSCSI layer, that layer may take full ownership of all the iSCSI-level resources tied to the connection, like I/O Buffers, for example.
3.1.8. Notice_Key_Values
Input qualifiers: Connection_Handle, number of keys, list of Key-
Value pairs
Return results: Not specifiedBoth iSCSI layers use this to ask the iSER layer to take note of the Key-Value pairs the iSCSI peers negotiated for the connection 📝
3.1.9. Deallocate_Task_Resources
Input qualifiers: Connection_Handle, ITT
Return results: Not specifiedThis one asks the iSER layer to free all the RCaP-specific resources it set aside for the task identified by ITT 🧽 The iSER layer might need a certain number of RCaP-specific resources tied to the ITT for every new iSCSI task. Normally, those task-level resources are handled automatically: allocated when a task starts and freed when it ends, as needed.
Things get messy in exception cases where the task doesn't finish with a SCSI Response PDU 😬 In those cases, the iSER layer has to be told about each individual task termination so it can manage its task-level resources properly. That's what this primitive is for. It isn't needed when a SCSI Response PDU normally wraps up the task ✨
Heads up: the iSER layer frees RCaP-specific task resources whenever a SCSI Response PDU normally ends a task, even if the SCSI status wasn't success. So don't expect a non-success status to change that cleanup 💀
3.2. Operational Primitives Used by iSER
The iSER layer MUST use the following Operational Primitives offered by the iSCSI protocol layer whenever the connection is in iSER-assisted mode 🫶
3.2.1. Control_Notify
Input qualifiers: Connection_Handle, an iSCSI control-type PDU
Return results: Not specifiedBoth iSER layers use this to tell the iSCSI layer that an inbound iSCSI control-type PDU is available 📬 A PDU counts as "available" to the iSCSI layer once the iSER layer has told it that the inbound PDU was received, along with an implementation-specific indication of where that PDU is.
3.2.2. Data_Completion_Notify
Input qualifiers: Connection_Handle, ITT, SN
Return results: Not specifiedThe iSER layer uses this to tell the iSCSI layer that an outbound data transfer finished. It only does this when the Put_Data invocation (see Section 3.1.2) was qualified with Notify_Enable set 🎉 For this case, SN refers to the DataSN associated with the SCSI Data-In PDU.
The iSER layer also uses it to tell the iSCSI layer that an inbound data transfer finished. That only happens when the Get_Data invocation (see Section 3.1.3) was qualified with Notify_Enable set 🥳 For this case, SN refers to the R2TSN associated with the R2T PDU.
3.2.3. Data_ACK_Notify
Input qualifier: Connection_Handle, ITT, DataSN
Return results: Not specifiedThe iSER layer at the target uses this to tell the iSCSI layer that a data acknowledgement arrived. That acknowledgement is defined in [iSCSI], and the iSCSI layer had asked for it earlier through a Put_Data invocation where the A-bit in the SCSI Data-In PDU was set to one 👑 See Section 7.3.5 for the details.
DataSN refers to the expected DataSN of the next SCSI Data-In PDU right after the SCSI Data-In PDU with the A-bit set that this notification corresponds to. The semantics follow the definition in [iSCSI] 🧠
3.2.4. Connection_Terminate_Notify
Input qualifiers: Connection_Handle
Return results: Not specifiedBoth iSER layers use this to tell the iSCSI layer about an unsolicited termination or failure of an iSCSI/iSER connection 💔 Before this primitive gets invoked, the iSER layer MUST free up the connection and task resources tied to the terminated connection.
This primitive is not invoked when the local iSCSI layer was the one that asked for the termination in the first place 🙅♀️
3.3. iSCSI Protocol Usage Requirements ✨
okay so, to even run in iSER-assisted mode, the iSCSI layers at both the initiator and the target MUST negotiate the RDMAExtensions key (see Section 6.3) to "Yes" on the leading connection. if the RDMAExtensions key isn't negotiated to "Yes", then iSER-assisted mode MUST NOT be used 🙅♀️. and if the RDMAExtensons key gets negotiated to "Yes", but the invocation of the Allocate_Connection_Resources Operational Primitive to the iSER layer fails, the iSCSI layer MUST fail the iSCSI Login process or terminate the connection as appropriate. see Section 10.1.3.1 for the deets 💅
if the RDMAExtensions key is negotiated to "Yes", the iSCSI layer MUST follow these protocol usage requirements from the iSER protocol, no exceptions bestie 🫶
- the iSCSI layer at the initiator MUST set ExpDataSN to zero in Task Management Function Requests for Task Allegiance Reassignment for read/bidirectional commands, so as to cause the target to send all unacknowledged read data 📦
- the iSCSI layer at the target MUST always return the SCSI status in a separate SCSI Response PDU for read commands. like, there MUST NOT be a "phase collapse" in concluding a SCSI Read Command 🙈
- the iSCSI layers at both the initiator and the target MUST support the keys as defined in Section 6 on Login/Text Operational Keys. if used as specified, these keys MUST NOT be answered with
NotUnderstood, and the semantics as defined MUST be followed for each iSER-assisted connection 🔑 - the iSCSI layer at the initiator MUST NOT issue SNACKs for PDUs. no SNACKs, period 🚫
4. Lower-Layer Interface Requirements 🌸
4.1. Interactions with the RCaP Layer 🎀
the iSER protocol layer sits on top of an RCaP layer (see Figure 1), and these are the key features that any RCaP layer is assumed to support ✨
- the RCaP layer supports all basic RDMA operations, including the RDMA Write Operation, RDMA Read Operation, and Send Operation.
- the RCaP layer provides reliable, in-order message delivery and direct data placement. 🧚
- when the iSER layer initiates an RDMA Read Operation following an RDMA Write Operation on one RCaP Stream, the RDMA Read Response Message processing on the remote node will only start after the preceding RDMA Write Message payload is placed in the memory of the remote node. the ordering is everything 🔥
- the RCaP layer encapsulates a single iSER Message into a single RCaP Message on the Data Source side. the RCaP layer decapsulates the iSER Message before delivering it to the iSER layer on the Data Sink side. 📬
- for an RCaP layer that supports the Send with Invalidate Message (e.g., iWARP), when the iSER layer provides the STag to be remotely invalidated to the RCaP layer for a Send with Invalidate Message, the RCaP layer uses this STag as the STag to be invalidated in the Send with Invalidate Message. 🦋
- the RCaP layer uses the STag and Tagged Offset provided by the iSER layer for the RDMA Write and RDMA Read Request Messages. 💜
- when the RCaP layer delivers the content of an RDMA Send Message to the iSER layer, the RCaP layer provides the length of the RDMA Send Message. this means the iSER layer doesn't have to carry a length field in the iSER header, which is so efficient, iconic behavior 👑
- when the RCaP layer delivers the Send Message to the iSER layer, it notifies the iSER layer with the mechanism provided on that interface. 🔔
- for an RCaP layer that supports the Send with Invalidate Message (e.g., iWARP), when the RCaP layer delivers a Send with Invalidate Message to the iSER layer, it passes the value of the STag that was invalidated. 🌈
- the RCaP layer propagates all status and error indications to the iSER layer. nothing gets hidden, we love transparency 🫶
- for a transport layer that operates in byte stream mode such as TCP, the RCaP implementation supports the enabling of the RDMA mode after connection establishment and the exchange of Login parameters in byte stream mode. for a transport layer that provides message delivery capability such as [IB], the RCaP implementation supports the direct use of the messaging capability by the iSCSI layer for the Login Phase after connection establishment and before enabling iSER-assisted mode. (in the specific example of InfiniBand [IB], the iSCSI layer uses IB messages to transfer iSCSI PDUs for the Login Phase after connection establishment and before enabling iSER-assisted mode.) 🧩
- whenever the iSER layer terminates the RCaP Stream, the RCaP layer terminates the associated connection. the stream and the connection go down together 💀
4.2. Interactions with the Transport Layer 🍓
after the iSER connection is established, the RCaP layer and the underlying transport layer are responsible for maintaining the connection and reporting any connection failures to the iSER layer. they've got the connection's back 🛡️
5. Connection Setup and Termination 🦋
5.1. iSCSI/iSER Connection Setup 💖
during connection setup, the iSCSI layer at the initiator is responsible for establishing a connection with the target. once the connection is established, the iSCSI layers at the initiator and the target enter the Login Phase using the same rules as outlined in [iSCSI]. the connection moves into the iSCSI Full Feature Phase in iSER-assisted mode after a successful login negotiation between the initiator and the target in which iSER-assisted mode is negotiated and the connection resources needed to support RCaP have been allocated at both the initiator and the target. the same connection MUST be used for both the iSCSI Login Phase and the subsequent iSER-assisted Full Feature Phase. one connection, two phases, zero drama 🎀
for a transport layer that operates in byte stream mode such as TCP, the RCaP implementation supports the enabling of the RDMA mode after connection establishment and the exchange of Login parameters in byte stream mode. for a transport layer that provides message delivery capability such as [IB], the RCaP implementation supports the use of the messaging capability by the iSCSI layer directly for the Login Phase after connection establishment and before enabling iSER-assisted mode. ✨
iSER-assisted mode MUST NOT be enabled unless it is negotiated on the leading connection during the LoginOperationalNegotiation stage of the iSCSI Login Phase. iSER-assisted mode is negotiated using the RDMAExtensions=<boolean-value> key. both the initiator and the target MUST exchange the RDMAExtensions key with the value set to "Yes" to enable iSER-assisted mode. if both the initiator and the target fail to negotiate the RDMAExtensions key set to "Yes", then the connection MUST continue with the login semantics as defined in [iSCSI]. if the RDMAExtensions key is not negotiated to Yes, then for some RCaP implementations (such as [IB]), the existing connection may need to be torn down and a new connection may need to be established in TCP-capable mode. (for InfiniBand, this will require a connection like [IPoIB].) 🔄
iSER-assisted mode is defined for a Normal session only, and the RDMAExtensions key MUST NOT be negotiated for a Discovery session. Discovery sessions are always conducted using the transport layer as described in [iSCSI]. 🙅♀️
an iSER-enabled node is not required to initiate the RDMAExtensions key exchange if its preference is for the Traditional iSCSI mode. the RDMAExtensions key, if offered, MUST be sent in the first available Login Response or Login Request PDU in the LoginOperationalNegotiation stage. this is because the value of some Login parameters might depend on whether or not iSER-assisted mode is enabled. timing matters, babe ⏰
iSER-assisted mode is a session-wide attribute. if both the initiator and the target negotiated RDMAExtensions="Yes" on the leading connection of a session, then all subsequent connections of the same session MUST enable iSER-assisted mode without having to exchange RDMAExtensions keys during the iSCSI Login Phase. conversely, if both the initiator and the target failed to negotiate RDMAExtensions to "Yes" on the leading connection of a session, then the RDMAExtensions key MUST NOT be negotiated further on any additional subsequent connection of the session. it's a whole-session vibe, not a one-connection thing 👑
when the RDMAExtensions key is negotiated to "Yes", the HeaderDigest and the DataDigest keys MUST be negotiated to "None" on all iSCSI/iSER connections participating in that iSCSI session. this is because, for an iSCSI/iSER connection, RCaP is responsible for providing error detection that is at least as good as a 32-bit CRC for all iSER Messages. furthermore, all SCSI Read data are sent using RDMA Write Messages instead of the SCSI Data-In PDUs, and all solicited SCSI Write data are sent using RDMA Read Response Messages instead of the SCSI Data-Out PDUs. HeaderDigest and DataDigest that apply to iSCSI PDUs would not be appropriate for RDMA Read and RDMA Write operations used with iSER. so the digests step aside, and RCaP handles the error checking 🧠💅
5.1.1. Initiator Behavior ✨
okay so if the iSCSI negotiation ends up turning on iSER-assisted mode, the initiator side has some homework to do first 📝. Before it sends the Login Request with the T (Transit) bit set to one and NSG (Next Stage) set to FullFeaturePhase, the iSCSI layer SHOULD ask the iSER layer to allocate the connection resources it needs to support RCaP, by invoking the Allocate_Connection_Resources Operational Primitive 🎀. Which resources those are is up to the implementation, and this spec doesn't cover them 🤷♀️. The iSCSI layer may also invoke the Notice_Key_Values Operational Primitive before the allocation call, so the iSER layer can take note of the negotiated iSCSI key values for the connection 🌸. Which keys get passed in is implementation dependent. They might include MaxOutstandingR2T and ErrorRecoveryLevel, but that's not the full list.
Now, the Inbound RDMA Read Queue Depth (IRD) is one of the connection resources at the initiator, and it's a big deal 👑. As Section 9.5.1 explains, the target turns R2Ts into RDMA Read operations. IRD caps how many incoming RDMA Read Requests from the target can be outstanding at once on an RCaP Stream. The exact IRD value is out of scope for iSER. The iSER layer at the initiator MUST set IRD to 1 or higher if R2Ts are going to be used on the connection 💜. BUT the iSER layer at the initiator MAY set IRD to zero based on implementation configuration, and setting it to zero just means no R2Ts will show up on that connection 🙈. To start, the iSER-IRD value at the initiator SHOULD match the IRD value at the initiator, and it MUST NOT be larger than that IRD value 💅.
The Outbound RDMA Read Queue Depth (ORD) is the other one, and it's giving chill vibes 😌. The ORD MAY be set to zero, because the iSER layer at the initiator never issues RDMA Read Requests to the target.
If the initiator can't allocate the connection resources it asked for, that's a login failure, and Section 10.1.3.1 covers how that gets handled 😭.
The iSER layer MUST return a success status to the iSCSI layer in response to the Allocate_Connection_Resources Operational Primitive ✨.
Once the target sends back the Login Response with the T bit set to one, NSG set to FullFeaturePhase, and a Status-Class of 0x00 (Success), the iSCSI layer MUST invoke the Enable_Datamover Operational Primitive with these qualifiers 🦋. (If the Status-Class isn't Success, see Section 10.1.4.6 instead.)
- a.
Connection_Handlethat identifies the iSCSI connection. - b.
Transport_Connection_Descriptorthat identifies the specific transport connection associated with theConnection_Handle.
The iSER layer MUST send the iSER Hello Message as the first iSER Message only if iSERHelloRequired is negotiated to "Yes" 🧚. Check Section 5.1.3 for all the iSER Hello Exchange details.
Here's the tea 🍵. If the initiator's iSCSI layer only allocates the RCaP connection resources after it gets the final Login Response PDU from the target, it might not be able to handle all the unexpected iSCSI control-type PDUs the target can send before the buffers get allocated on the initiator side. That count is declared by the MaxOutstandingUnexpectedPDUs key from the initiator. So in that case, iSERHelloRequired SHOULD be negotiated to "Yes", so the initiator can allocate its connection resources before it sends the iSER Hello Message 🫶. See Section 5.1.3 for more.
5.1.2. Target Behavior 🌈
Now the target side 🎯. If the iSCSI negotiation enables iSER-assisted mode, then before the target sends the Login Response with the T (Transit) bit set to one and NSG (Next Stage) set to FullFeaturePhase, the iSCSI layer MUST ask the iSER layer to allocate the resources it needs to support RCaP, by invoking the Allocate_Connection_Resources Operational Primitive 💖. Again, the implementation defines those resources, and they're outside this spec. Optionally, the iSCSI layer may invoke the Notice_Key_Values Operational Primitive before the allocation call, so the iSER layer can take note of the negotiated iSCSI keys for the connection. Which keys get passed in is implementation dependent, and they may include, but not limited to, MaxOutstandingR2T and ErrorRecoveryLevel 🍓.
Okay, this next part is serious, so pay attention 🚨. Allocating RCaP connection resources too early can expose an iSER target to a resource exhaustion attack. The attacker opens a bunch of iSER connections that only get as far as the point where the implementation allocates those RCaP resources, and then the resources run out 💀. The countermeasure is initiator authentication. So the iSCSI layer MUST NOT ask the iSER layer to allocate the connection resources needed for RCaP until the iSCSI layer is far enough along in the iSCSI Login Phase that it's reasonably certain the peer isn't an attacker 🔐. Specifically, if the Login Phase includes a SecurityNegotiation stage, the iSCSI layer MUST defer the connection resource allocation (that is, invoking the Allocate_Connection_Resources Operational Primitive) to the LoginOperationalNegotiation stage ([iSCSI]), so the allocation happens after authentication is finished ✨.
Among the connection resources allocated at the target is the Outbound RDMA Read Queue Depth (ORD). As Section 9.5.1 explains, the target turns R2Ts into RDMA Read operations. The ORD caps how many RDMA Read Requests from the target to the initiator can be outstanding at once on an RCaP Stream 🦋. To start, the iSER-ORD value at the target SHOULD match the ORD value at the target.
Meanwhile, the IRD at the target MAY be set to zero, since the iSER layer at the target never expects the initiator to issue RDMA Read Requests 🌸.
If the target can't allocate the connection resources it requested, that's a login failure, and Section 10.1.3.1 explains the handling 😩.
If the iSER layer at the target does successfully allocate the connection resources needed for RCaP, these events MUST happen in this exact order 👑:
- The iSER layer MUST return a success status to the iSCSI layer in response to the
Allocate_Connection_ResourcesOperational Primitive. - The iSCSI layer MUST invoke the
Enable_DatamoverOperational Primitive with these qualifiers:- a.
Connection_Handlethat identifies the iSCSI connection. - b.
Transport_Connection_Descriptorthat identifies the specific transport connection associated with theConnection_Handle. - c. The final transport-layer (e.g., TCP) message containing the Login Response with the
Tbit set to one and theNSGfield set toFullFeaturePhase.
- a.
- The iSER layer MUST send the final Login Response PDU in the native transport mode to wrap up the iSCSI Login Phase 🎀. If the underlying transport is TCP, then the iSER layer MUST send the final Login Response PDU in byte stream mode.
- After the iSER layer gets the iSER Hello Message from the initiator, it MUST respond with the iSER HelloReply Message, which gets sent as the first iSER Message if
iSERHelloRequiredis negotiated to"Yes". If the iSER layer gets an iSER Hello Message whileiSERHelloRequiredis negotiated to"No", that MUST be treated as an iSER protocol error 🙅♀️. See Section 5.1.3 on iSER Hello Exchange for more details.
Note: In the sequence above, items 3 and 4 MUST be performed atomically for iWARP connections. Skipping that can lead to race conditions, so don't sleep on it 😭🔥.
5.1.3. iSER Hello Exchange ✨
okay so the iSER Hello exchange is giving first-impressions energy 💅. If iSERHelloRequired is negotiated to "Yes", the very first iSER Message the iSER layer at the initiator sends to the target MUST be the iSER Hello Message 🎀. That's how the initiator introduces its iSER parameters to the target. Check Section 9.3 on iSER Header Format for the iSER Hello Message 🧚. Flip side: if iSERHelloRequired is negotiated to "No", then the iSER layer at the initiator MUST NOT send an iSER Hello Message 🙈.
The target gets its moment too 🦋. In response to the iSER Hello Message, the iSER layer at the target MUST return the iSER HelloReply Message as the first iSER Message it sends if iSERHelloRequired is negotiated to "Yes". That's how the target declares its iSER parameters to the initiator. See Section 9.4 on iSER Header Format for iSER HelloReply Message. If the iSER layer receives an iSER Hello Message when iSERHelloRequired is negotiated to "No", then this MUST be treated as an iSER protocol error 💀. See Section 10.1.3.4 on iSER Protocol Errors on for more details.
In the iSER Hello Message, the iSER layer at the initiator declares the iSER-IRD value to the target. 🌸
When the target gets the iSER Hello Message, the iSER layer at the target MUST set the iSER-ORD value to the minimum of two things: its own iSER-ORD value and the iSER-IRD value the initiator declared. To free up unused resources, the target MAY adjust (lower) its ORD value to match the iSER-ORD value, but only if the iSER-ORD value is smaller than the ORD value at the target. Resource-saving is very much a vibe here 🧹.
In the iSER HelloReply Message, the iSER layer at the target declares the iSER-ORD value to the initiator. 🍓
When the initiator gets the iSER HelloReply Message, the iSER layer at the initiator MAY adjust (lower) its IRD value to match the iSER-ORD value, in order to free up unused resources. This applies if the iSER-ORD value the target declared is smaller than the iSER-IRD value the initiator declared. 🫶
It is an iSER-level negotiation failure if the iSER parameters the initiator declared in the iSER Hello Message are unacceptable to the target. This includes the following:
- The initiator-declared iSER-IRD value is greater than 0, and the target-declared iSER-ORD value is 0.
- The initiator-supported and the target-supported iSER protocol versions do not overlap.
See Section 10.1.3.2 on the handling of the error situation. 🚨
Okay now the real tea ☕. An initiator that conforms to [RFC5046] allocates connection resources before sending the Login Request with the T (Transit) bit set to one and the NSG (Next Stage) field set to FullFeaturePhase. (For brevity, this is referred to as "early" connection allocation.) The current iSER specification relaxes this requirement, so an initiator can allocate connection resources after it receives the final Login Response PDU from the target. (For brevity, this is referred to as "late" connection allocation.) An initiator that uses "late" connection allocation may run into problems (e.g., RCaP connection closure) with a target that sends unexpected iSCSI PDUs right after it transitions to Full Feature Phase, which the negotiated value of the MaxOutstandingUnexpectedPDUs key permits. The only way to fully prevent this is to use iSER Hello Messages, since they let the initiator allocate its connection resources before sending its iSER Hello Message. The iSERHelloRequired key is how the initiator figures out whether it's dealing with a target that supports the iSER Hello exchanges. Luckily, known iSER target implementations don't fully use the number of allowed unexpected PDUs right after entering Full Feature Phase, which opens the door to an initiator workaround that uses fewer connection resources before Full Feature Phase, as explained below. 🦋✨
Here's the quick summary 📋. Where "late" connection allocation is practiced, an initiator that follows [RFC5046] is called an "old" initiator, and any other initiator is called a "new" initiator. Likewise, a target that doesn't support the iSERHelloRequired key (and responds with "NotUnderstood" when negotiating it) is called an "old" target, and any other target is called a "new" target. An "old" target can still support the iSER Hello exchanges, but the initiator doesn't know that. A "new" target can also respond with "No" when negotiating the iSERHelloRequired key. In that case, its behavior around "late" connection allocation looks like an "old" target's. 🧠
A "new" initiator works fine with a "new" target. 💖
For an "old" initiator and an "old" target, if the initiator fails to handle the number of unexpected iSCSI control-type PDUs the target sends before the buffer resources are allocated at the initiator, the iSER session can fail because the underlying RCaP connection gets closed. 😭 For the "old" target, there's a known implementation that sends one unexpected iSCSI control-type PDU after the final Login Response, then waits a bit before sending the next one. That somewhat eases the buffer allocation problem at the initiator. 🌈
For a "new" initiator and an "old" target, the same failure can happen: if the initiator doesn't handle the number of unexpected iSCSI control-type PDUs the target sends before its buffer resources are allocated, the iSER session can fail because the underlying RCaP connection gets closed. A "new" initiator MAY choose to terminate the connection. Otherwise, it SHOULD do one of the following:
- Allocate the connection resources before sending the final Login Request PDU.
- Allocate one or more buffers for receiving unexpected control-type PDUs from the target before sending the final Login Request PDU. This lowers the chance that the unexpected control-type PDUs close the RCaP connection before the connection resources are allocated.
For an "old" initiator and a "new" target, if the iSERHelloRequired key is not negotiated, a "new" target MUST still respond with the iSER HelloReply Message when it receives the iSER Hello Message. If the iSERHelloRequired key is negotiated to "No" or "NotUnderstood", a "new" target MAY choose to terminate the connection. Otherwise, it SHOULD delay sending any unexpected control-type PDUs until one of the following events has occurred:
- A PDU is received from the initiator after it sends the final Login Response PDU.
- A system-configurable timeout period (say, one second) has expired.
5.2. iSCSI/iSER Connection Termination 🌙
5.2.1. Normal Connection Termination at the Initiator
The iSCSI layer at the initiator ends an iSCSI/iSER connection normally by invoking the Send_Control Operational Primitive qualified with the Logout Request PDU. The iSER layer at the initiator MUST use a Send Message to send the Logout Request PDU to the target. The SendSE Message should be used if the RCaP layer supports it (e.g., iWARP). After the iSER layer at the initiator receives the Send Message containing the Logout Response PDU from the target, it MUST notify the iSCSI layer by invoking the Control_Notify Operational Primitive qualified with the Logout Response PDU. 👋
After the iSCSI logout process is complete, the iSCSI layer at the target is responsible for closing the iSCSI/iSER connection, as described in Section 5.2.2. After the RCaP layer at the initiator reports that the connection has been closed, the iSER layer at the initiator MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local Mappings (if any). Only then does it notify the iSCSI layer by invoking the Connection_Terminate_Notify Operational Primitive. 🧹
5.2.2. Normal Connection Termination at the Target
When the iSER layer at the target receives the Send Message containing the Logout Request PDU, it MUST notify the iSCSI layer at the target by invoking the Control_Notify Operational Primitive qualified with the Logout Request PDU. The iSCSI layer finishes the logout process by invoking the Send_Control Operational Primitive qualified with the Logout Response PDU. The iSER layer at the target MUST use a Send Message to send the Logout Response PDU to the initiator. The SendSE Message should be used if the RCaP layer supports it (e.g., iWARP). After the iSCSI logout process is complete, the iSCSI layer at the target MUST ask the iSER layer at the target to terminate the RCaP Stream by invoking the Connection_Terminate Operational Primitive. 🎬
As part of the termination process, the RCaP layer MUST close the connection. When the RCaP layer tells the iSER layer that the RCaP Stream and the associated connection are terminated, the iSER layer MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local and Remote Mappings (if any). Clean slate, bestie 🫧.
5.2.3. Termination without Logout Request/Response PDUs
5.2.3.1. Connection Termination Initiated by the iSCSI layer
The Connection_Terminate Operational Primitive MAY be invoked by the iSCSI layer to ask the iSER layer to terminate the RCaP Stream without having previously exchanged the Logout Request and Logout Response PDUs between the two iSCSI/iSER nodes. As part of the termination process, the RCaP layer will close the connection. When the RCaP layer tells the iSER layer that the RCaP Stream and the associated connection are terminated, the iSER layer MUST perform the following actions. 🚪
If the Connection_Terminate Operational Primitive is invoked by the iSCSI layer at the target, then the iSER layer at the target MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local and Remote Mappings (if any).
If the Connection_Terminate Operational Primitive is invoked by the iSCSI layer at the initiator, then the iSER layer at the initiator MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local Mappings (if any). 🫶
5.2.3.2. Connection Termination Notification to the iSCSI layer
If the iSCSI/iSER connection is terminated without the iSCSI layer invoking Connection_Terminate, the iSER layer MUST notify the iSCSI layer that the iSCSI/iSER connection has been terminated by invoking the Connection_Terminate_Notify Operational Primitive. 📣
Before invoking Connection_Terminate_Notify, the iSER layer at the target MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local and Remote Mappings (if any).
Before invoking Connection_Terminate_Notify, the iSER layer at the initiator MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local Mappings (if any).
If the remote iSCSI/iSER node started closing the connection (e.g., by sending a TCP FIN or TCP RST), the iSER layer MUST notify the iSCSI layer after the RCaP layer reports that the connection is closed, by invoking the Connection_Terminate_Notify Operational Primitive. 🌊
Another example of connection termination without a preceding logout is when the iSCSI layer at the initiator does an implicit logout (connection reinstatement). 🦋
6. Login/Text Operational Keys ✨
okay so this section is all about the iSCSI login/text keys, and some of them have restricted usage in iSER 🎀 Some keys get a glow-up for iSER, and a few new ones show up to make the iSER stuff work. Every other key from [iSCSI] that isn't covered here works on iSCSI/iSER connections with the same meaning, so you don't have to relearn those 💅
6.1. HeaderDigest and DataDigest 🦋
Irrelevant when: RDMAExtensions=Yesbestie, if the negotiation ends with RDMAExtensions=Yes for a session, then HeaderDigest=None and DataDigest=None are forced on every connection in that session 🔥 They override whatever the settings were, default or configured, so no arguing with it 💀
6.2. MaxRecvDataSegmentLength 🌸
Irrelevant when: RDMAExtensions=YesFor an iSCSI connection in a session where RDMAExtensions=Yes was negotiated on the leading connection, MaxRecvDataSegmentLength doesn't have to be declared during the Login Phase. If it does get declared anyway, it MUST be ignored 🙈 Instead, the InitiatorRecvDataSegmentLength (see Section 6.5) and TargetRecvDataSegmentLength (see Section 6.4) keys get negotiated. The local and remote MaxRecvDataSegmentLength values are worked out from those two keys, so the vibe is basically "the new keys are the source of truth now" 👑
In the Full Feature Phase, the initiator MUST consider its local MaxRecvDataSegmentLength (the one it would have declared to the target) to be the value of InitiatorRecvDataSegmentLength. It MUST also consider the remote MaxRecvDataSegmentLength (the one the target would have declared) to be the value of TargetRecvDataSegmentLength. The target flips it around the same way: it MUST consider its local MaxRecvDataSegmentLength (the one it would have declared to the initiator) to be the value of TargetRecvDataSegmentLength, and it MUST consider the remote one (the one the initiator would have declared) to be the value of InitiatorRecvDataSegmentLength ✨ Mirror mirror, iykyk.
Now the NOP-Out drama 😭 [RFC3720] says that when a target gets a NOP-Out request with a valid Initiator Task Tag, it has to answer with a NOP-In carrying that same Initiator Task Tag. On top of that, it returns the first MaxRecvDataSegmentLength bytes of the initiator-provided Ping Data. But iSER doesn't have one MaxRecvDataSegmentLength that both sides share, so the length of the data sent with a NOP-Out request MUST NOT exceed InitiatorMaxRecvDataSegmentLength. Not a single byte more 🫶
The MaxRecvDataSegmentLength key only applies to iSCSI control-type PDUs. Those are the only ones it cares about 💜
6.3. RDMAExtensions 🧚
Use: LO (leading only)
Senders: Initiator and Target
Scope: SW (session-wide)
RDMAExtensions=<boolean-value>
Irrelevant when: SessionType=Discovery
Default is No
Result function is ANDThis is the key that decides whether iSER-assisted mode is on. The initiator and target use it to negotiate support for that mode. To turn iSER-assisted mode on, both the initiator and the target MUST exchange RDMAExtensions=Yes. If either one offers RDMAExtensions=No, iSER-assisted mode MUST NOT be used. Main character energy for the Yes side, I guess 💅
An iSER-enabled node doesn't have to start the RDMAExtensions key exchange if it'd rather stick with Traditional iSCSI mode, totally valid 🦋 But if the RDMAExtensions key is going to be negotiated, the rules are strict. An initiator MUST offer the key in the first Login Request PDU in the LoginOperationalNegotiation stage of the leading connection. A target MUST offer the key in the first Login Response PDU it's allowed to send for that purpose, which means the first Login Response PDU issued after the first Login Request PDU with the C bit set to zero, again in the LoginOperationalNegotiation stage of the leading connection. When the other side offers RDMAExtensions=yes, an initiator MUST respond in the next Login Request PDU it's allowed to send, and a target MUST respond in the next Login Response PDU it's allowed to send 🎀
Negotiating RDMAExtensions first is the move 🔥 It lets a node pick the best values for the other keys. Keys like MaxBurstLength, MaxOutstandingR2T, ErrorRecoveryLevel, InitialR2T, ImmediateData, and so on might get negotiated differently depending on whether the connection is in Traditional iSCSI mode or iSER-assisted mode.
6.4. TargetRecvDataSegmentLength 🍓
Use: IO (Initialize only)
Senders: Initiator and Target
Scope: CO (connection-only)
Irrelevant when: RDMAExtensions=No
TargetRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>
Default is 8192 bytes
Result function is minimumThis key only matters for the iSCSI connection of an iSCSI session if RDMAExtensions=Yes was negotiated on the leading connection. The initiator and target use it to agree on the biggest data segment an initiator is allowed to send to the target in an iSCSI control-type PDU during the Full Feature Phase ✨
Here's the part that's a little bit extra 💀 For SCSI Command PDUs and SCSI Data-Out PDUs carrying non-immediate unsolicited data, which the initiator sends, the initiator MUST send all non-Final PDUs with a data segment size of exactly TargetRecvDataSegmentLength. That applies whenever those PDUs make up a data sequence bigger than TargetRecvDataSegmentLength. Very precise, very iconic 👑
6.5. InitiatorRecvDataSegmentLength 🌈
Use: IO (Initialize only)
Senders: Initiator and Target
Scope: CO (connection-only)
Irrelevant when: RDMAExtensions=No
InitiatorRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>
Default is 8192 bytes
Result function is minimumSame energy as the last one, just flipped 🦋 This key is relevant only for the iSCSI connection of an iSCSI session if RDMAExtensions=Yes was negotiated on the leading connection. The initiator and target use it to negotiate the maximum size of the data segment a target can send to the initiator in an iSCSI control-type PDU during the Full Feature Phase 💖
6.6. OFMarker and IFMarker 🫶
Irrelevant when: RDMAExtensions=YesIf negotiation ends with RDMAExtensions=Yes for a session, OFMarker=No and IFMarker=No get forced on every connection in that session. They override the settings, whether those are default or configured, so the markers are off no matter what 🙈
6.7. MaxOutstandingUnexpectedPDUs 🌸
Use: LO (leading only), Declarative
Senders: Initiator and Target
Scope: SW (session-wide)
Irrelevant when: RDMAExtensions=No
MaxOutstandingUnexpectedPDUs=
<numerical-value-from-2-to-(2**32-1) | 0>
Default is 0The initiator and target use this key to declare the maximum number of outstanding "unexpected" iSCSI control-type PDUs they can receive in the Full Feature Phase. The idea is to let the receiving side figure out how much buffer space it needs, on top of the normal flow control mechanism iSCSI already has ✨ A node should pick a value that doesn't put an unnecessary limit on the iSCSI layer under normal circumstances. A value of 0 means the declarer has no limit on how many outstanding "unexpected" iSCSI control-type PDUs it can receive. For how this gets used, check Sections 8.1.1 and 8.1.2. And note that iSER Hello and HelloReply Messages aren't iSCSI control-type PDUs, so this key doesn't touch them 🧚
For interoperability with implementations based on [RFC5046], this key SHOULD be negotiated. The reason is that the default value of 0 in [RFC5046] is a problem for most implementations, because it doesn't put any bound on the resources unexpected PDUs can use 😭
6.8. MaxAHSLength 💜
Use: LO (leading only), Declarative
Senders: Initiator and Target
Scope: SW (session-wide)
Irrelevant when: RDMAExtensions=No
MaxAHSLength=<numerical-value-from-2-to-(2**32-1) | 0>
Default is 256The initiator and target use this key to declare the maximum AHS size in an iSCSI control-type PDU they can receive in the Full Feature Phase. It helps the receiving side work out how much resource it needs for receive buffering 🍓 Same rule as before: pick a value that doesn't unnecessarily constrain the iSCSI layer under normal circumstances. A value of 0 means the declarer has no limit on the maximum AHS size in iSCSI control-type PDUs it can receive.
For interoperability with implementations based on [RFC5046], an initiator or target MAY terminate the connection if it expects MaxAHSLength to be greater than 256 and the key isn't understood by its peer. Okay, that's a real option, so it's good to know 👀
6.9. TaggedBufferForSolicitedDataOnly 🎀
Use: LO (leading only), Declarative
Senders: Initiator
Scope: SW (session-wide)
RDMAExtensions=<boolean-value>
Irrelevant when: RDMAExtensions=No
Default is NoThe initiator uses this key to tell the target how the Write Base Offset in the iSER header of an iSCSI control-type PDU is used 🦋 When it's set to No, the Base Offset goes with an I/O buffer holding all the write data, both unsolicited and solicited. When it's set to Yes, the Base Offset goes with an I/O buffer that only has solicited data. Two modes, one key, zero chill 💅
6.10. iSERHelloRequired 🌈
Use: LO (leading only), Declarative
Senders: Initiator
Scope: SW (session-wide)
RDMAExtensions=<boolean-value>
Irrelevant when: RDMAExtensions=No
Default is NoThis key is relevant only for the iSCSI connection of an iSCSI session if RDMAExtensions=Yes was negotiated on the leading connection. The initiator uses it to tell the target whether the iSER Hello Exchange is required 🫶 When it's set to Yes, the iSER layers MUST perform the iSER Hello Exchange as described in Section 5.1.3. When it's set to No, the iSER layers MUST NOT perform the iSER Hello Exchange. Clear instructions, no room for vibes 💀
7. iSCSI PDU Considerations 👑
When a connection is in the iSER-assisted mode, two types of message transfers are allowed between the iSCSI layer at the initiator and the iSCSI layer at the target 💖 They're called iSCSI data-type PDUs and iSCSI control-type PDUs, and the following sections explain what each one is 🧚
7.1. iSCSI Data-Type PDU ✨
okay bestie, so an iSCSI data-type PDU is an iSCSI PDU that makes data move between the peer iSCSI nodes during the Full Feature Phase of an iSCSI/iSER connection, and the remote iSCSI layer doesn't see it happen 💅 when the sending iSCSI layer asks for one to go out, the data transfer happens without the iSCSI layers at either the sending or the receiving node getting involved 🫶 that's because the PDU itself never gets handed to the iSCSI layer on the receiving side as-is. instead, the data transfer gets turned into the right RDMA operations, and the RDMA-Capable Controller handles those 🔥 the data-type PDUs are SCSI Data-In PDUs and R2T PDUs, and that's the whole squad 👑
if the iSCSI layer calls the Operational Primitive to have the iSER layer process a data-type PDU and it's qualified with Notify_Enable set, then once the RDMA operation finishes, the iSER layer at the target MUST tell the iSCSI layer at the target by invoking the Data_Completion_Notify Operational Primitive, qualified with the ITT and SN 🎀 there's no data completion notification at the initiator, though, since the RDMA-Capable Controller at the initiator handles those RDMA operations completely and the iSER layer at the initiator isn't involved in the data transfer for data-type PDUs 🦋
if the call is qualified with Notify_Enable cleared instead, then once the RDMA operation finishes, the iSER layer at the target MUST NOT notify the iSCSI layer at the target, and it MUST NOT invoke the Data_Completion_Notify Operational Primitive 🙈
and if anything goes wrong with an operation tied to a data-type PDU, for any reason at all, the contents of the Data Sink buffers for that operation are considered indeterminate 💀 so don't trust those bytes, bestie 😭
7.2. iSCSI Control-Type PDU 🌸
anything that's not a data-type PDU and also isn't a SCSI Data-Out PDU carrying solicited data is an iSCSI control-type PDU 🧚 the iSCSI layer calls the Send_Control Operational Primitive to have the iSER layer process one of these, and they get sent using Send Messages of RCaP 💜 the thing to note is that SCSI Data-Out PDUs carrying unsolicited data are counted as iSCSI control-type PDUs, so they follow this path. see Section 7.3.4 for how SCSI Data-Out PDUs get treated 🍓
when the iSER layer gets an iSCSI control-type PDU, it MUST tell the iSCSI layer by invoking the Control_Notify Operational Primitive, qualified with that iSCSI control-type PDU ✨
7.3. iSCSI PDUs 🦋
this section is all about how the iSER layer handles each iSCSI PDU type 💖 the iSCSI layer asks the iSER layer to process an iSCSI PDU by invoking the right Operational Primitive, and every one of those invocations MUST be qualified by a Connection_Handle 🫶 on top of that, the BHS and the optional AHS of the iSCSI PDU (as defined in [iSCSI]) MUST qualify each invocation too. the qualifying Connection_Handle, BHS, and AHS aren't spelled out in the sections that follow, so keep them in mind 🌈
7.3.1. SCSI Command 💅
Type: control-type PDU
PDU-specific qualifiers (for SCSI Write or bidirectional command):
ImmediateDataSize, UnsolicitedDataSize, DataDescriptorOut
PDU-specific qualifiers (for SCSI Read or bidirectional command):
DataDescriptorInthe iSER layer at the initiator MUST send the SCSI command in a Send Message to the target 🔥 the SendSE Message should be used if the RCaP layer supports it (e.g., iWARP) 🌸
for a SCSI Write or bidirectional command, the iSCSI layer at the initiator MUST invoke the Send_Control Operational Primitive like this:
- if there's immediate data to send for the SCSI write or bidirectional command, the
ImmediateDataSizequalifier MUST be used to say how many bytes of immediate unsolicited data go along with the write or bidirectional command. theDataDescriptorOutqualifier MUST be used to point at the initiator's I/O Buffer that holds the SCSI Write data 💜 - if there's unsolicited data to send for the SCSI Write or bidirectional command, the
UnsolicitedDataSizequalifier MUST be used to say how many bytes of immediate and non-immediate unsolicited data the command has. the iSCSI layer will send one or more SCSI Data-Out PDUs for the non-immediate unsolicited data. see Section 7.3.4 on SCSI Data-Out 🎀 - if there's solicited data to send for the SCSI Write or bidirectional command, which is the case when the Expected Data Transfer Length in the SCSI Command PDU is bigger than the value of
UnsolicitedDataSize, the iSER layer at the initiator MUST do the following:- a. It MUST allocate a Write STag for the I/O Buffer defined by the
DataDescriptorOutqualifier.DataDescriptorOutdescribes the I/O buffer, starting with the immediate unsolicited data (if any), then the non-immediate unsolicited data (if any), and then the solicited data. WhenTaggedBufferForSolicitedDataOnlyis negotiated to No, the Base Offset is associated with this I/O Buffer. WhenTaggedBufferForSolicitedDataOnlyis negotiated to Yes, the Base Offset is associated with an I/O Buffer that contains only solicited data 🦋 - b. It MUST establish a Local Mapping that associates the Initiator Task Tag (ITT) with the Write STag 🌈
- c. It MUST Advertise the Write STag and the Base Offset to the target by sending them in the iSER header of the iSER Message (the payload of the Send Message of RCaP) that carries the SCSI Write or bidirectional command PDU. The SendSE Message should be used if the RCaP layer supports it (e.g., iWARP). See Section 9.2 on iSER Header Format for iSCSI Control-Type PDU ✨
- a. It MUST allocate a Write STag for the I/O Buffer defined by the
for a SCSI Read or bidirectional command, the iSCSI layer at the initiator MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorIn, which defines the initiator's I/O Buffer for receiving the SCSI Read data. the iSER layer at the initiator MUST do the following:
- a. It MUST allocate a Read STag for the I/O Buffer and note the Base Offset for this I/O Buffer 💖
- b. It MUST establish a Local Mapping that associates the Initiator Task Tag (ITT) with the Read STag 🫶
- c. It MUST Advertise the Read STag and the Base Offset to the target by sending them in the iSER header of the iSER Message (the payload of the Send Message of RCaP) that carries the SCSI Read or bidirectional command PDU. The SendSE Message should be used if the RCaP layer supports it (e.g., iWARP). See Section 9.2 on iSER Header Format for iSCSI Control-Type PDU 🍓
if the amount of unsolicited data in a SCSI Command is bigger than TargetRecvDataSegmentLength, the iSCSI layer at the initiator MUST split the data into multiple iSCSI control-type PDUs. The data segment length in every generated PDU except the last one MUST be exactly TargetRecvDataSegmentLength 🌸 the last iSCSI control-type PDU carrying the unsolicited data can be up to TargetRecvDataSegmentLength 🧚
when the iSER layer at the target receives the SCSI Command, it MUST establish a Remote Mapping that associates the ITT with the Base Offset(s) and the Advertised STag(s) in the iSER header. The iSER layer at the target uses the Write STag while handling the data transfer tied to the R2T PDU(s), as described in Section 7.3.6. It uses the Read STag while handling the SCSI Data-In PDU(s) from the iSCSI layer at the target, as described in Section 7.3.5 💜
7.3.2. SCSI Response 🔥
Type: control-type PDU
PDU-specific qualifiers: DataDescriptorStatusthe iSCSI layer at the target MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorStatus, which defines the buffer holding the sense and response information 👑 the iSCSI layer at the target MUST always return the SCSI status for a SCSI command in a separate SCSI Response PDU 🎀 "Phase collapse" for sending SCSI status inside a SCSI Data-In PDU MUST NOT be used 🙈 the iSER layer at the target sends the SCSI Response PDU following these rules:
- if the initiator didn't Advertise any STags in the iSER Message that carried the SCSI command PDU, then the iSER layer at the target MUST send a Send Message containing the SCSI Response PDU. The SendSE Message should be used if the RCaP layer supports it (e.g., iWARP) 🦋
- if the initiator Advertised a Read STag in the iSER Message that carried the SCSI Command PDU, then the iSER layer at the target MUST send a Send Message containing the SCSI Response PDU. The header of that Send Message MUST carry the Read STag to be invalidated at the initiator. The Send with Invalidate Message can be used to automatically invalidate the STag, if the RCaP layer supports it (e.g., iWARP) 🌈
- if the initiator Advertised only the Write STag in the iSER Message that carried the SCSI command PDU, then the iSER layer at the target MUST send a Send Message containing the SCSI Response PDU. The header of that Send Message MUST carry the Write STag to be invalidated at the initiator. The Send with Invalidate Message can be used to automatically invalidate the STag, if the RCaP layer supports it (e.g., iWARP) ✨
when the iSCSI layer at the target invokes the Send_Control Operational Primitive to send the SCSI Response PDU, the iSER layer at the target MUST invalidate the Remote Mapping before it transfers the SCSI Response PDU to the initiator 💖
when the iSER layer at the initiator gets a Send Message containing the SCSI Response PDU from the target, it MUST invalidate the STag(s) named in the header 🫶 (if a Send with Invalidate Message is supported by the RCaP layer (e.g., iWARP) and is used to carry the SCSI Response PDU, then the RCaP layer at the initiator will invalidate the STag. The iSER layer at the initiator MUST still make sure the correct STag gets invalidated. If both the Read and the Write STags were Advertised earlier by the initiator, then the iSER layer at the initiator MUST explicitly invalidate the Write STag upon receiving the Send with Invalidate Message, because the header of that message can only carry one STag (in this case, the Read STag) to be invalidated.) 🍓
the iSER layer at the initiator MUST make sure the STag(s) used in a command get invalidated before it notifies the iSCSI layer at the initiator by invoking the Control_Notify Operational Primitive qualified with the SCSI Response. This rules out using the STag(s) after the command completes, since that use would cause data corruption 💀
when the iSER layer at the initiator receives a Send Message containing the SCSI Response PDU, it SHOULD invalidate the Local Mapping 🌸 the iSER layer MUST make sure that all local STag(s) associated with the ITT are invalidated before it notifies the iSCSI layer of the SCSI Response PDU by invoking the Control_Notify Operational Primitive qualified with the SCSI Response PDU 👑
7.3.3. Task Management Function Request/Response ✨
Type: control-type PDU
PDU-specific qualifiers (for TMF Request): DataDescriptorOut,
DataDescriptorInokay so this one's the TASK REASSIGN drama 🎭 The iSER layer MUST use a Send Message to send the Task Management Function Request/Response PDU. The SendSE Message should be used if the RCaP layer supports it (e.g., iWARP) 💅
For the Task Management Function Request with the TASK REASSIGN function, the iSER layer at the initiator MUST do the following:
- It MUST use the ITT as specified in the Referenced Task Tag from the Task Management Function Request PDU to locate the existing STags (if any) in the Local Mappings.
- It MUST invalidate the existing STags (if any) and the Local Mappings.
- It MUST allocate a Read STag for the I/O Buffer and note the Base Offset associated with the I/O Buffer as defined by the qualifier DataDescriptorIn if the Send_Control Operational Primitive invocation is qualified with DataDescriptorIn.
- It MUST allocate a Write STag for the I/O Buffer and note the Base Offset associated with the I/O Buffer as defined by the qualifier DataDescriptorOut if the Send_Control Operational Primitive invocation is qualified with DataDescriptorOut.
- If STags are allocated, it MUST establish new Local Mapping(s) that associate the ITT to the allocated STag(s).
- It MUST Advertise the STags and the Base Offsets, if allocated, to the target in the iSER header of the Send Message carrying the iSCSI PDU, as described in Section 9.2. The SendSE Message should be used if supported by the RCaP layer (e.g., iWARP).
For the Task Management Function Request with the TASK REASSIGN function for a SCSI Read or bidirectional command, the iSCSI layer at the initiator MUST set ExpDataSN to zero since the data transfer and acknowledgements happen transparently to the iSCSI layer at the initiator. This provides the flexibility to the iSCSI layer at the target to request transmission of only the unacknowledged data as specified in [iSCSI].
When the iSER layer at the target receives the Task Management Function Request with the TASK REASSIGN function, it MUST do the following:
- It MUST use the ITT as specified in the Referenced Task Tag from the Task Management Function Request PDU to locate the Local and Remote Mappings (if any).
- It MUST invalidate the local STags (if any) associated with the ITT.
- It MUST replace the Base Offset(s) and the Advertised STag(s) in the Remote Mapping with the Base Offset(s) and the Advertised STag(s) in the iSER header. The Write STag is used in the handling of the R2T PDU(s) from the iSCSI layer at the target as described in Section 7.3.6. The Read STag is used in the handling of the SCSI Data-In PDU(s) from the iSCSI layer at the target as described in Section 7.3.5.
7.3.4. SCSI Data-Out 🚚
Type: control-type PDU
PDU-specific qualifiers: DataDescriptorOutThe iSCSI layer at the initiator MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorOut, which defines the initiator's I/O Buffer containing unsolicited SCSI Write data.
If the amount of unsolicited data to be transferred as SCSI Data-Out exceeds TargetRecvDataSegmentLength, then the iSCSI layer at the initiator MUST segment the data into multiple iSCSI control-type PDUs, where the DataSegmentLength has the value of TargetRecvDataSegmentLength in all generated PDUs except the last one. The DataSegmentLength of the last iSCSI control-type PDU carrying the unsolicited data can be up to TargetRecvDataSegmentLength. The iSCSI layer at the target MUST perform the reassembly function for the unsolicited data.
For unsolicited data, the iSER layer at the initiator MUST use a Send Message to send the SCSI Data-Out PDU. If the F bit is set to 1, the SendSE Message should be used if supported by the RCaP layer (e.g., iWARP).
Note that for solicited data, the SCSI Data-Out PDUs are not used since R2T PDUs are not delivered to the iSCSI layer at the initiator. Instead, R2T PDUs are transformed by the iSER layer at the target into RDMA Read operations. (See Section 7.3.6.)
7.3.5. SCSI Data-In 📦
Type: data-type PDU
PDU-specific qualifiers: DataDescriptorInWhen the iSCSI layer at the target is ready to return the SCSI Read data to the initiator, it MUST invoke the Put_Data Operational Primitive qualified with DataDescriptorIn, which defines the SCSI Data-In buffer. See Section 7.1 on the general requirement on the handling of iSCSI data-type PDUs. SCSI Data-In PDU(s) are used in SCSI Read data transfer as described in Section 9.5.2.
The iSER layer at the target MUST do the following for each invocation of the Put_Data Operational Primitive:
- It MUST use the ITT in the SCSI Data-In PDU to locate the remote Read STag and the Base Offset in the Remote Mapping. The Remote Mapping was established earlier by the iSER layer at the target when the SCSI Read Command was received from the initiator.
- It MUST generate and send an RDMA Write Message containing the read data to the initiator. a. It MUST use the remote Read STag as the Data Sink STag of the RDMA Write Message. b. It MUST add the Buffer Offset from the SCSI Data-In PDU to the Base Offset from the Remote Mapping as the Data Sink Tagged Offset of the RDMA Write Message. c. It MUST use DataSegmentLength from the SCSI Data-In PDU to determine the amount of data to be sent in the RDMA Write Message.
- It MUST associate the DataSN and ITT from the SCSI Data-In PDU with the RDMA Write operation. If the Put_Data Operational Primitive invocation was qualified with Notify_Enable set, then when the iSER layer at the target receives a completion from the RCaP layer for the RDMA Write Message, the iSER layer at the target MUST notify the iSCSI layer by invoking the Data_Completion_Notify Operational Primitive qualified with the DataSN and ITT. Conversely, if the Put_Data Operational Primitive invocation was qualified with Notify_Enable cleared, then the iSER layer at the target MUST NOT notify the iSCSI layer on completion and MUST NOT invoke the Data_Completion_Notify Operational Primitive.
When the A-bit is set to one in the SCSI Data-In PDU, the iSER layer at the target MUST notify the iSCSI layer at the target when the data transfer is complete at the initiator. To perform this additional function, the iSER layer at the target can take advantage of the operational ErrorRecoveryLevel if previously disclosed by the iSCSI layer via an earlier invocation of the Notice_Key_Values Operational Primitive. There are two approaches that can be taken:
- If the iSER layer at the target knows that the operational ErrorRecoveryLevel is 2, or if the iSER layer at the target does not know the operational ErrorRecoveryLevel, then the iSER layer at the target MUST issue a zero-length RDMA Read Request Message following the RDMA Write Message. When the iSER layer at the target receives a completion for the RDMA Read Request Message from the RCaP layer, implying that the RDMA-Capable Controller at the initiator has completed processing the RDMA Write Message due to the completion ordering semantics of RCaP, the iSER layer at the target MUST notify the iSCSI layer at the target by invoking the Data_ACK_Notify Operational Primitive qualified with ITT and DataSN (see Section 3.2.3).
- If the iSER layer at the target knows that the operational ErrorRecoveryLevel is 1, then the iSER layer at the target MUST do one of the following: a. It MUST notify the iSCSI layer at the target by invoking the Data_ACK_Notify Operational Primitive qualified with ITT and DataSN (see Section 3.2.3) when it receives the local completion from the RCaP layer for the RDMA Write Message. This is allowed since digest errors do not occur in iSER (see Section 10.1.4.2) and a CRC error will cause the connection to be terminated and the task to be terminated anyway. The local RDMA Write completion from the RCaP layer guarantees that the RCaP layer will not access the I/O Buffer again to transfer the data associated with that RDMA Write operation. b. Alternatively, it MUST use the same procedure for handling the data transfer completion at the initiator as for ErrorRecoveryLevel 2.
It should be noted that the iSCSI layer at the target cannot set the A-bit to 1 if the ErrorRecoveryLevel=0.
SCSI status MUST always be returned in a separate SCSI Response PDU. The S bit in the SCSI Data-In PDU MUST always be set to zero. There MUST NOT be a "phase collapse" in the SCSI Data-In PDU.
Since the RDMA Write Message only transfers the data portion of the SCSI Data-In PDU but not the control information in the header, such as ExpCmdSN, if timely updates of such information are crucial, the iSCSI layer at the initiator MAY issue NOP-Out PDUs to request the iSCSI layer at the target to respond with the information using NOP-In PDUs. 🌸
7.3.6. Ready To Transfer (R2T) 🎀
Type: data-type PDU 💌
PDU-specific qualifiers: DataDescriptorOut ✨
okay so the R2T PDU is basically the target going "bestie, send me the write data pls" 🥺 To send one, the iSCSI layer at the target MUST invoke the Get_Data Operational Primitive qualified with DataDescriptorOut, which defines the I/O Buffer for receiving the SCSI Write data from the initiator. Check out Section 7.1 for the general rules on handling iSCSI data-type PDUs 📚
The iSER layer at the target MUST do the following for each invocation of the Get_Data Operational Primitive:
- It MUST ensure a valid local STag for the I/O Buffer and a valid Local Mapping. This may involve allocating a valid local STag and establishing a Local Mapping. 🧚
- It MUST use the ITT in the R2T to locate the remote Write STag and the Base Offset in the Remote Mapping. The Remote Mapping was set up earlier by the iSER layer at the target, when the iSER Message containing the Advertised Write STag, the Base Offset, and the SCSI Command PDU for a SCSI Write or bidirectional command arrived from the initiator 💜
- If the iSER-ORD value at the target is set to zero, the iSER layer at the target MUST terminate the connection and free up the resources associated with the connection (as described in Section 5.2.3) if it received the R2T PDU from the iSCSI layer at the target. Once the connection is terminated, the iSER layer at the target MUST notify the iSCSI layer at the target by invoking the Connection_Terminate_Notify Operational Primitive. 💀
- If the iSER-ORD value at the target is set to greater than 0, the iSER layer at the target MUST transform the R2T PDU into an RDMA Read Request Message. While transforming the R2T PDU, the iSER layer at the target MUST ensure that the number of outstanding RDMA Read Request Messages does not exceed the iSER-ORD value. To transform the R2T PDU, the iSER layer at the target:
- a. MUST derive the local STag and local Tagged Offset from the DataDescriptorOut that qualified the Get_Data invocation.
- b. MUST use the local STag as the Data Sink STag of the RDMA Read Request Message.
- c. MUST use the local Tagged Offset as the Data Sink Tagged Offset of the RDMA Read Request Message.
- d. MUST use the Desired Data Transfer Length from the R2T PDU as the RDMA Read Message Size of the RDMA Read Request Message.
- e. MUST use the remote Write STag as the Data Source STag of the RDMA Read Request Message.
- f. MUST add the Buffer Offset from the R2T PDU to the Base Offset from the Remote Mapping as the Data Source Tagged Offset of the RDMA Read Request Message.
- It MUST associate the R2TSN and ITT from the R2T PDU with the RDMA Read operation. If the Get_Data Operational Primitive invocation was qualified with Notify_Enable set, then when the iSER layer at the target receives a completion from the RCaP layer for the RDMA Read operation, the iSER layer at the target MUST notify the iSCSI layer by invoking the Data_Completion_Notify Operational Primitive qualified with the R2TSN and ITT. But if the Get_Data Operational Primitive invocation was qualified with Notify_Enable cleared, then the iSER layer at the target MUST NOT notify the iSCSI layer on completion and MUST NOT invoke the Data_Completion_Notify Operational Primitive. 🔥
When the RCaP layer at the initiator receives a valid RDMA Read Request Message, it will return an RDMA Read Response Message containing the solicited write data to the target. When the RCaP layer at the target receives that RDMA Read Response Message from the initiator, it will place the solicited data in the I/O Buffer referenced by the Data Sink STag in the RDMA Read Response Message. Iconic behavior 👑
Since the RDMA Read Request Message from the target does not carry the control information from the R2T PDU, such as ExpCmdSN, if timely updates of that information are crucial, the iSCSI layer at the initiator MAY issue NOP-Out PDUs to ask the iSCSI layer at the target to respond with the information using NOP-In PDUs. 🦋
Similarly, since the RDMA Read Response Message from the initiator only transfers the data and not the control information normally found in the SCSI Data-Out PDU, such as ExpStatSN, if timely updates of that information are crucial, the iSCSI layer at the target MAY issue NOP-In PDUs to ask the iSCSI layer at the initiator to respond with the information using NOP-Out PDUs. 🌸
7.3.7. Asynchronous Message 📣
Type: control-type PDU 💬
PDU-specific qualifiers: DataDescriptorSense 🧾
The iSCSI layer MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorSense, which defines the buffer holding the sense and iSCSI event information. The iSER layer MUST use a Send Message to send the Asynchronous Message PDU. The SendSE Message should be used if the RCaP layer supports it (e.g., iWARP) ⭐
7.3.8. Text Request and Text Response 📝
Type: control-type PDU 💌
PDU-specific qualifiers: DataDescriptorTextOut (for Text Request), DataDescriptorIn (for Text Response) 🎀
The iSCSI layer MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorTextOut (or DataDescriptorIn), which defines the Text Request (or Text Response) buffer. The iSER layer MUST use Send Messages to send the Text Request (or Text Response) PDUs. The SendSE Message should be used if the RCaP layer supports it (e.g., iWARP) 🫶
7.3.9. Login Request and Login Response 🔑
During the login negotiation, the iSCSI layer talks to the transport layer directly, and the iSER layer is not involved. See Section 5.1 on iSCSI/iSER Connection Setup. If the underlying transport is TCP, the Login Request PDUs and the Login Response PDUs are exchanged while the connection between the initiator and the target is still in the byte stream mode 🌊
The iSCSI layer MUST NOT send a Login Request (or a Login Response) PDU during the Full Feature Phase. A Login Request (or a Login Response) PDU, if used, MUST be treated as an iSCSI protocol error. The iSER layer MAY reject such a PDU from the iSCSI layer with an appropriate error code. If the iSCSI layer at the target receives a Login Request PDU, it MUST respond with a Reject PDU with a reason code of "protocol error". 🚫
7.3.10. Logout Request and Logout Response 👋
Type: control-type PDU 💌
PDU-specific qualifiers: None 🫧
The iSER layer MUST use a Send Message to send the Logout Request or Logout Response PDU. The SendSE Message should be used if the RCaP layer supports it (e.g., iWARP). Sections 5.2.1 and 5.2.2 explain how the initiator and the target handle the Logout Request and the Logout Response, plus the interactions between them to terminate a connection. Main character energy 🎬
7.3.11. SNACK Request 🙅♀️
Since HeaderDigest and DataDigest must be negotiated to "None", there are no digest errors when the connection is in iSER-assisted mode. Also, since RCaP delivers all messages in the order they were sent, there are no sequence errors when the connection is in iSER-assisted mode. So the iSCSI layer MUST NOT send SNACK Request PDUs. A SNACK Request PDU, if used, MUST be treated as an iSCSI protocol error. The iSER layer MAY reject such a PDU from the iSCSI layer with an appropriate error code. If the iSCSI layer at the target receives a SNACK Request PDU, it MUST respond with a Reject PDU with a reason code of "protocol error". 🙈
7.3.12. Reject 🚨
Type: control-type PDU 💌
PDU-specific qualifiers: DataDescriptorReject ✨
The iSCSI layer MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorReject, which defines the Reject buffer. The iSER layer MUST use a Send Message to send the Reject PDU. The SendSE Message should be used if the RCaP layer supports it (e.g., iWARP) 🥀
7.3.13. NOP-Out and NOP-In 🏓
Type: control-type PDU 💌
PDU-specific qualifiers: DataDescriptorNOPOut (for NOP-Out), DataDescriptorNOPIn (for NOP-In) 🍓
The iSCSI layer MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorNOPOut (or DataDescriptorNOPIn), which defines the Ping (or Return Ping) data buffer. The iSER layer MUST use Send Messages to send the NOP-Out (or NOP-In) PDU. The SendSE Message should be used if the RCaP layer supports it (e.g., iWARP) 🏓✨
8. Flow Control and STag Management 🌈
8.1. Flow Control for RDMA Send Messages 💅
Send Messages in RCaP are how the iSER layer moves iSCSI control-type PDUs around. Each Send Message in RCaP eats up an Untagged Buffer at the Data Sink. The catch is that neither the RCaP layer nor the iSER layer gives Send Messages an explicit flow control mechanism 😭 So the iSER layer SHOULD provision enough Untagged buffers for incoming Send Messages to prevent buffer exhaustion at the RCaP layer. If buffer exhaustion happens, it may result in the termination of the connection.
Buffer exhaustion can end the whole connection, so don't sleep on those Untagged buffers 💀
An implementation may choose to meet the buffer requirement with a common buffer pool shared across multiple connections, with usage limits on a per-connection basis and usage limits on the buffer pool itself. In that kind of implementation, going over the buffer usage limit for a connection or for the pool itself may trigger interventions from the iSER layer to refill the buffer pool and/or to isolate the connection causing the problem 🧠
iSER also provides the MaxOutstandingUnexpectedPDUs key, which the initiator and the target use to declare the maximum number of outstanding "unexpected" control-type PDUs they can receive. It's there so the receiving side can work out how many buffer resources it needs beyond the normal flow control mechanism that iSCSI already has 🎀
The buffer resources needed at both the initiator and the target because of control-type PDUs sent by the initiator are described in Section 8.1.1. The buffer resources needed at both the initiator and the target because of control-type PDUs sent by the target are described in Section 8.1.2. 🗺️
8.1.1. Flow Control for Control-Type PDUs from the Initiator 🧁
The control-type PDUs that an initiator can send to a target fall into the following categories:
- Regulated: Control-type PDUs in this category are regulated by the iSCSI CmdSN window mechanism, and the immediate flag is not set. 🚦
- Unregulated but Expected: Control-type PDUs in this category are not regulated by the iSCSI CmdSN window mechanism, but the target expects them. 🤍
- Unregulated and Unexpected: Control-type PDUs in this category are not regulated by the iSCSI CmdSN window mechanism, and the target finds them "unexpected". 👀
8.1.1.1. Control-Type PDUs from the Initiator in the Regulated Category
okay bestie, the regulated category is where the iSCSI CmdSN window runs the show 👑 Control-type PDUs the initiator sends here are regulated by that window, and the immediate flag is not set 🎀
The queuing capacity the target's iSCSI layer needs is described in Section 4.2.2.1 of [iSCSI] 🧚 For each control-type PDU the initiator can send in this category, the initiator MUST provision for the buffer resources needed by the matching control-type PDU that the target sends back. Here's the list of initiator PDUs and the target's replies:
- a. When an initiator sends a SCSI Command PDU, it expects a SCSI Response PDU from the target.
- b. When the initiator sends a Task Management Function Request PDU, it expects a Task Management Function Response PDU from the target.
- c. When the initiator sends a Text Request PDU, it expects a Text Response PDU from the target.
- d. When the initiator sends a Logout Request PDU, it expects a Logout Response PDU from the target.
- e. When the initiator sends a NOP-Out PDU as a ping request with
ITT != 0xffffffffandTTT = 0xffffffff, it expects a NOP-In PDU from the target with the sameITTandTTTas in the ping request. 🍓
The response from the target for any of the PDUs listed here may alternatively be a Reject PDU sent before the task is active, as described in Section 7.3 of [iSCSI] 🌸
8.1.1.2. Control-Type PDUs from the Initiator in the Unregulated but Expected Category
For the control-type PDUs in the Unregulated but Expected category, the amount of buffering resources the target needs can be predetermined ✨ Here's what's in this category:
- a. SCSI Data-Out PDUs are used by the initiator to send unsolicited data. The amount of buffer resources required by the target can be determined using
FirstBurstLength. Note that SCSI Data-Out PDUs are not used for solicited data since the R2T PDU, which is used for solicitation, is transformed into RDMA Read operations by the iSER layer at the target. See Section 7.3.4. 🦋 - b. A NOP-Out PDU with
TTT != 0xffffffffis sent as a ping response by the initiator to the NOP-In PDU sent as a ping request by the target. 💜
8.1.1.3. Control-Type PDUs from the Initiator in the Unregulated and Unexpected Category
PDUs in the Unregulated and Unexpected category are PDUs with the immediate flag set 🔥 The number of PDUs in this category that an initiator can send is controlled by the value of MaxOutstandingUnexpectedPDUs declared by the target. (See Section 6.7.) After a PDU in this category is sent by the initiator, it is outstanding until it is retired. At any time, the number of outstanding unexpected PDUs MUST NOT exceed the value of MaxOutstandingUnexpectedPDUs declared by the target. 🙈
The target uses the MaxOutstandingUnexpectedPDUs value it declared to work out how much buffer it needs for control-type PDUs in this category that an initiator can send. For the initiator, for each control-type PDU it can send in this category, the initiator MUST provision for the buffer resources, if required, for the matching control-type PDU that can be sent as a response from the target. 💅
Retiring an outstanding PDU in this category works like this. If the CmdSN of the PDU the initiator sent in this category is x, the PDU is outstanding until the initiator sends a non-immediate control-type PDU on the same connection with CmdSN = y (where y is at least x) and the target responds with a control-type PDU on any connection where ExpCmdSN is at least y+1. ⭐
When the number of outstanding unexpected control-type PDUs equals MaxOutstandingUnexpectedPDUs, the iSCSI layer at the initiator MUST NOT generate any unexpected PDUs that it otherwise would have generated, even if the unexpected PDU is intended for immediate delivery. 🫶
8.1.2. Flow Control for Control-Type PDUs from the Target
Control-type PDUs that a target can send and that the initiator expects are listed in the Regulated category. (See Section 8.1.1.1.) 🌈
For the control-type PDUs that a target can send and that the initiator does not expect, the number is controlled by MaxOutstandingUnexpectedPDUs declared by the initiator. (See Section 6.7.) After a PDU in this category is sent by a target, it is outstanding until it is retired. At any time, the number of outstanding unexpected PDUs MUST NOT exceed the value of MaxOutstandingUnexpectedPDUs declared by the initiator. The initiator uses the MaxOutstandingUnexpectedPDUs value it declared to work out how much buffer it needs for control-type PDUs in this category that a target can send. Here's the list of PDUs in this category and the rules for retiring each outstanding one:
- a. For an Asynchronous Message PDU with StatSN = x, the PDU is outstanding until the initiator sends a control-type PDU with
ExpStatSNset to at least x+1. 🧚 - b. For a Reject PDU with StatSN = x, which is sent after a task is active, the PDU is outstanding until the initiator sends a control-type PDU with
ExpStatSNset to at least x+1. 💖 - c. For a NOP-In PDU with
ITT = 0xffffffffand StatSN = x, the PDU is outstanding until the initiator responds with a control-type PDU on the same connection whereExpStatSNis at least x+1. But if the NOP-In PDU is sent as a ping request withTTT != 0xffffffff, the PDU can also be retired when the initiator sends a NOP-Out PDU with the sameITTandTTTas in the ping request. Note that when a target sends a NOP-In PDU as a ping request, it must provision a buffer for the NOP-Out PDU sent as a ping response from the initiator. 🎀
When the number of outstanding unexpected control-type PDUs equals MaxOutstandingUnexpectedPDUs, the iSCSI layer at the target MUST NOT generate any unexpected PDUs that it otherwise would have generated, even if its intent is to indicate an iSCSI error condition (e.g., Asynchronous Message, Reject). Task timeouts, as in the initiator's waiting for a command completion or other connection and session-level exceptions, will ensure that correct operational behavior will result in these cases despite not generating the PDU. This rule overrides any other requirements elsewhere that require that a Reject PDU MUST be sent. 🚨
(Implementation note: SCSI task timeout and recovery can be a lengthy process and hence SHOULD be avoided by proper provisioning of resources.)
(Implementation note: To ensure that the initiator has a means to inform the target that outstanding PDUs have been retired, the target should reserve the last unexpected control-type PDU allowable by the value ofMaxOutstandingUnexpectedPDUsdeclared by the initiator for sending a NOP-In ping request withTTT != 0xffffffffto allow the initiator to return the NOP-Out ping response with the currentExpStatSN.) 🧠
8.2. Flow Control for RDMA Read Resources
If iSERHelloRequired is negotiated to "Yes", then the total number of RDMA Read operations that can be active at the same time on an iSCSI/iSER connection depends on the amount of resources allocated, as declared in the iSER Hello exchange described in Section 5.1.3. Going over the number of RDMA Read operations allowed on a connection will get the connection terminated by the RCaP layer 💀 The iSER layer at the target maintains the iSER-ORD to keep track of the maximum number of RDMA Read Requests that the iSER layer can issue on a particular RCaP Stream. 👑
During connection setup (see Section 5.1), iSER-IRD is known at the initiator and iSER-ORD is known at the target. This happens after the iSER layers at the initiator and the target have each allocated the connection resources needed to support RCaP, as directed by the Allocate_Connection_Resources Operational Primitive from the iSCSI layer, before the end of the iSCSI Login Phase. In the Full Feature Phase, if iSERHelloRequired is negotiated to "Yes", then the first message the initiator sends is the iSER Hello Message (see Section 9.3), which contains the value of iSER-IRD. In response to the iSER Hello Message, the target sends the iSER HelloReply Message (see Section 9.4), which contains the value of iSER-ORD. The iSER layer at both the initiator and the target MAY adjust (lower) the resources associated with iSER-IRD and iSER-ORD, respectively, to match the iSER-ORD value declared in the HelloReply Message. The iSER layer at the target MUST control the flow of the RDMA Read Request Messages so that it does not exceed the iSER-ORD value at the target. 🦋
If iSERHelloRequired is negotiated to "No", then the maximum number of RDMA Read operations that can be active is negotiated by other means outside the scope of this document. For example, in InfiniBand, iSER connection setup uses InfiniBand Connection Manager (CM) Management Datagrams (MADs), with additional iSER information exchanged in the private data. 🌸
8.3. STag Management
An STag is an identifier of a Tagged Buffer used in an RDMA operation. If STags are exposed on the wire by being Advertised in the iSER header or declared in the header of an RCaP Message, then the allocation and the later invalidation of the STags are as specified in this document. 🍓
8.3.1. Allocation of STags
When the iSCSI layer at the initiator invokes the Send_Control Operational Primitive to ask the iSER layer at the initiator to process a SCSI Command, zero, one, or two STags may be allocated by the iSER layer. See Section 7.3.1 for details. The number of STags allocated depends on whether the command is unidirectional or bidirectional and whether or not solicited write data transfer is involved. 💜
When the iSCSI layer at the initiator invokes the Send_Control Operational Primitive to ask the iSER layer at the initiator to process a Task Management Function Request with the TASK REASSIGN function, then besides allocating zero, one, or two STags, the iSER layer MUST invalidate the existing STags (if any) associated with the ITT. See Section 7.3.3 for details. ✨
The iSER layer at the target allocates a local Data Sink STag when the iSCSI layer at the target invokes the Get_Data Operational Primitive to ask the iSER layer to process an R2T PDU. See Section 7.3.6 for details. 🫶
8.3.2. Invalidation of STags ✨
okay so invalidating the STags at the initiator when the target sends the SCSI Response PDU at the end of a command is already covered in Section 7.3.2, so go check that one bestie 💅
But what if a unidirectional or bidirectional command wraps up WITHOUT the target ever sending that SCSI Response PDU? 🙈 Then the iSCSI layer at the initiator MUST ask the iSER layer at the initiator to invalidate the STags, by invoking the Deallocate_Task_Resources Operational Primitive qualified with ITT. Next, the iSER layer at the initiator MUST locate any STags in the Local Mapping, and MUST invalidate those STags (if there are any) plus the Local Mapping itself. Clean sweep 🧹✨
For an RDMA Read operation that realizes a SCSI Write data transfer, the iSER layer at the target SHOULD invalidate the Data Sink STag once the RDMA Read operation that references it is done, so the buffer resources can be reused right away 🍓
For an RDMA Write operation that realizes a SCSI Read data transfer, the Data Source STag at the target is never declared to the initiator and never shows up on the wire 🙃 So invalidation of that STag is just not specified, full stop 🤷♀️
And the same vibe applies at the target 💖 When a unidirectional or bidirectional command ends WITHOUT the associated SCSI Response PDU being sent by the target, the iSCSI layer at the target MUST ask the iSER layer at the target to invalidate the STags, by invoking the Deallocate_Task_Resources Operational Primitive qualified with ITT. Then the iSER layer at the target MUST locate the local STags (if any) in the Local Mapping, and MUST invalidate the local STags (if any) and the Local Mapping. Bestie, that's a whole vibe 🌸
9. iSER Control and Data Transfer 🎀
okay this part is the big picture 🦋 For iSCSI data-type PDUs (see Section 7.1), the iSER layer uses RDMA Read and RDMA Write operations to move the solicited data around. For iSCSI control-type PDUs (see Section 7.2), the iSER layer uses Send Messages of RCaP instead 💜
9.1. iSER Header Format 💌
An iSER header MUST be present in every Send Message of RCaP, no exceptions bestie 🙅♀️ It lives in the first 28 bytes of the message payload of the Send Message of RCaP, and here's the layout 👑
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Opcode| Opcode Specific Fields |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Opcode Specific Fields (32 bits) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Opcode Specific Fields (64 bits) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Opcode Specific Fields (32 bits) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Opcode Specific Fields (64 bits) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2: iSER Header FormatOpcode - Operation Code: 4 bits 🔢
The Opcode field tells you what kind of iSER Message you're dealing with ✨ Here are the values:
0001b = iSCSI control-type PDU
0010b = iSER Hello Message
0011b = iSER HelloReply MessageAll other Opcodes are unassigned, so hands off 🙈
9.2. iSER Header Format for iSCSI Control-Type PDU 🧁
okay so the iSER layer uses Send Messages of RCaP to carry iSCSI control-type PDUs (see Section 7.2) 🦋 The message payload of each Send Message of RCaP used for an iSER Message has an iSER Header first, followed by an iSCSI control-type PDU.
The iSER header in a Send Message of RCaP carrying an iSCSI control-type PDU MUST have the format shown in Figure 3, no cap 💅
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |W|R| |
| 0001b |S|S| Reserved |
| |V|V| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Write STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Write Base Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Read STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Read Base Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 3: iSER Header Format for iSCSI Control-Type PDUNow let's break down each field 🎀
WSV - Write STag Valid flag: 1 bit 💜
This flag says whether the Write STag field and the Write Base Offset field are valid 💖 When it's set to one, both fields in this iSER Header are valid. When it's set to zero, the Write STag field and the Write Base Offset field in this iSER Header MUST be ignored at the receiver, so don't even look at them 🙈 The Write STag Valid flag is set to one when there's solicited data to transfer for a SCSI Write or bidirectional command, or when there's non-immediate unsolicited and solicited data to transfer for the referenced task in a Task Management Function Request with the TASK REASSIGN function.
RSV - Read STag Valid flag: 1 bit 🌈
This one is the Read version of the same idea 🦋 It says whether the Read STag field and the Read Base Offset field are valid. When it's set to one, both fields in this iSER Header are valid. When it's set to zero, the Read STag field and the Read Base Offset field in this iSER Header MUST be ignored at the receiver. The Read STag Valid flag is set to one for a SCSI Read or bidirectional command, or a Task Management Function Request with the TASK REASSIGN function.
Write STag - Write Steering Tag: 32 bits 🧚
This field holds the Write STag when the Write STag Valid flag is set to one 💅 For a SCSI Write or bidirectional command, the Write STag is used to Advertise the initiator's I/O Buffer that has the solicited data. For a Task Management Function Request with the TASK REASSIGN function, the Write STag is used to Advertise the initiator's I/O Buffer that has the non-immediate unsolicited data and solicited data. This Write STag gets used as the Data Source STag in the resulting RDMA Read operation(s). When the Write STag Valid flag is set to zero, this field MUST be set to zero and ignored on receive 🫶
Write Base Offset: 64 bits 🍓
This field holds the Base Offset linked to the I/O Buffer for the SCSI Write command when the Write STag Valid flag is set to one. When the Write STag Valid flag is set to zero, this field MUST be set to zero and ignored on receive.
Read STag - Read Steering Tag: 32 bits 🔥
This field holds the Read STag when the Read STag Valid flag is set to one. The Read STag is used to Advertise the initiator's Read I/O Buffer of a SCSI Read or bidirectional command, or a Task Management Function Request with the TASK REASSIGN function. This Read STag gets used as the Data Sink STag in the resulting RDMA Write operation(s). When the Read STag Valid flag is zero, this field MUST be set to zero and ignored on receive.
Read Base Offset: 64 bits 🌸
This field holds the Base Offset linked to the I/O Buffer for the SCSI Read command when the Read STag Valid flag is set to one. When the Read STag Valid flag is set to zero, this field MUST be set to zero and ignored on receive.
Reserved: 🫧
Reserved fields MUST be set to zero on transmit and MUST be ignored on receive. Easy peasy 🎀
9.3. iSER Header Format for iSER Hello Message 🌷
An iSER Hello Message MUST only contain the iSER header, and that header MUST have the format shown in Figure 4 ✨ If iSERHelloRequired is negotiated to "Yes", then the iSER Hello Message is the first iSER Message sent on the RCaP Stream from the iSER layer at the initiator to the iSER layer at the target. Main character energy, it goes first 👑
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | | | | |
| 0010b | Rsvd | MaxVer| MinVer| iSER-IRD |
| | | | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 4: iSER Header Format for iSER Hello MessageMaxVer - Maximum Version: 4 bits 🦋
This field says the maximum version of the iSER protocol that's supported 💖 It MUST be set to 10 to indicate the version of the specification described in this document.
MinVer - Minimum Version: 4 bits 🌈
This field says the minimum version of the iSER protocol that's supported 💜 It MUST be set to 10 to indicate the version of the specification described in this document.
iSER-IRD: 16 bits 🍓
This field holds the value of the iSER-IRD at the initiator.
Reserved (Rsvd): 🧁
Reserved fields MUST be set to zero on transmit and MUST be ignored on receive. Bestie, don't touch the reserved stuff 🫶
9.4. iSER Header Format for iSER HelloReply Message ✨
okay bestie, so an iSER HelloReply Message MUST only contain the iSER header, and that header MUST have the format shown in Figure 5 💅 If iSERHelloRequired is negotiated to "Yes", then the HelloReply is literally the first iSER Message sent on the RCaP Stream from the iSER layer at the target to the iSER layer at the initiator 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |R| | | |
| 0011b |Rsvd |E| MaxVer| CurVer| iSER-ORD |
| | |J| | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 5: iSER Header Format for iSER HelloReply MessageREJ - Reject flag: 1 bit 🙅
This flag tells you whether the target is rejecting this connection. If it's set to one, the target is rejecting the connection.
MaxVer - Maximum Version: 4 bits 🎀
This field specifies the maximum version of the iSER protocol supported. It MUST be set to 10 to indicate the version of the specification described in this document.
CurVer - Current Version: 4 bits 🎀
This field specifies the current version of the iSER protocol supported. It MUST be set to 10 to indicate the version of the specification described in this document.
iSER-ORD: 16 bits 🍓
This field contains the value of the iSER-ORD at the target.
Reserved (Rsvd): 🫶
Reserved fields MUST be set to zero on transmit and MUST be ignored on receive.
9.5. SCSI Data Transfer Operations 🚚
The iSER layer at the initiator and the iSER layer at the target handle each SCSI Write, SCSI Read, and bidirectional operation as described below.
9.5.1. SCSI Write Operation ✍️
The iSCSI layer at the initiator MUST invoke the Send_Control Operational Primitive to request the iSER layer at the initiator to send the SCSI Write Command. The iSER layer at the initiator MUST request the RCaP layer to transmit a Send Message with the message payload consisting of the iSER header followed by the SCSI Command PDU and immediate data (if any). The SendSE Message should be used if supported by the RCaP layer (e.g., iWARP). If there is solicited data, the iSER layer MUST Advertise the Write STag and the Base Offset in the iSER header of the Send Message, as described in Section 9.2. Upon receiving the Send Message, the iSER layer at the target MUST notify the iSCSI layer at the target by invoking the Control_Notify Operational Primitive qualified with the SCSI Command PDU. See Section 7.3.1 for details on the handling of the SCSI Write Command.
For the non-immediate unsolicited data, the iSCSI layer at the initiator MUST invoke a Send_Control Operational Primitive qualified with the SCSI Data-Out PDU. Upon receiving each Send Message containing the non-immediate unsolicited data, the iSER layer at the target MUST notify the iSCSI layer at the target by invoking the Control_Notify Operational Primitive qualified with the SCSI Data-Out PDU. See Section 7.3.4 for details on the handling of the SCSI Data-Out PDU.
For the solicited data, when the iSCSI layer at the target has an I/O Buffer available, it MUST invoke the Get_Data Operational Primitive qualified with the R2T PDU. See Section 7.3.6 for details on the handling of the R2T PDU.
When the data transfer associated with this SCSI Write operation is complete, the iSCSI layer at the target MUST invoke the Send_Control Operational Primitive when it is ready to send the SCSI Response PDU. Upon receiving a Send Message containing the SCSI Response PDU, the iSER layer at the initiator MUST notify the iSCSI layer at the initiator by invoking the Control_Notify Operational Primitive qualified with the SCSI Response PDU. See Section 7.3.2 for details on the handling of the SCSI Response PDU.
9.5.2. SCSI Read Operation 📖
The iSCSI layer at the initiator MUST invoke the Send_Control Operational Primitive to request the iSER layer at the initiator to send the SCSI Read Command. The iSER layer at the initiator MUST request the RCaP layer to transmit a Send Message with the message payload consisting of the iSER header followed by the SCSI Command PDU. The SendSE Message should be used if supported by the RCaP layer (e.g., iWARP). The iSER layer at the initiator MUST Advertise the Read STag and the Base Offset in the iSER header of the Send Message, as described in Section 9.2. Upon receiving the Send Message, the iSER layer at the target MUST notify the iSCSI layer at the target by invoking the Control_Notify Operational Primitive qualified with the SCSI Command PDU. See Section 7.3.1 for details on the handling of the SCSI Read Command.
When the requested SCSI data is available in the I/O Buffer, the iSCSI layer at the target MUST invoke the Put_Data Operational Primitive qualified with the SCSI Data-In PDU. See Section 7.3.5 for details on the handling of the SCSI Data-In PDU.
When the data transfer associated with this SCSI Read operation is complete, the iSCSI layer at the target MUST invoke the Send_Control Operational Primitive when it is ready to send the SCSI Response PDU. The SendInvSE Message should be used if supported by the RCaP layer (e.g., iWARP). Upon receiving the Send Message containing the SCSI Response PDU, the iSER layer at the initiator MUST notify the iSCSI layer at the initiator by invoking the Control_Notify Operational Primitive qualified with the SCSI Response PDU. See Section 7.3.2 for details on the handling of the SCSI Response PDU.
9.5.3. Bidirectional Operation 🔁
The initiator and the target handle the SCSI Write and the SCSI Read portions of this bidirectional operation the same as described in Sections 9.5.1 and 9.5.2, respectively.
10. iSER Error Handling and Recovery 🛟
RCaP provides the iSER layer with reliable in-order delivery. Therefore, the error management needs of an iSER-assisted connection are somewhat different than those of a Traditional iSCSI connection.
10.1. Error Handling 🧯
iSER error handling is described in the following sections, classified loosely based on the sources of errors:
- Those originating at the transport layer (e.g., TCP).
- Those originating at the RCaP layer.
- Those originating at the iSER layer.
- Those originating at the iSCSI layer.
10.1.1. Errors in the Transport Layer 🚧
If the transport layer is TCP, then TCP packets with detected errors are silently dropped by the TCP layer and result in retransmission at the TCP layer. This has no impact on the iSER layer. However, connection loss (e.g., link failure) and unexpected termination (e.g., TCP graceful or abnormal close without the iSCSI Logout exchanges) at the transport layer will cause the iSCSI/iSER connection to be terminated as well.
10.1.1.1. Failure in the Transport Layer Before RCaP Mode is Enabled 🫥
If the connection is lost or terminated before the iSCSI layer invokes the Allocate_Connection_Resources Operational Primitive, the login process is terminated and no further action is required.
If the connection is lost or terminated after the iSCSI layer has invoked the Allocate_Connection_Resources Operational Primitive, then the iSCSI layer MUST request the iSER layer to deallocate all connection resources by invoking the Deallocate_Connection_Resources Operational Primitive.
10.1.1.2. Failure in the Transport Layer After RCaP Mode is Enabled 💔
If the connection is lost or terminated after the iSCSI layer has invoked the Enable_Datamover Operational Primitive, the iSER layer MUST notify the iSCSI layer of the connection loss by invoking the Connection_Terminate_Notify Operational Primitive. Prior to invoking the Connection_Terminate_Notify Operational Primitive, the iSER layer MUST perform the actions described in Section 5.2.3.2.
10.1.2. Errors in the RCaP Layer 🫧
The RCaP layer does not have error recovery operations built in. If errors are detected at the RCaP layer, the RCaP layer will terminate the RCaP Stream and the associated connection.
10.1.2.1. Errors Detected in the Local RCaP Layer 🔍
If an error is encountered at the local RCaP layer, the RCaP layer MAY send a Send Message to the Remote Peer to report the error if possible. (For iWARP, see [RDMAP] for the list of errors where a Terminate Message is sent.) The RCaP layer is responsible for terminating the connection. After the RCaP layer notifies the iSER layer that the connection is terminated, the iSER layer MUST notify the iSCSI layer by invoking the Connection_Terminate_Notify Operational Primitive. Prior to invoking the Connection_Terminate_Notify Operational Primitive, the iSER layer MUST perform the actions described in Section 5.2.3.2.
10.1.2.2. Errors Detected in the RCaP Layer at the Remote Peer 🛰️
If an error is encountered at the RCaP layer at the Remote Peer, the RCaP layer at the Remote Peer may send a Send Message to report the error if possible. If it is unable to send a Send Message, the connection is terminated. This is treated the same as a failure in the transport layer after RDMA is enabled, as described in Section 10.1.1.2.
If an error is encountered at the RCaP layer at the Remote Peer and it is able to send a Send Message, the RCaP layer at the Remote Peer is responsible for terminating the connection. After the local RCaP layer notifies the iSER layer that the connection is terminated, the iSER layer MUST notify the iSCSI layer by invoking the Connection_Terminate_Notify Operational Primitive. Prior to invoking the Connection_Terminate_Notify Operational Primitive, the iSER layer MUST perform the actions described in Section 5.2.3.2.
10.1.3. Errors in the iSER Layer 🧩
The error handling due to errors at the iSER layer is described in the following sections.
10.1.3.1. Insufficient Connection Resources to Support RCaP at Connection Setup 🎀
Okay bestie, this one is about what happens when there aren't enough connection resources to even set up RCaP during login 😭. After the iSCSI layer at the initiator invokes Allocate_Connection_Resources during the iSCSI login negotiation phase, if the iSER layer at the initiator fails to allocate the connection resources needed to support RCaP, it MUST return a status of failure to the iSCSI layer at the initiator. The iSCSI layer at the initiator MUST terminate the connection as described in Section 5.2.3.1.
Now the target side 👀. After the iSCSI layer at the target invokes Allocate_Connection_Resources during the iSCSI login negotiation phase, if the iSER layer at the target fails to allocate the connection resources needed to support RCaP, it MUST return a status of failure to the iSCSI layer at the target. The iSCSI layer at the target MUST send a Login Response with a Status-Class of 0x03 (Target Error), and a Status-Code of 0x02 (Out of Resources). The iSCSI layers at the initiator and the target MUST terminate the connection as described in Section 5.2.3.1. 💀
10.1.3.2. iSER Negotiation Failures 🌸
If iSERHelloRequired is negotiated to "Yes" and the RCaP or iSER related parameters declared by the initiator in the iSER Hello Message are unacceptable to the iSER layer at the target, the iSER layer at the target MUST set the Reject (REJ) flag, as described in Section 9.4, in the iSER HelloReply Message. The following are the cases when the iSER layer MUST set the REJ flag to 1 in the HelloReply Message:
- The initiator-declared iSER-IRD value is greater than 0, and the target-declared iSER-ORD value is 0.
- The initiator-supported and the target-supported iSER protocol versions do not overlap.
After requesting the RCaP layer to send the iSER HelloReply Message, the handling of the error situation is the same as that for iSER format errors as described in Section 10.1.3.3. 🫶
10.1.3.3. iSER Format Errors 🦋
The following types of errors in an iSER header are considered format errors, so pay attention 👑:
- Illegal contents of any iSER header field
- Inconsistent field contents in an iSER header
- Length error for an iSER Hello or HelloReply Message (see Sections 9.3 and 9.4)
When a format error is detected, the following events MUST occur in the specified sequence:
- The iSER layer MUST request the RCaP layer to terminate the RCaP Stream. The RCaP layer MUST terminate the associated connection.
- The iSER layer MUST notify the iSCSI layer of the connection termination by invoking the
Connection_Terminate_NotifyOperational Primitive. Prior to invoking theConnection_Terminate_NotifyOperational Primitive, the iSER layer MUST perform the actions described in Section 5.2.3.2.
10.1.3.4. iSER Protocol Errors 🧚
If iSERHelloRequired is negotiated to "Yes", then the first iSER Message sent by the iSER layer at the initiator MUST be the iSER Hello Message (see Section 9.3). In this case the first iSER Message sent by the iSER layer at the target MUST be the iSER HelloReply Message (see Section 9.4). Failure to send the iSER Hello or HelloReply Message, as indicated by the wrong Opcode in the iSER header, is a protocol error. Conversely, if the iSER Hello Message is sent by the iSER layer at the initiator when iSERHelloRequired is negotiated to "No", the iSER layer at the target MAY treat this as a protocol error or respond with an iSER HelloReply Message. The handling of iSER protocol errors is the same as that for iSER format errors as described in Section 10.1.3.3.
If the sending side of an iSER-enabled connection acts in a manner not permitted by the negotiated or declared login/text operational key values as described in Section 6, this is a protocol error and the receiving side MAY handle this the same as for iSER format errors as described in Section 10.1.3.3. 💅
10.1.4. Errors in the iSCSI Layer ✨
The error handling due to errors at the iSCSI layer is described in the following sections. For error recovery, see Section 10.2.
10.1.4.1. iSCSI Format Errors 💖
When an iSCSI format error is detected, the iSCSI layer MUST request the iSER layer to terminate the RCaP Stream by invoking the Connection_Terminate Operational Primitive. For more details on connection termination, see Section 5.2.3.1.
10.1.4.2. iSCSI Digest Errors 🙈
In the iSER-assisted mode, the iSCSI layer will not see any digest error because both the HeaderDigest and the DataDigest keys are negotiated to "None". No drama there 💅.
10.1.4.3. iSCSI Sequence Errors 🌈
For Traditional iSCSI, sequence errors are caused by dropped PDUs due to header or data digest errors. Since digests are not used in iSER-assisted mode and the RCaP layer will deliver all messages in the order they were sent, sequence errors will not occur in iSER-assisted mode. Iconic behavior 👑.
10.1.4.4. iSCSI Protocol Error 💀
When the iSCSI layer handles certain protocol errors by dropping the connection, the error handling is the same as that for iSCSI format errors as described in Section 10.1.4.1.
When the iSCSI layer uses the iSCSI Reject PDU and response codes to handle certain other protocol errors, no special handling at the iSER layer is required.
10.1.4.5. SCSI Timeouts and Session Errors 🍓
This is handled at the iSCSI layer, and no special handling at the iSER layer is required.
10.1.4.6. iSCSI Negotiation Failures 🎀
For negotiation failures that happen during the Login Phase at the initiator after the iSCSI layer has invoked the Allocate_Connection_Resources Operational Primitive and before the Enable_Datamover Operational Primitive has been invoked, the iSCSI layer MUST request the iSER layer to deallocate all connection resources by invoking the Deallocate_Connection_Resources Operational Primitive. The iSCSI layer at the initiator MUST terminate the connection.
For negotiation failures during the Login Phase at the target, the iSCSI layer can use a Login Response with a Status-Class other than 0 (success) to terminate the Login Phase. If the iSCSI layer has invoked the Allocate_Connection_Resources Operational Primitive and has not yet invoked the Enable_Datamover Operational Primitive, the iSCSI layer at the target MUST request the iSER layer at the target to deallocate all connection resources by invoking the Deallocate_Connection_Resources Operational Primitive. The iSCSI layer at both the initiator and the target MUST terminate the connection.
During the iSCSI Login Phase, if the iSCSI layer at the initiator receives a Login Response from the target with a Status-Class other than 0 (Success) after the iSCSI layer at the initiator has invoked the Allocate_Connection_Resources Operational Primitive, the iSCSI layer MUST request the iSER layer to deallocate all connection resources by invoking the Deallocate_Connection_Resources Operational Primitive. The iSCSI layer MUST terminate the connection in this case.
For negotiation failures during the Full Feature Phase, the error handling is left to the iSCSI layer and no special handling at the iSER layer is required.
10.2. Error Recovery 🦋
Error recovery requirements of iSCSI/iSER are the same as that of Traditional iSCSI. All three ErrorRecoveryLevels as defined in [iSCSI] are supported in iSCSI/iSER.
- For ErrorRecoveryLevel 0, session recovery is handled by iSCSI and no special handling by the iSER layer is required.
- For ErrorRecoveryLevel 1, see Section 10.2.1 on PDU Recovery.
- For ErrorRecoveryLevel 2, see Section 10.2.2 on Connection Recovery.
The iSCSI layer may invoke the Notice_Key_Values Operational Primitive during connection setup to request the iSER layer to take note of the value of the operational ErrorRecoveryLevel, as described in Sections 5.1.1 and 5.1.2. 🧠
10.2.1. PDU Recovery 🌸
As described in Sections 10.1.4.2 and 10.1.4.3, digest and sequence errors will not occur in the iSER-assisted mode. If the RCaP layer detects an error, it will close the iSCSI/iSER connection, as described in Section 10.1.2. Therefore, PDU recovery is not useful in the iSER-assisted mode.
The iSCSI layer at the initiator SHOULD disable iSCSI timeout-driven PDU retransmissions. 🙅♀️
10.2.2. Connection Recovery 🫶
The iSCSI layer at the initiator MAY reassign connection allegiance for non-immediate commands that are still in progress and are associated with the failed connection by using a Task Management Function Request with the TASK REASSIGN function. See Section 7.3.3 for more details.
When the iSCSI layer at the initiator does a task reassignment for a SCSI Write command, it MUST qualify the Send_Control Operational Primitive invocation with DataDescriptorOut, which defines the I/O Buffer for both the non-immediate unsolicited data and the solicited data. This allows the iSCSI layer at the target to use recovery R2Ts to request data originally sent as unsolicited and solicited from the initiator. 🔥
When the iSCSI layer at the target accepts a reassignment request for a SCSI Read command, it MUST request the iSER layer to process SCSI Data-In for all unacknowledged data by invoking the Put_Data Operational Primitive. See Section 7.3.5 on the handling of SCSI Data-In.
When the iSCSI layer at the target accepts a reassignment request for a SCSI Write command, it MUST request the iSER layer to process a recovery R2T for any non-immediate unsolicited data and any solicited data sequences that have not been received by invoking the Get_Data Operational Primitive. See Section 7.3.6 on the handling of Ready To Transfer (R2T).
The iSCSI layer at the target MUST NOT issue recovery R2Ts on an iSCSI/iSER connection for a task for which the connection allegiance was never reassigned. The iSER layer at the target MAY reject such a recovery R2T received via the Get_Data Operational Primitive invocation from the iSCSI layer at the target, with an appropriate error code. 🙈
The iSER layer at the target will process the requests invoked by the Put_Data and Get_Data Operational Primitives for a reassigned task in the same way as for the original commands. ✨
11. Security Considerations 🔒✨
Okay bestie, security is where the real drama lives 😭🔐 When iSER sits on top of an RCaP layer and gives iSCSI its RDMA extensions, iSER's security considerations are the same as the ones for the RCaP layer underneath. For iWARP, that's covered in [RDMAP] and [RDDPSEC], plus the updates to both of those that live in [IPSEC-IPS]. 💅
Since iSER-assisted iSCSI is still basically iSCSI from a security standpoint, every iSCSI security requirement in [iSCSI] still applies 🫶 If iSER runs on a non-IP-based RCaP layer, then all the security protocol mechanisms that work for that RCaP layer also work for an iSCSI/iSER connection. Now if iSER runs on a non-IP protocol, the IPsec mechanism from [iSCSI] MUST be implemented at every point where iSER enters the IP network (like through gateways) 🚪 And the non-IP protocol SHOULD implement a packet-by-packet security protocol (optional to use) that is just as strong as the IPsec mechanism [iSCSI] describes. Strong, like, iconic 🔥
Okay, now the resource exhaustion part, which is seriously important ⚠️ To protect the target's RCaP connection resources from resource exhaustion attacks, allocating those resources for a new connection MUST be delayed until it's pretty certain the connection isn't part of an attack. (For example, that means waiting until after the SecurityNegotiation stage of Login.) See Section 5.1.2 🧱
A valid STag exposes I/O Buffer resources to the network so the RCaP can reach them 👀 The security measures for RCAP and iSER from the paragraphs above can protect the data in an I/O buffer from being disclosed or modified without permission. These measures matter even more for implementations that hold onto (like, cache) STags to use across multiple tasks (like iSCSI I/O operations), because those resources stay exposed to the network for a longer time 😰
Another way to control I/O Buffer exposure is to invalidate the STag after its task finishes, as described in Section 1.5.1 🧹 Send with Invalidate messages (which cause remote STag invalidation) are OPTIONAL. That means the iSER layer MUST NOT count on its Remote Peer using a Send with Invalidate to invalidate the STag locally 🙅♀️ If an STag should be invalid after a task is done, the iSER layer MUST check it and invalidate it if it's still valid. No loose ends, periodt 💜
12. IANA Considerations 📋
IANA added these entries to the "iSCSI Login/Text Keys" registry 🎀
IANA also updated these entries in the "iSCSI Login/Text Keys" registry so they point to this RFC 🌸
InitiatorRecvDataSegmentLengthMaxOutstandingUnexpectedPDUsRDMAExtensionsTargetRecvDataSegmentLength
And the reference to RFC 5046 for the "iSCSI Login/Text Keys" registry was switched over to point to this RFC too 🔄
IANA updated the registrations for iSER Opcodes 1-3 in the "iSER Opcodes" registry so they reference this RFC ✨ The reference to RFC 5046 for the "iSER Opcodes" registry also got changed to refer to this RFC 💖
13. References 📚
13.1. Normative References
- [RFC5046] Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah, H., and P. Thaler, "Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)", RFC 5046, October 2007.
- [iSCSI] Chadalapaka, M., Satran, J., Meth, K., and D. Black, "Internet Small Computer System Interface (iSCSI) Protocol (Consolidated)", RFC 7143, April 2014.
- [RDMAP] Recio, R., Metzler, B., Culley, P., Hilland, J., and D. Garcia, "A Remote Direct Memory Access Protocol Specification", RFC 5040, October 2007.
- [DDP] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct Data Placement over Reliable Transports", RFC 5041, October 2007.
- [MPA] Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, "Marker PDU Aligned Framing for TCP Specification", RFC 5044, October 2007.
- [RDDPSEC] Pinkerton, J. and E. Deleganes, "Direct Data Placement Protocol (DDP) / Remote Direct Memory Access Protocol (RDMAP) Security", RFC 5042, October 2007.
- [TCP] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981.
- [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
- [IPSEC-IPS] Black, D. and P. Koning, "Securing Block Storage Protocols over IP: RFC 3723 Requirements Update for IPsec v3", RFC 7146, April 2014.
13.2. Informative References
- [SAM5] INCITS Technical Committee T10, "SCSI Architecture Model - 5 (SAM-5)", T10/BSR INCITS 515 rev 04, Committee Draft.
- [iSCSI-SAM] Knight, F. and M. Chadalapaka, "Internet Small Computer System Interface (iSCSI) SCSI Features Update", RFC 7144, April 2014.
- [DA] Chadalapaka, M., Hufferd, J., Satran, J., and H. Shah, "DA: Datamover Architecture for the Internet Small Computer System Interface (iSCSI)", RFC 5047, October 2007.
- [IB] InfiniBand Architecture Specification Volume 1 Release 1.2, October 2004
- [IPoIB] Chu, J. and V. Kashyap, "Transmission of IP over InfiniBand (IPoIB)", RFC 4391, April 2006.
Appendix A. Summary of Changes from RFC 5046 📝
All of these changes are backward compatible with RFC 5046 except for item #8 🙈 Item #8 matches every known iSER implementation, and each of them already made that change even though RFC 5046 didn't have it. So a hypothetical implementation based on RFC 5046 won't interoperate with one based on this version of the spec 💀
- Removed the requirement that a connection be opened in "normal" TCP mode and transitioned to zero-copy mode. This lets the spec match existing implementations for both InfiniBand and iWARP. Changes were made in Sections 1, 3.1.6, 4.2, 5.1, 5.1.1, 5.1.2, 5.1.3, 10.1.3.4, and 11.
- Added a clause in Section 6.2 clarifying that
MaxRecvDataSegmentLengthmust be ignored if it's declared in the Login Phase. - Added a clause in Section 6.2 clarifying that the initiator must not send more than
InitiatorMaxRecvDataSegmentLengthworth of data when aNOP-Outrequest goes out with a valid Initiator Task Tag. SinceInitiatorMaxRecvDataSegmentLengthcan be smaller thanTargetMaxRecvDataSegmentLength, sending the original data back in the NOP-Out request could overflow the receive buffer unless the data sent with it is shorter thanInitiatorMaxRecvDataSegmentLength😬 - Added a SHOULD negotiate recommendation for
MaxOutstandingUnexpectedPDUsin Section 6.7. - Added the
MaxAHSLengthkey in Section 6.8 to set a limit on the AHS Length. This helps when posting receive buffers, since you know the maximum possible message length for a PDU that contains AHS 📦 - Added the
TaggedBufferForSolicitedDataOnlykey in Section 6.9 to say how the memory region will be used. An initiator can treat regions meant for unsolicited data and regions meant for solicited data differently, and can use different registration modes ✨ RFC 5046 treated the memory holding the data as one contiguous (or virtually contiguous, via scatter-gather mechanisms) and homogeneous region. This new key lets different memory models fit in 🧩 Changes were also made in Section 7.3.1. - Added the
iSERHelloRequiredkey in Section 6.10 so an initiator can allocate connection resources after the login process, by requiring iSER Hello messages before it sends iSCSI PDUs 🤝 The default is "No" because iSER Hello messages haven't been implemented and aren't in use. Changes were made in Sections 5.1.1, 5.1.2, 5.1.3, 8.2, 9.3, 9.4, 10.1.3.2, and 10.1.3.4. - Added two 64-bit fields to the iSER header in Section 9.2, for the Read Base Offset and the Write Base Offset, to support a non-zero Base Offset 🦋 This lets one implementation, like the Open Fabrics Enterprise Distribution (OFED) stack, work in both the InfiniBand and the iWARP environment.
Changes were made to the definitions of Base Offset, Advertisement, and Tagged Buffer. Changes were also made in Sections 1.5.1, 1.6, 1.7, 7.3.1, 7.3.3, 7.3.5, 7.3.6, 9.1, 9.3, 9.4, 9.5.1, and 9.5.2. This change is not backward compatible with RFC 5046, but it was part of all known implementations of iSER when this document was developed.
- Removed iWARP-specific behavior 🧹 Changes were made to the definitions of RDMA Operation and Send Message Type.
Clarifications were added in Section 1.5.2 about using SendSE and SendInvSE. These clarifications reflect removing the RFC 5046 requirements for using these messages, since implementations never followed RFC 5046 in this area. Changes affecting Send with Invalidate were made in Sections 1.5.1, 1.6, 1.7, 4.1, and 7.3.2. Changes affecting Terminate were made in Sections 10.1.2.1 and 10.1.2.2. Changes were made in Appendix B to remove iWARP headers.
- Removed the denial-of-service descriptions for the initiator in Section 5.1.1, since those only apply to the target 🎯
- Clarified in Section 1.5.1 that STag invalidation is the initiator's responsibility for security reasons, and that the initiator can't rely on the target using an Invalidate version of Send 🔐 Added text in Section 11 on Stag invalidation 💜
Appendix B. Message Format for iSER 💌
This section is for information only and is NOT part of the standard. 🌈
B.1. iWARP Message Format for iSER Hello Message
okay bestie, this is the iSER Hello Message wrapped up inside an iWARP SendSE Message 💖 the whole thing is literally giving layers 🧁
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0010b | Zeros | 0001b | 0001b | iSER-IRD |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| All Zeros |
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 6: SendSE Message Containing an iSER Hello Message
B.2. iWARP Message Format for iSER HelloReply Message
next up is the iSER HelloReply Message, also tucked inside an iWARP SendSE Message 🌸 for this one the Reject (REJ) flag is set to zero, so it's not rejecting anyone today, periodt 🫶
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0011b |Zeros|0| 0001b | 0001b | iSER-ORD |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 7: SendSE Message Containing an iSER HelloReply Message
B.3. iSER Header Format for SCSI Read Command PDU
this one's a SCSI Read Command PDU tucked into an iSER Message 🦋 in this example the iSER header is set up like this: the Write STag Valid flag is set to zero, the Read STag Valid flag is set to one, the Write STag field is all zeros, the Write Base Offset field is all zeros, the Read STag field holds a valid Read STag, and the Read Base Offset field holds a valid Base Offset for the Read Tagged Buffer. main character header energy ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0001b |0|1| All zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Read STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Read Base Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SCSI Read Command PDU |
// //
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 8: iSER Header Format for SCSI Read Command PDU
B.4. iSER Header Format for SCSI Write Command PDU
now the mirror image: a SCSI Write Command PDU inside an iSER Message 💜 for this example the Write STag Valid flag is set to one and the Read STag Valid flag is set to zero. The Write STag field holds a valid Write STag, and the Write Base Offset field holds a valid Base Offset for the Write Tagged Buffer. The Read STag field is all zeros since it isn't used, and the Read Base Offset field is all zeros too 🙈
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0001b |1|0| All zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Write STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Write Base Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SCSI Write Command PDU |
// //
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 9: iSER Header Format for SCSI Write Command PDU
B.5. iSER Header Format for SCSI Response PDU
last one in this batch: a SCSI Response PDU embedded in an iSER Message:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0001b |0|0| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SCSI Response PDU |
// //
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 10: iSER Header Format for SCSI Response PDU
Appendix C. Architectural Discussion of iSER over InfiniBand ✨
okay so this appendix is the big-picture tour of how an InfiniBand network with Gateways is set up 🧚 it's purely informational, but it's super helpful for seeing how iSER actually gets used in an InfiniBand environment. no cap, the diagrams are the moment 🔥
C.1. Host Side of iSCSI and iSER Connections in InfiniBand
Figure 11 lays out the topologies where iSCSI and iSER are able to operate on an InfiniBand Network 🌈
+---------+ +---------+ +---------+ +---------+ +--- -----+
| Host | | Host | | Host | | Host | | Host |
| | | | | | | | | |
+---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+
|HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA|
+-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+
|----+------|-----+-----|-----+-----|-----+-----|-----+---> To IB
IB| IB | IB | IB | IB | SubNet2 SWTCH
+-v-----------v-----------v-----------v-----------v---------+
| InfiniBand Switch for Subnet1 |
+---+-----+--------+-----+--------+-----+------------v------+
| TCA | | TCA | | TCA | |
+-----+ +-----+ +-----+ | IB
/ IB \ / IB \ / \ +--+--v--+--+
| iSER | | iSER | | IPoIB | | | TCA | |
| Gateway | | Gateway | | Gateway | | +-----+ |
| to | | to | | to | | Storage |
| iSCSI | | iSER | | IP | | Controller|
| TCP | | iWARP | |Ethernet | +-----+-----+
+---v-----| +---v-----| +----v----+
| EN | EN | EN
+--------------+---------------+----> to IP based storage
Ethernet links that carry iSCSI or iWARPFigure 11: iSCSI and iSER on IB
okay so in Figure 11 the Host systems hook up to the InfiniBand links through their InfiniBand Host Channel Adapters (HCAs) 💅 with the help of IB switch(es), those links connect each HCA to InfiniBand Target Channel Adapters (TCAs) sitting in gateways or Storage Controllers. an iSER-capable IB-IP Gateway converts iSER Messages wrapped in IB protocols into either standard iSCSI, or iSER Messages for iWARP 🎀 and an [IPoIB] Gateway converts the InfiniBand [IPoIB] protocol into IP protocol. in the iSCSI case, that lets iSCSI run over an IB Network between the Hosts and the [IPoIB] Gateway 🫶
C.2. Storage Side of iSCSI and iSER Mixed Network Environment ✨
okay bestie, picture a storage controller 🏰 with three different portal groups going on at once 😭 One is iSCSI only (TPG-4), one does iSER/iWARP or iSCSI (TPG-2), and one does iSER/IB (TPG-1). And yes, "TPG" just means "Target Portal Group" 🎀
| | |
| | |
+--+--v--+----------+--v--+----------+--v--+--+
| | IB | |iWARP| | EN | |
| | | | TCP | | NIC | |
| |(TCA)| | RNIC| | | |
| +-----| +-----+ +-----+ |
| TPG-1 TPG-2 TPG-4 |
| 9.1.3.3 9.1.2.4 9.1.2.6 |
| |
| Storage Controller |
| |
+---------------------------------------------+Figure 12: Storage Controller with TCP, iWARP, and IB Connections 🦋
The usual iSCSI portal group advertising processes (SLP, iSNS, or SendTargets) are all available to a Storage Controller 💅
C.3. Discovery Processes for an InfiniBand Host 🧚
so an InfiniBand Host can grab portal group IP addresses from SLP, iSNS, or SendTargets 🌸 It does this by using TCP/IP over [IPoIB] 💜 Once it has one or more remote portal IP addresses, the Initiator uses the standard IP mechanisms to figure out the local outgoing interface and the destination hardware address. That hardware address is either an Ethernet MAC or an InfiniBand Global Identifier (GID) of the target, or of a gateway leading to the target 👑 If the interface it resolves to is an [IPoIB] network interface, then the target portal is reachable through an InfiniBand fabric ✨ In that case the Initiator can set up an iSCSI/TCP or iSCSI/iSER session with the Target over that InfiniBand interface. It uses the hardware address (the InfiniBand GID) that it got through the standard Address Resolution Protocol (ARP) processes 🫶
If more than one IP address comes back from discovery, the Initiator should select a Target IP address that's on the same IP subnet as the Initiator, if one exists 🌈 This helps avoid the extra overhead of going through a gateway when a direct path already exists. no cap, that's just smart 🧠
Also, a user can set up manual static IP route entries if they want a particular path to the target 🍓
C.4. IBTA Connection Specifications 💖
this part is outside the scope of this document, but it's expected that the InfiniBand Trade Association (IBTA) has or will define the following 🔥
- The iSER ServiceID 🎀
- A means for letting a Host establish a connection with a peer InfiniBand end-node, and for that peer to indicate when that end-node supports iSER. That way the Host could fall back to iSCSI/TCP over [IPoIB] if it needs to 🌸
- A means for letting the Host establish connections with IB iSER connections on storage controllers or IB iSER-connected Gateways. These are preferred over IPoIB-connected Gateways/Bridges, or over connections to Target Storage Controllers that also accept iSCSI via [IPoIB] 💅
- A means for combining the IB ServiceID for iSER and the IP port number. This way the IB Host can use normal IB connection processes, and the iSER target peer can still actually connect to the required IP port number 🦋
Appendix D. Acknowledgments 🫶
the authors want to thank these lovely people for finding implementation issues and/or suggesting ways to fix the issues clarified in this document 💜 Robert Russell, Arne Redlich, David Black, Mallikarjun Chadalapaka, Tom Talpey, Felix Marti, Robert Sharp, Caitlin Bestler, Hemal Shah, Spencer Dawkins, Pete Resnick, Ted Lemon, Pete McCann, and Steve Kent 👑 Credit also goes to the authors of the original iSER Specification [RFC5046], which are Michael Ko, Mallikarjun Chadalapaka, John Hufferd, Uri Elzur, Hemal Shah, and Patricia Thaler ✨ This document benefited from all of their contributions 🥹
Authors' Addresses 🎀
Michael Ko
EMail: mkosjc@gmail.com
Alexander Nezhinsky
Mellanox Technologies
13 Zarchin St.
Raanana 43662
Israel
Phone: +972-74-712-9000
EMail: alexandern@mellanox.com, nezhinsky@gmail.com📦 where this page comes from
- ietf: rfc7145 2014-04
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms