RFC 3347

Small Computer Systems Interface protocol over the Internet (iSCSI) Requirements and Design Considerations

📅 published July 2002, from ietf

RFC 3347 🎀 Small Computer Systems Interface protocol over the Internet (iSCSI): Requirements and Design Considerations

Okay bestie, this is RFC 3347 💖 Its full title is "Small Computer Systems Interface protocol over the Internet (iSCSI) Requirements and Design Considerations." It came out of the Network Working Group in July 2002 🌸

It's marked Category: Informational, so it's sharing useful info, not an official standard 🧚 The authors are M. Krueger and R. Haagens from Hewlett-Packard Corporation, C. Sapuntzakis from Stanford, and M. Bakke from Cisco Systems 👑 The header has no Obsoletes or Updates line, so this RFC doesn't replace or change any other RFC 💅

Status of this Memo ✨

This memo gives info to the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited 🦋

Copyright (C) The Internet Society (2002). All Rights Reserved. 💜

Abstract 🍓

This doc lays out what iSCSI and its related infrastructure should be able to do, plus the design thinking behind the iSCSI protocol work 🧠 The IPS group wanted iSCSI adopted quickly, so for version one they chose to build on the existing SCSI architecture and commands, and on the existing TCP/IP transport layer 🚀 Both of these are widely deployed and well understood, so the plan is to avoid inventing much new stuff. That means faster adoption and better compatibility with Internet architecture, protocols, and equipment 🫶

Conventions used in this document 💅

This doc describes requirements for a protocol design but doesn't define a protocol standard 🎀 Still, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" are to be read as described in RFC-2119 [2] ✨

Table of Contents 📜

   1.   Introduction.................................................2
   2.   Summary of Requirements......................................3
   3.   iSCSI Design Considerations..................................7
   3.1. General Discussion...........................................7
   3.2. Performance/Cost.............................................9
   3.3. Framing.....................................................11
   3.4. High bandwidth, bandwidth aggregation.......................13
   4.   Ease of implementation/complexity of protocol...............14
   5.   Reliability and Availability................................15
   5.1. Detection of Data Corruption................................15
   5.2. Recovery....................................................15
   6.   Interoperability............................................16
   6.1. Internet infrastructure.....................................16
   6.2. SCSI........................................................16
   7.   Security Considerations.....................................18
   7.1. Extensible Security.........................................18
   7.2. Authentication..............................................18
   7.3. Data Integrity..............................................19
   7.4. Data Confidentiality........................................19
   8.   Management..................................................19
   8.1. Naming......................................................20
   8.2. Discovery...................................................21
   9.   Internet Accessibility......................................21
   9.1. Denial of Service...........................................21
   9.2. NATs, Firewalls and Proxy servers...........................22
   9.3. Congestion Control and Transport Selection..................22
   10.  Definitions.................................................22
   11.  References..................................................23
   12.  Acknowledgements............................................24
   13.  Author's Addresses..........................................25
   14.  Full Copyright Statement....................................26

1. Introduction 🧚‍♀️

The IP Storage Working group has a charter to build complete tech for moving block storage data over IP protocols 💫 This includes a protocol for carrying the Small Computer Systems Interface (SCSI) protocol over the Internet, called iSCSI. The first version of iSCSI will define how SCSI transport protocol maps onto TCP/IP. The goal is to let SCSI storage controllers, mainly disk and tape arrays and libraries, connect to IP networks, especially Gigabit Ethernet (GbE) and 10 Gigabit Ethernet (10 GbE) 🌟

iSCSI maps SCSI onto TCP, and that makes it a "SCSI transport." The ANSI T10 document, SCSI SAM-2 [SAM2, p. 3, "Transport Protocols"], defines that term 🔥

2. Summary of Requirements ✨

okay so this section is basically the checklist 📋 the iSCSI standard has to tick off. it's pulled from all the earlier design sections, and i'm keeping every single requirement, exact force and all, because implementers are literally going to build against these 🫶

From section 3.2 Performance/Cost 💸

  • MUST allow implementations to equal or improve on the current state of the art for SCSI interconnects.
  • MUST enable cost competitive implementations.
  • SHOULD minimize control overhead to enable low delay communications.
  • MUST provide high bandwidth and bandwidth aggregation.
  • MUST have low host CPU utilizations, equal to or better than current technology.
  • MUST be possible to build I/O adapters that handle the entire SCSI task.
  • SHOULD permit direct data placement architectures.
  • MUST NOT impose complex operations on host software.
  • MUST provide for full utilization of available link bandwidth.
  • MUST allow an implementation to exploit parallelism (multiple connections) at the device interfaces and within the interconnect fabric.

From section 3.4 High Bandwidth/Bandwidth Aggregation 🚀

  • MUST operate over a single TCP connection.
  • SHOULD support 'connection binding', and it MUST be optional to implement.

From section 4 Ease of Implementation/Complexity of Protocol 🧩

  • SHOULD keep the protocol simple.
  • SHOULD minimize optional features.
  • MUST specify feature negotiation at session establishment (login).
  • MUST operate correctly when no optional features are negotiated as well as when individual option negotiations are unsuccessful.

From section 5.1 Detection of Data Corruption 🔍

  • MUST support a data integrity check format for use in digest generation.
  • MAY use separate digest for data and headers.
  • iSCSI header format SHOULD be extensible to include other data integrity digest calculation methods.

From section 5.2 Recovery 🔄

  • MUST specify mechanisms to recover in a timely fashion from failures on the initiator, target, or connecting infrastructure.
  • MUST specify recovery methods for non-idempotent requests.
  • SHOULD take into account fail-over schemes for mirrored targets or highly available storage configurations.
  • SHOULD provide a method for sessions to be gracefully terminated and restarted that can be initiated by either the initiator or target.

From section 6 Interoperability 🤝

  • iSCSI protocol document MUST be clear and unambiguous.

From section 6.1 Internet Infrastructure 🌐

  • MUST:
    • be compatible with both IPv4 and IPv6
    • use TCP connections conservatively, keeping in mind there may be many other users of TCP on a given machine.
  • MUST NOT require changes to existing Internet protocols.
  • SHOULD minimize required changes to existing TCP/IP implementations.
  • MUST be designed to allow future substitution of SCTP (for TCP) as an IP transport protocol with minimal changes to iSCSI protocol operation, protocol data unit (PDU) structures and formats.

From section 6.2 SCSI 🦋

  • Any feature SAM2 requires in a valid transport mapping MUST be specified by iSCSI.
  • MUST specify strictly ordered delivery of SCSI commands over an iSCSI session between an initiator/target pair.
  • The command ordering mechanism SHOULD seek to minimize the amount of communication necessary across multiple adapters doing transport off-load.
  • MUST specify for each feature whether it is OPTIONAL, RECOMMENDED or REQUIRED to implement and/or use.
  • MUST NOT require changes to the SCSI-3 command sets and SCSI client code except where SCSI specifications point to "transport dependent" fields and behavior.
  • SHOULD track changes to SCSI and the SCSI Architecture Model.
  • MUST be capable of supporting all SCSI-3 command sets and device types.
  • SHOULD support ACA implementation.
  • MUST allow for the construction of gateways to other SCSI transports.
  • MUST reliably transport SCSI commands from the initiator to the target.
  • MUST correctly deal with iSCSI packet drop, duplication, corruption, stale packets, and re-ordering.

From section 7.1 Extensible Security 🔐

  • SHOULD require minimal configuration and overhead in the insecure operation.
  • MUST provide for strong authentication when increased security is required.
  • SHOULD allow integration of new security mechanisms without breaking backwards compatible operation.

From section 7.2 Authentication 🗝️

  • MAY support various levels of authentication security.
  • MUST support private authenticated login.
  • iSCSI authenticated login MUST be resilient against attacks.
  • MUST support data origin authentication of its communications; data origin authentication MAY be optional to use.

From section 7.3 Data Integrity 🛡️

  • SHOULD NOT preclude use of additional data integrity protection protocols (IPSec, TLS).

From section 7.4 Data Confidentiality 🙈

  • MUST provide for the use of a data encryption protocol such as TLS or IPsec ESP to provide data confidentiality between iSCSI endpoints

From section 8 Management 🛠️

  • SHOULD be manageable using standard IP-based management protocols.
  • iSCSI protocol document MUST NOT define the management architecture for iSCSI, or make explicit references to management objects such as MIB variables.

From section 8.1 Naming 🏷️

  • MUST support the naming architecture of SAM-2. The means by which an iSCSI resource is located MUST use or extend existing Internet standard resource location methods.
  • MUST provide a means of identifying iSCSI targets by a unique identifier that is independent of the path on which it is found.
  • The format for the iSCSI names MUST use existing naming authorities.
  • An iSCSI name SHOULD be a human readable string in an international character set encoding.
  • Standard Internet lookup services SHOULD be used to resolve iSCSI names.
  • SHOULD deal with the complications of the new SCSI security architecture.
  • iSCSI naming architecture MUST address support of SCSI 3rd party operations such as EXTENDED COPY.

From section 8.2 Discovery 🔭

  • MUST have no impact on the use of current IP network discovery techniques.
  • MUST provide some means of determining whether an iSCSI service is available through an IP address.
  • SCSI protocol-dependent techniques SHOULD be used for further discovery beyond the iSCSI layer.
  • MUST provide a method of discovering, given an IP end point on its well-known port, the list of SCSI targets available to the requestor. The use of this discovery service MUST be optional.

From section 9 Internet Accessibility 🚪

  • SHOULD be scrutinized for denial of service issues and they should be addressed.

From section 9.2 Firewalls and Proxy Servers 🧱

  • SHOULD allow deployment where functional and optimizing middleboxes such as firewalls, proxy servers and NATs are present.
  • use of IP addresses and TCP ports SHOULD be firewall friendly.

From section 9.3 Congestion Control and Transport Selection 🚦

  • MUST be a good network citizen with TCP-compatible congestion control (as defined in [RFC2914]).
  • iSCSI implementations MUST NOT use multiple connections as a means to avoid transport-layer congestion control.

3. iSCSI Design Considerations 🎀

3.1. General Discussion 🌸

okay so storage controllers (think disk array controllers and tape library controllers) have traditionally been speaking SCSI-3 and getting plugged into computers over a SCSI parallel bus or Fibre Channel. classic. 🙄

but the IP network is giving main character energy for block-oriented storage attachment 💖 it gets to tap into the performance and cost perks that come from competition in the Internet marketplace, and that could make storage networking cheaper because you only have to install and run one type of network instead of two. two networks is so last season 💅

the IP protocol suite also comes with a whole buffet of management, security, and QoS options 🎀 orgs might start out running iSCSI storage networks that are totally separate from their regular data networks, with the one exception of securely routing storage management traffic. they get the high performance and cost of IP gear AND a unified management setup, which is iconic ✨ as security and QoS keep evolving, it'll make sense to build combined networks that share infrastructure. still, I'm pretty sure the sophisticated users will keep their storage sub-networks isolated so they get the best control over security and QoS, keeping things tuned for storage traffic. high-performance environment or bust 🔥

mapping SCSI over IP also gives you:

  • Extended distance ranges
  • Connectivity to "carrier class" services that support IP

these are the iSCSI use cases being planned 🧚

  • Local storage access, consolidation, clustering and pooling (as in the data center)
  • Network client access to remote storage (eg. a "storage service provider")
  • Local and remote synchronous and asynchronous mirroring between storage controllers
  • Local and remote backup and recovery

and iSCSI is going to work over these topologies 🦋

  • Point-to-point direct connections
  • Dedicated storage LAN, consisting of one or more LAN segments
  • Shared LAN, carrying a mix of traditional LAN traffic plus storage traffic
  • LAN-to-WAN extension using IP routers or carrier-provided "IP Datatone"
  • Private networks and the public Internet

IP LAN-WAN routers can stretch the IP storage network out to the wide area 🌈 that gets you remote disk access (storage utility vibes), synchronous and asynchronous remote mirroring, and remote backup and restore (aka tape vaulting). and in the WAN, using TCP end-to-end is so smart because you don't need specialized protocol conversion gear, your data reliability is protected, it handles network congestion, and it has retransmission strategies that adapt to WAN delays. no notes 🫶

the iSCSI technology rollout is gonna involve these pieces:

  1. Conclusion of a complete protocol standard and supporting implementations;
  2. Development of Ethernet storage NICs and related driver and protocol software; [NOTE: high-speed applications of iSCSI are expected to require significant portions of the iSCSI/TCP/IP implementation in hardware to achieve the necessary throughput.]
  3. Development of compatible storage controllers; and
  4. The likely development of translating gateways to provide connectivity between the Ethernet storage network and the Fibre Channel and/or parallel-bus SCSI domains.
  5. Development of specifications for iSCSI device management such as MIBs, LDAP or XML schemas, etc.
  6. Development of management and directory service applications to support a robust SAN infrastructure.

products could launch first for Gigabit Ethernet attachment and then quickly move up to 10 GbE 🚀 to be performance-competitive with other SCSI transports, the performance path of the whole protocol stack has to run in hardware. these new storage NICs might do full-stack processing of an entire SCSI task (just like today's SCSI and Fibre Channel HBAs), and they might also support every host protocol that runs over TCP (NFS, CIFS, HTTP, etc). the hardware era is here bestie 💜

the IETF IP Storage Working Group (IPSWG) charter is all about mapping SCSI to IP using a transport with proven congestion avoidance behavior and broad implementation across a bunch of platforms 🍓 several transport options could be considered within that charter, but the initial IPS work focuses on TCP, and this requirements doc is restricted to that domain of interest. TCP only, babe 🎀

3.2. Performance/Cost 💅

Overall goal 🌟 in general, iSCSI MUST allow implementations to equal or improve on the current state of the art for SCSI interconnects. ngl that's a big ask. the goal breaks down into several types of requirement, so let's go through them 👑

Cost competitive with alternative storage network technologies 💸 in order to get adopted by vendors and the user community, the iSCSI protocol MUST enable cost competitive implementations when compared to other SCSI transports (Fibre Channel). if it's not cheap, nobody's switching 😭

Low delay communication ⏱️ conventional storage access is a stop-and-wait remote procedure call type of thing. apps usually don't pipeline their storage accesses much, so storage access delay directly hits performance. the delay from current storage interconnects, including protocol processing, is generally around 100 microseconds. caching in storage controllers means lots of accesses finish almost instantly, so the interconnect's delay can have a huge relative impact on overall performance 🙈 when stop-and-wait IO is used, the interconnect's delay affects performance. the iSCSI protocol SHOULD minimize control overhead, since that overhead adds to delay ⚡

Low host CPU utilization, equal to or better than current technology 🧠 for competitive performance, the iSCSI protocol MUST allow three key implementation goals to be realized:

  1. iSCSI MUST make it possible to build I/O adapters that handle an entire SCSI task, as alternative SCSI transport implementations do.
  2. The protocol SHOULD permit direct data placement ("zero-copy" memory architectures, where the I/O adapter reads or writes host memory exactly once per disk transaction.
  3. The protocol SHOULD NOT impose complex operations on the host software, which would increase host instruction path length relative to alternatives.

Direct data placement (zero-copy iSCSI) 🦋 direct data placement means iSCSI data gets placed directly "off the wire" into the allocated spot in memory with no intermediate copies. that seriously cuts down memory bus and I/O bus loading in the endpoint systems, which means better performance. it also reduces the memory NICs need, which could make these solutions cheaper 🤑

this is a big deal as an implementation goal. in an iSCSI system, each end node (like the host computer and the storage controller) should have ample memory, but the intervening nodes (NICs, switches) typically won't 💀

High bandwidth, bandwidth aggregation 🚄 the bandwidth (transfer rate, MB/sec) that storage controllers support is climbing fast, thanks to a few things:

  1. Increase in disk spindle and controller performance;
  2. Use of ever-larger caches, and improved caching algorithms;
  3. Increased scale of storage controllers (number of supported spindles, speed of interconnects).

the iSCSI protocol MUST provide for full utilization of available link bandwidth. the protocol MUST also allow an implementation to exploit parallelism (multiple connections) at the device interfaces and within the interconnect fabric. multiple connections, full bandwidth, no waste 🌸

the next two sections dig deeper into direct data placement and high bandwidth. stay tuned, bestie 🫧

3.3. Framing ✨

Okay bestie, framing is when you tuck extra info into a header (or the data stream) so the receiver can find where each iSCSI protocol data unit (PDU) starts and ends inside the TCP byte stream 📦. Two technical requirements are driving this: "interfacing needs" and "accelerated processing needs" 💅.

The "interfacing needs" part is about letting a message-based upper layer protocol (iSCSI) run on top of a byte streaming protocol (TCP) 🎀. TCP is reliable, so you can do this by putting a length field in the iSCSI header 🫶. The catch is that the receiver has to parse from the very beginning of the TCP data stream and never mess up (lose alignment on packet headers) 😭.

The other requirement, "accelerated processing," comes from data rates getting higher and higher in the physical media interface 🚀. Higher data rates cause two needs:

  1. LAN environment - NIC vendors want "zero-copy" ways to move data straight from the wire into application buffers 🔌
  2. WAN environment - high bandwidth, high latency, low bit error rate physical media puts huge buffer demands on the physical interface solutions 🌍

First, vendors are building network processing hardware that offloads network protocols so data can move faster 🏎️. The idea behind "zero-copy" is to store blocks of data in the right memory locations (aligned) directly off the wire, even when the data gets reordered because of packet loss. This is needed to drive actual data rates of 10 Gigabit/sec and beyond 🔥.

Secondly, for iSCSI to succeed in the WAN arena, it has to work efficiently over high bandwidth, high delay networks 🌏. Multi-gigabit IP networks with latencies in the tens to hundreds of milliseconds are a real challenge 😤. To fill those huge pipes, you need tens of megabytes of outstanding requests from the application. Some protocols also need tens of megabytes at the transport layer just to buffer and reassemble data that arrives out-of-order 📚.

In both cases, the goal is the same: minimize the amount of memory and memory bandwidth needed for iSCSI hardware solutions 💜.

Let's do the math, because it's wild 🤯. Consider that a network pipe at 10 Gbps x 200 msec holds 250 MB. The example assumes land-based communication with a spot halfway around the world at the equator 🌎. It ignores extra distance from cable routing, and ignores repeater and switching delays, so it only counts a speed-of-light delay of 5 microsec/km. The circumference of the globe at the equator is approx. 40000 km, and round-trip delay has to be considered to keep the pipe full. The result is 10 Gb/sec x 40000 km x 5 microsec/km x B / 8b = 250 MB.

Now the problem. In a conventional TCP implementation, if a TCP segment gets lost, stream processing MUST stop until that segment is recovered, which takes at least a time of <network round trip> 😩. Following the example above, an implementation would have to catch 250 MB of data into an anonymous buffer before it can resume stream processing. Later, that data would need to be moved to its proper location 🙃. That's why some iSCSI proponents want a way to put data directly where it belongs and avoid extra data movement when a segment is dropped. This is a key concept for understanding the debate behind framing methodologies 🧠.

The framing of iSCSI affects both the "interfacing needs" and the "accelerated processing needs." Including a length in a header may be enough for the "interfacing needs," but it won't serve the direct data placement needs 🙅‍♀️. So the framing mechanism developed should allow resynchronization of packet boundaries, even when a packet is temporarily missing from the incoming data stream 🦋.

3.4. High bandwidth, bandwidth aggregation 🌊

At today's block storage transport throughput, any single link can get saturated by storage traffic 📶. Scientific data applications and data replication are examples of storage applications that push throughput to its limits 🚄.

Some applications, like log updates, streaming tape, and replication, need updates to be ordered, which means the SCSI commands need to be ordered too 📋. An initiator can keep the order by waiting for each update to finish before sending the next one (a.k.a. synchronous updates). The catch? The throughput of synchronous updates drops as network distances get longer 🐌.

For more throughput, the SCSI task queuing mechanism lets an initiator have multiple commands outstanding at the target at the same time, and lets it express ordering constraints on how those commands run 🎟️. The task queuing mechanism only works if the commands arrive at the target in the same order they were presented to the initiator (FIFO order). So the iSCSI standard must provide an ordered transport of SCSI commands, even when commands are sent along different network paths (see Section 5.2 SCSI). This is referred to as "command ordering" 💅.

The iSCSI protocol MUST operate over a single TCP connection so implementations stay cheaper 💸. To enable higher performance storage devices, the protocol should specify a way to operate over multiple connections while still acting like a single SCSI port. That would let the initiator and target use multiple network interfaces and multiple paths through the network for more throughput 🚀. There are a few potential ways to meet the multiple path and ordering requirements 🗺️.

A popular way to meet the multiple-path requirement is to have a driver above the SCSI layer create multiple copies of the SCSI transport, each one talking to the target over a different path. "Wedge" drivers use this technique today to get high performance 🏆. The downside is that wedge drivers have to wait for an acknowledgement of each request's completion (stop-and-wait) to keep updates in order 😬.

Another approach is for the iSCSI protocol to use multiple instances of its underlying transport (e.g. TCP). The iSCSI layer would make these independent transport instances look like one SCSI transport instance, and still keep the ability to do ordered SCSI command queuing 🧩. For convenience, this document calls this technique "connection binding" 🎀.

The iSCSI protocol SHOULD support connection binding, and it MUST be optional to implement 🫶.

With connection binding, there are two ways to assign features to connections. In the symmetric approach, every connection is identical feature-wise. In the asymmetric model, connections have different features. For example, some connections might mainly handle data transfers, while others mainly handle SCSI commands 🔀.

Since the iSCSI protocol must support the case where there's only one transport connection, the protocol must have command, data, and status travel over the same connection 🔗.

With multiple connections, the iSCSI protocol must keep the command and its associated data and status on the same connection. This is called connection allegiance 🌈. Sending data and status on the same connection is desirable because it guarantees that status is received after the data (TCP provides ordered delivery) ✅. When each connection is managed by a separate processor, allegiance cuts down on the need for inter-processor communication 🖥️. This symmetric approach is a natural extension of the single connection approach 💖.

An alternate approach that got a lot of discussion sends all commands on a single connection, and the associated data and status on a different connection (asymmetric approach). In this scheme, the transport makes sure the commands arrive in order 📬. The protocol on the data and status connections is simpler, which might make it easier to build in hardware 🔧. One disadvantage is that the recovery procedure is different if a command connection fails versus a data connection 😵‍💫. Some folks argued this approach would need more inter-processor communication when connections are spread across processors 🤔.

If you want the full drama, the mail archives of the IPS mailing list from June through September of 2000 have extensive discussions on symmetric vs asymmetric connection models 📨🍿.

4. Ease of implementation/complexity of protocol 🧩

Experience has shown that adoption of a protocol by the Internet community is inversely proportional to its complexity 📉. Plus, the simpler the protocol, the easier it is to diagnose problems 🔍. The designers of iSCSI SHOULD strive to fulfill the requirements of creating a SCSI transport over IP, while keeping the protocol as simple as possible 💅.

In the interest of simplicity, iSCSI SHOULD minimize optional features 🧼. When features are deemed necessary, the protocol MUST specify feature negotiation at session establishment (login) 🤝. The iSCSI transport MUST operate correctly when no optional features are negotiated, and also when individual option negotiations are unsuccessful 💪.

5. Reliability and Availability 🛡️

5.1. Detection of Data Corruption 🔍

There have been several research papers suggesting that the TCP checksum calculation lets a certain number of bit errors slip through undetected [10] [11] 😱.

To protect against data corruption, the iSCSI protocol MUST support a data integrity check format for use in digest generation 🔐.

The iSCSI protocol MAY use separate digests for data and headers 🎀. In an iSCSI proxy or gateway situation, the iSCSI headers get removed and rebuilt, and the TCP stream is terminated on either side. This means even the TCP checksum gets removed and recomputed inside the gateway 🙈. To protect commands, data, and status, the iSCSI protocol MUST include a CRC or other digest mechanism that is computed on the SCSI data block itself, as well as on each command and status message 💖. Since gateways may strip iSCSI headers and rebuild them, a separate header CRC is required 🫧. Two header digests would protect the header: one for the invariant portions of the header (addresses) and one for the variant portion. That protects against changes to parts of the header that should never be changed by middle boxes (eg, addresses) 🛡️.

The iSCSI header format SHOULD be extensible to include other digest calculation methods 🌸

5.2. Recovery 🩹

okay so SCSI was built for a parallel bus that was super reliable, and SCSI apps literally assume transport errors never happen 🙈 when they do, recovering is expensive in time and compute. not a vibe 💀

iSCSI protocol design puts a lot of focus on simplicity, but it MUST still lead to timely recovery from failure of the initiator, the target, or the connecting network infrastructure (cabling, data path equipment like routers, etc) ✨

iSCSI MUST specify recovery methods for non-idempotent requests, like operations on tape drives 📼

the iSCSI protocol error recovery mechanism SHOULD take fail-over schemes into account for mirrored targets or highly available storage setups that give paths to target data through multiple "storage servers" 🫶 this gives a basis for layered stuff like high availability and clustering, which is so iconic

the iSCSI protocol SHOULD also provide a way for sessions to be gracefully terminated and restarted, and either the initiator or the target can start it 🌸 that means you can gracefully fail over an initiator or target, or reset a target after maintenance like upgrading software. main character energy 👑

6. Interoperability 🤝

initiators and targets that implement the required parts of the iSCSI spec must be able to interoperate 💅 yes it's obvious and it doesn't seem worth saying, BUT if the protocol wording is ambiguous, different implementations may not play nice together 😭

the iSCSI protocol document MUST be clear and unambiguous 🎀

6.1. Internet infrastructure 🌐

The iSCSI protocol MUST:

  • be compatible with both IPv4 and IPv6 🦋
  • use TCP connections conservatively, keeping in mind there may be many other users of TCP on a given machine 🧚

The iSCSI protocol MUST NOT require changes to existing Internet protocols, and SHOULD minimize required changes to existing TCP/IP implementations 🫧

iSCSI MUST be designed so SCTP (in place of TCP) can be swapped in as the IP transport protocol later, with minimal changes to iSCSI protocol operation, protocol data unit (PDU) structures and formats 🔮 not widely implemented yet, but SCTP has a bunch of design features that make it a lowkey iconic pick for future iSCSI upgrades ✨

6.2. SCSI 💾

to count as a SCSI transport, the iSCSI standard has to comply with the requirements of the SCSI Architecture Model [SAM-2] for a SCSI transport 📜 any feature SAM2 requires in a valid transport mapping MUST be specified by iSCSI. the iSCSI protocol document MUST specify, for each feature, whether it is OPTIONAL, RECOMMENDED or REQUIRED to implement and/or use 📝

the SCSI Architectural Model [SAM-2] expects the SCSI transport to provide ordering of commands at the initiator target-LUN granularity 🧩 there was SO much discussion on the IPS reflector and in working group meetings about how to make sure this ordering happens 😤 the rough consensus is that iSCSI MUST specify strictly ordered delivery of SCSI commands over an iSCSI session between an initiator/target pair, even when there are transport errors 🔥 this command ordering mechanism SHOULD seek to minimize the amount of communication needed across multiple adapters doing transport off-load. and if an iSCSI implementation doesn't need ordering, it can set up multiple sessions per initiator-target pair 🌈

iSCSI is meant to be a new SCSI "transport" [SAM2] 💖 as a mapping of SCSI over TCP, iSCSI has to deal with both T10 and IETF 🫶 but the iSCSI protocol MUST NOT require changes to the SCSI-3 command sets and SCSI client code, except where SCSI specs point to "transport dependent" fields and behavior 🙌 for example, SCSI docs will need changes to reflect longer iSCSI target names and potentially longer timeouts. collaborating with T10 is needed to hit this one 🤝

the iSCSI protocol SHOULD track changes to SCSI and the SCSI Architecture Model 👀

the iSCSI protocol MUST be capable of supporting all SCSI-3 command sets and device types 💅 the main focus is on the 'larger' devices: host computers and storage controllers (disk arrays, tape libraries) 🏢 however, other command sets (printers, scanners) must be supported too 🖨️ none of this MUST NOT be construed to mean that iSCSI has to be natively implementable on all of today's SCSI devices, which might have limited processing power or memory 🙈 lowkey important context fr

ACA (Auto Contingent Allegiance) is an optional SCSI mechanism that stops execution of a sequence of dependent SCSI commands when one of them fails 🛑 the situation around it is complex 😵‍💫 T10 specifies ACA in SAM2, so iSCSI must support it, and it should work hard to make sure ACA actually gets implemented enough (two independent interoperable implementations) to avoid dropping ACA in the move from Proposed Standard to Draft Standard 📈 this implies iSCSI SHOULD support ACA implementation 🧃

The iSCSI protocol MUST allow for building gateways to other SCSI transports, including parallel SCSI [SPI-X] and SCSI FCP[FCP, FCP-2] 🌉 it MUST be possible to build "translating" gateways so iSCSI hosts can interoperate with SCSI-X devices, so SCSI-X devices can talk over an iSCSI network, and so SCSI-X hosts can use iSCSI targets (where SCSI-X means parallel SCSI, SCSI-FCP, or SCSI over any other transport) 🔁 this is implied by support for SAM-2, but it's worth emphasizing 📣 these are true application protocol gateways, not just bridge/routers, since the different standards only share the SCSI-3 command set layer 💜 these gateways are not mere packet forwarders 🚫📦

The iSCSI protocol MUST reliably transport SCSI commands from the initiator to the target 🎯 according to [SAM-2, p. 17.] "The function of the service delivery subsystem is to transport an error-free copy of the request or response between the sender and the receiver" [SAM-2, p. 22]. the iSCSI protocol MUST correctly deal with iSCSI packet drop, duplication, corruption, stale packets, and re-ordering 🌪️ every single one of those is a plot twist and iSCSI has to handle all of them 😭🔥

7. Security Considerations 🔐

in the past, directly attached storage systems had minimal security checks because the physical connection left little chance for attack 🔒 transporting block storage (SCSI) over IP opens a whole new door for all kinds of malicious attacks 😈 attacks can be active (identity spoofing, man-in-the-middle) or passive (eavesdropping). this is serious, bestie, not a drill 🚨

7.1. Extensible Security 🧩

the security services needed for communications depend on the individual network configurations and environments 🌍 organizations are setting up Virtual Private Networks (VPN), also known as Intranets, that need one set of security functions for communication inside the VPN and possibly many different security functions for communication outside it, to support geographically separate components 🗺️ the iSCSI protocol applies to a wide range of internetworking environments that may use different security policies 💫 iSCSI MUST provide for strong authentication when increased security is required 🛡️ the protocol SHOULD require minimal configuration and overhead in insecure operation, and allow new security mechanisms to be integrated without breaking backwards compatible operation 🎀

7.2. Authentication 🪪

The iSCSI protocol MAY support various levels of authentication security, from no authentication all the way up to secure authentication using public or private keys 🗝️

the iSCSI protocol MUST support private authenticated login 🔑

authenticated login helps the target block unauthorized use of SCSI resources 🚧 "Private" authenticated login requires protected identity exchange (no clear text passwords, at a minimum) 🙅‍♀️ block storage confidentiality is considered critical in enterprises, and many IP networks may have access holes, so organizations will want to protect their iSCSI resources 💖

The iSCSI authenticated login MUST be resilient against attacks, since many IP networks are vulnerable to packet inspection 🕵️‍♀️

on top of that, the iSCSI protocol MUST support data origin authentication of its communications, but data origin authentication MAY be optional to use 📬 this is critical because IP networks are vulnerable to source spoofing, where a malicious third party pretends to send packets from the initiator's IP address 👻 these requirements should be met using standard Internet protocols like IPsec or TLS 🌷 the endpoints may negotiate the authentication method, optionally none ✅

7.3. Data Integrity 🧷

The iSCSI protocol SHOULD NOT preclude use of additional data integrity protection protocols (IPSec, TLS) 🙌

7.4. Data Confidentiality 🤫

block storage holds sensitive info, so data confidentiality is critical 💎 an application may encrypt the data blocks before writing them to storage, which gives the app the best protection 🔐 even if the storage or communications get compromised, the attacker will have a hard time reading the data 😤

in certain environments, encryption may be wanted for extra assurance of confidentiality ✨ an iSCSI implementation MUST provide for the use of a data encryption protocol such as TLS or IPsec ESP to give data confidentiality between iSCSI endpoints 🫶

8. Management 🗂️

iSCSI implementations SHOULD be manageable using standard IP-based management protocols 💻 however, the iSCSI protocol document MUST NOT define the management architecture for iSCSI within the network infrastructure 🙅‍♀️ iSCSI will be yet another resource service in a complex environment of network resources (printers, file servers, NAS, application servers, etc) 🖥️ there will definitely be efforts to design how the "block storage service" iSCSI devices provide fits into a comprehensive, shared model and network management environment 🌐 a "network administrator" (or "storage administrator") will want integrated apps for assigning user names, resource names, etc., and setting access rights 📋 iSCSI devices will presumably want to interact with these integrated network management apps 🤖 the iSCSI protocol document is not going to try to solve that set of problems, or specify how devices provide management agents 🙅‍♀️ and honestly? there should be no mention of MIBs or any other way of managing iSCSI devices as explicit references in the iSCSI protocol document, because management data and protocols change with the needs of the environment and the business models of the management apps 💅🌸

8.1. Naming 🏷️

okay so naming is a BIG deal bestie 💅 whenever possible, iSCSI MUST support the naming architecture of SAM-2. any deviations and uncertainties MUST be made explicit, and comments and resolutions need to be worked out between ANSI T10 and the IPS working group 🫶

the means by which an iSCSI resource is located MUST use or extend existing Internet standard resource location methods. RFC 2348 [12] specifies URL syntax and semantics which should be sufficiently extensible for the iSCSI resource ✨

the iSCSI protocol MUST provide a means of identifying an iSCSI storage device by a unique identifier that is independent of the path on which it is found. this name is what gets used to correlate alternate paths to the same device, so it's giving main character energy 👑 the format for iSCSI names MUST use existing naming authorities, so nobody has to create new central administrative tasks 🙈 an iSCSI name SHOULD be a human readable string in an international character set encoding 🌈

standard Internet lookup services SHOULD be used to resolve names. for example, Domain Name Services (DNS) MAY be used to resolve the <hostname> portion of a URL to one or multiple IP addresses. when a hostname resolves to multiple addresses, these addresses should be equivalent for functional (possibly not performance) purposes. this means the addresses can be used interchangeably as long as performance isn't a concern. for example, the same set of SCSI targets MUST be accessible from each of these addresses 🦋

an iSCSI device naming scheme MUST interact correctly with the proposed SCSI security architecture [99-245r9]. particular attention must be directed to the proxy naming architecture defined by the new security model. in this new model, a host is identified by an Access ID, and SCSI Logical Unit Numbers (LUNs) can be mapped in a manner that gives each AccessID a unique LU map. so a given LU within a target may be addressed by different LUNs 🧚

the iSCSI naming architecture MUST address support of SCSI 3rd party operations such as EXTENDED COPY. the key issue here is the naming architecture for SCSI LUs: iSCSI must provide a means of passing a name or handle between parties. iSCSI must specify a means of providing a name or handle that could be used in the XCOPY command and fit within the available space allocated by that command. and it must be possible, of course, for the XCOPY target (the third party) to de-reference the name to the correct target and LU 💖

8.2. Discovery 🔎

iSCSI MUST have no impact on the use of current IP network discovery techniques. network management platforms discover IP addresses and have various methods of probing the services available through these IP addresses. an iSCSI service should be evident using similar techniques ✨

the iSCSI specifications MUST provide some means of determining whether an iSCSI service is available through an IP address. it is expected that iSCSI will be a point of service in a host, just as SNMP, etc are points of services, associated with a well known port number 🌸

SCSI protocol-dependent techniques SHOULD be used for further discovery beyond the iSCSI layer. discovery is a complex, multi-layered process. the SCSI protocol specifications provide specific commands for discovering LUs, and the commands associated with this process will also work over iSCSI 🫶

the iSCSI protocol MUST provide a method of discovering, given an IP end point on its well-known port, the list of SCSI targets available to the requestor. the use of this discovery service MUST be optional.

further discovery guidelines are outside the scope of this document and may be addressed in separate Informational documents 📚

9. Internet Accessibility 🌐

9.1. Denial of Service 💀

as with all services, denial of service caused by either incorrect implementations or malicious agents is always a concern. all aspects of the iSCSI protocol SHOULD be scrutinized for potential denial of service issues, and guarded against as much as possible 🛡️

9.2. NATs, Firewalls and Proxy servers 🧱

NATs (Network Address Translators), firewalls, and proxy servers are a reality in today's Internet. these devices present a number of challenges to device access methods being developed for iSCSI. for example, specifying a URL syntax for iSCSI resource connection allows an initiator to address an iSCSI target device both directly and through an iSCSI proxy server or NAT. iSCSI SHOULD allow deployment where functional and optimizing middle-boxes such as firewalls, proxy servers and NATs are present 🌷

the iSCSI protocol's use of IP addressing and TCP port numbers MUST be firewall friendly. this means that all connection requests should normally be addressed to a specific, well-known TCP port. that way, firewalls can filter based on source and destination IP addresses, and destination (target) port number. additional TCP connections would require different source port numbers (for uniqueness), but could be opened after a security dialogue on the control channel 🔐

it's important that iSCSI operate through a firewall, because that provides a possible means of defending against Denial of Service (DoS) assaults from less-trusted areas of the network. it is assumed that a firewall will have much greater processing power for dismissing bogus connection requests than end nodes. 🚨

9.3. Congestion Control and Transport Selection 🚦

the iSCSI protocol MUST be a good network citizen with proven congestion control (as defined in [RFC2914]). in addition, iSCSI implementations MUST NOT use multiple connections as a means to avoid transport-layer congestion control 🙅‍♀️

10. Definitions 📖

certain definitions are offered here, with references to the original document where applicable, in order to clarify the discussion of requirements. definitions without references are the work of the authors and reviewers of this document 🤓

Logical Unit (LU): A target-resident entity that implements a device model and executes SCSI commands sent by an application client [SAM-2, sec. 3.1.50, p. 7].

Logical Unit Number (LUN): A 64-bit identifier for a logical unit [SAM-2, sec. 3.1.52, p. 7].

SCSI Device: A device that is connected to a service delivery subsystem and supports a SCSI application protocol [SAM-2, sec. 3.1.78, p. 9].

Service Delivery Port (SDP): A device-resident interface used by the application client, device server, or task manager to enter and retrieve requests and responses from the service delivery subsystem. Synonymous with port (SAM-2 sec. 3.1.61) [SAM-2, sec. 3.1.89, p. 9].

Target: A SCSI device that receives a SCSI command and directs it to one or more logical units for execution [SAM-2 sec. 3.1.97, p. 10].

Task: An object within the logical unit representing the work associated with a command or a group of linked commands [SAM-2, sec. 3.1.98, p. 10].

Transaction: A cooperative interaction between two objects, involving the exchange of information or the execution of some service by one object on behalf of the other [SAM-2, sec. 3.1.109, p. 10].

11. References 📚

  1. Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996.
  2. Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  3. [SAM-2] ANSI NCITS. Weber, Ralph O., editor. SCSI Architecture Model -2 (SAM-2). T10 Project 1157-D. rev 23, 16 Mar 2002.
  4. [SPC-2] ANSI NCITS. Weber, Ralph O., editor. SCSI Primary Commands 2 (SPC-2). T10 Project 1236-D. rev 20, 18 July 2001.
  5. [CAM-3] ANSI NCITS. Dallas, William D., editor. Information Technology - Common Access Method - 3 (CAM-3)). X3T10 Project 990D. rev 3, 16 Mar 1998.
  6. [99-245r8] Hafner, Jim. A Detailed Proposal for Access Controls. T10/99-245 revision 9, 26 Apr 2000.
  7. [SPI-X] ANSI NCITS. SCSI Parallel Interface - X.
  8. [FCP] ANSI NCITS. SCSI-3 Fibre Channel Protocol [ANSI X3.269:1996].
  9. [FCP-2] ANSI NCITS. SCSI-3 Fibre Channel Protocol - 2 [T10/1144-D].
  10. Paxon, V. End-to-end internet packet dynamics, IEEE Transactions on Networking 7,3 (June 1999) pg 277-292.
  11. Stone J., Partridge, C. When the CRC and TCP checksum disagree, ACM Sigcomm (Sept. 2000).
  12. Malkin, G. and A. Harkin, "TFTP Blocksize Option", RFC 2348, May 1998.
  13. Floyd, S., "Congestion Control Principles", BCP 14, RFC 2914, September 2000.

12. Acknowledgements 🫶

special thanks to Julian Satran, IBM and David Black, EMC for their extensive review comments 💜

13. Author's Addresses 📬

address comments to:

Marjorie Krueger
Hewlett-Packard Corporation
8000 Foothills Blvd
Roseville, CA 95747-5668, USA
Phone: +1 916 785-2656
EMail: marjorie_krueger@hp.com
Randy Haagens
Hewlett-Packard Corporation
8000 Foothills Blvd
Roseville, CA 95747-5668, USA
Phone: +1 916 785-4578
EMail: Randy_Haagens@hp.com
Costa Sapuntzakis
Stanford University
353 Serra Mall Dr #407
Stanford, CA 94305
Phone: 650-723-2458
EMail: csapuntz@stanford.edu
Mark Bakke
Cisco Systems, Inc.
6450 Wedgwood Road
Maple Grove, MN 55311
Phone: +1 763 398-1054
EMail: mbakke@cisco.com

Copyright (C) The Internet Society (2002). All Rights Reserved.

this document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. however, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards, in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

the limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

this document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 💅

Acknowledgement 💖

funding for the RFC Editor function is currently provided by the Internet Society 🌈

📦 where this page comes from