RFC 5042
Direct Data Placement Protocol (DDP) / Remote Direct Memory Access Protocol (RDMAP) Security
Direct Data Placement Protocol (DDP) / Remote Direct Memory Access Protocol (RDMAP) Security ✨
Okay bestie, this RFC is all about keeping RDMA safe 🔒. It was published in October 2007 by J. Pinkerton of Microsoft Corporation and E. Deleganes (writing as Self). It's a Standards Track document under the Network Working Group, which means it's an actual Internet standard and not just an idea someone had 💅. The title block doesn't list anything it obsoletes or updates, so it's not replacing or patching any older RFC. Iconic, clean start 🦋.
Status of This Memo
Okay, so this document is an Internet standards track protocol spec for the Internet community 🌸. It's also asking people to talk about it and send in suggestions for making it better, so your input matters 🫶. If you want to know where this protocol stands officially, check the current edition of the "Internet Official Protocol Standards" (STD 1), which tracks the standardization state and status of protocols 📋. And the best part? Distribution of this memo is unlimited 🎀, so share it with everyone who needs it.
Abstract
Here's the tea ☕. This document looks at the security issues that come up when you build or use the Direct Data Placement Protocol (DDP) and the Remote Direct Memory Access Protocol (RDMAP) 🔥. It starts by defining an architectural model for an RDMA Network Interface Card (RNIC), which can implement DDP or RDMAP and DDP 🧚. Then it goes through the attacks that can target the resources in that model, plus the countermeasures you can use to protect the system 🛡️. The attacks are sorted into three groups: ones you can stop with secure communication channels across the network, attacks from Remote Peers, and attacks from Local Peers 👑. The attack categories are spoofing, tampering, information disclosure, denial of service, and elevation of privilege. It's giving full security audit energy, no cap 💀.
Table of Contents
1. Introduction ....................................................4
2. Architectural Model .............................................6
2.1. Components .................................................7
2.2. Resources ..................................................9
2.2.1. Stream Context Memory ...............................9
2.2.2. Data Buffers .......................................10
2.2.3. Page Translation Tables ............................10
2.2.4. Protection Domain (PD) .............................11
2.2.5. STag Namespace and Scope ...........................11
2.2.6. Completion Queues ..................................12
2.2.7. Asynchronous Event Queue ...........................12
2.2.8. RDMA Read Request Queue ............................13
2.3. RNIC Interactions .........................................13
2.3.1. Privileged Control Interface Semantics .............13
2.3.2. Non-Privileged Data Interface Semantics ............13
2.3.3. Privileged Data Interface Semantics ................14
2.3.4. Initialization of RNIC Data Structures for
Data Transfer ......................................14
2.3.5. RNIC Data Transfer Interactions ....................16
3. Trust and Resource Sharing .....................................17
4. Attacker Capabilities ..........................................18
5. Attacks That Can Be Mitigated with End-to-End Security .........18
5.1. Spoofing ..................................................19
5.1.1. Impersonation ......................................19
5.1.2. Stream Hijacking ...................................20
5.1.3. Man-in-the-Middle Attack ...........................20
5.2. Tampering - Network-Based Modification of Buffer Content ..21
5.3. Information Disclosure - Network-Based Eavesdropping ......21
5.4. Specific Requirements for Security Services ...............21
5.4.1. Introduction to Security Options ...................21
5.4.2. TLS Is Inappropriate for DDP/RDMAP Security ........22
5.4.3. DTLS and RDDP ......................................23
5.4.4. ULPs That Provide Security .........................23
5.4.5. Requirements for IPsec Encapsulation of DDP ........23
6. Attacks from Remote Peers ......................................24
6.1. Spoofing ..................................................25
6.1.1. Using an STag on a Different Stream ................25
6.2. Tampering .................................................26
6.2.1. Buffer Overrun - RDMA Write or Read Response .......26
6.2.2. Modifying a Buffer after Indication ................27
6.2.3. Multiple STags to Access the Same Buffer ...........27
6.3. Information Disclosure ....................................28
6.3.1. Probing Memory Outside of the Buffer Bounds ........28
6.3.2. Using RDMA Read to Access Stale Data ...............28
6.3.3. Accessing a Buffer after the Transfer ..............28
6.3.4. Accessing Unintended Data with a Valid STag ........29
6.3.5. RDMA Read into an RDMA Write Buffer ................29
6.3.6. Using Multiple STags That Alias to the Same
Buffer .............................................29
6.4. Denial of Service (DOS) ...................................30
6.4.1. RNIC Resource Consumption ..........................30
6.4.2. Resource Consumption by Idle ULPs ..................31
6.4.3. Resource Consumption by Active ULPs ................32
6.4.3.1. Multiple Streams Sharing Receive Buffers ..32
6.4.3.2. Remote or Local Peer Attacking a
Shared CQ .................................34
6.4.3.3. Attacking the RDMA Read Request Queue .....36
6.4.4. Exercise of Non-Optimal Code Paths .................37
6.4.5. Remote Invalidate an STag Shared on
Multiple Streams ...................................37
6.4.6. Remote Peer Attacking an Unshared CQ ...............38
6.5. Elevation of Privilege ....................................38
7. Attacks from Local Peers .......................................38
7.1. Local ULP Attacking a Shared CQ ...........................39
7.2. Local Peer Attacking the RDMA Read Request Queue ..........39
7.3. Local ULP Attacking the PTT and STag Mapping ..............39
8. Security considerations ........................................40
9. IANA Considerations ............................................40
10. References ....................................................40
10.1. Normative References .....................................40
10.2. Informative References ...................................41
Appendix A. ULP Issues for RDDP Client/Server Protocols ...........43
Appendix B. Summary of RNIC and ULP Implementation Requirements ...46
Appendix C. Partial Trust Taxonomy ................................47
Acknowledgments ...................................................491. Introduction ✨
okay bestie, so RDMA lets two parties talk to each other in way more flexible ways than the usual networking stuff, like the stream model or the datagram model 💅. that flexibility is so cute, but it also brings new security issues you have to understand carefully when you're designing Upper Layer Protocols (ULPs) that use RDMA, or when you're implementing RDMA-aware NICs (RNICs) 🔥. quick note: for this security analysis, an RNIC might implement RDMAP [RDMAP] and DDP [DDP], or just DDP 🧚. and a ULP can be an application or a middleware library, both count 💖.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. Additionally, the security terminology defined in [RFC4949] is used in this specification 🎀.
the document starts by building an architectural model that's relevant for the security analysis 🦋. Section 2 breaks down the components, resources, and system properties that may be attacked. the document uses Local Peer to mean the RDMA/DDP protocol implementation on the local end of a Stream (implemented with a transport protocol, such as [RFC793] or [RFC4960]). the local Upper-Layer-Protocol (ULP) is the application or middle-ware layer sitting above the Local Peer 🌸.
the document doesn't try to split a Remote Peer from a Remote ULP (an RDMA/DDP protocol implementation on the remote end of a Stream versus the application on the remote end) 🙈. why? for a few reasons. often, the source of the attack is hard to know for sure, and no matter where it comes from, the mitigations the Local Peer or local ULP needs are the same. so the document just calls it a Remote Peer instead of trying to delineate the attacker any further 💜.
the document then defines, in Section 3, what resources a local ULP may share across Streams and what resources the local ULP may share with the Remote Peer across Streams 🍓.
sharing resources between multiple Streams on purpose can imply some level of trust between those Streams. but some types of resource sharing have attacks that can't be mitigated, which means you'd have to avoid sharing that specific type of resource unless the Streams sharing it have some level of trust between them 😭.
to talk about this concept, the document defines a brand new term, "Partial Mutual Trust" ✨:
Partial Mutual Trust - a collection of RDMAP/DDP Streams, which represent the local and remote end points of the Stream that are willing to assume that the Streams from the collection will not perform malicious attacks against any of the other Streams in the collection.
ULPs have explicit control over which collection of endpoints is in a Partial Mutual Trust collection, using the tools discussed in Appendix C, Partial Trust Taxonomy 👑.
an untrusted peer relationship is the right call when a ULP wants to stay robust and uncompromised even if its peer launches a deliberate attack 🛡️. for example, a single ULP that handles multiple unrelated Streams at once (like a server) would presumably treat each of its peers as an untrusted peer 🖥️. for a collection of Streams that share Partial Mutual Trust, the assumption is that any Stream not in the collection is untrusted. for the untrusted peer, Section 4 lists a brief set of capabilities 💅.
the rest of the document is all about analyzing attacks and recommending specific mitigations for them 🔍. attacks are sorted into three groups: attacks mitigated by end-to-end security, attacks initiated by Remote Peers, and attacks initiated by Local Peers. for each attack, possible countermeasures get reviewed 🧠.
ULPs within a host come in two categories: Privileged and Non-Privileged. both types can send and receive data and request resources. the key differences between them are:
- The Privileged ULP is trusted by the local system not to maliciously attack the operating environment, but it is not trusted to optimize resource allocation globally. for example, the Privileged ULP could be a kernel ULP, so the kernel presumably has in some way vetted the ULP before letting it execute 🧚.
- A Non-Privileged ULP's capabilities are a logical sub-set of the Privileged ULP's. the local system assumes that a Non-Privileged ULP is untrusted. all Non-Privileged ULP interactions with the RNIC Engine that could affect other ULPs need to go through a trusted intermediary that can verify the Non-Privileged ULP requests 🚨.
the appendices are focused summaries of this specification 📚. Appendix A, ULP Issues for RDDP Client/Server Protocols, is for implementers of traditional client/server protocols 🎀. Appendix B, Summary of RNIC and ULP Implementation Requirements, sums up all normative requirements in this specification ✅. Appendix C, Partial Trust Taxonomy, gives an abstract model for categorizing trust boundaries 🌈.
if an RDMAP/DDP protocol implementation uses the mitigations recommended in this document, it should not exhibit additional security vulnerabilities beyond those of an implementation of the transport protocol (i.e., TCP or SCTP) and the protocols beneath it (e.g., IP) without RDMAP/DDP 🫶.
2. Architectural Model 🏗️
this section describes an RDMA architectural reference model that we use as we examine security issues 🔎. it introduces the model's components, the resources that can be attacked, the types of interactions possible between components and resources, and the system properties that must be preserved 💎.
Figure 1 shows the components that make up the architecture, plus the interfaces where potential security attacks could be launched 🎯. external attacks can get injected into the system from a ULP that sits above the RNIC Interface, or from the network 🌐.
the goal here is to describe high-level components and capabilities that affect threat analysis, not to focus on specific implementation options 💅. also, the architectural model is an abstraction, and an actual implementation may choose to split its components along different boundary lines from the ones defined here. for example, the Privileged Resource Manager may be partially or completely encapsulated in the Privileged ULP. either way, the security analysis of the potential threats and countermeasures is still expected to apply 🙌.
the model below was derived from several specific RDMA implementations. a few of note are [VERBS-RDMAC], [VERBS-RDMAC-Overview], and [INFINIBAND] 📖.
+-------------+
| Privileged |
| Resource |
Admin<-+>| Manager | ULP Control Interface
| | |<------+-------------------+
| +-------------+ | |
| ^ v v
| | +-------------+ +-----------------+
+---------------->| Privileged | | Non-Privileged |
| | ULP | | ULP |
| +-------------+ +-----------------+
| ^ ^
|Privileged |Privileged |Non-Privileged
|Control |Data |Data
|Interface |Interface |Interface
RNIC | | |
Interface v v v
=================================================================
+--------------------------------------+
| |
| RNIC Engine |
| |
+--------------------------------------+
^
|
v
Internet
Figure 1 - RDMA Security Model2.1. Components 🧩
Okay bestie, let's start with the components in Figure 1 - RDMA Security Model 🎀
- RDMA Network Interface Controller Engine (RNIC) 💅 - the component that implements the RDMA protocol and/or DDP protocol.
- Privileged Resource Manager 👑 - the component in charge of managing and allocating resources tied to the RNIC Engine. The Resource Manager does not send or receive data. Whether it's an independent component, part of the RNIC, or part of the ULP is implementation dependent, so it's up to each build.
- Privileged ULP 🔑 - see Section 1, Introduction, for the definition. The local host infrastructure can let the Privileged ULP map a Data Buffer directly from the RNIC Engine to the host through the RNIC Interface, but it does NOT let the Privileged ULP directly consume RNIC Engine resources. (Lowkey the most important boundary in this whole list 🙈)
- Non-Privileged ULP 🦋 - see Section 1, Introduction, for the definition.
Here's the big goal: DDP and RDMAP are designed so that, under constrained conditions, a Non-Privileged ULP can send and receive data directly to and from the RDMA Engine without the Privileged Resource Manager stepping in, all while keeping the host secure 🔒 So the main point of this document is to analyze that usage model and figure out what enforcement the RNIC Engine needs in order to stay secure. It's giving careful balance ✨
DDP has two ways to transfer data 💖
- Untagged Data Transfer 📥 - the incoming payload just fills the first buffer in a queue of buffers, and that queue is in the order the receiving Peer specified (usually called the Receive Queue).
- Tagged Data Transfer 🎯 - the transmitting Peer explicitly says which destination buffer it's targeting, using an STag. STag-based transfers let the receiving ULP not care what order the opposite Peer sent the data in (or which messages it came in), and not care what order the packets arrive in either. Main character energy 👑
Both transfer mechanisms also work through RDMAP, just with extra control semantics 🌸 Usually Tagged Data Transfer is used for payload, and Untagged Data Transfer is best for control messages. But each Upper Layer Protocol gets to decide the optimal use of Tagged and Untagged messages for itself 💅 For more on which application fits which transfer mechanism, check [APPLICABILITY].
For DDP, the two forms are Untagged and Tagged DDP Messages. For RDMAP, they're Send Type Messages and RDMA Messages (either RDMA Read or RDMA Write Messages). Easy peasy 🧚
The host interfaces that could get exercised are:
- Privileged Control Interface 🛠️ - a Privileged Resource Manager uses the RNIC Interface to allocate and manage RNIC Engine resources, control the state inside the RNIC Engine, and monitor various events from it. It also uses this interface to act as a proxy for some operations a Non-Privileged ULP might need (after taking appropriate countermeasures).
- ULP Control Interface 📞 - a ULP uses this interface to talk to the Privileged Resource Manager when it wants to allocate RNIC Engine resources. The Privileged Resource Manager implements countermeasures so that if a Non-Privileged ULP launches an attack, the attack can't spread to other ULPs 🛡️
- Non-Privileged Data Transfer Interface 🦋 - a Non-Privileged ULP uses this interface to start data transfer operations and check how they're doing.
- Privileged Data Transfer Interface 👑 - a superset of what the Non-Privileged Data Transfer Interface offers. The ULP is allowed to directly manipulate RNIC Engine mapping resources to map an STag to a ULP Data Buffer. Very powerful, very trusted 🔥
If Internet control messages, like ICMP, ARP, RIPv4, etc., get processed by the RNIC Engine, the threat analyses for those protocols still apply. They're just outside the scope of this document 📝
2.2. Resources 💎
This section covers the main resources in the RNIC Engine that could get hurt if it's under attack 😭 For RDMAP, all the defined resources apply. For DDP, all the resources apply except the RDMA Read Queue.
2.2.1. Stream Context Memory 🧠
Each Stream's state lives in memory, and that memory can be in a bunch of places: on the NIC, inside RAM attached to the NIC, in host memory, or any mix of those three, depending on the implementation 🌈
Stream Context Memory also holds state tied to Data Buffers. For Tagged Buffers, that includes how STag names, Data Buffers, and Page Translation Tables (see Section 2.2.3) relate to each other. It also includes the list of Untagged Data Buffers posted for receiving Untagged Messages (commonly called the Receive Queue), and a list of operations to run to send data (commonly called the Send Queue) 📋
2.2.2. Data Buffers 📦
As I mentioned before, there are two different ways to expose a local ULP's Data Buffers for data transfer. One is Untagged Data Transfer, where a buffer gets exposed for receiving RDMAP Send Type Messages (a.k.a. DDP Untagged Messages) on DDP Queue zero. The other is Tagged Data Transfer, where the buffer gets exposed for remote access through STags (a.k.a. DDP Tagged Messages). This distinction matters a LOT because the attacks and the countermeasures against them are different depending on how the buffer gets exposed to the network 🙈
For the security discussion, a single logical Data Buffer is exposed with a single STag on a given Stream for Tagged Data Transfer. Real implementations might support scatter/gather so multiple physical data buffers can be reached with one STag, but for the threat analysis we assume that one STag enables access to one logical Data Buffer ✨
Either way, it's the Privileged Resource Manager's job to make sure no STag can be created that exposes memory the consumer had no authority to expose 🔐
A Data Buffer has specific access rights. The local ULP can control whether a Data Buffer is exposed for local access only, or for local and remote access. It can also assign specific access privileges (read, write, or read and write) on a per Stream basis 💅
For DDP, when an STag is Advertised, the Remote Peer is presumably given write access to the data (otherwise, what's the point of the Advertisement? 🤷♀️). For RDMAP, when a ULP Advertises an STag, it can enable write-only, read-only, or both write and read access 🦋
Some ULPs may also want to give a single buffer different access rights on a per Stream basis. For example, some Streams might get read-only access, some might get remote read and write access, and on other Streams only the local ULP/Local Peer is allowed in 🚪
2.2.3. Page Translation Tables 🗺️
Page Translation Tables are the structures the RNIC uses to access ULP memory for data transfer operations. Even though they're called "Page" Translation Tables, they might not reference a page at all. Conceptually, they map a ULP address space representation (e.g., a virtual address) of a buffer to the physical addresses the RNIC Engine uses to move data. If a specific system doesn't use a mapping, then only a subset of the attacks examined may apply 🌸 Also, the Page Translation Table may or may not be a shared resource 🤔
2.2.4. Protection Domain (PD) 🛡️
A Protection Domain (PD) is a local construct inside the RDMA implementation, and it's NEVER visible over the wire 🙅♀️ Protection Domains get assigned to three of the resources we care about: Stream Context Memory, STags associated with Page Translation Table entries, and Data Buffers. A correct implementation of a Protection Domain requires that resources belonging to one Protection Domain cannot be used on a resource belonging to another Protection Domain. The RNIC checks Protection Domain membership before taking any action involving such a resource, so this is a hard boundary 🧱 That means an STag can only be used to access its associated Data Buffer on one or more Streams in the same Protection Domain as the STag 💖
If an implementation chooses not to share resources between Streams, it is recommended that each Stream get its own unique Protection Domain. If an implementation does allow resource sharing, it is recommended that the Protection Domain be limited to the collection of Streams that have Partial Mutual Trust with each other 🤝
Also, a ULP (either Privileged or Non-Privileged) can potentially have multiple Protection Domains. This could be used, for example, so that multiple clients of a server can't corrupt each other. The server would allocate a Protection Domain per client, so resources in that Protection Domain can't be used by another (untrusted) client 🫶
2.2.5. STag Namespace and Scope 🏷️
The DDP specification defines a 32-bit namespace for the STag. Implementations can vary in how many STags they actually support. Either way, it's a bounded resource that can come under attack 😱 Depending on the STag namespace allocation algorithms, the actual namespace an attacker could go after may be a lot smaller than 2^32.
The scope of an STag is the set of DDP/RDMAP Streams on which the STag is valid. If an STag is valid on a particular DDP/RDMAP Stream, that Stream can modify the buffer, subject to the access rights it has for the STag (see Section 2.2.2, Data Buffers, for more info) 🔍
The analysis in this document assumes two ways to limit the scope of which Streams the STag is valid for:
- Protection Domain scope 🏰 - the STag is valid if it's used on any Stream within a specific Protection Domain, and invalid if it's used on any Stream that isn't a member of that Protection Domain.
- Single Stream scope 🎯 - the STag is valid on a single Stream, no matter what that Stream's Protection Domain association is. If it's used on any other Stream, it's invalid.
2.2.6. Completion Queues 📬
Completion Queues (CQ) are used in this document to conceptually represent how the RNIC Engine tells the ULP about the completion of data transmission, or the completion of data reception through the Data Transfer Interface. (Specifically for Untagged Data Transfer. Tagged Data Transfer cannot cause a completion to occur.) Since there could be lots of transmissions or receptions in flight at once, completions get modeled as a queue rather than a single event 🔁 An implementation can also use the Completion Queue to notify the ULP about other activities, like finishing a mapping of an STag to a specific ULP buffer ✨ Completion Queues might be shared by a group of Streams, or designated to handle one specific Stream's traffic. Limiting Completion Queue association to one, or a small number, of RDMAP/DDP Streams can prevent several forms of attack by sharply limiting how far the attack's effect reaches 🛑
Some implementations may let this queue be manipulated directly by both Non-Privileged and Privileged ULPs 😬
2.2.7. Asynchronous Event Queue ✨
Okay so the Asynchronous Event Queue is a queue that runs from the RNIC up to the Privileged Resource Manager, and it has a bounded size 💅 The RNIC uses it to tell the host about all sorts of events that might need some management action, like protocol violations, Stream state changes, local operation errors, low water marks on receive queues, and possibly other events too 🫶
Now the tea ☕ The Asynchronous Event Queue can be attacked, bestie, because Remote or Local Peers and/or ULPs can make events happen that have the potential to overflow the queue 😭🔥
Heads up, an implementation is totally free to build the functions of the Asynchronous Event Queue in a bunch of ways, including multiple queues or even simple callbacks 🧚 All the vulnerabilities identified are meant to apply no matter how the Asynchronous Event Queue is implemented. For example, a callback function can be viewed as just a very short queue, iykyk 🎀
2.2.8. RDMA Read Request Queue 🦋
The RDMA Read Request Queue is the memory that holds state information for one or more RDMA Read Request Messages that have arrived, but whose RDMA Read Response Messages haven't been completely sent yet. Because more than one RDMA Read Request can be outstanding at one time, this memory gets modeled as a queue of bounded size. Some implementations may enable sharing of a single RDMA Read Request Queue across multiple Streams 💜
2.3. RNIC Interactions 🌸
With the RNIC resources and interfaces defined, we can now look at the interactions supported by the generic RNIC functional interfaces through each of the 3 interfaces: Privileged Control Interface, Privileged Data Interface, and Non-Privileged Data Interface. As mentioned earlier in Section 2.1, Components, there are two data transfer mechanisms to examine, Untagged Data Transfer and Tagged Data Transfer 🍓
2.3.1. Privileged Control Interface Semantics 👑
Generally speaking, the Privileged Control Interface controls the RNIC's allocation, de-allocation, and initialization of RNIC global resources. That covers allocation and de-allocation of Stream Context Memory, Page Translation Tables, STag names, Completion Queues, RDMA Read Request Queues, and Asynchronous Event Queues ✨
The Privileged Control Interface is also typically used to manage Non-Privileged ULP resources for the Non-Privileged ULP (and possibly for the Privileged ULP as well). That includes initialization and removal of Page Translation Table resources, and managing RNIC events (possibly managing all events for the Asynchronous Event Queue) 💖
2.3.2. Non-Privileged Data Interface Semantics 🦋
The Non-Privileged Data Interface enables data transfer (transmit and receive), but it does not allow initialization of the Page Translation Table resources. However, once those resources have been initialized, the interface may enable a specific STag mapping to be enabled and disabled by communicating directly with the RNIC, or create an STag mapping for a buffer that was previously initialized in the RNIC. Main character energy 🎀
For RDMAP, ULP data can be sent by one of the previously described data transfer mechanisms: Untagged Data Transfer or Tagged Data Transfer. Two RDMAP data transfer mechanisms are defined. One uses Untagged Data Transfer (Send Type Messages), and the other uses Tagged Data Transfer (RDMA Read Responses and RDMA Writes) 🫶 ULP data reception through RDMAP can be done by receiving Send Type Messages into buffers that have been posted on the Receive Queue or Shared Receive Queue. So a Receive Queue or Shared Receive Queue can only be affected by Untagged Data Transfer 🙈 Data reception can also be done by receiving RDMA Write and RDMA Read Response Messages into buffers that have previously been exposed for external write access through Advertisement of an STag (i.e., Tagged Data Transfer). Additionally, to cause ULP data to be pulled (read) across the network, RDMAP uses an RDMA Read Request Message, which only contains the RDMAP control information needed to access the ULP buffer to be read, to cause an RDMA Read Response Message to be generated that contains the ULP data 🌈
For DDP, transmitting data means sending DDP Tagged or Untagged Messages. For data reception, DDP can receive Untagged Messages into buffers that have been posted on the Receive Queue or Shared Receive Queue. It can also receive Tagged DDP Messages into buffers that have previously been exposed for external write access through Advertisement of an STag 💅
Completion of data transmission or reception generally means informing the ULP of the completed work by placing completion information on the Completion Queue. For data reception, only an Untagged Data Transfer can cause completion information to be put in the Completion Queue 😭
2.3.3. Privileged Data Interface Semantics 👑
The Privileged Data Interface semantics are a superset of the Non-Privileged Data Transfer semantics ✨ The interface can do everything in the prior section, and it can also create and destroy buffer to STag mappings directly. This generally involves initialization or clearing of Page Translation Table state in the RNIC 🧚
2.3.4. Initialization of RNIC Data Structures for Data Transfer 🔥
Initialization of the mapping between an STag and a Data Buffer can be viewed in the abstract as two separate operations:
a. Initialization of the allocated Page Translation Table entries with the location of the Data Buffer, and
b. Initialization of a mapping from an allocated STag name to a set of Page Translation Table entry(s) or partial entries.
Note that an implementation may not have a Page Translation Table at all (i.e., it may support a direct mapping between an STag and a Data Buffer). If there is no Page Translation Table, then attacks based on changing its contents or exhausting its resources are not possible 🌸
Initialization of the contents of the Page Translation Table can be done either by the Privileged ULP or by the Privileged Resource Manager acting as a proxy for the Non-Privileged ULP. By definition, the Non-Privileged ULP is not trusted to directly manipulate the Page Translation Table. In general, the concern is that the Non-Privileged ULP may try to maliciously initialize the Page Translation Table to access a buffer it doesn't have permission for 🚨
The exact resource allocation algorithm for the Page Translation Table is outside the scope of this document. It may be allocated for a specific Data Buffer, or as a pooled resource to be consumed by potentially multiple Data Buffers, or be managed in some other way. This document attempts to abstract implementation-dependent issues and group them into higher level security issues, such as resource starvation and sharing of resources between Streams 💜
The next question is how an STag name gets associated with a Data Buffer. For an Untagged Data Buffer (i.e., Untagged Data Transfer), there is no wire-visible mapping between an STag and the Data Buffer. Note that there may, in fact, be an STag that represents the buffer, if an implementation chooses to internally represent Untagged Data Buffers using STags. However, because the STag, by definition, is not visible on the wire, this is a local host, implementation-specific issue that should be analyzed in the context of a local host implementation-specific security analysis, and so it is outside the scope of this document 🙈
For a Tagged Data Buffer (i.e., Tagged Data Transfer), either the Privileged ULP or the Privileged Resource Manager acting on behalf of the Non-Privileged ULP may initialize a mapping from an STag to a Page Translation Table, or may have the ability to simply enable/disable an existing STag to Page Translation Table mapping. There may also be multiple STag names that map to a specific group of Page Translation Table entries (or sub-entries). Specific security issues with this level of flexibility are examined in Section 6.2.3, Multiple STags to Access the Same Buffer 👀
There are a variety of implementation options for initializing Page Translation Table entries and mapping an STag to a group of them that have security repercussions. These include support for separating the mapping of an STag from the mapping of a set of Page Translation Table entries, and support for ULPs directly manipulating STag to Page Translation Table entry mappings (versus requiring access through the Privileged Resource Manager) 🌈
2.3.5. RNIC Data Transfer Interactions 🦋
RNIC Data Transfer operations can be subdivided into send and receive operations ✨
For send operations, there is typically a queue that enables the ULP to post multiple operation requests to send data (referred to as the Send Queue). Depending on the implementation, Data Buffers used in these operations may or may not have Page Translation Table entries associated with them, and may or may not have STags associated with them. Because this is a local host specific implementation issue rather than a protocol issue, the security analysis of threats and mitigations is left to the host implementation 🧚
Receive operations are different for Tagged Data Buffers versus Untagged Data Buffers (i.e., Tagged Data Transfer vs. Untagged Data Transfer). For Untagged Data Transfer, if more than one Untagged Data Buffer can be posted by the ULP, the DDP specification requires that they be consumed in sequential order (the RDMAP specification also requires this). So the most general implementation is a sequential queue of receive Untagged Data Buffers (Receive Queue). Some implementations may also support sharing that sequential queue between multiple Streams. In that case, defining "sequential" gets non-trivial. In general, the buffers for a single Stream are consumed from the queue in the order they were placed on the queue, but there is no consumption order guarantee between Streams 🫶
For receive Tagged Data Transfer (i.e., Tagged Data Buffers, RDMA Write Buffers, or RDMA Read Buffers), at some time prior to the data transfer, the mapping of the STag to specific Page Translation Table entries (if present), and the mapping from the Page Translation Table entries to the Data Buffer, must have been initialized (see Section 2.3.4 for interaction details) 💖
3. Trust and Resource Sharing 🔥
okay bestie, so the default assumption is that the Local and Remote Peer are both untrusted 🙈. that means attacks from either side should have mitigations in place. no cap, we plan for the worst 💅
ok, now a related vibe: resource sharing between multiple Streams 🎀. if local resources aren't shared, they're dedicated on a per Stream basis. resources are defined in Section 2.2, Resources ✨. the upside of NOT sharing is that there are fewer kinds of attacks possible 🦋. the downside is ULPs might literally run out of resources 😭. so there can be a strong incentive to share, as long as we can mitigate the security issues that come with sharing 💜
it's assumed in this doc that the component controlling sharing of the RNIC Engine resources is the Privileged Resource Manager 👑. the RNIC Engine exposes its resources through the RNIC Interface to the Privileged Resource Manager 🧚. all Privileged and Non-Privileged ULPs request resources from the Resource Manager. and yes, by definition both the Non-Privileged and the Privileged app might try to greedily consume resources, which creates a potential Denial of Service (DOS) attack 💀. the Resource Manager implements resource management policies to keep access fair 🌸. it should be designed with the security attacks detailed in this document in mind. note that for some systems the Privileged Resource Manager may be implemented within the Privileged ULP 🍓.
okay this next part is a big deal 🚨. all Non-Privileged ULP interactions with the RNIC Engine that could affect other ULPs MUST be done using the Privileged Resource Manager as a proxy. and all ULP resource allocation requests for scarce resources MUST also be done using a Privileged Resource Manager.
the sharing of resources across Streams should be under the control of the ULP 💖. that covers both the trust model the ULP wants to operate under and how much sharing it wants to give local processes. for more on trust models that mix partial trust and resource sharing, check Appendix C, Partial Trust Taxonomy 📚.
last one, and it's a MUST NOT 🙅♀️. the Privileged Resource Manager MUST NOT assume that different Streams share Partial Mutual Trust, unless there's a mechanism to make sure the Streams really do share Partial Mutual Trust. that can be done in several ways, including explicit notification from the ULP that owns the Streams 🫶.
4. Attacker Capabilities ✨
an attacker's capabilities decide what kinds of attacks they can actually pull off 🕵️♀️. RDMAP and DDP need the initial LLP Stream (and connection) to be set up before any RDMAP/DDP Messages get transferred. that means at least one round-trip handshake has to happen first 🤝.
if the attacker is NOT the Remote Peer that created the initial connection, their capabilities can be split into send only or send and receive 🎭. an attacker with send only capabilities has to guess the current LLP Stream parameters (like the TCP sequence number) before they can go after RNIC resources 🎯. if this class of attacker also has receive capabilities, and can pose as the receiver to the sender and as the sender to the receiver, they're usually called a "man-in-the-middle" attacker [RFC3552] 😈. a man-in-the-middle has a much wider ability to attack RNIC resources. the breadth of their attack is basically the same as an attacking Remote Peer (aka the Remote Peer that set up the initial LLP Stream) 😬.
5. Attacks That Can Be Mitigated with End-to-End Security 🔐
this section covers the RDMAP/DDP attacks where the only real solution is some form of end-to-end security 💅. the analysis gives a detailed description of each attack, what's being attacked, and the countermeasures that can stop it 🛡️.
some attacks involve a network-based attacker modifying the RDMAP or DDP payload, or just monitoring the traffic to snoop private info 👀. a strong way to keep things confidential is to encrypt the data stream, using things like IPsec encryption 🔒. authentication protocols, like IPsec authentication, are also super effective for making sure the remote entity is who they say they are, and that the payload isn't modified as it crosses the network ✨.
note that connection setup and tear down is presumed to happen in stream mode (so no RDMA encapsulation of the payload) 🧁. that means there are no new attacks on setup/tear down beyond what's already in the LLP (like TCP or SCTP). but! RDMAP/DDP parameters might be exchanged in stream mode, and if an attacker corrupts them, unintended consequences will follow 😱. so any existing mitigations for LLP Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, or Elevation of Privilege still apply (and they're out of scope for this doc) 🫡. the analysis in this section focuses on attacks that are present regardless of the LLP Stream type.
Tampering is any modification of legitimate traffic, whether it's machine internal or on the network ⚠️. a Spoofing attack is a special case of tampering where the attacker fakes the identity of the Remote Peer (the identity could be an IP address, a machine name, a ULP level identity, etc.) 🎭.
5.1. Spoofing 🎭
spoofing attacks can be launched by the Remote Peer, or by a network-based attacker 🌐. a network-based spoofing attack applies to all Remote Peers. this section looks at the different types of spoofing attacks that apply to RDMAP and DDP 🔍.
5.1.1. Impersonation 🥸
a network-based attacker can impersonate a legal RDMAP/DDP Peer by spoofing a legal IP address 👻. they can do this as a blind attack (see [RFC3552]) or by establishing an RDMAP/DDP Stream with the victim. since an RDMAP/DDP Stream needs an LLP Stream to be fully initialized (for [RFC793], that means it's in the ESTABLISHED state), the existing transport layer protections against blind attacks still stand 🙌.
for a blind attack to work, the attacker has to inject a valid transport layer segment (for TCP, that means matching at least the 4-tuple and guessing a sequence number within the window) while ALSO guessing valid RDMAP or DDP parameters 🎲. there are lots of ways to attack RDMAP/DDP if the transport protocol is assumed to be vulnerable 😤. for Tagged Messages, this means guessing the STag and TO values. if the attacker just wants to kill the connection, they can do it by correctly guessing the transport and network layer values and providing an invalid STag 💀. per the DDP specification, if an invalid STag is received, the Stream is torn down and the Remote Peer is notified with an error. if an attacker wants to overwrite an Advertised Buffer, they must successfully guess the correct STag and TO. since the TO will often start at zero, that's pretty straightforward 😬. the value of the STag should be chosen at random, as discussed in Section 6.1.1, Using an STag on a Different Stream 🎲. for Untagged Messages, if the MSN is invalid then the connection may be torn down. if it's valid, the receive buffers can be corrupted 💔.
end-to-end authentication (like IPsec or ULP authentication) protects against either the blind attack or the connected attack 🛡️✨.
5.1.2. Stream Hijacking 🏴☠️
stream hijacking happens when a network-based attacker eavesdrops on the LLP connection through the Stream establishment phase, then waits until the authentication phase (if there is one) finishes successfully 👀. then they spoof the IP address and redirect the Stream from the victim to their own machine 😈. for example, an attacker could wait until an iSCSI authentication completes successfully, and then hijack the iSCSI Stream 🥷.
the best protection against this is end-to-end integrity protection and authentication, like IPsec, to prevent spoofing 🔐. another option is a physically segregated network for security. physical security is out of scope for this doc 🙃.
because the connection and/or Stream itself is established by the LLP, some LLPs are harder to hijack than others 🧐. check out the relevant LLP documentation for security issues around connection and/or Stream hijacking 📖.
5.1.3. Man-in-the-Middle Attack 😈
if a network-based attacker can delete or modify packets that the LLP still accepts (like when the TCP sequence number is correct), then the Stream can be exposed to a man-in-the-middle attack 🚨. one style of attack is for the man-in-the-middle to send Tagged Messages (either RDMAP or DDP) 📨. if they can find a buffer that's been exposed for STag enabled access, the man-in-the-middle can use an RDMA Read operation to read the contents of the associated Data Buffer, perform an RDMA Write Operation to modify those contents, or invalidate the STag to disable further access to the buffer 😭.
the best protection against this is end-to-end integrity protection and authentication, like IPsec, to prevent spoofing or tampering 🔐. if authentication and integrity protections aren't used, then physical protection must be employed to prevent man-in-the-middle attacks 🧱.
because the connection/Stream itself is established by the LLP, some LLPs are more exposed to man-in-the-middle attacks than others 🫣. check out the relevant LLP documentation on security issues around connection and/or Stream hijacking 📖.
another approach is to restrict access to only the local subnet/link, and add some mechanism to limit access, like physical security or 802.1.x 🔑. honestly this model is an extremely limited deployment scenario, so it won't be examined further here 🙈.
5.2. Tampering - Network-Based Modification of Buffer Content 🎀
this is actually a man-in-the-middle attack, but only on the content of the buffer 🧩. the man-in-the-middle attack above is different because there both the signaling and the content can be modified. see Section 5.1.3, Man-in-the-Middle Attack 🔗.
5.3. Information Disclosure - Network-Based Eavesdropping 👀
an attacker who can eavesdrop on the network can read the content of all read and write accesses to a Peer's buffers 😱. to prevent information disclosure, the read/written data must be encrypted 🔒. see also Section 5.1.3, Man-in-the-Middle Attack. the encryption can be done either by the ULP, or by a protocol that provides security services to RDMAP and DDP (like IPsec) 💖.
5.4. Specific Requirements for Security Services 🛡️
generally speaking, Stream confidentiality protects against eavesdropping 🤫. Stream and/or session authentication and integrity protection is a countermeasure against various spoofing and tampering attacks 💪. how well authentication and integrity work against a specific attack depends on whether the authentication is machine level (like IPsec) or ULP authentication 🌈.
5.4.1. Introduction to Security Options ✨
okay so these are the security services you can actually apply to an RDMAP/DDP Stream 🔐
- Session confidentiality: protects against eavesdropping (Section 5.3). 🙈
- Per-packet data source authentication: protects against these spoofing attacks: network-based impersonation (Section 5.1.1) and Stream hijacking (Section 5.1.2). 💅
- Per-packet integrity: protects against tampering done by network-based modification of buffer content (Section 5.2). When it's combined with authentication, it also protects against man-in-the-middle attacks (Section 5.1.3). 🔥
- Packet sequencing: protects against replay attacks, which is a special case of the above tampering attack. 🔁
so if an RDMAP/DDP Stream might get hit by impersonation or Stream hijacking, it is recommended that the Stream be authenticated, integrity protected, and protected from replay attacks. it may also use confidentiality protection to guard against eavesdropping, especially if the RDMAP/DDP Stream travels over a public network 🌐✨
IPsec is a protocol suite that secures communication at the network layer between two peers 🛡️ It's specified across the IP Security Architecture [RFC2401], IKE [RFC2409], IPsec Authentication Header (AH) [RFC2402], and IPsec Encapsulating Security Payload (ESP) [RFC2406] documents. IKE is the key management protocol, while AH and ESP are used to protect IP traffic. For the full scoop, go read those RFCs, bestie 📚
IPsec can provide the above security services for IP and TCP traffic, respectively. ULP protocols can only provide part of those security services 🧚
5.4.2. TLS Is Inappropriate for DDP/RDMAP Security 🙅♀️
TLS [RFC4346] provides Stream authentication, integrity and confidentiality for TCP based ULPs. It supports one-way (server only) or mutual certificate based authentication 🎀
if TLS is layered underneath RDMAP, TLS's connection orientation makes it inappropriate for DDP/RDMA security 💀 If a stream cipher or a block cipher in CBC mode is used for bulk encryption, then a packet can only be decrypted after all the packets before it have already arrived. So if TLS is protecting DDP/RDMAP traffic, TCP has to gather all the out-of-order packets before TLS can decrypt them. Only after that can RDMAP/DDP place them into the ULP buffer. Which means one of the main perks of DDP/RDMAP, the flow-through architecture with little to no buffering, just can't happen if TLS is guarding the data stream 😭
if TLS is layered on top of RDMAP or DDP, TLS does not protect the RDMAP and/or DDP headers. That leaves room for a man-in-the-middle attack that modifies the RDMAP/DDP header to put data into the wrong buffer, which effectively corrupts the data stream 🫠
so for these reasons, it is not RECOMMENDED that TLS be layered on top of RDMAP or DDP 🚫
5.4.3. DTLS and RDDP 🦋
DTLS [DTLS] provides security services for datagram protocols, including unreliable datagram protocols. These services include anti-replay, based on a mechanism adapted from IPsec that is meant to operate on packets as they arrive from the network. For these and other reasons, DTLS is best applied to RDDP by putting DTLS beneath TCP, which gives you a layering of RDDP over TCP over DTLS over UDP/IP. That layering puts DTLS at roughly the same level in the protocol stack as IPsec, so from an RDDP standpoint, DTLS's security services are an alternative to IPsec's services 👯♀️
for RDDP, IPsec is the better choice for a security framework, so it is mandatory-to-implement (as specified elsewhere in this document) 👑 A big reason IPsec won over DTLS is that the non-RDDP versions of two early adopters of RDDP, iSCSI [iSCSI][iSER] and NFSv4 [NFSv4][NFSv4.1], are compatible with IPsec, but neither of those protocols currently uses TLS or DTLS. For iSCSI specifically, IPsec is the basis for mandatory-to-implement security services [RFC3723]. So this document and the RDDP protocol specifications contain mandatory implementation requirements for IPsec rather than for DTLS 🎀✨
5.4.4. ULPs That Provide Security 🧩
ULPs that provide integrated security but want to lean on lower-layer protocol security should be aware of security concerns around correlating a specific channel's security mechanisms with the authentication the ULP performs. See [NFSv4CHANNEL] for more on a promising approach called "channel binding" 🔗 From [NFSv4CHANNEL]:
"The concept of channel bindings allows applications to prove that the end-points of two secure channels at different network layers are the same by binding authentication at one channel to the session protection at the other channel. The use of channel bindings allows applications to delegate session protection to lower layers, which may significantly improve performance for some applications."
5.4.5. Requirements for IPsec Encapsulation of DDP 📦
The IP Storage working group put a lot of time and effort into defining the normative IPsec requirements for IP Storage [RFC3723]. Parts of that spec apply to a wide variety of protocols, including the RDDP protocol suite. To avoid duplicating all that work, an RNIC implementation MUST follow the requirements defined in RFC 3723, Section 2.3 and Section 5, including the associated normative references for those sections. Note that this means support for IPSEC ESP mode is normative 💖
also, since IPsec acceleration hardware may only be able to handle a limited number of active IKE Phase 2 SAs, Phase 2 delete messages may be sent for idle SAs to keep the number of active Phase 2 SAs to a minimum ⏳ Receiving an IKE Phase 2 delete message MUST NOT be treated as a reason to tear down a DDP/RDMA Stream. Instead, it's preferable to leave the Stream up and, if more traffic gets sent on it, bring up another IKE Phase 2 SA to protect it. This avoids the risk of constantly bringing Streams up and down 🙃
There are serious security issues if IPsec is not implemented end-to-end. For example, if IPsec is implemented as a tunnel in the middle of the network, any hosts between the Peer and the IPsec tunneling device can freely attack the unprotected Stream 🚨
the IPsec requirements for RDDP are based on the version of IPsec specified in RFC 2401 [RFC2401] and related RFCs, as profiled by RFC 3723 [RFC3723], even though a newer version of IPsec exists, specified in RFC 4301 [RFC4301] and related RFCs. One of the important early applications of the RDDP protocols is their use with iSCSI [iSER], and RDDP's IPsec requirements follow those of IPsec so that a common profile of IPsec can be used with iSCSI and the RDDP protocols 🤝 In the future, RFC 3723 may be updated to the newer version of IPsec. Any such update's IPsec security requirements should apply uniformly to iSCSI and the RDDP protocols 🌈
6. Attacks from Remote Peers 🏹
This section describes remote attacks that are possible against the RDMA system defined in Figure 1 - RDMA Security Model and the RNIC Engine resources defined in Section 2.2. The analysis includes a detailed description of each attack, what is being attacked, and a description of the countermeasures that can be taken to thwart the attack 🕵️♀️
The attacks are classified into five categories: Spoofing, Tampering, Information Disclosure, Denial of Service (DoS) attacks, and Elevation of Privileges. As mentioned previously, tampering is any modification of the legitimate traffic (machine internal or network). A spoofing attack is a special case of tampering where the attacker falsifies an identity of the Remote Peer (identity can be an IP address, machine name, ULP level identity, etc.) 🎭
6.1. Spoofing 🫣
This section analyzes the various types of spoofing attacks applicable to RDMAP and DDP. Spoofing attacks can be launched by the Remote Peer or by a network-based attacker. For countermeasures against a network-based attacker, see Section 5, Attacks That Can Be Mitigated with End-to-End Security 💅✨
6.1.1. Using an STag on a Different Stream 🦋
okay so the first style of attack is the Remote Peer trying to use STag values it isn't allowed to touch 👀. quick reminder: if the Remote Peer sends an invalid STag to the Local Peer, the DDP and RDMAP specs say the Stream has to get torn down 💀. so the real threat is when an STag got enabled for Remote Access on one Stream, and the Remote Peer can still use it on a totally unrelated Stream 🙅♀️. if that works, the attacker could maybe do RDMA Reads to read the Data Buffer, do RDMA Writes to change it, or invalidate the STag so nobody can get into the buffer anymore 😤.
now, whether a Remote Peer using an STag from a different Stream in the same Protection Domain is an attack depends on whether sharing resources was intended, aka whether the Streams shared Partial Mutual Trust 🤝. some ULPs actually want one STag used across multiple Streams in the same Protection Domain, and that's totally valid 💖. other ULPs would call it an attack. since it changes from ULP to ULP, the ULP usually needs to be able to control the scope of the STag 🎀.
if the implementation doesn't share resources between Streams (STags included), this attack gets shut down by giving each Stream its own Protection Domain 🔒. before remote access to the buffer is allowed, the Protection Domain of the Stream making the access gets matched against the Protection Domain of the STag. if they don't match, access is denied, an error is generated, and the RDMAP Stream tied to the attacking Stream gets terminated 🚫.
for implementations that DO share resources between Streams, giving every Stream its own Protection Domain might not be practical 🥺. but the ULP can still limit the scope of any STag it enables for remote access to a single Stream. once the scope is limited to one Stream, any attempt to use that STag on a different Stream throws an error, and the RDMAP Stream gets terminated 💅.
so, for implementations that don't share STags between Streams: each Stream MUST either be in a separate Protection Domain or the scope of an STag MUST be limited to a single Stream. iconic rule, no notes 📜.
An RNIC MUST ensure that a specific Stream in a specific Protection Domain cannot access an STag in a different Protection Domain.
An RNIC MUST ensure that, if an STag is limited in scope to a single Stream, no other Stream can use the STag.
one more thing 🙃: there's also the chance of unintended STag sharing (a bug in the ULP), or a bug in the Remote Peer that ends up using an off-by-one STag. for extra protection, an implementation should allocate STags in a way that makes the next allocated STag number hard to predict, and make sure STags get reused at as slow a rate as possible 🐢. any allocation method that would lead to intentional or unintentional reuse of an STag by the peer should be avoided. so no methods that always start at a given STag and monotonically increase it with each new allocation, and no methods that always use the same STag for each operation 🙅♀️.
6.2. Tampering 🖊️
a Remote Peer or a network-based attacker can try to tamper with the contents of Data Buffers on a Local Peer that have been enabled for remote write access 😈. the tampering attacks from a Remote Peer are laid out in the sections that follow. for countermeasures against a network-based attacker, see Section 5, Attacks That Can Be Mitigated with End-to-End Security 🔐.
6.2.1. Buffer Overrun - RDMA Write or Read Response 💥
this attack is the Remote Peer trying to do an RDMA Write or RDMA Read Response to memory outside the valid length range of the Data Buffer that was enabled for remote write access 😱. it can happen even when no resources are shared across Streams. it can also show up if the ULP has a bug 🐛.
the countermeasure for this type of attack has to live in the RNIC implementation, and it leverages the STag 🛠️. when the local ULP tells the RNIC the base address and the number of bytes in the buffer it wants to make accessible, the RNIC must ensure that the base and bounds check gets applied to any access to the buffer referenced by the STag, before the STag is enabled for access. when an RDMA data transfer operation (which includes an STag) shows up on a Stream, a base and bounds byte granularity access check must be performed, so the operation only touches memory locations inside the buffer described by that STag 📏.
so an RNIC implementation MUST ensure that a Remote Peer is not able to access memory outside of the buffer specified when the STag was enabled for remote access 🚧.
6.2.2. Modifying a Buffer after Indication 🕵️♀️
this attack can happen if a Remote Peer tries to change the contents of an STag referenced buffer, by doing an RDMA Write or an RDMA Read Response after it has already signaled to the Local Peer or local ULP (in a bunch of different ways) that the STag Data Buffer contents are ready to use 🙈. it can happen even when no resources are shared across Streams. and a bug in the Remote Peer, or network-based tampering, could also cause this problem 🫠.
for example: say the STag referenced buffer holds ULP control information as well as ULP payload, and the ULP's sequence is to validate the control information first and then operate on it 📋. if the Remote Peer can do one more RDMA Write or RDMA Read Response (which changes the buffer) after the validity checks are done but before the control data gets operated on, the Remote Peer could force the ULP down operational paths that were never meant to happen 😭.
the local ULP can protect itself from this by revoking remote access once the original data transfer finishes, and before it validates the buffer contents 🔐. it can do that either by explicitly revoking remote access rights for the STag when the Remote Peer says the operation is complete, or by checking that the Remote Peer invalidated the STag through the RDMAP Remote Invalidate capability. if the Remote Peer didn't invalidate the STag, the local ULP then explicitly revokes the STag's remote access rights. (See Section 6.4.5, Remote Invalidate an STag Shared on Multiple Streams for a definition of Remote Invalidate.)
the local ULP SHOULD follow the procedure above to protect the buffer before it validates the contents of the buffer (or uses the buffer in any way) ✅.
An RNIC MUST ensure that network packets using the STag for a previously Advertised Buffer can no longer modify the buffer after the ULP revokes remote access rights for the specific STag 🚫✨.
6.2.3. Multiple STags to Access the Same Buffer 🔁
see Section 6.3.6 Using Multiple STags That Alias to the Same Buffer for this analysis 🧐.
6.3. Information Disclosure 👀
the main place info can leak from is a local buffer that has been enabled for remote access 🫣. if a Remote Peer on another Stream can probe the buffer, then information disclosure is possible.
the attacks that could lead to unintended information disclosure, plus their countermeasures, are detailed in the sections below 📚.
6.3.1. Probing Memory Outside of the Buffer Bounds 🔍
this is basically the same attack as in Section 6.2.1, Buffer Overrun - RDMA Write or Read Response, except it uses an RDMA Read Request to pull it off 📖. the same countermeasure applies.
6.3.2. Using RDMA Read to Access Stale Data 🕰️
if a buffer is being used for some mix of reads and writes (remote or local), and it's exposed to a Remote Peer with at least remote read access rights before it's initialized with the correct data, there's a potential race condition 🏁. the Remote Peer could view the buffer's old contents. that becomes a security issue if those old contents weren't meant to be shared with the Remote Peer 😬.
to kill this race condition, the local ULP SHOULD make sure no stale data is in the buffer before remote read access rights are granted 🧹. one way is zeroing out the memory. that way the Remote Peer can't touch the buffer until the stale data is gone 🫧.
6.3.3. Accessing a Buffer after the Transfer 📦
if the Remote Peer has remote read access to a buffer and, somehow, tells the local ULP the transfer is done, but the local ULP doesn't disable remote access before it changes the data, the Remote Peer can grab the new data 😳.
this is similar to the attack in Section 6.2.2, Modifying a Buffer after Indication, so the same countermeasures apply 🔁. on top of that, the local ULP SHOULD grant remote read access rights only for as long as it takes to retrieve the data ⏱️.
6.3.4. Accessing Unintended Data with a Valid STag 🎯
if the ULP enables remote access to a buffer using an STag that references the entire buffer, but only wants a portion of it accessed, the Remote Peer can still get into the other parts of the buffer 😤.
to stop this, the ULP SHOULD set the base and bounds of the buffer when the STag is initialized, so only the data to be retrieved is exposed 🔒.
6.3.5. RDMA Read into an RDMA Write Buffer 🫢
one form of disclosure happens if the access rights on the buffer enable remote read when only remote write was intended 🙀. if the buffer held ULP data, or data from a transfer on an unrelated Stream, the Remote Peer could retrieve that data through an RDMA Read operation. side note: an RNIC implementation is not required to support STags that have both read and write access 🤷♀️.
the most obvious countermeasure is to not grant remote read access if the buffer is meant to be write-only 🚫📖. then the Remote Peer can't pull any data associated with the buffer. if it tries anyway, that gives an error, and the RDMAP Stream associated with the Stream gets terminated 💀.
so, if a ULP only intends a buffer to be exposed for remote write access, it MUST set the access rights to the buffer so that only remote write access is enabled 🧱. this requirement isn't meant to block zero-length RDMA Reads 🫶. zero-length RDMA Reads don't expose ULP data. they're meant to be a way to make sure all RDMA Writes have been received, and they don't even need a valid STag, so their use is allowed even when a buffer is only enabled for write access 🌸.
6.3.6. Using Multiple STags That Alias to the Same Buffer ✨
okay bestie, this one is a big deal 🫣 Multiple STags that point at the same buffer at the same time can cause unintentional information disclosure if those STags get used by different Remote Peers that don't trust each other. This mostly matters for client/server setups, where one server is talking to a bunch of clients who have no reason to trust one another 🙈
If only read access is enabled, the local ULP is fully in charge of who sees what 💅 So if a server deliberately shares the same data (aka the same buffer) with several clients by handing out multiple STags to that buffer, that doesn't create any new security problems beyond what we already covered in this document. But if the server did NOT mean to share the data with those clients, it should use separate buffers for each client (and separate STags) 🎀
Now, when one STag has remote read access enabled and a different STag has remote write access enabled to the same buffer, one Remote Peer can see the contents that another Remote Peer wrote 👀 Not it.
If both STags have remote write access enabled and the two Remote Peers don't mutually trust each other, one Remote Peer can overwrite what the other Remote Peer wrote 😭
So the rule is: a ULP with multiple Remote Peers that don't share Partial Mutual Trust MUST NOT grant write access to the same buffer through different STags 🚫 And a buffer should only be exposed to one untrusted Remote Peer at a time, so no information disclosure or information tampering can sneak in between peers 🔒
6.4. Denial of Service (DOS) 💀
DOS attacks are one of the biggest security risks for RDMAP, and it's giving high stakes. RNIC resources are valuable and scarce, and lots of ULP environments have to talk to untrusted Remote Peers. If the Remote Peer can be authenticated, or the ULP payload is encrypted, the DOS risk goes down, obviously. For this analysis though, we assume the RNIC has to be able to work in untrusted environments that are open to DOS-style attacks 🧚
DOS attacks against RNIC resources aren't the usual vibe of some random stranger spraying packets at a host (like a TCP SYN attack). The connection/Stream has to be fully established first (e.g., a 3-message transport layer handshake has occurred) 🤝 So the attacker either has to be able to both send and receive messages over that connection/Stream, or be able to guess a valid packet on an existing RDMAP Stream. Main character energy for the attacker, lowkey 😤
This section goes through the potential attacks and the countermeasures for dealing with each one 🌸
6.4.1. RNIC Resource Consumption 🍓
This section covers attacks in the general category of a local ULP trying to unfairly grab scarce (i.e., bounded) RNIC resources. The local ULP might be trying to allocate resources for itself, or for a Remote Peer. The resources in this category are Protection Domains, Stream Context Memory, Translation and Protection Tables, and the STag namespace. These attacks can come from local ULPs that are active right now, or from ones that allocated resources earlier and are now just sitting there idle 👀
This type of attack can happen whether or not resources are shared across Streams 💅
All allocation of scarce resources MUST be placed under the control of a Privileged Resource Manager. That lets the Privileged Resource Manager:
- prevent a local ULP from allocating more than its fair share of resources.
- detect if a Remote Peer is trying to launch a DOS attack by attempting to create an excessive number of Streams (with associated resources), and take corrective action (such as refusing the request or applying network layer filters against the Remote Peer) 🫶
This analysis assumes the Resource Manager is responsible for handing out Protection Domains, and that RNIC implementations will provide enough Protection Domains so the Resource Manager can give each unrelated, untrusted local ULP its own unique Protection Domain (for a bounded, reasonable number of local ULPs) ✨ It also assumes the Resource Manager has policies so untrusted local ULPs can't eat up all the Protection Domains through a DOS attack. And note that Protection Domain consumption can't come from a DOS attack launched by a Remote Peer, unless a local ULP is acting on the Remote Peer's behalf. Iykyk 🧠
6.4.2. Resource Consumption by Idle ULPs 😴
The simplest form of DOS attack, when you've got a fixed amount of resources, is for the Remote Peer to create an RDMAP Stream to a Local Peer, ask for dedicated resources, and then just... do nothing 🙃 This lets the Remote Peer stay super lightweight (it only negotiates resources, with no data transfer), while using up a disproportionate amount of resources at the Local Peer. Rude 😤
A general countermeasure for this style of attack is to monitor active RDMAP Streams and, if resources start getting low, reap the resources from RDMAP Streams that aren't transferring data, and possibly terminate the Stream. This would presumably be under administrative control 🧹
Check out Section 6.4.1 for the analysis and countermeasures for this style of attack on these RNIC resources: Stream Context Memory, Page Translation Tables, and STag namespace 🔍
Some RNIC resources aren't at risk from this type of attack by a Remote Peer, because the attack requires the Remote Peer to send messages in order to consume the resource. Receive Data Buffers, Completion Queue, and RDMA Read Request Queue resources are examples 🙌 These resources are still at risk from a local ULP that allocates resources and then goes idle. It could also happen if the ULP negotiates resource levels with the Remote Peer, which makes the Local Peer consume resources, but the Remote Peer never sends any data to use them. The general countermeasure from this section can free resources allocated by an idle Local Peer 🫧
6.4.3. Resource Consumption by Active ULPs 🔥
This section describes DOS attacks from Local and Remote Peers that are actively exchanging messages. Attacks on each RDMA NIC resource are examined, and specific countermeasures are identified 🎯 Attacks on Stream Context Memory, Page Translation Tables, and STag namespace are covered in Section 6.4.1, RNIC Resource Consumption, so they aren't included here.
6.4.3.1. Multiple Streams Sharing Receive Buffers 🧺
The Remote Peer can try to consume more than its fair share of receive Data Buffers (i.e., Untagged Buffers for DDP or Send Type Messages for RDMAP) if receive buffers are shared across multiple Streams 😬
If resources aren't shared across multiple Streams, this attack isn't possible, because the Remote Peer won't be able to consume more buffers than were allocated to its Stream. The worst case is that the Remote Peer consumes more receive buffers than the local ULP allowed, which leaves no buffers available. That could cause the Remote Peer's Stream to the Local Peer to be torn down, and all allocated resources to be released 💔
If local receive Data Buffers are shared among multiple Streams, the Remote Peer can try to consume more than its fair share of those receive buffers. That leaves a different Stream short of receive buffers, which could get that other Stream torn down. For example, if the Remote Peer sent enough one-byte Untagged Messages, it could eat all the locally shared receive queue resources with barely any effort 🥲
One method the Local Peer could use is to recognize that a Remote Peer is using more than its fair share of resources and terminate that Stream (which releases the allocated resources). But if the Local Peer is sufficiently slow, the Remote Peer may still be able to pull off a denial of service attack 🐢 One countermeasure that can protect against this is implementing a low-water notification. The low-water notification alerts the ULP if the number of buffers in the receive queue drops below a threshold 🚨
If all of the following conditions are true, the Local Peer or local ULP can size the amount of local receive buffers posted on the receive queue so a DOS attack can be stopped:
- A low-water notification is enabled, and
- The Local Peer is able to bound the amount of time it takes to replenish receive buffers, and
- The Local Peer maintains statistics to determine which Remote Peer is consuming buffers.
When those conditions hold, the low-water notification can arrive before resources run out, so the Local Peer or local ULP can take corrective action (e.g., terminate the Stream of the attacking Remote Peer) 💪
A different but similar attack happens when the Remote Peer sends a significant number of out-of-order packets and the RNIC is able to use the ULP buffer (i.e., the Untagged Buffer for DDP or the buffer consumed by a Send Type Message for RDMAP) as a reassembly buffer. In this case the Remote Peer can consume a bunch of ULP buffers but never send enough data for the ULP buffer to be completed and delivered to the ULP 😤
An effective countermeasure is to create a high-water notification that alerts the ULP if there are more than a specified number of receive buffers "in process" (partially consumed, but not completed). The notification is generated when more than the specified number of buffers are in process at the same time on a specific Stream (i.e., packets have started to arrive for the buffer, but the buffer hasn't been delivered to the ULP yet) 🫧
Another countermeasure is for the RNIC Engine to be able to limit the Remote Peer's ability to consume receive buffers on a per Stream basis. Unfortunately, this requires a large amount of state to be tracked in each RNIC on a per Stream basis, which is a lot 🥴
So if an RNIC Engine lets receive buffers be shared across multiple Streams, the combination of the RNIC Engine and the Privileged Resource Manager MUST be able to detect if the Remote Peer is trying to consume more than its fair share of resources, so the Local Peer or local ULP can apply countermeasures to detect and prevent the attack 🛡️✨
6.4.3.2. Remote or Local Peer Attacking a Shared CQ ✨
okay so if you want the big-picture vibe on the shared CQ attack model, peek at Section 7.1 🦋
the Remote Peer can go full main character and hog more than its fair share of CQ entries 💅 it has two sneaky ways to do it:
- the ULP protocol lets the Remote Peer make the local ULP reserve a set number of CQ entries, which can leave basically nothing for the other Streams sharing the CQ 😭
- if the Remote Peer, Local Peer, or local ULP (or any combo of them) floods the CQ with completions, then completion processing for every other Stream on that Completion Queue can get wrecked (like, the CQ overflows and just stops working) 💀
the first one can be dodged if the ULP doesn't let a Remote Peer reserve CQ entries, or if there's a trusted middleman, like a Privileged Resource Manager. the tea is that it's often unrealistic to say no to reservations, especially when the number of completion entries depends on other ULP-negotiated parameters, like how much buffering the ULP needs 🫠 so an implementation MUST implement a Privileged Resource Manager to control the allocation of CQ entries. check Section 2.1, Components, for the definition of a Privileged Resource Manager 🎀
a Local or Remote Peer can also try to bury a CQ in completions by sending minimum length RDMAP/DDP Messages, so it racks up as many completions per second as possible (receive completions for the Remote Peer, send completions for the Local Peer) 🏃♀️ if the Remote Peer is the one attacking, and we assume the Local Peer's receive queue(s) don't run out of receive buffers (if they do, that's a different attack, covered in Section 6.4.3.1 Multiple Streams Sharing Receive Buffers), then the Remote Peer might be able to eat more than its fair share of Completion Queue entries 🍓 depending on the CQ implementation, this either overflows the CQ (if it's too small for all the completions being generated), or another Stream can't generate CQ entries at all (if the RNIC had flow control on generating CQ entries into the CQ). either way, the CQ stops working correctly, and any Streams expecting completions on it stop working too 😩
this attack works no matter whether all the Streams on the CQ are in the same or different Protection Domains. the real problem is that the number of Completion Queue entries is less than the number of outstanding operations that can cause a completion 🙈
the Local Peer can protect itself using either of these:
- size the CQ to the right level, as specified below (note that if the CQ already exists and needs resizing, the resize isn't guaranteed to succeed in every case, so do the CQ resize before sizing the Send Queue and Receive Queue on the Stream), OR
- grant fewer resources than the Remote Peer asked for (don't supply the number of Receive Data Buffers requested).
how you size the CQ depends on whether the local ULP(s) will actually post as many resources to the various queues as each queue can hold 🧚 if the local ULP(s) can be trusted to post fewer resources than the size of a given queue, then a correctly sized CQ is big enough to hold completion status for all the outstanding Data Buffers (both send and receive buffers), like this:
CQ_MIN_SIZE = SUM(MaxPostedOnEachRQ)
+ SUM(MaxPostedOnEachSRQ)
+ SUM(MaxPostedOnEachSQ)where:
MaxPostedOnEachRQ = the maximum number of requests that
can cause a completion that will be posted on a
specific Receive Queue.
MaxPostedOnEachSRQ = the maximum number of requests that
can cause a completion that will be posted on a
specific Shared Receive Queue.
MaxPostedOnEachSQ = the maximum number of requests that
can cause a completion that will be posted on a
specific Send Queue.if the local ULP has to be able to completely fill the queues, or can't be trusted to observe a limit smaller than the queues, then the CQ has to be sized for the maximum number of operations that can ever be posted at one time 🫶 so the equation becomes:
CQ_MIN_SIZE = SUM(SizeOfEachRQ)
+ SUM(SizeOfEachSRQ)
+ SUM(SizeOfEachSQ)where:
SizeOfEachRQ = the maximum number of requests that
can cause a completion that can ever be posted
on a specific Receive Queue.
SizeOfEachSRQ = the maximum number of requests that
can cause a completion that can ever be posted
on a specific Shared Receive Queue.
SizeOfEachSQ = the maximum number of requests that
can cause a completion that can ever be posted
on a specific Send Queue.MaxPosted*OnEach*Q and SizeOfEach*Q can vary on a per Stream or per Shared Receive Queue basis 🌈
if the ULP shares a CQ across multiple Streams that don't share Partial Mutual Trust, then the ULP MUST implement a mechanism to make sure the Completion Queue doesn't overflow 🚨 it's still possible to share CQs even when the Remote Peers using them are untrusted, as long as one of the two formulas above is implemented. if the ULP can be trusted not to post more than MaxPostedOnEachRQ, MaxPostedOnEachSRQ, and MaxPostedOnEachSQ, then the first formula applies. if the ULP can't be trusted to obey those limits, then the second formula applies ✨
6.4.3.3. Attacking the RDMA Read Request Queue 🔥
the RDMA Read Request Queue can get attacked if the Remote Peer sends more RDMA Read Requests than the depth of the RDMA Read Request Queue at the Local Peer 😬 if that queue is a shared resource, this could corrupt it. if the queue isn't shared, the worst case is that the current Stream stops being functional (e.g., it gets torn down). one approach for the shared RDMA Read Request Queue would be to set up thresholds, similar to the ones in Section 6.4.3.1, Multiple Streams Sharing Receive Buffers. a simpler approach is to not share RDMA Read Request Queue resources among Streams, or to enforce hard consumption limits per Stream 💅 so RDMA Read Request Queue resource consumption MUST be controlled by the Privileged Resource Manager, so that RDMAP/DDP Streams that don't share Partial Mutual Trust don't share RDMA Read Request Queue resources. no sharing across trust boundaries, periodt 🙅♀️
if the problem is a bug in the Remote Peer's implementation rather than a malicious attack, it can be solved by requiring the Remote Peer's RNIC to throttle RDMA Read Requests 🌸 by properly configuring the Stream at the Remote Peer through a trusted agent, the RNIC can be made not to transmit RDMA Read Requests that exceed the depth of the RDMA Read Request Queue at the Local Peer. if the Stream is configured correctly and the Remote Peer submits more requests than the Local Peer's RDMA Read Request Queue can handle, those requests just wait in a queue at the Remote Peer's RNIC until earlier requests complete. if the Remote Peer's Stream isn't configured correctly, the RDMAP Stream gets terminated when more RDMA Read Requests arrive at the Local Peer than it can handle (assuming the recommendation in the paragraph before this one is implemented) 💀 so an RNIC implementation SHOULD provide a mechanism to cap the number of outstanding RDMA Read Requests. how that limit gets configured is outside the scope of this document 🧸
6.4.4. Exercise of Non-Optimal Code Paths 🐢
another flavor of DOS attack is trying to exercise data paths that eat a disproportionate amount of resources 😤 an example would be if error cases get handled on a "slow path" (using either host or RNIC computational resources), and an attacker generates a ton of errors just to drain those resources. note that for most RDMAP or DDP errors, the attacking Stream just gets torn down, so for this attack to actually work, the Remote Peer needs to exercise data paths that don't cause the Stream to be torn down 🙃
if an RNIC implementation has "slow paths" that don't result in the Stream being torn down, it is recommended that the implementation give admins a way to detect this condition and act on it 🫠 that can include potentially, administratively tearing down the RDMAP Stream that's exercising the data paths using a disproportionate amount of resources ✨
6.4.5. Remote Invalidate an STag Shared on Multiple Streams 🦋
if a Local Peer has enabled an STag for remote access, the Remote Peer could try to remotely invalidate that STag using the RDMAP Send with Invalidate or Send with SE and Invalidate Message 👀 if the STag is only valid on the current Stream, the only side effect is that the Remote Peer can't use the STag anymore, so there are no security issues 🌷
if the STag is valid across multiple Streams, though, the Remote Peer can stop other Streams from using that STag by using the Remote Invalidate functionality 😱
so if RDDP Streams don't share Partial Mutual Trust (i.e., the Remote Peer may try to remotely invalidate the STag prematurely), the ULP MUST NOT enable an STag that would be valid across multiple Streams 🚫
6.4.6. Remote Peer Attacking an Unshared CQ 💥
the Remote Peer can attack an unshared CQ if the Local Peer doesn't size the CQ correctly 😵 for example, if the Local Peer sets the CQ up to handle completions of received buffers, and the receive buffer queue is longer than the Completion Queue, then an overflow can happen. the effect on the attacker's own Stream is catastrophic. but if an RNIC doesn't have the proper protections in place, an attack that overflows the CQ can also cause corruption and/or termination of an unrelated Stream 🥲 so an RNIC MUST ensure that if a CQ overflows, any Streams that don't use the CQ MUST remain unaffected 🛡️
6.5. Elevation of Privilege 👑
the RDMAP/DDP Security Architecture explicitly splits privilege into three levels: Non-Privileged, Privileged, and the Privileged Resource Manager 🎀 if a Non-Privileged ULP can elevate its privilege level to a Privileged ULP, then mapping a physical address list to an STag can give local and remote access to any physical address location on the node 😱 if a Privileged Mode ULP can promote itself to be a Resource Manager, it can run denial of service type attacks where substantial amounts of local resources get consumed.
in general, elevation of privilege is a local implementation specific issue, so it's outside the scope of this document 🧚
7. Attacks from Local Peers 🌈
this section describes local attacks that are possible against the RDMA system defined in Figure 1 - RDMA Security Model and the RNIC Engine resources defined in Section 2.2 💜
7.1. Local ULP Attacking a Shared CQ 💅
Okay bestie, DOS attacks on a Shared Completion Queue (CQ - see Section 2.2.6, Completion Queues) can come from the local ULP or the Remote Peer, if either one tries to cause more completions than its fair share of the entries 😤. That could starve some totally unrelated ULP so no Completion Queue entries are left for it 💀.
A Completion Queue entry can get maliciously eaten by a completion from the Send Queue or a completion from the Receive Queue 🫣. For the first one, the attacker is the local ULP. For the second, it's the Remote Peer.
There's also a sneakier version where local ULPs eat up resources on the CQ. A local ULP that's slow to free those resources by not reaping its completion status fast enough could stall every other local ULP trying to use that CQ 🙈.
So for all these reasons, an RNIC MUST NOT enable sharing a CQ across ULPs that do not share Partial Mutual Trust 🚫✨
7.2. Local Peer Attacking the RDMA Read Request Queue 🦋
If RDMA Read Request Queue resources are pooled across multiple Streams, one attack is a local ULP trying to unfairly allocate them for its Streams. For example, it tries to grab every available resource on one specific RDMA Read Request Queue for its own Streams, which denies that resource to the ULPs sharing the queue 😡. The same idea applies even when the RDMA Read Request Queue isn't shared, if a local ULP tries to allocate all of the RNIC's resources when the queue gets created.
So, access to the interfaces that allocate RDMA Read Request Queue entries MUST be restricted to a trusted Local Peer, such as a Privileged Resource Manager. The Privileged Resource Manager SHOULD prevent a local ULP from allocating more than its fair share of resources 🫶
7.3. Local ULP Attacking the PTT and STag Mapping 🧚
If a Non-Privileged ULP can directly manipulate the RNIC Page Translation Tables (which translate from an STag to a host address), it's possible that it could point the Page Translation Table at an unrelated Stream's or ULP's buffers. Then it could gain access to the information of that unrelated Stream/ULP 👀.
As discussed in Section 2, Architectural Model, introducing a Privileged Resource Manager to arbitrate the mapping requests is an effective countermeasure. This lets the Privileged Resource Manager make sure a local ULP can only initialize the Page Translation Table (PTT) to point at its own buffers ✨
So if Non-Privileged ULPs are supported, the Privileged Resource Manager MUST verify that the Non-Privileged ULP has the right to access a specific Data Buffer before allowing an STag for which the ULP has access rights to be associated with that Data Buffer. This can be done when the Page Translation Table is initialized to access the Data Buffer, when the STag is initialized to point to a group of Page Translation Table entries, or both 💖
8. Security considerations 🔐
Go check out Sections 5, Attacks That Can be Mitigated with End-to-End Security; Section 6, Attacks from Remote Peers; and Section 7, Attacks from Local Peers, for a detailed analysis of attacks and the normative countermeasures that mitigate them. This is the serious stuff, so read it all 😭
The appendices also give a summary of the security requirements for specific audiences 🎀. Appendix A, ULP Issues for RDDP Client/Server Protocols, summarises implementation issues and requirements for applications that use a traditional client/server style of interaction. It adds extra insight into how the normative text in Sections 5, 6, and 7 applies. Appendix B, Summary of RNIC and ULP Implementation Requirements, is a handy summary of the normative requirements for implementers 🧠
9. IANA Considerations 📋
IANA considerations are not addressed by this document. Any IANA considerations resulting from the use of DDP or RDMA must be addressed in the relevant standards 🌸
10. References 📚
10.1. Normative References
[DDP]Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct Data Placement over Reliable Transports", RFC 5041, October 2007.[RDMAP]Recio, R., Culley, P., Garcia, D., and J. Hilland, "A Remote Direct Memory Access Protocol Specification", RFC 5040, October 2007.[RFC2401]Kent, S. and R. Atkinson, "Security Architecture for the Internet Protocol", RFC 2401, November 1998.[RFC2402]Kent, S. and R. Atkinson, "IP Authentication Header", RFC 2402, November 1998.[RFC2406]Kent, S. and R. Atkinson, "IP Encapsulating Security Payload (ESP)", RFC 2406, November 1998.[RFC2409]Harkins, D. and D. Carrel, "The Internet Key Exchange (IKE)", RFC 2409, November 1998.[RFC3723]Aboba, B., Tseng, J., Walker, J., Rangan, V., and F. Travostino, "Securing Block Storage Protocols over IP", RFC 3723, April 2004.[RFC4960]Stewart, R., Ed., "Stream Control Transmission Protocol", RFC 4960, September 2007.[RFC793]Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981.
10.2. Informative References
[RFC4301]Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, December 2005.[RFC4346]Dierks, T. and E. Rescorla, "The Transport Layer Security (TLS) Protocol Version 1.1", RFC 4346, April 2006.[RFC4949]Shirey, R., "Internet Security Glossary, Version 2", RFC 4949, August 2007.[APPLICABILITY]Bestler, C. and L. Coene, "Applicability of Remote Direct Memory Access Protocol (RDMA) and Direct Data Placement (DDP)", RFC 5045, October 2007.[NFSv4CHANNEL]Williams, N., "On the Use of Channel Bindings to Secure Channels", Work in Progress, July 2004.[VERBS-RDMAC]"RDMA Protocol Verbs Specification", RDMA Consortium standard, April 2003,<http://www.rdmaconsortium.org/home/draft-hilland-iwarp-verbs-v1.0-RDMAC.pdf>.[VERBS-RDMAC-Overview]"RDMA enabled NIC (RNIC) Verbs Overview", slide presentation by Renato Recio, April 2003,<http://www.rdmaconsortium.org/home/RNIC_Verbs_Overview2.pdf>.[RFC3552]Rescorla, E. and B. Korver, "Guidelines for Writing RFC Text on Security Considerations", BCP 72, RFC 3552, July 2003.[INFINIBAND]"InfiniBand Architecture Specification Volume 1", release 1.2, InfiniBand Trade Association standard,<http://www.infinibandta.org/specs>. Verbs are documented in chapter 11.[DTLS]Rescorla, E. and N. Modadugu, "Datagram Transport Layer Security", RFC 4347, April 2006.[iSCSI]Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and E. Zeidner, "Internet Small Computer Systems Interface (iSCSI)", RFC 3720, April 2004.[iSER]Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah, H., and P. Thaler, "Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)", RFC 5046, October 2007.[NFSv4]Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame, C., Eisler, M., and D. Noveck, "Network File System (NFS) version 4 Protocol", RFC 3530, April 2003.[NFSv4.1]Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "NFSv4 Minor Version 1", Work in Progress, September 2007.
Appendix A: ULP Issues for RDDP Client/Server Protocols ✨
Okay bestie, this section is a normative appendix 🎀 about client/server ULP implementation requirements, and its whole job is making sure a server stays secure 🔒.
The earlier sections laid out specific attacks and their countermeasures. This appendix recaps the ones that matter for building a secure ULP (aka application) server 💜. A ULP server is a ULP that has to talk to lots of clients who don't necessarily trust each other, and it has to make sure one client can't attack another client through the server 👑. The server might also want several Streams to talk to one specific client, and those Streams might share mutual trust 🫶. Quick note: this section assumes a compliant RNIC and Privileged Resource Manager implementation, so it's laser focused on ULP server (aka application) issues 🎯.
All of the earlier details on attacks and countermeasures apply to the server, so the requirements repeated here use non-normative "must", "should", and "may" 🌸. In some cases, normative SHOULD statements for the ULP from the main body get turned into MUST statements for the ULP server, because the operating conditions can be refined so the reasons for a SHOULD don't apply anymore ✨. If a prior SHOULD gets changed to a MUST in this section, it's explicitly noted and written as uppercase normative statements 💖.
The list below sums up the relevant attacks that clients can mount on the shared server, by restating the earlier normative statements in client/server terms 🦋. Each client/server ULP might use explicit RDMA Operations (RDMA Read, RDMA Write) in different ways, so where it makes sense, "Local ULP", "Local Peer", and "Remote Peer" get used in place of "server" or "client" to keep every requirement fully general 🧚.
- Spoofing
- Sections 5.1.1 to 5.1.3. For protection against many forms of spoofing attacks, enable IPsec.
- Section 6.1.1, Using an STag on a Different Stream. To ensure that one client cannot access another client's data via use of the other client's STag, the server ULP must either scope an STag to a single Stream or use a unique Protection Domain per client. If a single client has multiple Streams that share Partial Mutual Trust, then the STag can be shared between the associated Streams by using a single Protection Domain among the associated Streams (see Section 5.4.4, ULPs That Provide Security, for additional issues). To prevent unintended sharing of STags within the associated Streams, a server ULP should use STags in such a fashion that it is difficult to predict the next allocated STag number. 🔑
- Tampering
- 6.2.2 Modifying a Buffer after Indication. Before the local ULP operates on a buffer that was written by the Remote Peer using an RDMA Write or RDMA Read, the local ULP MUST ensure the buffer can no longer be modified by invalidating the STag for remote access (note that this is stronger than the SHOULD in Section 6.2.2). This can be done either by explicitly revoking remote access rights for the STag when the Remote Peer indicates the operation has completed, or by checking to make sure the Remote Peer Invalidated the STag through the RDMAP Invalidate capability. If the Remote Peer did not invalidate the STag, the local ULP then explicitly revokes the STag remote access rights. 🙅♀️
- Information Disclosure
- 6.3.2, Using RDMA Read to Access Stale Data. In a general purpose server environment, there is no compelling rationale not to require a buffer to be initialized before remote read is enabled (and an enormous downside of unintentionally sharing data). Thus, a local ULP MUST (this is stronger than the SHOULD in Section 6.3.2) ensure that no stale data is contained in a buffer before remote read access rights are granted to a Remote Peer (this can be done by zeroing the contents of the memory, for example). 🧹
- 6.3.3, Accessing a Buffer after the Transfer. This mitigation is already covered by Section 6.2.2 (above).
- 6.3.4, Accessing Unintended Data with a Valid STag. The ULP must set the base and bounds of the buffer when the STag is initialized to expose only the data to be retrieved.
- 6.3.5, RDMA Read into an RDMA Write Buffer. If a peer only intends a buffer to be exposed for remote write access, it must set the access rights to the buffer to only enable remote write access.
- 6.3.6, Using Multiple STags That Alias to the Same Buffer. The requirement in Section 6.1.1 (above) mitigates this attack. A server buffer is exposed to only one client at a time to ensure that no information disclosure or information tampering occurs between peers. 🙈
- 5.3, Network-Based Eavesdropping. Confidentiality services should be enabled by the ULP if this threat is a concern. 👀
- Denial of Service
- 6.4.3.1, Multiple Streams Sharing Receive Buffers. ULP memory footprint size can be important for some server ULPs. If a server ULP is expecting significant network traffic from multiple clients, using a receive buffer queue per Stream where there is a large number of Streams can consume substantial amounts of memory. Thus, a receive queue that can be shared by multiple Streams is attractive.
However, because of the attacks outlined in this section, sharing a single receive queue between multiple clients must only be done if a mechanism is in place to ensure that one client cannot consume receive buffers in excess of its limits, as defined by each ULP. For multiple Streams within a single client ULP (which presumably shared Partial Mutual Trust), this added overhead may be avoided. 🌊
- 7.1 Local ULP Attacking a Shared CQ. The normative RNIC mitigations require that the RNIC not enable sharing of a CQ if the local ULPs do not share Partial Mutual Trust. Thus, while the ULP is not allowed to enable this feature in an unsafe mode, if the two local ULPs share Partial Mutual Trust, they must behave in the following manner:
- The sizing of the completion queue is based on the size of the receive queue and send queues, as documented in 6.4.3.2, Remote or Local Peer Attacking a Shared CQ.
- The local ULP ensures that CQ entries are reaped frequently enough to adhere to Section 6.4.3.2's rules. 🧃
- 6.4.3.2, Remote or Local Peer Attacking a Shared CQ. There are two mitigations specified in this section. One requires a worst-case size of the CQ, and can be implemented entirely within the Privileged Resource Manager. The second approach requires cooperation with the local ULP server (not to post too many buffers), and enables a smaller CQ to be used.
In some server environments, partial trust of the server ULP (but not the clients) is acceptable, so the smaller CQ fully mitigates the remote attacker. In other environments, the local server ULP could also contain untrusted elements that can attack the local machine (or have bugs). In those environments, the worst-case size of the CQ must be used. 📏
- 6.4.3.3, Attacking the RDMA Read Request Queue. The section requires a server's Privileged Resource Manager not to allow sharing of RDMA Read Request Queues across multiple Streams that do not share Partial Mutual Trust for a ULP that performs RDMA Read operations to server buffers. However, because the server ULP knows which of its Streams best share Partial Mutual Trust, this requirement can be reflected back to the ULP. The ULP (i.e., server) requirement, in this case, is that it MUST NOT allow RDMA Read Request Queues to be shared between ULPs that do not have Partial Mutual Trust. 🚫
- 6.4.5, Remote Invalidate an STag Shared on Multiple Streams. This mitigation is already covered by Section 6.2.2 (above).
Appendix B: Summary of RNIC and ULP Implementation Requirements 🍓
This appendix is informative. 💜
Below is a summary of implementation requirements for the RNIC:
- 3 Trust and Resource Sharing
- 5.4.5 Requirements for IPsec Encapsulation of DDP
- 6.1.1 Using an STag on a Different Stream
- 6.2.1 Buffer Overrun - RDMA Write or Read Response
- 6.2.2 Modifying a Buffer after Indication
- 6.4.1 RNIC Resource Consumption
- 6.4.3.1 Multiple Streams Sharing Receive Buffers
- 6.4.3.2 Remote or Local Peer Attacking a Shared CQ
- 6.4.3.3 Attacking the RDMA Read Request Queue
- 6.4.6 Remote Peer Attacking an Unshared CQ
- 6.5 Elevation of Privilege 39
- 7.1 Local ULP Attacking a Shared CQ
- 7.3 Local ULP Attacking the PTT and STag Mapping
Below is a summary of implementation requirements for the ULP above the RNIC:
- 5.3 Information Disclosure - Network-Based Eavesdropping
- 6.1.1 Using an STag on a Different Stream
- 6.2.2 Modifying a Buffer after Indication
- 6.3.2 Using RDMA Read to Access Stale Data
- 6.3.3 Accessing a Buffer after the Transfer
- 6.3.4 Accessing Unintended Data with a Valid STag
- 6.3.5 RDMA Read into an RDMA Write Buffer
- 6.3.6 Using Multiple STags That Alias to the Same Buffer
- 6.4.5 Remote Invalidate an STag Shared on Multiple Streams
Appendix C: Partial Trust Taxonomy 🦋
This appendix is informative.
okay so Partial Trust is the vibe where one party is willing to assume another party will skip a specific attack or set of attacks. When that's true, the two are in a state of Partial Trust 🫶. Just know the partially trusted peer may still try a different set of attacks 👀. For a lot of ULPs that's fine, because if the betrayal happens, the damage is easily contained and doesn't put other clients or ULPs at risk 💅.
The Trust Models in this section have three primary distinguishing characteristics 🌈. The Trust Model is about a local ULP and a Remote Peer, which are meant to be the local and remote ULP instances talking to each other over RDMA/DDP ✨.
- Local Resource Sharing (yes/no): when local resources are shared, they're shared across a grouping of RDMAP/DDP Streams 🍓. If they're not shared, resources are dedicated on a per Stream basis. (Resources are defined in Section 2.2, Resources.) The upside of not sharing is that fewer attacks are possible 🛡️. The downside is that ULPs might run out of resources 😭.
- Local Partial Trust (yes/no): this one is decided by whether the local grouping of RDMAP/DDP Streams (which usually means one ULP or a group of ULPs) mutually trusts each other not to perform a specific set of attacks 🧚.
- Remote Partial Trust (yes/no): this level depends on whether the local ULP of a specific RDMAP/DDP Stream partially trusts the Remote Peer of that Stream (see the definition of Partial Trust in Section 1, Introduction) 💜.
Not every combination of these characteristics is expected to show up in practice 🙈. This document specifically looks at five ULP Trust Models that are expected to be in common use. Here they are, bestie:
- NS-NT: Non-Shared Local Resources, no Local Trust, no Remote Trust. This is typically a server ULP that wants to run in the safest mode possible 🔒. All attack mitigations are in place so it stays robust ✨.
- NS-RT: Non-Shared Local Resources, no Local Trust, Remote Partial Trust. This is typically a peer-to-peer ULP that has, by some method outside the scope of this document, authenticated the Remote Peer 🤝. Heads up: unless some form of key based authentication is used on a per RDMA/DDP Stream basis, man-in-the-middle attacks may not be possible to pull off 🕵️.
- S-NT: Shared Local Resources, no Local Trust, no Remote Trust. This is typically a server ULP running in an untrusted environment where the resources needed are either too large or too dynamic to dedicate to each RDMAP/DDP Stream 🌊.
- S-LT: Shared Local Resources, Local Partial Trust, no Remote Trust. This is typically a ULP that provides a session layer and uses multiple Streams to provide extra throughput or fail-over capabilities 🔁. All the Streams within the local ULP partially trust each other, but they don't trust the Remote Peer. This model may be appropriate for embedded environments 🧩.
- S-T: Shared Local Resources, Local Partial Trust, Remote Partial Trust. This is typically a distributed application, like a distributed database or a High Performance Computer (HPC) application, meant to run on a cluster 🖥️. Because the resource and performance requirements are extreme, the application usually authenticates with all of its peers and then runs in a highly trusted environment 👑. The application peers are all in a single application fault domain and depend on each other to behave well when accessing data structures. If a trusted Remote Peer has an implementation defect that causes poor behavior, the entire application could be corrupted 😱.
NS-NT and S-NT are the typical ones for Internet networking, since neither the local ULP nor the Remote Peer is trusted 🌍. Sometimes optimizations can be done that let multiple local ULPs share Page Translation Tables, so S-LT can be a real advantage 🎀. S-T is typically used when resource scaling across a large parallel ULP makes every other model infeasible 🚀. Those scaling issues can come from performance problems at scale, or simply from not having enough resources 📉. NS-RT is probably the least likely model to be used, but it's included for completeness 🥲.
Acknowledgments 🫶
Here are the wonderful humans who helped make this happen, and their contact details exactly as written:
Sara Bitan
Microsoft Corporation
EMail: sarab@microsoft.com
Allyn Romanow
Cisco Systems
170 W Tasman Drive
San Jose, CA 95134 USA
Phone: +1 (408) 525-8836
EMail: allyn@cisco.com
Catherine Meadows
Naval Research Laboratory
Code 5543
Washington, DC 20375 USA
EMail: meadows@itd.nrl.navy.mil
Patricia Thaler
Agilent Technologies, Inc.
1101 Creekside Ridge Drive, #100
M/S-RG10
Roseville, CA 95678 USA
Phone: +1 (916) 788-5662
EMail: pat_thaler@agilent.com
James Livingston
NEC Solutions (America), Inc.
7525 166th Ave. N.E., Suite D210
Redmond, WA 98052-7811 USA
Phone: +1 (425) 897-2033
EMail: james.livingston@necsam.com
John Carrier
Cray Inc.
411 First Avenue S, Suite 600
Seattle, WA 98104-2860
Phone: 206-701-2090
EMail: carrier@cray.com
Caitlin Bestler
Broadcom
49 Discovery
Irvine, CA 92618
EMail: cait@asomi.com
Bernard Aboba
Microsoft Corporation
One Microsoft Way USA
Redmond, WA 98052
Phone: +1 (425) 706-6606
EMail: bernarda@windows.microsoft.comAuthors' Addresses 🎀
And the authors themselves, bestie 💖:
James Pinkerton
Microsoft Corporation
One Microsoft Way
Redmond, WA 98052 USA
Phone: +1 (425) 705-5442
EMail: jpink@windows.microsoft.com
Ellen Deleganes
Self
P.O. Box 9245
Brooks, OR 97305
Phone: (503) 642-3950
EMail: deleganes@yahoo.comFull Copyright Statement ⚖️
Copyright (C) The IETF Trust (2007).
This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.
The legal fine print, verbatim because it matters 📜:
This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property 🔍
The IETF takes no position on the validity or scope of any Intellectual Property Rights or other rights that might be claimed to apply to the implementation or use of the technology described in this document, or on whether any license under such rights might or might not be available. It also doesn't claim to have made any independent effort to identify such rights 🕵️. Info on the procedures for rights in RFC documents can be found in BCP 78 and BCP 79 📚.
Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses to be made available, or the results of an attempt to get a general license or permission to use these proprietary rights by implementers or users of this specification can be found in the IETF online IPR repository at:
http://www.ietf.org/iprThe IETF invites any interested party to bring to its attention any copyrights, patents, patent applications, or other proprietary rights that may cover technology required to implement this standard. Send that info to the IETF at:
ietf-ipr@ietf.orgThat's a wrap on RFC 5042 🌸✨ periodt 💅
📦 where this page comes from
- ietf: rfc5042 2007-10
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms