RFC 4296
The Architecture of Direct Data Placement (DDP) and Remote Direct Memory Access (RDMA) on Internet Protocols
RFC 4296 🦋 The Architecture of Direct Data Placement (DDP) and Remote Direct Memory Access (RDMA) on Internet Protocols
okay bestie, let's get the important facts out of the way first 💅
- Network Working Group 🌸
- Request for Comments: 4296
- Authors: S. Bailey (Sandburst) and T. Talpey (NetApp) 👯
- Category: Informational
- Date: December 2005
- Obsoletes / Updates: this one doesn't obsolete or update anything. It's a standalone doc ✨
Status of This Memo 🧾
okay so this memo is just here to give info to the Internet community 📚. It is NOT a standard of any kind, so nobody is required to follow it, lol. Distribution is totally unlimited, so share it with everyone 💜
Copyright Notice ©️
The legal bit: Copyright (C) The Internet Society (2005). 📜
Abstract 🌈
this doc defines an abstract architecture for Direct Data Placement (DDP) and Remote Direct Memory Access (RDMA) protocols that run on Internet Protocol-suite transports ✨. It's a descriptive tool, not a blueprint for how you should actually build these things, so you can understand the protocols without being locked into one implementation 🧚
DDP lets data get placed efficiently into buffers that Upper Layer Protocols (like RDMA) designate 🍓. RDMA gives you the semantics for Remote Direct Memory Access between peers, in a way that matches what applications actually need 🔥
Table of Contents 📑
1. Introduction ....................................................2
1.1. Terminology ................................................2
1.2. DDP and RDMA Protocols .....................................3
2. Architecture ....................................................4
2.1. Direct Data Placement (DDP) Protocol Architecture ..........4
2.1.1. Transport Operations ................................6
2.1.2. DDP Operations ......................................7
2.1.3. Transport Characteristics in DDP ...................10
2.2. Remote Direct Memory Access (RDMA) Protocol Architecture ..12
2.2.1. RDMA Operations ....................................14
2.2.2. Transport Characteristics in RDMA ..................16
3. Security Considerations ........................................17
3.1. Security Services .........................................18
3.2. Error Considerations ......................................19
4. Acknowledgements ...............................................19
5. Informative References .........................................201. Introduction 🎀
so the intro says it again: this doc defines that same abstract architecture for DDP and RDMA running on Internet Protocol-suite transports 🦋. It's a descriptive tool, not the one true way to implement it 💅
quick heads-up: the doc uses C language notation as a shorthand to describe the architectural pieces of DDP and RDMA. But that's just for describing things. It does NOT mean these are concrete protocols or programming interfaces, so don't get it twisted 🙈
the doc is split into two parts. Part one covers the DDP architecture, including what it assumes about the transports it's built on 🧩. Part two covers the RDMA architecture, which sits on top of DDP ✨
1.1. Terminology 📖
before we get into the protocols, you need these definitions bestie 🫶
- Placement is writing to a data buffer 🖊️
- Operation is a protocol message, or a sequence of messages, that gives you an architectural semantic. Think reading or writing a data buffer 📦
- Delivery is telling any Upper Layer or application that a message is ready to use. So delivery is basically the "control" signal for a unit of data 🚦. Also, the order of delivery is defined more strictly than the order of placement, so pay attention to that one 👀
- Completion is telling any Upper Layer or application that an operation has finished. A completion might need several messages to be delivered, or it might mean some local processing is done too 🏁
- Data Sink is the peer where placement happens 🪣
- Data Source is the peer the placed data comes from 🌊
- Steering Tag is a "handle" that identifies the buffer being targeted for placement 🏷️. A "tagged" message is one that references one of these handles 💖
- RDMA Write is an Operation that places data from a local data buffer into a remote data buffer, and that remote buffer is specified by a Steering Tag 🚀
- RDMA Read is an Operation that places data into a local data buffer (specified by a Steering Tag) from a remote data buffer (specified by a different Steering Tag) 📥
- Send is an Operation that places data from a local data buffer into a remote data buffer of the data sink's choice. That's why sends are "untagged" 🙃
1.2. DDP and RDMA Protocols 🧚
the goal of DDP is to allow efficient placement of data into buffers designated by the protocols layered above it, like RDMA. The doc covers this in detail in [ROM] 📚. Efficiency here basically means minimizing how many times the data gets transferred over the receiver's system buses 🚌
the goal of RDMA is to provide the semantics that enable Remote Direct Memory Access between peers in a way that's consistent with application requirements 💜. It offers facilities that are immediately useful to existing and future networking, storage, and other application protocols. The references for that are [FCVI, IB, MYR, SDP, SRVNET, VI] ✨
DDP and RDMA work together to hit their goals 👯. DDP gives you facilities to safely steer payloads to specific buffers at the Data Sink. RDMA gives Upper Layers the facilities to identify those buffers, control how data moves between peers' buffers, support authorized bidirectional transfer between buffers, and signal completion 🌟. And if an Upper Layer Protocol doesn't need RDMA's features, it can be layered directly on top of DDP 🫧
both DDP and RDMA are transport independent 🦋. Here's how RDMA, DDP, Upper Layer Protocols, and Transport relate to each other:
+--------------------------------------------------+
| Upper Layer Protocol |
+---------+------------+---------------------------+
| | | RDMA |
| | +---------------------------+
| | DDP |
| +----------------------------------------+
| Transport |
+--------------------------------------------------+Wait, I need to keep the diagram exact. Here it is, character for character:
+--------------------------------------------------+
| Upper Layer Protocol |
+---------+------------+---------------------------+
| | | RDMA |
| | +---------------------------+
| | DDP |
| +----------------------------------------+
| Transport |
+--------------------------------------------------+2. Architecture 🏗️
the Architecture section comes in two parts 🧁: the Direct Data Placement Protocol architecture and the Remote Direct Memory Access Protocol architecture ✨
2.1. Direct Data Placement (DDP) Protocol Architecture 💎
okay this is the core idea of general-purpose DDP, and it's iconic 🔥. A data sender adds placement information to the data it sends. That lets the receiver's network interface put the data directly at its final destination, with no copying needed 🙌. DDP can steer received data to its final destination without needing layer-specific behavior for each different layer. Data sent with this DDP information is called "tagged" 🏷️
the central components of DDP are the "buffer" and a method called set(). A buffer is an object with beginning and ending addresses 📦. set() sets the value of one octet at an address ✍️. A lot of the time a buffer maps straight onto a chunk of host user memory, but DDP doesn't require that. A buffer could be a disk file, or literally anything you can view as an addressable collection of octets 🗂️. Here's the abstract interface:
typedef struct {
const address_t start;
const address_t end;
void set(address_t a, data_t v);
} ddp_buffer_t;and the two types in there are:
address_t
a reference to local memory
data_t
an octet data value.the protocol layering and in-line data flow of DDP looks like this 🧩:
DDP Client Protocol
(e.g., RDMA or Upper Layer Protocol)
| ^
untagged messages | | untagged message delivery
tagged messages | | tagged message delivery
v |
DDP+---> data placement
^
| transport messages
v
Transport
(e.g., SCTP, DCCP, framed TCP)
^
| IP datagrams
v
. . .in addition to that in-line data flow, the client protocol registers buffers with DDP. DDP then does buffer update (set()) operations whenever it receives tagged messages 🪄
a DDP message can get split into multiple, smaller DDP messages, and each one rides in its own transport message 🧃. But if the transport is unreliable or unordered, messages split across transport messages may or may not be useful 😬. It's the same deal as splitting arbitrary Upper Layer messages across unreliable or unordered transport messages: they may or may not be useful. So the same considerations apply whether you're building client protocols on different transport types with or without DDP 🤷♀️
here's what a DDP message split across transport messages looks like:
DDP message: Transport messages:
stag=s, offset=o, message 1:
notify=y, id=i |type=ddp |
message= |stag=s |
|aabbccddee|-------. |offset=o |
~ ... ~----. \ |notify=n |
|vvwwxxyyzz|-. \ \ |id=? |
| \ `--->|aabbccddee|
| \ ~ ... ~
| +----->|iijjkkllmm|
| |
+ | message 2:
\ | |type=ddp |
\ | |stag=s |
\ + |offset=o+n|
\ \ |notify=y |
\ \ |id=i |
\ `-->|nnooppqqrr|
\ ~ ... ~
`---->|vvwwxxyyzz|last bit 💜: even though the picture makes it look like DDP info rides in-line with the message payload, parts of the DDP info can also live in transport-specific fields. They can even be derived from transport-specific control information, as long as the transport allows it 🌈
that's the first chunk, bestie! 🫶 more parts are coming, so stay tuned ✨
2.1.1. Transport Operations ✨
Okay so first, we need to know what the transport is giving DDP to work with 💅. For this architecture, the transport gives us three functions, and they're the foundation of everything else:
void xpt_send(socket_t s, message_t m);
message_t xpt_recv(socket_t s);
msize_t xpt_max_msize(socket_t s);Here's the cast of characters 🦋:
socket_t A transport address, including IP addresses, ports and other transport-specific identifiers. Basically, it's who you're talking to 💌
message_t A string of octets. Just bytes, no fancy wrapper 🌸
msize_t (scalar) A message size. Simple but iconic 👑
Now the verbs:
xpt_send(socket_t s, message_t m) means send a transport message. Literally just sending 📮
xpt_recv(socket_t s) means receive a transport message. The other side of the group chat 🫶
xpt_max_msize(socket_t s) means get the current maximum transport message size. This one is lowkey important ✨ It corresponds, roughly, to the current path Maximum Transfer Unit (PMTU), adjusted by underlying protocol overheads. So it's the biggest message you can fit through the pipe right now, minus the packaging 🎀
Real implementations ofxpt_send()andxpt_recv()typically return error indications, but that is not relevant to this architecture. So yes, errors happen IRL, but we're not worrying about them here 🙈
2.1.2. DDP Operations 🌈
Now the main event 💖 The DDP layer provides these operations:
void ddp_send(socket_t s, message_t m);
void ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d,
ddp_notify_t n);
void ddp_post_recv(socket_t s, bdesc_t b);
ddp_ind_t ddp_recv(socket_t s);
bdesc_t ddp_register(socket_t s, ddp_buffer_t b);
void ddp_deregister(bhand_t bh);
msizes_t ddp_max_msizes(socket_t s);Let's go through the types first, because they're the vocab for the whole thing 🧚
ddp_addr_t This is the buffer address portion of a tagged message. It's a struct with two parts:
typedef struct {
stag_t stag;
address_t offset;
} ddp_addr_t;stag_t (scalar) A Steering Tag. This is the main character for tagged messages 👑 A stag_t identifies the destination buffer. stag_ts are generated when the buffer is registered, communicated to the sender by some client protocol convention, and inserted in DDP messages. The vibe is that stag_t values are assumed to be completely opaque to the client protocol, and implementation-dependent 🙈 However, particular implementations, such as DDP on a multicast transport (see below), may give the buffer holder some control in selecting stag_ts. So sometimes you get a little say, but not always 💅
ddp_notify_t This is the notification portion of a DDP message. It's used to signal that the message represents the final fragment of a multi-segmented DDP message. Think of it as the "I'm done talking" bit 🔥
typedef struct {
boolean_t notify;
ddp_msg_id_t i;
} ddp_notify_t;ddp_msg_id_t (scalar) A DDP message identifier. msg_id_ts are chosen by the DDP message receiver (buffer holder), communicated to the sender by some client protocol convention, and inserted in DDP messages. Whether a message reception indication is requested for a DDP message is a matter of client protocol convention. Unlike stag_ts, the structure of msg_id_ts is opaque to DDP, so it is completely in the hands of the client protocol 🦋
bdesc_t A description of a registered buffer:
typedef struct {
bhand_t bh;
ddp_addr_t a;
} bdesc_t;a.offset' is the starting offset of the registered buffer, which may have no relationship to the start' or `end' addresses of that buffer. However, particular implementations, such as DDP on a multicast transport (see below), may allow some client protocol control over the starting offset 🌸
bhand_t An opaque buffer handle used to deregister a buffer. Your ticket out 🎟️
recv_message_t A description of a completed untagged receive buffer:
typedef struct {
bdesc_t b;
length_t l;
} recv_message_t;ddp_ind_t This is an untagged message, a tagged message reception indication, or a tagged message reception error. It's a union, so it's one of these three at a time 🌈
typedef union {
recv_message_t m;
ddp_msg_id_t i;
ddp_err_t e;
} ddp_ind_t;ddp_err_t This indicates an error while receiving a tagged message. Usually that's offset' out of bounds, or stag' not being registered to the socket. Yikes 😭
msizes_t The maximum untagged and tagged messages that fit in a single transport message:
typedef struct {
msize_t max_untagged;
msize_t max_tagged;
} msizes_t;Now the operations, the fun part 💅
ddp_send(socket_t s, message_t m) sends an untagged message. Simple and clean 📮
ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n) sends a tagged message to remote buffer address d. Main character energy 👑
ddp_post_recv(socket_t s, bdesc_t b) posts a registered buffer to accept a single received untagged message. Each buffer is returned to the caller in a ddp_recv() untagged message reception indication, in the order in which it was posted. The same buffer may be enabled on multiple sockets, but receipt of an untagged message into the buffer from any of these sockets unposts the buffer from all sockets. So it's one buffer, first come first served 🫣
ddp_recv(socket_t s) gets the next received untagged message, tagged message reception indication, or tagged message error ✨
ddp_register(socket_t s, ddp_buffer_t b) registers a buffer for DDP on a socket. The same buffer may be registered multiple times on the same or different sockets. The same buffer registered on different sockets may result in a common registration. Different buffers may also refer to portions of the same underlying addressable object (buffer aliasing). It's giving "many doors, one room" 🚪🌸
ddp_deregister(bhand_t bh) removes a registration from a buffer. Bye bye 👋
ddp_max_msizes(socket_t s) gets the current maximum untagged and tagged message sizes that will fit in a single transport message 🧚
2.1.3. Transport Characteristics in DDP 🦋
Okay bestie, this part is the big one 🫶 Certain characteristics of the transport that DDP gets mapped onto determine the nature of the service provided to client protocols. Fundamentally, the characteristics of the transport will not be changed by the presence of DDP. So the transport choice is driven not by DDP, but by the requirements of the Upper Layer, which is employing the DDP service. DDP is just a guest here 💅
Specifically, transports are:
- reliable or unreliable,
- ordered or unordered,
- single source or multisource,
- single destination or multidestination (multicast or anycast).
Some transports support several combinations of these characteristics. For example, SCTP [SCTP] is reliable, single source, single destination (point-to-point), and supports both ordered and unordered modes. Versatile queen 👑
DDP messages carried by transport are framed for processing by the receiver, and may be further protected for integrity or privacy in accordance with the transport capabilities. DDP does not provide such functions, so don't look for them here 🙈
In general, transport characteristics equally affect transport and DDP message delivery. However, there are several issues specific to DDP messages 🔥
A key component of DDP is how the following operations on the receiving side are ordered among themselves, and how they relate to corresponding operations on the sending side:
- set()s,
- untagged message reception indications, and
- tagged message reception indications.
These relationships depend upon the characteristics of the underlying transport in a way that is defined by the DDP protocol. For example, if the transport is unreliable and unordered, the DDP protocol might specify that the client protocol is subject to the consequences of transport messages being lost or duplicated, rather than requiring that different characteristics be presented to the client protocol. So the client protocol gets the messy reality instead of a fake clean one 🌈
Buffer access must be implemented consistently across endpoint IP addresses on transports allowing multiple IP addresses per endpoint, for example, SCTP. In particular, the Steering Tag must be consistently scoped and must address the same buffer across all IP address associations belonging to the endpoint. Additionally, operation ordering relationships across IP addresses within an association (set(), get(), etc.) depend on the underlying transport. If the above consistency relationships cannot be maintained by a transport endpoint, then the endpoint is unsuitable for a DDP connection. No consistency, no DDP, sorry not sorry 😭
Multidestination data delivery is a transport characteristic that may require specific consideration in a DDP protocol. As mentioned above, the basic DDP model assumes that buffer address values returned by ddp_register() are opaque to the client protocol, and can be implementation dependent. The most natural way to map DDP to a multidestination transport is to require that all receivers produce the same buffer address when registering a multidestination destination buffer. Restricting the DDP model to accommodate multiple destinations involves engineering tradeoffs comparable to those of providing non-DDP multidestination transport capability. It's a whole balancing act 🤹♀️
A registered buffer is identified within DDP by its stag_t, which in turn is associated with a socket. Therefore, this registration grants a capability to the DDP peer. The socket, using the underlying properties of its chosen transport and possible security, identifies the peer and authenticates the stag_t. So the socket is basically the bouncer 🧐💜
The same buffer may be enabled by ddp_post_recv() on multiple sockets. In this case, any ddp_recv() untagged message reception indication may be provided on a different socket from the one on which the buffer was posted. Such indications are not ordered among multiple DDP sockets. Chaotic but make it make sense 🌀
When multiple sockets reference an untagged message reception buffer, local interfaces are responsible for managing the mechanisms of allocating posted buffers to received untagged messages, the handling of received untagged messages when no buffer is available, and resource management among multiple sockets. Where underprovisioning of buffers on multiple sockets is allowed, mechanisms should be provided to manage buffer consumption on a per-socket or group-of-related-sockets basis. That's the housekeeping part, and yes, it matters 🧹
Architecturally, therefore, DDP is a flexible and general paradigm that may be applied to any variety of transports. Implementations of DDP may, however, adapt themselves to these differences in ways appropriate to each transport. In all cases, the layering of DDP must continue to express the transport's underlying characteristics. So DDP can flex, but it can never pretend the transport is something it isn't 💖✨
2.2. Remote Direct Memory Access (RDMA) Protocol Architecture ✨
Okay bestie, so RDMA takes DDP and levels it up with two main moves 💅
First, it adds the ability to read from buffers registered to a socket (RDMA Read). This lets a client protocol do arbitrary, bidirectional data movement without involving the remote client 🦋. When RDMA is implemented in hardware, that data movement can happen without involving the remote host CPU at all 😭
RDMA also specifies a transport-independent untagged message service (Send). It's super efficient to build in hardware AND convenient for client protocols 🫶
The RDMA architecture is patterned after the traditional model for device programming. The client requests an operation using Send-like actions (programmed I/O), the server does the necessary data transfers for that operation (DMA reads and writes), and then it tells the client when it's done 🎀. The programmed I/O+DMA model supports a high degree of concurrency and flexibility for both the client and server, even when operations have a wide range of intrinsic latencies 💜
RDMA is layered as a client protocol on top of DDP 🧚:
Client Protocol
| ^
Sends | | Send reception indications
RDMA Read Requests | | RDMA Read Completion indications
RDMA Writes | | RDMA Write Completion indications
v |
RDMA
| ^
untagged messages | | untagged message delivery
tagged messages | | tagged message delivery
v |
DDP+---> data placement
^
| transport messages
v
. . .In addition to in-line data flow, read (get()) and update (set()) operations are performed on buffers registered with RDMA, as a result of RDMA Read Requests and RDMA Writes respectively 🌈
An RDMA buffer' extends a DDP buffer with a get() operation that retrieves the value of the octet at address a'. Here's the vibe 👑:
typedef struct {
const address_t start;
const address_t end;
void set(address_t a, data_t v);
data_t get(address_t a);
} rdma_buffer_t;2.2.1. RDMA Operations 🌸
The RDMA layer provides these 🍓:
void rdma_send(socket_t s, message_t m);
void rdma_write(socket_t s, message_t m, ddp_addr_t d,
rdma_notify_t n);
void rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d);
void rdma_post_recv(socket_t s, bdesc_t b);
rdma_ind_t rdma_recv(socket_t s);
bdesc_t rdma_register(socket_t s, rdma_buffer_t b,
bmode_t mode);
void rdma_deregister(bhand_t bh);
msizes_t rdma_max_msizes(socket_t s);Okay so, for clarity these data transfer interfaces are synchronous, but rdma_read() and possibly rdma_send() (when Send flow control is in play) can take an arbitrary amount of time to complete ⏳. To express the full concurrency and interleaving of RDMA data transfer, these interfaces should also be reentrant. For example, a client protocol may do an rdma_send() while an rdma_read() operation is still in progress. Main character multitasking 💅
rdma_notify_t is the RDMA Write notification info. It's used to signal that the message is the final fragment of a multi-segmented RDMA message:
typedef struct {
boolean_t notify;
rdma_write_id_t i;
} rdma_notify_t;It's identical in function to ddp_notify_t, except that the type rdma_write_id_t may not be equivalent to ddp_msg_id_t 🙈
rdma_write_id_t (scalar) is an RDMA Write identifier 🔥
rdma_ind_t is either a Send message or an RDMA error:
typedef union {
recv_message_t m;
rdma_err_t e;
} rdma_ind_t;rdma_err_t is an RDMA protocol error indication. RDMA errors include buffer addressing errors corresponding to ddp_err_ts, and buffer protection violations (e.g., RDMA Writing a buffer that's only registered for reading) 💀
bmode_t is the buffer registration mode (permissions). Any combination of permitting RDMA Read (BMODE_READ) and RDMA Write (BMODE_WRITE) operations is allowed ✨
rdma_send(socket_t s, message_t m): sends a message, delivering it to the next untagged RDMA buffer at the remote peer 📮
rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n): an RDMA Write to remote buffer address d 🦋
rdma_read(socket_t s, ddp_addr_t s, length_t l, ddp_addr_t d): an RDMA Read of l octets from remote buffer address s to local buffer address d 📖
rdma_post_recv(socket_t s, bdesc_t b): posts a registered buffer to accept a single Send message, to be filled and returned in-order to a subsequent caller of rdma_recv(). As with DDP, buffers may be enabled on multiple sockets, in which case ordering guarantees are relaxed. Also as with DDP, local interfaces must manage the mechanisms of allocation and management of buffers posted to multiple sockets 🧺
rdma_recv(socket_t s): gets the next received Send message, RDMA Write completion identifier, or RDMA error 📬
rdma_register(socket_t s, rdma_buffer_t b, bmode_t mode): registers a buffer for RDMA on a socket (for read access, write access or both). As with DDP, the same buffer may be registered multiple times on the same or different sockets, and different buffers may refer to portions of the same underlying addressable object 🌷
rdma_deregister(bhand_t bh): removes a registration from a buffer 🧹
rdma_max_msizes(socket_t s): gets the current maximum Send (max_untagged) and RDMA Read or Write (max_tagged) operations that will fit in a single transport message. The values returned by rdma_max_msizes() are closely related to the values returned by ddp_max_msizes(), but may not be equal 🎯
2.2.2. Transport Characteristics in RDMA 🚀
As with DDP, RDMA can be used on transports with a variety of different characteristics that show up directly in the service RDMA provides. Also, as with DDP, the fundamental characteristics of the transport will not be changed by the presence of RDMA 🫶
Like DDP, an RDMA protocol must specify how these things are ordered:
set()s,get()s,- Send messages, and
- RDMA Read completions
...are ordered among themselves, and how they relate to corresponding operations on the remote peer(s). These relationships are likely to be a function of the underlying transport characteristics 🧠
There are some additional characteristics of RDMA that may translate poorly to unreliable or multipoint transports, due to attendant complexities in managing endpoint state:
- Send flow control
- RDMA Read
These difficulties can be overcome by placing restrictions on the service provided by RDMA 🛠️. However, many RDMA clients, especially those that separate data transfer and application logic concerns, are likely to depend upon capabilities only provided by RDMA on a point-to-point, reliable transport. In other words, many potential Upper Layers, which might avail themselves of RDMA services, are naturally already biased toward these transport classes 💖
3. Security Considerations 🔒
Okay bestie, security is NOT a vibe you skip. Read this part slowly 💀
Fundamentally, the DDP and RDMA protocols themselves should not introduce additional vulnerabilities. They are intermediate protocols, so they should not perform or require functions such as authorization, which are the domain of Upper Layers. However, the DDP and RDMA protocols should allow mapping by strict Upper Layers that are not permissive of new vulnerabilities. DDP and RDMAP implementations should be prohibited from cutting corners' that create new vulnerabilities. Implementations must ensure that only supplied' resources (i.e., buffers) can be manipulated by DDP or RDMAP messages 🛡️
System integrity must be maintained in any RDMA solution. Mechanisms must be specified to prevent RDMA or DDP operations from impairing system integrity. For example, threats can include potential buffer reuse or buffer overflow, and these are not merely a security issue. Even trusted peers must not be allowed to damage local integrity. Any DDP and RDMA protocol must address the issue of giving end-systems and applications the capabilities to offer protection from such compromises 🚨
Because a Steering Tag exports access to a buffer, one critical aspect of security is the scope of this access. It must be possible to individually control specific attributes of the access provided by a Steering Tag on the endpoint (socket) on which it was registered, including remote read access, remote write access, and others that might be identified. DDP and RDMA specifications must provide both implementation requirements relevant to this issue, and guidelines to assist implementors in making the appropriate design decisions 🔐
For example, it must not be possible for DDP to enable evasion of buffer consistency checks at the recipient. The DDP and RDMA specifications must allow the recipient to rely on its consistent buffer contents, by explicitly controlling peer access to buffer regions at appropriate times 🌸
The use of DDP and RDMA on a transport connection may interact with any security mechanism, and vice versa. For example, if the security mechanism is implemented above the transport layer, the DDP and RDMA headers may not be protected. Therefore, such a layering may be inappropriate, depending on requirements 🥺
3.1. Security Services 🔐
okay so the RDMA security checklist is lowercase-iconic 💅. Here's what end-to-end security has to cover for DDP and RDMAP operation streams:
- Authentication of the data source 🪪, so nobody can impersonate a peer, hijack a stream, or sneak in as a man-in-the-middle by abusing what the RDMA implementation offers.
Peer connections that don't pass authentication and authorization checks must not be permitted to begin processing in RDMA mode with an inappropriate endpoint. Once things are associated, peer accesses to buffer regions must be authenticated and made subject to authorization checks in the context of the association and endpoint (socket) they're going to hit, and this has to happen before any transfer or data access. The RDMA protocols must ensure that these region protections stay under strict application control. Non-negotiable, bestie 🙅♀️
- Integrity 🧷, to protect against modification of the control content and buffer content.
Integrity matters for any transport, sure, but it's especially important for DDP and RDMAP. The RDMA control information in each operation has to be protected, because that's what directs the payloads to the right place. 🎯
- Sequencing 🔁, to protect against replay attacks (which are a special case of the above modifications).
- Confidentiality 🤫, to protect the stream from eavesdropping.
IPsec, which secures the connection packet by packet, is a natural fit for securing RDMA placement, since RDMA works hand in hand with the transport 🤝. RDMA lets an implementation avoid buffering, so it's preferable to do all the applicable security protection before each segment gets processed by the transport and RDMA layers. That layering is what makes the most efficient secure RDMA implementation possible ✨.
The TLS record protocol is a different story 😬. It sits on top of reliable transports and can't give that same security assurance until an entire record is available, which might mean buffering and/or assembling several distinct messages before TLS can process anything. That delays RDMA processing and adds the exact overhead RDMA is built to dodge. On top of that, TLS length restrictions on records add extra buffering and processing for long operations that have to span multiple records. So TLS is seen as potentially a less natural fit for protecting the RDMA protocols.
Any DDP and RDMAP specification must provide the means to satisfy the above security service requirements 📜.
IPsec is sufficient to provide the required security services to the DDP and RDMAP protocols, while enabling efficient implementations 🏆.
3.2. Error Considerations 🚨
Resource issues that lead to denial-of-service attacks, overwrites and other concurrent operations, the ordering of completions that the RDMA protocol requires, and the granularity of transfer are all squarely inside the scope of any security analysis of RDMA and DDP 👀.
The RDMA operations check what is essentially user information. That explicitly includes addressing information and operation type (read or write), and implicitly includes protection and attributes 🧐. The semantics for each class of error that could come out of a failed check must be clearly defined, and the expected action the protocols take in each case must be specified. No vibes-based error handling here 🙅♀️.
Sometimes this results in a catastrophic error on the RDMA association 💥. In other cases, a local or remote error may be signalled instead. Some of these errors may require considering abstract local semantics 🧠. The result of the error on the RDMA association must be carefully specified so that it provides useful behavior, while not constraining the implementation 🦋.
4. Acknowledgements 🫶
The authors wish to acknowledge the valuable contributions of Caitlin Bestler, David Black, Jeff Mogul, and Allyn Romanow. Iconic contributors, all of them 👑.
5. Informative References 📚
[FCVI] ANSI Technical Committee T11, "Fibre Channel Standard Virtual Interface Architecture Mapping", ANSI/NCITS 357-2001, March 2001, available from http://www.t11.org/t11/stat.nsf/fcproj.
[IB] InfiniBand Trade Association, "InfiniBand Architecture Specification Volumes 1 and 2", Release 1.1, November 2002, available from http://www.infinibandta.org/specs.
[MYR] VMEbus International Trade Association, "Myrinet on VME Protocol Specification", ANSI/VITA 26-1998, August 1998, available from http://www.myri.com/open-specs.
[ROM] Romanow, A., Mogul, J., Talpey, T., and S. Bailey, "Remote Direct Memory Access (RDMA) over IP Problem Statement", RFC 4297, December 2005.
[SCTP] Stewart, R., Xie, Q., Morneault, K., Sharp, C., Schwarzbauer, H., Taylor, T., Rytina, I., Kalla, M., Zhang, L., and V. Paxson, "Stream Control Transmission Protocol", RFC 2960, October 2000.
[SDP] InfiniBand Trade Association, "Sockets Direct Protocol v1.0", Annex A of InfiniBand Architecture Specification Volume 1, Release 1.1, November 2002, available from http://www.infinibandta.org/specs.
[SRVNET] R. Horst, "TNet: A reliable system area network", IEEE Micro, pp. 37-45, February 1995.
[VI] D. Cameron and G. Regnier, "The Virtual Interface Architecture", ISBN 0971288704, Intel Press, April 2002, more info at http://www.intel.com/intelpress/via/.
Authors' Addresses 🖊️
Stephen Bailey
Sandburst Corporation
600 Federal Street
Andover, MA 01810 USA
USA
Phone: +1 978 689 1614
EMail: steph@sandburst.com
Tom Talpey
Network Appliance
1601 Trapelo Road
Waltham, MA 02451 USA
Phone: +1 781 768 5329
EMail: thomas.talpey@netapp.comFull Copyright Statement 🌈
Copyright (C) The Internet Society (2005).
This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.
This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property 💜
The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.
Acknowledgement 🍓
Funding for the RFC Editor function is currently provided by the Internet Society. 💖
📦 where this page comes from
- ietf: rfc4296 2005-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms