RFC 4297
Remote Direct Memory Access (RDMA) over IP Problem Statement
Remote Direct Memory Access (RDMA) over IP Problem Statement 🎀
Okay so bestie, this is RFC 4297, and it's giving informational energy. 💅 It's not a standard, so nobody is required to follow it. It's a problem statement, which means it explains why we need RDMA over IP in the first place.
Title block, the tea ☕:
- Number and title: RFC 4297, "Remote Direct Memory Access (RDMA) over IP Problem Statement" 🦋
- Category: Informational. It's a "here's what's going on" doc, not a rulebook. 🌸
- Date: December 2005 🍓
- Authors: A. Romanow (Cisco), J. Mogul (HP), T. Talpey (NetApp), and S. Bailey (Sandburst) 👑
- Obsoletes and updates: The title block doesn't list any, so this doc isn't replacing or updating anything. 🧚
The original has the same facts, just in boring standards-body columns. I'm keeping everything exactly right, so the numbers and sections are all still here. ✨
Status of This Memo 💜
This memo gives info to the Internet community. It is NOT a standard of any kind. Distribution is unlimited, so share it with everyone 🫶
Copyright Notice 🌈
Copyright (C) The Internet Society (2005).
Abstract 🔥
Okay, the I/O bottleneck is real and it's so annoying. When data moves through the end-system network I/O path at high speeds, the overhead is huge. That has kept Internet protocols from being used in interconnection networks (and honestly, the Internet itself) in places that need high bandwidth, low latency, and/or low overhead. 😭
This doc looks at that overhead and proposes an architectural, IP-based "copy avoidance" fix: Remote Direct Memory Access (RDMA). ✨
Table of Contents 🍓
1. Introduction ....................................................2
2. The High Cost of Data Movement Operations in Network I/O ........4
2.1. Copy avoidance improves processing overhead. ...............5
3. Memory bandwidth is the root cause of the problem. ..............6
4. High copy overhead is problematic for many key Internet
applications. ...................................................8
5. Copy Avoidance Techniques ......................................10
5.1. A Conceptual Framework: DDP and RDMA ......................11
6. Conclusions ....................................................12
7. Security Considerations ........................................12
8. Terminology ....................................................14
9. Acknowledgements ...............................................14
10. Informative References ........................................151. Introduction 🧚
Okay so this section is about the high host processing overhead you get when user data moves to and from the network interface at high speed. People call this the "I/O bottleneck" [CT90]. 🫠
The overhead we care about here is data movement, aka copying. So if you're copying all the time, your throughput can get capped by that copying. This is NOT the same thing as TCP offload, and this doc doesn't cover TCP offload at all. 🙅♀️
"High speed" just means the link is fast compared to the host's CPU and memory bandwidth. With today's computers, 1 Gbits/s and up counts as high speed. 🚀
Copy costs are mostly a problem for big systems. Small stuff like rack-mounted PCs and small workstations would also benefit from less copying, but their benefit will mainly show up in the next few years as they scale up their bandwidth. 💅 Right now, it's the big multiprocessor and cluster machines with high-bandwidth feeds that get hit hard. Think database servers, storage servers, transaction and e-commerce app servers, web serving, content distribution, video distribution, backups, data mining, decision support, and scientific computing. The whole server squad, basically. 🎀
These servers almost always handle lots of concurrent sessions (transport connections), and together those add up to > 1 Gbits/s of communication. 🔥 But the copying cost for a given load is the same whether it comes from a few sessions or a lot of them. Not me being shocked. 💀
The I/O bottleneck and the role of data movement have been studied for about the last 14 years in research and industry, and we borrow freely from those results. 📚 The bottleneck shows up every time a new networking tech jacks up line rates: 100 Mbits/s Fast Ethernet and Fibre Distributed Data Interface [FDDI], 155 Mbits/s Asynchronous Transfer Mode [ATM], and 1 Gbits/s Ethernet. In earlier speed jumps, enough memory bandwidth let people put off the I/O problem. That's no longer the case. 😭 The I/O problem is already a big deal at 1 Gbits/s, but 10 Gbits/s Ethernet is what really got industry and research super hyped [IB, VI, CGY01, Ma02, MAF+02]. ✨
Because end-host processing overhead is so high in current implementations, the TCP/IP stack isn't used for high-speed transfer. Instead, people built special-purpose network fabrics using a tech generally known as Remote Direct Memory Access (RDMA), and those are widely used now. 🦋 RDMA is a set of mechanisms that let the network adapter, under application control, steer data directly into and out of application buffers. Examples include Fibre Channel [FIBRE] for block storage, Virtual Interface Architecture [VI] for database clusters, and Infiniband [IB], Compaq Servernet [SRVNET], and Quadrics [QUAD] for System Area Networks. These link-level techs limit how far apart nodes can be and how many there can be, so the number of nodes can't get arbitrarily large. 🙈
This problem statement backs up the claim that in network I/O processing, high overhead comes from data movement (specifically copying), and that copy avoidance significantly cuts that overhead. It explains when and why the high overheads happen, why they're a problem, and which applications are hit hardest. 🌸
The doc then gets into why this matters for the Internet and Internet-based apps. Apps that store, manage, and distribute Internet information are a great fit for the copy avoidance solution. 💖 They'd dodge the high processing overheads, which removes the limit on scaling tiered end-systems. Copy avoidance also removes latency for these systems, which can help distributed processing even more. 👑
This doc also introduces an architectural approach to the problem, which is developed in detail in [BT05]. It talks about how the proposed tech might bring security concerns and how those should be handled. 🔥
Last thing: there's a Terminology section to help with the several new terms RDMA brings in. 📖
2. The High Cost of Data Movement Operations in Network I/O 💅
Lots of research and industry data shows that copying eats up a lot of processing overhead. Even in carefully built systems, getting rid of copies noticeably cuts that overhead, as referenced below. 🧚
Clark et al. [CJRS89] (1989) showed that TCP [Po81] overhead comes from two places: operating system costs (interrupts, context switches, process management, buffer management, timer management) and the costs of processing individual bytes (computing the checksum and moving data in memory). They found that moving data in memory is the bigger cost, and their experiments show memory bandwidth is the biggest limiting factor. 😭 In their data, 64% of the measured microsecond overhead came from data-touching operations, and 48% came from copying. The system they measured was Berkeley TCP on a Sun-3/60 using 1460 Byte Ethernet packets. 🎀
In a well-built system, copying can happen between the network interface and the kernel, and between the kernel and application buffers. That's two copies, each one a pair of memory bus crossings (one for read, one for write). Sometimes you can do better, but usually two copies are needed on receive. 🫶
Later work consistently showed the same thing Clark found. Several studies report that data-touching operations (checksumming and data movement) dominate processing costs for messages longer than 128 Bytes [BS96, CGY01, Ch96, CJRS89, DAPP93, KP96]. For smaller messages, per-packet overheads dominate [KP96, CGY01]. 💜
The share of overhead from data-touching operations goes up as packet size goes up, since per-byte work scales linearly with message size [KP96]. For example, Chu [Ch96] reported big per-byte latency costs as a percentage of total networking software costs for an MTU-size packet on a SPARCstation/20, running memory-to-memory TCP tests over networks with 3 different MTU sizes. The share of total software cost from per-byte operations was:
1500 Byte Ethernet 18-25%
4352 Byte FDDI 35-50%
9180 Byte ATM 55-65%Lots of studies report data-touching operations together (checksumming and data movement), but plenty of work has focused just on copying [BS96, Br99, Ch96, TK95]. For example, [KP96] reports processing times for checksum and data movement separately. For the 1500 Byte Ethernet size, 20% of total processing overhead time is from copying. The study used 2 DECstations 5000/200 connected by an FDDI network. (In that study, checksum was 30% of processing time.) 🌈
2.1. Copy avoidance improves processing overhead. 💅
okay so the receipts on this are wild 🧾✨ a bunch of studies show that cutting out copies makes things way faster. The IO-Lite system [PDZ99] was built to make web servers faster, and it got a 43% throughput increase over an optimized web server and a 137% improvement over an Apache server 😭 It was built in a 4.4BSD-derived UNIX kernel, and the test setup was a server running on a 333MHz Pentium II PC, hooked up to a switched 100 Mbits/s Fast Ethernet 🖥️
There are so many other examples where getting rid of copying made systems way better [CFF+94, DP93, EBBV95, KSZ95, TK95, Wa97]. We're gonna look at one of them in detail so you can see just how big the boost is 🔍 [Ch02] 🦋
Next up, Chase et al. [CGY01] measured CPU utilization and found that avoiding copies dropped the CPU time spent on data access from 24% to 15% at 370 Mbits/s. That was with a 32 KBytes MTU, an AlphaStation XP1000, and a Myrinet adapter [BCF+95] 🌸 That's an absolute improvement of 9% from copy avoidance alone.
The total CPU utilization was 35%, and data access was 24% of that. So the relative importance of reducing copies is 26% 🧮 At 370 Mbits/s the system isn't really stressed, though. The relative improvement in achievable bandwidth is 34%, and that's the gain you'd see if copy avoidance got added while the machine was saturated by network I/O 🔥
Here's the thing: the bigger the overhead you're targeting is, the more the optimization matters 💜 That's what happens when other overhead, like checksumming, gets removed too. In [CGY01], once checksum overhead was gone, copy avoidance took CPU utilization from 26% down to 10% 🎀 That's a 16% absolute reduction, a 61% relative reduction, and a 160% relative improvement in achievable bandwidth. Absolutely unhinged numbers, bestie 😭
Today's network interface hardware usually offloads the checksum, which takes out the other source of per-byte overhead ✨ It also coalesces interrupts to cut down per-packet costs. So copying now makes up a bigger slice of CPU utilization than it used to, which means there's more to gain by reducing copies 🍓 (Yes, this argument falls apart if the overhead is tiny, but it's been shown to be substantial. [BS96, Br99, Ch96, KP96, TK95] 💅)
3. Memory bandwidth is the root cause of the problem. 👑
Data movement operations are expensive because memory bandwidth is scarce compared to network bandwidth and CPU bandwidth [PAC+97] 😤 This trend existed in the past, it's expected to keep going into the future [HP97, STREAM], and it's especially a thing in large multiprocessor systems 🧠
With copies crossing the bus twice per copy, network processing overhead gets high whenever network bandwidth is big compared to CPU and memory bandwidth 🚌 With today's end-systems, these effects show up at network speeds over 1 Gbits/s. Honestly it's giving bottleneck: with multiple bus crossings, the bus bandwidth can become the limiting factor for throughput. That stops an end-system from getting full network bandwidth and full application performance at the same time 💀
Quick question people always ask: does a faster CPU fix the high processing costs of network I/O? The answer is no. It's the memory bandwidth that's the actual problem 🙅♀️ Faster CPUs don't help if the CPU is spending most of its time waiting on memory [CGY01].
The gap between microprocessor performance and memory performance has been a widely recognized, well-understood problem for ages [PAC+97] 🥀 Hennessy [HP97] shows microprocessor performance grew at 60% per year from 1980-1998, while DRAM access time only improved at 10% per year. That gave us a growing "processor-memory performance gap" 📉
Another good source is the STREAM Benchmark Reference Information website, which covers the STREAM benchmark [STREAM] 📊 It's a simple synthetic benchmark program that measures sustainable memory bandwidth (in MBytes/s) plus the matching computation rate for simple vector kernels, measured in MFLOPS. The site tracks sustainable memory bandwidth for hundreds of machines from all the major vendors 🌈
The measured results: processing performance grew 50% per year on average from 1985-2001, and sustainable memory bandwidth grew 35% per year on average from 1975 to 2001, across all the systems measured ⭐ A similar 15% per year lead of processing bandwidth over memory bandwidth shows up in another stat, machine balance [Mc95]. That's a measure of the relative rate of CPU to memory bandwidth, written as (FLOPS/cycle) / (sustained memory ops/cycle) [STREAM] 🧚
Network bandwidth has been growing about 10-fold roughly every 8 years, which works out to a 40% per year growth rate 🚀
Here's a typical example of memory bandwidth looking really bad next to link speed 🙈 The STREAM benchmark shows that a modern uniprocessor PC, like the 1.2 GHz Athlon in 2001, moves the data 3 times during a receive operation: once for the network interface to deposit the data in memory, and twice for the CPU to copy it. With 1 GBytes/s of memory bandwidth (meaning one read or one write), the machine could handle about 2.67 Gbits/s of network bandwidth, which is one third of the copy bandwidth. But that assumes 100% utilization, which isn't possible, and more importantly the machine would be totally consumed! 😱 (A rule of thumb for databases is that 20% of the machine should be needed to service I/O, leaving 80% for the database application. And the less, the better. 🫶)
In 2001, 1 Gbits/s links were common 🔌 An application server might typically have two 1 Gbits/s connections: one backend connection to a storage server and one front-end, say for serving HTTP [FGM+99]. So the communications could use 2 Gbits/s. In our typical example, the machine could handle 2.7 Gbits/s at its theoretical maximum while doing nothing else. Which means the machine basically couldn't keep up with communication demands in 2001 🥺 And with the relative growth trends, the situation only gets worse 💔
4. High copy overhead is problematic for many key Internet applications ✨
okay so bestie, if a big chunk of a machine's resources gets eaten up by network I/O instead of actual application processing 😭, the app literally can't scale. It struggles to handle more clients or offer more services. not it 💅
Several years ago, the apps hit hardest were streaming multimedia, parallel file systems, and supercomputing on clusters [BS96]. Today it's giving even more central, since these apps store, manage, and distribute the Internet's and the enterprise's information 🌐. That includes database apps doing transaction processing, e-commerce, web serving, decision support, content distribution, video distribution, and backups. Clusters are usually used for this kind of app because they come with availability and scalability perks ✨.
These apps, which provide and manage Internet and corporate information, usually run in data centers organized into three logical tiers 🏢. Tier one is typically a set of web servers connecting to the WAN. Tier two is a set of application servers that run the specific apps, usually on more powerful machines. Tier three is the backend databases. Physically, the first two tiers (web server and application server) are usually combined [Pi01]. So an e-commerce server talks to a database server and a customer site, a content distribution server connects to a server farm, or an OLTP server connects to a database and a customer site.
When network I/O uses too much memory bandwidth, performance on network paths between tiers can suffer 📉. (Storage Area Network paths used by the database tier or the application tier can also have performance issues.) The high overhead from network-related memory copies pulls system resources away from other application processing, and it can create bottlenecks that limit total system performance 🚧.
There's a lot of motivation to max out the processing capacity of each CPU, because scaling by adding CPUs has drawbacks one way or another 🤔. For example, adding CPUs to a multiprocessor won't necessarily help, because a multiprocessor only improves performance when the memory bus has extra bandwidth to spare. Clustering can also add complexity to handling the apps.
To scale a cluster or multiprocessor, you have to proportionately scale the interconnect bandwidth 🔗. Interconnect bandwidth governs the performance of communication-intensive parallel apps. If it's too low (often talked about in terms of "bisection bandwidth"), adding more processors can't improve system throughput. Interconnect latency can also limit the performance of apps that share data between processors a lot.
So, excessive overhead on network paths in a "scalable" system can both force you to use more processors than optimal and shrink the marginal utility of those extra processors 🙃.
Copy avoidance scales a machine upwards by removing at least two-thirds of the bus bandwidth load from the "very best" 1-copy (on receive) implementations, and it removes at least 80% of the bandwidth overhead from the 2-copy implementations 🔥.
Removing bus bandwidth requirements removes bottlenecks from the network processing path and increases the machine's throughput. On a machine with limited bus bandwidth, the advantage of removing this load is immediately obvious, because the host can attain full network bandwidth. Even on a machine with enough bus bandwidth to sustain full network bandwidth, removing that load increases the machine's availability for user applications, sometimes dramatically 🦋.
An example showing poor performance with copies and better scaling with copy avoidance is super illustrative 🎀. The IO-Lite work [PDZ99] shows higher server throughput servicing more clients using a zero-copy system. In an experiment designed to mimic real-world web conditions by simulating the effect of TCP WAN connections on the server, they compared 3 servers. One was Apache, another was an optimized server called Flash, and the third was the Flash server running IO-Lite, called Flash-Lite with zero copy. They measured throughput in requests/second as a function of the number of slow background clients that could be served. As the table shows, Flash-Lite has better throughput, especially as the number of clients goes up 📊.
Apache Flash Flash-Lite
------ ----- ----------
#Clients Throughput reqs/s Throughput Throughput
0 520 610 890
16 390 490 890
32 360 490 850
64 360 490 890
128 310 450 880
256 310 440 820Traditional Web servers (which mostly send data and can keep most of their content in the file cache) are not the worst case for copy overhead 🙅♀️. Web proxies (which often receive as much data as they send) and complex Web servers based on System Area Networks or multi-tier systems will suffer more from copy overheads than in the example above 💀.
5. Copy Avoidance Techniques 🧚
There has been extensive research and industry experience with two main alternative approaches to eliminating data movement overhead, often alongside improving other Operating System processing costs 📚. In one approach, hardware and/or software changes within a single host reduce processing costs. In the other, memory-to-memory networking [MAF+02] lets hosts reduce processing costs by exchanging explicit data placement information between each other 🤝.
The single-host approaches range from new hardware and software architectures [KSZ95, Wa97, DWB+93] to new or modified software systems [BS96, Ch96, TK95, DP93, PDZ99]. In the approach based on using a networking protocol to exchange information, the network adapter, under control of the application, places data directly into and out of application buffers, which reduces the need for data movement. This is commonly called RDMA, Remote Direct Memory Access 🌸.
Research and industry experience has shown that copy avoidance techniques within the receiver processing path alone have proven to be problematic 😬. The research special-purpose host adapter systems had good performance and can be seen as precursors to the commercial RDMA-based adapters [KSZ95, DWB+93]. In software, many implementations have successfully achieved zero-copy transmit, but few have managed zero-copy receive. And the ones that did make strict alignment and no-touch requirements on the application, which greatly reduces the portability and usefulness of the implementation.
In contrast, experience has proven satisfactory with memory-to-memory systems that permit RDMA. Performance has been good, and there haven't been system or networking difficulties 🙌. RDMA is a single solution. Once it's implemented, it can be used with any OS and machine architecture, and it doesn't need to be revised when either of those changes.
In early work, one goal of the software approaches was to show that TCP could go faster with appropriate OS support [CJRS89, CFF+94]. That goal was achieved, but further investigation and experience showed that, though it's possible to craft software solutions, specific system optimizations have been complex, fragile, extremely interdependent with other system parameters in complicated ways, and often of only marginal improvement [CFF+94, CGY01, Ch96, DAPP93, KSZ95, PDZ99]. The network I/O system interacts with other aspects of the Operating System, such as machine architecture, file I/O, and disk I/O [Br99, Ch96, DP93] 🥴.
For example, the Solaris Zero-Copy TCP work [Ch96], which relies on page remapping, shows that the results are highly interdependent with other systems, such as the file system. It also shows that the particular optimizations are specific to particular architectures, which means that for each variation in architecture, the optimizations have to be re-crafted [Ch96] 🛠️.
With RDMA, application I/O buffers are mapped directly, and the authorized peer can access them without incurring additional processing overhead 🎯. When RDMA is implemented in hardware, arbitrary data movement can be performed without involving the host CPU at all 🤯.
A bunch of research projects and industry products have been based on the memory-to-memory approach to copy avoidance. These include U-Net [EBBV95], SHRIMP [BLA+94], Hamlyn [BJM+96], Infiniband [IB], and Winsock Direct [Pi01]. Several memory-to-memory systems have been widely used and have generally been found to be robust, to have good performance, and to be relatively simple to implement. These include VI [VI], Myrinet [BCF+95], Quadrics [QUAD], and Compaq/Tandem Servernet [SRVNET]. Networks based on these memory-to-memory architectures have been used widely in scientific applications and in data centers for block storage, file system access, and transaction processing 🏆.
By exporting direct memory access "across the wire", applications can direct the network stack to manage all data directly from application buffers 💫. A large and growing class of applications already takes advantage of these capabilities. It includes all the major databases, as well as network protocols such as Sockets Direct [SDP] ✨.
5.1. A Conceptual Framework: DDP and RDMA
An RDMA solution can be usefully viewed as two distinct components: "direct data placement (DDP)" and "remote direct memory access (RDMA) semantics" 🧩. They're distinct in purpose and in practice, and they may be implemented as separate protocols.
The more fundamental of the two is the direct data placement facility 🫶. This is the means by which memory is exposed to the remote peer in an appropriate fashion, and the means by which the peer can access it, for instance by reading and writing.
The RDMA control functions are semantically layered on top of direct data placement 🍰. They include operations that provide "control" features, such as connection and termination, the ordering of operations, and signaling their completions. A "send" facility is also provided.
While the functions (and potentially the protocols) are distinct, historically both aspects taken together have been referred to as "RDMA". The facilities of direct data placement are useful in and of themselves, and other upper-layer protocols may use them to facilitate data transfer. So it's often useful to refer to DDP as the data placement functionality and RDMA as the control aspect 🌈.
[BT05] develops an architecture for DDP and RDMA atop the Internet Protocol Suite, and it's a companion document to this problem statement 💜.
6. Conclusions ✨
okay so the big takeaway 💅 this Problem Statement wraps up by saying an IP-based, general solution for cutting down processing overhead on end-hosts is desirable 🎀
it shows that when processing network data gets super expensive, end-hosts become bottlenecks 😭 and honestly the copying of data is a huge reason why. machine bus bandwidth has historically been limited, and high-speed interconnects put a LOT of strain on it 💀
so an architectural solution is the best way to deal with this 🦋 and since today's interconnects are so fast and these hosts live on Internet Protocol-based networks, it makes sense to layer that solution on top of the Internet Protocol Suite 🌸 the architecture described in [BT05] is exactly that kind of proposal, iykyk 🧚
7. Security Considerations 🔒
bestie, this section is serious so pay attention 🙈
solutions for reducing copying overhead in high bandwidth transfers may introduce new security concerns. any proposed solution must be analyzed for security vulnerabilities, and any vulnerabilities found must be addressed. potential weaknesses need to be examined, described, and properly resolved. that includes resource issues that might lead to denial-of-service attacks 💀, overwrites and other concurrent operations, the ordering of completions as required by the RDMA protocol, the granularity of transfer, and any other vulnerabilities that get identified ✨
layered on top of Internet transport protocols, the RDMA protocols will gain leverage from Internet security standards like IPsec and TLS [IPSEC, TLS] and must permit integration with them 🫶 however, certain security approaches may have implementation ramifications for RDMA because of its copy avoidance 😬
IPsec secures the connection on a packet-by-packet basis, and it seems like a natural fit for securing RDMA placement, since RDMA works together with transport 🦋 because RDMA lets an implementation avoid buffering, it is preferable to do all applicable security protection before each segment gets processed by the transport and RDMA layers 👑 that layering enables the most efficient secure RDMA implementation, periodt 💅
the TLS record protocol is layered on top of reliable transports, and it can't provide that same security assurance until an entire record is available 😭 that might mean buffering and/or assembling several distinct messages before TLS processing even happens. this delays RDMA processing and adds the exact overheads RDMA is designed to avoid 🙈 so TLS is viewed as potentially a less natural fit for protecting the RDMA protocols 🌈
it is necessary to guarantee properties like confidentiality, integrity, and authentication on an RDMA communications channel ✨ however, those properties can't defend against every attack from properly authenticated peers, who might be malicious, compromised, or buggy 💀 so the RDMA design must address protection against such attacks. for example, an RDMA peer should not be able to read or write memory regions without prior consent 🛡️
and it must not be possible to evade memory consistency checks at the recipient 😤 the RDMA design must let the recipient rely on its consistent memory contents by explicitly controlling peer access to memory regions at appropriate times. main character energy for the recipient, honestly 👑
peer connections that don't pass authentication and authorization checks by upper layers must not be allowed to begin processing in RDMA mode with an inappropriate endpoint 🚫 once associated, peer accesses to memory regions must be authenticated and subject to authorization checks in the context of the association and connection they're performed on, before any transfer operation or data access happens 🔐
finally, the RDMA protocols must ensure that these region protections are under strict application control ✨ remote access to local memory by a network peer matters a LOT in the Internet context, because that access can be exported globally 🌍 not me getting chills rn 🥹
8. Terminology 📚
this section has the general terminology definitions for this document and for Remote Direct Memory Access in general 🎀
Remote Direct Memory Access (RDMA)
a method of accessing memory on a remote system where the local system specifies the location of the data to be transferred 🦋
RDMA Protocol
a protocol that supports RDMA Operations to transfer data between systems 🌸
Fabric
the collection of links, switches, and routers that connect a set of systems 🕸️
Storage Area Network (SAN)
a network where disks, tapes, and other storage devices are made available to one or more end-systems via a fabric 💾
System Area Network
a network where clustered systems share services, such as storage and interprocess communication, via a fabric 🧩
Fibre Channel (FC)
an ANSI standard link layer with associated protocols, typically used to implement Storage Area Networks. [FIBRE] ⭐
Virtual Interface Architecture (VI, VIA)
an RDMA interface definition developed by an industry group and implemented with a variety of differing wire protocols. [VI] 💜
Infiniband (IB)
an RDMA interface, protocol suite and link layer specification defined by an industry trade association. [IB] 🍓
9. Acknowledgements 🫶
Jeff Chase generously provided many useful insights and information 🧠✨ and thanks to Jim Pinkerton for many helpful discussions 💖
10. Informative References ✨
okay bestie, this is the reading list 📚🫶 every single paper and spec the authors leaned on, so if you're ever wondering where a claim came from, it's probably in here. Copied exactly, typos and all 💅
[ATM] The ATM Forum, "Asynchronous Transfer Mode Physical Layer
Specification" af-phy-0015.000, etc. available from
http://www.atmforum.com/standards/approved.html.
[BCF+95] N. J. Boden, D. Cohen, R. E. Felderman, A. E. Kulawik, C.
L. Seitz, J. N. Seizovic, and W. Su. "Myrinet - A
gigabit-per-second local-area network", IEEE Micro,
February 1995.
[BJM+96] G. Buzzard, D. Jacobson, M. Mackey, S. Marovich, J.
Wilkes, "An implementation of the Hamlyn send-managed
interface architecture", in Proceedings of the Second
Symposium on Operating Systems Design and Implementation,
USENIX Assoc., October 1996.
[BLA+94] M. A. Blumrich, K. Li, R. Alpert, C. Dubnicki, E. W.
Felten, "A virtual memory mapped network interface for the
SHRIMP multicomputer", in Proceedings of the 21st Annual
Symposium on Computer Architecture, April 1994, pp. 142-
153.
[Br99] J. C. Brustoloni, "Interoperation of copy avoidance in
network and file I/O", Proceedings of IEEE Infocom, 1999,
pp. 534-542.
[BS96] J. C. Brustoloni, P. Steenkiste, "Effects of buffering
semantics on I/O performance", Proceedings OSDI'96,
USENIX, Seattle, WA October 1996, pp. 277-291.
[BT05] Bailey, S. and T. Talpey, "The Architecture of Direct Data
Placement (DDP) And Remote Direct Memory Access (RDMA) On
Internet Protocols", RFC 4296, December 2005.
[CFF+94] C-H Chang, D. Flower, J. Forecast, H. Gray, B. Hawe, A.
Nadkarni, K. K. Ramakrishnan, U. Shikarpur, K. Wilde,
"High-performance TCP/IP and UDP/IP networking in DEC
OSF/1 for Alpha AXP", Proceedings of the 3rd IEEE
Symposium on High Performance Distributed Computing,
August 1994, pp. 36-42.
[CGY01] J. S. Chase, A. J. Gallatin, and K. G. Yocum, "End system
optimizations for high-speed TCP", IEEE Communications
Magazine, Volume: 39, Issue: 4 , April 2001, pp 68-74.
http://www.cs.duke.edu/ari/publications/end-
system.{ps,pdf}.
[Ch96] H.K. Chu, "Zero-copy TCP in Solaris", Proc. of the USENIX
1996 Annual Technical Conference, San Diego, CA, January
1996.
[Ch02] Jeffrey Chase, Personal communication.
[CJRS89] D. D. Clark, V. Jacobson, J. Romkey, H. Salwen, "An
analysis of TCP processing overhead", IEEE Communications
Magazine, volume: 27, Issue: 6, June 1989, pp 23-29.
[CT90] D. D. Clark, D. Tennenhouse, "Architectural considerations
for a new generation of protocols", Proceedings of the ACM
SIGCOMM Conference, 1990.
[DAPP93] P. Druschel, M. B. Abbott, M. A. Pagels, L. L. Peterson,
"Network subsystem design", IEEE Network, July 1993, pp.
8-17.
[DP93] P. Druschel, L. L. Peterson, "Fbufs: a high-bandwidth
cross-domain transfer facility", Proceedings of the 14th
ACM Symposium of Operating Systems Principles, December
1993.
[DWB+93] C. Dalton, G. Watson, D. Banks, C. Calamvokis, A. Edwards,
J. Lumley, "Afterburner: architectural support for high-
performance protocols", Technical Report, HP Laboratories
Bristol, HPL-93-46, July 1993.
[EBBV95] T. von Eicken, A. Basu, V. Buch, and W. Vogels, "U-Net: A
user-level network interface for parallel and distributed
computing", Proc. of the 15th ACM Symposium on Operating
Systems Principles, Copper Mountain, Colorado, December
3-6, 1995.
[FDDI] International Standards Organization, "Fibre Distributed
Data Interface", ISO/IEC 9314, committee drafts available
from http://www.iso.org.
[FGM+99] Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,
Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext
Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.
[FIBRE] ANSI Technical Committee T10, "Fibre Channel Protocol
(FCP)" (and as revised and updated), ANSI X3.269:1996
[R2001], committee draft available from
http://www.t10.org/drafts.htm#FibreChannel
[HP97] J. L. Hennessy, D. A. Patterson, Computer Organization and
Design, 2nd Edition, San Francisco: Morgan Kaufmann
Publishers, 1997.
[IB] InfiniBand Trade Association, "InfiniBand Architecture
Specification, Volumes 1 and 2", Release 1.1, November
2002, available from http://www.infinibandta.org/specs.
[IPSEC] Kent, S. and R. Atkinson, "Security Architecture for the
Internet Protocol", RFC 2401, November 1998.
[KP96] J. Kay, J. Pasquale, "Profiling and reducing processing
overheads in TCP/IP", IEEE/ACM Transactions on Networking,
Vol 4, No. 6, pp.817-828, December 1996.
[KSZ95] K. Kleinpaste, P. Steenkiste, B. Zill, "Software support
for outboard buffering and checksumming", SIGCOMM'95.
[Ma02] K. Magoutis, "Design and Implementation of a Direct Access
File System (DAFS) Kernel Server for FreeBSD", in
Proceedings of USENIX BSDCon 2002 Conference, San
Francisco, CA, February 11-14, 2002.
[MAF+02] K. Magoutis, S. Addetia, A. Fedorova, M. I. Seltzer, J.
S. Chase, D. Gallatin, R. Kisley, R. Wickremesinghe, E.
Gabber, "Structure and Performance of the Direct Access
File System (DAFS)", in Proceedings of the 2002 USENIX
Annual Technical Conference, Monterey, CA, June 9-14,
2002.
[Mc95] J. D. McCalpin, "A Survey of memory bandwidth and machine
balance in current high performance computers", IEEE TCCA
Newsletter, December 1995.
[PAC+97] D. Patterson, T. Anderson, N. Cardwell, R. Fromm, K.
Keeton, C. Kozyrakis, R. Thomas, K. Yelick , "A case for
intelligient RAM: IRAM", IEEE Micro, April 1997.
[PDZ99] V. S. Pai, P. Druschel, W. Zwaenepoel, "IO-Lite: a unified
I/O buffering and caching system", Proc. of the 3rd
Symposium on Operating Systems Design and Implementation,
New Orleans, LA, February 1999.
[Pi01] J. Pinkerton, "Winsock Direct: The Value of System Area
Networks", May 2001, available from
http://www.microsoft.com/windows2000/techinfo/
howitworks/communications/winsock.asp.
[Po81] Postel, J., "Transmission Control Protocol", STD 7, RFC
793, September 1981.
[QUAD] Quadrics Ltd., Quadrics QSNet product information,
available from
http://www.quadrics.com/website/pages/02qsn.html.
[SDP] InfiniBand Trade Association, "Sockets Direct Protocol
v1.0", Annex A of InfiniBand Architecture Specification
Volume 1, Release 1.1, November 2002, available from
http://www.infinibandta.org/specs.
[SRVNET] R. Horst, "TNet: A reliable system area network", IEEE
Micro, pp. 37-45, February 1995.
[STREAM] J. D. McAlpin, The STREAM Benchmark Reference Information,
http://www.cs.virginia.edu/stream/.
[TK95] M. N. Thadani, Y. A. Khalidi, "An efficient zero-copy I/O
framework for UNIX", Technical Report, SMLI TR-95-39, May
1995.
[TLS] Dierks, T. and C. Allen, "The TLS Protocol Version 1.0",
RFC 2246, January 1999.
[VI] D. Cameron and G. Regnier, "The Virtual Interface
Architecture", ISBN 0971288704, Intel Press, April 2002,
more info at http://www.intel.com/intelpress/via/.
[Wa97] J. R. Walsh, "DART: Fast application-level networking via
data-copy avoidance", IEEE Network, July/August 1997, pp.
28-38.Authors' Addresses 🦋
the squad behind this doc, with their contact details so you can slide into their inbox 💌 (pls be respectful, they're legends 🫶)
Stephen Bailey
Sandburst Corporation
600 Federal Street
Andover, MA 01810 USA
Phone: +1 978 689 1614
EMail: steph@sandburst.com
Jeffrey C. Mogul
HP Labs
Hewlett-Packard Company
1501 Page Mill Road, MS 1117
Palo Alto, CA 94304 USA
Phone: +1 650 857 2206 (EMail preferred)
EMail: JeffMogul@acm.org
Allyn Romanow
Cisco Systems, Inc.
170 W. Tasman Drive
San Jose, CA 95134 USA
Phone: +1 408 525 8836
EMail: allyn@cisco.com
Tom Talpey
Network Appliance
1601 Trapelo Road
Waltham, MA 02451 USA
Phone: +1 781 768 5329
EMail: thomas.talpey@netapp.comFull Copyright Statement 🌈
the legal bit, kept word-for-word because it matters 💅⚖️
Copyright (C) The Internet Society (2005).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.Intellectual Property 🔐
who owns what, and how to flag anything sus 👀
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at ietf-
ipr@ietf.org.Acknowledgement 🌸
the RFC Editor gets funding from the Internet Society, so a big thank-you to them for keeping the lights on 💖
Funding for the RFC Editor function is currently provided by the
Internet Society.📦 where this page comes from
- ietf: rfc4297 2005-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms