RFC 5532

RFC 5532 🎀 Network File System (NFS) Remote Direct Memory Access (RDMA) Problem Statement

okay bestie, this is RFC 5532 (part 1 of 3) 💅 It came from the Network Working Group and was written by T. Talpey and C. Juszczak. The date is May 2009 and the category is Informational, so it's just here to share info and isn't a standard 🧚 The title block doesn't list anything it obsoletes or updates, so no drama there ✨

Status of This Memo 🦋

This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited.

Copyright Notice 💜

Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved.

Okay, the legal bit, and it matters 🙈 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, as they stood on the publication date. You should read those carefully, because they spell out your rights and restrictions for this document. The license info lives at http://trustee.ietf.org/license-info.

This document may contain material from IETF Documents or IETF Contributions that were published or made publicly available before November 10, 2008. The people who control the copyright in some of that material may not have given the IETF Trust permission to allow modifications outside the IETF Standards Process. Unless you've got an adequate license from those copyright holders, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be made outside that process. The exception is formatting it for publication as an RFC, or translating it into languages other than English 🌈

Abstract 🌸

Okay so this doc is all about getting Remote Direct Memory Access (RDMA) working for the Network File System (NFS) protocols 💖 NFS has historically eaten a lot of overhead because of data copies on end-host systems, plus other processing costs. This document looks at how much RDMA could help these implementations, and why RDMA is especially well-suited to NFS and to network file protocols in general. It's giving main character energy ✨

Table of Contents 🧚

   1. Introduction ....................................................2
      1.1. Background .................................................3
   2. Problem Statement ...............................................4
   3. File Protocol Architecture ......................................5
   4. Sources of Overhead .............................................7
      4.1. Savings from TOE ...........................................8
      4.2. Savings from RDMA ..........................................9
   5. Application of RDMA to NFS .....................................10
   6. Conclusions ....................................................10
   7. Security Considerations ........................................11
   8. Acknowledgments ................................................12
   9. References .....................................................12
      9.1. Normative References ......................................12
      9.2. Informative References ....................................13

1. Introduction ✨

The Network File System (NFS) protocol (as described in [RFC1094], [RFC1813], and [RFC3530]) is one of several remote file access protocols used in the class of processing architecture sometimes called Network-Attached Storage (NAS) 🍓

Remote file access has historically been a convenient, cost-effective way to share information over a network. The popularity of the NFS protocol has proven that concept over time 🫶 However, there are issues with this kind of deployment.

Compared to a local, direct-attached file access architecture, NFS removes the overhead of managing the local on-disk file system state and its metadata. The catch is that it puts at least a transport network and two network endpoints between an application process and the files it's accessing 😭 Up to now, this trade-off has usually ended in a net performance loss, thanks to reduced bandwidth, higher application server CPU utilization, and other overheads.

A bunch of application classes have run into trouble moving to NFS architectures, especially ones supporting enterprise work in high-performance areas like database applications and shared clusters 💅 That's mostly because of the performance costs of NFS versus direct-attached files. Other reasons matter too, like the lack of strong consistency guarantees from NFS implementations.

Copying the local file access performance onto NAS with traditional network protocol stacks has been really hard. It's not because of protocol processing overhead, it's because of data copy costs in the network endpoints 🔥 This is especially true now, since host buses are often the main bottleneck in NAS architectures [MOG03] [CHA+01].

The External Data Representation [RFC4506] under NFS, and the Remote Procedure Call (RPC) [RFC5531], can add even more data copies, which makes the problem worse 🙈

Data copy-avoidance designs haven't been widely adopted, and there are a few reasons for that. [BRU99] points out that "many copy avoidance techniques for network I/O are not applicable or may even backfire if applied to file I/O". Other designs that get rid of unnecessary copies, like [PAI+00], are incompatible with existing APIs, so they force application changes 🌸

In recent years, the IETF chartered an effort to standardize a set of protocols for Remote Direct Memory Access (RDMA) over the standard Internet Protocol Suite [RDDP]. There is now a complete IP-based RDMA protocol suite in the published Standards Track specifications 🦋

RDMA is a general solution to the CPU overhead caused by data copies, mainly at the receiver. Plenty of research has looked at this and backed up how well the approach works. The "Remote Direct Memory Access (RDMA) over IP Problem Statement" [RFC4297] gives an overview of it 🧚

Besides the per-byte savings from offloading data copies, RDMA-enabled NICs (RNICS) also offload the underlying protocol layers (e.g., TCP). That cuts CPU overhead from NAS processing even further 💖

1.1. Background 💜

The RDDP Problem Statement [RFC4297] says:

High costs associated with copying are an issue primarily for large scale systems ... with high bandwidth feeds, usually multiprocessors and clusters, that are adversely affected by copying overhead. Examples of such machines include all varieties of servers: database servers, storage servers, application servers for transaction processing, for e-commerce, and web serving, content distribution, video distribution, backups, data mining and decision support, and scientific computing. Note that such servers almost exclusively service many concurrent sessions (transport connections), which, in aggregate, are responsible for > 1 Gbits/s of communication. Nonetheless, the cost of copying overhead for a particular load is the same whether from few or many sessions.

Each of those servers could be accessing its file data as an NFS client, serving data to such clients as NFS, or doing both 🌈

The CPU overhead of the NFS and TCP/IP protocol stacks (including data copies or reduced copy workarounds) becomes a big deal on these clients and servers 😭 File access over locally attached disks has relatively low overhead, thanks to the highly optimized I/O path and the direct memory access given to the storage controller. NFS isn't like that. It has to pass data to, and especially from, the network and network processing stack to the NFS stack. Data copies are often imposed on that transfer, and in some cases several copies get imposed in each direction 💅

Copies can show up in an NFS implementation that exchanges data with user address spaces, within kernel buffer caches, in eXternal Data Representation (XDR) marshalling and unmarshalling, and within network stacks and network drivers. Other overheads also show up, like serialization among multiple threads of execution sharing a single NFS mount point and transport connection ✨

Lots of upper-layer protocols get really high bandwidth and low overhead by using RDMA. [MAF+02] shows that the RDMA-based Direct Access File System (with a user-level implementation of the file system client) can outperform even a zero-copy implementation of NFS [CHA+01] [CHA+99] [GAL+99] [KM02]. Also, file data access implies the use of large Unequal Loss Protection (ULP) messages. Those big messages tend to amortize any increase in per-message costs from offloading protocol processing when using RNICs, and they still get the benefit of reduced per-byte costs 🔥 Finally, the direct memory addressing RDMA provides avoids many sources of contention on network resources 🦋

2. Problem Statement 👑

The main performance problem for NFS implementations is the CPU overhead needed to implement the protocol. The biggest source of that overhead is moving data from NFS protocol messages to its final destination in user buffers or aligned kernel buffers 🙈 Because of how RPC and XDR work, the NFS data payload arrives at arbitrary alignment, which forces a copy at the receiver. NFS requests also complete in an arbitrary sequence.

Those data copies eat up system bus bandwidth and CPU time, which leaves less system capacity for applications [RFC4297]. So far, getting zero-copy with NFS has needed sophisticated, version-specific "header cracking" hardware and/or extensive platform-specific virtual memory mapping tricks 💖 Those approaches get even harder with NFS version 4, because of the COMPOUND operation and the presence of Kerberos and other security information. Those things reduce alignment even more and make ULP offload way more complicated 😭

NFS is also challenged by high-speed network fabrics like 10 Gbits/s Ethernet. Even raw network I/O such as TCP is a problem at those speeds with today's hardware 🔥 The problem is fundamental, and it's why the IETF has been exploring RDMA [RFC4297] 🌸

Zero-copy techniques help file protocols a lot. They enable direct user I/O, reduce the overhead of protocol stacks, provide perfect alignment into caches, and more. Many studies have already shown the performance benefits of these techniques [SKE+01] [DCK+03] [FJNFS] [FJDAFS] [KM02] [MAF+02] 🧚

RDMA is compelling here for another reason. Hardware-offloaded networking support by itself doesn't avoid data copies, unless you implement part of the NFS protocol in the Network Interface Card (NIC) 🌈 Supporting RDMA in NFS gets the highest performance at the architecture level instead of the implementation level, and that makes for ubiquitous and interoperable solutions 👑

By providing file access performance equal to local file systems, NFS over RDMA will let applications running on a set of client machines interact through an NFS file system. That's the same way applications on a single machine interact through a local file system 💅✨

3. File Protocol Architecture ✨

okay so NFS runs as an Open Network Computing (ONC) RPC [RFC5531] application 🧚 And since it's a file access protocol, it is SO "rich" in data content compared to control info 💅

NFS messages can be super tiny (under 100 bytes) or hugely chunky (many kilobytes up to a megabyte or more) 😭 They're all wrapped inside an RPC message and follow a variable-length RPC header, which creates an alignment challenge for the data items in an NFS call (request) or reply (response) message. not me having to deal with alignment 🙈

On top of the control info in each NFS call or reply, there are sometimes big "chunks" of application file data, like read and write requests. With NFS version 4 (thanks to the COMPOUND operation), there can be several of these data chunks interspersed with control info. it's giving multitasking 🫶

ONC RPC is a remote procedure call protocol that has run over a bunch of different transports. Most implementations today use UDP or TCP 🌸 RPC messages are defined using eXternal Data Representation (XDR) [RFC4506], which gives you a canonical data representation across a variety of host architectures. The XDR data stream gets conveyed differently on each transport: on UDP, RPC messages are encapsulated inside datagrams, while on a TCP byte stream, they're delineated by a record-marking protocol. An RDMA transport also conveys RPC messages in its own unique fashion, and that has to be fully described if client and server implementations are going to interoperate 🦋

The RPC transport's job is to convey an RPC message from a sender to a receiver. An RPC message is either an RPC call from a client to a server, or an RPC reply from the server back to the client. An RPC message contains an RPC call header followed by arguments if it's a call, or an RPC reply header followed by results if it's a reply. The call header holds a transaction ID (XID), then the program and procedure number, plus a security credential. An RPC reply header begins with an XID that matches the RPC call message, followed by a security verifier and results. All data in an RPC message is XDR encoded 💖

Encoding XDR data into transport buffers is called "marshalling", and decoding XDR data from transport buffers into destination RPC procedure result buffers is called "unmarshalling". So marshalling happens at the sender of any message, whether it's an RPC request or an RPC response, and unmarshalling happens at the receiver 🎀

Normally, any bulk data gets moved (copied) as part of the unmarshalling process, because the destination address isn't known until the RPC code gets control and invokes the XDR unmarshalling routine. In other words, XDR-encoded data is not self-describing, and it carries no placement information. This results in a data copy in most NFS implementations. ngl that's a lot of copying 😩

One mechanism the RPC layer can use to get around this is for each request to include placement information, to be used for direct placement during XDR encode. This "write chunk" can avoid sending bulk data inline in an RPC message, and it generally results in one or more RDMA Write operations 🔥

Similarly, a "read chunk" can be conveyed, carrying placement information for bulk data that may be directly fetched via one or more RDMA Read operations during XDR decode. The "read chunk" is therefore useful in both RPC calls and replies, while the "write chunk" is used solely in replies 💜

These "chunks" are the key concept in an existing proposal [RPCRDMA]. They convey what are effectively pointers to remote memory across the network. They let cooperating peers exchange data outside of XDR encodings while still using XDR to describe the data being transferred. And through use of XDR, they keep a large degree of on-the-wire compatibility. main character energy of a design 👑

The central concept of the RDMA transport is to provide additional encoding conventions that convey this placement information in a transport-specific encoding, and to modify the XDR handling of bulk data 🌈

Here's the layout, bestie 🦋

                           Block Diagram

   +------------------------+-----------------------------------+
   |         NFS            |            NFS + RDMA             |
   +------------------------+----------------------+------------+
   |           Operations / Procedures             |            |
   +-----------------------------------------------+            |
   |                   RPC/XDR                     |            |
   +--------------------------------+--------------+            |
   |       Stream Transport         |      RDMA Transport       |
   +--------------------------------+---------------------------+

---

4. Sources of Overhead 💅

Network and file protocol costs can be sorted into these buckets:

• per-byte costs - data touching costs such as checksum or data copy. Today's network interface hardware commonly offloads the checksum, which leaves the other major source of per-byte overhead, data copy. 🍓
• per-packet costs - interrupts and lower-layer processing (LLP). Today's network interface hardware also commonly coalesces interrupts to reduce per-packet costs. 🧚
• per-message (request or response) costs - LLP and ULP processing. 💜

Optimizing gets more important when the overhead it targets is a bigger share of the total cost. As other sources of overhead, like the checksumming and interrupt handling above, get eliminated, the leftover overheads (mainly data copy) loom larger 👀

With copies crossing the bus twice per copy, network processing overhead is high whenever network bandwidth is large compared to CPU and memory bandwidths. Generally, with today's end-systems, the effects are observable at network speeds at or above 1 Gbit/s 🔥

A common question is whether more CPU processing power fixes the high processing costs of network I/O. The answer is no, it is the memory bandwidth that is the issue. Faster CPUs don't help if the CPU spends most of its time waiting for memory [RFC4297] 😤

TCP offload engine (TOE) technology aims to offload the CPU by moving TCP/IP protocol processing to the NIC. However, TOE technology by itself does nothing to avoid necessary data copies within upper-layer protocols. [MOG03] provides a description of the role TOE can play in reducing per-packet and per-message costs. Beyond the offloads commonly provided by today's network interface hardware, TOE alone (without RDMA) helps with protocol header processing, but this has been shown to be a minority component of the total protocol processing overhead. [CHA+01] 🙃

Lots of software approaches to optimizing network throughput have been tried. Experience has shown that network I/O interacts with other aspects of system processing, such as file I/O and disk I/O [BRU99] [CHU96]. Zero-copy optimizations based on page remapping [CHU96] can depend on machine architecture, and they are not scalable to multi-processor architectures. Correct buffer alignment and sizing together are needed to optimize the performance of zero-copy movement mechanisms [SKE+01]. The NFS message layout described above doesn't facilitate splitting headers from data, and it doesn't facilitate providing correct data buffer alignment either. ugh, the layout is not it 💀

4.1. Savings from TOE

The expected improvement of TOE specifically for NFS protocol processing can be quantified, and it's shown to be fundamentally limited. [SHI+03] presents a set of "LAWS" parameters that illustrate the issues. In the TOE case, the copy cost can be viewed as part of the application processing "a". Application processing increases the LAWS "gamma", which the paper shows results in a diminished benefit for TOE 📉

For example, if the overhead is 20% TCP/IP, 30% copy, and 50% real application work, then gamma is 80/20 or 4, which means the maximum benefit of TOE is 1/gamma, or only 25% 😭

For RDMA (with embedded TOE) and the same example, the "overhead" (o) offloaded or eliminated is 50% (20% + 30%). Therefore, in the RDMA case, gamma is 50/50 or 1, and the inverse gives the potential benefit of 1 (100%), a factor of two. RDMA really said "let me in" 🦋

                    CPU Overhead Reduction Factor

               No Offload   TCP Offload   RDMA Offload
               -----------+-------------+-------------
                  1.00x        1.25x         2.00x

The analysis in the paper shows that RDMA could improve throughput by the same factor of two, even when the host is (just) powerful enough to drive the full network bandwidth without RDMA. The speedup may also be higher if network bandwidth grows faster than Moore's Law, although the higher benefits will apply to a narrow range of applications 🌸

4.2. Savings from RDMA

Performance measurements directly comparing an NFS-over-RDMA prototype with conventional network-based NFS processing are described in [CAL+03]. The study compared Read throughput and CPU overhead on two types of Gigabit Ethernet adapters: a conventional adapter, and another type with RDMA capability. The prototype RDMA protocol performed all transfers via RDMA Read. The NFS layer was measured while performing read transfers, varying the transfer size and readahead depth across ranges used by typical NFS deployments 🧪

In these results, conventional network-based throughput was severely limited by the client's CPU being saturated at 100% for all transfers. Read throughput reached no more than 60 MBytes/s 😤

   I/O Type        Size     Read Throughput     CPU Utilization
   Conventional    2 KB         20 MB/s              100%
   Conventional   16 KB         40 MB/s              100%
   Conventional  256 KB         60 MB/s              100%

However, over RDMA, throughput rose to the theoretical maximum throughput of the platform, while saturating the single-CPU system only at maximum throughput. slay 👑

    I/O Type       Size     Read Throughput     CPU Utilization
   RDMA            2 KB         10 MB/s               45%
   RDMA           16 KB         40 MB/s               70%
   RDMA          256 KB        100 MB/s              100%

The lower relative throughput of the RDMA prototype at the small blocksize may be attributable to the RDMA Read imposed by the prototype protocol, which reduced the operation rate since it introduces additional latency. As well, it may reflect the relative increase of per-packet setup costs within the DMA portion of the transfer. ok the small-block tea is real ☕💜

5. Application of RDMA to NFS 🦋

Okay, efficient file protocols need efficient data positioning and movement, and this is where it gets good. The client knows the client memory address where the app has data to be written, or where it wants read data to land. The server knows the server memory address where the local file system will accept write data, or where it has data ready to be read. 😭 Neither side knows where the other's data is supposed to go in the current NFS, RPC, or XDR protocols. Existing NFS implementations have really struggled with the performance cost of data copies over traditional Ethernet transports. It's giving bottleneck 💀

With faster networks coming, the network I/O bottleneck is only going to get worse. Luckily, new transports that support RDMA have shown up, and they're iconic 👑 RDMA is amazing at bulk transfer efficiency. It's an efficient way to do direct data placement, which removes a major part of the problem: data copies. RDMA also takes care of other overheads, like underlying protocol offload, and it separates control information from data. Main character energy 🌈

The current NFS message layout opens the door to a performance upgrade: an NFS-over-RDMA protocol that separates control information from data chunks while still meeting the alignment needs of both. The data chunks can be copied "directly" between the client and server memory addresses above (with a single occurrence on each memory bus), and the control information can be passed "inline". [RPCRDMA] describes exactly this kind of protocol. 🫶

6. Conclusions 💜

So, NFS version 4 [RFC3530] has been granted "Proposed Standard" status, and it's honestly a serious glow-up 🌸 The NFSv4 protocol was built around several key design points. The big ones: effective operation over wide-area networks, including the Internet itself; strong security built right into the protocol; extensive cross-platform interoperability, including integrated locking semantics that work with multiple operating systems; and (this is key) protocol extension. 🔥

NFS version 4 is an excellent base for adding the performance enhancements and improved semantics described above. The minor versioning support defined in NFS version 4 was designed to support protocol improvements without disrupting the installed base [NFSv4.1]. Evolving the protocol through minor versioning is a conservative, careful approach to current and future problems and shortcomings. Not reckless, just smart 🧚

There are a lot of arguments about how well the file abstraction will meet the future needs of enterprise data services and the Internet. Fine-grained Quality of Service (QoS) policies (for example, data delivery, retention, availability, security, etc.) are high on that list. ✨

It is vital that the NFS protocol keeps giving these benefits to a wide range of applications, without its usefulness being compromised by concerns about performance and semantic inadequacies. This can reasonably be handled within the existing NFS protocol framework. A cautious, evolutionary improvement of performance and semantics builds on the value NFS already has, while addressing new requirements that have come up as networking technology has changed. Building on what works is the move 🦋

7. Security Considerations 🔐

Okay, bestie, this section is serious, so listen up 🙈 The NFS protocol, together with its layering on RPC, gives applications and systems a rich and widely interoperable security model. Any layering of NFS-over-RDMA transports must address the NFS security requirements, and it must also ensure that no new vulnerabilities are introduced. For RDMA, the integrity, and any privacy, of the data stream are especially important. ⚠️

The core goals of an NFS-to-RDMA binding are to reduce overhead and to enable high performance. Supporting those goals while still keeping the required NFS security protection is a real challenge. Historically, integrity and privacy have been implemented within the RPC layer, and that operation requires local processing of messages exchanged with the RPC peer. That processing adds memory and processing overhead to every single message, which is exactly the overhead RDMA is designed to avoid. 💀

So, it is a requirement that the RDMA transport binding provide a way to delegate integrity and privacy processing to the RDMA hardware. That keeps the high level of performance we want from this approach, while still providing the highest levels of security NFS already requires. This in turn needs a way for the RPC layer to invoke these services from the RDMA provider, and for the NFS layer to negotiate their use end-to-end. 🎀

The "Channel Binding" concept [RFC5056], together with "IPsec Channel Connection Latching" [BTNSLATCH], provides a way for the RPC and NFS layers to delegate their session protection to the lower RDMA layers. An extension to the RPCSEC_GSS protocol [RFC5403] may be used to negotiate the use of these bindings and to establish the shared secrets needed to protect the sessions. 🧠

The protocol described in [RPCRDMA] specifies the use of these mechanisms, and they are required to implement the protocol. 📌

There's also another consideration: protecting the integrity and privacy of local memory by the RDMA transport itself. The use of RDMA by NFS must not introduce any vulnerabilities to system memory contents, or to memory owned by user processes. These protections come from the RDMA layer specifications, and specifically their security models. It is required that any RDMA provider used for NFS transport be conformant to the requirements of [RFC5042] in order to satisfy these protections. Protect the memory, periodt 💅

8. Acknowledgments 🫶

The authors wish to thank Jeff Chase, who provided many useful suggestions. Thank you, Jeff! 🌷

9. References 📚

9.1. Normative References

• [RFC3530] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame, C., Eisler, M., and D. Noveck, "Network File System (NFS) version 4 Protocol", RFC 3530, April 2003.
• [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol Specification Version 2", RFC 5531, May 2009.
• [RFC4506] Eisler, M., Ed., "XDR: External Data Representation Standard", STD 67, RFC 4506, May 2006.
• [RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS Version 3 Protocol Specification", RFC 1813, June 1995.
• [RFC5403] Eisler, M., "RPCSEC_GSS Version 2", RFC 5403, February 2009.
• [RFC5056] Williams, N., "On the Use of Channel Bindings to Secure Channels", RFC 5056, November 2007.
• [RFC5042] Pinkerton, J. and E. Deleganes, "Direct Data Placement Protocol (DDP) / Remote Direct Memory Access Protocol (RDMAP) Security", RFC 5042, October 2007.

9.2. Informative References

• [BRU99] J. Brustoloni, "Interoperation of copy avoidance in network and file I/O", in Proc. INFOCOM '99, pages 534-542, New York, NY, Mar. 1999., IEEE. Also available from http://www.cs.pitt.edu/~jcb/publs.html.
• [BTNSLATCH] Williams, N., "IPsec Channels: Connection Latching", Work in Progress, November 2008.
• [CAL+03] B. Callaghan, T. Lingutla-Raj, A. Chiu, P. Staubach, O. Asad, "NFS over RDMA", in Proceedings of ACM SIGCOMM Summer 2003 NICELI Workshop.
• [CHA+01] J. S. Chase, A. J. Gallatin, K. G. Yocum, "Endsystem optimizations for high-speed TCP", IEEE Communications, 39(4):68-74, April 2001.
• [CHA+99] J. S. Chase, D. C. Anderson, A. J. Gallatin, A. R. Lebeck, K. G. Yocum, "Network I/O with Trapeze", in 1999 Hot Interconnects Symposium, August 1999.
• [CHU96] H.K. Chu, "Zero-copy TCP in Solaris", Proc. of the USENIX 1996 Annual Technical Conference, San Diego, CA, January 1996.
• [DCK+03] M. DeBergalis, P. Corbett, S. Kleiman, A. Lent, D. Noveck, T. Talpey, M. Wittle, "The Direct Access File System", in Proceedings of 2nd USENIX Conference on File and Storage Technologies (FAST '03), San Francisco, CA, March 31 - April 2, 2003.
• [FJDAFS] Fujitsu Prime Software Technologies, "Meet the DAFS Performance with DAFS/VI Kernel Implementation using cLAN", available from http://www.pst.fujitsu.com/english/dafsdemo/index.html, 2001.
• [FJNFS] Fujitsu Prime Software Technologies, "An Adaptation of VIA to NFS on Linux", available from http://www.pst.fujitsu.com/english/nfs/index.html, 2000.
• [GAL+99] A. Gallatin, J. Chase, K. Yocum, "Trapeze/IP: TCP/IP at Near-Gigabit Speeds", 1999 USENIX Technical Conference (Freenix Track), June 1999.
• [KM02] K. Magoutis, "Design and Implementation of a Direct Access File System (DAFS) Kernel Server for FreeBSD", in Proceedings of USENIX BSDCon 2002 Conference, San Francisco, CA, February 11-14, 2002.
• [MAF+02] K. Magoutis, S. Addetia, A. Fedorova, M. Seltzer, J. Chase, D. Gallatin, R. Kisley, R. Wickremesinghe, E. Gabber, "Structure and Performance of the Direct Access File System (DAFS)", in Proceedings of 2002 USENIX Annual Technical Conference, Monterey, CA, June 9-14, 2002.
• [MOG03] J. Mogul, "TCP offload is a dumb idea whose time has come", 9th Workshop on Hot Topics in Operating Systems (HotOS IX), Lihue, HI, May 2003. USENIX.
• [NFSv4.1] Shepler, S., Eisler, M., and D. Noveck, "NFSv4 Minor Version 1", Work in Progress, September 2008.
• [PAI+00] V. S. Pai, P. Druschel, W. Zwaenepoel, "IO-Lite: a unified I/O buffering and caching system", ACM Trans. Computer Systems, 18(1):37-66, Feb. 2000.
• [RDDP] RDDP Working Group charter, http://www.ietf.org/html.charters/rddpcharter.html.
• [RFC4297] Romanow, A., Mogul, J., Talpey, T., and S. Bailey, "Remote Direct Memory Access (RDMA) over IP Problem Statement", RFC 4297, December 2005.
• [RFC1094] Sun Microsystems, "NFS: Network File System Protocol specification", RFC 1094, March 1989.
• [RPCRDMA] Talpey, T. and B. Callaghan, "Remote Direct Memory Access Transport for Remote Procedure Call", Work in Progress, April 2008.
• [SHI+03] P. Shivam, J. Chase, "On the Elusive Benefits of Protocol Offload", Proceedings of ACM SIGCOMM Summer 2003 NICELI Workshop, also available from http://issg.cs.duke.edu/publications/niceli03.pdf.
• [SKE+01] K.-A. Skevik, T. Plagemann, V. Goebel, P. Halvorsen, "Evaluation of a Zero-Copy Protocol Implementation", in Proceedings of the 27th Euromicro Conference - Multimedia and Telecommunications Track (MTT'2001), Warsaw, Poland, September 2001.

Authors' Addresses 🎀

Tom Talpey
170 Whitman St.
Stow, MA 01775 USA

Phone: +1 978 821-8577
EMail: tmtalpey@gmail.com


Chet Juszczak
P.O. Box 1467
Merrimack, NH 03054

Phone: +1 603 253-6602
EMail: chetnh@earthlink.net