RFC 893

Trailer Encapsulations 🎀

Okay bestie, let's get the paperwork out of the way first 💅 This is RFC 893, written by Samuel J. Leffler and Michael J. Karels at the University of California at Berkeley, dated April 1984. Its status is "for information only," and it's explicitly NOT an official protocol for the ARPA Internet community. The title block doesn't list any RFC it obsoletes or updates, so it's standing on its own 🧚

Part 1 of 2 ✨

Status of this Memo 📜

This doc talks about why "trailer encapsulations" are useful on local-area networks and how they get implemented on different media 🌸 It's for information only, so it's not a rulebook. It is NOT an official protocol for the ARPA Internet community, bestie. Just vibes and facts 💜

Introduction 🦋

Okay so a trailer encapsulation is a link level packet format that 4.2BSD UNIX (among others) uses 🍓 A system might generate one under certain conditions, trying to cut down on the number and size of memory-to-memory copy operations that a receiving host has to do when it processes a data packet. The whole point is that trailers are strictly a link level packet format, so they're not visible to higher level protocol processing, as long as they're implemented properly ✨

This note explains the motivation behind trailer encapsulation and describes the trailer packet formats currently in use on the 3 Mb/s Experimental Ethernet, the 10 Mb/s Ethernet, and the 10 Mb/s V2LNI ring networks [1]. Iconic stuff 👑

The idea of using a trailer encapsulation was suggested by Greg Chesson, and Bill Joy designed the encapsulation described here 🫶

Motivation 💖

Okay the big reason trailers exist is the overhead you can get during protocol processing when one or more memory to memory copies have to happen 😭 Copying can show up at lots of levels, from moving data between the network medium and host memory, all the way to passing data between the operating system and user address spaces. An optimal network implementation would ideally incur zero copy operations between the moment a data packet lands in host memory and the moment the right data gets handed to the receiving process. Not every packet can dodge copying, but when the host has decent memory management support, it can often skip the copy just by playing with the right virtual memory hardware 🔥

In a page mapped virtual memory environment, two prerequisites usually have to be met to hit that zero copy goal during packet processing. First, data headed for a receiving agent has to be aligned on a page boundary. Second, it has to have a size that's a multiple of the hardware page size (or be filled out to a page boundary). That second restriction assumes virtual memory protection is maintained at the page level, and different architectures might change these prerequisites 🦋

Segmenting data for transmission across a network is pretty easy to do at the right size. The problem is that unless the encapsulating protocol header is a fixed size, page alignment is basically impossible 🙈 Headers can vary in size because multiple protocols each bring their own header, or because of an agreement, like when optional information gets included in the header. To make page alignment work, the header info sitting in front of the data for the receiver has to be reduced to a fixed size, which is normally what happens at the link level of a network. So the trick is to take all that possibly variable length header information and move it after the data segment. Then the sending host can "do its best" to give the receiving host a chance to receive data on a page aligned boundary 🧚 Rearranging data at the link level so the variable length header info "trails" the data is the whole substance of trailer encapsulation ✨

There are a few implicit assumptions in that argument, though:

1. The receiving host must be willing to accept trailers. Because this is a link level encapsulation, unless a host to host negotiation happens (ideally at the link level so it doesn't break layering principles), only certain hosts will be able to talk to each other. Their communication could also get seriously impaired if trailer packets get mixed in with non-trailer packets 💀
2. The cost of receiving data on a page aligned boundary should be comparable to receiving data on a non-page aligned boundary. If the overhead of making sure alignment happens is too high, the savings from avoiding copy operations might not be cost effective 💅
3. The size of the variable length header information should be significantly less than the size of the data segment being transmitted. You can move trailing info without physically copying it, but implementation constraints and the characteristics of the underlying network hardware often rule out just remapping the header(s) 🌈
4. The memory to memory copying overhead that the receiver is expected to perform must be significant enough to justify the added complexity in both the sending and receiving host software 🎀

The first point is well known, and it's the reason this note exists 💜

Thought has been put into negotiating the use of trailers on a per host basis, using a variant of the Address Resolution Protocol [2] (actually augmenting that protocol). For now, though, every system that uses trailers requires that hosts sharing a network medium either all accept trailers or never transmit them 🫶 (The latter is easy to do at boot time in 4.2BSD without touching the operating system source code, which is a lowkey nice bonus 🍓)

The second point is, to the authors' knowledge, insignificant 🦋 A host may not be able to take advantage of the alignment and size properties of a trailer packet, but it should still never get in the way of them 🌸

For the third point, let's assume the trailing header information gets copied rather than remapped, and look at the header overhead in the TCP/IP protocols as a representative example [3] 🧚 If we say both the TCP and IP protocol headers count as variable length header information, then the smallest trailer packet (generated by a VAX) would carry 512 bytes of data plus 40+ bytes of header information, along with the trailer header described later ✨ The trailing header could have IP and/or TCP options included, but that would normally be rare, since you'd expect most TCP options to show up in the initial connection setup exchange. Either way it's certainly much smaller than 512 bytes 👑 If the data segment is larger, the ratio drops and the expected gain from fewer copies on the receiving end goes up 🔥 Given the relative overheads of a memory to memory copy operation versus a page map manipulation (including translation buffer invalidation), the advantage is obvious 😭

The fourth issue, the authors believe, is actually a non-issue 💅 In their implementation, the extra code needed to support trailer encapsulation comes to about a dozen lines in each link level "network interface driver." The performance improvement that results more than justifies that tiny investment in software 🎀

It's worth knowing that changing the network (and normal link) level format of a packet this way forces the receiving host to buffer the entire packet before processing it 🙈 Clever implementations might parse protocol headers as the packet arrives, to find out the actual size (or network level packet type) of an incoming message. That lets them avoid preallocating maximum sized buffers for incoming packets they can recognize as unacceptable 🧚 But implementations that parse the network level format on the fly are breaking layering principles that have been celebrated in design for a long time (even if they're often broken in implementation). Postponing link level type recognition is a valid criticism 💀 That said, with network hardware that supports DMA, the entire packet is always received before processing begins anyway, so it's not really a dealbreaker there 💖

Summary 💖

Okay so bestie, the big idea is this: we've described a link level encapsulation that lines data up so it plays nicely with the alignment that virtual memory hardware needs to work efficiently 🧠✨ It's giving practical, and it's already everywhere. Lots of systems use it, and it's a standard feature in 4.2BSD UNIX 💅

The payoff is that cooperating hosts on a local network can get some seriously noticeable performance boosts from it 🚀 The catch, ngl, is that right now every host on the network has to be on board, so it's all-or-nothing 🙈 The dream is a per host negotiation mechanism someday, so hosts that are into it can use the encapsulation even when the network isn't uniform 🫶

---

References 📚

[1] "The Ethernet - A Local Area Network", Version 1.0, Digital Equipment Corporation, Intel Corporation, Xerox Corporation, September 1980. 🌸

[2] Plummer, David C., "An Ethernet Address Resolution Protocol", RFC-826, Symbolics Cambridge Research Center, November 1982. 🦋

[3] Postel, J., "Internet Protocol", RFC-791, USC/Information Sciences Institute, September 1981. 💜