RFC 3819
Advice for Internet Subnetwork Designers
RFC 3819: Advice for Internet Subnetwork Designers 🎀
Part 1 of 16 💖 (more parts are coming, so keep your eyes peeled!)
Okay bestie, let's start with the title block 🦋. This is RFC 3819, Advice for Internet Subnetwork Designers, and it's a Best Current Practice (that's BCP: 89) from the Network Working Group, dated July 2004. P. Karn (Qualcomm) is the editor, and the co-authors are C. Bormann (Universitaet Bremen TZI), G. Fairhurst (University of Aberdeen), D. Grossman (Motorola, Inc.), R. Ludwig (Ericsson Research), J. Mahdavi (Novell), G. Montenegro (Sun Microsystems Laboratories, Europe), J. Touch (USC/ISI), and L. Wood (Cisco Systems). This header doesn't list anything it obsoletes or updates, so no drama there 🌸
Status of this Memo ✨
So this doc is a Best Current Practice for the whole Internet community 🌈 It's basically asking everyone to talk about it and send over ideas for making it better. And it can be shared freely, no restrictions 💅
Copyright Notice 💜
Copyright (C) The Internet Society (2004).
Abstract 🍓
This doc is advice for people building digital communication equipment, link-layer protocols, and packet-switched local networks. We're calling all of that "subnetworks" 🧚 These designers want to support the Internet protocols, but they might not know much about the Internet architecture or how their design choices affect the performance and efficiency of the Internet. Bestie, that's exactly who this is for 🫶
Table of Contents 🌸
1. Introduction and Overview. . . . . . . . . . . . . . . . . . . 2
2. Maximum Transmission Units (MTUs) and IP Fragmentation . . . . 4
2.1. Choosing the MTU in Slow Networks. . . . . . . . . . . . 6
3. Framing on Connection-Oriented Subnetworks . . . . . . . . . . 7
4. Connection-Oriented Subnetworks. . . . . . . . . . . . . . . . 9
5. Broadcasting and Discovery . . . . . . . . . . . . . . . . . . 10
6. Multicasting . . . . . . . . . . . . . . . . . . . . . . . . . 11
7. Bandwidth on Demand (BoD) Subnets. . . . . . . . . . . . . . . 13
8. Reliability and Error Control. . . . . . . . . . . . . . . . . 14
8.1. TCP vs Link-Layer Retransmission . . . . . . . . . . . . 14
8.2. Recovery from Subnetwork Outages . . . . . . . . . . . . 17
8.3. CRCs, Checksums and Error Detection. . . . . . . . . . . 18
8.4. How TCP Works. . . . . . . . . . . . . . . . . . . . . . 20
8.5. TCP Performance Characteristics. . . . . . . . . . . . . 22
8.5.1. The Formulae . . . . . . . . . . . . . . . . . . 22
8.5.2. Assumptions. . . . . . . . . . . . . . . . . . . 23
8.5.3. Analysis of Link-Layer Effects on TCP
Performance. . . . . . . . . . . . . . . . . . . 24
9. Quality-of-Service (QoS) Considerations. . . . . . . . . . . . 26
10. Fairness vs Performance. . . . . . . . . . . . . . . . . . . . 29
11. Delay Characteristics. . . . . . . . . . . . . . . . . . . . . 30
12. Bandwidth Asymmetries. . . . . . . . . . . . . . . . . . . . . 31
13. Buffering, Flow and Congestion Control . . . . . . . . . . . . 31
14. Compression. . . . . . . . . . . . . . . . . . . . . . . . . . 34
15. Packet Reordering. . . . . . . . . . . . . . . . . . . . . . . 36
16. Mobility . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
17. Routing. . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
18. Security Considerations. . . . . . . . . . . . . . . . . . . . 41
19. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 44
20. Informative References . . . . . . . . . . . . . . . . . . . . 45
21. Contributors' Addresses. . . . . . . . . . . . . . . . . . . . 57
22. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 58
23. Full Copyright Statement . . . . . . . . . . . . . . . . . . . 601. Introduction and Overview 👑
Okay so, IP (the Internet Protocol, [RFC791] [RFC2460]) is the core protocol of the Internet, and it's giving "simple but iconic" energy ✨ It defines a "connectionless" packet-switched network. The Internet's success is mostly credited to how simple IP is, to the "end-to-end principle" [SRC81] the Internet is built on, and to how easily IP can run over all kinds of subnetworks, even ones that weren't designed with IP in mind 🦋 A subnetwork is any network that sits right below the IP layer and connects two or more IP systems (end hosts or routers). In its simplest form, that's just two IP systems directly connected, like with a length of cable or over a wireless medium 🌈
This doc defines a subnetwork as a layer 2 network, meaning one that doesn't need IP routers to forward packets between its parts. IP routers can still bridge frames at Layer 2 between parts of a subnetwork, though 💅 Sometimes it's handy to group a bunch of these subnetworks into one logical subnetwork. IP routing protocols (like OSPF, IS-IS, and PIM) can be set up to do that grouping, but they usually show you a layer-3 subnetwork instead of a layer-2 one. This can also mean one specific packet goes over the same layer-2 subnetwork several times, by way of an intermediate layer-3 gateway (router). Since that grouping needs layer-3 parts, the issues that come with it are outside the scope of this doc 🧚
That said, even though a lot of subnetworks carry IP, they don't always do it with maximum efficiency, minimum complexity, or low cost. They also often skip features that newer Internet stuff really needs, like multicasting or quality of service 😭
The Internet has grown explosively, so IP packets are now a bigger and bigger share of the traffic on the world's telecom networks 📈 It totally makes sense to optimize both existing and brand-new subnetwork technologies for IP as much as possible 🔥
Optimizing a subnetwork for IP comes down to three complementary things:
- Providing enough functionality to carry IP.
- Getting rid of unnecessary functions that add cost or complexity.
- Choosing subnetwork parameters that maximize the performance of the Internet protocols.
Because IP is so simple, consideration 2 is more of a problem than consideration 1. In other words, subnetwork designers make way more errors of commission than errors of omission 🙈 Still, some Internet features, like multicasting and quality-of-service, get a huge boost from support in the underlying subnetworks, beyond what's needed to carry "traditional" unicast, best-effort IP 💖
A big thing to think about when designing any layered communication network is which layer (or layers) should handle a given function. This was first tackled in the legendary paper "End-to-End Arguments in System Design" [SRC81] 🎀 That paper argued that a lot of functions can only be implemented properly on an end-to-end basis, meaning at the highest protocol layers, outside the subnetwork. These include making sure data is delivered reliably and using cryptography for confidentiality and message integrity 🔐
Those functions can't be built purely by chaining together hop-by-hop services. Copying them at the lower layers (inside the subnetwork) can be needlessly redundant, or even harmful to cost and performance 💀
Now, partial duplication of functionality in a lower layer can sometimes be justified by performance, security, or availability reasons ✨ Examples include link-layer retransmission to improve performance on an unusually lossy channel (like mobile radio), link-level encryption meant to thwart traffic analysis, and redundant transmission links to improve availability, increase throughput, or guarantee performance for certain classes of traffic 🌸 Duplicating protocol functions should only happen with a real understanding of the system-level implications, including possible interactions with higher-layer mechanisms. Don't just copy stuff without thinking, bestie 🙅♀️
The original Internet architecture was shaped by the end-to-end principle [SRC81], and in our view, that's a big part of why the Internet has been so successful 🌈👑
The rest of this doc digs into the different subnetwork design issues that the authors think matter for efficient IP support. Let's gooo 🚀💖
2. Maximum Transmission Units (MTUs) and IP Fragmentation ✨
okay so IPv4 packets come in all sizes, from a tiny 20 bytes (that's just the IPv4 header, no snacks 🍓) all the way up to 65535 bytes. But subnetworks don't have to handle maximum-sized (64KB) packets 💅 because IP has a trick: it chops packets that are too big for a subnetwork into fragments. Those fragments travel as independent IP packets and get put back together at the destination. 🧩 The biggest packet a subnetwork can carry is called its Maximum Transmission Unit (MTU), and it's a big deal, bestie.
Subnetworks may indicate the length of each packet they carry, but they don't have to. Ethernet with the super popular DIX [DIX82] header (not IEEE 802.3 [IEEE8023]) is an example that doesn't do it. That header has no length field to show the real data length when a packet gets padded up to a minimum of 60 bytes. 🙈 It's not a problem for uncompressed IP, though, because every IP packet carries its own length field, so it's giving self-sufficient.
Now, if you use optional header compression [RFC1144] [RFC2507] [RFC2508] [RFC3095], the link framing is required to indicate the frame length. That's non-negotiable, because it's needed to rebuild the original header. 🫶
In IP version 4 (the version everyone is still using 😭), fragmentation can happen at the sending host or at an intermediate router. Fragments can even get fragmented again at later routers if they need to.
IP version 6 [RFC2460] is a bit stricter. Fragmentation can only happen at the sending host. It can't happen in a router, which this document calls "router fragmentation." Not me being picky, but that's the rule 👑
Both IPv4 and IPv6 have a "path MTU discovery" procedure [RFC1191] [RFC1435] [RFC1981]. It lets the sending host dodge fragmentation by finding the smallest MTU along a path and shrinking its packets to match. This procedure is optional in both IPv4 and IPv6, so it's totally a choice.
Path MTU discovery is widely deployed, but it has its moments 💀 Some routers don't generate the ICMP messages that carry path MTU info back to the sender, and sometimes overly strict firewalls block those ICMP messages altogether. The result can be a "Path MTU Black Hole" [RFC2923] [RFC1435]. Packets just vanish into the void and nobody knows why, which is so unbothered of them 🌑
The path MTU discovery procedure, the fact that path MTU black holes keep showing up, and the removal of router fragmentation in IPv6 all show that the Internet technical community agrees router fragmentation is best avoided. ✨ That means subnetworks have to support MTUs that are "reasonably" large. Every IPv4 end host is required to accept and reassemble IP packets of 576 bytes [RFC791], but honestly, a number that small would be clearly inefficient 🙈 IPv6 doesn't let routers fragment, so [RFC2460] sets a bigger minimum MTU of 1280 bytes. Any subnetwork whose internal packet payload is smaller than 1280 bytes must have a mechanism that fragments and reassembles IP packets to and from subnetwork frames, if it wants to support IPv6. Iconic behavior from the subnetwork, honestly 🦋
If a subnetwork can't directly support a "reasonable" MTU with its native framing, it should fragment internally. In other words, it should transparently break IP packets into internal data elements and put them back together at the other end of the subnetwork. 🧚
So what counts as a "reasonable" MTU? That's the real question 🤔 Ethernet (10 and 100 Mb/s) has an MTU of 1500 bytes. Because Ethernet is everywhere, few Internet paths have MTUs bigger than that. This really limits how useful the bigger MTUs from other subnetworks are. Meanwhile, bigger MTUs are increasingly wanted on high-speed subnetworks, because they cut down the per-packet processing overhead in host computers. So implementers are encouraged to provide them, even though they may not be usable when Ethernet is also in the path. Main character energy, but it's complicated 💜
Then there are "tunneling" schemes, like GRE [RFC2784] or IP Security in tunnel mode [RFC2406], which treat IP as a subnetwork for IP. 🎀 Tunneling adds header overhead, so it can trigger fragmentation, even when the same physical subnetworks (for example, Ethernet) are on both sides of the host doing the IPsec encapsulation. Tunneling has made it harder to avoid router fragmentation and has increased the number of path MTU black holes [RFC2401] [RFC2923]. Bigger subnetwork MTUs may help ease this problem, so there's hope 🌈
2.1. Choosing the MTU in Slow Networks
In slow networks, even the largest possible packet can take a seriously long time to send. This is known as channelisation or serialisation delay. ⏳ Total end-to-end interactive response time should not go over the well-known human factors limit of 100 to 200 ms. That limit includes every source of delay: electromagnetic propagation delay, queuing delay, serialisation delay, and the store-and-forward time, which means the time it takes to transmit a packet at link speed.
At low link speeds, store-and-forward delays can dominate the total end-to-end delay, and those delays are directly influenced by the maximum transmission unit (MTU) size. 😩 Even when an interactive packet gets a higher queuing priority, it may still have to wait for a big bulk transfer packet to finish sending. That worst-case wait can be set by picking an appropriate MTU.
Let's do the math, because it's wild 🧮 If the MTU is 1500 bytes, an MTU-sized packet takes about 8 milliseconds to send on a T1 (1.536 Mb/s) link. But if the link speed is 19.2kb/s, the transmission time jumps to 625 ms -- way above our 100-200 ms limit. A 256-byte MTU would bring that delay down to a little over 100 ms. 💖 Still, care should be taken not to lower the MTU excessively. That increases header overhead and triggers frequent router fragmentation (if Path MTU discovery is not in use). This is likely to be the case with multicast, where Path MTU discovery is ineffective. 🙃
One way to limit delay for interactive traffic, without shrinking the MTU, is to give that traffic priority and to preempt (abort) the transmission of a lower-priority packet when a higher-priority packet shows up in the queue. ⚡ The catch is that the link resources used to send the aborted packet are lost, so overall throughput will decrease.
Another way to limit delay is a link-level multiplexing scheme that lets several packets be in progress at the same time, with transmission priority given to segments of higher-priority IP packets. For links using the Point-To-Point Protocol (PPP) [RFC1661], multi-class multilink [RFC2686] [RFC2687] [RFC2689] provides this kind of facility. 🫧
ATM (asynchronous transfer mode) is another example of this technique. SNDUs get fragmented and interleaved across smaller 53-byte ATM cells. 🌸 But ATM is generally used on high-speed links, where store-and-forward delays are already minimal, and it adds significant overhead (~9%) because each 48-byte ATM cell carries 5 bytes of cell overhead. Not the vibe for every situation 💅
A third example is the Data-Over-Cable Service Interface Specification (DOCSIS), with typical upstream bandwidths of 2.56 Mb/s or 5.12 Mb/s. 📡 To reduce the impact of a 1500-byte MTU in DOCSIS 1.0 [DOCSIS1], a data link layer fragmentation mechanism was specified in DOCSIS 1.1 [DOCSIS2]. But to accommodate the installed base, DOCSIS 1.1 must be backward compatible with DOCSIS 1.0 cable modems, which generally do not support fragmentation. When DOCSIS 1.0 and DOCSIS 1.1 coexist, the unfragmented large data packets from DOCSIS 1.0 cable modems may affect the quality of service for voice packets from DOCSIS 1.1 cable modems. It has been shown in [DOCSIS3] that bandwidth allocation algorithms can mitigate this effect. Not every problem is a one-and-done situation, iykyk 🎧
To summarize, there is a fundamental tradeoff between efficiency and latency in subnetwork design, and the designer should keep this tradeoff in mind. 👑🫶
3. Framing on Connection-Oriented Subnetworks ✨
okay so IP needs subnetworks to mark where each variable-length, asynchronous IP packet starts and ends 🎀 Some links and subnetworks don't do this on their own, and these are the main examples:
- leased lines carrying a synchronous bit stream;
- ISDN B-channels carrying a synchronous octet stream;
- dialup telephone modems carrying an asynchronous octet stream;
and
- Asynchronous Transfer Mode (ATM) networks carrying an asynchronous stream of fixed-sized "cells".
The Internet community has already made packet framing methods for all of these 🌈 The Point-To-Point Protocol (PPP) [RFC1661] uses a variant of HDLC and works on bit synchronous, octet-synchronous, and octet asynchronous links (aka examples 1-3 above) 💜 PPP is one of the preferred framing methods for IP, since a big pile of systems already work with PPP, so it's giving compatibility 🫶 ATM has its own framing methods, described in [RFC2684] [RFC2364] 🦋
At high speeds, a subnetwork should provide a framed interface that can carry asynchronous, variable-length IP datagrams ✨ The maximum packet size that interface supports is covered above in the MTU/Fragmentation section 🍓 The subnetwork can build this however it likes 💅
IP packet boundaries don't have to line up with any framing or synchronization stuff going on inside the subnetwork 🧚 When the subnetwork uses variable sized data units, the most straightforward move is to put exactly one IP packet into each subnetwork data unit (SNDU), and then lean on the subnetwork's existing way of marking SNDU edges to mark the IP packet edges too 👑 Ethernet is a great example of this 🔥 Some subnetworks, though, only have SNDUs of one or more fixed sizes, because of switching, forward error correction and/or interleaving stuff 🙈 ATM is one of these, with a single cell payload of 48 octets plus a 5-octet header 💀 IS-95 digital cellular is another one, with two "rate sets" of four fixed frame sizes each, and you can pick one on 20 millisecond boundaries 😭
Since IP packets come in all kinds of lengths, they won't always fit into a whole number of fixed-sized SNDUs 🌸 So you need an "adaptation layer" to turn IP packets into SNDUs while marking where each IP packet ends 🫶
There are a few ways to handle this 💖 The first one is to encode each IP packet into one or more SNDUs, with no SNDU holding pieces of more than one IP packet, and to pad out the last SNDU of the packet as needed ✨ Bits in a control header added to each SNDU say where that chunk of data belongs in the IP packet 🎀 If the subnetwork gives you in-order, at-most-once delivery, the header can be as simple as a pair of bits saying whether the SNDU is the first and/or the last in the IP packet 🦋 Or, for subnetworks that don't reorder the fragments of an SNDU, you could mark only the last SNDU of the packet, because that implicitly tells you the next SNDU starts a new IP packet 🌈 The AAL5 (ATM Adaptation Layer 5) scheme used with ATM is an example of this approach, though it adds other features too, including a payload length field and a payload CRC 💅
In AAL5, the ATM User-User Indication (stored in the Payload Type field of an ATM cell) tells you which cell is the last one of a packet 🍓 The packet trailer sits at the end of the SNDU and holds the packet length and a CRC 🧚
Another framing technique is to add per-segment overhead to signal that a segment option is there 🌟 When it shows up, the option carries a pointer to the end of the packet 🔥 This is different from AAL5 because another packet is allowed to follow within the same segment 👑 MPEG-2 Transport Streams [EN301192] [ISO13818] support this style of fragmentation, and they can either use padding (so each MPEG transport stream packet only carries part of one IP packet), or let a second IP packet start in the same Transport Stream packet (no padding) 💜
A third approach is to put a special flag sequence into the data stream between each IP packet, then pack the resulting data stream into SNDUs without caring about SNDU boundaries 🌸 This can have implications when frames get lost 😭 The flag sequence can also pad the unused space at the end of an SNDU 🫶 If the special flag shows up in the user data, it gets escaped to an alternate sequence (usually bigger than a flag) so it doesn't get mistaken for a flag 🙈 The HDLC-based framing schemes used in PPP are all examples of this approach 💖
All three adaptation schemes add overhead 💀 How much depends on the spread of IP packet sizes, the size(s) of the SNDUs, and, in the HDLC-like approaches, what's inside the IP packet (since flag-like sequences in the packet have to be escaped, which makes them bigger) 🎀 The designer also has to weigh implementation complexity and performance when choosing and designing an adaptation layer 🦋
4. Connection-Oriented Subnetworks 💖
IP has no concept of a "connection" 🌈 It's a purely connectionless protocol, full stop 💅 When an application needs a connection, it's usually provided by TCP [RFC793], the Transmission Control Protocol, running on top of IP end-to-end 👑
Connection-oriented subnetworks can carry IP (and lots of them do), but it often comes with a lot of complexity 🔥 Subnetworks with only a few nodes can just open a permanent connection between each pair of nodes 🧚 This is done a lot with ATM 🍓 But the number of connections grows as the square of the number of nodes, so this is clearly impractical for big subnetworks 😭 That's why a "shim" layer between IP and the subnetwork is required to manage connections 🫶 This is one of the most common jobs of a Subnetwork Dependent Convergence Function (SNDCF) sublayer between IP and a subnetwork 💜
SNDCFs usually open subnetwork connections as needed, when an IP packet is queued for transmission, and close them after an idle timeout ✨ There's no relationship between subnetwork connections and any connections that might exist at higher layers (e.g., TCP) 🎀
Internet traffic is typically bursty and transaction-oriented, so it's often hard to pick an optimal idle timeout 🙈 If the timeout is too short, subnetwork connections get opened and closed super fast, which could over-stress the subnetwork connection management system (especially if it was designed for voice traffic call holding times) 💀 If the timeout is too long, subnetwork connections sit idle most of the time, wasting whatever resources the subnetwork dedicated to them 😭
Purely connectionless subnets (like Ethernet), which have no state and dynamically share resources, are optimal for supporting best-effort IP, which is stateless and dynamically shares resources too 🌸 Connection-oriented packet networks (like ATM and Frame Relay), which have state and dynamically share resources, are less optimal, since best-effort IP doesn't get any benefit from the overhead of creating and maintaining state 🦋 Connection-oriented circuit-switched networks (including the PSTN and ISDN) have state and statically allocate resources for a call, so they need state creation and maintenance overhead, but they don't get the efficiencies of statistical multiplexing sharing of capacity that IP has 🌈
Either way, if an SNDCF that opens and closes subnet connections is used to support IP, care should be taken to make sure connection processing in the subnet can keep up with relatively short holding times 🧚💅
5. Broadcasting and Discovery 📡
Subnetworks fall into two categories: point-to-point and shared 🍓 A point-to-point subnet has exactly two endpoint components (hosts or routers) ✨ A shared link has more than two endpoint components, using either an inherently broadcast medium (e.g., Ethernet, radio) or a switching layer hidden from the network layer (e.g., switched Ethernet, Myrinet [MYR95], ATM) 🔥 Switched subnetworks handle broadcast by copying broadcast packets, giving each interface that supports one or more systems (hosts or routers) its own copy of each packet 👑
Several Internet protocols for IPv4 use broadcast capabilities, including link-layer address lookup (ARP), auto-configuration (RARP, BOOTP, DHCP), and routing (RIP) 🫶
Without broadcast capability, these protocols can get slower or stop working entirely (e.g., DHCP) 💀 ARP-like link address lookup can be handled by a centralized database instead, but that comes with potentially higher response latency and the need for nodes to know the ARP server address explicitly 😭 Shared links should support native, link-layer subnet broadcast 💖
A matching set of IPv6 protocols uses multicasting (see next section) instead of broadcasting, which gives similar functions with better scaling in large networks 🌈🦋
6. Multicasting ✨
Okay bestie, so the Internet model has this thing called "multicasting" 💖, where IP packets get sent to all the members of a multicast group [RFC1112] [RFC3376] [RFC2710]. Multicast is an option in IPv4 but a standard feature of IPv6 🌈. Right now IPv4 multicast is used by multimedia, teleconferencing, gaming, and file distribution (web, peer-to-peer sharing) apps, plus some key network and host protocols (e.g., RIPv2, OSPF, NTP) 🎀. IPv6 also leans on multicast for network configuration (DHCP-like autoconfiguration) and link-layer address discovery [RFC2461] (replacing ARP) 🧚. The big slay here is that with IPv6, autoconfiguration and address discovery can span across routers, whereas IPv4's broadcast-based services can't do that without ad-hoc router support [RFC1812] 😭.
Multicast-enabled IP routers organize each multicast group into a spanning tree 🌸. They route multicast packets by making copies of each packet and forwarding the copies to each output interface that includes at least one downstream member of the group ✨.
Multicasting is considerably more efficient when a subnetwork explicitly supports it 💅. For example, a router relaying a multicast packet onto an Ethernet segment only needs to send one copy, no matter how many group members are connected to that segment 🙈. Without native multicast support, routers and switches on shared links would have to use broadcast with software filters, so every multicast packet sent creates software overhead for every node on the subnetwork, even nodes that aren't in the group 💀. The alternative is for the router to transmit a separate copy to every member of the group on the segment, which is what happens on multicast-incapable switched subnets 🫶.
Subnetworks using shared channels (e.g., radio LANs, Ethernets) are especially suitable for native multicasting, and their designers should make every effort to support it 🔥. This involves designating a section of the subnetwork's own address space for multicasting 👑. On these networks, multicast is basically broadcast on the medium, with Layer-2 receiver filters 🦋.
Subnet interfaces also need to accept packets addressed to some number of multicast addresses, in addition to the unicast packets specifically addressed to them 💖. How many a host needs depends on what that host is doing, and at least several dozen will meet most current needs ✨.
On low-speed networks, the multicast address recognition function can be readily implemented in host software 🌈, but on high-speed networks it should be implemented in subnetwork hardware 🔥. That hardware doesn't have to be complete 🧚. For example, many Ethernet interfaces implement a "hashing" function, where the IP layer receives all of the multicast (and unicast) traffic the host subscribes to, plus some small fraction of multicast traffic it doesn't subscribe to 🙈. Host/router software then has to discard the unwanted packets that pass the Layer-2 multicast address filter [RFC1112] 🗑️.
There doesn't need to be a one-to-one mapping between a Layer-2 multicast address and an IP multicast address 💜. An address overlap can significantly degrade the filtering capability of a receiver's hardware multicast address filter 😬. A subnetwork supporting only broadcast should use this service for multicast and must rely on software filtering 💀.
Switched subnetworks must also provide a mechanism for copying multicast packets so they reach at least all members of a multicast group 🦋. One option is to "flood" multicast packets in the same manner as broadcast 🌊. This can lead to unnecessary transmissions on some subnetwork links (notably non-multicast-aware Ethernet switches) 😵. Some subnetworks therefore allow multicast filter tables to control which links receive packets belonging to a specific group 💅. Configuring this automatically requires access to Layer-3 group membership information (e.g., IGMP [RFC3376], or MLD [RFC2710]) 🧠.
There are various implementation options for giving a subnet node a list of mappings from multicast addresses to ports/interfaces 🗺️. They use a range of approaches, including signaling from end hosts (e.g., IEEE 802 GARP/GMRP [802.1p]), signaling from switches (e.g., CGMP [CGMP] and RGMP [RFC3488]), interception and proxy of IP group membership packets (e.g., IGMP/MLD Proxy [MAGMA-PROXY]), and enabling Layer-2 devices to snoop/inspect/peek into forwarded Layer-3 protocol headers (e.g., IGMP, MLD, PIM) so they can infer Layer-3 multicast group membership [MAGMA-SNOOP] 🕵️♀️. These approaches differ in their complexity, flexibility, and ability to support new protocols 🎀.
7. Bandwidth on Demand (BoD) Subnets 🦋
Some subnets allow a number of subnet nodes to share a channel efficiently by assigning transmission opportunities dynamically 🌸. A subnet node requests one when it has packets to send 📦. The subnet schedules and grants transmission opportunities sufficient to allow the transmitting node to send one or more packets (or packet fragments) 🧚. We call these subnets Bandwidth on Demand (BoD) subnets 👑. Examples of BoD subnets include Demand Assignment Multiple Access (DAMA) satellite and terrestrial wireless networks, IEEE 802.11 point coordination function (PCF) mode, and DOCSIS 📡. A connection-oriented network (such as the PSTN, ATM or Frame Relay) reserves resources on a much longer timescale, so it is not a BoD subnet in our taxonomy 🙅♀️.
The design parameters for BoD are similar to those in connection-oriented subnetworks, although implementations may vary significantly ✨. In BoD, the user typically requests access to the shared channel for some duration 💖. Access may be allocated for a period of time at a specific rate, for a certain number of packets, or until the user releases the channel 🎀. Access may be coordinated through a central management entity or with a distributed algorithm amongst the users 🫶. Examples of the resource that may be shared include a terrestrial wireless hop, an upstream channel in a cable television system, a satellite uplink, and an end-to-end satellite channel 🛰️.
Long-delay BoD subnets pose problems similar to connection-oriented subnets in anticipating traffic 😩. While connection-oriented subnets hold idle channels open expecting new data to arrive, BoD subnets request channel access based on buffer occupancy (or expected buffer occupancy) on the sending port 📦. Poor performance will likely result if the sender does not anticipate additional traffic arriving at that port during the time it takes to grant a transmission request 💀. It is recommended that the algorithm have the capability to extend a hold on the channel for data that has arrived after the original request was generated (this may be done by piggybacking new requests on user data) 🔥.
There is a wide variety of BoD protocols available 🌈. However, there has been relatively little comprehensive research on the interactions between BoD mechanisms and Internet protocol performance 🤔. Research on some specific mechanisms is available (e.g., [AR02]) 📚. One item that has been studied is TCP's retransmission timer [KY02] ⏱️. BoD systems can cause spurious timeouts when adjusting from a relatively high data rate to a relatively low data rate 😱. In this case, TCP's transmitted data takes longer to get through the network than predicted by the TCP sender's computed retransmission timeout. Therefore, the TCP sender is prone to resending a segment prematurely 🙈.
8. Reliability and Error Control 🛡️
In the Internet architecture, the ultimate responsibility for error recovery is at the end points [SRC81] 👑. The Internet may occasionally drop, corrupt, duplicate, or reorder packets, and the transport protocol (e.g., TCP) or application (e.g., if UDP is used as the transport protocol) must recover from these errors on an end-to-end basis [RFC3155] 🔁. Error recovery in the subnetwork is therefore justifiable only to the extent that it can enhance overall performance 🤓. It is important to recognize that a subnetwork can go too far in attempting to provide error recovery services in the Internet environment 😤. Subnet reliability should be "lightweight", i.e., it only has to be "good enough", not perfect 💖.
In this section, we discuss how to analyze characteristics of a subnetwork to determine what is "good enough" 🔍. The discussion below focuses on TCP, which is the most widely-used transport protocol in the Internet 💜. It is widely believed (and is a stated goal within the IETF) that non-TCP transport protocols should attempt to be "TCP-friendly" and have many of the same performance characteristics ✨. Thus, the discussion below should be applicable even to portions of the Internet where TCP may not be the predominant protocol 🌸.
8.1. TCP vs Link-Layer Retransmission 🦋
Okay bestie, so error recovery is all about sending extra redundant info that's computed from the user data. 🧠 Depending on how much of that extra info gets sent and how it's made, the receiver can reliably detect transmission errors, correct up to some max number of errors, or do both. The whole vibe is called Error Control Coding, or ECC. 💖
When you use ECC to detect errors so the sender can request retransmissions (which will hopefully be error-free 🙏), that's widely known as "ARQ" (Automatic Repeat Request). 🔁
When there's enough ECC info that the receiver can fix some errors without asking for a retransmission, that's called Forward Error Correction (FEC). ✨ FEC is complicated, and its design has to be tailored to the specific modem and channel, so it's traditionally been built into special-purpose hardware inside a modem. That basically makes it part of the physical layer. 🔧
Unlike ARQ, FEC was rarely used in telecommunications outside of space links before the 1990s. 🚀 Now it's nearly everywhere: telephone, cable and DSL modems, digital satellite links, and digital mobile phones. It's also super heavily used in optical and magnetic storage, where "retransmissions" just aren't possible. 💿
Some systems mix ARQ on top of FEC. V.90 dialup modems (in the upstream direction) with V.42 error control are one example. 📞 Most errors get fixed by the trellis (FEC) code inside the V.90 modem, and most of the leftover errors get detected and corrected by the ARQ mechanisms in V.42. It's giving teamwork 🫶
Work is now underway to apply FEC above the physical layer, mainly for reliable multicasting [RFC3048] [RFC3450-RFC3453], where conventional ARQ mechanisms are inefficient or hard to implement. 🌈 For this discussion, though, we'll assume that if FEC is present, it's implemented within the physical layer. 💅
Depending on which layer does the error control, it can work end-to-end or over a shorter span, like a single link. 🔗 TCP is the most important example of an end-to-end protocol that uses an ARQ strategy. 👑
Lots of link-layer protocols use ARQ, usually some flavor of HDLC [ISO3309]. Examples include the X.25 link layer, the AX.25 protocol used in amateur packet radio, 802.11 wireless LANs, and the reliable link layer specified in IEEE 802.2. 📡
Only end-to-end error recovery can ensure reliable service to the application (see Section 8). 💯 However, some subnetworks (e.g., many wireless links) also use link-layer error recovery as a performance boost [RFC3366]. For example, lots of cellular links have tiny physical frame sizes (< 100 bytes) and pretty high frame loss rates. 😭 Relying only on end-to-end error recovery can clearly lead to performance degradation, because retransmissions across the whole end-to-end path take way longer to arrive than link-layer retransmissions do. So link-layer error recovery can often make end-to-end performance better. ⭐ As a result, link-layer and end-to-end recovery often co-exist, and that can lead to possible inefficient interactions between the two layers of ARQ protocols. 🙈
Now here's the drama 🎭 This inter-layer "competition" can lead to a wasteful situation. When the link layer retransmits (parts of) a packet, the link latency momentarily goes up. TCP bases its retransmission timeout on prior measurements of total end-to-end latency, including that of the link in question. So this sudden latency spike can trigger an unnecessary TCP retransmission of a packet the link layer is still busy retransmitting. 💀 These spurious end-to-end retransmissions create unnecessary load and reduce end-to-end throughput. Honestly, the link layer might even end up with multiple copies of the same packet sitting in the same link queue at the same time. 😵 In general, the competing error recovery happens because an inner control loop (link-layer error recovery) reacts to the same signal as an outer control loop (end-to-end error recovery), with zero coordination between them. 🙃 Just to be clear, this is solely an efficiency issue. TCP still provides reliable end-to-end delivery over these links. 🫶
This raises the question of how persistent a link-layer sender should be when retransmitting [RFC3366]. We define the link-layer (LL) ARQ persistency as the maximum time that a particular link will spend trying to transfer a packet before it can be discarded. ⏳ This deliberately simplified definition says nothing about the maximum number of retransmissions, retransmission strategies, queue sizes, queuing disciplines, transmission delays, or anything like that. 🤷♀️ The reason we use the term "LL ARQ persistency," instead of something like "maximum link-layer packet holding time," is that the definition is closely tied to link-layer error recovery. For example, on links with straightforward error recovery strategies, LL ARQ persistency will often match a maximum number of retransmissions permitted per link-layer frame. 🎀
For link layers that can't tell flows apart (e.g., because of network layer encryption), the LL ARQ persistency should be small. 🔒 This avoids harmful effects or performance degradation from indiscriminate high persistence. A detailed discussion of all this is in [RFC3366]. 📚
But when a link layer can identify individual flows and apply ARQ selectively [LKJK02], then the link ARQ persistency should be high for a flow using reliable unicast transport protocols (e.g., TCP), and must be low for all other flows. 🧚 Setting the link ARQ persistency larger than the largest link outage lets TCP restore transmission quickly without waiting for a retransmission timeout. ⚡ This generally improves TCP performance when outages are transient. However, excessively high persistence can be disadvantageous, so a practical upper limit of 30-60 seconds may be desirable. 🤔 Implementing schemes like this is still a research issue. (See also the following section "Recovery from Subnetwork Outages.") 🔍
Lots of subnetwork designers have chances to reduce the probability of packet loss, e.g., with FEC, ARQ, and interleaving, but it costs increased delay. 🎯 TCP performance improves as loss decreases, but it gets worse as end-to-end delay increases, so it's important to find the right balance through analysis and simulation. ⚖️
8.2. Recovery from Subnetwork Outages 🌸
Some types of subnetworks, especially mobile radio, get hit with frequent temporary outages. 📵 For example, an active cellular data user might drive or walk into an area (like a tunnel) that's out of range of every base station. No packets get delivered successfully until the user comes back to an area with coverage. 🚇
The Internet protocols currently provide no standard way for a subnetwork to explicitly tell an upper layer protocol (e.g., TCP) that it's having an outage and not just severe congestion. 🙅♀️
In these situations, TCP waits even longer after each unsuccessful retransmission before trying again. That's its "exponential back-off" algorithm. 📉 Even worse, TCP won't find out that the subnetwork outage has ended until its next retransmission attempt. If TCP has backed off, that can take a while, and it can lead to extremely poor TCP performance over these subnetworks. 😩
So it's highly desirable that a subnetwork subject to outages does not silently discard packets during an outage. ✋ Ideally, the subnetwork should define an interface to the next higher layer (i.e., IP) that lets it refuse packets during an outage, and automatically ask IP for new packets when it can deliver them again. 💫 If it can't do that, then the subnetwork should hold onto at least some of the packets it accepts during an outage and try to deliver them once the outage ends. 📦 When packets are discarded, IP should be notified so the appropriate ICMP messages can be sent. 📬
Note that it is not necessary to completely avoid dropping packets during an outage. 🙌 The point of holding onto a packet during an outage, either in the subnetwork or at the IP layer, is so that its eventual delivery implicitly tells TCP that the subnetwork is operational again. This is to enhance performance, not to ensure reliability. Reliability, as discussed earlier, can only be ensured on an end-to-end basis. 💜
Only a few packets per TCP connection, including ACKs, need to be held this way so the TCP sender can recover from the additional losses once the flow resumes [RFC3366]. 🦋
Because it would be a layering violation (and possibly a performance hit) for IP or a subnetwork layer to look at TCP headers (which would be impossible anyway if IPsec encryption [RFC2401] is in use), it would be reasonable for the IP or subnetwork layers to pick, as a design parameter, some small number of packets to retain during an outage. 🧁
8.3. CRCs, Checksums and Error Detection ✨
okay bestie, so TCP [RFC793], UDP [RFC768], ICMP, and IPv4 [RFC791] all use the same simple 16-bit 1's complement checksum algorithm [RFC1071] to catch corrupted packets 🧐💖. The IPv4 header checksum only protects the IPv4 header. The TCP, ICMP, and UDP checksums give you end-to-end error detection for both the transport pseudo header (the network and transport layer info) and the payload data 🎀. Protecting the data is optional for apps using UDP [RFC768] over IPv4, but it is required for IPv6 🫶.
The Internet checksum is not very strong from a coding theory standpoint, ngl 💀. But it's super easy to compute in software, and every attempt to replace it with a stronger checksum has failed 😭. Also, undetected errors can and do happen in packets that end hosts receive [SP2000] 🙈.
IPv6 has no IP header checksum, because that would mean extra work for every router 🧚. Instead, the destination host catches the "important" errors in the IP header, like a packet getting delivered to the wrong destination. It does this by putting the IP source and destination addresses (the pseudo header) into the checksum computation of the TCP or UDP header, which IPv4 was already doing ✨. Errors in other IPv6 header fields may go undetected inside the network. This was considered a reasonable price to pay for a big drop in processing for each router, and it was assumed that subnetworks would use a strong link CRC 👑.
One way to add extra protection to an IPv4 or IPv6 header is the authentication and packet integrity services of the IP Security (IPsec) protocol [RFC2401] 🔐. But honestly, that may not be a choice available to the subnetwork designer 😤.
Most subnetworks detect errors right above the physical layer 🌸. Packets corrupted in transit get detected and discarded before they ever reach the IP layer. A 16-bit cyclic redundancy check (CRC) is usually the minimum for error detection. That's way more robust against most error patterns than the 16-bit Internet checksum 💅. Note that the error detection properties of a specific CRC get weaker as the frame size goes up 🦋. The Point-to-Point Protocol [RFC1662] requires support for a 16-bit CRC on each link frame, with a 32-bit CRC as an option. (PPP is often used with a dialup modem, which brings its own error control, so it's not all on PPP 📞.) Other subnetworks, including 802.3/Ethernet, AAL5/ATM, FDDI, Token Ring, and PPP over SONET/SDH, all use a 32-bit CRC 🌈. Lots of subnetworks can also use other mechanisms to boost the error detection of the link CRC, like FEC in dialup modems, mobile radio, and satellite channels 🛰️.
So any new subnetwork designed to carry IP should provide error detection for each IP packet that is at least as strong as the 32-bit CRC specified in [ISO3309] 💜. This gets you a very low undetected packet error rate from transmission errors. It won't (and doesn't need to) get a very low packet loss rate, though, because the Internet protocols are way better at handling lost packets than corrupted ones [SRC81] 🫠.
Packet corruption can also come from bugs in host and router hardware and software 🙃. Even if every subnetwork had strong error detection, it's still essential that end-to-end checksums get used at the receiving end host [SP2000] 🫶.
Designers of complex subnetworks, made of internal links and packet switches, should consider error detection on an edge-to-edge basis, covering an entire SNDU (or IP packet) 🧩. A CRC would be generated at the entry point to the subnetwork and checked at the exit endpoint. This can be used instead of, or together with, error detection at the interface to each physical link. The big perk of an edge-to-edge check is that it protects against errors introduced anywhere inside the subnetwork, not just on its transmission links 🎯. Examples include the way the Ethernet CRC-32 gets handled by LAN bridges [802.1D]. ATM AAL5 [ITU-I363] also uses an edge-to-edge CRC-32 🍓.
Some specific apps can handle residual errors in the data they exchange. But removing the link CRC may expose the network to an undesirable increase in undetected errors in the IP and transport headers 😬. Apps may also need a high level of error protection for control info exchanged by protocols above the transport layer. One example is a voice codec, which is robust against bit errors in the speech samples 🎤. For mechanisms like that to work, the receiving app has to be able to tolerate corrupted data. It also requires that the app uses a mechanism to signal that payload corruption is permitted, and to indicate the coverage (headers and data) that needs to be protected by the subnetwork CRC 🚦. The UDP-Lite protocol [RFC3828] is the first Internet standards track transport protocol that supports partial payload protection 🦋. Receiving corrupt data in arbitrary application protocols is seriously dangerous, though. A subnet could deliver data with errors that the application never detects, which corrupts the communicated data 💀 [SRC81].
8.4. How TCP Works 🚀
One of TCP's jobs is end-host based congestion control for the Internet 👑. This is a critical part of the overall stability of the Internet, so it's important that link-layer designers understand TCP's congestion control algorithms 🧠.
TCP assumes that, at the most abstract level, the network is made of links and queues 🧃. Queues provide output buffering on links that are momentarily oversubscribed. They smooth out instantaneous traffic bursts so they fit the link bandwidth. When demand exceeds link capacity for long enough to fill the queue, packets have to be dropped 😵. The traditional move of dropping the most recent packet ("tail dropping") is no longer recommended [RFC2309] [RFC2914], but it's still widely practiced 🙄.
TCP uses sequence numbering and acknowledgments (ACKs) on an end-to-end basis to provide reliable, sequenced delivery 📬. TCP ACKs are cumulative, i.e., each one implicitly ACKs every segment received so far ✅. If a packet with an unexpected sequence number arrives, the ACK field in the packets sent back by the receiver stops advancing 🛑. With an optional enhancement, TCP can send selective acknowledgments (SACKs) [RFC2018] to say which segments have arrived at the receiver 🎯.
Since the most common cause of packet loss is congestion, TCP treats packet loss as a sign of potential Internet congestion along the path between the TCP end hosts 🚧. This happens automatically, and the subnetwork doesn't need to know anything about IP or TCP. A subnetwork node simply drops packets whenever it has to, though some packet-dropping strategies (e.g., RED) are fairer to competing flows than others ⚖️.
TCP recovers from packet loss in two different ways 💫. The most important one is the retransmission timeout. If an ACK doesn't show up after a certain amount of time, TCP retransmits the oldest unacked packet ⏰. Taking that as a hint that the network is congested, TCP waits for the retransmission to be ACKed before it continues, and it gradually increases the number of packets in flight as long as a timeout doesn't happen again 📈.
A retransmission timeout can be a big performance hit, since the sender sits idle during the timeout interval and restarts with a congestion window of one TCP segment after the timeout 😩. To recover faster from the occasional lost packet in a bulk transfer, an alternate scheme called "fast recovery" was introduced [RFC2581] [RFC2582] [RFC2914] [TCPF98] ⚡.
Fast recovery relies on the fact that when a single packet is lost in a bulk transfer, the receiver keeps returning ACKs to later data packets that don't actually acknowledge any newly received data 🔁. These are known as "duplicate acknowledgments" or "dupacks" 👯. The sending TCP can use dupacks as a hint that a packet was lost, and retransmit it without waiting for a timeout 🏃♀️. Dupacks basically act as a negative acknowledgment (NAK) for the packet sequence number in the acknowledgment field 🙅♀️. TCP waits until a certain number of dupacks (currently 3) have been seen before it assumes a loss has happened. This helps avoid an unnecessary retransmission during out-of-sequence delivery 🫧.
A technique called "Explicit Congestion Notification" (ECN) [RFC3168] lets routers signal congestion directly to hosts without dropping packets 📣. This is done by setting a bit in the IP header 🪄. Since ECN support is likely to stay optional, the lack of an ECN bit must never be read as a lack of congestion 🚨. So for the foreseeable future, TCP has to interpret a lost packet as a signal of congestion 🫡.
The TCP "congestion avoidance" [RFC2581] algorithm keeps a congestion window (cwnd) that controls how much data TCP can have in flight at any moment 🪟. Shrinking cwnd lowers the overall bandwidth the connection gets. Raising cwnd boosts performance, up to the limit of the available capacity 📊.
TCP probes for available network capacity by starting cwnd at one or two packets, then growing it by one packet for each ACK the receiver returns 🐣. This is TCP's "slow start" mechanism 🌱. When a packet loss is detected (or congestion is signaled some other way), cwnd resets to one, and slow start repeats until cwnd reaches one half of its previous setting before the reset 🔄. After that point cwnd keeps growing, but way more slowly than before 🐢. If no more losses happen, cwnd will eventually reach the window size the receiver advertised 🏁.
This is an "Additive Increase, Multiplicative Decrease" (AIMD) algorithm 💖. The steep drop in cwnd in response to congestion keeps the network stable, and the AIMD algorithm also keeps things fair between long-running TCP connections sharing the same path ⚖️✨.
8.5. TCP Performance Characteristics 📈
Caveat 🫣
Here we present a current "state-of-the-art" understanding of TCP performance 🧪. This analysis tries to describe how TCP connections perform over links with different characteristics 🌐.
Link designers may wish to use the techniques in this section to predict what TCP/IP may achieve over a new link-layer design 🔮. Such analysis is encouraged 🙌. Because this is a relatively new analysis, and the theory is based on single-stream TCP connections under "ideal" conditions, it should be recognized that the results may differ from actual performance on the Internet 🫠. That being said, we did our best to give designers helpful information for an accurate picture of TCP's capabilities and limitations under various conditions 💅🎀.
8.5.1. The Formulae ✨
okay bestie, TCP's AIMD Congestion Avoidance algorithm has been analyzed extensively 🤓 and the current best formula for Reno TCP (the TCP specified in [RFC2581]) comes from Padhye, et al. [PFTK98]. it's math, but it's the iconic kind 💅
MSS
BW = --------------------------------------------------------
RTT*sqrt(1.33*p) + RTO*p*[1+32*p^2]*min[1,3*sqrt(.75*p)]where:
BW is the maximum TCP throughout achievable by an
individual TCP flow
MSS is the TCP segment size being used by the connection
RTT is the end-to-end round trip time of the TCP connection
RTO is the packet timeout (based on RTT)
p is the packet loss rate for the path
(i.e., .01 if there is 1% packet loss)ngl the speed of the links on the Internet path doesn't show up in this formula at all 🫠 but that's because sending faster than the slowest link just makes the queue grow at the transmitter driving the bottleneck. 📈 that bigger queue raises the RTT, and the higher RTT is what drags your achievable throughput down. 🌸
this is currently considered the best approximate formula for Reno TCP performance 👑 and it's commonly simplified by assuming RTO is approximately 5*RTT ✨
TCP is always getting upgraded 🦋 so there's also a simpler formula that gives an upper bound on the performance of any AIMD algorithm that's likely to be implemented in TCP in the future. that one was derived by Ott, et al. [MSMO97]:
MSS 1
BW = C --- -------
RTT sqrt(p)where C is 0.93. 💖
8.5.2. Assumptions
okay so both formulae assume the TCP Receiver Window isn't the thing limiting the connection 🙈 and since the receiver window is totally controlled by end-hosts, we assume hosts will max out the announced receiver window to get the best network performance. (sus, but iykyk 😌)
both formulae let BW go infinite if there's no loss 🫨 but a real Internet path will drop packets at bottlenecked queues if the load is too high. so a completely lossless TCP/IP network can never happen (unless the network is being underutilized). 🥲
the RTT used here is the arithmetic average, including queuing delays 🧚
the formulae are for a single TCP connection. if a path carries many TCP connections, each one follows the formulae above independently 💜
the formulae also assume long-running TCP connections. for connections that are extremely short (<10 packets) and don't lose any packets, performance is driven by the TCP slow-start algorithm 🚀 and for connections of medium length, where on average only a few segments are lost, single connection performance will actually be slightly better than the formulae predict. ✨
the difference between the simple and complex formulae is that the complex one includes the effects of TCP retransmission timeouts ⏰ for really low packet loss (significantly less than 1%), timeouts are unlikely to happen, so the formulae give very similar results 🌈 at higher packet losses (1% and above), the complex formula gives a more accurate estimate of performance, and it'll always come out significantly lower than the simple formula. 😭
Note that these formulae break down as p approaches 100%. 💀
8.5.3. Analysis of Link-Layer Effects on TCP Performance
okay here's a juicy example 🍓 a designer invents a new wireless link layer which, on average, loses 1% of IP packets. the link layer supports packets of up to 1040 bytes and has a one-way delay of 20 msec.
if this link gets used on an Internet path with a round trip time greater than 80ms, the upper bound can be computed like this:
- For MSS, use 1000 bytes to exclude the 40 bytes of minimum IPv4 and TCP headers.
- For RTT, use 120 msec (80 msec for the Internet part, plus 20 msec each way for the new wireless link).
- For p, use .01. For C, assume 1.
the simple formula gives:
BW = (1000 * 8 bits) / (.120 sec * sqrt(.01)) = 666 kbit/secand the more complex formula gives:
BW = 402.9 kbit/secif this were a 2 Mb/s wireless LAN, the designers might be somewhat disappointed. 😬
some observations on performance, bestie 🎀
- We have assumed that the packet losses on the link layer are interpreted as congestion by TCP. This is a "fact of life" that must be accepted. 💅
- The equations for TCP performance are all expressed in terms of packet loss, but many subnetwork designers think in terms of bit-error ratio. If channel bit errors are independent, then the probability of a packet being corrupted is:
p = 1 - ([1 - BER]^[FRAME_SIZE*8])Here we assume FRAME_SIZE is in bytes and "^" represents exponentiation. It includes the user data and all headers (TCP, IP and subnetwork). (Note: this analysis assumes the subnetwork does not perform ARQ or transparent fragmentation [RFC3366].) If the inequality
BER * [FRAME_SIZE*8] << 1holds, the packet loss probability p can be approximated by:
p = BER * [FRAME_SIZE*8]These equations can be used to apply BER to the performance equations above. 🧮
Note that FRAME_SIZE can vary from one packet to the next. Small packets (such as TCP acks) generally have a smaller probability of packet error than, say, a TCP packet carrying one MSS (maximum segment size) of user data. So a flow of small TCP acks can be expected to be slightly more reliable than a stream of larger TCP data segments. 🦋
It bears repeating that the above analysis assumes bit errors are statistically independent. Because this isn't true for many real links, our computation of p is actually an upper bound, not the exact probability of packet loss. 🫶
There are many reasons bit errors aren't independent on real links. Lots of radio links are affected by propagation fading or by interference that lasts over many bit times. Also, links with Forward Error Correction (FEC) generally have very non-uniform bit error distributions that depend on the type of FEC, but in general the uncorrected errors tend to occur in bursts even when channel symbol errors are independent. In all such cases, our computation of p from BER can only place an upper limit on the packet loss rate. ⚠️
If the distribution of errors under the FEC scheme is known, you could apply the same type of analysis as above, using the correct distribution function for the BER. It's more likely in these FEC cases, however, that empirical methods are needed to determine the actual packet loss rate. 🔬
- Note that the packet size plays an important role. If the subnetwork loss characteristics are such that large packets have the same probability of loss as smaller packets, then larger packets will yield improved performance. 📦
- We have chosen a specific RTT that might occur on a wide-area Internet path within the USA. It's important to recognize that a variety of RTT values are experienced on the Internet. 🌍
For example, RTTs are typically less than 10 msec in a wired LAN environment when communicating with a local host. International connections may have RTTs of 200 msec or more. Modems and other low-capacity links can add considerable delay due to their long packet transmission (serialisation) times. 🐢
Links over geostationary repeater satellites have one-way speed-of-light delays of around 250ms, a minimum of 125ms propagation delay up to the satellite and 125ms down. The RTT of an end-to-end TCP connection that includes such a link can be expected to be greater than 250ms. 🛰️
Queues on heavily-congested links may back up, increasing RTTs. Finally, virtual private networks (VPNs) and other forms of encryption and tunneling can add significant end-to-end delay to network connections. 🔐✨
9. Quality-of-Service (QoS) considerations ✨
okay bestie, real-time multimedia, toll-quality telephony, and other performance-critical apps genuinely need specific service guarantees 🎀. Making QoS work on the Internet is still a super active area of research and standardization 🔬. The IETF hasn't landed on one service model, one set of services, or one mechanism that gives apps useful guarantees AND scales to the whole Internet 😭. The IETF doesn't even have one single definition of Quality of Service 💀. [RFC2990] is the one that lays out the current understanding of the challenges in architecting QoS for the Internet 📚.
There are currently two architectural approaches to giving the Internet QoS mechanisms 🦋.
IP Integrated Services (Intserv) [RFC1633] gives fine-grained service guarantees to individual flows 💖. Flows get identified by a flow specification (flowspec), which creates a stateful association between packets by matching fields in the packet header 🧚. Capacity is reserved for the flow, and appropriate traffic conditioning and scheduling gets installed in routers along the path 👑. The ReSerVation Protocol (RSVP) [RFC2205] [RFC2210] is usually used to install the flow QoS state, but it doesn't have to be 🌸. Intserv defines two services, on top of the Default (best effort) service 🍓.
- Guaranteed Service (GS) [RFC2212] offers hard upper bounds on delay to flows that conform to a traffic specification (TSpec) 🔥. It uses a fluid-flow model to relate the TSpec and reserved bandwidth (RSpec) to variable delay 🧠. Non-conforming packets get forwarded on a best-effort basis 💅.
- Controlled Load Service (CLS) [RFC2211] gives flows that conform to a TSpec delay and packet loss equivalent to an unloaded network, but with no hard bounds 🌈. Non-conforming packets get forwarded on a best-effort basis too 🫶.
Intserv requires installing state information in every participating router 🙈. Performance guarantees cannot be made unless that state is present in every router along the path 😬. This, along with RSVP processing and the need for usage-based accounting, is believed to have scalability problems, particularly in the core of the Internet [RFC2208] 🥺.
IP Differentiated Services (Diffserv) [RFC2475] gives you a "toolkit" of coarse-grained controls for aggregates of flows ✨. Diffserv by itself does not provide QoS guarantees, but it can be used to build services that do have them across a Diffserv domain 💜. It tries to fix the scaling issues of Intserv by only requiring state awareness at the edge of a Diffserv domain 🦋. At the edge, packets get classified into flows, and the flows get conditioned (marked, policed, or shaped) to a traffic conditioning specification (TCS) 🎀. A Diffserv Codepoint (DSCP), which identifies a per-hop behavior (PHB), is set in each packet header 🏷️. The DSCP is carried in the DS-field, taking over six bits of the former Type-of-Service (ToS) byte [RFC791] of the IP header [RFC2474] 💅. The PHB is the forwarding behavior applied to the packet at each node in the Diffserv domain 🌸. There's a "recommended" DSCP associated with each PHB, but the mappings from DSCPs to PHBs get defined by the DS-domain 👑. Several DSCPs can even map to the same PHB 🙈. Diffserv presently defines three PHBs 🎀.
- The class selector PHB [RFC2474] replaces the IP precedence field of the former ToS byte, and offers relative forwarding priorities 🫶.
- The Expedited Forwarding (EF) PHB [RFC3246] [RFC3248] guarantees a well-defined minimum departure rate which, if not exceeded, keeps the associated queues short or empty 🚀. EF is meant to support services with tightly-bounded loss, delay, and delay jitter ✨.
- The Assured Forwarding (AF) PHB group [RFC2597] offers different levels of forwarding assurance for each aggregated flow of packets 💖. Each AF group gets its own independently allocated forwarding resources 🧁. Packets get marked with one of three drop precedences, and the ones with the highest drop precedence are dropped with lower probability than those marked with the lowest drop precedence 🌈. DSCPs are recommended for four independent AF groups, though a DS domain can have more or fewer AF groups 💜.
Ongoing work in the IETF is addressing ways to support Intserv with Diffserv 🔍. There's some belief (e.g., as expressed in [RFC2990]) that this approach will let individual flows get service guarantees and still scale to the global Internet 🌍.
The QoS guarantees the IP layer can offer are a product of two factors 🧮:
- the concatenation of the QoS guarantees offered by the subnets along the path of a flow. This implies that a subnet may wish to offer multiple services (with different QoS guarantees) to the IP layer, which can then determine which flows use which subnet service. To put it another way, forwarding behavior in the subnet needs to be "clued" by the forwarding behavior (service or PHB) at the IP layer, and
- the operation of a set of cooperating mechanisms, such as bandwidth reservation and admission control, policy management, traffic classification, traffic conditioning (marking, policing and/or shaping), selective discard, queuing, and scheduling. Note that support for QoS in subnets may require similar mechanisms, especially when these subnets are general topology subnets (e.g., ATM, frame relay, or MPLS) or shared media subnets.
Many subnetwork designers face inherent tradeoffs between delay, throughput, reliability, and cost 😵💫. Other subnetworks have parameters that manage bandwidth, internal connection state, and the like. So the following subnetwork capabilities may be desirable, although some might be trivial or moot if the subnet is a dedicated point-to-point link 🔗.
- The subnetwork should have the ability to reserve bandwidth for a connection or flow and schedule packets accordingly 📅.
- Bandwidth reservations should be based on a one- or two-token bucket model, depending on whether the service is intended to support constant-rate or bursty traffic 🪣.
- If a connection or flow does not use its reserved bandwidth at a given time, the unused bandwidth should be available for other flows 🤝.
- Packets in excess of a connection or flow's agreed rate should be forwarded as best-effort or discarded, depending on the service offered by the subnet to the IP layer 🗑️.
- If a subnet contains error control mechanisms (retransmission and/or FEC), it should be possible for the IP layer to influence the inherent tradeoffs between uncorrected errors, packet losses, and delay. These capabilities at the subnet/IP layer service boundary correspond to selection of more or less error control and/or to selection of particular error control mechanisms within the subnetwork 🛠️.
- The subnet layer should know, and be able to inform the IP layer, how much fixed delay and delay jitter it offers for a flow or connection. If the Intserv model is used, the delay jitter component may be best expressed in terms of the TSpec/RSpec model described in [RFC2212] 📏.
- Support of the Diffserv class selectors [RFC2474] suggests that the subnet might consider mechanisms that support priorities 🥇.
10. Fairness vs Performance 💅
Subnetwork designers should be aware of the tradeoffs between fairness and efficiency inherent in many transmission scheduling algorithms ⚖️. For example, many local area networks use contention protocols to resolve access to a shared transmission channel. These protocols represent overhead 😩. Limiting how much data a subnet node may transmit per contention cycle helps assure timely access to the channel for each subnet node, but it also increases contention overhead per unit of data sent 🙃.
In some mobile radio networks, capacity is limited by interference, which in turn depends on average transmitter power 📡. Some receivers may require considerably more transmitter power (generating more interference and consuming more channel capacity) than others 🔋.
In each case, the scheduling algorithm designer has to balance competing objectives: giving each subnet node a fair share of capacity while maximizing the total capacity of the network 🎯. One approach for balancing performance and fairness is outlined in [ES00] 🦋.
11. Delay Characteristics 🕐✨
okay so bestie, TCP's sender sets its retransmission timeout (RTO) by measuring the round trip delay that earlier packets actually experienced 💅 that's how it adapts to the super wide range of delays you find across the Internet 🌈 the recommended algorithms are in [RFC2988], and if you want to see people evaluating TCP's retransmission timer, check out [AP99] and [LS00] 📚
these algorithms treat the delay along an Internet path like a normally-distributed random variable, with a mean and standard deviation that drift slowly over time 📈 TCP estimates those two numbers by exponentially smoothing each individual delay measurement, then sets the RTO to the estimated mean delay plus some fixed number of standard deviations 🎀 (the algorithm actually uses mean deviation as a stand-in for standard deviation, because it's easier to compute 🙈)
the goal is giving "small enough to catch lost packets, but not so small that we spuriously retransmit packets that were just running late" 💖 those goals fight each other, but the algorithm works really well when the delay variance on the path is low, or when the packet loss rate is low 🦋
if the path's delay variance is high, TCP picks an RTO way bigger than the mean of the measured delays 😭 but if the loss rate is low, that big RTO doesn't matter much, since timeouts only happen rarely 🌸 and if the delay variance is low, TCP recovers from lost packets quickly, so again it's working great ✨ the tea is that when delay variance AND packet loss rate are both high, these algorithms perform poorly, especially if the mean delay is high too 💀
because TCP uses the returning acknowledgments as a "clock" to time when it sends more data, really high delays (even with low variance) also mess with TCP's ability to fully use a high-speed pipe 🚀 they slow down recovery of lost packets too, even when the variance is small 🙃
so subnetwork designers should minimize all three things: delay, delay variance, and packet loss, as much as they possibly can 🫶
but ngl, in lots of subnetworks these three are literally in conflict 😤 like on a mobile radio channel, the designer can use retransmission (ARQ) and/or forward error correction (FEC) to trade off delay, delay variance, and packet loss, all to try to make TCP happier 🔥 ARQ increases delay variance, but FEC does not 🧚 however, FEC (especially combined with interleaving) often increases mean delay, even on good channels where ARQ retransmissions aren't needed at all, and where ARQ wouldn't increase delay or delay variance either 🙈
the tradeoffs between these error control mechanisms, and how they interact with TCP, are super complex and still the subject of a lot of ongoing research 🔬 so we recommend that subnetwork designers give these mechanisms as much flexibility as possible, and provide access to them as discussed above in the section on Quality of Service 🎀💜
12. Bandwidth Asymmetries 🦋
some subnetworks give asymmetric bandwidth (or make TCP packet flows experience asymmetric capacity), and the Internet protocol suite will generally still work just fine 🌸 BUT there's one case where that kind of setup can actually tank TCP performance 😭 TCP data segments get "clocked" out by the returning acknowledgments, so TCP senders are limited by how fast ACKs can make it back [BPK98] 🐢
so when the available capacity of the Internet path carrying the data is too big compared to the bandwidth of the acknowledgment return path, the slow ACKs directly hurt performance 💀 ACKs are usually smaller than data segments, so TCP can handle some asymmetry 💅 but as a general rule, subnetwork designers should know that subnetworks with significant asymmetry can cause reduced performance, unless they take steps to mitigate it [RFC3449] ✨
there are several strategies for reducing the impact of asymmetry on the path between two TCP end hosts, like the ones in [RFC3449] 🧚 these techniques try to cut down the number of ACKs sent over the return path (the low bandwidth channel) by making changes at the end hosts, and/or by changing how the subnetwork forwards packets 🎀 they can help with the asymmetry problems, BUT they come with costs and may have other implications too 🙈 a fuller discussion of the strategies and their implications is in [RFC3449], so go check that out bestie 💖
13. Buffering, flow and congestion control ✨
Okay bestie, so subnets are often a whole mix of links, with different traffic demands and sometimes different speeds 🦋 Every link there must have a queuing system, which means buffering, scheduling, and a way to discard excess subnet packets 🗑️ Those queues may also be part of a subnet flow control or congestion control scheme, so they're kind of multitaskers 💅
For this discussion, we're talking about packets without caring whether they're a complete IP packet or a subnetwork frame 🌸 At each queue, a packet waits for a delay that depends on competing traffic and the scheduling discipline, and it also gets hit by a local discarding policy 💀
Some subnets have flow or congestion control mechanisms on top of packet dropping 🎀 These can work on subnet-layer parts like schedulers, shapers, or discarders, and they can affect how IP forwarders at the edges of the subnet behave. Still, the only exception is Explicit Congestion Notification [RFC3168] (more on that below), because IP has no way to pass explicit congestion or flow control signals to TCP 😭
TCP traffic is super bursty, especially when it's aggregated, so instantaneous queue depths can swing hard even on networks that look stable 📈 For the best performance, packets should be dropped in a controlled way, not only when there's no buffer space left 🫶 How much buffer space to supply is still up for debate, but as a rule of thumb, each node should have enough buffering to hold one link_bandwidth*link_delay product's worth of data for each TCP connection sharing the link ✨
This is honestly hard to estimate, because it depends on stuff the subnet doesn't control or even know about 🙈 Internet nodes generally don't implement admission control policies, so they can't limit how many TCP connections use them. In general, it's wise to err on the side of too much buffering rather than too little 💜 It may also help for subnets to include mechanisms that measure propagation delays, which can assist with buffer sizing calculations 🧚
There's a rough consensus in the research community that active queue management matters for fairness, link utilization, and throughput [RFC2309] 👑 There are questions and concerns about how well active queue management works (e.g., [MBDL99]), but it's widely considered an improvement over tail-drop discard policies 🔥
One form of active queue management is the Random Early Detection (RED) algorithm [RED93], which is actually a family of related algorithms 🌈 In one version of RED, an exponentially-weighted moving average of the queue depth is maintained:
- When this average queue depth is between a maximum threshold max_th and a minimum threshold min_th, the probability of packets that are dropped is proportional to the amount by which the average queue depth exceeds min_th.
- When this average queue depth is equal to max_th, the drop probability is equal to a configurable parameter max_p.
- When this average queue depth is greater than max_th, packets are always dropped.
Tbh there are loads of RED variants out there, plus other active queue management algorithms that claim various advantages over RED [GM02] 💅
With active queue management, dropped packets become a feedback signal that tells the TCPs in the end hosts to change their congestion behavior 🎯 Randomizing the drops tends to break up the tendency of TCP windows from different connections to synchronize because of correlated drops 🦋 It also imposes a degree of fairness on connections that implement TCP congestion avoidance properly 💖 Another big win is that active queue management tries to keep average queue depths short while still handling large short-term bursts ✨
TCP doesn't know or care whether congestive packet loss happens at the IP layer or inside a subnet, so it may be a good idea for subnets that queue and discard to consider some form of active queue management 🍓 This is especially true when large aggregates of TCP connections are likely to share the same queue. However, active queue management may be less effective when there are many queues carrying smaller aggregates of TCP connections, for example in an ATM switch that implements per-VC queuing 🙃
Also, the performance of active queue management algorithms is highly sensitive to the settings of their configurable parameters, and to factors like RTT [MBB00] [FB00] 🧐
Some subnets, most notably ATM, do segmentation and reassembly at the subnetwork edges, so discard policies need extra care here 😬 If the subnet discards a fragment of an IP packet, the remaining fragments become an unproductive load on the subnet, and that can seriously degrade end-to-end performance [RF95] 💀 So subnetworks should try to discard these extra fragments whenever one of them has to be discarded. If the IP packet has already been partially forwarded when discarding becomes necessary, then every remaining fragment except the one marking the end of the IP packet should also be discarded. For ATM subnets, this specifically means using Early Packet Discard and Partial Packet Discard [ATMFTM] 🎀
Some subnets include flow control mechanisms that basically require the rate of traffic flows to be shaped when they enter the subnet 🫧 One example is the ATM Available Bit rate (ABR) service category [ATMFTM] 🦋 These flow control mechanisms make the subnet nearly lossless by pushing congestion into the IP routers at the edges of the subnet 😤 In that case, those routers need adequate buffering and discard policies to deal with a subnet that appears to have varying bandwidth 🌸
Whether this kind of flow control actually helps is controversial, and there are loads of simulation and analytical studies that go both ways 🙈 It seems some of the reasons for these different results include sensitivity to ABR parameters, whether binary or explicit rate feedback is used, whether per-VC queuing is used (or not), and which specific ATM switch algorithms were picked for the study 🤓 Anecdotally, some large networks that used IP over ABR to carry TCP traffic have claimed it's been successful, but they haven't published any results 🤷♀️
Another possible approach to flow control in the subnet would be to work with TCP Explicit Congestion Notification (ECN) semantics [RFC3168] by using explicit congestion indicators in subnet frames 🧚 Instead of shaping, routers at the edges of the subnet would set the explicit congestion bit in those IP packets that were received in subnet frames carrying an ECN indication. Nodes in the subnet would need to implement an active queue management protocol that marks subnet frames instead of dropping them 💖
ECN is currently a proposed standard, but it's not yet widely deployed 🌈
14. Compression ✨
okay so application data compression is something that can usually be skipped in the subnetwork 🫶. The endpoints typically have way more CPU and memory to run a compression algorithm, plus they understand what's being compressed much better 🧠. End-to-end compression helps every network element on the path, while subnetwork-layer compression, by definition, only helps a single subnetwork 💅.
Data handed to the subnetwork layer might already be compressed 🎀, like a JPEG file, or compressed at the application layer (think the optional "gzip", "compress", and "deflate" compression in HTTP/1.1 [RFC2616]), or compressed at the IP layer (the IP Payload Compression Protocol [RFC3173] supports DEFLATE [RFC2394] and LZS [RFC2395]). Compressing at the subnetwork edges is no help for any of these cases 😭.
The subnetwork might also process data the application encrypted 🔐: OpenPGP [RFC2440] or S/MIME [RFC2633], just above TCP (SSL, TLS [RFC2246]), or just above IP (IPsec ESP [RFC2406]).
Ciphers generate high-entropy bit streams with no patterns a compression algorithm can exploit 🙈.
BUT, much data is still sent uncompressed over the Internet, so subnetwork compression may be beneficial 🌈. Any subnetwork compression algorithm must not expand uncompressible data, e.g., data that has already been compressed or encrypted ❌.
We make a strong recommendation that subnetworks operating at low speed or with small MTUs compress IP and transport-level headers (TCP and UDP) using several header compression schemes developed within the IETF [RFC3150]. An uncompressed 40-byte TCP/IP header takes about 33 milliseconds to send at 9600 bps 🐢. "VJ" TCP/IP header compression [RFC1144] squishes most headers down to 3-5 bytes, so transmission time drops to several milliseconds on dialup modem links ⚡. This is especially helpful for small, latency-sensitive packets in interactive sessions 💬.
Similarly, RTP compression schemes like CRTP [RFC2508] and ROHC [RFC3095] compress most IP/UDP/RTP headers to 1-4 bytes 🎧. The savings are especially big when audio packets are kept small to cut store-and-forward latency 🎵.
Designers should consider the effect of subnetwork error rate on header compression performance 🧐. TCP normally recovers from lost packets by retransmitting only the ones that were actually lost. Packets arriving correctly after a loss sit on a resequencing queue and don't need retransmitting 🙌. But with VJ TCP/IP [RFC1144] header compression, the receiver can't explicitly tell the sender about data corruption and the resulting loss of sync between compressor and decompressor 😩. It relies on TCP retransmission to resync the decompressor instead. After a packet is lost, the decompressor has to throw away every later packet, even if the subnetwork makes no more errors, until the sending TCP retransmits to resync it. This can really magnify subnetwork losses if the sending TCP window is large, which it often is on a path with a large bandwidth*delay product [LRKOJ99] 📈.
Alternate schemes like those in [RFC2507] include an explicit request for retransmission of an uncompressed packet, so decompressors can resync without waiting for a TCP retransmission 🔄. However, these schemes aren't in widespread use yet 🤷♀️.
Both TCP header compression schemes don't compress widely used TCP options like selective acknowledgements (SACK) 🙅. Both also fail to compress TCP traffic that uses explicit congestion notification (ECN). Work is under way in the IETF ROHC WG to fix these shortcomings in a ROHC header compression scheme for TCP [RFC3095] [RFC3096] 🛠️.
Subnetwork error rate matters for RTP header compression too 🎤. CRTP uses delta encoding, so a packet loss on the link creates uncertainty about subsequent packets. Those often must be discarded until the decompressor has notified the compressor and the compressor has sent resync info. That typically takes slightly more than the end-to-end path round-trip time ⏳. For links that combine significant error rates with latencies that require multiple packets in flight at once, this leads to significant error propagation, i.e., subsequent losses caused by an initial loss 💥.
For links that are both high-latency (multiple packets in flight from a typical RTP stream) and error-prone, RTP ROHC offers a more robust way of RTP header compression, at the cost of higher complexity at the compressor and decompressor 🧩. For example, within a talk spurt, only extended losses of (depending on the mode chosen) 12-64 packets typically cause error propagation 🗣️.
15. Packet Reordering ✨
The Internet architecture doesn't guarantee that packets arrive in the same order they were sent, so transport protocols like TCP have to account for this 🔀.
However, reordering does come at a cost with TCP as it's currently defined 😬. Because TCP returns a cumulative acknowledgment (ACK) for the last in-order segment that arrived, out-of-order segments make a TCP receiver send a duplicate acknowledgment. When the TCP sender sees three duplicate acknowledgments, it assumes a segment was dropped by the network and uses the fast retransmit algorithm [Jac90] [RFC2581] to resend it. On top of that, the congestion window gets cut in half, effectively halving TCP's sending rate 📉. If a subnetwork reorders segments enough to generate three duplicate ACKs, the TCP sender needlessly shrinks the congestion window and performance suffers 😤.
Packet reordering happens a lot in parts of the Internet and seems hard or impossible to eliminate [BPS99]. So research on improving TCP's behavior when reordering happens [LK00] [BA02] has begun 🔬.
[BPS99] cites reasons it may even be undesirable to eliminate reordering 🤯. There are situations where average packet latency can drop, link efficiency can go up, and/or reliability can improve if reordering is permitted. Examples include certain high speed switches in the Internet backbone and the parallel links many Internet paths use for load splitting and redundancy 🛤️.
This suggests subnetwork implementers should try to avoid packet reordering whenever possible, but not if doing so compromises efficiency, impairs reliability, or increases average packet delay 💡.
Note that every header compression scheme currently standardized for the Internet requires in-order packet delivery on the link between compressor and decompressor 📦. PPP is frequently used to carry compressed TCP/IP packets. Since it was originally designed for point-to-point and dialup links, it's assumed to provide in-order delivery. For this reason, subnetwork implementers who provide PPP interfaces to VPNs and other more complex subnetworks must also maintain in-order delivery of PPP frames 🔒.
16. Mobility ✨
Internet users are increasingly mobile 📱. Not only are many Internet nodes laptops, but pocket organizers and mobile embedded systems are becoming nodes too 🧚. These nodes may connect to many different access points over time, and they expect this to be pretty transparent to what they're doing. Except when they aren't connected to the Internet at all, and except for performance differences when they are connected, they expect everything to just work regardless of their current attachment point or local subnetwork technology 🦋.
Changing a host's Internet attachment point involves one or more of the following steps 👀.
First, if use of the local subnetwork is restricted, the user's credentials must be verified and access granted. There are many ways to do this. A trivial example is an "Internet cafe" that grants physical access to the subnetwork for a fee ☕. Subnetworks may implement technical access controls of their own; one example is IEEE 802.11 Wireless Equivalent Privacy [IEEE80211]. It's common practice for both cellular telephone and Internet service providers (ISPs) to agree to serve one another's users. RADIUS [RFC2865] is the standard method for ISPs to exchange authorization information 🤝.
Second, the host may have to be reconfigured with IP parameters appropriate for the local subnetwork. This usually includes setting an IP address, default router, and domain name system (DNS) servers ⚙️.
On multiple-access networks, the Dynamic Host Configuration Protocol (DHCP) [RFC2131] is almost universally used for this purpose. On PPP links, these functions are performed by the IP Control Protocol (IPCP) [RFC1332] 📡.
Third, traffic destined for the mobile host must be routed to its current location. This roaming function is the most common meaning of the term "Internet mobility" 🗺️.
Internet mobility can be provided at any of several layers in the Internet protocol stack, and there's ongoing debate about which is the most appropriate and efficient 🥊. Mobility is already a feature of certain application layer protocols. The Post Office Protocol (POP) [RFC1939] and the Internet Message Access Protocol (IMAP) [RFC3501] were created specifically to provide mobility in receiving email 📬.
Mobility can also be provided at the IP layer [RFC3344]. This gives greater transparency, viz., IP addresses that stay fixed as the nodes move. The cost is potentially significant network overhead and increased delay, because of the sub-optimal network routing and tunneling involved 🐢.
Some subnetworks may provide internal mobility, transparent to IP, as a feature of their own internal routing mechanisms. To the extent these simplify routing at the IP layer, reduce the need for mechanisms like Mobile IP, or exploit mechanisms unique to the subnetwork, this is generally desirable 💖. This is especially true when the subnetwork covers a relatively small geographic area and users move rapidly between attachment points within that area. Examples include Ethernet switching and intra-system handoff in cellular telephony 📶.
However, if the subnetwork is physically large and connects to other parts of the Internet at multiple geographic points, care should be taken to optimize the wide-area routing of packets between nodes on the external Internet and nodes on the subnet. This is generally done with "nearest exit" routing strategies. Because a given subnetwork may be unaware of the actual physical location of a destination on another subnetwork, it simply routes packets bound for the other subnetwork to the nearest router between the two. This implies some awareness of IP addressing and routing within the subnetwork. The subnetwork may wish to use IP routing internally for wide area routing and restrict subnetwork-specific routing to constrained geographic areas where the effects of suboptimal routing are minimized 🌸.
17. Routing 🛣️✨
okay bestie, any subnetwork connecting more than two systems must provide its own internal Layer-2 forwarding 🔥, either implicitly (e.g., broadcast) or explicitly (e.g., switched). 💅 And since routing is the major function of the Internet layer, it's giving big "who does what" energy: how does routing at the Internet layer interact with routing inside the subnet, and how should the job be split between the two? 🤔
Layer-2 subnetworks can be point-to-point, connecting two systems, or multipoint. 🦋 Multipoint subnetworks can be broadcast (e.g., shared media or emulated) or non-broadcast. Generally, IP treats multipoint subnetworks as broadcast, with shared-medium Ethernet as the canonical (and historical) example 🌈, and point-to-point subnetworks as a degenerate case. Non-broadcast subnetworks may require additional mechanisms, e.g., above IP at the routing layer [RFC2328]. 🧚
IP is totally ignorant of the topology of the subnetwork layer 🙈. In particular, reconfiguration of subnetwork paths is not tracked by the IP layer. IP only cares whether it can send and receive packets to the remotely connected systems via the subnetwork interface (i.e., the reachability from one router to another). IP also assumes subnetworks are largely static 💜, meaning both their membership and existence are stable at routing timescales (tens of seconds). Changes to these are considered re-provisioning, rather than routing. 🛠️
Routing functionality in a subnetwork is related to addressing in that subnetwork. 🏠 Resolution of addresses on subnetwork links is required for forwarding IP packets across links (e.g., ARP for IPv4, or ND for IPv6). There is unlikely to be direct interaction between subnetwork routing and IP routing. Where broadcast is provided or explicitly emulated, address resolution can be used directly. Where it's not provided, the link layer routing may interface to a protocol for resolution, e.g., to the Next-Hop Resolution Protocol [RFC2322] to provide context-dependent address resolution capabilities. 🍓
Subnetwork routing can either complement or compete with IP routing. 🤝 It complements IP when a subnetwork encapsulates its internal routing, and the effects of that routing are not visible at the IP layer. 🫶 However, if different paths in the subnetwork have characteristics that affect IP routing, it can affect or even inhibit the convergence of IP routing. 😭
Routing protocols generally consider Layer-2 subnetworks (i.e., with subnet masks and no intermediate IP hops) to have uniform routing metrics to all members. 📏 Routing can break when a link's characteristics don't match the routing metric, for example when some member pairs have different path characteristics. Picture this: a virtual Ethernet subnetwork that includes both nearby (sub-millisecond latency) and remote (100's of milliseconds away) systems. 🙀 Presenting that group as a single subnetwork means some routing protocols will assume all pairs have the same delay, and that the delay is small. Because this is not the case, the routing tables constructed may be suboptimal or may even fail to converge. 💀
When a subnetwork is used for transit between a set of routers, it conventionally provides the equivalent of a full mesh of point-to-point links. 🕸️ Simplicity of the internal subnet structure can be used (e.g., via NHRP [RFC2332]) to reduce the size of address resolution tables, but routing exchanges will continue to reflect the full mesh they emulate. In general, subnetworks should not be used as transit among a set of routers where routing protocols would break if a full mesh of equivalent point-to-point links were used. 🚫
Some subnetworks have special features that allow the use of more effective or responsive routing mechanisms, which can't be implemented in IP because IP needs to stay general. ✨ One example is the self-learning bridge algorithm widely used in Ethernet networks. Learning bridges perform Layer-2 subnetwork forwarding, avoiding the need for dynamic routing at each subnetwork hop. 🌸 Another is the "handoff" mechanism in cellular telephone networks, particularly the "soft handoff" scheme in IS-95 CDMA. 📱
Subnetworks that cover large geographic areas or include links of widely-varying capabilities should be avoided. 🌍 IP routing generally considers all multipoint subnets equivalent to a local, shared-medium link with uniform metrics between any pair of systems, and ignores internal subnetwork topology. Where a subnetwork diverges from that assumption, it is the obligation of subnetwork designers to provide compensating mechanisms. Not doing so can affect the scalability and convergence of IP routing, as noted above. 😬
The subnetwork designer who decides to implement internal routing should consider whether a custom routing algorithm is warranted, or whether an existing Internet routing algorithm or protocol may suffice. 🧐 The designer should also consider whether this decision is meant to reduce the address resolution table size (possible, but with additional protocol support required), or is trying to reduce routing table complexity. The latter may be better achieved by partitioning the subnetwork, either physically or logically, and using network-layer protocols to support partitioning (e.g., AS's in BGP). 🧩 Protocols and routing algorithms can be notoriously subtle, complex, and difficult to implement correctly, so bestie, much work can be avoided if existing protocols or implementations can be readily reused. 👑🫶
18. Security Considerations 🔐
okay bestie, security is a HUGE priority for the Internet rn 💅 It's massive, and loads of different orgs and people own and run different bits of it. The group basically agreed on a "security placement principle" 🫶 The idea is that a security mechanism works best when it sits as close as possible to the asset it protects, and when the owner of that asset is the one in control of it.
So here's the corollary: you can't get end-to-end security (like confidentiality, authentication, integrity, and access control) using subnetwork security mechanisms alone 🙈 End-to-end mechanisms are way more tightly tied to the end-user stuff they protect, AND they cover more ground. Like, they fill the gaps that show up when perfectly fine subnetwork mechanisms get strung together. This is a really important use of the end-to-end principle [SRC81] ✨
A bunch of end-to-end security mechanisms are already out there and getting more and more love 💖 The big ones are:
- Secure Sockets Layer (SSL) [SSL2] [SSL3] and TLS [RFC2246], mostly used to protect web commerce
- Pretty Good Privacy (PGP) [RFC1991] and S/MIME [RFCs-2630-2634], mostly used to protect and authenticate email and software distributions
- Secure Shell (SSH), used for secure remote access and file transfer
- IPsec [RFC2401], a general purpose encryption and authentication mechanism that sits just above IP so any IP application can use it. (IPsec can actually run either end-to-end or between security gateways that don't include one or both end systems.)
Even so, end-to-end security isn't used as much as it probably should be 😭 The group couldn't agree on whether subnetwork designers should be actively encouraged to build mechanisms that protect user data.
The working group's clear consensus was that subnetwork security mechanisms, especially weak or badly implemented ones [BGW01], might actually backfire 💀 The worry is that they can lull end users into a false sense of security, make people less motivated to deploy good end-to-end mechanisms, and encourage "risky" uses of the Internet that people wouldn't attempt if they really understood the built-in limits of subnetwork security.
The other side says subnetwork security is still worth it because it beats the default, which is all too often no security at all 🌸 Users of especially vulnerable subnets (like consumers with wireless home networks and/or shared media Internet access) often control at most one endpoint, usually a client, so they can't enforce end-to-end mechanisms on their own. Subnet security can be totally adequate for protecting low-value assets against the most likely threats. Either way, subnet mechanisms don't rule out end-to-end ones, which are usually what gets used to protect high-value assets. This view also notices that lots of security policies quietly assume the whole end-to-end path is just a chain of links that are physically secured. In other words, those policies assume every link endpoint is trusted and that attackers find it hard to get at the physical medium. To make those policies work, links that lack physical protection (especially shared medium links) need explicit mechanisms. That's the reasoning behind Wired Equivalent Privacy (WEP) in the IEEE 802.11 [IEEE80211] wireless LAN standard, and the Baseline Privacy Interface in the DOCSIS [DOCSIS1] [DOCSIS2] data over cable television networks standards 🎀
So our recommendation is that subnetwork designers who choose to build security mechanisms to protect user data be as candid as they can about how those mechanisms work and about the inherent limits of even the most secure mechanism when it's implemented in a subnetwork instead of end-to-end 🫶
Following the "placement principle," there's also a clear consensus on another subnetwork security job: protecting the subnetwork itself ✨ Threats to subnetwork assets include theft of service and denial of service, and shared media subnets tend to be especially exposed to these attacks. In some cases, mechanisms that protect subnet assets can also improve (but can't guarantee) end-to-end security.
There's one service the subnetwork can provide that helps with a bigger Internet problem: subnetwork security should provide a way to authenticate the source of a subnetwork frame 🧚 Some current protocols are missing this. For example, ARP [RFC826] is used to link an IPv4 address to a MAC address, and it doesn't have it. The IPv6 Neighbor Discovery (ND) [RFC2461] does something similar.
ARP has well-known security flaws [Wilbur89]. Still, including subnetwork frame source authentication would make a secure subnetwork address possible 🔥
Another potential job for subnetwork security is protecting users against traffic analysis 👀 That's figuring out who is talking to whom, plus their communication patterns and volumes, even when the contents are locked down by strong end-to-end security. Lower-layer security can be more effective here because it can pool the communications of multiple parties sharing the same physical facilities, while hiding higher-layer protocol information that points to specific endpoints, like IP addresses and TCP/UDP port numbers.
Traffic analysis is notoriously sneaky and hard to understand and beat, way more so than threats to confidentiality and integrity 🙈 So we really urge extreme care when designing subnetwork security mechanisms that are specifically meant to stop traffic analysis.
Subnetwork designers have to remember that designing and implementing for security is hard [Schneier00]. [Schneier95] describes protocols and algorithms that are considered well-understood and believed to be sound 🤓
Poor design process, subtle design errors, and flawed implementation can all lead to gaping vulnerabilities 💀 Over the last few years, a bunch of subnet standards have had problems exposed. Here are some examples of mistakes that have actually happened:
- Using weak and untested algorithms [Crypto9912] [BGW01]. For a variety of reasons, some algorithms were picked that had subtle flaws, making them open to all kinds of attacks.
- Relying on "security by obscurity" [Schneier00] [Crypto9912]. A super common mistake is assuming that keeping cryptographic algorithms secret makes them more secure. It sounds logical, but it's wrong 😤 Full public disclosure early in the design process invites peer review from knowledgeable cryptographers. The flaws that review exposes are worth way more than any imagined benefit from making attackers reverse engineer the algorithm.
- Including trapdoors [Schneier00] [Crypto9912]. Trapdoors are flaws that someone deliberately slips into an algorithm so it can be broken. This might be done to recover lost keys or to allow secret access by government agencies. Trapdoors can also be found and exploited by malicious attackers.
- Sending passwords or other identifying information in clear text. For years, analog cellular phones could be cloned and used to steal service. The cloners just eavesdropped on registration protocols that exchanged everything in clear text.
- Using keys that are shared by all systems on a subnet [BGW01].
- Using a sound mechanism incorrectly. For example [BGW01], one subnet standard includes an initialization vector that's poorly designed and poorly specified. A determined attacker can easily recover multiple ciphertexts encrypted with the same key stream and run statistical attacks to decipher them.
- Sending identifying information in clear text that can be traced back to an individual, identifiable device. This opens the door to attacks aimed at that device (or its owner).
- Being unable to renew and revoke shared secret information.
- Using insufficient key length.
- Failing to address "man-in-the-middle" attacks, for example with mutual authentication.
- Failing to provide some form of replay detection, for example to stop a receiver from accepting packets from an attacker who just resends previously captured network traffic.
- Failing to provide integrity mechanisms when providing confidentiality schemes [Bel98].
This list is nowhere near complete 🫠 Design problems are hard to avoid, but expert review is usually invaluable for catching them.
Even well-designed security protocols can be broken by implementation defects 😭 Examples include using predictable pseudo-random numbers [RFC1750], being vulnerable to buffer overflow attacks because of unsafe use of certain I/O system calls [WFBA2000], and inadvertently exposing secret data.
19. Contributors 🌈
This document is the consensus of the members of the IETF Performance Implications of Link Characteristics (PILC) working group 💜
This document would not have been possible without a huge number of contributions from people in the Performance Implications of Link Characteristics Working Group 🫶 Some people gave major contributions of text, editing, and advice: Mark Allman provided the final editing to complete this document. Carsten Bormann provided text on robust header compression. Gorry Fairhurst provided text on broadcast and multicast issues, routing, and many valuable comments on the entire document. Aaron Falk provided text on bandwidth on demand. Dan Grossman provided text on many facets of the document. Reiner Ludwig provided thorough document review and text on TCP vs. Link-Layer Retransmission. Jamshid Mahdavi provided text on TCP performance calculations. Saverio Mascolo provided feedback on the document. Gabriel Montenegro provided feedback on the document. Marie-Jose Montpetit provided text on bandwidth on demand. Joe Touch provided text on multicast, broadcast, and routing, and Lloyd Wood provided many valuable comments on versions of the document 👑
20. Informative References 📚
References of the form RFCnnnn are Internet Request for Comments (RFC) documents available online at www.rfc-editor.org 🍓
[802.1D]Information Technology Telecommunications and information exchange between systems Local and metropolitan area networks, Common specifications Media access control (MAC) bridges, IEEE 802.1D, 1998. ISO 15802-3.[802.1p]IEEE, 802.1p, Standard for Local and Metropolitan Area Networks - Supplement to Media Access Control (MAC) Bridges: Traffic Class Expediting and Multicast.[AP99]Allman, M. and V. Paxson, On Estimating End-to-End Network Path Properties, In Proceedings of ACM SIGCOMM 99.
okay so the reference list keeps going and it's giving library energy 📚✨ here's the next batch, every citation exactly as it was, bestie 🫶
[AR02]Acar, G. and C. Rosenberg, Weighted Fair Bandwidth-on-Demand (WFBoD) for Geo-Stationary Satellite Networks with On-Board Processing, Computer Networks, 39(1), 2002.[ATMFTM]The ATM Forum, "Traffic Management Specification, Version 4.0", April 1996, document af-tm-0056.000.http://www.atmforum.com/[BA02]Blanton, E. and M. Allman, On Making TCP More Robust to Packet Reordering. ACM Computer Communication Review, 32(1), January 2002.[Bel98]Bellovin, S., "Cryptography and the Internet", in Proceedings of CRYPTO '98, August 1998.http://www.research.att.com/~smb/papers/inet-crypto.pdf[BGW01]Borisov, N., Goldberg, I. and D. Wagner, "Intercepting Mobile Communications: The Insecurity of 802.11," In Proceedings of ACM MobiCom, July 2001.[BPK98]Balakrishnan, H., Padmanabhan, V. and R. Katz. "The Effects of Asymmetry on TCP Performance." ACM Mobile Networks and Applications (MONET), 1998.[BPS99]Bennet,, J.C.R., Partridge, C. and N. Shectman, "Packet Reordering is Not Pathological Network Behavior", IEEE/ACM Transactions on Networking, Vol. 7, No. 6, December 1999.[CGMP]Farinacci D., Tweedly A. and T. Speakman, "Cisco Group Management Protocol (CGMP)", 1996/1997.ftp://ftpeng.cisco.com/ipmulticast/specs/cgmp.txt[Crypto9912]Schneier, B., "European Cellular Encryption Algorithms" Crypto-Gram, December 15, 1999.http://www.counterpane.com[DIX82]Digital Equipment Corp, Intel Corp, Xerox Corp, Ethernet Local Area Network Specification Version 2.0, November 1982.[DOCSIS1]Data-Over-Cable Service Interface Specifications, Radio Frequency Interface Specification 1.0, SP-RFI-I05-991105, November 1999, Cable Television Laboratories, Inc.[DOCSIS2]Data-Over-Cable Service Interface Specifications, Radio Frequency Interface Specification 1.1, SP-RFIv1.1-I05-000714, July 2000, Cable Television Laboratories, Inc.[DOCSIS3]Lai, W.S., "DOCSIS-Based Cable Networks: Impact of Large Data Packets on Upstream Capacity", 14th ITC Specialists Seminar on Access Networks and Systems, Barcelona, Spain, April 25-27, 2001.[EN301192]ETSI, European Broadcasting Union, Digital Video Broadcasting (DVB); DVB Specification for Data Broadcasting, European Standard (Telecommunications Series) EN 301 192 v1.2.1(1999-06).[ES00]Eckhardt, D. and P. Steenkiste, "Effort-limited Fair (ELF) Scheduling for Wireless Networks, Proceedings of IEEE Infocom 2000.[FB00]Firoiu V. and M. Borden, "A Study of Active Queue Management for Congestion Control" to appear in Infocom 2000.[GM02]Grieco1, L. and S. Mascolo, "TCP Westwood and Easy RED to Improve Fairness in High-Speed Networks", Proceedings of the 7th International Workshop on Protocols for High-Speed Networks, April 2002.[IEEE8023]IEEE 802.3 CSMA/CD Access Method.http://standards.ieee.org/[IEEE80211]IEEE 802.11 Wireless LAN standard.http://standards.ieee.org/[ISO3309]ISO/IEC 3309:1991(E), "Information Technology - Telecommunications and information exchange between systems - High-level data link control (HDLC) procedures - Frame structure", International Organization For Standardization, Fourth edition 1991-06-01.[ISO13818]ISO/IEC, ISO/IEC 13818-1:2000(E) Information Technology - Generic coding of moving pictures and associated audio information: Systems, Second edition, 2000-12-01 International Organization for Standardization and International Electrotechnical Commission.[ITU-I363]ITU-T I.363.5 B-ISDN ATM Adaptation Layer Specification Type AAL5, International Standards Organisation (ISO), 1996.[Jac90]Jacobson, V., Modified TCP Congestion Avoidance Algorithm. Email to the end2end-interest mailing list, April 1990.ftp://ftp.ee.lbl.gov/email/vanj.90apr30.txt[KY02]Khafizov, F. and M. Yavuz, Running TCP Over IS-2000, Proceedings of IEEE ICC, 2002.[LK00]Ludwig, R. and R. H. Katz, "The Eifel Algorithm: Making TCP Robust Against Spurious Retransmissions", ACM Computer Communication Review, Vol. 30, No. 1, January 2000.[LKJK02]Ludwig, R., Konrad, A., Joseph, A. D. and R. H. Katz, "Optimizing the End-to-End Performance of Reliable Flows over Wireless Links", Kluwer/ACM Wireless Networks Journal, Vol. 8, Nos. 2/3, pp. 289-299, March-May 2002.[LRKOJ99]Ludwig, R., Rathonyi, B., Konrad, A., Oden, K. and A. Joseph, Multi-Layer Tracing of TCP over a Reliable Wireless Link, pp. 144-154, In Proceedings of ACM SIGMETRICS 99.[LS00]Ludwig, R. and K. Sklower, The Eifel Retransmission Timer, ACM Computer Communication Review, Vol. 30, No. 3, July 2000.[MAGMA-PROXY]Fenner, B., He, H., Haberman, B. and H. Sandick, "IGMP/MLD-based Multicast Forwarding ("IGMP/MLD Proxying")", Work in Progress.[MAGMA-SNOOP]Christensen, M., Kimball, K. and F. Solensky, "Considerations for IGMP and MLD Snooping Switches", Work in Progress.[MBB00]May, M., Bonald, T. and J-C. Bolot, "Analytic Evaluation of RED Performance", INFOCOM 2000.[MBDL99]May, M., Bolot, J., Diot, C. and B. Lyles, "Reasons not to deploy RED", Proc. of 7th. International Workshop on Quality of Service (IWQoS'99), June 1999.[MSMO97]Mathis, M., Semke, J., Mahdavi, J. and T. Ott, "The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm", Computer Communication Review, Vol. 27, number 3, July 1997.[MYR95]Boden, N., Cohen, D., Felderman, R., Kulawik, A., Seitz, C., et al. MYRINET: A Gigabit per Second Local Area Network, IEEE-Micro, Vol. 15, No.1, February 1995, pp. 29-36.[PFTK98]Padhye, J., Firoiu, V., Towsley, D. and J. Kurose, "Modeling TCP Throughput: a Simple Model and its Empirical Validation", UMASS CMPSCI Tech Report TR98-008, Feb. 1998.[RED93]Floyd, S. and V. Jacobson, "Random Early Detection gateways for Congestion Avoidance", IEEE/ACM Transactions in Networking, Vol. 1 No. 4, August 1993.http://www.aciri.org/floyd/papers/red/red.html[RF95]Romanow, A. and S. Floyd, "Dynamics of TCP Traffic over ATM Networks". IEEE Journal of Selected Areas in Communication, Vol.13 No. 4, May 1995, p. 633-641.[RFC791]Postel, J., "Internet Protocol", STD 5, RFC 791, September 1981.[RFC793]Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981.[RFC768]Postel, J., "User Datagram Protocol", STD 6, RFC 768, August 1980.[RFC826]Plummer, D.C., "Ethernet Address Resolution Protocol: Or converting network protocol addresses to 48-bit Ethernet address for transmission on Ethernet hardware", STD 37, RFC 826, November 1982.[RFC1071]Braden, R., Borman, D. and C. Partridge, "Computing the Internet checksum", RFC 1071, September 1988.[RFC1112]Deering, S., "Host Extensions for IP Multicasting", STD 5, RFC 1112, August 1989.[RFC1144]Jacobson, V., "Compressing TCP/IP Headers for Low-Speed Serial Links", RFC 1144, February 1990.[RFC1191]Mogul, J. and S. Deering, "Path MTU Discovery", RFC 1191, November 1990.[RFC1332]McGregor, C., "The PPP Internet Protocol Control Protocol (IPCP)", RFC 1332, May 1992.[RFC1435]Knowles, S., "IESG Advice from Experience with Path MTU Discovery", RFC 1435, March 1993.[RFC1633]Braden, R., Clark, D. and S. Shenker, "Integrated Services in the Internet Architecture: an Overview", RFC 1633, June 1994.[RFC1661]Simpson, W., "The Point-to-Point Protocol (PPP)", STD 51, RFC 1661, July 1994.[RFC1662]Simpson, W., Ed., "PPP in HDLC-like Framing", STD 51, RFC 1662, July 1994.[RFC1750]Eastlake 3rd, D., Crocker, S. and J. Schiller, "Randomness Recommendations for Security", RFC 1750, December 1994.[RFC1812]Baker, F., Ed., "Requirements for IP Version 4 Routers", RFC 1812, June 1995.[RFC1939]Myers, J. and M. Rose, "Post Office Protocol - Version 3", STD 53, RFC 1939, May 1996.[RFC1981]McCann, J., Deering, S. and J. Mogul, "Path MTU Discovery for IP version 6", RFC 1981, August 1996.[RFC1991]Atkins, D., Stallings, W. and P. Zimmermann, "PGP Message Exchange Formats", RFC 1991, August 1996.[RFC2018]Mathis, M., Mahdavi, J., Floyd, S. and A. Romanow, "TCP Selective Acknowledgement Options", RFC 2018, October 1996.[RFC2131]Droms, R., "Dynamic Host Configuration Protocol", RFC 2131, March 1997.[RFC2205]Braden, R., Ed., Zhang, L., Berson, S., Herzog, S. and S. Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 Functional Specification", RFC 2205, September 1997.[RFC2208]Mankin, A., Baker, F., Braden, B., Bradner, S., O`Dell, M., Romanow, A., Weinrib, A. and L. Zhang, "Resource ReSerVation Protocol (RSVP) -- Version 1 Applicability Statement Some Guidelines on Deployment", RFC 2208, September 1997.[RFC2210]Wroclawski, J., "The Use of RSVP with IETF Integrated Services", RFC 2210, September 1997.[RFC2211]Wroclawski, J., "Specification of the Controlled-Load Network Element Service", RFC 2211, September 1997.[RFC2212]Shenker, S., Partridge, C. and R. Guerin, "Specification of Guaranteed Quality of Service", RFC 2212, September 1997.[RFC2246]Dierks, T. and C. Allen, "The TLS Protocol Version 1.0", RFC 2246, January 1999.
Okay bestie, here's the full reading list 📚✨ Every single one is here, no skipping, because these are the papers the whole spec leans on 🫶
[RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, S., Wroclawski, J. and L. Zhang, "Recommendations on Queue Management and Congestion Avoidance in the Internet", RFC 2309, April 1998. 🌸
[RFC2322] van den Hout, K., Koopal, A. and R. van Mook, "Management of IP numbers by peg-dhcp", RFC 2322, 1 April 1998. 🙈 (yes, it's an April Fools one, iconic)
[RFC2328] Moy, J., "OSPF Version 2", STD 54, RFC 2328, April 1998. 👑
[RFC2332] Luciani, J., Katz, D., Piscitello, D., Cole, B. and N. Doraswamy, "NBMA Next Hop Resolution Protocol (NHRP)", RFC 2332, April 1998. 🦋
[RFC2364] Gross, G., Kaycee, M., Li, A., Malis, A. and J. Stephens, "PPP Over AAL5", RFC 2364, July 1998. 💜
[RFC2394] Pereira, R., "IP Payload Compression Using DEFLATE", RFC 2394, December 1998. ✨
[RFC2395] Friend, R. and R. Monsour, "IP Payload Compression Using LZS", RFC 2395, December 1998. ✨
[RFC2401] Kent, S. and R. Atkinson, "Security Architecture for the Internet Protocol", RFC 2401, November 1998. 🔥
[RFC2406] Kent, S. and R. Atkinson, "IP Encapsulating Security Payload (ESP)", RFC 2406, November 1998. 🔥
[RFC2440] Callas, J., Donnerhacke, L., Finney, H. and R. Thayer, "OpenPGP Message Format", RFC 2440, November 1998. 🎀
[RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", RFC 2460, December 1998. 🌈
[RFC2461] Narten, T., Nordmark, E. and W. Simpson, "Neighbor Discovery for IP Version 6 (IPv6)", RFC 2461, December 1998. 🧚
[RFC2474] Nichols, K., Blake, S., Baker, F. and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, December 1998. 💅
[RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z. and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, December 1998. 💅
[RFC2507] Degermark, M., Nordgren, B. and S. Pink, "IP Header Compression", RFC 2507, February 1999. 🍓
[RFC2508] Casner, S. and V. Jacobson, "Compressing IP/UDP/RTP Headers for Low-Speed Serial Links", RFC 2508, February 1999. 🍓
[RFC2581] Allman, M., Paxson, V. and W. Stevens, "TCP Congestion Control", RFC 2581, April 1999. ⭐
[RFC2582] Floyd, S. and T. Henderson, "The NewReno Modification to TCP's Fast Recovery Algorithm", RFC 2582, April 1999. ⭐
[RFC2597] Heinanen, J., Baker, F., Weiss, W. and J. Wroclawski, "Assured Forwarding PHB Group", RFC 2597, June 1999. 🦋
[RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P. and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. 🌸
[RFC2630] Housley, R., "Cryptographic Message Syntax", RFC 2630, June 1999. 🔐
[RFC2631] Rescorla, E., "Diffie-Hellman Key Agreement Method", RFC 2631, June 1999. 🔐
[RFC2632] Ramsdell, B., Ed., "S/MIME Version 3 Certificate Handling", RFC 2632, June 1999. 💌
[RFC2633] Ramsdell, B., "S/MIME Version 3 Message Specification", RFC 2633, June 1999. 💌
[RFC2634] Hoffman, P., "Enhanced Security Services for S/MIME", RFC 2634, June 1999. 💌
[RFC2684] Grossman, D. and J. Heinanen, "Multiprotocol Encapsulation over ATM Adaptation Layer 5", RFC 2684, September 1999. 💜
[RFC2686] Bormann, C., "The Multi-Class Extension to Multi-Link PPP", RFC 2686, September 1999. 🌈
[RFC2687] Bormann, C., "PPP in a Real-time Oriented HDLC-like Framing", RFC 2687, September 1999. 🌈
[RFC2689] Bormann, C., "Providing Integrated Services over Low-bitrate Links", RFC 2689, September 1999. 🌈
[RFC2710] Deering, S., Fenner, W. and B. Haberman, "Multicast Listener Discovery (MLD) for IPv6", RFC 2710, October 1999. 📣
[RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D. and P. Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, March 2000. 🧁
[RFC2865] Rigney, C., Willens, S., Rubens, A. and W. Simpson, "Remote Authentication Dial In User Service (RADIUS)", RFC 2865, June 2000. 🔑
[RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, RFC 2914, September 2000. 🌟
[RFC2923] Lahey, K., "TCP Problems with Path MTU Discovery", RFC 2923, September 2000. 🛠️
[RFC2988] Paxson, V. and M. Allman, "Computing TCP's Retransmission Timer", RFC 2988, November 2000. ⏱️
[RFC2990] Huston, G., "Next Steps for the IP QoS Architecture", RFC 2990, November 2000. 🎯
[RFC3048] Whetten, B., Vicisano, L., Kermode, R., Handley, M., Floyd, S. and M. Luby, "Reliable Multicast Transport Building Blocks for One-to-Many Bulk-Data Transfer", RFC 3048, January 2001. 📦
[RFC3095] Bormann, C., Ed., Burmeister, C., Degermark, M., Fukushima, H., Hannu, H., Jonsson, L-E., Hakenberg, R., Koren, T., Le, K., Liu, Z., Martensson, A., Miyazaki, A., Svanbro, K., Wiebke, T., Yoshimura, T. and H. Zheng, "RObust Header Compression (ROHC): Framework and four profiles: RTP, UDP, ESP, and uncompressed", RFC 3095, July 2001. 🍓
[RFC3096] Degermark, M., Ed., "Requirements for robust IP/UDP/RTP header compression", RFC 3096, July 2001. 🍓
[RFC3150] Dawkins, S., Montenegro, G., Kojo, M. and V. Magret, "End-to-end Performance Implications of Slow Links", BCP 48, RFC 3150, July 2001. 🐢
[RFC3155] Dawkins, S., Montenegro, G., Kojo, M., Magret, V. and N. Vaidya, "End-to-end Performance Implications of Links with Errors", BCP 50, RFC 3155, August 2001. ⚡
[RFC3168] Ramakrishnan, K., Floyd, S. and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, September 2001. 🚦
[RFC3173] Shacham, A., Monsour, B., Pereira, R. and M. Thomas, "IP Payload Compression Protocol (IPComp)", RFC 3173, September 2001. ✨
[RFC3246] Davie, B., Charny, A., Bennet, J.C.R., Benson, K., Le Boudec, J.Y., Courtney, W., Davari, S., Firoiu, V. and D. Stiliadis, "An Expedited Forwarding PHB (Per-Hop Behavior)", RFC 3246, March 2002. 🚀
[RFC3248] Armitage, G., Carpenter, B., Casati, A., Crowcroft, J., Halpern, J., Kumar, B. and J. Schnizlein, "A Delay Bound alternative revision of RFC 2598", RFC 3248, March 2002. 🚀
[RFC3344] Perkins, C., Ed., "IP Mobility Support for IPv4", RFC 3344, August 2002. 🧳
[RFC3366] Fairhurst, G. and L. Wood, "Advice to link designers on link Automatic Repeat reQuest (ARQ)", BCP 62, RFC 3366, August 2002. 🔁
[RFC3376] Cain, B., Deering, S., Kouvelas, I., Fenner, B. and A. Thyagarajan, "Internet Group Management Protocol, Version 3", RFC 3376, October 2002. 📣
[RFC3449] Balakrishnan, H., Padmanabhan, V., Fairhurst, G. and M. Sooriyabandara, "TCP Performance Implications of Network Path Asymmetry", BCP 69, RFC 3449, December 2002. ⚖️
[RFC3450] Luby, M., Gemmell, J., Vicisano, L., Rizzo, L. and J. Crowcroft, "Asynchronous Layered Coding (ALC) Protocol Instantiation", RFC 3450, December 2002. 🧅
[RFC3451] Luby, M., Gemmell, J., Vicisano, L., Rizzo, L., Handley, M. and J. Crowcroft, "Layered Coding Transport (LCT) Building Block", RFC 3451, December 2002. 🧅
[RFC3452] Luby, M., Vicisano, L., Gemmell, J., Rizzo, L., Handley, M. and J. Crowcroft, "Forward Error Correction (FEC) Building Block", RFC 3452, December 2002. 🛡️
[RFC3453] Luby, M., Vicisano, L., Gemmell, J., Rizzo, L., Handley, M. and J. Crowcroft, "The Use of Forward Error Correction (FEC) in Reliable Multicast", RFC 3453, December 2002. 🛡️
[RFC3488] Wu, I. and T. Eckert, "Cisco Systems Router-port Group Management Protocol (RGMP)", RFC 3488, February 2003. 📣
[RFC3501] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1", RFC 3501, March 2003. 📬
[RFC3828] Larzon, L-A., Degermark, M., Pink, S., Jonsson, L-E., Ed. and G. Fairhurst, Ed., "The User Datagram Protocol (UDP)-Lite Protocol", RFC 3828, June 2004. 🦋
[Schneier95] Schneier, B., Applied Cryptography: Protocols, Algorithms and Source Code in C (John Wiley and Sons, October 1995). 📖
[Schneier00] Schneier, B., Secrets and Lies: Digital Security in a Networked World (John Wiley and Sons, August 2000). 📖
[SP2000] Stone, J. and C. Partridge, "When the CRC and TCP Checksum Disagree", ACM SIGCOMM, September 2000. http://www.acm.org/sigcomm/sigcomm2000/conf/paper/sigcomm2000-9-1.pdf
[SRC81] Saltzer, J., Reed D. and D. Clark, "End-to-End Arguments in System Design". Second International Conference on Distributed Computing Systems (April, 1981) pages 509-512. Published with minor changes in ACM Transactions in Computer Systems 2, 4, November, 1984, pages 277-288. Reprinted in Craig Partridge, editor Innovations in internetworking. Artech House, Norwood, MA, 1988, pages 195-206. ISBN 0-89006-337-0. 🫶
[SSL2] Hickman, K., "The SSL Protocol", Netscape Communications Corp., Feb 9, 1995. 🔐
[SSL3] Frier, A., Karlton, P. and P. Kocher, "The SSL 3.0 Protocol", Netscape Communications Corp., Nov 18, 1996. 🔐
[TCPF98] Lin, D. and H.T. Kung, "TCP Fast Recovery Strategies: Analysis and Improvements", IEEE Infocom, March 1998. http://www.eecs.harvard.edu/networking/papers/infocom-tcp-final-198.pdf
[WFBA2000] Wagner, D., Foster, J., Brewer, E. and A. Aiken, "A First Step Toward Automated Detection of Buffer Overrun Vulnerabilities", Proceedings of NDSS2000. http://www.isoc.org/isoc/conferences/ndss/2000/proceedings/039.pdf 🕵️
[Wilbur89] Wilbur, Steve R., Jon Crowcroft, and Yuko Murayama. "MAC layer Security Measures in Local Area Networks", Local Area Network Security, Workshop LANSEC '89 Proceedings, Springer-Verlag, April 1989, pp. 53-64. 💖
21. Contributors' Addresses ✨
okay bestie, the contributors are up first 🫶 here are their deets, so you know exactly who to thank 🎀
Aaron Falk
USC/Information Sciences Institute
4676 Admiralty Way
Marina Del Rey, CA 90292
Phone: 310-448-9327
EMail: falk@isi.edu
Saverio Mascolo
Dipartimento di Elettrotecnica ed Elettronica,
Politecnico di Bari Via Orabona 4, 70125 Bari, Italy
Phone: +39 080 596 3621
EMail: mascolo@poliba.it
URL: http://www-dee.poliba.it/dee-web/Personale/mascolo.html
Marie-Jose Montpetit
MJMontpetit.com
EMail: marie@mjmontpetit.com22. Authors' Addresses 💖
and now the authors!! these are the people who actually wrote it, so iconic 👑
Phil Karn, Editor
Qualcomm 5775 Morehouse Drive
San Diego CA 92121
Phone: 858 587 1121
EMail: karn@qualcomm.com
Carsten Bormann
Universitaet Bremen TZI
Postfach 330440
D-28334 Bremen, Germany
Phone: +49 421 218 7024
Fax: +49 421 218 7000
EMail: cabo@tzi.org
Godred (Gorry) Fairhurst
Department of Engineering, University of Aberdeen,
Aberdeen, AB24 3UE, United Kingdom
EMail: gorry@erg.abdn.ac.uk
URL: http://www.erg.abdn.ac.uk/users/gorry
Dan Grossman
Motorola, Inc.
111 Locke Drive
Marlboro, MA 01752
EMail: Dan.Grossman@motorola.com
Reiner Ludwig
Ericsson Research
Ericsson Allee
1 52134 Herzogenrath, Germany
Phone: +49 2407 575 719
EMail: Reiner.Ludwig@ericsson.com
Jamshid Mahdavi
Novell, Inc.
EMail: jmahdavi@earthlink.net
Gabriel Montenegro
Sun Microsystems Laboratories, Europe
180, Avenue de l'Europe
38334 Saint Ismier CEDEX
France
EMail: gab@sun.com
Joe Touch
USC/Information Sciences Institute
4676 Admiralty Way
Marina del Rey CA 90292
Phone: 310 448 9151
EMail: touch@isi.edu
URL: http://www.isi.edu/touch
Lloyd Wood
Cisco Systems
9 New Square Park, Bedfont Lakes
Feltham TW14 8HA
United Kingdom
Phone: +44 (0)20 8824 4236
EMail: lwood@cisco.com
URL: http://www.ee.surrey.ac.uk/Personal/L.Wood/23. Full Copyright Statement 🔥
legal time, but I'll keep it real 💅 The Internet Society owns the copyright here, dated 2004. The document is covered by BCP 78's rights, licenses and restrictions. Outside of that, the authors keep all their rights. 🌈
The doc and everything in it is handed over "AS IS". The contributor, the org they represent or sponsor them (if any), the Internet Society and the IETF all say there are no warranties, express or implied. That covers not infringing anyone's rights, and also merchantability or fitness for a particular purpose. 😭
Intellectual Property 🎀
the IETF doesn't pick a side on whether any patents or other rights covering this tech are valid or how far they reach. It also doesn't say whether licenses are available, and it hasn't independently checked for any such rights. If you want the rules on rights in RFCs, look at BCP 78 and BCP 79 🧚
Copies of IPR disclosures sent to the IETF Secretariat are available, along with any promised licenses. You can also find the results of any attempt to get a general license or permission for implementers or users of this spec. All of that lives in the IETF's online IPR repository at http://www.ietf.org/ipr 🦋
The IETF also invites anyone to flag copyrights, patents, patent applications or other proprietary rights that might cover tech needed to implement this standard. Send that info to ietf-ipr@ietf.org 💜
Acknowledgement 🌸
the RFC Editor function is currently funded by the Internet Society. thank you, Internet Society 🫶✨
📦 where this page comes from
- ietf: rfc3819 2004-07
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms