RFC 3439

Some Internet Architectural Guidelines and Philosophy

📅 published December 2002, from ietf
Network Working Group                                            R. Bush
Request for Comments: 3439                                      D. Meyer
Updates: 1958                                              December 2002
Category: Informational


         Some Internet Architectural Guidelines and Philosophy

okay bestie, this one is a whole vibe 💖 RFC 3439 is an Informational doc from December 2002, written by R. Bush and D. Meyer. It doesn't set any standard, it just shares guidance and philosophy. It updates RFC 1958 (it doesn't obsolete anything), and it's basically an extension of that one, so you'll see it reference RFC 1958 a lot 🧚

Status of this Memo 📜

This memo is here to give information to the Internet community. It is not a specification of any Internet standard, and anyone can distribute it freely ✨

Copyright (C) The Internet Society (2002). All Rights Reserved.

Abstract 🦋

This doc extends RFC 1958 by laying out some philosophical guidelines that people designing and building Internet backbone networks should follow. It introduces the Simplicity Principle, which says that complexity is the main thing blocking efficient scaling. Then it looks at what that means for architecture, design, and engineering in huge Internet backbones 🫶

Table of Contents 🌸

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . .  2
2. Large Systems and The Simplicity Principle . . . . . . . . .  3
2.1. The End-to-End Argument and Simplicity   . . . . . . . . .  3
2.2. Non-linearity and Network Complexity   . . . . . . . . . .  3
2.2.1. The Amplification Principle. . . . . . . . . . . . . . .  4
2.2.2. The Coupling Principle . . . . . . . . . . . . . . . . .  5
2.3. Complexity lesson from voice. . . . .  . . . . . . . . . .  6
2.4. Upgrade cost of complexity. . . . . .  . . . . . . . . . .  7
3. Layering Considered Harmful. . . . . . . . . . . . . . . . .  7
3.1. Optimization Considered Harmful . . .  . . . . . . . . . .  8
3.2. Feature Richness Considered Harmful .  . . . . . . . . . .  9
3.3. Evolution of Transport Efficiency for IP.  . . . . . . . .  9
3.4. Convergence Layering. . . . . . . . . . .  . . . . . . . .  9
3.4.1. Note on Transport Protocol Layering. . . . . . . . . . . 11
3.5. Second Order Effects   . . . . . . . . . . . . . . . . . . 11
3.6. Instantiating the EOSL Model with IP   . . . . . . . . . . 12
4. Avoid the Universal Interworking Function. . . . . . . . . . 12
4.1. Avoid Control Plane Interworking . . . . . . . . . . . . . 13
5. Packet versus Circuit Switching: Fundamental Differences . . 13
5.1. Is PS is inherently more efficient than CS?  . . . . . . . 13
5.2. Is PS simpler than CS? . . . . . . . . . . . . . . . . . . 14
5.2.1. Software/Firmware Complexity . . . . . . . . . . . . . . 15
5.2.2. Macro Operation Complexity . . . . . . . . . . . . . . . 15
5.2.3. Hardware Complexity. . . . . . . . . . . . . . . . . . . 15
5.2.4. Power. . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.2.5. Density. . . . . . . . . . . . . . . . . . . . . . . . . 16
5.2.6. Fixed versus variable costs. . . . . . . . . . . . . . . 16
5.2.7. QoS. . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.2.8. Flexibility. . . . . . . . . . . . . . . . . . . . . . . 17
5.3. Relative Complexity  . . . . . . . . . . . . . . . . . . . 17
5.3.1. HBHI and the OPEX Challenge. . . . . . . . . . . . . . . 18
6. The Myth of Over-Provisioning. . . . . . . . . . . . . . . . 18
7. The Myth of Five Nines . . . . . . . . . . . . . . . . . . . 19
8. Architectural Component Proportionality Law. . . . . . . . . 20
8.1. Service Delivery Paths . . . . . . . . . . . . . . . . . . 21
9. Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . 21
10. Security Considerations . . . . . . . . . . . . . . . . . . 22
11. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . 23
12. References. . . . . . . . . . . . . . . . . . . . . . . . . 23
13. Authors' Addresses. . . . . . . . . . . . . . . . . . . . . 27
14. Full Copyright Statement. . . . . . . . . . . . . . . . . . 28

1. Introduction ✨

RFC 1958 [RFC1958] is the one that lays out the basic principles of Internet architecture. This note builds on it by sharing philosophical guidelines that architects and designers of Internet backbone networks should stick to 💅 Lowkey, a lot of what's in here might be controversial, but the big unifying idea is not. That idea is treating complexity as the thing you control to keep costs down and reliability up. Complexity in carrier networks can come from all kinds of places. But, as [DOYLE2002] puts it, "Complexity in most systems is driven by the need for robustness to uncertainty in their environments and component parts far more than by basic functionality". 🤯

So the main goal of this doc is to get people aware of how complicated some of our current architectures are, and to look at how that complexity is almost certainly going to affect the IP carrier industry's chances of winning 👑

The rest of the doc goes like this. Section 2 explains the Simplicity Principle and what it means for designing really big systems. The stuff after that covers the high-level results of the Simplicity Principle, and how it should shape the way large-scale network architecture and design gets done 🌈

2. Large Systems and The Simplicity Principle 🎀

The Simplicity Principle was probably first put into words by Mike O'Dell, former Chief Architect at UUNET 🧠 It says that complexity is the main thing that gets in the way of efficient scaling. Because of that, complexity is also the main driver of growth in both capital expenditures (CAPEX) and operational expenditures (OPEX). So for carrier IP networks, the takeaway is pretty clear: if we want to succeed, we need to push our architectures and designs toward the simplest solutions possible 🍓

2.1. The End-to-End Argument and Simplicity 🦋

The end-to-end argument is described in [SALTZER], and also in RFC 1958 [RFC1958]. It says that "end-to-end protocol design should not rely on the maintenance of state (i.e., information about the state of the end-to-end communication) inside the network. Such state should be maintained only in the end points, in such a way that the state can only be destroyed when the end point itself breaks." 😭 This property is also connected to Clark's "fate-sharing" idea [CLARK]. You can see that the end-to-end principle leads straight to the Simplicity Principle when you look at the so-called "hourglass" shape of Internet architecture [WILLINGER2002]. In that picture, the skinny waist of the hourglass is the (minimalist) IP layer, and any extra complexity gets stacked on top of IP. Basically, the complexity of the Internet belongs at the edges, and the IP layer should stay as simple as it can be 💜

Also, the end-to-end argument doesn't mean the core of the Internet won't keep state. The core actually keeps a ton of coarse-grained state, like routing state. The important thing is that this coarse-grained state is almost orthogonal to the state the end-points (like hosts) keep. Keeping that interaction minimal is a big part of what makes things simple. So when you look at protocols like Network Address Translation (NAT), which reduce the transparency between the network and hosts, it's crucial to think about how "core vs. end-point" state interacts 🙈

2.2. Non-linearity and Network Complexity 🌸

Complex architectures and designs have been, and still are, some of the biggest and hardest barriers to building cost-effective large-scale IP networks 😤 Take building a large-scale packet network, for example. Industry experience shows that building one is a different activity from building a small or medium network. It needs a different skill set, and it just doesn't have the same properties. The biggest networks show architecture, design, and engineering non-linearities, both in theory and in practice, that don't show up at smaller scales. We call this Architecture, Design, and Engineering (ADE) non-linearity. In other words, systems like the Internet could be described as highly self-dissimilar, with extremely different scales and levels of abstraction [CARLSON]. The ADE non-linearity property is based on two well-known principles from non-linear systems theory [THOMPSON]:

2.2.1. The Amplification Principle 🔥

The Amplification Principle says that there are non-linearities that happen at large scale and don't happen at small to medium scale.

COROLLARY: In many large networks, even small things can and do cause huge events. In system-theoretic terms, in large systems like these, even small perturbations on the input to a process can destabilize the system's output.

An important example of the Amplification Principle is non-linear resonant amplification. It's a powerful process that can transform dynamic systems, like large networks, in surprising ways with seemingly small fluctuations. These small fluctuations may slowly accumulate, and if they line up with other cycles, they may produce major changes. Resonant phenomena are examples of non-linear behavior where small fluctuations can get amplified and have influences far beyond their starting size. The natural world is full of resonant behavior that can produce system-wide changes, like the destruction of the Tacoma Narrows bridge (due to resonant amplification of small gusts of wind). Other examples include the gaps in the asteroid belts and the rings of Saturn, which are created by non-linear resonant amplification. Some features of human behavior and most pilgrimage systems are influenced by resonant phenomena involving the dynamics of the solar system, like solar days, the 27.3 day (sidereal) and 29.5 day (synodic) cycles of the moon, or the 365.25 day cycle of the sun. 🌍

In the Internet domain, it's been shown that more inter-connectivity leads to more complex and often slower BGP routing convergence [AHUJA]. A related result is that even a small amount of inter-connectivity makes the output of a routing mesh much more complex than its input [GRIFFIN]. One important way to reduce amplification is to make sure local changes only have local effects. That's the opposite of systems where local changes have global effects. And ATM is a great example of an amplification effect: if you lose one cell, you destroy the entire packet. It gets worse, too. Without mechanisms like Early Packet Discard [ROMANOV], you'll keep carrying the packet that's already damaged 💀

Another interesting example of amplification comes from engineering, described in [CARLSON]. They look at the Boeing 777, a "fly-by-wire" aircraft with as many as 150,000 subsystems and about 1000 CPUs. What they found is that the 777 is robust to big atmospheric disturbances, turbulence boundaries, and changes in cargo loads (to name a few). But it could be catastrophically disabled by microscopic alterations in a very few large CPUs. As they point out, thankfully this is a very rare occurrence. This example shows the issue "that complexity can amplify small perturbations, and the design engineer must ensure such perturbations are extremely rare." [CARLSON] ✈️🫶

2.2.2. The Coupling Principle 🧶

Okay so the Coupling Principle says that as things get bigger, their parts often end up way more dependent on each other 🫶

COROLLARY: the more events that happen at the same time, the bigger the chance that two or more of them will interact. This has also been called "unforeseen feature interaction" [WILLINGER2002]. 💀

Most of the non-linear behaviour we see in large systems is mostly because of coupling. Coupling has a horizontal side and a vertical side. In networking, horizontal coupling shows up between components of the same protocol layer, and vertical coupling happens between layers. ✨

Coupling pops up all over natural systems too 🌈 Think plasma macro-instabilities (hydro-magnetic, e.g., kink, fire-hose, mirror, ballooning, tearing, trapped-particle effects) [NAVE], or all kinds of electrochemical systems (like the custom fluorescent nucleotide synthesis/nucleic acid labeling problem [WARD]). Coupling of clock physical periodicity has been observed as well [JACOBSON], and so has coupling of various types of biological cycles.

Networks have their own classic examples 🌸 The synchronization of various control loops is one, such as routing update synchronization and TCP Slow Start synchronization [FLOYD,JACOBSON]. The big takeaway here is that coupling is super intimately related to synchronization. Injecting randomness into these systems is one way to reduce coupling. 🎀

Now here's a fun one 🍓 Charles Perrow, while analyzing risk factors for the Public Switched Telephone Network (PSTN), breaks the complexity problem down along two related axes, which he calls "interactions" and "coupling" [PERROW]. Perrow says interactions and coupling are big factors in how reliable a complex system is (and especially the PSTN). In his model, interactions means the dependencies between components (linear or non-linear), while coupling means how much flexibility a system has. Systems with simple, linear interactions have components that only affect other components that are functionally downstream. Complex system components, on the other hand, interact with many other components in different and possibly distant parts of the system. Loosely coupled systems have more flexibility when it comes to time constraints, sequencing, and environmental assumptions than tightly coupled systems do. Also, systems with complex interactions and tight coupling are likely to have unforeseen failure states (of course, complex interactions allow more complications to develop and make the system hard to understand and predict). This behaviour is also described in [WILLINGER2002]. Tight coupling also means the system has less flexibility when it comes to recovering from failure states. 🙈

The PSTN's SS7 control network is an iconic example of what can go wrong with a tightly coupled complex system 😭 Outages like the well-publicized 1991 outage of AT&T's SS7 show the phenomenon: the outage was caused by software bugs in the switches' crash recovery code. One switch crashed due to a hardware glitch. When this switch came back up, it (plus a reasonably probable timing event) caused its neighbors to crash. When the neighboring switches came back up, they caused their neighbors to crash, and so on [NEUMANN]. (The root cause turned out to be a misplaced 'break' statement, which is an excellent example of cross-layer coupling. 💅) This is similar to the phase-locking of weakly coupled oscillators, where random variations in sequence times play an important role in system stability [THOMPSON].

2.3. Complexity lesson from voice 📞

In the 1970s and 1980s, the voice carriers competed by adding features, and that drove substantial increases in the complexity of the PSTN, especially in the Class 5 switching infrastructure. This complexity was typically software-based rather than hardware-driven, so its cost curves were worse than Moore's Law. 📉 In summary, poor margins on voice products today are due to OPEX and CAPEX costs not dropping the way we'd expect from simple, hardware-bound implementations.

2.4. Upgrade cost of complexity 💸

Let's think about what it costs to add new features to a complex network. The traditional voice network has very little intelligence in its edge devices (phone instruments) and a very smart core. The Internet is the opposite 🧚 It has smart edges (computers with operating systems, applications, etc.) and a simple core, which consists of a control plane and packet forwarding engines. So adding a new Internet service is just distributing an application to the few consenting desktops that want it. Compare that to adding a service to voice, where you have to upgrade the entire core. 👑

3. Layering Considered Harmful 🚫

There are several generic properties of layering, or vertical integration as applied to networking. In general, a layer (as defined in our context) implements one or more of the following:

    Error Control:     The layer makes the "channel" more reliable
                       (e.g., reliable transport layer)

    Flow Control:      The layer avoids flooding slower peer (e.g.,
                       ATM flow control)

    Fragmentation:     Dividing large data chunks into smaller
                       pieces, and subsequent reassembly (e.g., TCP
                       MSS fragmentation/reassembly)

    Multiplexing:      Allow several higher level sessions share
                       single lower level "connection" (e.g., ATM PVC)

    Connection Setup:  Handshaking with peer (e.g., TCP three-way
                       handshake, ATM ILMI)

    Addressing/Naming: Locating, managing identifiers associated
                       with entities (e.g., GOSSIP 2 NSAP Structure
                       [RFC1629])

Layering like this does have various conceptual and structuring advantages. 🌷 However, in the data networking context, structured layering means that the functions of each layer are carried out completely before the protocol data unit is passed to the next layer. This means that the optimization of each layer has to be done separately. Those ordering constraints clash with efficient implementation of data manipulation functions. One could accuse the layered model (e.g., TCP/IP and ISO OSI) of causing this conflict. In fact, the operations of multiplexing and segmentation both hide vital information that lower layers may need to optimize their performance. For example, layer N may duplicate lower level functionality, e.g., error recovery hop-hop versus end-to-end error recovery. Also, different layers may need the same information (e.g., time stamp): layer N may need layer N-2 information (e.g., lower layer packet sizes), and the like [WAKEMAN].

A related and even more ironic statement comes from Tennenhouse's classic paper, "Layered Multiplexing Considered Harmful" [TENNENHOUSE] 😤

"The ATM approach to broadband networking is presently being pursued within the CCITT (and elsewhere) as the unifying mechanism for the support of service integration, rate adaptation, and jitter control within the lower layers of the network architecture. This position paper is specifically concerned with the jitter arising from the design of the "middle" and "upper" layers that operate within the end systems and relays of multi-service networks (MSNs)."

As a result of inter-layer dependencies, increased layering can quickly lead to violating the Simplicity Principle. Industry experience has taught us that increased layering frequently increases complexity, and that leads to increases in OPEX, exactly as the Simplicity Principle predicts. 📈 A corollary is stated in RFC 1925 [RFC1925], section 2(5):

"It is always possible to agglutinate multiple separate problems into a single complex interdependent solution. In most cases this is a bad idea."

So the first-order conclusion is that horizontal (as opposed to vertical) separation may be more cost-effective and reliable in the long term. ✨

3.1. Optimization Considered Harmful ⚡

A corollary of the layering arguments above is that optimization can also be considered harmful. Optimization introduces complexity, and it also introduces tighter coupling between components and layers. 🧵

There's an important, related effect of optimization described by the Law of Diminishing Returns. It says that if one factor of production is increased while the others stay constant, the overall returns will relatively decrease after a certain point [SPILLMAN]. The implication? Trying to squeeze out efficiency past that point only adds complexity, and therefore leads to less reliable systems. 🙃

3.2. Feature Richness Considered Harmful 🎁

Adding any new feature may be considered a gain (and in fact, it frequently differentiates vendors of various types of equipment), but there's a danger 😬 The danger is increased system complexity.

3.3. Evolution of Transport Efficiency for IP 🛣️

The evolution of transport infrastructures for IP is a great example of how decreasing vertical integration has led to various efficiencies. In particular:

    | IP over ATM over SONET  -->
    | IP over SONET over WDM  -->
    | IP over WDM
    |
   \|/
   Decreasing complexity, CAPEX, OPEX

The key point here is that layers get removed, and that results in CAPEX and OPEX efficiencies. 🦋

3.4. Convergence Layering 🦋

Okay so convergence is basically the layering stuff we just talked about, because it happens through a "convergence layer" 🧚 The endgame of this whole convergence argument is called Everything Over Some Layer (EOSL) 💅 Conduit, DWDM, fiber, ATM, MPLS, and even IP have all been pitched as convergence layers 🌈 Here's the tea: layering usually pushes OPEX up, so we expect convergence to do the same thing 💀 And honestly, that matches what the industry has seen 🙈

There are lots of notable convergence layer fails, and the most germane one is IP over ATM 😭 The immediate and most obvious consequence of ATM layering is the so-called cell tax. First, the complete answer on ATM efficiency depends on packet size distributions. Let's assume typical Internet traffic patterns, which tend to have high percentages of packets at 40, 44, and 552 bytes. Recent data [CAIDA] shows that about 95% of WAN bytes and 85% of packets are TCP. Much of this traffic is made of 40/44 byte packets, so it's giving tiny packets everywhere 🎀

Now consider a DS3 backbone with PLCP turned on. Then the maximum cell rate is 96,000 cells/sec. If you multiply this value by the number of bits in the payload, you get: 96000 cells/sec * 48 bytes/cell * 8 = 36.864 Mbps. This, however, is unrealistic since it assumes perfect payload packing 🙃 Two other things add to the ATM overhead (cell tax): the wasted padding and the 8 byte SNAP header.

It is the SNAP header that causes most of the problems (and you can't do anything about this) ✨ It forces most small packets to consume two cells, with the second cell mostly empty padding. This interacts really poorly with the data quoted above, e.g., that most packets are 40-44 byte TCP Ack packets 💀 It causes a loss of about another 16% from the 36.8 Mbps ideal throughput.

So the total throughput ends up being (for a DS3):

         DS3 Line Rate:              44.736
         PLCP Overhead              - 4.032
         Per Cell Header:           - 3.840
         SNAP Header & Padding:     - 5.900
                                   =========
                                     30.960 Mbps

Result: With a DS3 line rate of 44.736 Mbps, the total overhead is about 31% 😩

Another way to look at this is that since a large fraction of WAN traffic is made of TCP ACKs, you can do a different but related calculation. IP over ATM requires:

IP data (40 bytes in this case)
8 bytes SNAP
8 bytes AAL5 stuff
5 bytes for each cell
+ as much more as it takes to fill out the last cell

On ATM, this becomes two cells: 106 bytes to convey 40 bytes of information 🥲 The next most common size seems to be one of several sizes in the 504-556 byte range, which is 636 bytes to carry IP, TCP, and a 512 byte TCP payload. Messages larger than 1000 bytes come in third 🍓

One would imagine that 87% payload (556 byte message size) is better than 37% payload (TCP Ack size), but it's not the 95-98% that customers are used to, and the predominance of TCP Acks skews the average 📉

3.4.1. Note on Transport Protocol Layering

Protocol layering models are often cast as "X over Y" models. In these cases, protocol Y carries protocol X's protocol data units (and possibly control data) over Y's data plane, which means Y is a "convergence layer". Examples include Frame Relay over ATM, IP over ATM, and IP over MPLS. While X over Y layering has met with only marginal success [TENNENHOUSE,WAKEMAN], there have been a few notable instances where efficiency can be and is gained 🌸 In particular, "X over Y efficiencies" can be realized when there is a kind of "isomorphism" between X and Y (i.e., there is a small convergence layer). In these cases X's data, and possibly control traffic, are "encapsulated" and transported over Y. Examples include Frame Relay over ATM, and Frame Relay, AAL5 ATM and Ethernet over L2TPv3 [L2TPV3]. The simplifying factors here are that there is no requirement that a shared clock be recovered by the communicating end points, and that control-plane interworking is minimized. An alternative is to interwork X's and Y's control and data planes, and control-plane interworking is discussed below 🧠

3.5. Second Order Effects

IP over ATM is an excellent example of unanticipated second order effects 🙈 In particular, Romanov and Floyd's classic study on TCP good-put [ROMANOV] on ATM showed a few things. Large UBR buffers (larger than one TCP window size) are required to achieve reasonable performance. Packet discard mechanisms (such as Early Packet Discard, or EPD) improve the effective usage of the bandwidth. And more elaborate service and drop strategies than FIFO+EPD, such as per VC queuing and accounting, might be required at the bottleneck to ensure both high efficiency and fairness. Though all studies clearly indicate that a buffer size not less than one TCP window size is required, the amount of extra buffer needed naturally depends on the packet discard mechanism used, and it is still an open issue 👀

Examples of this kind of problem with layering abound in practical networking. Consider the effect of IP transport's implicit assumptions about lower layers. In particular:

  • Packet loss: TCP assumes that packet losses are indications of congestion, but sometimes losses come from corruption on a wireless link [RFC3115].
  • Reordered packets: TCP assumes that significantly reordered packets are indications of congestion. This is not always the case [FLOYD2001].
  • Round-trip times: TCP measures round-trip times, and assumes that the lack of an acknowledgment within a period of time based on the measured round-trip time is a packet loss, and therefore an indication of congestion [KARN].
  • Congestion control: TCP congestion control implicitly assumes that all the packets in a flow are treated the same by the network, but this is not always the case [HANDLEY].

3.6. Instantiating the EOSL Model with IP

While IP is being proposed as a transport for almost everything, the base assumption that Everything over IP (EOIP) will result in OPEX and CAPEX efficiencies needs critical examination 🔍 In particular, many protocols can be efficiently transported over an IP network (specifically, those that do not need to recover synchronization between the communication end points, such as Frame Relay, Ethernet, and AAL5 ATM). But the Simplicity and Layering Principles suggest that EOIP may not be the most efficient convergence strategy for arbitrary services. Rather, a more CAPEX and OPEX efficient convergence layer might be much lower (again, this behavior is predicted by the Simplicity Principle) 💖

An example of where EOIP would not be the most OPEX and CAPEX efficient transport is a service or protocol that needed SONET-like restoration times (e.g., 50ms). It is not hard to imagine that it would cost more to build and operate an IP network with this kind of restoration and convergence property (if that were even possible) than it would to build the SONET network in the first place 😵

4. Avoid the Universal Interworking Function 🚫

While there have been many implementations of the Universal Interworking Function (UIWF), IWF approaches have been problematic at large scale. This concern is codified in the Principle of Minimum Intervention [BRYANT]:

"To minimise the scope of information, and to improve the efficiency of data flow through the Encapsulation Layer, the payload should, where possible, be transported as received without modification."

4.1. Avoid Control Plane Interworking

This corollary is best understood in the context of the integrated solutions space. In this case, the architecture and design frequently achieves the worst of all possible worlds 💀 This is because integrated solutions perform poorly at both ends of the performance/CAPEX/OPEX spectrum: the protocols with the least switching demand may have to bear the cost of the most expensive, while the protocols with the most stringent requirements often must make concessions to those with different requirements. Add the various control plane interworking issues and you have a big opportunity for failure. In summary, interworking functions should be restricted to data plane interworking and encapsulations, and these functions should be carried out at the edge of the network 🧚

As described above, interworking models have been successful when there is a kind of "isomorphism" between the layers being interworked. The trade-off here, frequently described as the "Integrated vs. Ships In the Night trade-off", has been examined at various times and at various protocol layers. In general, there are few cases where such integrated solutions have proven efficient. Multi-protocol BGP [RFC2283] is a subtly different but notable exception 🌈 In this case, the control plane is independent of the format of the control data. That is, no control plane data conversion is required, in contrast with control plane interworking models such as the ATM/IP interworking envisioned by some soft-switch manufacturers, and the so-called "PNNI-MPLS SIN" interworking [ATMMPLS].

5. Packet versus Circuit Switching: Fundamental Differences ⚡

Conventional wisdom holds that packet switching (PS) is inherently more efficient than circuit switching (CS), mainly because of the efficiencies that can be gained by statistical multiplexing, and because routing and forwarding decisions are made independently in a hop-by-hop fashion [[MOLINERO2002]. Further, it is widely assumed that IP is simpler than circuit switching, and hence should be more economical to deploy and manage [MCK2002]. However, if you examine these and related assumptions, a different picture emerges (see for example [ODLYZKO98]) 👀 The following sections discuss these assumptions.

5.1. Is PS is inherently more efficient than CS? 🤔✨

Okay so, everyone knows packet switches make super efficient use of scarce bandwidth [BARAN]. That efficiency comes from the statistical multiplexing that's built into packet switching. BUT 😭 the Internet backbones seem to be really underused, and that's lowkey confusing. First question: what's the average utilization of Internet backbones, and how does that compare to long distance voice networks? Odlyzko and Coffman [ODLYZKO,COFFMAN] report that links in IP networks sat somewhere between 3% and 20% utilized. Corporate intranets were around 3%, and commercial Internet backbones ran around 15-20%. Long haul voice lines averaged about 33%. For 2002, optical networks (all services) were hovering around 11%, while their historical average was about 15% [ML2002]. So why is this happening, especially if PS is supposed to be inherently more efficient than CS? 🧐 Odlyzko and Coffman point to a few reasons:

  • (i) Internet traffic is extremely asymmetric and bursty, but links are symmetric and of fixed capacity (i.e., you don't know the traffic matrix or the required link capacities) 🫠
  • (ii) It's hard to predict traffic growth on a link, so operators tend to add bandwidth aggressively 🚀
  • (iii) Falling prices for coarser bandwidth granularity make it look more economical to add capacity in big chunks 💸

Other static factors also add to the need to "over-provision" [MC2001]. Those include protocol overhead, other kinds of equipment granularity, restoration capacity, and provisioning lag time. Honestly, it's a whole mess, and I'm obsessed 💅

5.2. Is PS simpler than CS? 🧩

So the end-to-end principle can be read as saying the complexity of the Internet belongs at the edges. But today's Internet backbone routers are extremely complex, no cap 😩 And that complexity scales with line rate. Since nobody has managed to directly analyze how complex circuit switching and packet switching are relative to each other, we look at some artifacts of each as complexity metrics instead. The ones we look at are software complexity, macro operation complexity, hardware complexity, power consumption, and density. Each one gets its own section below 🎀

5.2.1. Software/Firmware Complexity 💻

One way to measure software/firmware complexity is by counting the instructions needed to program the device. A typical software image for an Internet router needs somewhere between eight and ten million instructions (including firmware). A typical transport switch needs about three million on average [MCK2002]. 😳

That difference in software complexity has tended to make Internet routers unreliable, and it has some other notable second order effects too (e.g., it can take a long time to reboot one of these routers) 🥱 For comparison, the AT&T (Lucent) 5ESS class 5 switch has a huge number of calling features, but it only needs about twice as many lines of code as an Internet core router [EICK]. Not bad, right? 🌸

Last but not least, since routers are as much software as hardware (or even more), the cost of routers benefits less from Moore's Law than devices that are less software-intensive. That creates a bandwidth/device trade-off that favors bandwidth more than it does for less software-intensive devices 🤓

5.2.2. Macro Operation Complexity 🛠️

An Internet router's line card has to do a lot of complex stuff. That includes processing the packet header, longest prefix match, generating ICMP error messages, processing IP header options, and buffering the packet so TCP congestion control actually works. That last one typically means a buffer proportional to the line rate times the RTT, so the buffer holds around 250 ms of packet data 🤯 And that's before you even count route and packet filtering, or any QoS or VPN filtering. Like, wow 😭

A transport switch, on the other hand, only needs to map ingress time-slots to egress time-slots and interfaces. So it can be considerably less complex ✨

5.2.3. Hardware Complexity 🔌

One way to measure hardware complexity is by counting the logic gates on a line card [MOLINERO2002]. Let's look at a high-speed Internet router line card. An OC192 POS router line card has at least 30 million gates in ASICs, at least one CPU, 300 Mbytes of packet buffers, 2 Mbytes of forwarding table, and 10 Mbytes of other state memory 😵‍💫

Now compare that to a comparable transport switch line card, which has 7.5 million logic gates, no CPU, no packet buffer, no forwarding table, and on-chip state memory. Instead, the line card of an electronic transport switch usually has a SONET framer, a chip that maps ingress time-slots to egress time-slots, and an interface to the switch fabric. Simple vibes 🦋

5.2.4. Power ⚡

Transport switches have traditionally been built from simpler hardware components, so they also consume less power [PMC]. 🌿

5.2.5. Density 📦

The highest capacity transport switches have about four times the capacity of an IP router [CISCO,CIENA], and they sell for about one-third as much per Gigabit/sec. 💰 Optical (OOO) technology pushes this complexity difference even further (e.g., tunable lasers and MEMs switches, e.g., [CALIENT]). DWDM multiplexers also provide the technology to build extremely high capacity, low power transport switches 🌈

Physical footprint is another related metric. Generally, because of their higher density, transport switches have a smaller "per-gigabit" physical footprint 🏠

5.2.6. Fixed versus variable costs 🧾

Packet switching would seem to have high variable cost. That means it costs more to send the n-th piece of information with packet switching than it might on a circuit switched network. A lot of that advantage comes from how static circuit switching is. For example, circuit switching can take advantage of the pre-scheduled arrival of information to skip operations that would otherwise have to be done on incoming information. In the circuit switched case, there's no need to buffer incoming information, perform loop detection, resolve next hops, modify fields in the packet header, and so on 🙌 Many circuit switched networks also pair relatively static configuration with out-of-band control planes (e.g., SS7), which really simplifies data-plane switching. The bottom line? As data rates get large, switching packets gets more and more complex, while circuit switching scales more or less linearly 📈

5.2.7. QoS 🎟️

Look, the components of a complete solution for Internet QoS are call admission control, efficient packet classification, and scheduling algorithms. They've been the subject of extensive research and standardization for more than 10 years, but end-to-end signaled QoS for the Internet still isn't a reality 😔 QoS, meanwhile, has been part of the circuit switched infrastructure almost from the very beginning. QoS is usually deployed to decide which queuing disciplines get used when there isn't enough bandwidth to support traffic. But voice traffic isn't the same as TCP traffic here. Packet drop or severe delay can hit TCP traffic way harder, because of its congestion-aware feedback loop (in particular, TCP backoff/slow start) 🫣

5.2.8. Flexibility 🦄

Flexibility is a little harder to measure, but it's a big deal. The Internet's flexibility is a big reason it grew so fast, but that flexibility comes with a relatively high cost at the edge: you need highly trained support people. A standard rule of thumb is that in an enterprise setting, one support person is enough to provide telephone service for a group, while you need ten computer networking experts to handle the networking needs of that same group [ODLYZKO98A]. This phenomenon is also described in [PERROW]. Stressful, but iconic 💀

5.3. Relative Complexity 🧠

It's been hard to describe the relative computational complexity of circuit switching compared to packet switching in formal terms [PARK]. So the sections above describe complexity in terms of observable artifacts instead. With that in mind, it's clear that the main thing driving the complexities we just covered is the hop-by-hop independence (HBHI) that's built into the IP architecture. That's different from end to end architectures like ATM or Frame Relay 🔍

[WILLINGER2002] describes this in terms of the robustness requirement of the original Internet design, and how that requirement has driven the network's complexity. They describe a "complexity/robustness" spiral: more complexity creates more and more serious sensitivities, which then require additional robustness. That's why it's a spiral 🌀

The big takeaway from this section is that the Simplicity Principle matters for both circuit and packet switching, but it's crucial for controlling the complexity of packet networks. And with it, the OPEX and CAPEX properties too 💜 It's reinforced by the fact that packet switching is a younger, less mature discipline than circuit switching. The trend in packet switches is toward more complex line cards, while the complexity of circuit switches seems to scale linearly with line rates and aggregate capacity 📊

5.3.1. HBHI and the OPEX Challenge 🧯

Because of HBHI, we have to approach IP networks in a fundamentally different way than circuit based networks. The major OPEX challenge for IP networks is that debugging a large-scale IP network still takes a lot of expertise and understanding. That's because of the hop-by-hop independence built into packet architecture. (Note that this hop-by-hop independence isn't present in virtual circuit networks like ATM or Frame Relay.) For example, you might have to visit a big set of your routers only to find out the problem is external to your own network 😩 The debugging tools used to diagnose problems are also complex and kind of primitive. And IP has to deal with people having problems with their DNS, their mail, their news, or some new application, which usually isn't the case for TDM/ATM/etc. With IP, this can be eased by improving automation (note that much of what we mention is customer facing). Overall, lots of variables outside the network affect OPEX 🌍

Finally, the quantitative relationship between CAPEX, OPEX, and a network's inherent complexity isn't well understood. In fact, there aren't any agreed upon, quantitative metrics for describing a network's complexity, so a precise relationship between CAPEX, OPEX, and complexity is still elusive 🫶

6. The Myth of Over-Provisioning ✨

okay so, as [MC2001] and other sources point out, a huge chunk of the complexity we see on today's Internet is there to squeeze out more bandwidth utilization 🚀. Because of that, network engineers like to keep utilization below 50%, and people have been calling that "over-provisioning" 💅. But that's a total misnomer, bestie! In modern Internet backbones, the unused capacity is actually protection capacity 🛡️. You could honestly call it "1:1 protection at the IP layer" 🦋. Seen that way, an IP network run at 50% utilization is no more over-provisioned than the typical SONET network. And the perks you get from running an IP network like that are seriously iconic: close to speed of light delay and close to zero packet loss [FRALEIGH] 😭✨. Those benefits can be seen as a "side-effect" of 1:1 protection provisioning 💖.

There are also other, system-theoretic reasons to provide 1:1-like protection provisioning 🧠. The biggest one is that packet-switched networks with in-band control loops can become unstable, and they can experience oscillations and synchronization when they get congested 🌪️. The traffic dynamics are complex and non-linear, so congestion in one part of the network will spread into other parts of the network 💀. When routing protocol packets get lost because of congestion or route-processor overload, the routing state ends up inconsistent, and that can lead to traffic loops, black holes, and lost connectivity 🕳️. So yeah, statistical multiplexing can in theory give you higher network utilization 🤓. But in practice, if you want consistent performance and a reasonably stable network, the dynamics of Internet backbones favor 1:1 provisioning and its side effects 💜. That's how you keep the network stable and the delay low 🌸.

7. The Myth of Five Nines 🙈

Paul Baran, in his classic paper, "SOME PERSPECTIVES ON NETWORKS-- PAST, PRESENT AND FUTURE", stated that "The tradeoff curves between cost and system reliability suggest that the most reliable systems might be built of relatively unreliable and hence low cost elements, if it is system reliability at the lowest overall system cost that is at issue" [BARAN77] 📜✨.

Today we call this phenomenon "the myth of five nines" 🧚. Basically, so-called five nines reliability in packet network elements is considered a myth, and here's why 👀. First, since 80% of unscheduled outages are caused by people or process errors [SCOTT], there's only a 20% window in which to optimize. So if you want to increase component reliability, you add complexity (because optimization frequently leads to complexity) 🙃. And that complexity is the root cause of 80% of the unplanned outages 😭. That effectively narrows the 20% window, which means you increase the likelihood of people and process failure 💀. This is also described as a "complexity/robustness" spiral [WILLINGER2002], where more complexity creates further and more serious sensitivities, which then require additional robustness, and so on. That's why it's called a spiral 🌀.

So the conclusion is that while a system like the Internet can reach five-nines-like reliability, it's undesirable (and likely impossible) to try to make any individual component, especially the most complex ones, hit that reliability standard 🫶.

8. Architectural Component Proportionality Law 🎀

As we noted in the previous section, the computational complexity of packet switched networks like the Internet has proven hard to describe in formal terms 🥺. Still, an intuitive, high-level definition of architectural complexity might be this: the complexity of an architecture is proportional to its number of components, and the probability of achieving a stable implementation of an architecture is inversely proportional to its number of components 📈📉. As described above, components include discrete elements such as hardware elements, space and power requirements, as well as software, firmware, and the protocols they implement 🔧.

Stated more abstractly, it looks like this:

    Let

      A   be a representation of architecture A,

      |A| be number of distinct components in the service
          delivery path of architecture A,

      w   be a monotonically increasing function,

      P   be the probability of a stable implementation of an
          architecture, and let

    Then

      Complexity(A) = O(w(|A|))
      P(A)          = O(1/w(|A|))

    where

    O(f) = {g:N->R | there exists c > 0 and n such that g(n)
    < c*f(n)}

So, in plain English: O(f) is the set of functions g for which there exists a constant c and a number n, such that g(n) is smaller than or equal to c*f(n) for all n 🤓. Basically, O(f) is the set of all functions that don't grow faster than f, ignoring constant factors 💅.

Interestingly, the Highly Optimized Tolerance (HOT) model [HOT] tries to characterize complexity in general terms 🧐. HOT is one recent attempt to build a general framework for studying complexity, and it belongs to a family of abstractions generally called "the new science of complexity" or "complex adaptive systems" 🌈. In HOT semantics, tolerance means that "robustness in complex systems is a constrained and limited quantity that must be carefully managed and protected." 🛡️ One focus of the HOT model is characterizing heavy-tailed distributions, like Complexity(A) in the example above. Other examples include forest fires, power outages, and Internet traffic distributions 🔥⚡. In particular, Complexity(A) tries to map the extreme heterogeneity of the parts of the system (the Internet), and the effect of organizing those parts into highly structured networks, with hierarchies and multiple scales 🪆.

8.1. Service Delivery Paths 🛤️

The Architectural Component Proportionality Law (ACPL) states that the complexity of an architecture is proportional to its number of components 📏.

COROLLARY: Minimize the number of components in a service delivery path, where the service delivery path can be a protocol path, a software path, or a physical path.

This corollary is a big deal, because it follows from the ACPL 👑. The path between a customer and the desired service is especially sensitive to the number and complexity of the elements along it 😬. That's because the complexity "smoothing" we find at high levels of aggregation [ZHANG] goes missing as you move closer to the edge, and there are complex interactions with backoffice and CRM systems too 📞. Some architectures never found a market because of this effect, including TINA-based CRM systems and CORBA/TINA based service architectures 💀. The basic lesson was that the only ways to deploy these systems were "Limited scale deployments (such) as in Starvision can avoid coping with major unproven scalability issues", or "Otherwise need massive investments (like the carrier-grade ORB built almost from scratch)" [TINA] 😵‍💫. In other words, these systems had complex service delivery paths, and they were too complex to be feasibly deployed 🚫.

9. Conclusions 🎉

This document is basically an attempt to write down long-understood Internet architectural principles 📝. The unifying principle here is best expressed by the Simplicity Principle, which says complexity must be controlled if you hope to scale a complex object efficiently 🧩. The idea that simplicity itself can lead to some form of optimality has been a common theme throughout history, and it has been stated in many other ways along many dimensions 🌟. For example, there's the maxim known as Occam's Razor, formulated by the medieval English philosopher and Franciscan monk William of Ockham (ca. 1285-1349), which states "Pluralitas non est ponenda sine neccesitate" or "plurality should not be posited without necessity." (That's why Occam's Razor is sometimes called "the principle of unnecessary plurality" and "the principle of simplicity" 🤌.) A more contemporary version says the simplest explanation for a phenomenon is the one preferred by nature 🌿. Other versions of the same idea include the KISS (Keep It Simple Stupid) principle and the Principle of Least Astonishment, which says the most usable system is the one that least often leaves users astonished 😮. [WILLINGER2002] gives a more theoretical discussion of "robustness through simplicity" 🧠, and when discussing the PSTN, [KUHN87] says that in most systems, "a trade-off can be made between simplicity of interactions and looseness of coupling" 🔗.

When applied to packet switched network architectures, the Simplicity Principle has implications that some people may consider heresy 😱. For example, highly converged approaches are likely to be less efficient than "less converged" solutions. Otherwise stated, the "optimal" convergence layer may be much lower in the protocol stack than conventional belief holds 🪜. The analysis above also leads to several conclusions that go against the conventional wisdom around packet networking 🙅‍♀️. Perhaps the most significant is the belief that packet switching is simpler than circuit switching. That belief has led to conclusions like "since packet is simpler than circuit, it must cost less to operate" 💸. This study finds the opposite: by examining the metrics described above, packet switching turns out to be more complex than circuit switching 🤯. And this conclusion is backed up by the fact that normalized OPEX for data networks is typically significantly greater than for voice networks [ML2002] 📊.

Finally, the important takeaway of this work is that for packet networks of the scale of today's Internet or larger, we must strive for the simplest possible solutions if we hope to build cost-effective infrastructures 💅✨. This idea is eloquently stated in [DOYLE2002]: "The evolution of protocols can lead to a robustness/complexity/fragility spiral where complexity added for robustness also adds new fragilities, which in turn leads to new and thus spiraling complexities". That's exactly the phenomenon the Simplicity Principle is designed to avoid 🫶🌈.

10. Security Considerations 🔐

This document does not directly affect the security of any existing Internet protocol 🙂. However, following the Simplicity Principle does have a direct effect on our ability to implement secure systems 🚨. As a system's complexity grows, it becomes more difficult to model and analyze. That makes it harder to find and understand the security implications inherent in its architecture, design, and implementation 😰💀.

11. Acknowledgments 💜

Many of the ideas for comparing the complexity of circuit switched and packet switched networks were inspired by conversations with Nick McKeown 💭. Scott Bradner, David Banister, Steve Bellovin, Steward Bryant, Christophe Diot, Susan Harris, Ananth Nagarajan, Andrew Odlyzko, Pete and Natalie Whiting, and Lixia Zhang all made many helpful comments on early drafts of this document 🎀🫶.

12. References 📚✨

okay bestie, this is the reading list 🫶 every source the RFC leans on, all still here and all in their original form so you can go find them yourself 🧚

  • [AHUJA] "The Impact of Internet Policy and Topology on Delayed Routing Convergence", Labovitz, et. al. Infocom, 2001.
  • [ATMMPLS] "ATM-MPLS Interworking Migration Complexities Issues and Preliminary Assessment", School of Interdisciplinary Computing and Engineering, University of Missouri-Kansas City, April 2002
  • [BARAN] "On Distributed Communications", Paul Baran, Rand Corporation Memorandum RM-3420-PR, http://www.rand.org/publications/RM/RM3420", August, 1964.
  • [BARAN77] "SOME PERSPECTIVES ON NETWORKS--PAST, PRESENT AND FUTURE", Paul Baran, Information Processing 77, North-Holland Publishing Company, 1977,
  • [BRYANT] "Protocol Layering in PWE3", Bryant et al, Work in Progress.
  • [CAIDA] http://www.caida.org
  • [CALLIENT] http://www.calient.net/home.html
  • [CARLSON] "Complexity and Robustness", J.M. Carlson and John Doyle, Proc. Natl. Acad. Sci. USA, Vol. 99, Suppl. 1, 2538-2545, February 19, 2002. http://www.pnas.org/cgi/doi/10.1073/pnas.012582499
  • [CIENA] "CIENA Multiwave CoreDiretor", http://www.ciena.com/downloads/products/coredirector.pdf
  • [CISCO] http://www.cisco.com
  • [CLARK] "The Design Philosophy of the DARPA Internet Protocols", D. Clark, Proc. of the ACM SIGCOMM, 1988.
  • [COFFMAN] "Internet Growth: Is there a 'Moores Law' for Data Traffic", K.G. Coffman and A.M. Odlyzko, pp. 47-93, Handbook of Massive Data Stes, J. Elli, P. M. Pardalos, and M. G. C. Resende, Editors. Kluwer, 2002.
  • [DOYLE2002] "Robustness and the Internet: Theoretical Foundations", John C. Doyle, et. al. Work in Progress.
  • [EICK] "Visualizing Software Changes", S.G. Eick, et al, National Institute of Statistical Sciences, Technical Report 113, December 2000.
  • [MOLINERO2002] "TCP Switching: Exposing Circuits to IP", Pablo Molinero-Fernandez and Nick McKeown, IEEE January, 2002.
  • [FLOYD] "The Synchronization of Periodic Routing Messages", Sally Floyd and Van Jacobson, IEEE ACM Transactions on Networking, 1994.
  • [FLOYD2001] "A Report on Some Recent Developments in TCP Congestion Control, IEEE Communications Magazine, S. Floyd, April 2001.
  • [FRALEIGH] "Provisioning IP Backbone Networks to Support Delay-Based Service Level Agreements", Chuck Fraleigh, Fouad Tobagi, and Christophe Diot, 2002.
  • [GRIFFIN] "What is the Sound of One Route Flapping", Timothy G. Griffin, IPAM Workshop on Large-Scale Communication Networks: Topology, Routing, Traffic, and Control, March, 2002.
  • [HANDLEY] "On Inter-layer Assumptions (A view from the Transport Area), slides from a presentation at the IAB workshop on Wireless Internetworking", M. Handley, March 2000.
  • [HOT] J.M. Carlson and John Doyle, Phys. Rev. E 60, 1412-1427, 1999.
  • [ISO10589] "Intermediate System to Intermediate System Intradomain Routing Exchange Protocol (IS-IS)".
  • [JACOBSON] "Congestion Avoidance and Control", Van Jacobson, Proceedings of ACM Sigcomm 1988, pp. 273-288.
  • [KARN] "TCP vs Link Layer Retransmission" in P. Karn et al., Advice for Internet Subnetwork Designers, Work in Progress.
  • [KUHN87] "Sources of Failure in the Public Switched Telephone Network", D. Richard Kuhn, EEE Computer, Vol. 30, No. 4, April, 1997.
  • [L2TPV3] Lan, J., et. al., "Layer Two Tunneling Protocol (Version 3) -- L2TPv3", Work in Progress.
  • [MC2001] "U.S Communications Infrastructure at A Crossroads: Opportunities Amid the Gloom", McKinsey&Company for Goldman-Sachs, August 2001.
  • [MCK2002] Nick McKeown, personal communication, April, 2002.
  • [ML2002] "Optical Systems", Merril Lynch Technical Report, April, 2002.
  • [NAVE] "The influence of mode coupling on the non-linear evolution of tearing modes", M.F.F. Nave, et al, Eur. Phys. J. D 8, 287-297.
  • [NEUMANN] "Cause of AT&T network failure", Peter G. Neumann, http://catless.ncl.ac.uk/Risks/9.62.html#subj2
  • [ODLYZKO] "Data networks are mostly empty for good reason", A.M. Odlyzko, IT Professional 1 (no. 2), pp. 67-69, Mar/Apr 1999.
  • [ODLYZKO98A] "Smart and stupid networks: Why the Internet is like Microsoft". A. M. Odlyzko, ACM Networker, 2(5), December, 1998.
  • [ODLYZKO98] "The economics of the Internet: Utility, utilization, pricing, and Quality of Service", A.M. Odlyzko, July, 1998. http://www.dtc.umn.edu/~odlyzko/doc/networks.html
  • [PARK] "The Internet as a Complex System: Scaling, Complexity and Control", Kihong Park and Walter Willinger, AT&T Research, 2002.
  • [PERROW] "Normal Accidents: Living with High Risk Technologies", Basic Books, C. Perrow, New York, 1984.
  • [PMC] "The Design of a 10 Gigabit Core Router Architecture", PMC-Sierra, http://www.pmc-sierra.com/products/diagrams/CoreRouter_lg.html
  • [RFC1629] Colella, R., Callon, R., Gardner, E. and Y. Rekhter, "Guidelines for OSI NSAP Allocation in the Internet", RFC 1629, May 1994.
  • [RFC1925] Callon, R., "The Twelve Networking Truths", RFC 1925, 1 April 1996.
  • [RFC1958] Carpenter, B., Ed., "Architectural principles of the Internet", RFC 1958, June 1996.
  • [RFC2283] Bates, T., Chandra, R., Katz, D. and Y. Rekhter, "Multiprotocol Extensions for BGP4", RFC 2283, February 1998.
  • [RFC3155] Dawkins, S., Montenegro, G., Kojo, M. and N. Vaidya, "End-to-end Performance Implications of Links with Errors", BCP 50, RFC 3155, May 2001.
  • [ROMANOV] "Dynamics of TCP over ATM Networks", A. Romanov, S. Floyd, IEEE JSAC, vol. 13, No 4, pp.633-641, May 1995.
  • [SALTZER] "End-To-End Arguments in System Design", J.H. Saltzer, D.P. Reed, and D.D. Clark, ACM TOCS, Vol 2, Number 4, November 1984, pp 277-288.
  • [SCOTT] "Making Smart Investments to Reduce Unplanned Downtime", D. Scott, Tactical Guidelines, TG-07-4033, Gartner Group Research Note, March 1999.
  • [SPILLMAN] "The Law of Diminishing Returns:, W. J. Spillman and E. Lang, 1924.
  • [STALLINGS] "Data and Computer Communications (2nd Ed)", William Stallings, Maxwell Macmillan, 1989.
  • [TENNENHOUSE] "Layered multiplexing considered harmful", D. Tennenhouse, Proceedings of the IFIP Workshop on Protocols for High-Speed Networks, Rudin ed., North Holland Publishers, May 1989.
  • [THOMPSON] "Nonlinear Dynamics and Chaos". J.M.T. Thompson and H.B. Stewart, John Wiley and Sons, 1994, ISBN 0471909602.
  • [TINA] "What is TINA and is it useful for the TelCos?", Paolo Coppo, Carlo A. Licciardi, CSELT, EURESCOM Participants in P847 (FT, IT, NT, TI)
  • [WAKEMAN] "Layering considered harmful", Ian Wakeman, Jon Crowcroft, Zheng Wang, and Dejan Sirovica, IEEE Network, January 1992, p. 7-16.
  • [WARD] "Custom fluorescent-nucleotide synthesis as an alternative method for nucleic acid labeling", Octavian Henegariu*, Patricia Bray-Ward and David C. Ward, Nature Biotech 18:345-348 (2000).
  • [WILLINGER2002] "Robustness and the Internet: Design and evolution", Walter Willinger and John Doyle, 2002.
  • [ZHANG] "Impact of Aggregation on Scaling Behavior of Internet Backbone Traffic", Sprint ATL Technical Report TR02-ATL-020157 Zhi-Li Zhang, Vinay Ribeiroj, Sue Moon, Christophe Diot, February, 2002.

13. Authors' Addresses 🖊️

the two people who wrote this, and how to reach them 💌

Randy Bush EMail: randy@psg.com

David Meyer EMail: dmm@maoz.com

This bit is legal text, so I've kept it word-for-word. It's not mine to remix 🫶

Copyright (C) The Internet Society (2002). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement 🌸

Funding for the RFC Editor function is currently provided by the Internet Society. 💖

📦 where this page comes from