RFC 7980

A Framework for Defining Network Complexity

📅 published October 2016, from ietf

1. Introduction ✨

Okay so bestie, let's get into it 🎀 This is Part 1 of the RFC 7980 glow-up, and the title block comes first because it's giving "who's who" energy 💅

Title block 🦋 This is an Independent Submission and it's Informational, which means it's published for info, not as an Internet Standard. The authors are M. Behringer and A. Retana from Cisco Systems, R. White from Ericsson, and G. Huston from APNIC. It's from October 2016, and the ISSN is 2070-1721. Also, nothing about obsoletes or updates is listed in the title block, so it doesn't replace or change any other RFC 🌸

The title is A Framework for Defining Network Complexity 🧚 and the big idea is that complexity is SUCH a popular word in network design, but nobody has a generally agreed definition for it 💀 Like, everyone uses it, nobody can pin it down. The abstract says the document gives a framework for research on the topic, plus some practical trade-off examples 💜

Then there's the status bit: this is a snapshot of the IRTF's Network Complexity Research Group (NCRG) at the time it closed, not final results 🍓 It's a basis for future work, so lowkey it's a starting point, not the finish line ✨

1. Introduction 🌈

Network design is basically the art of finding the simplest solution to a problem 🫶 Complexity is already baked into the design process, so engineers don't ask if there should be complexity, they ask how much is needed 💅 But then, how do you even measure "how much"? That's assuming there's a way to characterize complexity, and honestly that's still a research thing rather than a solved problem 😭 Right now, design decisions are based on rough guesses about a network's complexity instead of a solid understanding 🙈

Here's the plan for the doc 🎀 It starts with general considerations and foundational definitions, then gives examples of the trade-offs engineers make all the time. The point is that there's no single answer to complexity. It's a managed trade-off between a bunch of parameters 💖 After that, it lists parameters engineers should think about when they try to measure complexity or build a framework around it. The list isn't complete, but it's a guide to known areas of investigation AND areas that still need digging into 🔥

Two purposes here, bestie ✨ First, it helps researchers aim their work at what network designers actually care about, which makes their research way more useful 💜 It might even inspire research into stuff nobody has looked at yet 🦋 Second, it helps network engineers spot where complexity might be "hiding" in their networks and understand how it interacts with design and deployment 👑

The IRTF Network Complexity Research Group (NCRG) [ncrg] was meant to define a framework for network complexity research, while also recognizing that metrics for overall network complexity may be impossible 😭 This doc summarizes that group's work up to its closure in 2014 🌸 Again, it's not final results, just a snapshot of an ongoing activity as a basis for future work ⭐

Lots of existing research on network complexity is listed on the Network Complexity Wiki [wiki] 🧚 The wiki also has background info on past meetings, previous research, and more, so go check it out if you're obsessed like me 💅

2. General Considerations 💖

2.1. The Behavior of a Complex Network 🦋

There's no generally accepted definition of network complexity, but we do have some understanding of how a complex network behaves 🌈 A complex network has some or all of these properties:

  • Self-Organization 🎀: A network runs some protocols and processes without external control, like a routing process or failover mechanisms. Those mechanisms interacting with each other can lead to complex behavior 💅
  • Unpredictability 🔮: In a complex network, a local change can have unpredictable effects on the behavior of the global network 😭
  • Emergence 🌸: The behavior of the system as a whole isn't reflected in the behavior of any single component 🧚
  • Non-linearity 📈: An input into the network produces a non-linear result 💀
  • Fragility 💔: A small local input can break the entire system 🙈

2.2. Complex versus Complicated 🍓

"Complex" and "complicated" get used interchangeably all the time, but they describe different (though overlapping) properties ✨ The RG made some statements about the two terms, but they'd need more refinement before they count as formal definitions 💜

  • A "complicated" system is a deterministic system that can be understood with an appropriate level of analysis 🫶 It's often an externally applied attribute rather than an intrinsic property of a system, and it's typically associated with systems that need deep or significant analysis 🔍
  • A "complex" system, by comparison, is an intrinsic property of a system 👑 It's typically associated with emergent behaviors, where the behavior of the whole isn't fully described by the sum of its components. Complex systems are often linked to components that show high levels of interaction and feedback 🔥

2.3. Robust Yet Fragile 🌈

Networks typically follow the "robust yet fragile" paradigm 💅 They're designed to be robust against a set of failures, but they're very vulnerable to other failures 😭 Doyle [Doyle] explains it with an example: the Internet is robust against single-component failure but fragile to targeted attacks 🎀 This property also touches on the fact that all network designs are necessarily making trade-offs between different design goals ✨

The simplest one is "Good, Fast, Cheap: Pick any two (you can't have all three)", as described in "The Twelve Networking Truths" [RFC1925] 💖 In real network design, you have to juggle trade-offs between lots of aspects, like scope, time, and cost across the planning, design, implementation, and management of a network platform 🧚 Section 3 gives some trade-off examples, and parameters are covered in Section 4 🦋

That's the end of Part 1! 🌈💖 Part 2 is coming next, so stay tuned, bestie 🫶

2.4. The Complexity Cube ✨

okay so complex tasks on a network can get done in different places 🧚 for example, routing can be controlled by central algorithms with the result pushed out to everyone (that's the OpenFlow model 💅). Or the routing algorithm can run completely distributed, with every router figuring it out together (routing protocols like OSPF or IS-IS 🦋). Or a human operator could calculate the routing tables and set them up statically by hand 🙈. Behringer [Behringer] calls these three axes a "complexity cube", with the sides being network elements, central systems, and human operators. Any function can be built on any of those three axes, and that choice probably changes how complex the whole system ends up being 💜

bestie, when you're talking about network complexity, there are SO many influencing factors you have to look at before you get the full picture 👀 here are some of the big ones:

  • State in the Network: this covers the network elements, like routers, switches (along with their OS, including the protocols), lines, central systems, and so on. It also includes how many protocols run on network devices and how algorithmically complicated those protocols are 💖
  • Human Operators: complexity often shows up as a network that human operators don't completely understand 💀 human error is a primary source of catastrophic failures, so it must be taken into account, no exceptions 🫶
  • Classes/Templates: honestly, it's not about counting lines in a config or counting hardware boxes. What matters more is how many classes those things can be derived from ✨ so 1000 interfaces that are all configured identically are probably LESS complex than 5 that are each configured completely differently 🙃 iconic.
  • Dependencies and Interactions: how many dependencies there are between elements, plus how those elements interact with each other, affects how complex the network gets 🕸️
  • Total Cost of Ownership (TCO): TCO could be a good metric for network complexity, but only if the calculation counts every influencing factor, like how long it takes staff to train up so they can maintain the network 🎀
  • Benchmark Unit Cost (BUC): BUC is a related metric that shows what it costs to operate a certain component 💅 if it's calculated well, it reflects at least part of that component's complexity. So the way TCO or BUC gets calculated can help you come up with a complexity metric 🌈
  • Churn / Rate of Change: the rate of change in a network can add to its complexity all on its own, especially when a bunch of components in the overall network are interacting 🌪️

Networks also differ in their intended purpose (think enterprise networks versus public carrier networks 🏢) and in their intended roles (think "access" networks versus "core" transit networks 🛣️). Those differences in purpose and role can change how much complexity people will tolerate, and even which metrics for complexity make sense in each scenario 🫠 so no, this isn't a space where one single way of measuring complexity, with one single acceptability threshold, is appropriate. main character energy would be pretending there's one answer 🙅‍♀️

2.6. Technical Debt 💸

okay this one's so real 😭 lots of changes in a network get made while depending on the network that already exists. Often a suboptimal decision gets made, because the optimal decision is hard or even impossible to pull off at that moment. Over time, all those suboptimal changes start causing serious complexity by themselves, and that complexity wouldn't have existed if the optimal solution had been implemented in the first place 💔

The term "technical debt" refers to that accumulated complexity from suboptimal changes over time. Just like financial debt, the idea is that technical debt has to be repaid one day, by cleaning up the network or the software 🧾✨ the bill always comes eventually, periodt.

2.7. Layering Considerations 🍓

when you look at the bigger picture (applications, transport services, network services, and media services), you can engineer responses to certain types of desired application behavior in lots of different ways, and those ways can involve different layers of the so-called network protocol stack 🥞 for example, Quality of Service (QoS) could be engineered at any of these layers, or even across a bunch of different combinations of layers 🎀

complexity shows up when mutually incompatible measures get used together (like error detection and retransmission at the media layer, combined with the TCP transport protocol 😵‍💫), or when an assumption made in one layer gets violated by another layer. That leads to surprising outcomes that can create complex interactions, like oscillation, because different layers use different timers for retransmission 🌀 these issues are why people have the perspective that increased layering frequently increases complexity [RFC3439] 🫣

and yes, this research is focused on network complexity, but the interactions between the network and the end-to-end transport protocols, application layer protocols, and media properties are relevant too 🦋

3. Trade-Offs ⚖️

network complexity is a system-level problem, not a component-level one 👑 the overall system complexity can end up being more than just the sum of the complexity of each individual piece 🤯

there are two basic ways to tackle a system-level problem: interfaces and continuums 🧩 with interfaces, you treat each piece of the system as a "black box" and work hard to fully understand the interfaces between those black boxes. With a continuum, you try to understand how a single change or element affects the entire system as a set of trade-offs 🎢

network complexity can be approached from either of those perspectives, but in this document we chose the continuums of trade-offs approach for looking at the system-level impact of network complexity 💜 in theory, modifying the network to fix one particular problem (or class of problems) adds complexity, and that complexity raises the likelihood (or the appearance) of a different class of problems 🙃 so the key steps are discovering these continuums of trade-offs, and then figuring out how to measure each one. That's how you understand and measure system-level complexity this way ✨

the following sections describe five such continuums, but more may be possible 🌈

  • Control-Plane State versus Optimal Forwarding Paths (or its opposite measure, stretch)
  • Configuration State versus Failure Domain Separation
  • Policy Centralization versus Optimal Policy Application
  • Configuration State versus Per-Hop Forwarding Optimization
  • Reactivity versus Stability

3.1. Control-Plane State versus Optimal Forwarding Paths (Stretch) 🦋

control-plane state is the aggregate amount of information the control plane carries through the network in order to produce the forwarding table on each device 📦 every extra piece of information added to the control plane adds to the complexity of that plane. That includes more-specific reachability information, policy information, additional control planes for virtualization and tunneling, or more precise topology information 🧠 and that added complexity adds to the burden of monitoring, understanding, troubleshooting, and managing the network 😩

removing control-plane state isn't always a net positive for the network as a system, though 🙅‍♀️ removing it almost always makes forwarding and handling of packets less optimal. That decreased optimality can be termed "stretch", which is defined as the difference between the absolute shortest (or best) path traffic could take through the network and the path the traffic actually takes 🛣️ stretch is expressed as the difference between the optimal path and the actual path. Here's an example of this trade-off 👇

                            +---R1---+
                            |        |
    (aggregate: 192.0.2/24) R2       R3 (aggregate: 192.0.2/24)
                            |        |
                            R4-------R5
                            |
   (announce: 192.0.2.1/32) R6

assume each link is of equal cost in this figure, and that R6 is advertising 192.0.2.1/32 📣

for R1, the shortest path to 192.0.2.1/32 (advertised by R6) is along the path [R1,R2,R4,R6] 🏃‍♀️

now, say the network administrator decides to aggregate reachability information at R2 and R3, advertising 192.0.2.0/24 towards R1 from both of those points 🙌 this cuts down the overall complexity of the control plane, since less information gets carried past these two routers (at R1 only, in this case) ✂️

but here's the twist 😳 aggregating at R2 and R3 can make both routes towards 192.0.2.1/32 look like equal cost paths to R1. There's no particular reason R1 should pick the shortest path through R2 over the longer path through R3 🤷‍♀️ and that, in effect, increases the stretch of the network. The shortest path from R1 to R6 is 3 hops, and it's the path that will always get chosen before aggregation is configured. Assuming half the traffic goes through R2 (3 hops) and half through R3 (4 hops), the network is stretched by ((3+4)/2) - 3), or .5, a "half a hop" 📏💀

last thing, traffic engineering through various tunneling mechanisms is, at a broad level, adding control-plane state to get more optimal forwarding (or better network utilization) 🎀 optimizing network utilization may mean detuning stretch (intentionally increasing stretch) to boost overall utilization and efficiency. That's just another instance of control-plane state (and therefore complexity) being weighed against optimal forwarding through the network ⚖️✨

3.2. Configuration State versus Failure Domain Separation ✨

Okay so, bestie, a failure domain is basically the group of devices that feel it when the topology or configuration changes in the control plane. 🫶 Bigger failure domains are way more prone to cascading failures, so smaller ones are normally the move. 💅

The main tool for shrinking a failure domain is information hiding, and there are two big things you can hide: reachability information and topology information. 🦋 Aggregating reachability is giving summarization. Like, 192.0.2.1/32, 192.0.2.2/32, and 192.0.2.3/32 can all get squished into the single route 192.0.2.0/24, with the metrics of the component routes aggregated too. 🎀 Honestly, aggregation is super natural in IP networks, starting from individual hosts getting grouped into a subnet at the network edge. 🌸 Topology aggregation looks like summarizing routes at a link-state flooding domain boundary, or just not having topology info at all in a distance-vector protocol. 🧚

Now here's the tea. Limiting failure domains looks like an absolute good for network complexity, but there's a real trade-off in configuration complexity. ✨ The more failure domain edges you make, the more complex the configuration gets. 😭 It's especially true when you redistribute routing info between multiple control-plane processes to make those boundaries, because moving between different control-plane types wrecks the consistent metrics that most of them use to build loop-free paths. 💀 Redistribution is the real menace here since it opens the door to very destructive positive feedback loops inside the control plane. 🔥 Some examples of the complexity you get from making failure domain boundaries: route filters, routing aggregation configuration, and metric modifications to engineer traffic across those boundaries. 🙈

Going back to the network from the previous section, aggregating routing info at R2 and R3 splits the network into two failure domains: (R1, R2, R3) and (R2, R3, R4, R5). 🦋 A failure at R5 should have no impact on the forwarding information at R1. 👑

But wait, there's a catch! 🙈 A false failure domain separation happens when the metric of the aggregate route advertised by R2 and R3 depends on one of the routes inside the aggregate. For example, if the metric of the 192.0.2.0/24 aggregate is derived from the metric of the component 192.0.2.1/32, then a failure of that one component changes the forwarding table at R1. 💀 So the control plane never really got split into two distinct failure domains. 😭 The extra complexity in this illustration network is managing the configuration needed to aggregate the control-plane info, plus managing the metrics so the control plane is actually separated into two distinct failure domains. 💅

Replacing aggregation with redistribution adds the complexity of managing the feedback of routing info redistributed between the failure domains. 🌈 For instance, if R1, R2, and R3 run one routing protocol while R2, R3, R4, R5, and R6 run another, R2 and R3 could be configured to redistribute reachability info between those two control planes. 🎀 This can split the control plane into multiple failure domains (depending on exactly how redistribution is configured), but the cost is creating and managing that redistribution config. 🫶 And on top of that, R3 must be configured to block routing info redistributed at R2 towards R1 from being redistributed (again) towards R4 and R5. 🔥

3.3. Policy Centralization versus Optimal Policy Application ✨

Another big area where control-plane complexity meets optimal network utilization is QoS. 💖 Two specific actions are required to optimize the flow of traffic through a network: marking and Per Hop Behaviors (PHBs). 🦋 Instead of examining each packet at each forwarding device, packets are often marked, or classified, in some way (typically through Type of Service bits) so they can be handled consistently at all forwarding devices. 🧚

Packet-marking policies must be configured on specific forwarding devices throughout the network. 💅 Distributing marking closer to the edge means configuring and managing more devices, but it gives optimal forwarding at a bigger number of network devices. 🌸 Moving marking toward the network core means packets get marked for proper handling across fewer devices. ✨ Same deal with PHBs: every device a packet passes through with the correct PHBs configured makes packet handling more consistent through the network, but it also means more devices to configure and manage for those PHBs. 🎀 Here's the network we'll use to illustrate this:

                              +----R1----+
                              |          |
                           +--R2--+   +--R3--+
                           |      |   |      |
                           R4     R5  R6     R7

In this network, marking and PHB configuration may be configured on any device, R1 through R7. 🫶

Say marking is configured at the network edge. Then four devices (R4, R5, R6, R7) must be configured, including ongoing configuration management, to mark packets. 👑 Moving packet marking to R2 and R3 halves the number of devices where marking config must be managed, but the cost is inconsistent packet handling at the inbound interfaces of R2 and R3 themselves. 🙈

So, reducing the number of devices that need managed configs for packet marking also reduces optimal packet flow through the network. 💀 Assuming packet marking is actually configured along the edge of this network, configuring PHBs on different devices has the same trade-off of managed config versus optimal traffic flow. 🌈 If the correct PHBs are configured on R1, R2, and R3, packets get handled correctly at each hop, but the cost is managing PHB config on three devices. 🦋 Configuring a single device for the correct PHBs (R1, for instance) cuts down the config management, but at the cost of less than optimal packet handling along the entire path. 😭

3.4. Configuration State versus Per-Hop Forwarding Optimization ✨

The number of PHBs configured along a forwarding path has the same complexity versus optimality trade-off we just covered. 💅 The more classes (or queues) traffic gets divided into, the more fine-grained that traffic can be managed as it moves through the network. 🎀 But every class of service has to be managed, both in terms of configuration and in how it interacts with the other classes configured in the network. 🫶

3.5. Reactivity versus Stability ⚡

okay so bestie, this is the big one 😭 how fast should the control plane react when something changes? sounds like an easy question but it is NOT, it's a whole thing 💅

the control plane's convergence (how long it takes the network to figure out the new situation) breaks down into four essential parts:

  • Detecting the change
  • Propagating information about the change
  • Determining the best path(s) through the network after the change
  • Changing the forwarding path at each network element along the modified paths

every one of these can be sped up to make convergence faster 🏎️ but some of those upgrades come with extra complexity, so it's never free bestie 🫶

detecting changes can get super quick with faster echo (or hello) mechanisms, lower-layer physical detection, and other methods 🦋 BUT each of these only works if you're ready to evaluate and manage false positives and high rates of topology change. so you trade speed for drama 🙈

like, if the state of a link change can be detected in 10 ms, the link could theoretically flip 50 times in a single second 💀 and it would be impossible to tune a control plane to react to topology changes at that rate. feeding topology change info into the control plane that fast can destabilize the control plane, and hence the whole network 😭 so to counter this, most techniques that quickly detect link-down events include some form of dampening mechanism 🧊 and yes, configuring and managing those dampeners adds complexity too. literally everything has a cost 🥲

propagating changes is its own vibe 🌊 the info has to reach every device along the path so each one can compute new forwarding tables. in high-speed environments this can happen in tens of milliseconds, so multiple changes could get propagated per second. pushing info at that rate into the control plane risks overloading the processes and devices taking part, and it can create destructive positive feedback loops in the network 💀 to avoid all that, most control-plane protocols regulate how fast any individual device is allowed to transmit information about network changes 🚦

a recent innovation here is exponential backoff ✨ the first change goes out quickly, and subsequent changes get sent more slowly. these techniques all keep the destabilizing effects of rapid information flows in check, but the price is the added complexity of configuring and managing the rate at which the control plane propagates information about network changes 🎀

every control plane needs some kind of algorithmic calculation to find the best path through the network to any given destination 🗺️ these algorithms are often lightweight, but they still need some memory and computational power to run. rapid changes can overwhelm the devices running them, especially if changes arrive faster than the algorithm can finish 😵 once a device becomes processor or memory bound, it could fail completely and cause a bigger network outage 🚨 so to prevent computational overload, control-plane protocols use timers that limit how often they can compute the best path. these timers are often exponential too, so the first computation runs quickly while later ones get delayed 🐢 and yep, configuring and managing those timers is another source of complexity 🌀

another way to make the control plane react faster is to precompute alternate paths at each device and maybe preinstall forwarding info into local forwarding tables 🏃‍♀️ but precomputing alternate paths needs additional state, and often extra algorithms and techniques get configured and deployed too. all of that adds some complexity to configuring and managing the network 📋

and in some situations (for some topologies), a tunnel is required to pass traffic around a network failure or topology change 🚇 the tunnels aren't configured by hand, but they still add complexity to both the forwarding and control planes. not me having to deal with more moving parts 😩

4. Parameters 📊

okay so in Section 3 we went through a bunch of network design trade-offs to show the real choices operators have to make 💅 the list of parameters to weigh in those trade-offs is HUGE, so a complete listing might not be possible 😭 also, the dependencies between the metrics are super complex and still need more study 🧠 this document tries to define a methodology and an overall high-level structure, and that's the goal here ✨

to analyze trade-offs, we have to formalize them 📐 the parameters are long and can be complicated themselves. like, "cost" can be a simple one-dimensional metric, but "extensibility" and "optimal forwarding state" are way harder to pin down 🤔

a list of parameters to trade off includes metrics like:

  • State: How much state needs to be held in the control plane, forwarding plane, configuration, etc.? 🗃️
  • Cost: How much does the network cost to build and run (i.e., capital expenditure (capex) and operating expenses (opex))? 💸
  • Bandwidth/Delay/Jitter: Traffic characteristics between two points (average, max, etc.) 📶
  • Configuration Complexity: How hard is it to configure and maintain the configuration? 🛠️
  • Susceptibility to Denial of Service: How easy is it to attack the service? 🥊
  • Security (Confidentiality/Integrity): How easy is it to sniff/modify/insert the data flow? 🔐
  • Scalability: To what size can I grow the network/service? 📈
  • Stability: How stable is the network under the influence of local change? 🧘
  • Reactivity: How fast does the network converge or adapt to new situations? ⚡
  • Extensibility: Can I use the network for other services in the future? 🌱
  • Ease of Troubleshooting: Are failure domains separated? How hard is it to find and correct problems? 🔍
  • Optimal Per-Hop Forwarding Behavior 🎯
  • Predictability: If I change a parameter, what will happen? 🔮
  • Clean Failure: When a problem arises, does the root cause lead to deterministic failure? 🧯

5. Elements of Complexity 🧩

okay bestie, complexity can show up in all sorts of elements in a networked system 💖 like, the configuration of a network element reflects some of the complexity in the system, or an algorithm a protocol uses might be more or less complex 🤓 when classifying complexity, the first question to ask is "WHAT is complex?" ❓ this section gives you a method to answer that 🫶

5.1. The Physical Network (Hardware) 🔌

the set of network devices and wiring has a certain complexity of its own 🧵 for example, adding a redundant link between two locations makes the network more complex, but it also gives you more redundancy 🛟 and network devices can be more or less modular, which affects complexity in a trade-off against ease of maintenance, availability, and upgradability 🧱

5.2. Algorithms 🧮

the behavior of the physical network isn't just defined by the hardware, it's also defined by algorithms running on network elements and in central locations 🖥️ every algorithm has some intrinsic complexity, and that's the subject of research on software complexity 📚

5.3. State in the Network 🗂️

the way a network element treats traffic is largely defined by the state in the network, in the form of configuration, routing state, security measures, and so on 🛡️ Section 3.1 shows an example where more control-plane state allows for more precise forwarding 🎯

5.4. Churn 🌪️

the rate of change is itself a parameter in complexity, and it needs to be weighed against the other parameters ⚖️ Section 3.5 explains a trade-off between how fast changes get communicated through the network and how stable the network stays 🧘‍♀️

5.5. Knowledge 🧠

some complexity parameters are strongly linked to the human side of networking 🙋‍♀️ for example, the more options and parameters a protocol has, the harder it is to configure and troubleshoot 😵‍💫 so there's a trade-off between the knowledge operational staff have to maintain and the functionality you want 💅 the knowledge network operators need is therefore a big part of complexity considerations 💜

6. Location of Complexity 📍

the previous section looked at the forms complexity can take 👀 this section focuses on WHERE that complexity lives in a network 🗺️ an algorithm can run centrally, distributed, or even in the head of a network administrator 🤯 when you classify a network's complexity, the location of a component can change the overall complexity, so this section offers a method to find WHERE the complex component is located 🔎

6.1. Topological Location 🌐

an algorithm can run distributed, like a routing protocol such as OSPF running on all routers in a network 🛣️ but it can also live in a central location, like a Network Operations Center (NOC) 🏢 the physical location affects other parameters like availability (local changes might be faster than going through a remote NOC) and ease of operation, because it's easier to understand and troubleshoot one central entity than many remote ones 🧐

the example in Section 3.3 shows how the location of state (in this case, configuration) affects the precision of policy enforcement and the state required to do it 📏 enforcement closer to the edge needs more network-wide state, but it's more precise 🎯

6.2. Logical Location 🧩

independent of physical location, the logical location can also make a difference to complexity 🗂️ a controller function, for example, can live in a NOC and also on a network element 💻 organizing a network into separate logical entities is generally considered a positive, because it makes the network easier to understand and therefore easier to troubleshoot and configure 🌟 for example, a BGP route reflector is a separate logical entity from a BGP speaker, but it may sit on the same physical node 🫧

6.3. Layering Considerations 🥞

the layer of the TCP/IP stack where a function is implemented can also affect the complexity of the overall network 📚 some functions get implemented across several layers in slightly different ways, and that can lead to unexpected results 😬

as an example, a link failure gets detected at various layers: L1, L2, the IGP, BGP, and potentially more 🔗 those layers depend on each other, so different link failure detection times can cause undesired effects 😵 dependencies get covered in more detail in the next section 👇

7. Dependencies 🕸️

dependencies are generally regarded as related to overall complexity 🧶 a system with fewer dependencies is generally considered less complex 🥰 this section proposes a way to analyze dependencies in a network 🔬

for example, [Chun] states: "We conjecture that the complexity particular to networked systems arises from the need to ensure state is kept in sync with its distributed dependencies." 🤓

in this document, we distinguish three types of dependencies: local dependencies, network-wide dependencies, and network-external dependencies 💜

7.1. Local Dependencies

okay so local dependencies are all about ONE node 💅 Think of a single router. An interface on it has an IP address, and that address might get used in other bits of the config too. Change the interface address and every dependent config piece has to change with it, or things get messy fr fr 🙈

Same deal with QoS policies, access-control lists, and the names and numbers of config parts. It's the same drama, just in different outfits 👗✨

7.2. Network-Wide Dependencies

Routing protocols, failover protocols, and a bunch of others depend on each other across the whole network 🌐 If one node has a problem, it can ripple out to everyone else, kind of like a group chat where one message sets off a chain reaction 💀

These protocols are usually designed to handle weird unexpected stuff, so they're not likely to cause trouble all by themselves. BUT sometimes a bunch of complexity issues pile up at once (like different timers running on different layers 🫠) and that's when you get really unexpected behavior 😭

7.3. Network-External Dependencies

Some dependencies live OUTSIDE the network entirely 🌍 Like an external NTP clock source, or an Authentication, Authorization, and Accounting (AAA) server. And here's the tradeoff, bestie: take AAA for login authentication. We cut down the config (state) on each node, especially the user-specific stuff, but now every node depends on an AAA server 🫣

In a network with lots of admins, an AAA server is honestly the only manageable way to keep track of everyone 🧚 But the cost is this external dependency, which means admin access can vanish for ALL devices at the same time when the AAA server goes down. Yikes 😱

Even so, centralizing user info (and logging) is still worth it compared to spreading user info across every device 💜 To cover for the central dependency being unavailable, other solutions exist too. For example, a secondary authentication mode with a single root-level password for when the AAA server is not around 🔑

8. Management Interactions

A static network is usually pretty stable 😌 Changes, on the other hand, bring uncertainty, so they need a closer look 🔍 Troubleshooting a network also shows you how complex it really is, kind of intuitively 🎀 This section proposes a way to classify management interactions based on how they relate to network complexity 🦋

8.1. Configuration Complexity

Configuration can be thought of as state spread across network devices, where the administrator has direct say over how the network behaves 👑 Changing the config can make the network behave better overall, or make it worse. In the worst case, one misconfiguration could take down the entire network 💀 So it's super important that a human admin can manage config complexity well 🫶

The configuration reflects most of the local and global dependencies in the network, as explained in Section 7. Tracking those dependencies in the config helps you understand the overall network complexity 🌸

8.2. Troubleshooting Complexity

Unexpected behavior can come from a bunch of places 🙃 The config might have errors, the operating system (its algorithms) might have bugs, or the hardware might be faulty, and that includes everything from broken fibers to faulty line cards 🔌 In serious problems, a combo of causes can show up as one single visible symptom, which is so annoying 😩 Tracking down the root causes can be extremely difficult, and that's basically the network's complexity showing its face 🦋

So finding the source of a problem takes a solid understanding of how complex the network is 🧠 The configuration complexity from the last section is only PART of the overall problem space ✨

8.3. Monitoring Complexity

Even when nothing is erroring, you should keep an eye on the network's state so you can catch error conditions, ideally BEFORE services get affected 👀 For example, a single link-down event might not cause a service disruption in a well-designed network, but you still need to fix it quickly to get redundancy back 🔧

Monitoring has its own complexity, too 🤯 The issues are about scale, the variety of devices to monitor, the variety of methods used to collect info, the inevitable loss of information when reports get aggregated centrally, and the knowledge needed to understand the network, its dependencies, and how it interacts with users and other external inputs 🌈

8.4. Complexity of System Integration

A network isn't just network devices 🙄 It also includes a huge array of backend and support systems. It connects to a large variety of user devices, plus a bunch of human interfaces, both for the users and customers AND for the network admins 🧑‍💻 A system integration job is needed to make sure the whole network delivers the overall service that's expected 💖

All those systems and interactions have to be modeled so you can understand the interdependencies and complexities in the network 🕸️ This is a big area for future research 🔭

9. External Interactions

A network isn't a self-contained little world 🌍 It exists to give connectivity and services to users and to other networks, and a network admin doesn't directly control any of them 🙅‍♀️ The user experience also shows its own kind of interaction, with its own complexity 💅

External interactions fall into these categories:

  • User Interactions: Users need a way to request a service, get their problems resolved, and possibly get billed for their usage 💸 There are a bunch of human interfaces to think about, which depend somewhat on the network, for example, for troubleshooting or for monitoring usage 📊
  • Interactions with End Systems: The network also talks to the devices that plug into it 📱 Usually, a device gets an IP address from the network, plus info on how to resolve domain names, and maybe other services too. Those interactions are pretty simple, BUT there are sooo many types of end devices that keeping track of them is a complicated space 😵‍💫
  • Internetwork Interactions: Most networks connect to other networks 🤝 There are lots of interactions between networks, both technical (like running a routing protocol) AND non-technical (like tracing problems across network boundaries) 🧩

For a fully operational network that serves users, the external interactions and dependencies are an integral part of the overall complexity of the network service 🫧 One big example is the root DNS servers, which are critical to how the Internet works 👑 Practically all Internet users have an implicit dependency on them, which is why they're frequent targets for attacks 🎯 Understanding the overall complexity of a network includes understanding all of these external dependencies 🫶 With the root DNS servers, though, there's honestly not much a network operator can influence 🥲

10. Examples

In the foreseeable future, it's unlikely that anyone will define a single, objective metric that covers ALL the relevant aspects of complexity 📏 Without a global metric like that, a comparative approach could be easier 💜

For example, you can compare a centralized system, where algorithms run centrally and the results get distributed to the network nodes, with a distributed algorithm 🌸 The type of algorithm might be similar, but the location is different, so you end up with a different dependency graph 🕸️ The supporting hardware might be the same, so it could be ignored for this exercise 🔧 Layering is likely to be the same too 🧱 The management interactions, though, would be WILDLY different in the two cases 😤

The classification in this document also makes it easier to survey existing research on which areas of complexity are covered 📚 That could help identify open areas for research 🔭✨

11. Security Considerations

This document does not discuss any specific security considerations. 🚨 Heads up, this one is not a free pass! Just because this document doesn't cover security doesn't mean security isn't a big deal. Complexity and security are tangled up with each other, so keep that in mind when you're designing and running networks 🔐💀

12. Informative References

[Behringer] Behringer, M., "Classifying Network Complexity",
            Proceedings of the 2009 Workshop on Re-architecting the
            Internet (Re-Arch '09), ACM, DOI 10.1145/1658978.1658983,
            December 2009.

[Chun]      Chun, B-G., Ratnasamy, S., and E. Eddie, "NetComplex: A
            Complexity Metric for Networked System Designs",
            Proceedings of the 5th USENIX Symposium on Networked
            Systems Design and Implementation (NSDI '08), pp.
            393-406, April 2008, <http://usenix.org/events/nsdi08/
            tech/full_papers/chun/chun.pdf>.

[Doyle]     Doyle, J., Anderson, D., Li, L., Low, S., Roughnan, M.,
            Shalunov, S., Tanaka, R., and W. Willinger, "The 'robust
            yet fragile' nature of the Internet", Proceedings of the
            National Academy of Sciences of the United States of
            America (PNAS), Volume 102, Number 41,
            DOI 10.1073/pnas.0501426102, October 2005.

[ncrg]      IRTF, "IRTF Network Complexity Research Group (NCRG)
            [CONCLUDED]", <https://irtf.org/concluded/ncrg>.

[RFC1925]   Callon, R., "The Twelve Networking Truths", RFC 1925,
            DOI 10.17487/RFC1925, April 1996,
            <http://www.rfc-editor.org/info/rfc1925>.

[RFC3439]   Bush, R. and D. Meyer, "Some Internet Architectural
            Guidelines and Philosophy", RFC 3439,
            DOI 10.17487/RFC3439, December 2002,
            <http://www.rfc-editor.org/info/rfc3439>.

[wiki]      "Network Complexity - The Wiki",
            <http://networkcomplexity.org/>.

Acknowledgements ✨

The motivations and framework of this overview of studies into network complexity came out of SO many meetings and discussions with way too many people to list here 🫶 Still, some key contributions came from John Doyle, Dave Meyer, Jon Crowcroft, Mark Handley, Fred Baker, Paul Vixie, Lars Eggert, Bob Briscoe, Keith Jones, Bruno Klauser, Stephen Youell, Joel Obstfeld, and Philip Eardley 🌟

The authors would also like to thank Rana Sircar, Ken Carlberg, and Luca Caviglione for their contributions to the preparation of this document 💖

Authors' Addresses

Michael H. Behringer
Cisco Systems
Building D, 45 Allee des Ormes
Mougins  06250
France

Email: mbehring@cisco.com


Alvaro Retana
Cisco Systems
7025 Kit Creek Rd.
Research Triangle Park, NC  27709

United States of America
Email: aretana@cisco.com


Russ White
Ericsson
144 Warm Wood Lane
Apex, NC   27539
United States of America

Email: russ@riw.us
URI:   http://www.ericsson.com


Geoff Huston
Asia Pacific Network Information Centre
6 Cordelia St
South Brisbane, QLD  4101
Australia

Email: gih@apnic.net
URI:   http://www.apnic.net

That's the whole RFC, bestie! 🌈🦋 You made it through every complexity bit, every dependency, and all the authors 💅✨

📦 where this page comes from