RFC 3272

RFC 3272: Overview and Principles of Internet Traffic Engineering 🦋

Part 1 of 20 🎀

okay bestie, RFC 3272 is giving main character energy 💅 It's titled Overview and Principles of Internet Traffic Engineering and was published in May 2002 by the Network Working Group. The authors are D. Awduche (Movaz Networks), A. Chiu (Celion Networks), A. Elwalid (Lucent Technologies), I. Widjaja (Lucent Technologies), and X. Xiao (Redback Networks) 🫶 It's Informational, not a standard ✨ and its title block doesn't list anything it obsoletes or updates 🌸

Status of this Memo ✨

okay so this memo is just giving info to the Internet community 💖 It doesn't specify an Internet standard of any kind, and it can be shared with literally anyone 🦋

Copyright Notice 💜

The Internet Society holds the copyright (2002), and all rights are reserved 🌈

Abstract 🌸

this memo is all about the principles of Traffic Engineering (TE) on the Internet 🧚 It's meant to help people actually get the issues around traffic engineering in IP networks, and to give everyone a shared starting point for building TE capabilities for the Internet 🔥 The principles, architectures, and methods for measuring and optimizing how operational IP networks perform are covered all the way through 👑

Table of Contents 🍓

   1.0 Introduction...................................................3
      1.1 What is Internet Traffic Engineering?.......................4
      1.2 Scope.......................................................7
      1.3 Terminology.................................................8
   2.0 Background....................................................11
      2.1 Context of Internet Traffic Engineering....................12
      2.2 Network Context............................................13
      2.3 Problem Context............................................14
         2.3.1 Congestion and its Ramifications......................16
      2.4 Solution Context...........................................16
         2.4.1 Combating the Congestion Problem......................18
      2.5 Implementation and Operational Context.....................21

   3.0 Traffic Engineering Process Model.............................21
      3.1 Components of the Traffic Engineering Process Model........23
      3.2 Measurement................................................23
      3.3 Modeling, Analysis, and Simulation.........................24
      3.4 Optimization...............................................25
   4.0 Historical Review and Recent Developments.....................26
      4.1 Traffic Engineering in Classical Telephone Networks........26
      4.2 Evolution of Traffic Engineering in the Internet...........28
         4.2.1 Adaptive Routing in ARPANET...........................28
         4.2.2 Dynamic Routing in the Internet.......................29
         4.2.3 ToS Routing...........................................30
         4.2.4 Equal Cost Multi-Path.................................30
         4.2.5 Nimrod................................................31
      4.3 Overlay Model..............................................31
      4.4 Constraint-Based Routing...................................32
      4.5 Overview of Other IETF Projects Related to Traffic
          Engineering................................................32
         4.5.1 Integrated Services...................................32
         4.5.2 RSVP..................................................33
         4.5.3 Differentiated Services...............................34
         4.5.4 MPLS..................................................35
         4.5.5 IP Performance Metrics................................36
         4.5.6 Flow Measurement......................................37
         4.5.7 Endpoint Congestion Management........................37
      4.6 Overview of ITU Activities Related to Traffic
          Engineering................................................38
      4.7 Content Distribution.......................................39
   5.0 Taxonomy of Traffic Engineering Systems.......................40
      5.1 Time-Dependent Versus State-Dependent......................40
      5.2 Offline Versus Online......................................41
      5.3 Centralized Versus Distributed.............................42
      5.4 Local Versus Global........................................42
      5.5 Prescriptive Versus Descriptive............................42
      5.6 Open-Loop Versus Closed-Loop...............................43
      5.7 Tactical vs Strategic......................................43
   6.0 Recommendations for Internet Traffic Engineering..............43
      6.1 Generic Non-functional Recommendations.....................44
      6.2 Routing Recommendations....................................46
      6.3 Traffic Mapping Recommendations............................48
      6.4 Measurement Recommendations................................49
      6.5 Network Survivability......................................50
         6.5.1 Survivability in MPLS Based Networks..................52
         6.5.2 Protection Option.....................................53
      6.6 Traffic Engineering in Diffserv Environments...............54
      6.7 Network Controllability....................................56
   7.0 Inter-Domain Considerations...................................57
   8.0 Overview of Contemporary TE Practices in Operational
       IP Networks...................................................59

   9.0 Conclusion....................................................63
   10.0 Security Considerations......................................63
   11.0 Acknowledgments..............................................63
   12.0 References...................................................64
   13.0 Authors' Addresses...........................................70
   14.0 Full Copyright Statement.....................................71

1.0 Introduction ✨

bestie, this memo is all about the principles of Internet traffic engineering 🎀 The goal is to lay out the big-picture issues and principles, and where it makes sense, to offer recommendations, guidelines, and options for building online and offline Internet traffic engineering capabilities and the support systems around them 💅

Service providers can use this document to plan and carry out traffic engineering solutions for their networks 🌸 Networking hardware and software vendors will also find it super helpful when they're building the mechanisms and support systems that let the Internet do traffic engineering 🧚

This document gives you a terminology for describing common Internet traffic engineering concepts, so everyone's speaking the same language 🫶 It also offers a taxonomy of known traffic engineering styles ✨ A traffic engineering style pulls out the important parts of a traffic engineering methodology 🦋 Styles can be looked at in different ways depending on the context and what they're for, and combining the styles and views gives a natural taxonomy of traffic engineering systems 👑

Even though Internet traffic engineering works best end-to-end 💖, the first focus here is intra-domain traffic engineering, meaning traffic engineering inside a single autonomous system 🔥 But since a huge chunk of Internet traffic is inter-domain (starting in one autonomous system and ending in another) 😭, the document also gives an overview of inter-domain traffic engineering 🌈

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 💜 These capital words are the official rule-decoder ring, so read them the way RFC 2119 says 🙈

1.1. What is Internet Traffic Engineering? ✨

okay bestie, let's talk about what Internet traffic engineering actually is 🧚 It's the part of Internet network engineering that deals with evaluating performance and optimizing performance of operational IP networks. Traffic engineering wraps up the use of technology and scientific principles for measuring, characterizing, modeling, and controlling Internet traffic [RFC-2702, AWD2] 💜

Making an operational network perform better, at both the traffic and resource levels, is the big goal here 🎀 You get there by handling traffic-oriented performance requirements while using network resources economically and reliably. The traffic-oriented performance measures include delay, delay variation, packet loss, and throughput 📊

An important goal of Internet traffic engineering is to make network operations reliable 🫶 Reliable operations can be supported by mechanisms that strengthen network integrity and by policies that put emphasis on network survivability. The result is that the network becomes less vulnerable to service outages caused by errors, faults, and failures happening inside the infrastructure 🛡️ [RFC-2702]

Here's the core vibe: the Internet exists to move information from source nodes to destination nodes 🦋 So one of the most significant things the Internet does is route traffic from ingress nodes to egress nodes. That makes one of the most distinctive jobs of Internet traffic engineering the control and optimization of routing, so traffic gets steered through the network in the most effective way possible 🔥

Ultimately, the performance of the network as seen by end users of network services is what truly matters most 💖 This crucial point should be considered throughout the development of traffic engineering mechanisms and policies. The characteristics that end users can see are the emergent properties of the network, meaning the characteristics of the network when you view it as a whole 🌈 So a central goal of the service provider is to enhance those emergent properties while taking economic considerations into account 💅

Here's why that matters so much: special care must be taken when choosing which network performance measures to optimize 😭 Optimizing the wrong measures might hit certain local objectives, but it can have disastrous consequences on the emergent properties of the network, and therefore on the quality of service that end-users perceive. Not me watching someone optimize the wrong thing and tanking the whole vibe 💀

A subtle but practical perk of applying traffic engineering concepts systematically to operational networks is that it helps identify and structure goals and priorities around improving the quality of service delivered to end users 🌸 Those concepts also help with measuring and analyzing whether those goals are being achieved ✨

The optimization aspects of traffic engineering can be achieved through capacity management and traffic management 🍓 In this document, capacity management includes capacity planning, routing control, and resource management. The network resources of particular interest include link bandwidth, buffer space, and computational resources. Also in this document, traffic management includes (1) nodal traffic control functions such as traffic conditioning, queue management, and scheduling, and (2) other functions that regulate traffic flow through the network or that arbitrate access to network resources between different packets or between different traffic streams 👑

The optimization objectives of Internet traffic engineering should be viewed as a continual and iterative process of network performance improvement, not just a one-time goal 🔁 Traffic engineering also demands ongoing development of new technologies and new methodologies for enhancing network performance. It's giving "never stop improving" energy 🌟

The optimization objectives of Internet traffic engineering may change over time as new requirements are imposed, as new technologies emerge, or as new insights are brought to bear on the underlying problems 🦄 Different networks may also have different optimization objectives, depending on their business models, capabilities, and operating constraints. Regardless of the specific optimization goals in any particular environment, the optimization aspects of traffic engineering are ultimately concerned with network control 🎀

So the optimization aspects of traffic engineering can be viewed from a control perspective 🧠 The control aspect of Internet traffic engineering can be pro-active and/or reactive. In the pro-active case, the traffic engineering control system takes preventive action to obviate predicted unfavorable future network states. It may also take perfective action to induce a more desirable state in the future 🔮 In the reactive case, the control system responds correctively and perhaps adaptively to events that have already transpired in the network 🛠️

The control dimension of Internet traffic engineering responds to network events at multiple levels of temporal resolution ⏱️ Certain aspects of capacity management, such as capacity planning, respond at very coarse temporal levels, ranging from days to possibly years. The introduction of automatically switched optical transport networks (e.g., based on the Multi-protocol Lambda Switching concepts) could significantly reduce the lifecycle for capacity planning by expediting provisioning of optical bandwidth ⚡ Routing control functions operate at intermediate levels of temporal resolution, ranging from milliseconds to days. Finally, the packet level processing functions (e.g., rate shaping, queue management, and scheduling) operate at very fine levels of temporal resolution, ranging from picoseconds to milliseconds, while responding to the real-time statistical behavior of traffic 🏎️

The subsystems of Internet traffic engineering control include capacity augmentation, routing control, traffic control, and resource control (including control of service policies at network elements) 🧩 When capacity is to be augmented for tactical purposes, it may be desirable to devise a deployment plan that expedites bandwidth provisioning while minimizing installation costs 💸

Inputs into the traffic engineering control system include network state variables, policy variables, and decision variables 📥

One major challenge of Internet traffic engineering is realizing automated control capabilities that adapt quickly and cost effectively to significant changes in a network's state, while still maintaining stability 🧗‍♀️ That's the hard part, babe!

Another critical dimension of Internet traffic engineering is network performance evaluation 🔍 It's important for assessing the effectiveness of traffic engineering methods, and for monitoring and verifying compliance with network performance goals. Results from performance evaluation can be used to identify existing problems, guide network re-optimization, and aid in predicting potential future problems 🎯

Performance evaluation can be achieved in many different ways 🌈 The most notable techniques include analytical methods, simulation, and empirical methods based on measurements. When analytical methods or simulation are used, network nodes and links can be modeled to capture relevant operational features such as topology, bandwidth, buffer space, and nodal service policies (link scheduling, packet prioritization, buffer management, etc.). Analytical traffic models can be used to depict dynamic and behavioral traffic characteristics, such as burstiness, statistical distributions, and dependence 📈

Performance evaluation can get quite complicated in practical network contexts 😵‍💫 A number of techniques can be used to simplify the analysis, such as abstraction, decomposition, and approximation. For example, simplifying concepts such as effective bandwidth and effective buffer [Elwalid] may be used to approximate nodal behaviors at the packet level and simplify the analysis at the connection level. Network analysis techniques using, for example, queuing models and approximation schemes based on asymptotic and decomposition techniques can make the analysis even more tractable 🧮 In particular, an emerging set of concepts known as network calculus [CRUZ], based on deterministic bounds, may simplify network analysis relative to classical stochastic techniques ✨ When using analytical techniques, care should be taken to ensure that the models faithfully reflect the relevant operational characteristics of the modeled network entities 🫡

Simulation can be used to evaluate network performance or to verify and validate analytical approximations. Simulation can, however, be computationally costly and may not always provide sufficient insights 🥲 An appropriate approach to a given network performance evaluation problem may involve a hybrid combination of analytical techniques, simulation, and empirical methods 🍰

As a general rule, traffic engineering concepts and mechanisms must be sufficiently specific and well defined to address known requirements, but simultaneously flexible and extensible to accommodate unforeseen future demands 🌷 Balance is everything, fr fr 💅

---

1.2. Scope 🎀

The scope of this document is intra-domain traffic engineering, meaning traffic engineering within a given autonomous system in the Internet 🏠 This document will discuss concepts pertaining to intra-domain traffic control, including issues such as routing control, micro and macro resource allocation, and the control coordination problems that arise as a consequence 🧩

This document will describe and characterize techniques already in use or in advanced development for Internet traffic engineering 📚 The way these techniques fit together will be discussed, and scenarios in which they are useful will be identified 🗺️

While this document considers various intra-domain traffic engineering approaches, it focuses more on traffic engineering with MPLS 💜 Traffic engineering based upon manipulation of IGP metrics is not addressed in detail. That topic may be addressed by other working group document(s) 👀

Although the emphasis is on intra-domain traffic engineering, Section 7.0 provides an overview of the high level considerations pertaining to inter-domain traffic engineering 🌍 Inter-domain Internet traffic engineering is crucial to the performance enhancement of the global Internet infrastructure 🚀

Whenever possible, relevant requirements from existing IETF documents and other sources will be incorporated by reference 🔗

1.3 Terminology 📖

okay bestie, this is the glossary 🧚 these are the terms that matter for Internet traffic engineering. the definitions here apply to this document, but these words can mean other things elsewhere, so don't panic if you see them somewhere else and they hit different 💅

• Baseline analysis: a study done as a baseline to compare against how the network actually behaves 📊
• Busy hour: a one hour window (usually inside a 24 hour interval) where the traffic load on a network or sub-network is at its highest 🔥
• Bottleneck: a network element whose input traffic rate tends to be bigger than its output rate. it's the spot where things pile up 🚧
• Congestion: a state where the traffic hitting a resource is more than its output capacity over some period of time. so much traffic, not enough room 😭
• Congestion avoidance: a congestion management approach that tries to stop congestion from happening in the first place 🛡️
• Congestion control: a congestion management approach that tries to fix congestion problems that have already happened 🩹
• Constraint-based routing: a class of routing protocols that pick routes by looking at specified traffic attributes, network constraints, and policy constraints. it works for traffic aggregates as well as flows, and it's a generalization of QoS routing. literally the main character of routing 👑
• Demand side congestion management: a congestion management scheme that deals with congestion by regulating or conditioning the offered load 🎀
• Effective bandwidth: the minimum bandwidth you can assign to a flow or traffic aggregate and still deliver 'acceptable service quality' to it ✨
• Egress traffic: traffic leaving a network or network element 🚪
• Hot-spot: a network element or subsystem that is currently in a state of congestion 🌶️
• Ingress traffic: traffic coming into a network or network element 🫶
• Inter-domain traffic: traffic that starts in one Autonomous system and ends up in another 🌉
• Loss network: a network without enough buffering, so traffic that runs into a busy resource gets dropped instead of queued. no waiting in line, just gone 🦋
• Metric: a parameter defined in terms of standard units of measurement 📏
• Measurement Methodology: a repeatable measurement technique used to derive one or more metrics of interest 🔬
• Network Survivability: the ability to keep a prescribed level of QoS for existing services after a given number of failures happen in the network. the network refuses to fold 💪
• Offline traffic engineering: a traffic engineering system that sits outside of the network 🌙
• Online traffic engineering: a traffic engineering system that lives inside the network, usually running on or as add-ons to operational network elements ⚡
• Performance measures: metrics that give quantitative or qualitative measures of how well systems or subsystems of interest are performing 📈
• Performance management: a systematic approach to getting better at reaching specific networking goals tied to performance improvement 🌟
• Performance Metric: a performance parameter defined in terms of standard units of measurement 💜
• Provisioning: the process of assigning or configuring network resources to meet certain requests 🍓
• QoS routing: a class of routing systems that choose the paths a flow uses based on that flow's QoS requirements 🎯
• Service Level Agreement: a contract between a provider and a customer that guarantees specific levels of performance and reliability at a certain cost. the pinky promise with receipts 📝
• Stability: an operational state where the network doesn't oscillate in a disruptive way from one mode to another mode 🧘
• Supply side congestion management: a congestion management scheme that provisions extra network resources to handle existing and/or anticipated congestion problems 🏗️
• Transit traffic: traffic whose origin and destination are both outside of the network being considered. just passing through 🧳
• Traffic characteristic: a description of the temporal behavior of a traffic flow or traffic aggregate, or a description of its attributes 🕰️
• Traffic engineering system: a collection of objects, mechanisms, and protocols that are used together to accomplish traffic engineering objectives. it's a team effort 🤝
• Traffic flow: a stream of packets between two end-points that can be characterized in a certain way. a micro-flow is more specific: it's a stream of packets with the same source and destination addresses, the same source and destination ports, and the same protocol ID. basically identical twins 👯
• Traffic intensity: a measure of traffic loading against a resource capacity over a specified period of time. in classical telephony systems, it's measured in units of Erlang 📞
• Traffic matrix: a representation of the traffic demand between a set of origin and destination abstract nodes. an abstract node can be made up of one or more network elements 🗺️
• Traffic monitoring: the process of watching traffic characteristics at a given point in a network and collecting the traffic information so it can be analyzed and acted on 👀
• Traffic trunk: an aggregation of traffic flows that belong to the same class and get forwarded through a common path. it can be characterized by an ingress and egress node, plus a set of attributes that decide its behavior and what it needs from the network 🚂

---

2.0 Background 🌐

okay so the Internet has glowed up SO hard 💖 it's now a super critical communications infrastructure that supports huge economic, educational, and social activities. meanwhile, delivering Internet communication services has gotten super competitive, and end-users are demanding very high quality service from their providers 😤 because of that, performance optimization of large scale IP networks, especially public Internet backbones, has become a seriously important problem. network performance requirements are multi-dimensional, complex, and sometimes contradictory, which makes traffic engineering very challenging 🧩

the network has to convey IP packets from ingress nodes to egress nodes efficiently, expeditiously, and economically 🚀 and in a multiclass service environment (for example, Diffserv capable networks), the resource sharing parameters of the network must be appropriately determined and configured according to prevailing policies and service models. this is how you resolve resource contention issues caused by mutual interference between packets moving through the network 🎀 so consideration must be given to resolving competition for network resources between traffic streams in the same service class (intra-class contention resolution) and traffic streams in different classes (inter-class contention resolution). it's giving sharing drama, but make it infrastructure 💅

2.1 Context of Internet Traffic Engineering 🧭

the context of Internet traffic engineering is all about the scenarios where traffic engineering gets used 🌸 a traffic engineering methodology sets up appropriate rules to resolve traffic performance issues that show up in a specific context. the context of Internet traffic engineering includes:

1. A network context 🕸️ this defines the universe of discourse, and in particular the situations where traffic engineering problems happen. it covers network structure, network policies, network characteristics, network constraints, network quality attributes, and network optimization criteria.
2. A problem context 🔍 this defines the general and concrete issues traffic engineering addresses. it includes identification, abstraction of relevant features, representation, formulation, specification of the requirements on the solution space, and specification of the desirable features of acceptable solutions.
3. A solution context 💡 this suggests how to address the issues identified by the problem context. it includes analysis, evaluation of alternatives, prescription, and resolution.
4. An implementation and operational context 🛠️ this is where the solutions get methodologically instantiated. it includes planning, organization, and execution.

the context of Internet traffic engineering and the different problem scenarios are discussed in the following subsections ✨

2.2 Network Context

okay bestie, IP networks come in every size 🦋. Some are just a tiny cluster of routers sitting in one location, and others are thousands of interconnected routers, switches, and other components spread all over the world 🌍✨

So at the most basic level of abstraction, you can picture an IP network as a distributed dynamical system 🧚 with three parts: (1) a set of interconnected resources that provide transport services for IP traffic, subject to certain constraints, (2) a demand system, which is the offered load that needs to be carried through the network, and (3) a response system made of network processes, protocols, and related mechanisms that help traffic actually move through 💅 [see also AWD2].

The network elements and resources can have specific characteristics that limit how demand gets handled 🙈. On top of that, network resources can be equipped with traffic control mechanisms that oversee how demand is served 👑. These mechanisms can, for example, control various packet processing activities inside a given resource, referee the fights when different packets contend for access to that resource, and regulate traffic behavior through it. A configuration management and provisioning system can let internal or external entities change the settings of those traffic control mechanisms, so they can steer how network elements respond to internal and external stimuli 🎀.

The details of how the network provides transport services for packets are written in the policies of the network administrators, and they get installed through network configuration management and policy-based provisioning systems 📋. The types of services a network can offer also depend on the technology and characteristics of its network elements and protocols, the prevailing service and utility models, and whether the network administrators can actually turn policies into network configurations. So it's a whole team effort 🫶.

Contemporary Internet networks have three big characteristics 🔥: (1) they provide real-time services, (2) they've become mission critical, and (3) their operating environments are super dynamic 😭. The dynamic nature of IP networks can be blamed partly on fluctuations in demand, on the interaction between various network protocols and processes, on the rapid evolution of the infrastructure (which constantly needs new technologies and new network elements), and on transient and persistent impairments that happen inside the system 🌈.

Packets contend for network resources as they get conveyed through the network 🏃‍♀️. A network resource is considered congested if the arrival rate of packets exceeds the output capacity of the resource over an interval of time. When congestion happens, some of the arriving packets may get delayed or even dropped 💀.

Congestion increases transit delays, delay variation, and packet loss, and it reduces how predictable network services are. Clearly, congestion is a highly undesirable phenomenon 🙅‍♀️.

Combating congestion at a reasonable cost is a major objective of Internet traffic engineering 🎯.

Efficient sharing of network resources by multiple traffic streams is a basic economic premise for packet switched networks in general, and for the Internet in particular 💖. A fundamental challenge in network operation, especially in a large scale public IP network, is to increase the efficiency of resource utilization while minimizing the possibility of congestion ✨.

The Internet is increasingly going to have to function with different classes of traffic that have different service requirements 🌸. The arrival of Differentiated Services [RFC-2475] makes this requirement especially intense. So packets may be grouped into behavior aggregates, where each behavior aggregate can have a common set of behavioral characteristics or a common set of delivery requirements 🧁. In practice, the delivery requirements of a specific set of packets may be specified explicitly or implicitly. The two most important traffic delivery requirements are capacity constraints and QoS constraints 👀.

Capacity constraints can be expressed statistically as peak rates, mean rates, or burst sizes, or as some deterministic notion of effective bandwidth 📊. QoS requirements can be expressed in terms of (1) integrity constraints, such as packet loss, and (2) temporal constraints, such as timing restrictions for the delivery of each packet (delay) and timing restrictions for the delivery of consecutive packets belonging to the same traffic stream (delay variation) ⏱️.

2.3 Problem Context

Okay so there are some real fundamental problems when you try to operate a network described by the simple model from the previous subsection 🧐. This subsection walks through the problem context as it relates to the traffic engineering function 🎀.

The identification, abstraction, representation, and measurement of network features relevant to traffic engineering is a significant issue 📏.

One especially important class of problems is about how to explicitly formulate the problems that traffic engineering is trying to solve 🧩. It's also about how to identify the requirements on the solution space, how to specify the desirable features of good solutions, how to actually solve the problems, and how to measure and characterize how effective the solutions are 🏆.

Another class of problems is about how to measure and estimate relevant network state parameters 📡. Effective traffic engineering relies on a good estimate of the offered traffic load, plus a view of the underlying topology and its associated resource constraints 🗺️. A network-wide view of the topology is also a must for offline planning 💜.

Still another class of problems is about how to characterize the state of the network and how to evaluate its performance under a variety of scenarios 🔍. The performance evaluation problem is two-fold 👯‍♀️. One aspect relates to evaluating the system level performance of the network. The other aspect relates to evaluating the resource level performance, which only looks at the performance analysis of individual network resources. In this memo, we refer to the system level characteristics of the network as the "macro-states" and the resource level characteristics as the "micro-states" 🌟. The system level characteristics are also known as the emergent properties of the network, as noted earlier. So we'll call the traffic engineering schemes that deal with network performance optimization at the systems level "macro-TE," and the schemes that optimize at the individual resource level "micro-TE" 🫧. Under certain circumstances, system level performance can be derived from resource level performance using appropriate rules of composition, depending on the particular performance measures of interest 🧮.

Another fundamental class of problems is about how to effectively optimize network performance 🚀. Performance optimization may involve translating solutions to specific traffic engineering problems into network configurations. It may also involve some degree of resource management control, routing control, and/or capacity augmentation 🛠️.

As noted previously, congestion is an undesirable phenomenon in operational networks 😤. So the next subsection addresses the issue of congestion and its ramifications within the problem context of Internet traffic engineering 💅.

2.3.1 Congestion and its Ramifications

Congestion is one of the most significant problems in an operational IP context 🚨. A network element is said to be congested if it experiences sustained overload over an interval of time. And congestion almost always makes service quality worse for end users 😩.

Congestion control schemes can include demand side policies and supply side policies 🧃. Demand side policies may restrict access to congested resources, and/or dynamically regulate the demand to ease the overload situation 🚦. Supply side policies may expand or augment network capacity to better accommodate offered traffic 🏗️. They may also re-allocate network resources by redistributing traffic over the infrastructure 🔀. Traffic redistribution and resource re-allocation serve to increase the 'effective capacity' seen by the demand ✨👑.

The emphasis of this memo is primarily on congestion management schemes that fall within the scope of the network, rather than on congestion management systems that depend on sensitivity and adaptivity from end-systems 💻. In other words, the congestion management aspects this memo looks at are the solutions that can be provided by control entities operating on the network, and by the actions of network administrators and network operations systems 🧑‍💻🎀.

2.4. Solution Context ✨

Okay so the solution context for Internet traffic engineering is basically the thinking part 🧠💅 It's all about analysis, weighing up alternatives, and then choosing which course of action to take. The vibe is making reasonable guesses about what the network is doing now or will be doing later, and then making good decisions based on those guesses. Sometimes that means picking between different sets of actions. 🎀

More specifically, the solution context needs reasonable estimates of traffic workload, a characterization of network state, and solutions to traffic engineering problems. Those problems might be formulated implicitly or explicitly. It might also mean actually putting a set of control actions into effect. 🦋 Control actions can involve changing parameters tied to routing, controlling tactical capacity acquisition, and controlling the traffic management functions. Main character energy, honestly 👑

The following list of instruments may be applicable to the solution context of Internet traffic engineering. 🌸

1. A set of policies, objectives, and requirements (which may be context dependent) for network performance evaluation and performance optimization. 📋
2. A collection of online and possibly offline tools and mechanisms for measurement, characterization, modeling, and control of Internet traffic, plus control over the placement and allocation of network resources, and control over the mapping or distribution of traffic onto the infrastructure. 🧰
3. A set of constraints on the operating environment, the network protocols, and the traffic engineering system itself. 🚧
4. A set of quantitative and qualitative techniques and methodologies for abstracting, formulating, and solving traffic engineering problems. 🧮
5. A set of administrative control parameters which may be manipulated through a Configuration Management (CM) system. The CM system itself may include a configuration control subsystem, a configuration repository, a configuration accounting subsystem, and a configuration auditing subsystem. 🗂️
6. A set of guidelines for network performance evaluation, performance optimization, and performance improvement. ✨

Deriving traffic characteristics through measurement and/or estimation is very useful in the solution space for traffic engineering, ngl 🔥 Traffic estimates can come from customer subscription information, traffic projections, traffic models, and actual empirical measurements. The empirical measurements can be taken at the traffic aggregate level or at the flow level, so you get traffic statistics at different levels of detail. 📊 Measurements at the flow level, or on small traffic aggregates, may be performed at edge nodes, which is where traffic enters and leaves the network. Measurements at large traffic aggregate levels may be performed within the core of the network, where potentially numerous traffic flows may be in transit at the same time. 🌈

To conduct performance studies and to support planning of existing and future networks, a routing analysis may be performed. It figures out the path(s) the routing protocols will choose for various traffic demands, and it checks how much network resources get used as traffic is routed through the network. 🛣️ The routing analysis should capture the selection of paths through the network, the assignment of traffic across multiple feasible routes, and the multiplexing of IP traffic over traffic trunks (if such constructs exists) and over the underlying network infrastructure. 🧩 A network topology model is a necessity for routing analysis, bestie 🫶 A network topology model may be extracted from network architecture documents, from network designs, from information contained in router configuration files, from routing databases, from routing tables, or from automated tools that discover and depict network topology information. Topology information may also come from servers that monitor network state, and from servers that perform provisioning functions. 🧚

Routing in operational IP networks can be administratively controlled at various levels of abstraction, including manipulating BGP attributes and manipulating IGP metrics. 🎛️ For path oriented technologies such as MPLS, routing can be controlled even further by manipulating the relevant traffic engineering parameters, resource parameters, and administrative policy constraints. Within the context of MPLS, the path of an explicit label switched path (LSP) can be computed and established in various ways, including:

1. manually, 🙈
2. automatically online, using constraint-based routing processes implemented on label switching routers, and 💜
3. automatically offline, using constraint-based routing entities implemented on external traffic engineering support systems. 🌸

2.4.1 Combating the Congestion Problem ✨

okay so minimizing congestion is literally a huge part of Internet traffic engineering 🚦 This subsection gives an overview of the general approaches that have been used or proposed to fight congestion problems 💅

Congestion management policies can be sorted using these criteria (see e.g., [YARE95] for a more detailed taxonomy of congestion control schemes) 📚: (1) response time scale, which can be long, medium, or short; (2) reactive versus preventive, which relates to congestion control and congestion avoidance; and (3) supply side versus demand side congestion management schemes. We'll go through each one in the next paragraphs 🫶

(1) Congestion Management based on Response Time Scales ⏰

• Long (weeks to months): Capacity planning works on a pretty long time scale 🐢 It expands network capacity based on estimates or forecasts of future traffic demand and traffic distribution. Router and link provisioning takes time and is usually expensive, so these upgrades are typically done on the weeks-to-months or even years time scale.
• Medium (minutes to days): Several control policies live in this medium time scale category 🎯 Examples include: (1) adjusting IGP and/or BGP parameters to route traffic away from or towards certain segments of the network; (2) setting up and/or adjusting some explicitly routed label switched paths (ER-LSPs) in MPLS networks to route some traffic trunks away from possibly congested resources or towards possibly more favorable routes; (3) re-configuring the logical topology of the network so it correlates more closely with the spatial traffic distribution, using for example some underlying path-oriented technology such as MPLS LSPs, ATM PVCs, or optical channel trails. Many of these adaptive medium time scale response schemes rely on a measurement system that monitors changes in traffic distribution, traffic shifts, and network resource utilization 📊 It then provides feedback to the online and/or offline traffic engineering mechanisms and tools, which use this feedback to trigger certain control actions within the network. These traffic engineering mechanisms and tools can be implemented in a distributed fashion or a centralized fashion, and may have a hierarchical structure or a flat structure. The comparative merits of distributed and centralized control structures for networks are well known 🧠 A centralized scheme may have global visibility into the network state and may produce potentially more optimal solutions. However, centralized schemes are prone to single points of failure and may not scale as well as distributed schemes. Moreover, the information used by a centralized scheme may be stale and may not reflect the actual state of the network 😬 It is not an objective of this memo to recommend between distributed and centralized schemes. That's a choice network administrators must make based on their specific needs 🫡
• Short (picoseconds to minutes): This category covers packet level processing functions and events on the order of several round trip times 🏎️ It includes router mechanisms such as passive and active buffer management. These mechanisms control congestion and/or signal congestion to end systems, so they can adaptively regulate the rate at which traffic is injected into the network. One of the most popular active queue management schemes, especially for TCP traffic, is Random Early Detection (RED) [FLJA93], which supports congestion avoidance by controlling the average queue size 🌊 During congestion (but before the queue is full), the RED scheme chooses arriving packets to "mark" according to a probabilistic algorithm that takes the average queue size into account. For a router that does not utilize explicit congestion notification (ECN) see e.g., [FLOY94], the marked packets can simply be dropped to signal the start of congestion to end systems. On the other hand, if the router supports ECN, it can set the ECN field in the packet header 🎀 Several variations of RED have been proposed to support different drop precedence levels in multi-class environments [RFC-2597], e.g., RED with In and Out (RIO) and Weighted RED. There is general consensus that RED provides congestion avoidance performance that is not worse than traditional Tail-Drop (TD) queue management (drop arriving packets only when the queue is full). Importantly, though, RED reduces the possibility of global synchronization and improves fairness among different TCP sessions 🤝 However, RED by itself cannot prevent congestion and unfairness caused by sources that are unresponsive to RED, e.g., UDP traffic and some misbehaved greedy connections 🙄 Other schemes have been proposed to improve performance and fairness in the presence of unresponsive traffic. Some of these were proposed as theoretical frameworks and are typically not available in existing commercial products 🧪 Two such schemes are Longest Queue Drop (LQD) and Dynamic Soft Partitioning with Random Drop (RND) [SLDC98].

(2) Congestion Management: Reactive versus Preventive Schemes 🔄

• Reactive: Reactive (recovery) congestion management policies react to existing congestion problems to improve them. All the policies described in the long and medium time scales above can be categorized as reactive, especially if the policies are based on monitoring and identifying existing congestion problems, and on initiating relevant actions to ease the situation 🩹
• Preventive: Preventive (predictive/avoidance) policies take proactive action to prevent congestion based on estimates and predictions of future potential congestion problems 🔮 Some of the policies described in the long and medium time scales fall into this category. They do not necessarily respond immediately to existing congestion problems. Instead, forecasts of traffic demand and workload distribution are considered, and action may be taken to prevent potential congestion problems in the future. The schemes described in the short time scale (e.g., RED and its variations, ECN, LQD, and RND) are also used for congestion avoidance, since dropping or marking packets before queues actually overflow would trigger corresponding TCP sources to slow down 🐌

(3) Congestion Management: Supply Side versus Demand Side Schemes ⚖️

• Supply side: Supply side congestion management policies increase the effective capacity available to traffic in order to control or obviate congestion 📈 This can be done by augmenting capacity. Another way is to minimize congestion by having a relatively balanced distribution of traffic over the network. For example, capacity planning should aim to provide a physical topology and associated link bandwidths that match the estimated traffic workload and traffic distribution based on forecasting (subject to budgetary and other constraints) 💰 However, if actual traffic distribution does not match the topology derived from capacity planning (due to forecasting errors or facility constraints, for example), then the traffic can be mapped onto the existing topology using routing control mechanisms, using path oriented technologies (e.g., MPLS LSPs and optical channel trails) to modify the logical topology, or by using some other load redistribution mechanisms 🔀
• Demand side: Demand side congestion management policies control or regulate the offered traffic to alleviate congestion problems 🚪 For example, some of the short time scale mechanisms described earlier (such as RED and its variations, ECN, LQD, and RND), as well as policing and rate shaping mechanisms, attempt to regulate the offered load in various ways. Tariffs may also be applied as a demand side instrument. To date, however, tariffs have not been used as a means of demand side congestion management within the Internet 🙃

In summary, a variety of mechanisms can be used to address congestion problems in IP networks 💖 These mechanisms may operate at multiple time-scales.

---

2.5 Implementation and Operational Context 🛠️

The operational context of Internet traffic engineering is characterized by constant change, which occurs at multiple levels of abstraction 🌪️ The implementation context demands effective planning, organization, and execution. Planning may involve determining the sets of actions to take ahead of time to achieve desired objectives 🗓️ Organizing involves arranging and assigning responsibility to the various components of the traffic engineering system, and coordinating their activities to accomplish the desired TE objectives 🧩 Execution involves measuring and applying corrective or perfective actions to attain and maintain the desired TE goals 🏁

3.0 Traffic Engineering Process Model(s) 🦋

okay bestie, this section is literally the blueprint for how traffic engineering actually happens in real life 💅 it's a generic process model, meaning it's the broad stuff most TE methods share, even though every network does the details its own way 🧚 and it can be run by a robot, a human, or both, no cap ✨

The TE process model is iterative, so the four phases just keep going on repeat forever 🔁💖 Here they are, in order, slay 👑

The first phase is defining the control policies that run the network 📋 These can depend on a bunch of stuff: the business model, what the network costs, the operating constraints, the utility model, and the optimization criteria. it's giving "set the rules before you play the game" 🎀

The second phase is the feedback loop, where you collect measurement data from the live network 📊 If the real data isn't easy to get, you can use synthetic workloads instead, which copy either the current workload or the expected one 🌸 You can make those by estimating or extrapolating from old real data, or by using mathematical models of traffic, or other methods 🫶

The third phase is analyzing the network state and characterizing the traffic workload 🔍 Performance analysis can be proactive and/or reactive, and the difference is so fun 😭 Proactive analysis spots potential problems that don't exist yet but could show up later 🔮 Reactive analysis finds problems that already exist, figures out what caused them through diagnosis, and checks out different ways to fix them if needed 🛠️ You can use all kinds of quantitative and qualitative techniques here, including modeling-based analysis and simulation 💜 This phase can look at how traffic is concentrated and spread across the whole network or parts of it, figure out what the offered traffic workload looks like, find bottlenecks, and call out network pathologies like ineffective link placement or single points of failure 🙈 Those pathologies can come from bad architecture, bad design, or config problems 💀 You might also build a traffic matrix during analysis 🧩 And analysis can be descriptive or prescriptive, either way is valid 🌈

The fourth phase is performance optimization, which is where the magic happens ✨ You pick and carry out a set of actions from a bunch of alternatives 🎯 Those actions can control the offered traffic or control how traffic is distributed across the network 🦋 They can also mean adding links or boosting link capacity, deploying more hardware like routers and switches, systematically tweaking routing stuff like IGP metrics and BGP attributes, and adjusting traffic management parameters 💅 Optimization can also kick off a network planning process to improve the architecture, design, capacity, technology, and element configs so the network can handle what's happening now and the future growth 🌱 main character energy 👑

---

3.1. Components of the Traffic Engineering Process Model 🧩

okay so the process model has three main components 🫶 There's a measurement subsystem, a modeling and analysis subsystem, and an optimization subsystem 💖 The next subsections break down how each one fits into the TE process model ✨

3.2. Measurement 📊

measurement is literally crucial to TE, no cap 😭 You can only really know the operational state of a network by measuring it 🔍 It's also key for optimization because it gives the feedback data that TE control subsystems use 🔁 That data lets the network adapt its performance in response to things happening inside and outside the network 🌍 Measurement also tells you the quality of network services and whether your TE policies are actually working 💅 Experience says measurement works best when it's done systematically 🎀

When you're building a measurement system for TE in IP networks, there's a whole list of questions to think through carefully 💭 Why do you need measurement in this specific context? What parameters are you measuring? How should it get done? Where and when should it happen? How often should the monitored variables get checked? What accuracy and reliability level is desirable, and what's actually realistic to achieve? How much can the measurement system interfere with the components and variables it's watching? And what's an acceptable cost? 💸 The answers decide which tools and methods make sense for any given TE situation 🧚

Also, measurement and evaluation are NOT the same thing, bestie 🙅‍♀️ Measurement gives you the raw data about the state parameters and variables of the network elements being watched 📈 Evaluation takes that raw data and makes inferences about the system 🧠

Measurement for TE can happen at different levels of abstraction 🌈 You can derive packet-level characteristics, flow-level characteristics, user or customer-level characteristics, traffic aggregate characteristics, component-level characteristics, and network-wide characteristics 🦋 It's giving "zoom in, zoom out" energy 🔭

3.3. Modeling, Analysis, and Simulation 🧪

modeling and analysis are a huge deal in Internet TE 💖 Modeling means building an abstract or physical representation that shows the relevant traffic characteristics and network attributes 🎀

A network model is an abstract representation of the network that captures the important features, attributes, and characteristics, like link and nodal attributes and constraints 🧩 It can make analysis and/or simulation easier, and those can predict how the network will perform under different conditions and help guide expansion plans 🗺️

Generally, Internet TE models can be classified as either structural or behavioral 🫧 Structural models focus on how the network and its components are organized 🏗️ Behavioral models focus on the dynamics of the network and the traffic workload 🌊 Modeling for Internet TE can also be formal or informal ✨

Accurate behavioral models for traffic sources are especially useful for analysis 🔬 Building behavioral traffic source models that match empirical data from operational networks is a major research topic in Internet TE 📚 These source models should also be tractable and amenable to analysis 💜 The topic of source models for IP traffic is a research topic, so it's outside the scope of this document 🙈 But its importance absolutely needs to be emphasized 📣

Network simulation tools are extremely useful for TE 🔥 Realistic quantitative analysis of network behavior is super complicated, so some aspects of network performance studies can only be done effectively with simulation 🧠 A good network simulator can mimic and visualize network characteristics under various conditions in a safe, non-disruptive way 👀 For example, it can show congested resources and hot spots and give hints about possible solutions to performance problems 🔥 A good simulator can also validate planned fixes without messing with the operational network, or stop you from starting an expensive upgrade that might not hit the goals 💅 And during network planning, a simulator can reveal pathologies like single points of failure that need more redundancy, and bottlenecks and hot spots that need more capacity 🛡️

Routing simulators are especially useful in large networks 🌐 A routing simulator can identify planned links that might never actually be used to route traffic by the existing routing protocols 🙃 Simulators can also run scenario-based and perturbation-based analysis, plus sensitivity studies 🧪 The simulation results can kick off appropriate actions in various ways 🚀 For example, a big application of network simulation tools is figuring out how best to make the network evolve and grow to handle projected future demands 🌱✨

3.4 Optimization

Network performance optimization is all about fixing network issues by turning them into things you can actually solve: figuring out what the problem is, finding a solution, and then putting that solution in place 🔧✨. It can be corrective or perfective. Corrective optimization is when the goal is to remedy a problem that has already happened or is brewing (incipient = about to happen 👀). Perfective optimization is when the goal is to improve network performance even when there are no explicit problems and none are expected 💅.

Network performance optimization is a continual process, as noted previously 🔁. Optimization iterations may include real-time optimization sub-processes and non-real-time network planning sub-processes 🧩. The big difference between real-time optimization and network planning is mostly the time scale they work on and how granular their actions are ⏱️. One goal of a real-time optimization sub-process is to control how traffic gets mapped and distributed over the existing network infrastructure, so it can avoid and/or relieve congestion, assure satisfactory service delivery, and optimize resource utilization 🚦. Real-time optimization is needed because random incidents like fiber cuts or shifts in traffic demand will happen no matter how well a network is designed 💥. These incidents can cause congestion and other problems to show up in an operational network. Real-time optimization has to solve such problems on small to medium time-scales, ranging from micro-seconds to minutes or hours ⚡. Examples include queue management, IGP/BGP metric tuning, and using technologies such as MPLS explicit LSPs to change the paths of some traffic trunks [XIAO] 🛤️.

One of the jobs of the network planning sub-process is to kick off actions that systematically evolve the architecture, technology, topology, and capacity of a network 🏗️. When there's a problem in the network, real-time optimization should provide an immediate remedy 🚑. Because a prompt response is necessary, the real-time solution may not be the best possible one. Network planning may then be needed to refine the solution and improve the situation 🧠. Planning is also required to expand the network to support traffic growth and changes in traffic distribution over time 📈. As previously noted, a change in the topology and/or capacity of the network may be the outcome of network planning 🗺️.

Clearly, network planning and real-time performance optimization are mutually complementary activities, aka they're besties 🫶. A well-planned and designed network makes real-time optimization easier, while a systematic approach to real-time network performance optimization lets planning focus on long term issues instead of tactical considerations. Systematic real-time network performance optimization also provides valuable inputs and insights toward network planning 💜.

Stability is an important consideration in real-time network performance optimization. This aspect will be repeatedly addressed throughout this memo 🌈.

---

4.0 Historical Review and Recent Developments ✨

This section briefly reviews different traffic engineering approaches that have been proposed and implemented in telecommunications and computer networks 🕰️. The discussion isn't meant to be comprehensive. It's mainly here to illuminate pre-existing perspectives and prior art concerning traffic engineering in the Internet and in legacy telecommunications networks 📚.

4.1 Traffic Engineering in Classical Telephone Networks

This subsection gives a brief overview of traffic engineering in telephone networks, which often relates to how user traffic is steered from an originating node to the terminating node 📞. For a detailed description of the various routing strategies used in telephone networks, check out the book by G. Ash [ASH2] 📖.

The early telephone network relied on static hierarchical routing, where routing patterns stayed fixed no matter the state of the network or the time of day 🔒. The hierarchy was meant to accommodate overflow traffic, improve network reliability via alternate routes, and prevent call looping by employing strict hierarchical rules 🔁🚫. The network was typically over-provisioned, since a given fixed route had to be dimensioned so it could carry user traffic during a busy hour of any busy day 😮‍💨. Hierarchical routing was found to be too rigid once digital switches and stored program control showed up, since those could manage more complicated traffic engineering rules 💻.

Dynamic routing was introduced to fix that routing inflexibility so the network could operate more efficiently 🙌. This resulted in significant economic gains [HUSS87]. Dynamic routing typically reduces the overall loss probability by 10 to 20 percent (compared to static hierarchical routing) 📉. It can also improve network resilience by recalculating routes on a per-call basis and periodically updating routes 🛟.

There are three main types of dynamic routing in the telephone network: time-dependent routing, state-dependent routing (SDR), and event dependent routing (EDR) 🎀.

In time-dependent routing, regular variations in traffic loads (such as time of day or day of week) are exploited in pre-planned routing tables 🗓️. In state-dependent routing, routing tables are updated online according to the current state of the network (e.g., traffic demand, utilization, etc.) 🔄. In event dependent routing, routing changes are incepted by events (such as call setups encountering congested or blocked links), whereupon new paths are searched out using learning models 🧠. EDR methods are real-time adaptive, but they don't require global state information the way SDR does 🌍. Examples of EDR schemes include dynamic alternate routing (DAR) from BT, the state-and-time dependent routing (STR) from NTT, and the success-to-the-top (STT) routing from AT&T 🏆.

Dynamic non-hierarchical routing (DNHR) is an example of dynamic routing that was introduced in the AT&T toll network in the 1980's to respond to time-dependent information, such as regular load variations as a function of time ☎️. Time-dependent information in terms of load may be divided into three time scales: hourly, weekly, and yearly 📆. Correspondingly, three algorithms are defined to pre-plan the routing tables. The network design algorithm operates over a year-long interval, while the demand servicing algorithm operates on a weekly basis to fine tune link sizes and routing tables to correct forecast errors on the yearly basis. At the smallest time scale, the routing algorithm is used to make limited adjustments based on daily traffic variations 🌞. Network design and demand servicing are computed using offline calculations. Typically, those calculations require extensive searches on possible routes 🔍. Routing, on the other hand, may need online calculations to handle crankback ⚙️. DNHR adopts a "two-link" approach, whereby a path can consist of two links at most ✌️. The routing algorithm presents an ordered list of route choices between an originating switch and a terminating switch 📋. If a call overflows, a via switch (a tandem exchange between the originating switch and the terminating switch) would send a crankback signal to the originating switch. That switch would then select the next route, and so on, until there are no alternative routes available, in which case the call is blocked 🚫.

4.2 Evolution of Traffic Engineering in Packet Networks

This subsection reviews related prior work that was intended to improve the performance of data networks 📦. Fun fact: optimization of data network performance started way back in the early days of the ARPANET 🕹️. Other early commercial networks, such as SNA, also recognized the importance of performance optimization and service differentiation 💼.

In terms of traffic management, the Internet has been a best effort service environment until recently 🌬️. In particular, very limited traffic management capabilities existed in IP networks to provide differentiated queue management and scheduling services to packets belonging to different classes 😬.

In terms of routing control, the Internet has employed distributed protocols for intra-domain routing 🌐. These protocols are highly scalable and resilient 💪. However, they're based on simple algorithms for path selection, which have very limited functionality to allow flexible control of the path selection process 🙃.

In the following subsections, the evolution of practical traffic engineering mechanisms in IP networks and its predecessors is reviewed 🧚.

4.2.1 Adaptive Routing in the ARPANET

The early ARPANET recognized the importance of adaptive routing, where routing decisions are based on the current state of the network [MCQ80] 🧠. Early minimum delay routing approaches forwarded each packet to its destination along a path for which the total estimated transit time was the smallest 🏃‍♀️. Each node maintained a table of network delays, representing the estimated delay that a packet would experience along a given path toward its destination 📊. The minimum delay table was periodically transmitted by a node to its neighbors 📬. The shortest path, in terms of hop count, was also propagated to give the connectivity information 🔗.

One drawback to this approach is that dynamic link metrics tend to create "traffic magnets", causing congestion to be shifted from one location of a network to another location, resulting in oscillation and network instability 🧲💀.

4.2.2 Dynamic Routing in the Internet ✨

okay so the Internet grew out of the APARNET 🌐 and adopted dynamic routing algorithms with distributed control to figure out which paths packets should take on the way to their destinations 🗺️. these routing algorithms are basically adaptations of shortest path algorithms, where the costs are based on link metrics. a link metric can be based on static or dynamic quantities. a link metric based on static quantities may be assigned administratively according to local criteria, and a link metric based on dynamic quantities may be a function of a network congestion measure such as delay or packet loss 📉.

it was apparent early on that static link metric assignment was inadequate, because it can easily lead to unfavorable scenarios where some links become congested while others stay lightly loaded 😬. one of the many reasons static link metrics fall short is that link metric assignment was often done without considering the traffic matrix in the network. the routing protocols also didn't take traffic attributes and capacity constraints into account when making routing decisions. this means traffic concentration gets localized in subsets of the network infrastructure, which can potentially cause congestion 🚦. even if link metrics are assigned in accordance with the traffic matrix, unbalanced loads can still happen due to a number of factors, including:

• Resources may not be deployed in the most optimal locations from a routing perspective.
• Forecasting errors in traffic volume and/or traffic distribution.
• Dynamics in traffic matrix due to the temporal nature of traffic patterns, BGP policy change from peers, etc.

the inadequacy of the legacy Internet interior gateway routing system is one of the factors motivating the interest in path oriented technology with explicit routing and constraint-based routing capability, such as MPLS 🎀.

4.2.3 ToS Routing 🍓

Type-of-Service (ToS) routing is when different routes go to the same destination, with the selection depending on the ToS field of an IP packet [RFC-2474] 💅. the ToS classes may be classified as low delay and high throughput. each link is associated with multiple link costs, and each link cost is used to compute routes for a particular ToS. a separate shortest path tree is computed for each ToS. the shortest path algorithm has to be run for each ToS, which makes the computation very expensive 🥵. honestly, classical ToS-based routing is now outdated, because the IP header field has been replaced by a Diffserv field 💀. effective traffic engineering is also difficult in classical ToS-based routing, because each class still relies exclusively on shortest path routing, which localizes traffic concentration within the network 🙈.

4.2.4 Equal Cost Multi-Path 🦋

Equal Cost Multi-Path (ECMP) is another technique that tries to address the deficiency in the Shortest Path First (SPF) interior gateway routing systems [RFC-2328] 🔧. in the classical SPF algorithm, if two or more shortest paths exist to a given destination, the algorithm will choose one of them. ECMP modifies this slightly so that if two or more equal cost shortest paths exist between two nodes, the traffic between those nodes is distributed among the multiple equal-cost paths ⚖️. traffic distribution across the equal-cost paths is usually done in one of two ways: (1) packet-based, in a round-robin fashion, or (2) flow-based, using hashing on source and destination IP addresses and possibly other fields of the IP header. the first approach can easily cause out-of-order packets, while the second approach depends on the number and distribution of flows. flow-based load sharing may be unpredictable in an enterprise network, where the number of flows is relatively small and less heterogeneous (for example, hashing may not be uniform). but it is generally effective in core public networks, where the number of flows is large and heterogeneous 🌈.

in ECMP, link costs are static and bandwidth constraints are not considered. so ECMP tries to spread the traffic as equally as possible among the equal-cost paths, independent of the congestion status of each path 😅. as a result, given two equal-cost paths, one of them may end up more congested than the other. another drawback of ECMP is that load sharing cannot be achieved on multiple paths that have non-identical costs.

4.2.5 Nimrod 🧚

Nimrod is a routing system developed to provide heterogeneous service-specific routing in the Internet, while taking multiple constraints into account [RFC-1992] 👑. essentially, Nimrod is a link state routing protocol that supports path oriented packet forwarding. it uses the concept of maps to represent network connectivity and services at multiple levels of abstraction. mechanisms are provided to allow restriction of the distribution of routing information 🔒.

even though Nimrod never really got deployed in the public Internet, a number of its key concepts, like explicit routing (which lets you select paths at the originating nodes), are starting to show up in some recent constraint-based routing initiatives ✨. main character comeback, no cap 🫶.

4.3 Overlay Model 🧶

in the overlay model, a virtual-circuit network, such as ATM, frame relay, or WDM, provides virtual-circuit connectivity between routers that are located at the edges of a virtual-circuit cloud ☁️. in this mode, two routers connected through a virtual circuit see a direct adjacency between themselves, no matter what physical route the virtual circuit takes through the ATM, frame relay, or WDM network. so the overlay model basically decouples the logical topology that routers see from the physical topology that the ATM, frame relay, or WDM network manages 🫧. the overlay model based on ATM or frame relay lets a network administrator or an automaton use traffic engineering concepts to do path optimization, by reconfiguring or rearranging the virtual circuits so that a virtual circuit on a congested or sub-optimal physical link can be rerouted to a less congested or more optimal one 🛣️. in the overlay model, traffic engineering is also used to establish relationships between the traffic management parameters (e.g., PCR, SCR, and MBS for ATM) of the virtual-circuit technology and the actual traffic that traverses each circuit. these relationships can be established based on known or projected traffic profiles, and some other factors 📊.

the overlay model using IP over ATM requires managing two separate networks with different technologies (IP and ATM), which means increased operational complexity and cost 😩. in the fully-meshed overlay model, each router would peer with every other router in the network, so the total number of adjacencies is a quadratic function of the number of routers 📈. some of the issues with the overlay model are discussed in [AWD2] 📚.

4.4 Constrained-Based Routing 🎯

constraint-based routing refers to a class of routing systems that compute routes through a network subject to satisfying a set of constraints and requirements. in the most general setting, constraint-based routing may also try to optimize overall network performance while minimizing costs 💸.

the constraints and requirements may be imposed by the network itself or by administrative policies. constraints may include bandwidth, hop count, delay, and policy instruments such as resource class attributes. constraints may also include domain specific attributes of certain network technologies and contexts, which impose restrictions on the solution space of the routing function 🧩. path oriented technologies such as MPLS have made constraint-based routing feasible and attractive in public IP networks 💜.

the concept of constraint-based routing within the context of MPLS traffic engineering requirements in IP networks was first defined in [RFC-2702] 🥇.

unlike QoS routing (for example, see [RFC-2386] and [MA]), which generally addresses routing individual traffic flows to satisfy prescribed flow based QoS requirements subject to network resource availability, constraint-based routing is applicable to traffic aggregates as well as flows, and may be subject to a wide variety of constraints, which may include policy restrictions 🎀.

4.5 Overview of Other IETF Projects Related to Traffic Engineering 🌸

this subsection reviews a number of IETF activities pertinent to Internet traffic engineering 📝. these activities are primarily intended to evolve the IP architecture to support new service definitions, which allow preferential or differentiated treatment to be given to certain types of traffic 💅.

4.5.1 Integrated Services 🛎️

the IETF Integrated Services working group developed the integrated services (Intserv) model. this model requires resources, such as bandwidth and buffers, to be reserved a priori for a given traffic flow, so the quality of service requested by that flow is satisfied 🔐. the integrated services model includes components beyond those used in the best-effort model, such as packet classifiers, packet schedulers, and admission control. a packet classifier is used to identify flows that are to receive a certain level of service. a packet scheduler handles scheduling of service to different packet flows so QoS commitments are met. admission control is used to determine whether a router has the necessary resources to accept a new flow 🚪.

two services have been defined under the Integrated Services model: guaranteed service [RFC-2212] and controlled-load service [RFC-2211] 🫶.

the guaranteed service can be used for applications that require bounded packet delivery time. for this kind of application, data delivered after a pre-defined amount of time has elapsed is usually considered worthless ⏰. so guaranteed service was intended to provide a firm quantitative bound on the end-to-end packet delay for a flow. this is accomplished by controlling the queuing delay on network elements along the data flow path. the guaranteed service model does not, however, provide bounds on jitter (inter-arrival times between consecutive packets) 🙅‍♀️.

the controlled-load service can be used for adaptive applications that can tolerate some delay but are sensitive to traffic overload conditions. this type of application typically works fine when the network is lightly loaded, but its performance degrades significantly when the network is heavily loaded 📉. controlled-load service was therefore designed to provide approximately the same service as best-effort service in a lightly loaded network, regardless of actual network conditions. controlled-load service is described qualitatively, in that no target values of delay or loss are specified 🌤️.

the main issue with the Integrated Services model has been scalability [RFC-2998], especially in large public IP networks, which may potentially have millions of active micro-flows in transit concurrently 🌊.

a notable feature of the Integrated Services model is that it requires explicit signaling of QoS requirements from end systems to routers [RFC-2753] 📣. the Resource Reservation Protocol (RSVP) performs this signaling function and is a critical component of the Integrated Services model. the RSVP protocol is described next 🦋.

4.5.2 RSVP ✨

okay so RSVP is a soft state signaling protocol [RFC-2205] 💖 It supports receiver initiated establishment of resource reservations for both multicast and unicast flows. It was originally built as a signaling protocol inside the integrated services framework, so apps could tell the network what QoS they need, and the network could reserve the right resources to make it happen 🦋

Here's how the handshake goes. The sender (or source node) sends a PATH message to the receiver, using the same source and destination addresses as the traffic it's about to generate 🧚 The PATH message carries three things: (1) a sender Tspec describing the characteristics of the traffic, (2) a sender Template describing the format of the traffic, and (3) an optional Adspec, which supports the "one pass with advertising" (OPWA) idea [RFC-2205]. Every intermediate router along the path forwards the PATH message to the next hop chosen by the routing protocol. Once the receiver gets the PATH message, it replies with a RESV message containing a flow descriptor that asks for the reservation. The RESV message travels back to the sender or source node, going the opposite way along the path the PATH message took 🌸

Every intermediate router along the path gets a say. Each one can reject or accept the reservation request in the RESV message 💅 If a router rejects it, that router sends an error message to the receiver and the signaling process ends there. If a router accepts it, link bandwidth and buffer space get allocated for the flow, and the related flow state information gets installed in the router. Very dramatic, very precise 😭

Scalability was the big drama with the original RSVP spec. Reservations were required for micro-flows, so the amount of state network elements had to keep tended to grow linearly with the number of micro-flows. These issues are described in [RFC-2961].

Lowkey iconic, though, because RSVP has been modified and extended in several ways to fix those scaling problems 🔥 It's becoming a versatile signaling protocol for the Internet, no cap. For example, RSVP has been extended to reserve resources for aggregations of flows, to set up MPLS explicit label switched paths, and to handle other signaling jobs within the Internet. There are also a bunch of proposals to cut down on the amount of refresh messages needed to keep established RSVP sessions alive [RFC-2961] 🎀

A number of IETF working groups have been working on RSVP-related stuff. These include the original RSVP working group, the MPLS working group, the Resource Allocation Protocol working group, and the Policy Framework working group 🌈

4.5.3 Differentiated Services ✨

The goal of the Differentiated Services (Diffserv) effort in the IETF is to come up with scalable ways to sort traffic into behavior aggregates. That lets each behavior aggregate get treated differently, especially when resources like link bandwidth and buffer space are running low [RFC-2475] 🫶 One of the main reasons for Diffserv was to find other ways to do service differentiation on the Internet that avoid the scalability problems people ran into with the Intserv model 💜

The IETF Diffserv working group defined a Differentiated Services field in the IP header, called the DS field. It's made of six bits from the part of the IP header that used to be called the TOS octet. The DS field says what forwarding treatment a packet should get at a node [RFC-2474]. The Diffserv working group also standardized a bunch of Per-Hop Behavior (PHB) groups. With PHBs, you can define several classes of service using different classification, policing, shaping, and scheduling rules 🍓

If an end user wants to get Differentiated Services from its Internet Service Provider (ISP), the user may need a Service Level Agreement (SLA) with that ISP. An SLA may explicitly or implicitly specify a Traffic Conditioning Agreement (TCA), which defines classifier rules plus metering, marking, discarding, and shaping rules 💖

Packets get classified, and possibly policed and shaped, at the ingress to a Diffserv network ✨ When a packet crosses the boundary between different Diffserv domains, its DS field may be re-marked according to the existing agreements between those domains 🦋

Differentiated Services only lets the DS field indicate a finite number of service classes. The big win of Diffserv over Intserv is scalability: resources get allocated per class, and the amount of state information scales with the number of classes instead of the number of application flows 👑

It should be obvious by now that the Diffserv model basically handles traffic management on a per hop basis. The Diffserv control model is a collection of micro-TE control mechanisms 🌸 Other traffic engineering capabilities, like capacity management (including routing control), are also needed to deliver acceptable service quality in Diffserv networks. The concept of Per Domain Behaviors was introduced to better capture differentiated services across a whole domain [RFC-3086] 🧚

4.5.4 MPLS ✨

MPLS is an advanced forwarding scheme that also includes extensions to conventional IP control plane protocols. It extends the Internet routing model and boosts packet forwarding and path control [RFC-3031] 🔥

At the ingress to an MPLS domain, label switching routers (LSRs) classify IP packets into forwarding equivalence classes (FECs). They do this based on a variety of factors, including, e.g., a combination of the information in the packets' IP header and the local routing information the LSRs maintain 💅 An MPLS label then gets prepended to each packet according to its forwarding equivalence class. In a non-ATM/FR environment, the label is 32 bits long and contains a 20-bit label field, a 3-bit experimental field (formerly known as the Class-of-Service or CoS field), a 1-bit label stack indicator, and an 8-bit TTL field. In an ATM (FR) environment, the label is information encoded in the VCI/VPI (DLCI) field. An MPLS capable router (an LSR) examines the label and possibly the experimental field, then uses that info to make packet forwarding decisions 🎀

An LSR makes forwarding decisions by using the label prepended to a packet as the index into a local next hop label forwarding entry (NHLFE). The packet then gets processed as the NHLFE specifies. The incoming label may be swapped for an outgoing label, and the packet may be switched to the next LSR. This label-switching process is very similar to the label (VCI/VPI) swapping process in ATM networks 🦋 Before a packet leaves an MPLS domain, its MPLS label may be removed. A Label Switched Path (LSP) is the path a labeled packet travels between an ingress LSR and an egress LSR. The path of an explicit LSP is defined at the originating (ingress) node of the LSP. MPLS can use a signaling protocol like RSVP or LDP to set up LSPs 🌈

MPLS is a seriously powerful technology for Internet traffic engineering, because explicit LSPs let constraint-based routing be implemented efficiently in IP networks [AWD2]. The requirements for traffic engineering over MPLS are described in [RFC-2702]. Extensions to RSVP that support instantiation of explicit LSPs are discussed in [RFC-3209]. Extensions to LDP, known as CR-LDP, that support explicit LSPs are presented in [JAM] 💜

4.5.5 IP Performance Metrics ✨

The IETF IP Performance Metrics (IPPM) working group has been developing a set of standard metrics for monitoring the quality, performance, and reliability of Internet services 🍓 Network operators, end users, and independent testing groups can all apply these metrics. They give users and service providers a common understanding of the performance and reliability of the Internet component 'clouds' they use or provide [RFC-2330]. The criteria for performance metrics the IPPM WG developed are described in [RFC-2330] too 🫶 Examples of performance metrics include one-way packet loss [RFC-2680], one-way delay [RFC-2679], and connectivity measures between two nodes [RFC-2678]. Other metrics include second-order measures of packet loss and delay ✨

Some of the performance metrics the IPPM WG specified are useful for writing Service Level Agreements (SLAs). SLAs are sets of service level objectives negotiated between users and service providers, where each objective is a combination of one or more performance metrics, possibly subject to certain constraints 👑

4.5.6 Flow Measurement ✨

The IETF Real Time Flow Measurement (RTFM) working group produced an architecture document. It defines a way to specify traffic flows, plus a number of components for flow measurement: meters, meter readers, and managers [RFC-2722] 🦋 A flow measurement system lets network traffic flows get measured and analyzed at the flow level for all kinds of purposes. As noted in RFC 2722, a flow measurement system can be super useful in these contexts: (1) understanding the behavior of existing networks, (2) planning for network development and expansion, (3) quantifying network performance, (4) verifying the quality of network service, and (5) attributing network usage to users 💅

A flow measurement system is made up of meters, meter readers, and managers 🧚 A meter watches packets passing through a measurement point, sorts them into groups, accumulates usage data for each group (like the number of packets and bytes), and stores that usage data in a flow table. A group might represent a user application, a host, a network, a group of networks, and so on. A meter reader gathers usage data from various meters so it can be made available for analysis. A manager is in charge of configuring and controlling meters and meter readers 🌸 The instructions a meter gets from a manager include a flow specification, meter control parameters, and sampling techniques. The instructions a meter reader gets from a manager include the address of the meter whose data is to be collected, how often to collect data, and the types of flows to be collected 🎀

4.5.7 Endpoint Congestion Management ✨

[RFC-3124] is meant to provide a set of congestion control mechanisms that transport protocols can use 🌈 It also aims to develop mechanisms for unifying congestion control across a subset of an endpoint's active unicast connections, called a congestion group. A congestion manager keeps a constant eye on the state of the path for each congestion group under its control 💜 The manager uses that info to tell a scheduler how to split up bandwidth among the connections in that congestion group 🫶

4.6 Overview of ITU Activities Related to Traffic Engineering 🎀

Okay so now we're going over what the ITU-T has already done for traffic engineering in the old-school telephone-style networks. This section gives you the highlights 💅

The ITU-T has three big recs on this: E.600 [ITU-E600], E.701 [ITU-E701], and E.801 [ITU-E801]. E.600 is basically the vocabulary list 📚, and E.701 is where you find reference connections, Grade of Service (GOS), and traffic parameters for ISDN ✨. The reference connection idea is so clever. It lets you point at a representative kind of connection without caring how it's physically built. Bestie, that's giving abstraction goals 🦋

Here's how E.600 defines a connection: "a connection is an association of resources providing means for communication between two or more devices in, or attached to, a telecommunication network." 🌸 It also defines a resource as "any set of physically or conceptually identifiable entities within a telecommunication network, the use of which can be unambiguously determined" [ITU-E600]. Connections can look really different from each other, because the number and types of resources in a connection can vary 🧚

Connections usually cross different network segments. A connection can be local, national, or international 🌍 The whole point of reference connections is to clarify and specify traffic performance issues at the various interfaces between network domains. And each domain might be made of one or more service provider networks. So many layers, periodt 💜

Reference connections are also the base for defining GoS parameters for traffic engineering in the ITU-T framework. E.600 says "GoS refers to a number of traffic engineering variables which are used to provide a measure of the adequacy of a group of resources under specified conditions." 🫶 The GoS variables can be things like probability of loss, dial tone, or delay 📞 They matter a ton for designing and running the network internally, and for specifying how individual components perform 🔥

Now the drama: GoS is NOT the same thing as quality of service (QoS) in the ITU world 💀 QoS is the performance a telecom service user actually perceives, and it shows how happy the user is with the service 😭 QoS parameters look at what you can observe at the service access points and network interfaces, not at what's causing it inside the network. GoS is the network-side stuff. It measures how adequate a group of resources is under specified conditions 🍓 For a network to really serve its users well, GoS and QoS values have to line up, and GoS parameters usually make a major contribution to QoS. Main character relationship, honestly ✨

E.600 also says a set of GoS parameters has to be selected and defined end-to-end for each major service category the network provides. That helps the network provider make the network more efficient and effective 👑 Starting from a selected set of reference connections, you assign suitable target values to the chosen GoS parameters under both normal and high load conditions. Then those end-to-end GoS target values get split up among the individual resource components of the reference connections so they can be dimensioned 🌈

---

4.7 Content Distribution 🌸

Bestie, the Internet is basically run on client-server vibes. Web traffic is the biggest chunk of it right now, and in the future more fancy media servers might take over 📺 Where the major information servers live, and how well they perform, changes the traffic patterns across the whole Internet. It also changes how good the service feels to end users 💅

People have come up with a bunch of dynamic load balancing techniques to make replicated information servers perform better 🎀 These techniques can make spatial traffic characteristics way more dynamic across the Internet. Why? Because information servers can get picked on the fly, based on where the clients are, where the servers are, how busy each server is, how well different networks are doing, and how well different parts of a network are doing 🦋 Assigning distributed servers to clients like this is called Traffic Directing. And it works at the application layer 🧚

Traffic Directing schemes that give clients servers spread across several geographic locations might need empirical network performance statistics to make smarter choices ✨ In the future, network measurement systems may need to provide that kind of info. The exact parameters needed aren't defined yet, so this is still a open question 🔮

When the network is congested, Traffic Directing and Traffic Engineering systems should act in a coordinated manner. That's giving teamwork 🫶 But this topic is for further study, so it's still on the to-do list 📝

Location and replication of information servers, especially web servers, matter a lot for Internet traffic engineering. These servers account for a substantial proportion of Internet traffic, so they're a huge deal 🔥

---

5.0 Taxonomy of Traffic Engineering Systems 🌈

Okay, this section is a quick taxonomy of traffic engineering systems. You can build one based on the traffic engineering styles and views listed below 👑

• Time-dependent vs State-dependent vs Event-dependent
• Offline vs Online
• Centralized vs Distributed
• Local vs Global Information
• Prescriptive vs Descriptive
• Open Loop vs Closed Loop
• Tactical vs Strategic

Each of these classification systems gets a deeper dive in the subsections that follow ✨

---

5.1 Time-Dependent Versus State-Dependent Versus Event Dependent 💜

So traffic engineering methods can be time-dependent, state-dependent, or event-dependent 🦋 In this document, every TE scheme counts as dynamic. Static TE means no traffic engineering methodology or algorithm is being applied at all 🙈

Time-dependent TE uses historical info about periodic traffic variations, like time of day, to pre-program routing plans and other TE control mechanisms ⏰ Customer subscriptions or traffic projections can also come into play. Pre-programmed routing plans usually change on a relatively long time scale, like daily cycles. Time-dependent algorithms don't try to adapt to random traffic variations or changing network conditions 🌙 An example is a global centralized optimizer where the input is a traffic matrix and multi-class QoS requirements, as described in [MR99] 🧚

State-dependent TE adapts the routing plans for packets based on the network's current state 🌸 That current state tells you about actual traffic variations, the stuff that deviates from the usual patterns and couldn't be predicted from historical info 📈 Constraint-based routing is an example of state-dependent TE running on a relatively long time scale. An example on a relatively short time scale is the load-balancing algorithm described in [MATE] ⚡

The network state can be based on things like utilization, packet delay, packet loss, and so on. There are a few ways to get these values. For example, each router may flood the parameters to other routers periodically or when something triggers it 📡 Another option is for a particular router doing adaptive TE to send probe packets along a path to find out that path's state 🔍 And there's also the option of a management system gathering the relevant info from network elements 🗂️

Getting state info gathered and distributed quickly and accurately is critical for adaptive TE, because network conditions are so dynamic 🏃‍♀️ State-dependent algorithms can be applied to boost network efficiency and resilience 💪 Time-dependent algorithms suit predictable traffic variations better, while state-dependent algorithms are better at adapting to whatever the network state is at the moment 🎯

Event-dependent TE methods can also be used to pick TE paths 🦋 They're different from time-dependent and state-dependent methods in how they select paths. These algorithms are adaptive and distributed in nature, and they usually use learning models to find good TE paths in a network 🧠 State-dependent TE models usually rely on available-link-bandwidth (ALB) flooding to pick paths. Event-dependent methods don't need ALB flooding. Instead, they go searching for capacity with learning models, like the success-to-the-top (STT) method 🏆 ALB flooding is resource intensive, honestly. It eats link bandwidth to carry LSAs, processor capacity to process LSAs, and the overhead can limit area or autonomous system (AS) size 😭 Modeling results suggest event-dependent TE methods could cut ALB flooding overhead without losing network throughput performance [ASH3] 🌈

---

5.2 Offline Versus Online 🎀

Traffic engineering needs routing plans to be computed. That computation can happen offline or online ✨ Offline computation works when the routing plans don't have to run in real time. For example, routing plans built from forecast info can be computed offline. Offline computation is also typically used for extensive searches across multi-dimensional solution spaces 🔎

Online computation is required when routing plans have to adapt to changing network conditions, like in state-dependent algorithms ⚡ Offline computation can be super computationally demanding. Online computation, on the other hand, is geared toward relatively simple and fast calculations 💨 It's for selecting routes, fine-tuning resource allocations, and balancing load 🫶

---

5.3 Centralized Versus Distributed 👑

Centralized control has one central authority that decides the routing plans, and maybe other TE control parameters, on behalf of each router 🏛️ That authority collects network-state info from all routers periodically and sends routing info back to them. The routing update cycle is a critical parameter that directly affects how well the network being controlled performs ⏱️ Centralized control may need high processing power and high bandwidth control channels 💪

Distributed control is different. Each router picks its own routes autonomously, based on its view of the network state 🦋 The router can get that state info by using a probing method, or other routers can distribute it periodically using link state advertisements 📣 Network state info may also be disseminated under exceptional conditions 🚨

5.4 Local Versus Global ✨

okay so traffic engineering algos might need local or global network-state info 🧠

Local info is about a portion of the domain, like the bandwidth and packet loss rate of one particular path. honestly local state can be enough for some distributed-controlled TEs 🎀

Global info is about the whole domain being engineered. think a global traffic matrix plus loading info on every link in the domain of interest 🌍 global state is typically required with centralized control, and distributed TE systems may also need it in some cases 💅

5.5 Prescriptive Versus Descriptive 🔮

TE systems can also be classed as prescriptive or descriptive, bestie 💜

Prescriptive TE evaluates alternatives and recommends a course of action. it can be further split into corrective or perfective 🩹

• Corrective TE prescribes a course of action to address an existing or predicted anomaly 🚨
• Perfective TE prescribes a course of action to evolve and improve network performance even when no anomalies are evident ✨

Descriptive TE is the chill one 🧋 it characterizes the state of the network and assesses the impact of various policies, but it does NOT recommend any particular course of action 🙈

5.6 Open-Loop Versus Closed-Loop 🔁

Open-loop TE control is where the control action does not use feedback info from the current network state. it may still use its own local info for accounting purposes though 🧾

Closed-loop TE control is where the control action uses feedback info from the network state 🫶 that feedback can be historical info or current measurement 📊

5.7 Tactical vs Strategic 🎯

Tactical TE is all about fixing specific performance problems (like hot-spots) that pop up in the network, without thinking about the overall strategic big picture 🔥 without proper planning and insights, tactical TE tends to be ad hoc in nature 😭

Strategic TE is way more organized and systematic 👑 it takes into account the immediate and longer term consequences of specific policies and actions 🌸

6.0 Recommendations for Internet Traffic Engineering 🌈

this section gives high level recommendations for traffic engineering on the Internet 🦋 they're presented in general terms, so no stress 🫧

the recommendations describe the capabilities needed to solve a TE problem or hit a TE objective. broadly speaking they split into two groups: functional and non-functional 💖

Functional recommendations describe the functions a TE system should perform. those functions are needed to realize TE objectives by tackling TE problems 🧚

Non-functional recommendations relate to the quality attributes or state characteristics of a TE system 💅 these can contain conflicting assertions and may be hard to quantify precisely, which is very relatable tbh 💀

6.1 Generic Non-functional Recommendations 💎

the generic non-functional recommendations for Internet TE include: usability, automation, scalability, stability, visibility, simplicity, efficiency, reliability, correctness, maintainability, extensibility, interoperability, and security 🧩 in a given context some of these may be critical while others are optional, so prioritization may be required during the development phase to tailor the system to a specific operational context 🎀

here's a quick run-through of some of the aspects of these non-functional recommendations 🫶

Usability: usability is a human factor aspect of TE systems 🙈 it's about how easily a TE system can be deployed and operated. it's desirable to have a TE system that can be readily deployed in an existing network, and one that's easy to operate and maintain 🍓

Automation: whenever feasible, a TE system should automate as many TE functions as possible so the human effort needed to control and analyze operational networks is minimized 🤖 automation is especially imperative in large scale public networks, because human aspects of network operations are expensive and human errors can cause high risk network problems 😬 automation may mean building automatic feedback and intelligence into some components of the TE system 🧠

Scalability: contemporary public networks are growing very fast in both network size and traffic volume 🚀 so a TE system should be scalable to stay applicable as the network evolves 🌱 in particular, it should stay functional as the network expands in the number of routers and links, and as traffic volume grows. it should have a scalable architecture, should not adversely impair other functions and processes in a network element, and should not consume too much network resources when collecting and distributing state info or when exerting control 🛠️

Stability: stability is a very important consideration for TE systems that respond to changes in network state ⚖️ state-dependent TE methodologies typically mandate a tradeoff between responsiveness and stability. it is strongly recommended that when tradeoffs are warranted, the tradeoff be made in favor of stability (especially in public IP backbone networks) 🫶

Flexibility: a TE system should be flexible enough to allow for changes in optimization policy 🦎 in particular, it should provide sufficient configuration options so a network administrator can tailor it to a particular environment 🎛️ it may also be desirable to have both online and offline TE subsystems that can be independently enabled and disabled 🔀 TE systems used in multi-class networks should also have options to support class based performance evaluation and optimization 📚

Visibility: as part of the TE system, mechanisms should exist to collect statistics from the network and analyze them to determine how well the network is functioning 👀 derived statistics such as traffic matrices, link utilization, latency, packet loss, and other performance measures determined from network measurements can be used as indicators of prevailing network conditions 📈 other status info worth observing includes existing functional routing information (and, in the context of MPLS, existing LSP routes too), etc. 🗺️

Simplicity: generally, a TE system should be as simple as possible 🫧 more importantly, it should be relatively easy to use, meaning clean, convenient, and intuitive user interfaces ✨ simplicity in the user interface does not necessarily mean the TE system uses naive algorithms 🧁 when complex algorithms and internal structures are used, that complexity should be hidden as much as possible from the network administrator through the user interface 🙌

Interoperability: whenever feasible, TE systems and their components should be developed with open standards based interfaces, so they can interoperate with other systems and components 🤝

Security: security is a critical consideration in TE systems 🔐 these systems typically exert control over certain functional aspects of the network to achieve the desired performance objectives, so adequate measures must be taken to safeguard the integrity of the TE system 🛡️ adequate measures must also be taken to protect the network from vulnerabilities that originate from security breaches and other impairments within the TE system 🚨

the rest of this section will focus on some of the high level functional recommendations for traffic engineering 💜

6.2 Routing Recommendations ✨

Okay bestie, routing control is a huge deal in Internet traffic engineering 💅 It affects the key performance stuff like throughput, delay, and utilization. Honestly, it's really hard to get good service quality across a wide area network without effective routing control 😭 A desirable routing system is one that takes traffic characteristics and network constraints into account during route selection, and it still keeps things stable. That's the dream 🫶

Traditional shortest path first (SPF) interior gateway protocols are based on shortest path algorithms, and they have limited control capabilities for traffic engineering [RFC-2702, AWD2]. These limitations include:

1. The well known issues with pure SPF protocols, which don't take network constraints and traffic characteristics into account during route selection. For example, since IGPs always use the shortest paths (based on administratively assigned link metrics) to forward traffic, load sharing can't be done among paths of different costs. Using shortest paths to forward traffic conserves network resources, but it can cause these problems: 1) if traffic from a source to a destination exceeds the capacity of a link along the shortest path, that link (and so the shortest path) becomes congested, while a longer path between those two nodes may be under-utilized; 2) the shortest paths from different sources can overlap at some links. If the total traffic from those sources exceeds the capacity of any of these links, congestion will happen. Problems can also show up because traffic demand changes over time while the network topology and routing configuration can't be changed as quickly. This makes the topology and routing configuration sub-optimal over time, which can lead to persistent congestion problems 🙈
2. The Equal-Cost Multi-Path (ECMP) capability of SPF IGPs supports sharing traffic among equal cost paths between two nodes. However, ECMP tries to divide the traffic as equally as possible among the equal cost shortest paths. Generally, ECMP doesn't support configurable load sharing ratios among equal cost paths. So one of the paths can end up carrying a lot more traffic than the others, because it may also be carrying traffic from other sources. That can cause congestion along the busier path 😬
3. Modifying IGP metrics to control traffic routing tends to have network-wide effects. That means undesirable and unanticipated traffic shifts can get triggered as a result. Recent work described in Section 8.0 may be capable of better control [FT00, FT01] 👀

Because of these limitations, new capabilities are needed to enhance the routing function in IP networks. Some of these capabilities have been described elsewhere, and they're summarized below 💖

Constraint-based routing is desirable for evolving the routing architecture of IP networks, especially public IP backbones with complex topologies [RFC-2702]. Constraint-based routing computes routes to fulfill requirements subject to constraints. Constraints may include bandwidth, hop count, delay, and administrative policy instruments such as resource class attributes [RFC-2702, RFC-2386]. This makes it possible to select routes that satisfy a given set of requirements while following network and administrative policy constraints. Routes computed through constraint-based routing are not necessarily the shortest paths. Constraint-based routing works best with path oriented technologies that support explicit routing, such as MPLS 🦋

Constraint-based routing can also be used to redistribute traffic onto the infrastructure (even for best effort traffic). For example, if the bandwidth requirements for path selection and the reservable bandwidth attributes of network links are appropriately defined and configured, then congestion problems caused by uneven traffic distribution may be avoided or reduced. In this way, the performance and efficiency of the network can be improved ✨

A number of enhancements are needed to conventional link state IGPs, such as OSPF and IS-IS, so they can distribute the additional state information required for constraint-based routing. These extensions to OSPF were described in [KATZ], and the ones for IS-IS in [SMIT]. Essentially, these enhancements require propagating additional information in link state advertisements. Specifically, in addition to normal link-state information, an enhanced IGP is required to propagate the topology state information needed for constraint-based routing. Some of the additional topology state information includes link attributes such as reservable bandwidth and the link resource class attribute (an administratively specified property of the link). The resource class attribute concept was defined in [RFC-2702]. The additional topology state information is carried in new TLVs and sub-TLVs in IS-IS, or in the Opaque LSA in OSPF [SMIT, KATZ] 🎀

An enhanced link-state IGP may flood information more frequently than a normal IGP. That's because even without changes in topology, changes in reservable bandwidth or link affinity can trigger the enhanced IGP to start flooding. A tradeoff is typically required between how fresh the flooded information is and how often it's flooded. You want to avoid excessive consumption of link bandwidth and computational resources and, more importantly, avoid instability 🫠

In a TE system, it's also desirable for the routing subsystem to make the load splitting ratio among multiple paths (with equal cost or different cost) configurable. This capability gives network administrators more flexibility in controlling traffic distribution across the network. It can be very useful for avoiding or relieving congestion in certain situations. Examples can be found in [XIAO] 🌈

The routing system should also be able to control the routes of subsets of traffic without affecting the routes of other traffic, if sufficient resources exist for this purpose. This capability allows more refined control over the distribution of traffic across the network. For example, being able to move traffic from a source to a destination off its original path and onto another path (without affecting other traffic paths) lets traffic move from resource-poor network segments to resource-rich ones. Path oriented technologies such as MPLS inherently support this capability, as discussed in [AWD2] 🧚

Additionally, the routing subsystem should be able to select different paths for different classes of traffic (or for different traffic behavior aggregates) if the network supports multiple classes of service (different behavior aggregates) 👑

---

6.3 Traffic Mapping Recommendations 🍓

Traffic mapping is about assigning traffic workload onto pre-established paths to meet certain requirements. So while constraint-based routing deals with path selection, traffic mapping deals with assigning traffic to established paths, which may have been selected by constraint-based routing or by some other means. Traffic mapping can be performed by time-dependent or state-dependent mechanisms, as described in Section 5.1 💜

An important part of traffic mapping is being able to establish multiple paths between an originating node and a destination node, and being able to distribute the traffic between those two nodes across the paths according to some policies. A precondition for this scheme is having flexible mechanisms to partition traffic and then assign those traffic partitions onto the parallel paths. This requirement was noted in [RFC-2702]. When traffic is assigned to multiple parallel paths, it is recommended that special care should be taken to make sure packets belonging to the same application (or micro-flow) arrive in proper order at the destination node of the parallel paths 🫶

As a general rule, mechanisms that perform traffic mapping should aim to map traffic onto the network infrastructure in a way that minimizes congestion. If the total traffic load can't be accommodated, or if the routing and mapping functions can't react fast enough to changing traffic conditions, then a traffic mapping system may rely on short time scale congestion control mechanisms (such as queue management, scheduling, etc.) to mitigate congestion. So mechanisms that perform traffic mapping should complement existing congestion control mechanisms. In an operational network, it's generally desirable to map traffic onto the infrastructure so that intra-class and inter-class resource contention are minimized 💅

When traffic mapping techniques that depend on dynamic state feedback (e.g., MATE and such like) are used, special care must be taken to guarantee network stability. This one's serious, so don't skip it 🙈

---

6.4 Measurement Recommendations 📊

Measurement has been a big theme throughout this document when it comes to traffic engineering 🔥 Mechanisms should be provided to measure and collect statistics from the network to support the traffic engineering function. Extra capabilities may be needed to help analyze those statistics. The actions of these mechanisms should not adversely affect the accuracy and integrity of the statistics collected. And the mechanisms for statistical data acquisition should be able to scale as the network evolves 🌸

Traffic statistics may be classified according to long-term or short-term time scales. Long-term time scale traffic statistics are super useful for traffic engineering 🧁 They can capture or reflect periodicity in network workload (like hourly, daily, and weekly variations in traffic profiles) as well as traffic trends. Aspects of the monitored traffic statistics may also show class of service characteristics for a network that supports multiple classes of service. Analysis of the long-term traffic statistics MAY yield secondary statistics such as busy hour characteristics, traffic growth patterns, persistent congestion problems, hot-spots, and imbalances in link utilization caused by routing anomalies 🔍

A mechanism for constructing traffic matrices for both long-term and short-term traffic statistics should be in place. In multi-service IP networks, traffic matrices may be constructed for different service classes. Each element of a traffic matrix represents a statistic of traffic flow between a pair of abstract nodes. An abstract node may represent a router, a collection of routers, or a site in a VPN 🗺️

Measured traffic statistics should provide reasonable and reliable indicators of the current state of the network on the short-term scale. Some short term traffic statistics may reflect link utilization and link congestion status. Examples of congestion indicators include excessive packet delay, packet loss, and high resource utilization. Examples of mechanisms for distributing this kind of information include SNMP, probing techniques, FTP, IGP link state advertisements, etc. 🦋

6.5. Network Survivability 🛡️

Okay bestie, survivability is basically "can the network keep the vibes going when something breaks" 💅. It's the ability of a network to keep service continuous even when faults pop up 🦋. You get there by bouncing back quickly from impairments and keeping the required QoS for existing services once everything is restored ✨. This has become a huge deal in the Internet community because people want mission critical, real-time, and other high priority traffic riding the Internet 😭. You can tackle it at the device level by building network elements that are more reliable, and at the network level by baking redundancy into the architecture, design, and operation of networks 🫶. It is recommended that a philosophy of robustness and survivability be adopted when architecting, designing, and operating traffic engineering that controls IP networks (especially public IP networks) 👑. Different contexts need different amounts of survivability, so the mechanisms for it should be flexible enough to be tailored to different needs 🌈.

Failure protection and restoration tools now show up at multiple layers, and that's been getting better as network technologies improve 🔥. At the bottom of the stack, optical networks can do dynamic ring and mesh restoration at the wavelength level, plus the classic protection stuff 🌸. At the SONET/SDH layer, survivability comes from Automatic Protection Switching (APS) and self-healing ring and mesh architectures ⭐. Layer 2 technologies like ATM offer similar functionality, though they generally have slower mean restoration times 🙈. At the IP layer, rerouting is the traditional way to restore service after link and node outages. The catch is that IP rerouting only kicks in after routing convergence, which can take seconds to minutes 💀. Some newer MPLS developments make it possible to recover at the IP layer before convergence even finishes [SHAR] 🧚.

To meet advanced survivability requirements, path-oriented technologies like MPLS can help strengthen IP networks in a potentially cost effective way 💜. The perks of path-oriented technologies like MPLS for IP restoration get even more obvious when you need class based protection and restoration 👑.

Recently, a common suite of control plane protocols was proposed for both MPLS and optical transport networks, under the name Multi-protocol Lambda Switching [AWD1] 🦋. This new paradigm is expected to support even fancier mesh restoration at the optical layer for emerging IP over WDM network architectures ✨.

Another big thing about multi-layer survivability is that each layer protects and restores at different temporal granularities (different time scales) and different bandwidth granularities (from packet level all the way to wavelength level) 💅. Protection and restoration can also depend on different service classes and different network utility models 🎀.

The impact of an outage depends a lot on the service class and how long the outage lasts 😭. An outage can run from milliseconds (minor impact) to seconds (possible call drops for IP telephony and session time-outs for connection oriented transactions) to minutes and hours (potentially huge social and business impact) 💀.

Getting protection and restoration across multiple layers to work together cohesively, while keeping network survivability intact and the cost reasonable, is a genuinely hard job 🫶. Coordinating across layers may not always be feasible, because networks at different layers can belong to different administrative domains 🙈.

Here are some general recommendations for protection and restoration coordination 🍓:

• Protection and restoration capabilities from different layers should be coordinated whenever feasible and appropriate, so survivability comes out flexible and cost effective ✨. Minimizing function duplication across layers is one way to get there. Escalating alarms and other fault indicators from lower layers to higher layers can also be done in a coordinated manner. And setting a temporal order for restoration trigger timing across layers is another way to coordinate multi-layer protection and restoration 💖.
• Spare capacity at higher layers is often treated as working traffic at lower layers 🌈. Putting protection and restoration functions into many layers may increase redundancy and robustness, but it should not cause significant and avoidable inefficiencies in network resource utilization 😬.
• It is generally desirable to have protection and restoration schemes that are bandwidth efficient 💜.
• Failure notification throughout the network should be timely and reliable ⭐.
• Alarms and other fault monitoring and reporting capabilities should be provided at appropriate layers 🔥.

6.5.1. Survivability in MPLS Based Networks

MPLS is an important emerging technology that makes IP networks better in terms of features, capabilities, and services 🧚. Because MPLS is path-oriented, it can potentially offer faster and more predictable protection and restoration than conventional hop by hop routed IP systems 🦋. This subsection covers some basic aspects and recommendations for MPLS networks around protection and restoration. For a deeper dive on MPLS based recovery, check out [SHAR] 💅.

Protection types for MPLS networks can be sorted into link protection, node protection, path protection, and segment protection 🎀.

• Link Protection: The goal of link protection is to protect an LSP from a given link failure ✨. Under link protection, the path of the protection or backup LSP (the secondary LSP) is disjoint from the path of the working or operational LSP at the particular link that needs protecting 🫶. When the protected link fails, traffic on the working LSP gets switched over to the protection LSP at the head-end of the failed link 🔥. This is a local repair method, so it can be fast 🌸. It might fit better when some network elements along a path are less reliable than others 🙈.
• Node Protection: The goal of LSP node protection is to protect an LSP from a given node failure 👑. Under node protection, the path of the protection LSP is disjoint from the path of the working LSP at the node being protected 💜. The secondary path is also disjoint from the primary path at all links associated with the node being protected 🌈. When the node fails, traffic on the working LSP gets switched over to the protection LSP at the upstream LSR directly connected to the failed node ⭐.
• Path Protection: The goal of LSP path protection is to protect an LSP from failure at any point along its routed path 💖. Under path protection, the protection LSP's path is completely disjoint from the working LSP's path 💅. The big win is that the backup LSP covers the working LSP against all possible link and node failures along the path 😭. The exceptions are failures at the ingress and egress LSRs, and correlated failures that could hit the working and backup paths at the same time 💀. Since path selection is end-to-end, path protection might also be more efficient in terms of resource usage than link or node protection ✨. The tradeoff is that path protection may be slower than link and node protection in general 🙈.
• Segment Protection: An MPLS domain can be partitioned into multiple protection domains, so a failure in one protection domain gets fixed within that domain 🧚. When an LSP crosses multiple protection domains, a protection mechanism inside a domain only needs to protect the segment of the LSP that lies within that domain 🦋. Segment protection will generally be faster than path protection, since recovery happens closer to the fault 🔥.

6.5.2. Protection Option

Another thing to think about is protection options 🎀. The notation for a protection option is m:n protection, where m is the number of protection LSPs used to protect n working LSPs 💜. These are the feasible protection options 🍓:

• 1:1: one working LSP is protected/restored by one protection LSP 💖.
• 1:n: one protection LSP is used to protect/restore n working LSPs 🌈.
• n:1: one working LSP is protected/restored by n protection LSPs, possibly with a configurable load splitting ratio ✨. When more than one protection LSP is used, it may be desirable to share traffic across the protection LSPs when the working LSP fails to satisfy the bandwidth requirement of the traffic trunk tied to it 🫶. This can be especially handy when it isn't feasible to find one path that satisfies the bandwidth requirement of the primary LSP 👑.
• 1+1: traffic is sent concurrently on both the working LSP and the protection LSP 🔥. The egress LSR then picks one of the two LSPs using a local traffic integrity decision process, which compares the traffic received from both and flags any discrepancies 🌸. It's unlikely this option gets used a lot in IP networks because of its resource utilization inefficiency 😬. But if bandwidth becomes plentiful and cheap, this option might become quite viable and attractive in IP networks 🦋.

6.6. Traffic Engineering in Diffserv Environments ✨

okay so this section is the overview of the traffic engineering stuff that's specifically for Differentiated Services (Diffserv) [RFC-2475] capable IP networks 🫶

the vibe is that the internet keeps needing to support multiple classes of traffic, like best effort and mission critical data 💅 so IP networks have to tell traffic apart based on some criteria and give certain types of traffic preferential treatment 👑 lots of flows can get bundled together into a few behavior aggregates. the bundling is based on common performance requirements (packet loss ratio, delay, and jitter) or on common fields in the IP packet headers 🦋

as Diffserv evolves and shows up in operational networks, traffic engineering is going to be critical for making sure the SLAs defined within a given Diffserv service model actually get met 😭 service classes (CoS) can be supported in a Diffserv environment by concatenating per-hop behaviors (PHBs) along the routing path, using service provisioning mechanisms, and by appropriately configuring edge functionality such as traffic classification, marking, policing, and shaping 🌸 a PHB is the forwarding behavior a packet gets at a DS node (a Diffserv-compliant node), and it's done with buffer management and packet scheduling mechanisms 🔥 in this context, packets in a class are the ones that belong to a corresponding ordering aggregate ✨

traffic engineering can be a complement to Diffserv mechanisms to improve how network resources get used 🎀 but it isn't a necessary element in general, so no pressure 🙈 when it's used, it can run on an aggregated basis across all service classes [RFC-3270] or on a per service class basis. the aggregated version gives a better distribution of the total traffic load across the network resources (see [RFC-3270] for the detailed mechanisms for aggregate traffic engineering). the per service class version is the one we're talking about next, since it's specific to Diffserv and it's called Diffserv-aware traffic engineering [DIFF_TE] 💖

for some Diffserv networks, you might want to control how well certain service classes perform by enforcing certain relationships between the traffic workload each class brings in and the network resources allocated or provisioned for that class 🧚 those relationships between demand and resource allocation can be enforced with a combination of things like: (1) traffic engineering mechanisms on a per service class basis that enforce the relationship you want between the traffic a class contributes and the resources allocated to it, and (2) mechanisms that dynamically adjust the resources allocated to a service class so they track how much traffic that class is sending 🍓

it may also be worth limiting how much high priority traffic hurts the relatively low priority traffic 😤 one way is controlling the percentage of high priority traffic routed through a given link. another is increasing link capacities so lower priority traffic can still get decent service quality 💜 when the ratio of traffic workload from different service classes varies a lot from router to router, relying only on conventional IGP routing protocols or on traffic engineering mechanisms that ignore service classes might not cut it 💀 instead, it may be smart to do traffic engineering (especially routing control and mapping functions) on a per service class basis 🌈 in a domain that supports both MPLS and Diffserv, one way to do this is to define class specific LSPs and map traffic from each class onto one or more LSPs that match that service class. an LSP for a given service class can then be routed and protected/restored in a class dependent manner, following specific policies 👑

doing traffic engineering per class might mean some per-class parameters need to be distributed 💅 it's super common for some classes to share an aggregate constraint (like a maximum bandwidth requirement) without enforcing that constraint on each individual class. those classes can be grouped into a class-type, and then per-class-type parameters get distributed instead, which helps scalability 🫶 it also allows better bandwidth sharing between classes in the same class-type ✨ a class-type is a set of classes that meet these two conditions:

1. Classes in the same class-type have common aggregate requirements to satisfy required performance levels.
2. There is no requirement to be enforced at the level of individual class in the class-type. Note that it is still possible, nevertheless, to implement some priority policies for classes in the same class-type to permit preferential access to the class-type bandwidth through the use of preemption priorities. 🎀

an example of a class-type is a low-loss class-type that includes both AF1-based and AF2-based Ordering Aggregates 🦋 with that class-type, you could set up a priority policy that gives AF1-based traffic trunks higher preemption priority than AF2-based ones, or the other way around, or the same priority for both 🍓

for detailed requirements on Diffserv-aware traffic engineering, see [DIFF-TE] 💖

---

6.7. Network Controllability 🎀

okay this one is so important 😭 off-line (and on-line) traffic engineering considerations are pretty much useless if the network can't be controlled well enough to actually carry out the TE decisions and hit the performance goals 💅 capacity augmentation is the coarse grained solution. it's simple, and it can be a great move if bandwidth is abundant and cheap or if current or expected workload needs it 🌸 but bandwidth isn't always abundant and cheap, and the workload doesn't always demand more capacity, so it's not a magic fix 🙈

adjusting administrative weights and other routing protocol parameters gives you finer grained control 🧚 the catch is that it's hard to use and kind of imprecise, because of the routing interactions that happen across the network 💀 in some network contexts, more flexible, finer grained approaches that give more precise control over how traffic maps to routes, and over how routes are selected and placed, may be the right move 👑

control mechanisms can be manual (like administrative configuration), partially-automated (like scripts), or fully-automated (like policy based management systems) 🌈 automated mechanisms are especially needed in large scale networks 🔥 multi-vendor interoperability gets a boost when you develop and deploy standardized management systems (e.g., standard MIBs) and policies (PIBs) to support the control functions needed for TE objectives like load distribution and protection/restoration 🫶

last but not least, network control functions should be secure, reliable, and stable 🔒 they're often needed to keep operating correctly during network impairments, like network congestion or security attacks, so they really can't flake out when things get messy 😭🔥

7.0 Inter-Domain Considerations ✨

okay so inter-domain traffic engineering is all about optimizing performance for traffic that starts in one administrative domain and ends up in a different one 🌍💖

Traffic exchange between autonomous systems on the Internet happens through exterior gateway protocols. Right now, [BGP4] is the standard exterior gateway protocol for the Internet 👑 BGP comes with a bunch of attributes and capabilities (route filtering, for example) that you can use for inter-domain traffic engineering. Basically, BGP lets you control routing information and traffic exchange between Autonomous Systems (AS's) on the Internet 🦋 It also has a sequential decision process that works out how preferred the different routes to a given destination network are 🧠 There are two fundamental aspects to inter-domain TE with BGP:

• Route Redistribution: controlling the import and export of routes between AS's, and controlling the redistribution of routes between BGP and other protocols within an AS.
• Best path selection: selecting the best path when there are multiple candidate paths to a given destination network. Best path selection is performed by the BGP decision process based on a sequential procedure, taking a number of different considerations into account. Ultimately, best path selection under BGP boils down to selecting preferred exit points out of an AS towards specific destination networks. The BGP path selection process can be influenced by manipulating the attributes associated with the BGP decision process. These attributes include: NEXT-HOP, WEIGHT (Cisco proprietary, which is also implemented by some other vendors), LOCAL-PREFERENCE, AS-PATH, ROUTE-ORIGIN, MULTI-EXIT-DESCRIMINATOR (MED), IGP METRIC, etc. 💅

Route-maps give you the flexibility to implement complex BGP policies based on pre-configured logical conditions ✨ Specifically, Route-maps can be used to control import and export policies for incoming and outgoing routes, control the redistribution of routes between BGP and other protocols, and influence the selection of best paths by manipulating the attributes associated with the BGP decision process. Very complex logical expressions that implement various types of policies can be built using a combination of Route-maps, BGP-attributes, Access-lists, and Community attributes 🎀

---

Okay, now the plot twist 🙈 When you're looking at possible strategies for inter-domain TE with BGP, it's worth noting that the outbound traffic exit point is controllable, whereas the interconnection point where inbound traffic is received from an EBGP peer typically is not, unless a special arrangement is made with the peer sending the traffic 😭 So it's up to each individual network to implement sound TE strategies that deal with the efficient delivery of outbound traffic from one's customers to one's peering points 🫶 The vast majority of TE policy is based on a "closest exit" strategy, which offloads interdomain traffic at the nearest outbound peer point towards the destination autonomous system. Most methods of manipulating the point where inbound traffic enters a network from an EBGP peer (inconsistent route announcements between peering points, AS pre-pending, and sending MEDs) are either ineffective, or not accepted in the peering community 💀

Inter-domain TE with BGP is generally effective, but it's usually applied in a trial-and-error fashion 🥲 A systematic approach for inter-domain traffic engineering has yet to be devised.

Inter-domain TE is inherently harder than intra-domain TE under the current Internet architecture, and it's both technical and administrative 😤 Technically, while topology and link state information help with mapping traffic more effectively, BGP does not propagate that information across domain boundaries, for stability and scalability reasons. Administratively, there are differences in operating costs and network capacities between domains. Generally, what counts as a good solution in one domain may not be a good solution in another domain. Moreover, it would generally be considered inadvisable for one domain to let another domain influence the routing and management of traffic in its network 🚫

---

Now for the MPLS glow-up 🧚 MPLS TE-tunnels (explicit LSPs) can potentially add a degree of flexibility in the selection of exit points for inter-domain routing. The concept of relative and absolute metrics can be applied to this purpose. The idea is that if BGP attributes are defined so the BGP decision process depends on IGP metrics to select exit points for inter-domain traffic, then some inter-domain traffic destined to a given peer network can be made to prefer a specific exit point. You do this by establishing a TE-tunnel from the router making the selection to the peering point, and assigning the TE-tunnel a metric that's smaller than the IGP cost to all other peering points 🌸 If a peer accepts and processes MEDs, then a similar MPLS TE-tunnel based scheme can be applied to make certain entrance points preferred, by setting MED to be an IGP cost that has been modified by the tunnel metric 🔥

Similar to intra-domain TE, inter-domain TE is best accomplished when a traffic matrix can be derived to depict the volume of traffic from one autonomous system to another 📊

Redistribution of inter-domain traffic generally requires coordination between peering partners 🤝 An export policy in one domain that results in load redistribution across peer points with another domain can significantly affect the local traffic matrix inside the domain of the peering partner. That, in turn, affects the intra-domain TE because the spatial distribution of traffic has changed. So it's mutually beneficial for peering partners to coordinate with each other before attempting any policy changes that might cause significant shifts in inter-domain traffic. In certain contexts, this coordination can be quite challenging, for both technical and non-technical reasons 😩

And finally, the big open question 🔮 It's a matter of speculation whether MPLS, or similar technologies, can be extended to allow selection of constrained paths across domain boundaries 🌈

8.0 Overview of Contemporary TE Practices in Operational IP Networks ✨

okay so this section is a little tour of some contemporary traffic engineering practices in IP networks 💅 The focus is mostly on the parts that deal with controlling the routing function in real operational settings. It's just an overview of the stuff people commonly do, not a complete list, so don't panic if your favourite trick isn't here 🫶

Service providers are already using a bunch of the TE mechanisms we've covered in this document to make their IP networks perform better 🚀 The lineup looks like this: capacity planning for long time scales, routing control using IGP metrics and MPLS for medium time scales, the overlay model (also medium time scales), and traffic management mechanisms for short time scales 🦋

When a service provider wants to build a new IP network or grow an existing one, effective capacity planning should be a big part of the process 🎀 The plan may take a bunch of things into account, like where any new nodes go (if there are any), existing and predicted traffic patterns, costs, link capacity, topology, routing design, and survivability 🌸

Performance optimization of operational networks is usually an ongoing process 🔄 Traffic statistics, performance parameters, and fault indicators are constantly collected from the network, and that empirical data gets analyzed and used to trigger various TE mechanisms 📊 Tools that do what-if analysis can help the TE process too, because they let people review different scenarios before a new set of configurations goes live in the operational network 🔮

Traditionally, intra-domain real-time TE with IGP was done by increasing the OSPF or IS-IS metric of a congested link until enough traffic had been diverted away from it 😅 This approach has some limitations, as discussed in Section 6.2 🙈 Recently, some new intra-domain TE approaches/tools have been proposed [RR94][FT00][FT01][WANG]. These take a traffic matrix, network topology, and network performance objective(s) as input, and produce some link metrics and possibly some unequal load-sharing ratios to be set at the head-end routers of some ECMPs as output. These new progresses open up new possibilities for doing intra-domain TE with IGP in a more systematic way 🧠✨

The overlay model (IP over ATM or IP over Frame relay) is another approach that's commonly used in practice [AWD2] 🧁 The IP over ATM technique is no longer viewed favorably, thanks to recent advances in MPLS and router hardware technology 💀

MPLS for traffic engineering has started to get deployed in some service provider networks 🌈 One operational scenario is to deploy MPLS in conjunction with an IGP (IS-IS-TE or OSPF-TE) that supports the traffic engineering extensions. You pair that with constraint-based routing for explicit route computations, plus a signaling protocol (e.g., RSVP-TE or CRLDP) for LSP instantiation 🔥

In contemporary MPLS traffic engineering, network administrators specify and configure link attributes and resource constraints, like maximum reservable bandwidth and resource class attributes, for links (interfaces) within the MPLS domain 🛠️ A link state protocol with TE extensions (IS-IS-TE or OSPF-TE) propagates information about network topology and link attributes to all routers in the routing area 📡 Network administrators also specify all the LSPs that originate at each router. For each LSP, they specify the destination node and the attributes of the LSP, which indicate the requirements to be satisfied during the path selection process 📝 Then each router uses a local constraint-based routing process to compute explicit paths for all LSPs originating from it. After that, a signaling protocol is used to instantiate the LSPs. By assigning proper bandwidth values to links and LSPs, congestion caused by uneven traffic distribution can generally be avoided or mitigated 💖

The bandwidth attributes of LSPs used for TE can be updated periodically 🕰️ The basic concept is that the bandwidth assigned to an LSP should relate in some manner to the bandwidth requirements of the traffic that actually flows through it 🎯 The traffic attribute of an LSP can be modified to accommodate traffic growth and persistent traffic shifts. If network congestion happens because of some unexpected events, existing LSPs can be rerouted to ease the situation, or the network administrator can configure new LSPs to divert some traffic to alternative paths 🛣️ The reservable bandwidth of congested links can also be reduced to force some LSPs to get rerouted to other paths 🚧

In an MPLS domain, a traffic matrix can also be estimated by monitoring the traffic on LSPs 📈 Those traffic statistics are useful for a variety of purposes, including network planning and network optimization. Current practice suggests that deploying an MPLS network with hundreds of routers and thousands of LSPs is feasible 🙌 In summary, recent deployment experience suggests the MPLS approach is very effective for traffic engineering in IP networks [XIAO] 👑

---

As mentioned previously in Section 7.0, you usually have no direct control over the distribution of inbound traffic 🙃 So the main goal of contemporary inter-domain TE is to optimize the distribution of outbound traffic across multiple inter-domain links. When you're operating a global network, keeping the ability to run it in a regional fashion where desired, while still enjoying the perks of being global, also becomes an important objective 🌍

Inter-domain TE with BGP usually starts with placing multiple peering interconnection points in locations that have high peer density, are close to where originating and terminating traffic lives on your own network, and are the lowest in cost 📍 There are generally several locations in each region of the world where the vast majority of major networks congregate and interconnect. Some location-decision problems that come up with inter-domain routing are discussed in [AWD5] 🗺️

Once the interconnect locations are picked and the circuits are implemented, you decide how best to handle the routes heard from the peer, and how to propagate the peers' routes within your own network 💅 One way to engineer outbound traffic flows on a network with many EBGP peers is to create a hierarchy of peers. Generally, the Local Preferences of all peers are set to the same value so that the shortest AS paths get chosen to forward traffic 🧩 Then, by overwriting the inbound MED metric (Multi-exit-discriminator metric, also referred to as "BGP metric". Both terms are used interchangeably in this document) with BGP metrics on routes received at different peers, the hierarchy can be formed 🪜 For example, all Local Preferences can be set to 200, preferred private peers can be assigned a BGP metric of 50, the rest of the private peers can be assigned a BGP metric of 100, and public peers can be assigned a BGP metric of 600 💖

"Preferred" peers might be defined as those peers with the most available capacity, whose customer base is larger in comparison to other peers, whose interconnection costs are the lowest, and with whom upgrading existing capacity is the easiest 🌟 In a network with low utilization at the edge, this works really well. The same concept can be applied to a network with higher edge utilization by creating more levels of BGP metrics between peers, which gives you more granularity in selecting the exit points for traffic bound for a dual homed customer on a peer's network 🎀

By only replacing inbound MED metrics with BGP metrics, only the exit points of routes with equal AS-Path length are being changed. (The BGP decision considers Local Preference first, then AS-Path length, and then BGP metric) 🤓 For example, assume a network has two possible egress points, peer A and peer B. Each peer has 40% of the Internet's routes exclusively on its network, while the remaining 20% of the Internet's routes are from customers who dual home between A and B. Assume that both peers have a Local Preference of 200 and a BGP metric of 100. If the link to peer A is congested, increasing its BGP metric while leaving the Local Preference at 200 will ensure that the 20% of total routes belonging to dual homed customers prefer peer B as the exit point 😌 The example above would be used in a situation where all exit points to a given peer are close to congestion levels, and traffic needs to be shifted away from that peer entirely.

When there are multiple exit points to a given peer and only one of them is congested, it isn't necessary to shift traffic away from the peer entirely, just from the one congested circuit 🎯 This can be achieved by using passive IGP-metrics, AS-path filtering, or prefix filtering.

Occasionally, more drastic changes are needed, for example when dealing with a "problem peer" who is difficult to work with on upgrades or is charging high prices for connectivity to their network 😤 In that case, the Local Preference to that peer can be reduced below the level of other peers. This effectively reduces the amount of traffic sent to that peer to only originating traffic (assuming no transit providers are involved) 🙅‍♀️

⚠️ This type of change can affect a large amount of traffic, and is only used after other methods have failed to provide the desired results. Don't reach for it first! 💀

Although it's not much of an issue in regional networks, the propagation of a peer's routes back through the network must be considered when a network is peering on a global scale 🌐 Sometimes business considerations can influence the choice of BGP policies in a given context. For example, it may be imprudent, from a business perspective, to operate a global network and provide full access to the global customer base to a small network in a particular country 💼 However, for the purpose of providing one's own customers with quality service in a particular region, good connectivity to that in-country network may still be necessary. This can be achieved by assigning a set of communities at the edge of the network, which have a known behavior when routes tagged with those communities propagate back through the core 🏷️ Routes heard from local peers are prevented from propagating back to the global network, whereas routes learned from larger peers may be allowed to propagate freely throughout the entire global network. By implementing a flexible community strategy, the benefits of using a single global AS Number (ASN) can be realized, while the benefits of operating regional networks can also be taken advantage of 🧚 An alternative to doing this is to use different ASNs in different regions, with the consequence that the AS path length for routes announced by that service provider will increase 💜

9.0 Conclusion 🎀

okay bestie, this is the wrap-up 💖 The document laid out principles for traffic engineering on the Internet ✨ It gave an overview of the big issues around TE in IP networks, explained the context of TE, and presented a TE process model and a taxonomy of TE styles 🧠 It also did a brief history tour of relevant developments 🕰️ and a survey of contemporary TE techniques in operational networks 🔍 On top of all that, it specified a set of generic requirements, recommendations, and options for Internet traffic engineering 👑 Iconic, honestly 💅

10.0 Security Considerations 🔐

ngl this one is short and sweet 🙈 This document does not introduce new security issues 🫶

11.0 Acknowledgments 🌸

the authors wanted to give a shoutout to everyone who helped out, and we are not going to skip anyone 😭🫶 Jim Boyle gave inputs on the recommendations section 💜 Francois Le Faucheur helped with Diffserv aspects 🦋 Blaine Christian helped with measurement 📊 Gerald Ash gave inputs on routing in telephone networks and wrote text on event-dependent TE methods ⭐ Steven Wright contributed inputs on network controllability 🎀 and Jonathan Aufderheide contributed inputs on inter-domain TE with BGP 🌈 Special thanks to Randy Bush for proposing the TE taxonomy based on "tactical vs strategic" methods 👑 The subsection describing an "Overview of ITU Activities Related to Traffic Engineering" was adapted from a contribution by Waisum Lai 🧚 Useful feedback and pointers to relevant materials came from J. Noel Chiappa 🍓 Additional comments came from Glenn Grotefeld during the working last call process 💕 Finally, the authors thanked Ed Kern, the TEWG co-chair, for his comments and support 🥰✨

12.0 References ✨

Okay bestie, the reference list is basically the whole library card catalogue for this doc, and it's giving scholarly main character energy 📚💅. Every single citation stays exactly as written, so you can chase down any of them 🔍.

[ASH2] J. Ash, Dynamic Routing in Telecommunications Networks, McGraw Hill, 1998.

[ASH3] Ash, J., "TE & QoS Methods for IP-, ATM-, & TDM-Based Networks", Work in Progress, March 2001.

[AWD1] D. Awduche and Y. Rekhter, "Multiprocotol Lambda Switching: Combining MPLS Traffic Engineering Control with Optical Crossconnects", IEEE Communications Magazine, March 2001.

[AWD2] D. Awduche, "MPLS and Traffic Engineering in IP Networks", IEEE Communications Magazine, Dec. 1999.

[AWD5] D. Awduche et al, "An Approach to Optimal Peering Between Autonomous Systems in the Internet", International Conference on Computer Communications and Networks (ICCCN'98), Oct. 1998.

[CRUZ] R. L. Cruz, "A Calculus for Network Delay, Part II: Network Analysis", IEEE Transactions on Information Theory, vol. 37, pp. 132-141, 1991.

[DIFF-TE] Le Faucheur, F., Nadeau, T., Tatham, M., Telkamp, T., Cooper, D., Boyle, J., Lai, W., Fang, L., Ash, J., Hicks, P., Chui, A., Townsend, W. and D. Skalecki, "Requirements for support of Diff-Serv-aware MPLS Traffic Engineering", Work in Progress, May 2001.

[ELW95] A. Elwalid, D. Mitra and R.H. Wentworth, "A New Approach for Allocating Buffers and Bandwidth to Heterogeneous, Regulated Traffic in an ATM Node", IEEE IEEE Journal on Selected Areas in Communications, 13:6, pp. 1115-1127, Aug. 1995.

[FGLR] A. Feldmann, A. Greenberg, C. Lund, N. Reingold, and J. Rexford, "NetScope: Traffic Engineering for IP Networks", IEEE Network Magazine, 2000.

[FLJA93] S. Floyd and V. Jacobson, "Random Early Detection Gateways for Congestion Avoidance", IEEE/ACM Transactions on Networking, Vol. 1 Nov. 4., p. 387-413, Aug. 1993.

[FLOY94] S. Floyd, "TCP and Explicit Congestion Notification", ACM Computer Communication Review, V. 24, No. 5, p. 10-23, Oct. 1994.

[FT00] B. Fortz and M. Thorup, "Internet Traffic Engineering by Optimizing OSPF Weights", IEEE INFOCOM 2000, Mar. 2000.

[FT01] B. Fortz and M. Thorup, "Optimizing OSPF/IS-IS Weights in a Changing World", www.research.att.com/~mthorup/PAPERS/papers.html.

[HUSS87] B.R. Hurley, C.J.R. Seidl and W.F. Sewel, "A Survey of Dynamic Routing Methods for Circuit-Switched Traffic", IEEE Communication Magazine, Sep. 1987.

[ITU-E600] ITU-T Recommendation E.600, "Terms and Definitions of Traffic Engineering", Mar. 1993.

[ITU-E701] ITU-T Recommendation E.701, "Reference Connections for Traffic Engineering", Oct. 1993.

[ITU-E801] ITU-T Recommendation E.801, "Framework for Service Quality Agreement", Oct. 1996.

[JAM] Jamoussi, B., Editior, Andersson, L., Collon, R. and R. Dantu, "Constraint-Based LSP Setup using LDP", RFC 3212, January 2002.

[KATZ] Katz, D., Yeung, D. and K. Kompella, "Traffic Engineering Extensions to OSPF", Work in Progress, February 2001.

[LNO96] T. Lakshman, A. Neidhardt, and T. Ott, "The Drop from Front Strategy in TCP over ATM and its Interworking with other Control Features", Proc. INFOCOM'96, p. 1242-1250, 1996.

[MA] Q. Ma, "Quality of Service Routing in Integrated Services Networks", PhD Dissertation, CMU-CS-98-138, CMU, 1998.

[MATE] A. Elwalid, C. Jin, S. Low, and I. Widjaja, "MATE: MPLS Adaptive Traffic Engineering", Proc. INFOCOM'01, Apr. 2001.

[MCQ80] J.M. McQuillan, I. Richer, and E.C. Rosen, "The New Routing Algorithm for the ARPANET", IEEE. Trans. on Communications, vol. 28, no. 5, pp. 711-719, May 1980.

[MR99] D. Mitra and K.G. Ramakrishnan, "A Case Study of Multiservice, Multipriority Traffic Engineering Design for Data Networks", Proc. Globecom'99, Dec 1999.

[RFC-1458] Braudes, R. and S. Zabele, "Requirements for Multicast Protocols", RFC 1458, May 1993.

[RFC-1771] Rekhter, Y. and T. Li, "A Border Gateway Protocol 4 (BGP-4)", RFC 1771, March 1995.

[RFC-1812] Baker, F., "Requirements for IP Version 4 Routers", STD 4, RFC 1812, June 1995.

[RFC-1992] Castineyra, I., Chiappa, N. and M. Steenstrup, "The Nimrod Routing Architecture", RFC 1992, August 1996.

[RFC-1997] Chandra, R., Traina, P. and T. Li, "BGP Community Attributes", RFC 1997, August 1996.

[RFC-1998] Chen, E. and T. Bates, "An Application of the BGP Community Attribute in Multi-home Routing", RFC 1998, August 1996.

[RFC-2205] Braden, R., Zhang, L., Berson, S., Herzog, S. and S. Jamin, "Resource Reservation Protocol (RSVP) - Version 1 Functional Specification", RFC 2205, September 1997.

[RFC-2211] Wroclawski, J., "Specification of the Controlled-Load Network Element Service", RFC 2211, September 1997.

[RFC-2212] Shenker, S., Partridge, C. and R. Guerin, "Specification of Guaranteed Quality of Service", RFC 2212, September 1997.

[RFC-2215] Shenker, S. and J. Wroclawski, "General Characterization Parameters for Integrated Service Network Elements", RFC 2215, September 1997.

[RFC-2216] Shenker, S. and J. Wroclawski, "Network Element Service Specification Template", RFC 2216, September 1997.

[RFC-2328] Moy, J., "OSPF Version 2", STD 54, RFC 2328, July 1997.

[RFC-2330] Paxson, V., Almes, G., Mahdavi, J. and M. Mathis, "Framework for IP Performance Metrics", RFC 2330, May 1998.

[RFC-2386] Crawley, E., Nair, R., Rajagopalan, B. and H. Sandick, "A Framework for QoS-based Routing in the Internet", RFC 2386, August 1998.

[RFC-2474] Nichols, K., Blake, S., Baker, F. and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, December 1998.

[RFC-2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z. and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, December 1998.

[RFC-2597] Heinanen, J., Baker, F., Weiss, W. and J. Wroclawski, "Assured Forwarding PHB Group", RFC 2597, June 1999.

[RFC-2678] Mahdavi, J. and V. Paxson, "IPPM Metrics for Measuring Connectivity", RFC 2678, September 1999.

[RFC-2679] Almes, G., Kalidindi, S. and M. Zekauskas, "A One-way Delay Metric for IPPM", RFC 2679, September 1999.

[RFC-2680] Almes, G., Kalidindi, S. and M. Zekauskas, "A One-way Packet Loss Metric for IPPM", RFC 2680, September 1999.

[RFC-2702] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M. and J. McManus, "Requirements for Traffic Engineering over MPLS", RFC 2702, September 1999.

[RFC-2722] Brownlee, N., Mills, C. and G. Ruth, "Traffic Flow Measurement: Architecture", RFC 2722, October 1999.

[RFC-2753] Yavatkar, R., Pendarakis, D. and R. Guerin, "A Framework for Policy-based Admission Control", RFC 2753, January 2000.

[RFC-2961] Berger, L., Gan, D., Swallow, G., Pan, P., Tommasi, F. and S. Molendini, "RSVP Refresh Overhead Reduction Extensions", RFC 2961, April 2000.

[RFC-2998] Bernet, Y., Ford, P., Yavatkar, R., Baker, F., Zhang, L., Speer, M., Braden, R., Davie, B., Wroclawski, J. and E. Felstaine, "A Framework for Integrated Services Operation over Diffserv Networks", RFC 2998, November 2000.

[RFC-3031] Rosen, E., Viswanathan, A. and R. Callon, "Multiprotocol Label Switching Architecture", RFC 3031, January 2001.

[RFC-3086] Nichols, K. and B. Carpenter, "Definition of Differentiated Services Per Domain Behaviors and Rules for their Specification", RFC 3086, April 2001.

[RFC-3124] Balakrishnan, H. and S. Seshan, "The Congestion Manager", RFC 3124, June 2001.

[RFC-3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V. and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP Tunnels", RFC 3209, December 2001.

[RFC-3210] Awduche, D., Hannan, A. and X. Xiao, "Applicability Statement for Extensions to RSVP for LSP-Tunnels", RFC 3210, December 2001.

[RFC-3213] Ash, J., Girish, M., Gray, E., Jamoussi, B. and G. Wright, "Applicability Statement for CR-LDP", RFC 3213, January 2002.

[RFC-3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaahanen, P., Krishnan, R., Cheval, P. and J. Heinanen, "Multi- Protocol Label Switching (MPLS) Support of Differentiated Services", RFC 3270, April 2002.

[RR94] M.A. Rodrigues and K.G. Ramakrishnan, "Optimal Routing in Shortest Path Networks", ITS'94, Rio de Janeiro, Brazil.

[SHAR] Sharma, V., Crane, B., Owens, K., Huang, C., Hellstrand, F., Weil, J., Anderson, L., Jamoussi, B., Cain, B., Civanlar, S. and A. Chui, "Framework for MPLS Based Recovery", Work in Progress.

[SLDC98] B. Suter, T. Lakshman, D. Stiliadis, and A. Choudhury, "Design Considerations for Supporting TCP with Per-flow Queueing", Proc. INFOCOM'98, p. 299-306, 1998.

[SMIT] Smit, H. and T. Li, "IS-IS extensions for Traffic Engineering", Work in Progress.

[WANG] Y. Wang, Z. Wang, L. Zhang, "Internet traffic engineering without full mesh overlaying", Proceedings of INFOCOM'2001, April 2001.

[XIAO] X. Xiao, A. Hannan, B. Bailey, L. Ni, "Traffic Engineering with MPLS in the Internet", IEEE Network magazine, Mar. 2000.

[YARE95] C. Yang and A. Reddy, "A Taxonomy for Congestion Control Algorithms in Packet Switching Networks", IEEE Network Magazine, p. 34-45, 1995.

---

13.0 Authors' Addresses 🎀

Okay so here's where the authors live, and honestly the addresses are giving very professional-but-make-it-cute vibes 💖. Every line stays exactly as written, so if you ever need to reach them, you know where to look 📬.

Daniel O. Awduche Movaz Networks 7926 Jones Branch Drive, Suite 615 McLean, VA 22102

Phone: 703-298-5291 EMail: awduche@movaz.com

Angela Chiu Celion Networks 1 Sheila Dr., Suite 2 Tinton Falls, NJ 07724

Phone: 732-747-9987 EMail: angela.chiu@celion.com

Anwar Elwalid Lucent Technologies Murray Hill, NJ 07974

Phone: 908 582-7589 EMail: anwar@lucent.com

Indra Widjaja Bell Labs, Lucent Technologies 600 Mountain Avenue Murray Hill, NJ 07974

Phone: 908 582-0435 EMail: iwidjaja@research.bell-labs.com

XiPeng Xiao Redback Networks 300 Holger Way San Jose, CA 95134

Phone: 408-750-5217 EMail: xipeng@redback.com

14.0. Full Copyright Statement ✨

okay bestie, this is the legal bit at the end of the doc and it is NOT the fun part, but we gotta keep it all 🎀 the copyright belongs to The Internet Society (2002), and everything is reserved. 💅

So basically: you're allowed to copy and share this document and translations of it, and you can even make derivative works that comment on or explain it or help people implement it. 🦋 You can publish and distribute those in whole or in part, no restrictions of any kind, as long as the copyright notice and this paragraph stay on every copy and derivative work. 🫶

BUT the document itself can't be modified in any way. 😭 So no removing the copyright notice, and no scrubbing the references to the Internet Society or other Internet orgs. The only exceptions are if you need it to develop Internet standards (then you follow the copyright procedures in the Internet Standards process), or if you're translating it into a language other than English. 🌍

Also, the permissions granted above are perpetual and won't be revoked by the Internet Society or its successors or assigns. 💜

Last bit, the disclaimer: this document and the info in it is provided "AS IS" 🙈 and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED. That includes (but isn't limited to) any warranty that using the info won't infringe any rights, and any implied warranties of merchantability or fitness for a particular purpose. Not gonna lie, that's a lot of "no" energy. 💀

Acknowledgement ✨

Funding for the RFC Editor function is currently provided by the Internet Society. 🌸 So shoutout to them for keeping the lights on! 🔥

---