RFC 7424

Mechanisms for Optimizing Link Aggregation Group (LAG) and Equal-Cost Multipath (ECMP) Component Link Utilization in Networks

📅 published January 2015, from ietf

RFC 7424: Mechanisms for Optimizing Link Aggregation Group (LAG) and Equal-Cost Multipath (ECMP) Component Link Utilization in Networks ✨

Okay bestie, here's the vibe check 🎀 This one is Informational, not a standard, and it's dated January 2015. Its ISSN is 2070-1721. It's an IETF document, and it's RFC 7424 💖

The authors are R. Krishnan (Brocade Communications), L. Yong (Huawei USA), A. Ghanwani (Dell), N. So (Vinci Systems), and B. Khasnabish (ZTE Corporation) 🫶 The title block doesn't list anything it obsoletes or updates, so it doesn't replace or change any other RFC 🌸

Abstract

Okay so the internet is getting absolutely thirsty 😭 Rich media and data-center-to-data-center traffic are growing like crazy, and that's bringing bandwidth-hungry apps with it. Networks lean hard on link aggregation groups and equal-cost multipaths to scale their bandwidth, so using that bandwidth well is a huge deal. This doc explores some mechanisms that help with exactly that 🦋

Status of This Memo

This document is not an Internet Standards Track specification. It's published for informational purposes only 📚

It's a product of the Internet Engineering Task Force (IETF) and reflects the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every document the IESG approves is a candidate for any level of Internet Standard, so check Section 2 of RFC 5741 if you're curious 🧚

For the current status, any errata, and how to send feedback, head to http://www.rfc-editor.org/info/rfc7424 🌈

Copyright Notice

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, which you can find at http://trustee.ietf.org/license-info. Those rules are the ones in effect on the date it was published. Please read them carefully, since they spell out your rights and restrictions for this document 👀

Code components pulled out of this document have to include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions. They're provided without warranty, as described in the Simplified BSD License 🔥

   1. Introduction ....................................................4
      1.1. Acronyms ...................................................4
      1.2. Terminology ................................................5
   2. Flow Categorization .............................................6
   3. Hash-Based Load Distribution in LAG/ECMP ........................6
   4. Mechanisms for Optimizing LAG/ECMP Component Link Utilization ...8
      4.1. Differences in LAG vs. ECMP ................................9
      4.2. Operational Overview ......................................10
      4.3. Large Flow Recognition ....................................11
           4.3.1. Flow Identification ................................11
           4.3.2. Criteria and Techniques for Large Flow
                  Recognition ........................................12
           4.3.3. Sampling Techniques ................................12
           4.3.4. Inline Data Path Measurement .......................14
           4.3.5. Use of Multiple Methods for Large Flow
                  Recognition ........................................15
      4.4. Options for Load Rebalancing ..............................15
           4.4.1. Alternative Placement of Large Flows ...............15
           4.4.2. Redistributing Small Flows .........................16
           4.4.3. Component Link Protection Considerations ...........16
           4.4.4. Algorithms for Load Rebalancing ....................17
           4.4.5. Example of Load Rebalancing ........................17
   5. Information Model for Flow Rebalancing .........................18
      5.1. Configuration Parameters for Flow Rebalancing .............18
      5.2. System Configuration and Identification Parameters ........19
      5.3. Information for Alternative Placement of Large Flows ......20
      5.4. Information for Redistribution of Small Flows .............21
      5.5. Export of Flow Information ................................21
      5.6. Monitoring Information ....................................21
           5.6.1. Interface (Link) Utilization .......................21
           5.6.2. Other Monitoring Information .......................22
   6. Operational Considerations .....................................23
      6.1. Rebalancing Frequency .....................................23
      6.2. Handling Route Changes ....................................23
      6.3. Forwarding Resources ......................................23
   7. Security Considerations ........................................23
   8. References .....................................................24
      8.1. Normative References ......................................24
      8.2. Informative References ....................................25
   Appendix A.  Internet Traffic Analysis and Load-Balancing
                Simulation ...........................................28
   Acknowledgements ..................................................28
   Contributors ......................................................28
   Authors' Addresses ................................................29

1. Introduction ✨

Networks lean hard on link aggregation groups (LAGs) [802.1AX] and equal-cost multipaths (ECMPs) [RFC2991] to scale capacity 💅 For the problems this doc tackles, network traffic mostly falls into two types: long-lived large flows and everything else. The everything-else bucket includes long-lived small flows, short-lived small flows, and short-lived large flows, and the doc calls all of those "small flows" 🌸 Long-lived large flows are just called "large flows" 🦋

Stateless hash-based techniques [ITCOM] [RFC2991] [RFC2992] [RFC6790] are commonly used to spread both large and small flows across the component links in a LAG/ECMP. The catch is that traffic doesn't always split evenly across those component links, because of how the traffic behaves 😭

This document describes mechanisms for optimizing LAG/ECMP component link utilization when hash-based techniques are in use. The mechanisms have two steps: 1) recognize large flows in a router, and 2) when a component link on the router gets congested, either assign the large flows to specific LAG/ECMP component links or redistribute the small flows 🔥

Keep in mind that in typical use cases, large flows eat up a big chunk of a link's bandwidth, for example more than 5% of the link bandwidth 👑 That means there aren't many of them. You'd expect something on the order of 10s or 100s per LAG/ECMP. So the number of large flows is NOT expected to be on the order of millions 🙈 Examples would be IPsec tunnels in service provider backbone networks, or storage backup traffic in data center networks 💜

1.1. Acronyms

   DoS:    Denial of Service

   ECMP:   Equal-Cost Multipath

   GRE:    Generic Routing Encapsulation

   IPFIX:  IP Flow Information Export

   LAG:    Link Aggregation Group

   MPLS:   Multiprotocol Label Switching

   NVGRE:  Network Virtualization using Generic Routing Encapsulation
   PBR:    Policy-Based Routing

   QoS:    Quality of Service

   STT:    Stateless Transport Tunneling

   VXLAN:  Virtual eXtensible LAN

1.2. Terminology 🧚

Central management entity: This is an entity that can monitor link utilization and flow info across routers in the network. It may also be able to make traffic-engineering decisions about where large flows go. It may include the functions of a collector [RFC7011] 💖

ECMP component link: This is one individual next hop inside an ECMP group. An ECMP component link can itself be a LAG 🫶

ECMP table: This is a table used as the next hop of an ECMP route. It holds the set of ECMP component links plus a weight for each one. You look it up using the packet's hash value, and the weights decide which hash values map to which ECMP component link 🌈

Flow (large or small): This is a sequence of packets that should keep their ordered delivery, for example packets that belong to the same TCP connection 🎀

LAG component link: This is one individual link inside a LAG. Usually it's a physical link ✨

LAG table: This is a table used as the output port, which is a LAG. It holds the set of LAG component links plus a weight for each one. You look it up using the packet's hash value, and the weights decide which hash values map to which LAG component link 🦋

Large flow(s): This refers to long-lived large flow(s) 💅

Small flow(s): This refers to any one of, or any mix of, long-lived small flow(s), short-lived small flows, and short-lived large flow(s) 🌸

2. Flow Categorization 💖

Based on size and duration, a flow can fall into any one of these four types, as shown in Figure 1:

short-lived large flow (SLLF),
short-lived small flow (SLSF),
long-lived large flow (LLLF), and
long-lived small flow (LLSF).

     Flow Bandwidth
         ^
         |--------------------|--------------------|
         |                    |                    |
   Large |      SLLF          |       LLLF         |
   Flow  |                    |                    |
         |--------------------|--------------------|
         |                    |                    |
   Small |      SLSF          |       LLSF         |
   Flow  |                    |                    |
         +--------------------+--------------------+-->Flow Duration
              Short-Lived            Long-Lived
              Flow                   Flow

            Figure 1: Flow Categorization

Like we said earlier, in this doc we call long-lived large flows "large flows", and everything else (long-lived small flows, short-lived small flows, and short-lived large flows) gets called "small flows" 🎀✨

3. Hash-Based Load Distribution in LAG/ECMP ✨

okay so hash-based techniques are everywhere for load balancing 💖 when traffic needs to pick among several paths in a LAG/ECMP group, routers love to hash. the big perks are that packets in a flow stay in order 🎀 and the distribution happens in real time without the router keeping per-flow state. the way it works is that the router grabs a combo of fields from the packet headers to identify the flow, then runs a hash function over them to get a number that points to one link or path in the group. the catch is that it's a many-to-one mapping, so lots of flows can end up squished onto the same component link 🫠

hash-based techniques give good utilization across the individual component links when:

the traffic mix is made of flows where the hash function spreads out fairly evenly, so a similar number of flows lands on each component link,
the individual flow rates are much smaller than the link capacity, and
the differences in flow rates are not dramatic.

but if one or more of those conditions doesn't hold, hash-based techniques may leave the loads on individual component links imbalanced 😭

here's the example in Figure 2 🦋 we've got two routers, R1 and R2, with a LAG between them made of three component links, (1), (2), and (3). ten flows need to get spread across this LAG, and hashing does the following:

Component link (1) has three flows (two small flows and one large flow), and the link utilization is normal. ✨
Component link (2) has three flows (three small flows and no large flows), and the link utilization is light.
- no large flows means this link is underutilized 🥀
Component link (3) has four flows (two small flows and two large flows), and the link capacity is exceeded, which means congestion. 🚨
- two large flows on one link = congestion, bestie 💀

          +-----------+ ->     +-----------+
          |           | ->     |           |
          |           | ===>   |           |
          |        (1)|--------|(1)        |
          |           | ->     |           |
          |           | ->     |           |
          |   (R1)    | ->     |     (R2)  |
          |        (2)|--------|(2)        |
          |           | ->     |           |
          |           | ->     |           |
          |           | ===>   |           |
          |           | ===>   |           |
          |        (3)|--------|(3)        |
          |           |        |           |
          +-----------+        +-----------+

    Where: ->   small flow
           ===> large flow

        Figure 2: Unevenly Utilized Component Links

this document brings in mechanisms to fix that imbalance from commonly used hash-based techniques for LAG/ECMP, like the one shown in the example above 🌸 the mechanisms use large flow awareness to even things out again 👑

4. Mechanisms for Optimizing LAG/ECMP Component Link Utilization 💅

the mechanisms suggested in this document are local optimization solutions 🎀 "local" here means that both spotting the large flows and rebalancing the load can happen entirely inside individual routers, with no need to chat with other routers.

that approach may not give a global optimization for where large flows get placed across multiple routers in a network, which might be what some networks want 🤔 on the flip side, a local approach can be totally adequate in some environments, for these reasons:

Different links within a network run at different levels of utilization, so a "targeted" solution is needed for those hot spots. an example is a LAG between two routers that needs optimizing. 🔥
Some networks may lack end-to-end visibility, e.g., when a certain network, under the control of a given operator, is a transit network for traffic from other networks that aren't under the same operator's control. 🌈

4.1. Differences in LAG vs. ECMP

the mechanisms here work for both LAGs and ECMP groups, but there are some key differences between the two that can change how well they work 🦋 part of that comes from the localized info these mechanisms are meant to operate on.

a LAG is usually set up across links between two adjacent routers. so the problem of optimizing bandwidth on the component links is pretty narrow 🧚 it just means rebalancing the load across the component links between those two routers, and there's no impact whatsoever on other parts of the network. the scheme works equally well for unicast and multicast flows. ✨

ECMP is a different story 😬 redistributing load across component links in an ECMP group may affect traffic patterns at all the routers downstream of the given router, between it and the destination. the local optimization might cause congestion at a downstream node. (in its simplest form, an ECMP group may be used to spread traffic over component links between two adjacent routers, and then the ECMP group is no different than a LAG for this discussion. it's also worth noting that an ECMP component link may itself be a LAG, in which case the scheme may be applied further to the component links within that LAG.)

to show the limits of local optimization, check out the two-level Clos network in Figure 3 🌸 it has three leaf routers (L1, L2, and L3) and two spine routers (S1 and S2). assume all of the links are 10 Gbps.

let L1 have two flows of 4 Gbps each toward L3, and let L2 have one flow of 7 Gbps also toward L3. if L1 balances the load optimally between S1 and S2, and L2 sends its flow via S1, then the downlink from S1 to L3 would get congested, resulting in packet discards. 💀 but if L1 had sent both its flows toward S1 and L2 had sent its flow toward S2, there would have been no congestion at either S1 or S2. 🫶

                +-----+     +-----+
                | S1  |     | S2  |
                +-----+     +-----+
                 / \ \       / /\
                / +---------+ /  \
               / /  \  \     /    \
              / /    \  +------+   \
             / /      \    /    \   \
          +-----+    +-----+   +-----+
          | L1  |    | L2  |   | L3  |
          +-----+    +-----+   +-----+

          Figure 3: Two-Level Clos Network

the other issue with applying this scheme to ECMP groups is that it may not apply equally to unicast and multicast traffic, because of the way multicast trees are constructed. 🌲

finally, a single physical link can take part as a component link in multiple ECMP groups, whereas with LAGs, a link can be a component link of only one LAG. 🧩

4.2. Operational Overview

here's the rundown on the steps for optimizing LAG/ECMP component link utilization in networks 📝

Step 1: this step is about recognizing large flows in routers and keeping track of the mapping from each large flow to the component link it uses. recognition of large flows is explained in Section 4.3. 👀

Step 2: the egress component links get scanned periodically for link utilization, and the imbalance for the LAG/ECMP group gets monitored. if the imbalance goes over a certain threshold, rebalancing gets triggered. measuring the imbalance is discussed further in Section 5.1. on top of the imbalance, other criteria (such as the maximum utilization of any of the component links) may also be used to decide whether to trigger rebalancing. the use of sampling techniques to measure egress component link utilization, including the issues of depending on ingress sampling for these measurements, are discussed in Section 4.3.3. 📊

Step 3: as part of rebalancing, the operator can choose to move the large flows onto lightly loaded component links of the LAG/ECMP group, redistribute the small flows on the congested link to other component links of the group, or do a combination of both. 🔄

all of the steps above can be done locally within the router itself, or they could involve a central management entity. 🖥️

giving large flow info to a central management entity gives the capability to globally optimize flow distribution, as described in Section 4.1. ooh, here's an example 🍓 a router may have three ECMP next hops that lead down paths P1, P2, and P3. a couple of hops downstream on path P1, there may be a congested link, while paths P2 and P3 may be underutilized. the local router can't see that, because it has no visibility there. with the help of a central management entity, the operator could redistribute some flows from P1 to P2 and/or P3, resulting in a more optimized flow of traffic. 🧚‍♀️✨

these steps are especially useful when bundling links of different bandwidths, e.g., 10 Gbps and 100 Gbps as described in [RFC7226]. 💜

4.3. Large Flow Recognition

4.3.1. Flow Identification

flows are typically identified using one or more fields from the packet header, for example: 🔍

Layer 2: Source Media Access Control (MAC) address, destination MAC address, VLAN ID.
IP header: IP protocol, IP source address, IP destination address, flow label (IPv6 only).
Transport protocol header: Source port number, destination port number. These apply to protocols such as TCP, UDP, and the Stream Control Transmission Protocol (SCTP).
MPLS labels.

for tunneling protocols like Generic Routing Encapsulation (GRE) [RFC2784], Virtual eXtensible LAN (VXLAN) [RFC7348], Network Virtualization using Generic Routing Encapsulation (NVGRE) [NVGRE], Stateless Transport Tunneling (STT) [STT], Layer 2 Tunneling Protocol (L2TP) [RFC3931], etc., flow identification is possible based on inner and/or outer headers, as well as fields introduced by the tunnel header, since any or all of those fields may be used for load balancing decisions [RFC5640]. 🎀

the list above is not exhaustive. 🌈

the mechanisms described in this document are agnostic to the fields used for flow identification. 🫶

this method of flow identification is consistent with that of IPFIX [RFC7011]. 🦋

4.3.2. Criteria and Techniques for Large Flow Recognition ✨

Okay so, bestie, to decide what counts as a large flow 🦋, we set up an observation interval and measure how much bandwidth the flow uses during that window. If a flow goes over a certain minimum bandwidth threshold during that interval, it gets labelled a large flow 💅.

Both of those knobs, the observation interval and the minimum bandwidth threshold, should be programmable 🎀. That way you can tune them for different use cases and different kinds of traffic. For example, a flow sitting at or above 10% of link bandwidth for at least one second could be declared a large flow [DEVOFLOW] ✨.

Now here's the lowkey clever part 🧠. To avoid excessive churn when rebalancing, a flow that has already been recognized as large should stay recognized as large for as long as the traffic it gets during an observation interval stays above some fraction of the bandwidth threshold, for example, 80% of the bandwidth threshold 🫶. No flip-flopping!

There are several techniques for spotting large flows, and they're covered in Sections 4.3.3, 4.3.4, and 4.3.5 💖.

4.3.3. Sampling Techniques 🌸

A bunch of routers support sampling methods like sFlow [sFlow-v5] [sFlow-LAG], Packet Sampling (PSAMP) [RFC5475], and NetFlow Sampling [RFC3954] 📊. For large flow recognition, sampling has to be turned on at all of the egress ports on the router where you want those measurements 🔌.

Taking sFlow as the example, a sFlow collector can give you an approximate picture of which large flows are mapped to each component link in each LAG/ECMP group 👀. If there are enough control plane resources available, that collector function can run in the router's control plane instead, so you lean less on a central management entity 🧚.

If egress sampling isn't available, ingress sampling can be good enough 💅. The central management entity that runs the sampling technique usually sees across multiple routers in the network, so it can use samples from an immediately downstream router to measure the egress traffic at the local router 🦋.

Heads up: ingress sampling might not be an option if the downstream router belongs to a different operator, or if that downstream device doesn't support sampling at all 💀.

Another angle: sampling techniques need each sample to be annotated with the packet's egress port information, so ingress sampling may also do the job 🍓. The catch is that sampling then has to be enabled on all ports, not just the ports where you actually want monitoring 😭. And there's one situation where this approach breaks down. If the given router has tunnels originating from it, and the resulting tunnel is the large flow, ingress sampling at that router can't reveal that 🙈. For this case, if egress sampling is unavailable, ingress sampling from the downstream router has to be used instead 🫶.

Let's look at Figure 2 to see ingress versus egress sampling in action 🎀. Since we're rebalancing flows at R1, R1 needs egress sampling on ports (1), (2), and (3). If egress sampling isn't available and R2 is under the same administrator's control, enabling ingress sampling on R2's ports (1), (2), and (3) would also work 💜. The downside is that R1 would then need a central management entity to get the large flow information for each of its links 🌈. Last option: R1 can enable ingress sampling on all of its ports (not only the ports in the LAG/ECMP group being monitored), and that's enough as long as the sampling technique tags the samples with egress port information ✨.

Here are the pros and cons of sampling, bestie 👑:

Advantages:

Supported in most existing routers 🌸.
Requires minimal router resources 💅.

Disadvantage:

To keep the error from sampling under control, there's a minimum delay before large flows are recognized, and also in the time it takes to react to that information 😭.

With sampling, large flows can be detected on the order of one second [DEVOFLOW] ⏱️. If you want to figure out the right sampling frequency, check out [SAMP-BASIC] 📚.

4.3.4. Inline Data Path Measurement 🔥

Implementations are allowed to recognize large flows by measuring traffic right in the router's data path 💅. This is expected to run at the interface speed on every interface, and it accounts for every packet the router's data path processes 🦋. IPFIX [RFC5470] is an example of an approach like this 🌸.

With inline data path measurement, you may get a faster and more accurate picture of which large flows are mapped to each component link in a LAG/ECMP group, compared to the sampling-based approach 🚀.

Here's the pros and cons, iykyk 🙈:

Advantages:

As link speeds climb, sampling rates usually get dialed down to keep the number of samples manageable, and that puts a lower bound on detection time 😩. With inline data path measurement, large flows can be recognized in shorter windows on higher link speeds, since every packet is counted [NDTM] 💖.
Inline data path measurement removes the potential dependence on a central management entity for large flow recognition 🧚.

Disadvantage:

Inline data path measurement uses more resources, specifically in the table sizes needed to monitor all flows 💀.

Like we said earlier, the observation interval and the bandwidth threshold for classifying a large flow should be programmable parameters in a router 🎀.

How inline data path measurement of large flows is implemented is vendor dependent, and that's beyond the scope of this document 🤷‍♀️.

4.3.5. Use of Multiple Methods for Large Flow Recognition 🌈

A router might have line cards that support a sampling technique, while other line cards support inline data path measurement 🫶. As long as the router can reliably work out the mapping of large flows to component links of a LAG/ECMP group, it's fine for it to use more than one method for large flow recognition ✨.

If both methods are supported, inline data path measurement may be the better pick because it detects faster [FLOW-ACC] 🔥.

4.4. Options for Load Rebalancing ⚖️

The subsections that follow describe suggested techniques for load balancing 🎀. Equipment vendors may implement more than one technique, including ones not described in this document, and let the operator choose between them 💅.

Heads up, though: regardless of the method, perfect rebalancing of large flows may not be possible, because flows arrive and leave at different times ⏰. Also, any flow that gets moved from one component link to another may experience momentary packet reordering 😬.

4.4.1. Alternative Placement of Large Flows 🦋

Within a LAG/ECMP group, the member component links with the least average link utilization get identified 👀. Then some large flow(s) from the heavily loaded component links are moved onto those lightly loaded member component links, using a PBR rule in the ingress processing element(s) of the routers 🧚.

With this approach, only certain large flows experience momentary flow reordering 💖.

Here's the catch, bestie 😭. Moving a large flow increases the utilization of the link it lands on, which can tip things into imbalance all over again 🙈. So when you move a large flow, you have to account for both the existing load and the future load after the move 🫶. And the appearance of new large flows may force a rearrangement of where existing flows are placed 🌪️.

Here's an example 🍓. Say there's a LAG made up of four 10 Gbps component links, and four large flows, each at 1 Gbps. Each flow gets placed on one of the component links 💅. Then a fifth large flow at 2 Gbps is recognized. To keep the load distribution fair, one of the existing 1 Gbps flows may need to move to a different component link 🔄. Even so, that still leaves some imbalance in utilization across the component links 😩.

4.4.2. Redistributing Small Flows 🌸

Sometimes large flows can eat up the entire bandwidth of a component link 🔥. In that case, it'd be ideal for the small flows to stay off the congested component link(s) 🙏.

The LAG/ECMP table gets modified to include only the non-congested component link(s). Small flows hash into this table to get mapped to a destination component link. Alternatively, if certain component links are heavily loaded but not congested, the output of the hash function can be adjusted to account for the large flow loading on each of the component links 🧚.
The PBR rules for large flows (refer to Section 4.4.1) must have strict precedence over the LAG/ECMP table lookup result 👑.

This method works on some existing router hardware 💻. The whole idea is to prevent, or at least lower the probability of, a small flow hashing into the congested component link(s) 🦋.

Just so you know, with this approach the small flows that get moved would be subject to reordering 😬.

4.4.3. Component Link Protection Considerations 🛡️

If you want, certain component links can be reserved for link protection 💜. These reserved component links aren't used for any flows when there are no failures 🌙. When one or more component links fail, all the flows on the failed component link(s) get moved onto the reserved component link(s) 🚑. The mapping table of large flows to component links just swaps the failed component link for the reserved one 🔁. The LAG/ECMP table does the same swap, replacing the failed component link with the reserved component link ✨.

4.4.4. Algorithms for Load Rebalancing 🧮

Specific algorithms for placing large flows are out of the scope of this document 🤷‍♀️. One possibility is to frame large flow placement as the well-known bin-packing problem, and use the various heuristics that exist for that problem [BIN-PACK] 🎀.

4.4.5. Example of Load Rebalancing ✨

Okay so bestie, let's see the rebalancing in action 🎀 Optimizing LAG/ECMP component utilization for the use case in Figure 2 is shown below in Figure 4. The large flow rebalancing explained in Section 4.4.1 is what does the work here 💅 The improved link utilization looks like this:

- Component link (1) has three flows (two small flows and one large flow), and the link utilization is normal.

- Component link (2) has four flows (three small flows and one large flow), and the link utilization is normal now.

- Component link (3) has three flows (two small flows and one large flow), and the link utilization is normal now.

            +-----------+ ->     +-----------+
            |           | ->     |           |
            |           | ===>   |           |
            |        (1)|--------|(1)        |
            |           |        |           |
            |           | ===>   |           |
            |           | ->     |           |
            |           | ->     |           |
            |   (R1)    | ->     |     (R2)  |
            |        (2)|--------|(2)        |
            |           |        |           |
            |           | ->     |           |
            |           | ->     |           |
            |           | ===>   |           |
            |        (3)|--------|(3)        |
            |           |        |           |
            +-----------+        +-----------+

      Where: ->   small flow
             ===> large flow

          Figure 4: Evenly Utilized Composite Links

Basically 🦋 the mechanisms from Section 4.4.1 did a glow-up on the traffic. One of the large flows on component link (3) was previously congested, so it got moved over to component link (2), which was previously underutilized. Iconic rebalancing, honestly 😭✨

5. Information Model for Flow Rebalancing 🧚

Okay, so for a router to do flow rebalancing based on an external system, the two need to talk to each other 💬 This section gives an example information model covering the different pieces needed for that. It's informational only, so think of it as a guide if you're building a data model 🌸

5.1. Configuration Parameters for Flow Rebalancing 🎀

The following parameters are required to configure this feature:

Large flow recognition parameters:
- Observation interval: The observation interval is the time period in seconds over which packet arrivals are observed for the purpose of large flow recognition. ⏱️
- Minimum bandwidth threshold: The minimum bandwidth threshold is configured as a percentage of link speed and translated into a number of bytes over the observation interval. A flow for which the number of bytes received over a given observation interval exceeds this number is recognized as a large flow 🔥
- Minimum bandwidth threshold for large flow maintenance: This threshold provides hysteresis for large flow recognition. Once a flow is recognized as a large flow, it keeps being recognized as one until it falls below this threshold. It is also configured as a percentage of link speed and is typically lower than the minimum bandwidth threshold defined above. Lowkey clever 🫶
Imbalance threshold: A measure of the deviation of the component link utilizations from the utilization of the overall LAG/ECMP group. Since component links can be different speeds, the imbalance can be computed as follows. Let the utilization of each component link in a LAG/ECMP group with n links of speed b_1, b_2 .. b_n be u_1, u_2 .. u_n. The mean utilization is computed as:

u_ave = [ (u_1 * b_1) + (u_2 * b_2) + .. + (u_n * b_n) ] /
        [b_1 + b_2 + .. + b_n].

The imbalance is then computed as:

max_{i=1..n} | u_i - u_ave |.

Rebalancing interval: The minimum amount of time between rebalancing events. This parameter makes sure rebalancing isn't invoked too frequently, since it impacts packet ordering 🙈

These parameters may be configured on a system-wide basis or may apply to an individual LAG/ECMP group. They may be applied to an ECMP group, provided that the component links are not shared with any other ECMP group. 💜

5.2. System Configuration and Identification Parameters 🍓

The following parameters are useful for router configuration and operation when using the mechanisms in this document.

IP address: The IP address of a specific router that the feature is being configured on or that the large flow placement is being applied to.
LAG ID: Identifies the LAG on a given router. The LAG ID may be required when configuring this feature (to apply a specific set of large flow identification parameters to the LAG) and will be required when specifying flow placement to achieve the desired rebalancing.
Component Link ID: Identifies the component link within a LAG or ECMP group. This is required when specifying flow placement to achieve the desired rebalancing.
Component Link Weight: The relative weight to be applied to traffic for a given component link when using hash-based techniques for load distribution.
ECMP group: Identifies a particular ECMP group. The ECMP group may be required when configuring this feature (to apply a specific set of large flow identification parameters to the ECMP group) and will be required when specifying flow placement to achieve the desired rebalancing. We note that multiple ECMP groups can share an overlapping set (or non-overlapping subset) of component links. This document does not deal with the complexity of addressing such configurations. 🧩

The feature may be configured globally for all LAGs and/or for all ECMP groups, or it may be configured specifically for a given LAG or ECMP group.

5.3. Information for Alternative Placement of Large Flows 🦄

In cases where large flow recognition is handled by a central management entity (see Section 4.3.3), an information model for flows is required to allow the import of large flow information to the router.

Typical fields used for identifying large flows were discussed in Section 4.3.1. The IPFIX information model [RFC7012] can be leveraged for large flow identification.

Large flow placement is achieved by specifying the relevant flow information along with the following:

For LAG: router's IP address, LAG ID, LAG component link ID.
For ECMP: router's IP address, ECMP group, ECMP component link ID.

In the case where the ECMP component link itself comprises a LAG, we would have to specify the parameters for both the ECMP group as well as the LAG to which the large flow is being directed. 🌈

5.4. Information for Redistribution of Small Flows 🌸

Redistribution of small flows is done using the following:

For LAG: The LAG ID and the component link IDs along with the relative weight of traffic to be assigned to each component link ID are required.
For ECMP: The ECMP group and the ECMP next hop along with the relative weight of traffic to be assigned to each ECMP next hop are required.

It is possible to have an ECMP next hop that itself comprises a LAG. In that case, we would have to specify the new weights for both the ECMP component links and the LAG component links.

In the case where an ECMP component link itself comprises a LAG, we would have to specify new weights for both the component links within the ECMP group as well as the component links within the LAG. 🍭

5.5. Export of Flow Information 📤

Exporting large flow information is required when large flow recognition is being done on a router but the decision to rebalance is being made in a central management entity. Large flow information includes flow identification and the component link ID that the flow is currently assigned to. Other information such as flow QoS and bandwidth may be exported too.

The IPFIX information model [RFC7012] can be leveraged for large flow identification.

5.6. Monitoring Information 📊

5.6.1. Interface (Link) Utilization 💖

The incoming bytes (ifInOctets), outgoing bytes (ifOutOctets), and interface speed (ifSpeed) can be obtained, for example, from the Interfaces table (ifTable) in the MIB module defined in [RFC1213].

The link utilization can then be computed as follows:

Incoming link utilization = (delta_ifInOctets * 8) / (ifSpeed * T)

Outgoing link utilization = (delta_ifOutOctets * 8) / (ifSpeed * T)

Where T is the interval over which the utilization is being measured, delta_ifInOctets is the change in ifInOctets over that interval, and delta_ifOutOctets is the change in ifOutOctets over that interval.

For high-speed Ethernet links, the etherStatsHighCapacityTable in the MIB module defined in [RFC3273] can be used.

Similar results may be achieved using the corresponding objects of other interface management data models such as YANG [RFC7223] if those are used instead of MIBs.

For scalability, it is recommended to use the counter push mechanism in [sFlow-v5] for the interface counters. Doing so would help avoid counter polling through the MIB interface.

The outgoing link utilization of the component links within a LAG/ECMP group can be used to compute the imbalance (see Section 5.1) for the LAG/ECMP group.

5.6.2. Other Monitoring Information 🔍

Additional monitoring information that is useful includes:

Number of times rebalancing was done.
Time since the last rebalancing event.
The number of large flows currently rebalanced by the scheme.
A list of the large flows that have been rebalanced, including:
- the rate of each large flow at the time of the last rebalancing for that flow,
- the time that rebalancing was last performed for the given large flow, and
- the interfaces that the large flows was (re)directed to.
The settings for the weights of the interfaces within a LAG/ECMP group used by the small flows that depend on hashing.

6. Operational Considerations 🧠

6.1. Rebalancing Frequency 🎀

Okay, the big rule here: flows should be rebalanced only when the imbalance in utilization across component links exceeds a certain threshold 💅 Rebalancing over and over to get perfectly equal utilization could be counterproductive, because it may move flows back and forth between component links, which messes with packet ordering and system stability 😭 This applies whether large flows or small flows are being redistributed. It's worth noting that reordering is a concern for TCP flows even with just a few packets, because three out-of-order packets would trigger enough duplicate ACKs to the sender to cause a retransmission [RFC5681]. Not me getting anxious over three packets 💀

The operator would have to experiment with different values for the large flow recognition parameters (minimum bandwidth threshold, minimum bandwidth threshold for large flow maintenance, and observation interval) and for the imbalance threshold across component links, so the solution gets tuned for their environment 🌟

6.2. Handling Route Changes 🛣️

okay bestie, this one is a big-brain moment 🧠✨ large flow rebalancing has to keep an eye on the Forwarding Information Base (FIB) 👀 if the next hop of a route stops pointing at the LAG or an ECMP group, then any PBR entries added as described in Sections 4.4.1 and 4.4.2 must be withdrawn, or you'll create forwarding loops 🌀 and that is NOT the vibe 💀

6.3. Forwarding Resources 💜

so the hash-based load balancing you get with LAG/ECMP is usually stateless 🫶 the mechanisms in this document need extra forwarding-plane resources on the routers so they can create PBR rules that override the hash-based forwarding decision 🔧 those resources may limit how many flows can get rebalanced, and they may also add latency, because packets need the additional lookups that are required 🐢

7. Security Considerations 🔐

this document does not directly impact the security of the Internet infrastructure or its applications 🌸 honestly it could even help 🦋 if there's a DoS attack pattern that causes a hash imbalance and heavily overloads large flows onto certain LAG/ECMP component links, this approach can help push things back into balance

⚠️ okay now the serious part, bestie. an attacker who knows the large flow recognition algorithm and any stateless distribution method can generate flows that get distributed in a way that overloads one specific path 😱 this could be used to cause the creation of PBR rules that exhaust the available PBR rule capacity on routers in the network. if those PBR rules are then discarded, the result could be congestion on the attacker-selected path. or, tracking large numbers of PBR rules could cause performance degradation. so yes, this one is dramatic, and we take it seriously 🫶

8. References 📚

8.1. Normative References

[802.1AX]    IEEE, "IEEE Standard for Local and metropolitan area
             networks - Link Aggregation", IEEE Std 802.1AX-2008,
             2008.

[RFC2991]    Thaler, D. and C. Hopps, "Multipath Issues in Unicast
             and Multicast Next-Hop Selection", RFC 2991, November
             2000, <http://www.rfc-editor.org/info/rfc2991>.

[RFC7011]    Claise, B., Ed., Trammell, B., Ed., and P. Aitken,
             "Specification of the IP Flow Information Export (IPFIX)
             Protocol for the Exchange of Flow Information", STD 77,
             RFC 7011, September 2013,
             <http://www.rfc-editor.org/info/rfc7011>.

[RFC7012]    Claise, B., Ed., and B. Trammell, Ed., "Information
             Model for IP Flow Information Export (IPFIX)", RFC 7012,
             September 2013,
             <http://www.rfc-editor.org/info/rfc7012>.

8.2. Informative References

[BIN-PACK]   Coffman, Jr., E., Garey, M., and D. Johnson.
             "Approximation Algorithms for Bin-Packing -- An Updated
             Survey" (in "Algorithm Design for Computer System
             Design"), Springer, 1984.

[CAIDA]      "Caida Traffic Analysis Research",
             <http://www.caida.org/research/traffic-analysis/>.

[DEVOFLOW]   Mogul, J., Tourrilhes, J., Yalagandula, P., Sharma, P.,
             Curtis, R., and S. Banerjee, "DevoFlow: Cost-Effective
             Flow Management for High Performance Enterprise
             Networks", Proceedings of the ACM SIGCOMM, 2010.

[FLOW-ACC]   Zseby, T., Hirsch, T., and B. Claise, "Packet Sampling
             for Flow Accounting: Challenges and Limitations",
             Proceedings of the 9th international Passive and Active
             Measurement Conference, 2008.

[ITCOM]      Jo, J., Kim, Y., Chao, H., and F. Merat, "Internet
             traffic load balancing using dynamic hashing with flow
             volume", SPIE ITCOM, 2002.

[NDTM]       Estan, C. and G. Varghese, "New Directions in Traffic
             Measurement and Accounting", Proceedings of ACM SIGCOMM,
             August 2002.

[NVGRE]      Garg, P. and Y. Wang, "NVGRE: Network Virtualization
             using Generic Routing Encapsulation", Work in Progress,
             draft-sridharan-virtualization-nvgre-07, November 2014.

[RFC2784]    Farinacci, D., Li, T., Hanks, S., Meyer, D., and P.
             Traina, "Generic Routing Encapsulation (GRE)", RFC 2784,
             March 2000, <http://www.rfc-editor.org/info/rfc2784>.

[RFC6790]    Kompella, K., Drake, J., Amante, S., Henderickx, W., and
             L. Yong, "The Use of Entropy Labels in MPLS Forwarding",
             RFC 6790, November 2012,
             <http://www.rfc-editor.org/info/rfc6790>.

[RFC1213]    McCloghrie, K. and M. Rose, "Management Information Base
             for Network Management of TCP/IP-based internets:
             MIB-II", STD 17, RFC 1213, March 1991,
             <http://www.rfc-editor.org/info/rfc1213>.

[RFC2992]    Hopps, C., "Analysis of an Equal-Cost Multi-Path
             Algorithm", RFC 2992, November 2000,
             <http://www.rfc-editor.org/info/rfc2992>.

[RFC3273]    Waldbusser, S., "Remote Network Monitoring Management
             Information Base for High Capacity Networks", RFC 3273,
             July 2002, <http://www.rfc-editor.org/info/rfc3273>.

[RFC3931]    Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed.,
             "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", RFC
             3931, March 2005,
             <http://www.rfc-editor.org/info/rfc3931>.

[RFC3954]    Claise, B., Ed., "Cisco Systems NetFlow Services Export
             Version 9", RFC 3954, October 2004,
             <http://www.rfc-editor.org/info/rfc3954>.

[RFC5470]    Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek,
             "Architecture for IP Flow Information Export", RFC 5470,
             March 2009, <http://www.rfc-editor.org/info/rfc5470>.

[RFC5475]    Zseby, T., Molina, M., Duffield, N., Niccolini, S., and
             F. Raspall, "Sampling and Filtering Techniques for IP
             Packet Selection", RFC 5475, March 2009,
             <http://www.rfc-editor.org/info/rfc5475>.

[RFC5640]    Filsfils, C., Mohapatra, P., and C. Pignataro, "Load-
             Balancing for Mesh Softwires", RFC 5640, August 2009,
             <http://www.rfc-editor.org/info/rfc5640>.

[RFC5681]    Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
             Control", RFC 5681, September 2009,
             <http://www.rfc-editor.org/info/rfc5681>.

[RFC7223]    Bjorklund, M., "A YANG Data Model for Interface
             Management", RFC 7223, May 2014,
             <http://www.rfc-editor.org/info/rfc7223>.

[RFC7226]    Villamizar, C., Ed., McDysan, D., Ed., Ning, S., Malis,
             A., and L. Yong, "Requirements for Advanced Multipath in
             MPLS Networks", RFC 7226, May 2014,
             <http://www.rfc-editor.org/info/rfc7226>.

[SAMP-BASIC] Phaal, P. and S. Panchen, "Packet Sampling Basics",
             <http://www.sflow.org/packetSamplingBasics/>.

[sFlow-v5]   Phaal, P. and M. Lavine, "sFlow version 5", July 2004,
             <http://www.sflow.org/sflow_version_5.txt>.

[sFlow-LAG]  Phaal, P. and A. Ghanwani, "sFlow LAG Counters
             Structure", September 2012,
             <http://www.sflow.org/sflow_lag.txt>.

[STT]        Davie, B., Ed., and J. Gross, "A Stateless Transport
             Tunneling Protocol for Network Virtualization (STT)",
             Work in Progress, draft-davie-stt-06, April 2014.

[RFC7348]    Mahalingam, M., Dutt, D., Duda, K., Agarwal, P.,
             Kreeger, L., Sridhar, T., Bursell, M., and C. Wright,
             "Virtual eXtensible Local Area Network (VXLAN): A
             Framework for Overlaying Virtualized Layer 2 Networks
             over Layer 3 Networks", RFC 7348, August 2014,
             <http://www.rfc-editor.org/info/rfc7348>.

[YONG]       Yong, L. and P. Yang, "Enhanced ECMP and Large Flow
             Aware Transport", Work in Progress,
             draft-yong-pwe3-enhance-ecmp-lfat-01, March 2010.

Appendix A. Internet Traffic Analysis and Load-Balancing Simulation 📊

okay so internet traffic [CAIDA] has been analyzed to get flow stats like the number of packets in a flow and how long the flow lasts ⏱️ flows get identified by the 5-tuple in the packet header (IP source address, IP destination address, transport protocol source port number, transport protocol destination port number, and IP protocol) 🎀 and the analysis shows that < ~2% of the flows take ~30% of total traffic volume, while the rest of the flows (> ~98%) contribute ~70% [YONG] 🤯

the simulation showed that, with Internet traffic patterns, the hash-based technique does not evenly spread flows over ECMP paths 😭 some paths may be > 90% loaded while others are < 40% loaded. the more ECMP paths you have, the messier the load distribution gets, and this means hash-based distribution can make some paths congested while others are underused [YONG] 💀

but the good news 🌈 the simulation also shows a big improvement using the large flow-aware, hash-based distribution technique described in this document. with the same simulated traffic, the improved rebalancing can achieve < 10% load differences among the paths. it proves that large flow-aware, hash-based distribution can effectively make up for the uneven load balancing caused by hashing and the traffic characteristics [YONG] 👑

Acknowledgements 🫶

the authors would like to thank the following individuals for their review and valuable feedback on earlier versions of this document: Shane Amante, Fred Baker, Michael Bugenhagen, Zhen Cao, Brian Carpenter, Benoit Claise, Michael Fargano, Wes George, Sriganesh Kini, Roman Krzanowski, Andrew Malis, Dave McDysan, Pete Moyer, Peter Phaal, Dan Romascanu, Curtis Villamizar, Jianrong Wong, George Yum, and Weifeng Zhang. as a part of the IETF Last Call process, valuable comments were received from Martin Thomson and Carlos Pignataro 💖

Contributors 🎀

Sanjay Khanna
Cisco Systems
EMail: sanjakha@gmail.com

Authors' Addresses 💌

Ram Krishnan
Brocade Communications
San Jose, CA 95134
United States
Phone: +1-408-406-7890
EMail: ramkri123@gmail.com


Lucy Yong
Huawei USA
5340 Legacy Drive
Plano, TX 75025
United States
Phone: +1-469-277-5837
EMail: lucy.yong@huawei.com


Anoop Ghanwani
Dell
5450 Great America Pkwy
Santa Clara, CA 95054
United States
Phone: +1-408-571-3228
EMail: anoop@alumni.duke.edu


Ning So
Vinci Systems
2613 Fairbourne Cir
Plano, TX 75093
United States
EMail: ningso@yahoo.com


Bhumip Khasnabish
ZTE Corporation
New Jersey 07960
United States
Phone: +1-781-752-8003
EMail: vumip1@gmail.com

📦 where this page comes from

ietf: rfc7424 2015-01
💖 girlified by claude-haiku-5-5 on 2026-10-18: a language model's rewrite of the original, under its terms