RFC 2260
Scalable Support for Multi-homed Multi-provider Connectivity
RFC 2260 Title Block ✨
Okay bestie, let's start with the vibe check 🎀. This is RFC 2260, titled Scalable Support for Multi-homed Multi-provider Connectivity 🦋. It was written by T. Bates and Y. Rekhter, both at Cisco Systems 💜. It's in the Informational category 🌸, so it's sharing info rather than being a standard. The date is January 1998 📅. The title block doesn't list anything it obsoletes or updates, so there's no drama there 💅.
Status of This Memo
This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited 🌈.
Copyright Notice
Copyright (C) The Internet Society (1998). All Rights Reserved 💖.
2. Abstract 🧚
This document describes addressing and routing strategies for multi-homed enterprises attached to multiple Internet Service Providers (ISPs) that are intended to reduce the routing overhead due to these enterprises in the global Internet routing system 🔥.
3. Motivations 🌸
Okay so, an enterprise might get its Internet connectivity from more than one Internet Service Provider (ISP), and there are a few reasons for that 🫶. Being connected through more than one ISP could be seen as a way to make Internet connectivity more reliable. That way, if the connection through one ISP fails, the connection through the other ISP(s) lets the enterprise keep its Internet access 🙌. On top of reliability, having more than one ISP could also let the enterprise spread its load across multiple connections. And for enterprises spread over a wide geographical area, this could enable better (more optimal) routing 🗺️✨.
Now here's the tea 🍵. Internet connectivity keeps getting cheaper, so more and more enterprises are becoming multi-homed to multiple ISPs. At the same time, the routing overhead these enterprises put on the Internet routing system keeps getting bigger and bigger 📈💀. Scaling the Internet and supporting a growing number of these enterprises needs some mechanism(s) to keep that overhead contained 🔒. This document assumes that an approach where routers in the "default-free" zone of the Internet would be required to maintain a route for every multi-homed enterprise connected to multiple ISPs doesn't scale well enough 🙅♀️. Also, given how the Internet works, this document assumes that any approach to handle routing for these enterprises should minimize the amount of coordination among ISPs, especially the ISPs that aren't directly connected to these enterprises 🤝.
There's a difference of opinion on whether the things driving multi-homing to multiple ISPs could be handled just by multi-homing to a single ISP, which would then get rid of the negative impact of multi-homing on the Internet routing system 🤷♀️. That discussion is beyond the scope of this document, so we're not going there today 🙈.
The focus of this document is on the routing and addressing strategies that could reduce the routing overhead caused by multi-homed enterprises connected to multiple ISPs in the Internet routing system 🎯.
The strategies described in this document are equally applicable to both IPv4 and IPv6 💜.
4. Address allocation and assignment 🎀
A multi-homed enterprise connected to a set of ISPs would be allocated a block of addresses (an address prefix) by each of these ISPs. So an enterprise connected to N ISPs would get N different blocks 🧩. The address allocation from the ISPs to the enterprise would be based on the "address-lending" policy [RFC2008] 📜. The allocated addresses would then be used for address assignment within the enterprise 🏢.
One possible address assignment plan the enterprise could use is to treat the topological proximity of a node (host) to a particular ISP (to the interconnect between the enterprise and the ISP) as a criterion for choosing which of the address prefixes to use when assigning an address to that node 🗺️. A particular node (host) may be assigned address(es) out of a single prefix, or it may have addresses from different prefixes 🦋.
5. Routing information exchange 🌈
The issue of routing information exchange between an enterprise and its ISPs breaks down into the following components 💅:
- a) reachability information that an enterprise border router advertises to a border router within an ISP
- b) reachability information that a border router within an ISP advertises to an enterprise border router
The primary focus of this document is on (a). (b) is only covered as needed by this document 🔍.
5.1. Advertising reachability information by enterprise border routers 🧚♀️
When an enterprise border router connected to a particular ISP determines that its connectivity between the enterprise and the Internet is up through all of its ISPs, the router advertises reachability to only the address prefix that the ISP allocated to the enterprise. It sends this to the border router of that ISP 💖. This way, in a steady state, the routes the enterprise injects into its ISPs get aggregated by those ISPs, and they don't get propagated into the "default-free" zone of the Internet 🌸✨.
When an enterprise border router connected to a particular ISP determines that its connectivity to the Internet through one or more of its other ISPs is down, the router starts advertising reachability to the address prefixes that those other ISPs allocated to the enterprise 😬. This would result in injecting additional routing information into the "default-free" zone of the Internet 📈. But here's the silver lining 🌤️: the probability of all multi-homed enterprises in the Internet losing connectivity through one or more of their ISPs at the same time is fairly small. So on average, the number of additional routes in the "default-free" zone due to multi-homed enterprises is expected to be a small fraction of the total number of such enterprises 🥰.
The approach described above depends on the assumption that an enterprise border router has a mechanism(s) for figuring out (a) whether connectivity to the Internet through some other border router of that enterprise is up or down, and (b) the address prefix that was allocated to the enterprise by the ISP connected to that other border router 🔑. One such possible mechanism could be provided by BGP [RFC1771] 🛠️. In this case, border routers within the enterprise would have an IBGP peering with each other 👯♀️. Whenever one border router determines that the intersection between the set of reachable destinations it receives via its EBGP peerings (from its directly connected ISP) and the set of reachable destinations it receives from another border router in the same enterprise via IBGP is empty, that border router starts advertising reachability to its external peer for the address prefix that was allocated to the enterprise by the ISP connected to the other border router 🚨. The other border router advertises (via IBGP) the address prefix that was allocated to the enterprise by the ISP connected to that router 📣. This approach is known as "auto route injection" 💜.
For an illustration, let's say an enterprise is connected to two ISPs, ISP-A and ISP-B 🎀. We'll call the enterprise border router that connects the enterprise to ISP-A BR-A, and the one that connects the enterprise to ISP-B BR-B 🏷️. We'll call the address prefix that ISP-A allocated to the enterprise Pref-A, and the one ISP-B allocated Pref-B 🧁. When the set of routes BR-A receives from ISP-A (via EBGP) has a non-empty intersection with the set of routes BR-A receives from BR-B (via IBGP), BR-A advertises only the reachability to Pref-A to ISP-A 🙌. When the intersection becomes empty, BR-A advertises reachability to both Pref-A and Pref-B to ISP-A 😱. This continues for as long as the intersection stays empty. Once the intersection becomes non-empty again, BR-A stops advertising reachability to Pref-B to ISP-A, but it still keeps advertising reachability to Pref-A to ISP-A 🥳. Figure 1 below shows this method visually 👀.
+-------+ +-------+ +-------+ +-------+
( ) ( ) ( ) ( )
( ISP-A ) ( ISP-B ) ( ISP-A ) ( ISP-B )
( ) ( ) ( ) ( )
+-------+ +-------+ +-------+ +-------+
| /\ | /\ | /\ |
| || | || | Pref-A (connection
| Pref-A | Pref-B | Pref-B broken)
| || | || | || |
+-----+ +-----+ +-----+ +-----+
| BR-A|------|BR-B | | BR-A|------|BR-B |
+-----+ IBGP +-----+ +-----+ IBGP +-----+
non-empty intersection empty intersection
Figure 1: Reachability information advertisedAlthough calculating the intersection is strictly an implementation detail, it could potentially be a costly operation for a large set of routes 🥵. An alternate solution is to use a selected single (or more) address prefix received from an ISP (the ISP's backbone route, for example), and to configure the enterprise border router to perform auto route injection if the selected prefix is not present via IBGP 🛡️. Let's suppose ISP-B has a well-known address prefix, ISP-Pref-B, for its backbone. ISP-B advertises this to BR-B, and BR-B in turn advertises it via IBGP to BR-A. If BR-A sees a withdraw for ISP-Pref-B, it advertises Pref-B to ISP-A 🧠.
The approach described in this section may produce less than full Internet-wide connectivity in the presence of ISPs that filter out routes based on the length of their address prefixes 🚧. One could observe, however, that this would be a problem regardless of how the enterprise sets up its routing and addressing 🤔.
5.2. Further improvements ✨
okay so the approach in the last section already cuts the routing overhead in the "default-free" zone a LOT for multi-homed enterprises 💅 but this next one? it completely eliminates that overhead 😭🫶 the way that it just wipes it out is so iconic
so here's the vibe. an enterprise border router would keep EBGP peering not only with the directly connected border router of an ISP, but also with the border router(s) in one or more ISPs whose border routers are directly connected to the other border routers inside the enterprise. we call that "non-direct" EBGP 🦋
an ISP that has both direct and non-direct EBGP peering with a particular enterprise advertises the same set of routes over both of those peerings 🌸 and an enterprise border router that has either direct or non-direct peering with an ISP advertises reachability to the address prefix that ISP allocated to the enterprise. so it's giving consistency, bestie 💖
now the preferences. inside the ISP, routes received over direct peering should be preferred over routes received over non-direct peering ✨ and inside the enterprise, routes received over direct peering should be preferred over routes received over non-direct peering too 👑 direct always wins, no debate
forwarding along a route learned over non-direct peering should be accomplished via encapsulation [RFC1773] 🎀
let's do an example because it'll make way more sense 🧚 say an enterprise is connected to two ISPs, ISP-A and ISP-B. the enterprise border router that connects to ISP-A is E-BR-A, and the ISP-A border router connected to E-BR-A is ISP-BR-A. on the other side, the enterprise border router connecting to ISP-B is E-BR-B, and the ISP-B border router connected to E-BR-B is ISP-BR-B. the prefix ISP-A allocated to the enterprise is Pref-A, and the prefix ISP-B allocated to the enterprise is Pref-B 🍓
so E-BR-A has direct EBGP peering with ISP-BR-A and advertises reachability to Pref-A over that peering. E-BR-A also keeps a non-direct EBGP peering with ISP-BR-B and advertises reachability to Pref-B over that one. meanwhile E-BR-B has direct EBGP peering with ISP-BR-B and advertises reachability to Pref-B there, AND it keeps a non-direct EBGP peering with ISP-BR-A and advertises reachability to Pref-A over that one 🌈 it's a whole web of peerings and honestly I'm obsessed
when both ISPs and the enterprise are fully connected, traffic for hosts numbered out of Pref-A flows through ISP-A to ISP-BR-A to E-BR-A, then into the enterprise ✨ and traffic for hosts numbered out of Pref-B flows through ISP-B to ISP-BR-B to E-BR-B, then into the enterprise 💜
now the plot twist 😭 what happens if the connection between ISP-BR-B and E-BR-B goes down? traffic for hosts in Pref-A is handled exactly as before 🥰 but traffic for hosts in Pref-B flows through ISP-B to ISP-BR-B, and then ISP-BR-B encapsulates it and sends it to E-BR-A. E-BR-A decapsulates it and sends it into the enterprise. no dropped packets, no drama, just a little detour 🦋 here's the picture:
+---------+ +---------+
( ) ( )
( ISP-A ) ( ISP-B )
( ) ( )
+---------+ +---------+
| |
+--------+ +--------+
|ISP-BR-A| |ISP-BR-B|
+--------+ +--------+
| /+/ |
/\ | Pref-B /+/ |
|| | /+/ \./
Pref-A| /+/ non- /.\
|| | /+/ direct |
| /+/ EBGP |
+------+ +-------+
|E-BR-A|-----------|E-BR-B |
+------+ IBGP +-------+
Figure 2: Reachability information advertised via non-direct EBGPObserve that with this scheme there is NO additional routing information from multi-homed enterprises that has to be carried in the "default-free" zone of the Internet 🙌 and it also doesn't degrade when ISPs filter out routes based on the length of their address prefixes. literally no extra routing state up there, periodt 💅
one more thing 🌟 the set of routers inside an ISP that keep non-direct peering with an enterprise's border routers doesn't have to be limited to the ISP's border routers that have direct peering with that enterprise's border routers. the non-direct peering could be kept up with any router in the ISP, and doing that could improve overall robustness when things fail inside the ISP 🛡️✨
5.3. Combining the two 🌸
okay so the approach in Section 5.2 completely eliminates the routing overhead that multi-homed enterprises would otherwise put into the "default-free" zone of the Internet 🎀 BUT it can result in suboptimal routing when links fail 🥺 so the fix is to combine it with a slightly modified version of the approach from Section 5.1 🫶
the modification is about constraining how far the additional routes go. when an enterprise border router detects problems with Internet connectivity through its other border routers, it advertises extra routes, and the scope of propagation for those routes gets constrained. one way to constrain the scope is by using the BGP Community attribute [RFC1997] 💜
5.4. Better (more optimal) routing in steady state 💖
this document assumes that, in a steady state, an enterprise border router advertises to a directly connected ISP border router only the reachability to the address prefix that this ISP allocated to the enterprise 🌷 the result? traffic from other enterprises connected to that same ISP, heading for the parts of the enterprise numbered out of other prefixes, wouldn't enter the enterprise at this border router, and the paths can end up being potentially suboptimal 😬
to fix that, the border router MAY (in steady state) advertise reachability not only to the prefix its own ISP allocated, but also to the prefixes allocated by some other ISPs (the ones directly connected to other border routers inside the enterprise). BUT distributing those advertisements should be carefully constrained, or else it could create significant additional routing information that has to be kept in the "default-free" part of the Internet 🙈 one way to constrain it is again by using the BGP Community attribute [RFC1997] 🧚
6. Comparison with other approaches 🍓
CIDR [RFC1518] proposes several possible address allocation strategies for multi-homed enterprises that are connected to multiple ISPs 🌈 so here's a quick tour of the alternatives people use today, and how they compare to the approaches above 👑
6.1. Solution 1 🦋
one solution suggested in [RFC1518] is for each multi-homed enterprise to get its IP address space independently from the ISPs it's attached to 🫶 this lets the enterprise base all of its IP assignments on a single prefix, so it can summarize every IP address reachable inside the enterprise under one prefix ✨
the catch? since the enterprise's address has no relationship to any particular ISP's addresses, the reachability information the enterprise advertises can't be aggregated with anything except a default route. that means the routing overhead in the "default-free" zone of the Internet is O(N), where N is the total number of multi-homed enterprises across the whole Internet that are connected to multiple ISPs 😭💀
so this approach can't really be a viable option for everyone, only for the enterprises that get a high enough degree of address aggregation 🙃 and since by definition there are probably not many of those, this approach isn't viable for most multi-homed enterprises connected to multiple ISPs 💅
6.2. Solution 2 🌸
another solution from [RFC1518] is to give each multi-homed enterprise a single address prefix based on one of its connections to one of its ISPs 🎀 the other ISPs the enterprise is attached to keep a routing table entry for the organization, but they're super selective about which other ISPs get told about that route, and they'd need to do "proxy" aggregation 🥺
most of the complexity here comes from needing to do that "proxy" aggregation, which in turn needs additional inter-ISP coordination and more complicated router configuration 😵💫 not the easiest vibe, honestly 💜
7. Discussion 💬
the approach in this document assumes that the addresses an enterprise uses are allocated under the "address lending" policy 🫶 which means that whenever an enterprise changes its ISP, it would need to renumber the part of its network that was numbered out of the address block the old ISP allocated 😬 BUT these issues aren't specific to multihoming, and they should be considered accepted practice on today's Internet 🌈 the approach in this document basically erases any difference between single-homed and multi-homed enterprises when it comes to the renumbering impact of changing ISPs 🦋✨
the approach also requires careful address assignment inside the enterprise, because address assignment affects how traffic gets distributed across the multiple connections between an enterprise and its ISPs 🍓
both address assignment and renumbering could be handled by using network address translation (NAT) appropriately 🧚 but using NAT for multi-homed enterprises is beyond the scope of this document, so we're leaving that one alone 💅
using auto route injection (as described in Section 5.1) does increase the number of routers in the "default-free" zone of the Internet that could be affected by changes in a multi-homed enterprise's connectivity, compared to using provider-independent addresses (as described in Section 6.1) 😭 specifically, with auto route injection, when a multi-homed enterprise loses its connectivity through one of its ISPs, the auto-injected route has to be propagated to ALL the routers in the "default-free" zone 🥺 with provider-independent addresses, only some (but not all) of the routers in that zone would see routing changes when the enterprise loses a connection 🌷
to suppress excessive routing load caused by link flapping, the auto-injected route has to stay advertised until the other connection (the one that went down and triggered the auto route injection) becomes stable again 🛡️💖
and finally, using the non-direct EBGP approach (as described in Section 5.2) eliminates route flapping caused by multi-homed enterprises in the "default-free" zone of the Internet 🙌 so non-direct EBGP has better routing stability properties than using provider-independent addresses (as described in Section 6.1) 👑🌈
8. Applications to multi-homed ISPs 🌸
okay so this bit is about small to medium ISPs that are hooked up to a few upstream ISPs at once (multi-homed, bestie 💅). The ISP gets blocks of addresses, aka address prefixes, from those upstreams and hands them out to its own customers. When it peers with its upstreams, it can use auto route injection, or the non-direct EBGP approach, or a mix of both. That gives the ISP's customers routability without messing up the overall scalability of the Internet routing system 🦋 no cap, that's the whole point 🙌
9. Security Considerations 🔐
⚠️ Security is serious business, so read this part carefully, bestie 🫶
okay so the non-direct EBGP approach (see Section 5.2) needs EBGP sessions between routers that are more than one IP hop away from each other. Routers that keep these sessions up should use an appropriate authentication mechanism(s) for BGP peer authentication 🔥
Security issues for IBGP peering, and for EBGP peering between routers that are one IP hop apart, are outside the scope of this document 💀 that's a different story for a different RFC, iykyk 💜
10. Acknowledgments 🎀
the authors aren't claiming they came up with these ideas first 🙈 if somebody thought about this stuff before, they deserve all the credit, periodt ✨
11. References 📚
[RFC1518]
Rekhter, Y., and T. Li, "An Architecture for IP Address
Allocation with CIDR", RFC 1518, September 1993.
[RFC1771]
Rekhter, Y., and T. Li, "A Border Gateway Protocol 4 (BGP-4)",
RFC 1771, March 1995.
[RFC1773]
Hanks, S., Li, T., Farinacci, T., and P. Traina, "Generic
Routing Encapsulation over IPv4 networks", RFC 1773, October
1994.
[RFC1918]
Rekhter, Y., Moskowitz, B., Karrenberg, D., de Groot G.J., and
E. Lear, "Address Allocation for Private Internets", RFC 1918,
February 1996.
[RFC1997]
Chandra, R., Traina, P., and T. Li, "BGP Communities Attribute",
RFC 1997, August 1996.
[RFC2008]
Rekhter, Y., and T. Li, "Implications of Various Address
Allocation Policies for Internet Routing", BCP 7, RFC 2008,
October 1996.12. Authors' Addresses 💌
the two authors, Tony Bates and Yakov Rekhter, both at Cisco Systems, Inc. 🌈 here are their deets, exactly as written:
Tony Bates
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134
EMail: tbates@cisco.com
Yakov Rekhter
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134
EMail: yakov@cisco.com13. Full Copyright Statement ⭐
okay the legal part, but it's still important so here it is in full 🧚 the copyright belongs to The Internet Society (1998), and the wording is kept exactly as it appears in the original:
Copyright (C) The Internet Society (1998). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.that's the whole doc, bestie 🌈💖 you're officially an RFC 2260 expert now 👑
📦 where this page comes from
- ietf: rfc2260 1998-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms