RFC 6179
The Internet Routing Overlay Network (IRON)
The Internet Routing Overlay Network (IRON) 🦋
Okay bestie, let's get into RFC 6179! 🫶 First, the title block. It's from the Internet Research Task Force (IRTF), and the editor is F. Templin, Ed., from Boeing Research & Technology. The category is Experimental 🧪, it's dated March 2011, and the ISSN is 2070-1721. The title block doesn't list any obsoletes or updates, so there's nothing else to chase up here 💅.
Abstract 💖
Okay so the Internet has to keep handling more and more demand, which means our current routing architectures and operational practices need an upgrade 📈. This document proposes an Internet Routing Overlay Network (IRON) that supports sustainable growth without needing any changes to end systems or to the existing routing system 🌸. IRON also tackles other big stuff: routing scaling, mobility management, multihoming, traffic engineering, and NAT traversal ✨. Business considerations matter a lot for whether people actually adopt it, but they're out of scope for this document 💼. And it's a product of the IRTF Routing Research Group 🎀.
Status of This Memo 📝
This one isn't an Internet Standards Track specification, babe 🙈. It's published so people can examine it, try out experimental implementations, and evaluate it.
This document defines an Experimental Protocol for the Internet community 🧪. It's a product of the Internet Research Task Force (IRTF), which publishes the results of Internet-related research and development. Those results might not be suitable for deployment, so be careful 💀. This RFC is the individual opinion of one or more members of the Internet Research Task Force (IRTF) Research Group of the Internet Research Task Force (IRTF). Documents approved for publication by the IRSG are not a candidate for any level of Internet Standard; see Section 2 of RFC 5741 🙅♀️.
Info about the current status of this document, any errata, and how to give feedback is at http://www.rfc-editor.org/info/rfc6179 🔗.
Copyright Notice ©
Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved 💜.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, since they describe your rights and restrictions with respect to this document 📜.
Table of Contents 🗺️
1. Introduction ....................................................4
2. Terminology .....................................................5
3. The Internet Routing Overlay Network ............................7
3.1. IRON Client ................................................9
3.2. IRON Serving Router .......................................10
3.3. IRON Relay Router .........................................10
4. IRON Organizational Principles .................................11
5. IRON Initialization ............................................13
5.1. IRON Relay Router Initialization ..........................13
5.2. IRON Serving Router Initialization ........................14
5.3. IRON Client Initialization ................................15
6. IRON Operation .................................................15
6.1. IRON Client Operation .....................................16
6.2. IRON Serving Router Operation .............................17
6.3. IRON Relay Router Operation ...............................18
6.4. IRON Reference Operating Scenarios ........................18
6.4.1. Both Hosts within IRON EUNs ........................19
6.4.2. Mixed IRON and Non-IRON Hosts ......................21
6.5. Mobility, Multihoming, and Traffic Engineering
Considerations ............................................24
6.5.1. Mobility Management ................................24
6.5.2. Multihoming ........................................25
6.5.3. Inbound Traffic Engineering ........................25
6.5.4. Outbound Traffic Engineering .......................25
6.6. Renumbering Considerations ................................25
6.7. NAT Traversal Considerations ..............................26
6.8. Multicast Considerations ..................................26
6.9. Nested EUN Considerations .................................26
6.9.1. Host A Sends Packets to Host Z .....................28
6.9.2. Host Z Sends Packets to Host A .....................28
7. Implications for the Internet ..................................29
8. Additional Considerations ......................................30
9. Related Initiatives ............................................30
10. Security Considerations .......................................31
11. Acknowledgements ..............................................31
12. References ....................................................32
12.1. Normative References .....................................32
12.2. Informative References ...................................32
Appendix A. IRON VPs over Internetworks with Different
Address Families ......................................35
Appendix B. Scaling Considerations ................................361. Introduction ✨
Okay so the number of entries in the Internet routing system keeps growing, and that's raising real concerns about unsustainable routing scaling [RADIR] 😬. A big reason is operational practice: more and more multihoming with Provider-Independent (PI) addressing means more and more fine-grained prefixes getting injected into the routing system from more and more end user networks 🧩. Then the public IPv4 address space is running low, which causes address space fragmentation (more routing table entries, ugh) and an impending run-out scenario 💀. Meanwhile the IPv6 routing system is starting to grow too [BGPMON], and that has to be managed so we don't repeat the IPv4 scaling mess 🙈. Since the Internet has to keep scaling for all this demand, new routing methodologies and operational practices are needed, no cap 💅.
Several related works have looked at routing scaling. Virtual Aggregation (VA) [GROW-VA] and Aggregation in Increasing Scopes (AIS) [EVOLUTION] are global routing proposals that introduce routing overlays with Virtual Prefixes (VPs) to cut down the number of entries needed in each router's Forwarding Information Base (FIB) and Routing Information Base (RIB) 🌈. Routing and Addressing in Networks with Global Enterprise Recursion (RANGER) [RFC5720] looks at recursive arrangements of enterprise networks that can apply to a very broad set of use-case scenarios [RFC6139] 🦋. IRON specifically adopts the RANGER Non-Broadcast, Multiple Access (NBMA) tunnel virtual-interface model, and uses Virtual Enterprise Traversal (VET) [INTAREA-VET] and the Subnetwork Adaptation and Encapsulation Layer (SEAL) [INTAREA-SEAL] as its functional building blocks 🧱.
This document proposes IRON with goals of supporting sustainable growth while requiring no changes to the existing routing system 👑. IRON borrows concepts from VA and AIS, and also from the Internet Vastly Improved Plumbing (Ivip) [IVIP-ARCH] architecture proposal, along with its associated Translating Tunnel Router (TTR) mobility extensions [TTRMOB]. Honestly, the TTR model inspired the IRON mobility architecture design discussed in this document to a great degree 🫶. The Network Address Translator (NAT) traversal techniques adapted for IRON were inspired by the Simple Address Mapping for Premises Legacy Equipment (SAMPLE) proposal [SAMPLE] ✨.
IRON supports scalable addressing without changing the current BGP [RFC4271] routing system 🌸. It observes the Internet Protocol standards [RFC0791][RFC2460]. Other network-layer protocols that can be encapsulated within IP packets (e.g., OSI/CLNP (Connectionless Network Protocol) [RFC1070], etc.) are also within scope 🧚.
The IRON is a global routing system made of virtual overlay networks managed by Virtual Prefix Companies (VPCs). Those VPCs own and manage Virtual Prefixes (VPs), and End User Network (EUN) prefixes (EPs) are delegated from those VPs to customer sites 🍓. IRON is motivated by growing customer demand for multihoming, mobility management, and traffic engineering, all while using stable addressing to minimize dependence on network renumbering [RFC4192][RFC5887] 💖. The IRON uses the existing IPv4 and IPv6 global Internet routing systems as virtual NBMA links for tunneling inner network protocol packets within outer IPv4 or IPv6 headers (see Section 3) 🎀. It requires deployment of a small number of new BGP core routers and supporting servers, plus IRON-aware routers and servers in customer EUNs 🖥️. No modifications to hosts are needed, and no modifications to most routers are required either 🙌.
Note: This document is offered in compliance with Internet Research Task Force (IRTF) document stream procedures [RFC5743]. It is not an IETF product and is not a standard. The views in this document were considered controversial by the IRTF Routing Research Group (RRG), but the RG reached a consensus that the document should still be published. The document will undergo a period of review within the RRG and through selected expert reviewers prior to publication.
The following sections discuss details of the IRON architecture 🔥.
2. Terminology ✨
okay so before we get into the good stuff, the doc sets up some vocab 🎀 these are the terms it uses:
End User Network (EUN): an edge network that hooks up an organization's devices (computers, routers, printers, the whole crew 🖨️) to the Internet 🌐
End User Network Prefix (EP): a more specific inner network-layer prefix that comes from a Virtual Prefix (VP) (think an IPv4 /28 or an IPv6 /56, etc.) and gets handed out to an EUN by a Virtual Prefix Company (VPC) 💅
End User Network Prefix Address (EPA): a network-layer address that belongs to an EP and is assigned to the interface of an end system in an EUN 🦋
Forwarding Information Base (FIB): a data structure holding network prefix to next-hop mappings. It's usually kept in a router's fast-path processing lookup tables ⚡
Internet Routing Overlay Network (IRON): a composite virtual overlay network. It's the union of all the VPC overlay networks configured over one common Internetwork 👑 The IRON routes by encapsulating inner packets (the ones with EPA addresses) inside outer headers that use locator addresses 🎁
IRON Client Router/Host ("Client"): a customer's router or host that logically connects the customer's EUNs and their EPs to the IRON through an NBMA tunnel virtual interface 🫶
IRON Serving Router ("Server"): a VPC's overlay network router that provides forwarding and mapping services for the EPs owned by customer Clients 🧚
IRON Relay Router ("Relay"): a VPC's overlay network router that acts as a relay between the IRON and the native Internet 🌉
IRON Agent (IA): the catch-all term for any of an IRON Client, Server, or Relay 🫧
Internet Service Provider (ISP): a service provider that connects customer EUNs to the underlying Internetwork. Put simply, an ISP is responsible for basic Internet connectivity for customer EUNs 📡
Locator: an IP address assigned to the interface of a router or end system within a public or private network. Locators from public IP prefixes are routable on a global basis, while locators from private IP prefixes are made public through Network Address Translation (NAT) 🔐
Routing and Addressing in Networks with Global Enterprise Recursion (RANGER): an architectural look at virtual overlay networks applied to enterprise network scenarios, with implications for a wider variety of use cases 🌈
Subnetwork Encapsulation and Adaptation Layer (SEAL): an encapsulation sublayer that provides extended packet identification and a Control Message Protocol to ensure deterministic network-layer feedback 💜
Virtual Enterprise Traversal (VET): a method for discovering border routers and forming dynamic tunnel-neighbor relationships over enterprise networks (or sites) with varying properties 🦋
Virtual Prefix (VP): a prefix block (think an IPv4 /16, an IPv6 /20, an OSI Network Service Access Protocol (NSAP) prefix, etc.) that is owned and managed by a Virtual Prefix Company (VPC) 🍓
Virtual Prefix Company (VPC): a company that owns and manages a set of VPs, and delegates EPs out of them to EUNs 💖
VPC Overlay Network: a specialized set of routers deployed by a VPC to service customer EUNs through a virtual overlay network configured over an underlying Internetwork (for example, the global Internet) 🌸
3. The Internet Routing Overlay Network 🌍
the IRON is a whole system of virtual overlay networks sitting on top of a common Internetwork ✨ the doc talks about it in terms of the public global Internet, but the same principles work for any autonomous Internetwork too 💅 so from here on, "Internet" and "Internetwork" are used interchangeably unless a difference actually matters 🎀
the IRON is made of IRON Agents (IAs) that automatically tunnel the packets of end-to-end communication sessions inside encapsulating headers that are used for Internet routing 🫶 IAs use the Virtual Enterprise Traversal (VET) [INTAREA-VET] virtual NBMA link model together with the Subnetwork Encapsulation and Adaptation Layer (SEAL) [INTAREA-SEAL] to wrap inner network-layer packets in outer headers, as shown in Figure 1 🎁
+-------------------------+
| Outer headers with |
~ locator addresses ~
| (IPv4 or IPv6) |
+-------------------------+
| SEAL Header |
+-------------------------+ +-------------------------+
| Inner Packet Header | --> | Inner Packet Header |
~ with EP addresses ~ --> ~ with EP addresses ~
| (IPv4, IPv6, OSI, etc.) | --> | (IPv4, IPv6, OSI, etc.) |
+-------------------------+ +-------------------------+
| | --> | |
~ Inner Packet Body ~ --> ~ Inner Packet Body ~
| | --> | |
+-------------------------+ +-------------------------+
Inner packet before Outer packet after
encapsulation encapsulation
Figure 1: Encapsulation of Inner Packets within Outer IP HeadersVET is the one that specifies the automatic tunneling mechanisms used for encapsulation 🌊 SEAL is the one that specifies the SEAL header's format and usage, plus a set of control messages 💜 The big highlight is that IAs use the SEAL Control Message Protocol (SCMP) to deterministically exchange and authenticate control messages, like route redirections, indications of Path Maximum Transmission Unit (PMTU) limitations, destination unreachables, etc. 🔥 IAs show up as neighbors on an NBMA virtual link and form bidirectional and/or unidirectional tunnel-neighbor relationships 🤝
the IRON is the union of all the virtual overlay networks configured over a common underlying Internet and owned and managed by Virtual Prefix Companies (VPCs) 👑 Each of those overlay networks is a set of IAs spread throughout the Internet to serve highly aggregated Virtual Prefixes (VPs) 🌈 VPCs delegate sub-prefixes out of their VPs, and lease them to customers as End User Network Prefixes (EPs) 🎀 Then the customers assign the EPs to their customer edge IAs, which connect their End User Networks (EUNs) to the IRON 🫧
VPCs may have no affiliation with the ISP networks that customers get their basic Internet connectivity from 🙈 So a customer could buy its summary network services through a common broker or through separate entities 💅 In that case, the VPC can open for business and start serving customers right away, without needing to coordinate with ISPs or other VPCs 🌸 Further details on business considerations are out of scope for this document 💼
the IRON needs no changes to end systems or to most routers on the Internet 🙌 Instead, the IRON is made of IAs that get deployed either as new platforms or as modifications to existing platforms 🍓 IAs may be deployed incrementally without disturbing the existing Internet routing system, and they act as waypoints (or "cairns") for navigating the IRON 🗺️ The functional roles for IAs are described in the following sections 🧚
3.1. IRON Client
an IRON client (or just "Client") is a customer's router or host that logically connects the customer's EUNs and their associated EPs to the IRON through tunnels, as shown in Figure 2 🫶 Client routers get EPs from VPCs and use them to number subnets and interfaces within their EUNs 🔢 A Client can run on the same physical platform that connects the customer's EUNs to its ISPs, but it can also be a separate router or even a standalone server system located within the EUN 🖥️ (This model applies even if the EUN connects to the ISP through a Network Address Translator (NAT) -- see Section 6.7 🔀) Finally, a Client may also be a simple end system that connects a singleton EPA and has the outward appearance of a host 🦋
.-.
,-( _)-.
+--------+ .-(_ (_ )-.
| Client |--(_ ISP )
+---+----+ `-(______)-'
| <= T \ .-.
.-. u \ ,-( _)-.
,-( _)-. n .-(_ (- )-.
.-(_ (_ )-. n (_ Internet )
(_ EUN ) e `-(______)-
`-(______)-' l ___
| s => (:::)-.
+----+---+ .-(::::::::)
| Host | .-(::::::::::::)-.
+--------+ (:::: The IRON ::::)
`-(::::::::::::)-'
`-(::::::)-'
Figure 2: IRON Client Router Connecting EUN to the IRON3.2. IRON Serving Router
an IRON serving router (or just "Server") is a VPC's overlay network router that provides forwarding and mapping services for the EPs owned by customer Client routers 🧚 In typical deployments, a VPC will deploy lots of Servers around the IRON in a globally distributed way (as depicted in Figure 3), so that Clients can discover the ones that are nearby 🌍
+--------+ +--------+
| Boston | | Tokyo |
| Server | | Server |
+--+-----+ ++-------+
+--------+ \ /
| Seattle| \ ___ /
| Server | \ (:::)-. +--------+
+------+-+ .-(::::::::)------+ Paris |
\.-(::::::::::::)-. | Server |
(:::: The IRON ::::) +--------+
`-(::::::::::::)-'
+--------+ / `-(::::::)-' \ +--------+
| Moscow + | \--- + Sydney |
| Server | +----+---+ | Server |
+--------+ | Cairo | +--------+
| Server |
+--------+
Figure 3: IRON Serving Router Global Distribution Exampleeach Server acts as a tunnel-endpoint router that forms a bidirectional tunnel-neighbor relationship with each of its Client customers 🤝 Each Server also associates with a set of Relays, which can forward packets from the IRON out to the native Internet and back again, as we'll get into in the next section 🌉✨
3.3. IRON Relay Router ✨
okay so an IRON Relay Router (or just "Relay" 💅) is the VPC's overlay network router that acts as a relay between the IRON and the native Internet 🌉. That also makes it an Autonomous System Border Router (ASBR), and the VPC owns and manages it. Main character energy 👑
Each VPC sets up one or more Relays that advertise the company's VPs into the IPv4 and IPv6 global Internet BGP routing systems 🦋. Every Relay connects to all of the VPC's overlay network Servers, like through tunnels over the IRON or a direct interconnect such as an Ethernet cable 🔌. The Relay role (and how it hooks up to the Servers) is shown in Figure 4 ✨
.-.
,-( _)-.
.-(_ (_ )-.
(_ Internet )
`-(______)-' | +--------+
| |--| Server |
+----+---+ | +--------+
| Relay |----| +--------+
+--------+ |--| Server |
_|| | +--------+
(:::)-. (Ethernet)
.-(::::::::)
+--------+ .-(::::::::::::)-. +--------+
| Server |=(:::: The IRON ::::)=| Server |
+--------+ `-(::::::::::::)-' +--------+
`-(::::::)-'
|| (Tunnels)
+--------+
| Server |
+--------+
Figure 4: IRON Relay Router Connecting IRON to Native Internet4. IRON Organizational Principles 🧚
okay this section is the big picture 🖼️. The IRON is the union of all VPC overlay networks running over a shared Internetwork (like the public Internet, fr). Each overlay network is its own distinct "patch" on the Internet "quilt" 🧵, and tunnels over the links, routers, bridges, etc. that connect the underlying Internetwork are what stitch the patches together. Whenever a new VPC overlay network gets deployed, it's just another patch on the quilt 🌈. So the IRON is a composite overlay network made of lots of patches, and each patch coordinates its own stuff independently of the others. The one exception is that the Servers of each patch must be aware of all VPs in the IRON 👀. To keep all the VPC overlay networks cooperating nicely, sufficient address space portions of the inner network-layer protocol (e.g., IPv4, IPv6, etc.) should be set aside and designated as VP space 🎀.
Each VPC overlay network in the IRON keeps a set of Relays and Servers that serve their Client customers 💖. For solid customer service levels, the VPC should do a traffic scaling analysis and spread out enough Relays and Servers for the overlay network globally across the Internet 🌍. Figure 5 shows how Relays, Servers, and Clients are arranged in an IRON virtual overlay network 💜
.-.
,-( _)-.
.-(_ (_ )-.
(__ Internet _)
`-(______)-'
<------------ Relays ------------>
________________________
(::::::::::::::::::::::::)-.
.-(:::::::::::::::::::::::::::::)
.-(:::::::::::::::::::::::::::::::::)-.
(::::::::::: The IRON :::::::::::::::)
`-(:::::::::::::::::::::::::::::::::)-'
`-(::::::::::::::::::::::::::::)-'
<------------ Servers ------------>
.-. .-. .-.
,-( _)-. ,-( _)-. ,-( _)-.
.-(_ (_ )-. .-(_ (_ )-. .-(_ (_ )-.
(__ ISP A _) (__ ISP B _) ... (__ ISP x _)
`-(______)-' `-(______)-' `-(______)-'
<----------- NATs ------------>
<----------- Clients and EUNs ----------->
Figure 5: Virtual Overlay Network OrganizationEach Relay in the VPC overlay network hooks the overlay directly up to the underlying IPv4 and IPv6 Internets 🔗. It also advertises the VPC overlay network's IPv4 VPs into the IPv4 BGP routing system, and advertises the overlay network's IPv6 VPs into the IPv6 BGP routing system 📣. So Relays get packets with EPA destination addresses from end systems on the Internet and send them along toward the EPA-addressed end systems connected to the VPC overlay network 🚀.
Each VPC overlay network also manages a set of Servers. These connect their Clients and associated EUNs to the IRON and to the IPv6 and IPv4 Internets through their links with Relays 🤝. IRON Servers don't need to be BGP routers themselves, they can just be simple commodity hardware platforms 📦 (so chill). Also, the Server and Relay functions can be deployed together on the same physical platform as one unified gateway 🎁, or they can live on separate platforms, like for load balancing purposes ⚖️.
Each Server keeps a working set of Clients, and it caches EP-to-Client mappings in its Forwarding Information Base (FIB) 🗂️. Each Server also passes the list of EPs in its working set along to each of the Relays in the VPC overlay network. It does this through a dynamic routing protocol, like an overlay network internal BGP instance that carries only the EP-to-Server mappings and doesn't interact with the external BGP routing system 🔒. So each Server only has to track the EPs for its current working set of Clients, while each Relay maintains a full EP-to-Server mapping table covering reachability information for all EPs in the VPC overlay network 🧠.
Customers set up Clients that get their basic Internet connectivity from ISPs and connect to Servers to attach their EUNs to the IRON 💻. Each EUN can connect to the IRON through one or multiple Clients, as long as those Clients coordinate with each other (for example, to mitigate EUN partitions) 🫶. Unlike Relays and Servers, Clients may use private addresses behind one or several layers of NATs 🙈. Each Client first discovers a list of nearby Servers through an anycast discovery process (described below) 🔍. Then it picks one of those nearby Servers and forms a bidirectional tunnel-neighbor relationship with that Server through an initial exchange, followed by periodic keepalives 💞.
Once the Client picks a Server, it sends its initial outbound packets from its EUNs by tunneling them to the Server ✈️. The Server then forwards them to the nearest Relay within the IRON that serves the final destination 🎯. After that, the Client gets redirect messages telling it about a more direct route through a Server that serves the final destination EUN 💫.
The IRON can also carry VPs of network-layer address families that can't be routed natively in the underlying Internetwork (for example, OSI/CLNP over the public Internet, IPv6 over IPv4-only Internetworks, IPv4 over IPv6-only Internetworks, and so on) 🌀. More details on supporting IRON VPs of one address family over Internetworks based on other address families are in Appendix A 📖
5. IRON Initialization 🌸
IRON initialization covers the startup actions of IAs within the VPC overlay network and of customer EUNs 🚦. The sub-sections below walk through these startup procedures 👇
5.1. IRON Relay Router Initialization 🎬
Before its first operational use, each Relay in a VPC overlay network gets provisioned with the list of VPs it will serve, plus the locators for all Servers that belong to the same overlay network 📝. The Relay also gets provisioned with external BGP interconnections, same as any BGP router would 🔧.
On startup, the Relay does BGP routing exchanges with its peers in the IPv4 and IPv6 Internets, just like any BGP router 🤙. Then it connects to all of the Servers in the overlay network (for example, via a TCP connection over a bidirectional tunnel, or via an Internal BGP (IBGP) route reflector, etc.) so it can discover EP-to-Server mappings 🔎. Once the Relay has fully populated its EP-to-Server mapping information database, it's said to be "synchronized" with regard to its VPs ✅.
After that initial synchronization, the Relay advertises the overlay network's VPs externally 📡. Specifically, it advertises the IPv6 VPs into the IPv6 BGP routing system and the IPv4 VPs into the IPv4 BGP routing system. The Relay also advertises an IPv4 /24 companion prefix (for example, 192.0.2.0/24) into the IPv4 routing system, and an IPv6 ::/64 companion prefix (for example, 2001:DB8::/64) into the IPv6 routing system 🌷. (These may also be sub-prefixes taken from a VP, fyi.) Then the Relay sets host number '1' in the IPv4 companion prefix (for example, as 192.0.2.1) and interface identifier '0' in the IPv6 companion prefix (for example, as 2001:DB8::0). It assigns the resulting addresses as subnet-router anycast addresses [RFC3068][RFC2526] for the VPC overlay network 🎀. (See Appendix A for more on discovering and using companion prefixes 📚.) At this point the Relay gets on with ordinary packet-forwarding operations 🚀✨
5.2. IRON Serving Router Initialization 🫧
Before its first operational use, each Server in a VPC overlay network gets provisioned with the locators for all Relays that aggregate the overlay network's VPs 🗺️. To support route optimization, the Server also has to be provisioned with the list of all VPs in the IRON (not just the VPs of its own overlay network) so it can tell EPA and non-EPA addresses apart 🕵️♀️. (So the Server could be way simpler if the list of VPs could be covered by a small number of very short prefixes, for example one or a few IPv6 ::/20's 🧁.) The Server must also discover the VP companion prefix relationships discussed in Section 5.1, for example through a global database like the one discussed in Appendix A 🔑.
On startup, each Server must connect to all of the Relays within its overlay network (for example, via a TCP connection or an IBGP route reflector, etc.) so it can report its EP-to-Server mappings 📤. Then the Server actively listens for Client customers that register their EP prefixes as part of setting up a bidirectional tunnel-neighbor relationship 👂. When a new Client registers its EP prefixes, the Server announces the new EP additions to all Relays 📣💖. When an existing Client unregisters its EP prefixes, the Server withdraws its announcements 🙅♀️
5.3. IRON Client Initialization ✨
okay bestie, before a Client can do anything operational, it has to grab one or more EPs from its VPC, plus the companion prefixes that go with the VPC overlay network (see Section 5.1) 🎀. It also must get a certificate and a public/private key pair from the VPC so it can later prove it actually owns its EPs 🔐. This implies that each VPC must run its own public key infrastructure, and it's only for checking that customers really have the right to use an EP. Hence, the VPC need not coordinate its public key infrastructure with any other organization. so independent 💅
Upon startup, the Client sends an SCMP Router Solicitation (SRS) message to the VPC overlay network subnet-router anycast address to find the nearest Relay 🔍. The Relay will send back an SCMP Router Advertisement (SRA) message that lists the locator addresses of one or more nearby Servers. (This list is analogous to the Intra-Site Automatic Tunnel Addressing Protocol (ISATAP) Potential Router List (PRL) [RFC5214].) it's giving "list of options" energy 📜
After the Client receives an SRA message from the nearby Relay listing the locator addresses of nearby Servers, it starts a short transaction with one of those Servers over a reliable transport protocol such as TCP, so they can establish a bidirectional tunnel-neighbor relationship 🤝. The protocol details of the transaction are specific to the VPC, and hence out of scope for this document. (that's a VPC problem, not ours 🙈)
Note that it is essential that the Client select one and only one Server. This is to allow the VPC overlay network mapping system to have one and only one active EP-to-Server mapping at any point in time, which shares fate with the Server itself. If this Server fails, the Client can select a new one that will automatically update the VPC overlay network mapping system with a new EP-to-Server mapping. the one-Server rule is non-negotiable, periodt 👑
6. IRON Operation 🌈
Following the IRON initialization detailed in Section 5, IAs get into the steady-state groove of receiving and forwarding packets 🦋. All IAs forward encapsulated packets over the IRON using the mechanisms of VET [INTAREA-VET] and SEAL [INTAREA-SEAL], while Relays (and in some cases Servers) additionally forward packets to and from the native IPv6 and IPv4 Internets. IAs also use SCMP to coordinate with other IAs, including sending and receiving redirect messages, error messages, etc. (Note however that an IA must not send an SCMP message in response to an SCMP error message. no replying to errors with more messages, that's just chaos 💀) Each IA operates as specified in the following sub-sections.
6.1. IRON Client Operation
After selecting its Server as specified in Section 5.3, the Client should register each of its ISP connections with the Server for multihoming purposes 🌸. To do so, it sends periodic beacons (e.g., SRS messages) to its Server via each of its ISPs to establish additional tunnel-neighbor state. This implies that a single tunnel-neighbor identifier (i.e., a "nonce") is used to represent the set of all ISP paths between the Client and the Server. Therefore, the nonce names this "bundle" of ISP paths 🧺.
If the Client stops receiving acknowledgements from its Server via a specific ISP connection, it marks the Server as unreachable from that address and therefore over that ISP connection. (The Client should also inform its Server of this outage via one of its working ISP connections.) If the Client stops receiving acknowledgements from its Server via multiple ISP connections, it marks the Server as unusable and quickly tries to register with a new Server 😭. The act of registering with a new Server will automatically purge the stale mapping state associated with the old Server, since dynamic routing will propagate the new client/server relationship to the VPC overlay network Relay Routers. clean breakup, no leftover mapping drama 💅
When an end system in an EUN sends a flow of packets to a correspondent, the packets are forwarded through the EUN via normal routing until they reach the Client, which then tunnels the initial packets to its Server as the next hop 🚀. In particular, the Client encapsulates each packet in an outer header with its locator as the source address and the locator of its Server as the destination address. Note that after sending the initial packets of a flow, the Client may receive important SCMP messages, such as indications of PMTU limitations, redirects that point to a better next hop, etc. 👀
The Client uses the mechanisms specified in VET and SEAL to encapsulate each forwarded packet. The Client further uses the SCMP protocol to coordinate with Servers, including accepting redirects and other SCMP messages. When the Client receives an SCMP message, it checks the nonce field of the encapsulated packet-in-error to verify that the message corresponds to the tunnel-neighbor state for its Server, and it accepts the message if the nonce matches ✅. (Note however that the outer source and destination addresses of the packet-in-error may be different than those in the original packet due to possible Server and/or Relay address rewritings. so don't panic if the addresses look different, it's the rewriting being extra 🙈)
6.2. IRON Serving Router Operation
After the Server is initialized, it responds to SRSs from Clients by sending SRAs 🎀. When the Server receives a SEAL-encapsulated packet from one of its Client tunnel neighbors, it examines the inner destination address. If the inner destination address is not an EPA, the Server decapsulates the packet and forwards it unencapsulated into the Internet if it is able to do so without loss due to ingress filtering. Otherwise, the Server re-encapsulates the packet (i.e., it removes the outer header and replaces it with a new outer header of the same address family) and sets the outer destination address to the locator address of a Relay within its VPC overlay network. It then forwards the re-encapsulated packet to the Relay, which will, in turn, decapsulate it and forward it into the Internet 🌍.
If the inner destination address is an EPA, however, the Server rewrites the outer source address to one of its own locator addresses and rewrites the outer destination address to the subnet-router anycast address taken from the companion prefix associated with the inner destination address (where the companion prefix of the same address family as the outer IP protocol is used). The Server then forwards the revised encapsulated packet into the Internet via a default or more specific route, where it will be directed to the closest Relay within the destination VPC overlay network 🧭. After sending the packet, the Server may then receive an SCMP error or redirect message from a Relay/Server within the destination VPC overlay network. In that case, the Server verifies that the nonce in the message matches the Client that sent the original inner packet, and it discards the message if the nonce does not match. Otherwise, the Server re-encapsulates the SCMP message in a new outer header that uses the source address, destination address, and nonce parameters associated with the Client's tunnel-neighbor state; it then forwards the message to the Client. This arrangement is necessary to allow SCMP messages to flow through any NATs on the path 🦋.
When a Server ('A') receives a SEAL-encapsulated packet from a Relay or from the Internet, if the inner destination address matches an EP in its FIB, 'A' re-encapsulates the packet in a new outer header and forwards it to a Client ('B'). B then decapsulates the packet and forwards it to the correct end system in the EUN 📦. However, if 'B' has left notice with 'A' that it has moved to a new Server ('C'), 'A' will instead forward the packet to 'C' and also send an SCMP redirect message back to the source of the packet. In this way, 'B' can leave behind forwarding information when changing between Servers 'A' and 'C' (e.g., due to mobility events) without exposing packets to loss. leaving a forwarding note on the way out is such a main character move 🧚
6.3. IRON Relay Router Operation
After each Relay has synchronized its VPs (see Section 5.1), it advertises the full set of the company's VPs and companion prefixes into the IPv4 and IPv6 Internet BGP routing systems 📡. These prefixes will be represented as ordinary routing information in the BGP, and any packets originating from the IPv4 or IPv6 Internet destined to an address covered by one of the prefixes will be forwarded to one of the VPC overlay network's Relays.
When a Relay receives a packet from the Internet destined to an EPA covered by one of its VPs, it behaves as an ordinary IP router 🤓. In particular, the Relay looks in its FIB to discover a locator of the Server that serves the EP covering the destination address. The Relay then simply encapsulates the packet with its own locator as the outer source address and the locator of the Server as the outer destination address, and forwards the packet to the Server.
When a Relay receives a packet from the Internet destined to one of its subnet-router anycast addresses, it discards the packet if it is not SEAL encapsulated 🚫. If the packet is an SCMP SRS message, the Relay instead sends an SRA message back to the source listing the locator addresses of nearby Servers, then discards the message. The Relay otherwise discards all other SCMP messages.
If the packet is an ordinary SEAL packet (i.e., one that encapsulates an inner packet), the Relay sends an SCMP redirect message of the same address family back to the source, with the locator of the Server that serves the EPA destination in the inner packet as the redirected target 🎯. The source and destination addresses of the SCMP redirect message use the outer destination and source addresses of the original packet, respectively. After sending the redirect message, the Relay then rewrites the outer destination address of the SEAL-encapsulated packet to the locator of the Server and forwards the revised packet to the Server. Note that in this arrangement, any errors that occur on the path between the Relay and the Server will be delivered to the original source, but with a different destination address due to this Relay address rewriting 🙈.
6.4. IRON Reference Operating Scenarios
The IRON supports communications when one or both hosts are located within EP-addressed EUNs, regardless of whether the EPs are provisioned by the same VPC or by different VPCs 🌈. When both hosts are within IRON EUNs, route redirections that eliminate unnecessary Servers and Relays from the path are possible. When only one host is within an IRON EUN, however, route optimization cannot be used. The following sections discuss the two scenarios. stay tuned, it's about to get spicy 🔥
6.4.1. Both Hosts within IRON EUNs ✨
okay so when both hosts are inside IRON EUNs, you only need to trace the packets one way, from the source host to the destination host 💅 The reverse direction is its own thing, but it gets the exact same considerations as the forward one, so you don't need to think about it twice 🫶
Here's the vibe. The first packets of a flow from a source host in an EUN connected to the IRON by a Client must go through both the Server of the source host and a Relay of the destination host. BUT route optimization can kick those middlemen off the path for the packets that come after, which is so iconic 👑 Figure 6 shows the initial packets going from host A to host B inside two IRON EUNs. And it's the same story whether the two EUNs live in the same VPC overlay network or in different ones 🧚
________________________________________
.-( .-. )-.
.-( ,-( _)-. )-.
.-( +========+(_ (_ +=====+ )-.
.( || (_|| Internet ||_) || ).
.( || ||-(______)-|| vv ).
.( +--------++--+ || || +------------+ ).
( +==>| Server(A) | vv || | Server(B) |====+ )
( // +---------|\-+ +--++----++--+ +------------+ \\ )
( // .-. | \ | Relay(B) | .-. \\ )
( //,-( _)-. | \ +-v----------+ ,-( _)-\\ )
( .||_ (_ )-. | \____| .-(_ (_ ||. )
( _|| ISP A .) | (__ ISP B ||_))
( ||-(______)-' | (redirect) `-(______)|| )
( || | | | vv )
( +-----+-----+ | +-----+-----+ )
| Client(A) | <--+ | Client(B) |
+-----+-----+ The IRON +-----+-----+
| ( (Overlaid on the Native Internet) ) |
.-. .-( .-) .-.
,-( _)-. .-(________________________)-. ,-( _)-.
.-(_ (_ )-. .-(_ (_ )-.
(_ IRON EUN A ) (_ IRON EUN B )
`-(______)-' `-(______)-'
| |
+---+----+ +---+----+
| Host A | | Host B |
+--------+ +--------+
Figure 6: Initial Packet Flow before RedirectsSo here's the tea ☕ Host A sends its packets toward host B through its network interface on EUN A. Routing in EUN A sends them to Client(A), which is the default router for the EUN. Client(A) then uses VET and SEAL to encapsulate them, with its own locator as the outer source address and the locator address of Server(A) as the outer destination address. After that, Client(A) just forwards the encapsulated packets out through the ISP connection that gave it its locator. The ISP lets them through without filtering because the (outer) source address is topologically correct 🌸 Once they're in the Internet, routing takes them to Server(A).
Server(A) gets the encapsulated packets from Client(A), then rewrites the outer source address to one of its own locator addresses. It also rewrites the outer destination address to the subnet-router anycast address of the right address family for the inner destination address. Then it sends the revised packets into the Internet, and routing gets them to Relay(B), which serves the VPC overlay network that host B is in 🦋
Relay(B) catches the encapsulated packets from Server(A). It checks its FIB for an entry that covers inner destination address B with Server(B) as the next hop. Then Relay(B) sends SCMP redirect messages back to Server(A) (*), rewrites the outer destination address to the locator address of Server(B), and forwards the packets over to Server(B) 🧚♀️
Server(B) receives the encapsulated packets from Relay(B). It checks its FIB for an entry that covers destination address B with Client(B) as the next hop. Then Server(B) re-encapsulates the packets in a brand new outer header. That header uses the source address, destination address, and nonce parameters tied to the tunnel-neighbor state for Client(B). The re-encapsulated packets go out into the Internet, and routing gets them to Client(B). Client(B) then decapsulates them and passes the inner packets to host B through EUN B 💖
(*) Note that once the initial flow is done, Server(A) will have gotten one or more SCMP redirect messages from Relay(B) that name Server(B) as a better next hop. Server(A) then forwards those redirects to Client(A). Client(A) sets up unidirectional tunnel-neighbor state and from then on sends its encapsulated packets straight to the locator address of Server(B). Neither Server(A) nor Relay(B) is involved anymore, as shown in Figure 7 🔥
________________________________________
.-( .-. )-.
.-( ,-( _)-. )-.
.-( +=============> .-(_ (_ )-.======+ )-.
.( // (__ Internet _) || ).
.( // `-(______)-' vv ).
.( // +------------+ ).
( // | Server(B) |====+ )
( // +------------+ \\ )
( // .-. .-. \\ )
( //,-( _)-. ,-( _)-\\ )
( .||_ (_ )-. .-(_ (_ ||. )
( _|| ISP A .) (__ ISP B ||_))
( ||-(______)-' `-(______)|| )
( || | | vv )
( +-----+-----+ The IRON +-----+-----+ )
| Client(A) | (Overlaid on the native Internet) | Client(B) |
+-----+-----+ +-----+-----+
| ( ) |
.-. .-( .-) .-.
,-( _)-. .-(________________________)-. ,-( _)-.
.-(_ (_ )-. .-(_ (_ )-.
(_ IRON EUN A ) (_ IRON EUN B )
`-(______)-' `-(______)-'
| |
+---+----+ +---+----+
| Host A | | Host B |
+--------+ +--------+
Figure 7: Sustained Packet Flow after Redirects6.4.2. Mixed IRON and Non-IRON Hosts 🍓
Now things get a little spicy 🌶️ This is when one host is inside an IRON EUN and the other is in a non-IRON EUN, which means it connects to the native Internet instead of the IRON. Which IA elements get involved depends on which way the packets are flowing, so the next sub-sections break it down case by case 💜
6.4.2.1. From IRON Host A to Non-IRON Host B
Figure 8 shows the IRON reference operating scenario for packets flowing from host A in an IRON EUN to host B in a non-IRON EUN 🦋
_________________________________________
.-( )-. )-.
.-( +-------)----+ )-.
.-( | Relay(A) |--------------+ )-.
.( +------------+ \ ).
.( +=======>| Server(A) | \ ).
.( // +--------)---+ \ ).
( // ) \ )
( // The IRON ) \ )
( // .-. ) \ .-. )
( //,-( _)-. ) \ ,-( _)-. )
( .||_ (_ )-. ) The Native Internet .-|_ (_ )-. )
( _|| ISP A ) ) (_ | ISP B ))
( ||-(______)-' ) |-(______)-' )
( || | )-. v | )
( +-----+ ----+ )-. +-----+-----+ )
| Client(A) |)-. | Router B |
+-----+-----+ +-----+-----+
| ( ) |
.-. .-(____________________________________)-. .-.
,-( _)-. ,-( _)-.
.-(_ (_ )-. .-(_ (_ )-.
(_ IRON EUN A ) (_non-IRON EUN B)
`-(______)-' `-(______)-'
| |
+---+----+ +---+----+
| Host A | | Host B |
+--------+ +--------+
Figure 8: From IRON Host A to Non-IRON Host BHere's how it goes 🌈 Host A sends its packets to host B through its network interface on IRON EUN A. Routing in EUN A sends them to Client(A), the default router for the EUN. Client(A) uses VET and SEAL to encapsulate them, with its locator as the outer source address and the locator address of Server(A) as the outer destination address. The ISP passes them along without filtering because the (outer) source address is topologically correct. Once the packets are released into the native Internet, routing gets them to Server(A) 🫶
Server(A) takes the encapsulated packets from Client(A), re-encapsulates them, and forwards them to Relay(A). Relay(A) just decapsulates them and sends the unencapsulated packets into the Internet. Once they're released, routing delivers them to the final destination B. (Note that Server(A) and Relay(A) are drawn in Figure 8 as two halves of one unified gateway. In that case, the "forwarding" between Server(A) and Relay(A) is a zero-instruction imaginary operation inside the gateway 🧚)
This scenario always involves a Server and Relay owned by the VPC that provides service to IRON EUN A. So it imparts a cost that has to be borne by either the VPC or its customers 💀
6.4.2.2. From Non-IRON Host B to IRON Host A ✨
Okay bestie, time for the reverse trip 🦋 Figure 9 shows the IRON reference scenario for packets going from host B, who lives in a non-IRON EUN, to host A, who lives in an IRON EUN 💌
_______________________________________
.-( )-. )-.
.-( +-------)----+ )-.
.-( | Relay(A) |<-------------+ )-.
.( +------------+ \ ).
.( +========| Server(A) | \ ).
.( // +--------)---+ \ ).
( // ) \ )
( // The IRON ) \ )
( // .-. ) \ .-. )
( //,-( _)-. ) \ ,-( _)-. )
( .||_ (_ )-. ) The Native Internet .-|_ (_ )-. )
( _|| ISP A ) ) (_ | ISP B ))
( ||-(______)-' ) |-(______)-' )
( vv | )-. | | )
( +-----+ ----+ )-. +-----+-----+ )
| Client(A) |)-. | Router B |
+-----+-----+ +-----+-----+
| ( ) |
.-. .-(____________________________________)-. .-.
,-( _)-. ,-( _)-.
.-(_ (_ )-. .-(_ (_ )-.
(_ IRON EUN A ) (_non-IRON EUN B)
`-(______)-' `-(_______)-'
| |
+---+----+ +---+----+
| Host A | | Host B |
+--------+ +--------+
Figure 9: From Non-IRON Host B to IRON Host ASo here's the vibe: host B fires off packets toward host A through its network interface plugged into non-IRON EUN B 📡 Routing then steers those packets to Relay(A), which passes them along to Server(A), and it uses encapsulation along the way if it's needed 🎀
Next up, Server(A) checks its FIB for an entry that covers destination address A with Client(A) as the next hop 👀 Then it (re-)encapsulates the packets in an outer header, using the source address, destination address, and nonce parameters tied to the tunnel-neighbor state for Client(A) 💅 Those (re-)encapsulated packets get forwarded out into the Internet, where routing sends them to Client(A) 🌈 From there, Client(A) decapsulates the packets and forwards the inner packets to host A through its network interface connected to IRON EUN A 🫶
This scenario always involves a Server and Relay owned by the VPC that provides service to IRON EUN A. Therefore, it imparts a cost that would need to be borne by either the VPC or its customers. 💀
6.5. Mobility, Multihoming, and Traffic Engineering Considerations 🧚
Servers and Relays in IRON are basically fixed infrastructure, but Clients might need to move between different network points of attachment, connect to multiple ISPs, or manage their traffic flows on purpose 🌸 The next few sections cover mobility, multihoming, and traffic engineering for IRON client routers 💜
6.5.1. Mobility Management
When a Client changes its network point of attachment (e.g., due to a mobility event), it configures one or more new locators 🦋 If the Client hasn't gone too far from its previous point of attachment, it just tells its Server about any locator additions or deletions. This operation is performance sensitive and should be conducted immediately to avoid packet loss 😭
If the Client has moved far away from its previous network point of attachment, though, it re-issues the anycast discovery procedure described in Section 6.1 to find out whether its candidate set of Servers has changed 🗺️ If the Client's current Server is also in the new list received from the VPC, that's a sign the Client hasn't moved far enough to justify switching to a new Server 🫠 Otherwise, the Client may wish to move to a new Server in order to reduce routing stretch. This operation isn't performance critical, so it can be done over a matter of seconds/minutes instead of milliseconds/microseconds ⭐
To move to a new Server, the Client first goes through the EP registration process with the new Server, as described in Section 5.3 📝 Then it tells its former Server that it has moved, by giving it the locator address of the new Server, again via a VPC-specific reliable transaction 🤝 The former Server will then garbage-collect the stale FIB entries when their lifetime expires 🧹
This lets the former Server redirect existing correspondents to the new Server, so no packets are lost 🎀
6.5.2. Multihoming
A Client may register multiple locators with its Server 🌈 It can assign metrics with its registrations to tell the Server which locators it prefers, and it can pick outgoing locators based on its own local preferences 💅 So multihoming is naturally supported, iykyk 👑
6.5.3. Inbound Traffic Engineering
A Client can dynamically adjust the priorities of its prefix registrations with its Server to influence inbound traffic flows 📈 It can also switch between Servers when more than one is available, but it should strive for stability in its Server selection to limit VPC network routing churn 🫶
6.5.4. Outbound Traffic Engineering
A Client can select outgoing locators based on current Quality-of-Service (QoS) considerations, e.g., minimizing one-way delay or one-way delay variance 🚀
6.6. Renumbering Considerations 🔁
As new link-layer technologies and/or service models show up, customers will be motivated to pick their service providers through healthy competition between ISPs 🏆 But if a customer's EUN addresses are tied to one specific ISP, the customer may be forced to go through a painstaking EUN renumbering process just to switch to a different ISP [RFC4192][RFC5887] 😩
When a customer gets EP prefixes from a VPC, it can switch between ISPs seamlessly, without needing to renumber ✨ If the VPC itself applies unreasonable costing structures for use of the EPs, though, the customer may be pushed to look for a different VPC and would again have to deal with a renumbering scenario 🙃 The IRON approach to renumbering avoidance therefore depends on VPCs running ethical businesses and offering reasonable rates 💖
6.7. NAT Traversal Considerations 🕳️
The Internet today is a global public IPv4 routing and addressing system, with non-IRON EUNs that use either public or private IPv4 addressing 🌍 The private ones connect to the public Internet through Network Address Translators (NATs) 🚪 When a Client sits behind a NAT, it selects Servers using the same procedures as Clients with public addresses, e.g., it can send SRS messages to Servers in order to get SRA messages back 📬 The only requirement is that the Client must configure its SEAL encapsulation to use a transport protocol that supports NAT traversal, namely UDP 💜
Since the Server keeps state about its Client customers, it can discover locator information for each Client by looking at the UDP port number and IP address in the outer headers of the Client's encapsulated packets 🔍 When there's a NAT in the path, the UDP port number and IP address in each encapsulated packet will match state in the NAT box and might not match the actual values assigned to the Client 🙈 The Server can then encapsulate packets destined to hosts in the Client's EUN within outer headers that use this IP address and UDP port number 🎯 The NAT box receives the packets, translates the values in the outer headers, and forwards the packets to the Client 📦 In this sense, the Server's "locator" for the Client is the concatenation of the IP address and UDP port number 🧩
IRON doesn't add any new issues to the complications already raised for NAT traversal or for applications that embed address referrals in their payload 🫧
6.8. Multicast Considerations 📻
IRON Servers and Relays are topologically positioned to provide Internet Group Management Protocol (IGMP) / Multicast Listener Discovery (MLD) proxying for their Clients [RFC4605] 🎶 Further multicast considerations for IRON (e.g., interactions with multicast routing protocols, traffic scaling, etc.) will be discussed in a separate document 📚
6.9. Nested EUN Considerations 🪆
Each Client configures a locator that may come from an ordinary non-EPA address assigned by an ISP, or from an EPA address taken from an EP assigned to another Client 🧸 When that happens, the Client is said to be "nested" within the EUN of another Client, and recursive nestings of multiple layers of encapsulations may be necessary 🌀
Take the network scenario in Figure 10 as an example: Client(A) configures a locator EPA(B), taken from the EP assigned to EUN(B) 🧁 Client(B) in turn configures a locator EPA(C), taken from the EP assigned to EUN(C) 🍓 Finally, Client(C) configures a locator ISP(D), taken from a non-EPA address delegated by an ordinary ISP(D) 🍬 Using this example, we have to look at the "nested-IRON" case, where a host A, which configures the address EPA(A) within EUN(A), exchanges packets with host Z located elsewhere in the Internet 😍
.-.
ISP(D) ,-( _)-.
+-----------+ .-(_ (_ )-.
| Client(C) |--(_ ISP(D) )
+-----+-----+ `-(______)-'
| <= T \ .-.
.-. u \ ,-( _)-.
,-( _)-. n .-(_ (- )-.
.-(_ (_ )-. n (_ Internet )
(_ EUN(C) ) e `-(______)-'
`-(______)-' l ___
| EPA(C) s => (:::)-.
+-----+-----+ .-(::::::::)
| Client(B) | .-(::::::::::::)-. +-----------+
+-----+-----+ (:::: The IRON ::::) | Relay(Z) |
| `-(::::::::::::)-' +-----------+
.-. `-(::::::)-' +-----------+
,-( _)-. | Server(Z) |
.-(_ (_ )-. +-----------+ +-----------+
(_ EUN(B) ) | Server(C) | +-----------+
`-(______)-' +-----------+ | Client(Z) |
| EPA(B) +-----------+ +-----------+
+-----+-----+ | Server(B) | +--------+
| Client(A) | +-----------+ | Host Z |
+-----------+ +-----------+ +--------+
| | Server(A) |
.-. +-----------+
,-( _)-. EPA(A)
.-(_ (_ )-. +--------+
(_ EUN(A) )---| Host A |
`-(______)-' +--------+
Figure 10: Nested EUN ExampleThe two cases, host A sending packets to host Z and host Z sending packets to host A, must be considered separately, as described below 🫶🌷
6.9.1. Host A Sends Packets to Host Z ✨
okay so bestie, Host A kicks things off by forwarding a packet with source EPA(A) and destination Z into EUN(A) 📦. Routing inside EUN(A) sends it to Client(A), which wraps it in an outer header with EPA(B) as the outer source and Server(A) as the outer destination. Then it forwards this once-encapsulated packet into EUN(B) 💌.
Routing inside EUN(B) sends it to Client(B), which wraps it again, this time with EPA(C) as the outer source and Server(B) as the outer destination. It then forwards this twice-encapsulated packet into EUN(C) 🎀. Routing inside EUN(C) sends it to Client(C), which adds one more outer header with ISP(D) as the outer source and Server(C) as the outer destination. Client(C) then sends this triple-encapsulated packet into the ISP(D) network, where it gets routed across the Internet to Server(C) 🚀. The layers are literally stacking up like a whole lasagna 🍝.
When Server(C) gets the triple-encapsulated packet, it peels off the outer layer and forwards the twice-encapsulated packet across the Internet to Server(B) 🧁. Next, Server(B) removes its outer layer and forwards the once-encapsulated packet across the Internet to Server(A) 🌷.
Now it's Server(A)'s turn to check the address type of the inner address 'Z' 👀. If Z is a non-EPA address, Server(A) just decapsulates the packet and forwards it into the Internet, easy peasy 💅. Otherwise, Server(A) rewrites the outer source and destination addresses of the once-encapsulated packet and forwards it to Relay(Z) 🦋. Relay(Z) then rewrites the outer destination address of the packet to the locator for Server(Z), forwards the packet, and sends a redirect to Server(A), which passes the redirect along to Client(A) 🔀. After that, Server(Z) re-encapsulates the packet and forwards it to Client(Z), which decapsulates it and forwards the inner packet to host Z 🎉.
The best part? Subsequent packets from Client(A) use Server(Z) as the next hop toward host Z, which means Server(A) and Relay(Z) get cut out of the path entirely 💜. Main character energy for the shortcut 👑.
6.9.2. Host Z Sends Packets to Host A 💫
So here's the flip side. Whether or not host Z configures an EPA address, its packets destined to host A will eventually reach Server(A) 🌊. Server(A) has a mapping that lists Client(A) as the next hop toward EPA(A). It then encapsulates the packet with EPA(B) as the outer destination address and forwards it into the Internet 📮.
Internet routing carries this once-encapsulated packet to Server(B), which has a mapping that lists Client(B) as the next hop toward EPA(B). Server(B) encapsulates the packet with EPA(C) as the outer destination address and forwards it into the Internet 🌸. Internet routing then delivers this twice-encapsulated packet to Server(C), which has a mapping that lists Client(C) as the next hop toward EPA(C). Server(C) encapsulates the packet with ISP(D) as the outer destination address and forwards it into the Internet, and Internet routing delivers this triple-encapsulated packet to Client(C) 🧚.
When the triple-encapsulated packet arrives at Client(C), it strips the outer layer and forwards the twice-encapsulated packet to EPA(C), which is the locator address of Client(B) 🎈. Client(B) receives it, strips the outer layer, and forwards the once-encapsulated packet to EPA(B), which is the locator address of Client(A) 🫶. Finally, Client(A) strips the last outer layer and forwards the unencapsulated packet to EPA(A), which is the host address of host A. Packet delivered, we ate that 🍓💅.
7. Implications for the Internet 🌍
The IRON architecture is envisioning a hybrid routing and mapping system, and honestly it's giving best-of-both-worlds 🤩. It takes the shortest-path routing you get from pure dynamic routing systems and the routing-scaling suppression you get from pure mapping systems. So IRON is going after the elusive "sweet spot" that pure routing and pure mapping systems can't hit on their own 🎯.
The IRON system requires new routers and servers to be deployed throughout the Internet and/or provider networks, so the overlay networks stay well-balanced 🏗️. The good news is these routers and servers can be rolled out incrementally without disrupting the existing Internet infrastructure, and they can be managed to deliver acceptable service levels to customers 💖.
End-to-end traffic that goes through an IRON virtual overlay network may see delay variance between the initial packets and the subsequent packets of a flow 🌀. Why? Because the IRON system lets initial packets take a longer path stretch, and then makes timely route optimizations so later packets can use better next hop routers and servers 🚦.
IRON virtual overlay networks also play nicely with existing and emerging services in the native Internet ✨. Customers served by IRON virtual overlay networks get the same service that customers of non-IRON service providers enjoy 🫧. And Internet services already deployed in the native Internet don't need to change anything to accommodate IRON virtual overlay network customers 🙌.
The IRON system operates between routers within provider networks and end user networks 🏢. Inside these networks, the underlying paths that the virtual overlay networks travel may include links with varying MTUs 📏. The IRON system adds some per-packet overhead, which can make packets slightly larger than the underlying path can handle. IRON routers have a method for naturally detecting and tuning out all instances of path MTU underruns 🔍. In some cases these MTU underruns may need to be reported back to the original hosts 📣. The system will also allow MTUs much larger than those typically available in current Internet paths to be discovered and used, as more links with larger MTUs get deployed 🚀.
Finally, and perhaps most importantly, the IRON system provides built-in mobility management and multihoming capability 💜. This lets end user devices and networks move around freely while keeping routing oscillations minimal and generally maintaining shortest-path routes 🦋. The mobility management comes from the very nature of the IRON customer/provider relationship, so it doesn't need any adjunct mechanisms 🙅♀️. The mobility management and multihoming capabilities are further supported by forward-path reachability detection, which provides "hints of forward progress" in the same spirit as IPv6 Neighbor Discovery (ND) 🔮.
8. Additional Considerations 📚
Considerations for the scalability of Internet Routing due to multihoming, traffic engineering, and provider-independent addressing are discussed in [RADIR]. Other scaling considerations specific to IRON are discussed in Appendix B 🧐.
Route optimization considerations for mobile networks are found in [RFC5522] 🛰️.
9. Related Initiatives 🤝
IRON builds upon the concepts of the RANGER architecture [RFC5720] [RFC6139], and therefore inherits the same set of related initiatives 🌈. The Internet Research Task Force (IRTF) Routing Research Group (RRG) mentions IRON in its recommendation for a routing architecture [RFC6115] 🫶.
Virtual Aggregation (VA) [GROW-VA] and Aggregation in Increasing Scopes (AIS) [EVOLUTION] provide the basis for the Virtual Prefix concepts 🧩.
Internet Vastly Improved Plumbing (Ivip) [IVIP-ARCH] has contributed valuable insights, including the use of real-time mapping ⏱️. The use of Servers as mobility anchor points is directly influenced by Ivip's associated TTR mobility extensions [TTRMOB] 🥰.
[RO-CR] discusses a route optimization approach using a Correspondent Router (CR) model 🔁. The IRON Server construct is similar to the CR concept described in this work, though the way customer EUNs coordinate with Servers is different and based on the redirection model associated with NBMA links 🔀.
Numerous publications have proposed NAT traversal techniques 🛠️. The NAT traversal techniques adapted for IRON were inspired by the Simple Address Mapping for Premises Legacy Equipment (SAMPLE) proposal [SAMPLE] 💡.
10. Security Considerations 🔐
Okay, bestie, this section is SERIOUS. Read every line 🚨😭.
Security considerations that apply to tunneling in general are discussed in [V6OPS-TUN-SEC]. Additional considerations that apply also to IRON are discussed in RANGER [RFC5720] [RFC6139], VET [INTAREA-VET] and SEAL [INTAREA-SEAL] 🛡️.
The IRON system further depends on mutual authentication of IRON Clients to Servers and Servers to Relays 🤝. This is handled through initial authentication exchanges, followed by tunnel-neighbor nonces that can be used to detect off-path attacks 🕵️♀️. As for all Internet communications, the IRON system also depends on Relays acting with integrity and not injecting false advertisements into the BGP (for example, to mount traffic siphoning attacks) 🚫💀.
Each VPC overlay network requires a means for assuring the integrity of the interior routing system, so that all Relays and Servers in the overlay have a consistent view of Client<->Server bindings 🧷. Finally, Denial-of-Service (DoS) attacks on IRON Relays and Servers can occur when packets with spoofed source addresses arrive at high data rates 🌊😱. However, this issue is no different than for any border router in the public Internet today 🙃.
11. Acknowledgements 🫶
The ideas behind this work have benefited greatly from discussions with colleagues, some of which appear on the RRG and other IRTF/IETF mailing lists 💬. Robin Whittle and Steve Russert co-authored the TTR mobility architecture, which strongly influenced IRON 🙌. Eric Fleischman pointed out the opportunity to leverage anycast for discovering topologically close Servers 📍. Thomas Henderson recommended a quantitative analysis of scaling properties 📊.
The following individuals provided essential review input: Jari Arkko, Mohamed Boucadair, Stewart Bryant, John Buford, Ralph Droms, Wesley Eddy, Adrian Farrel, Dae Young Kim, and Robin Whittle 🎀💖.
12. References 📖
12.1. Normative References
- [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, September 1981 🧾.
- [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", RFC 2460, December 1998 ✨.
12.2. Informative References ✨
okay bestie, these are the extra reading materials 📚🦋 Not required to get the core spec, but they're the receipts behind the ideas. Some are still works in progress, so they could have changed by now 🙈
[BGPMON] net, B., "BGPmon.net - Monitoring Your Prefixes,
http://bgpmon.net/stat.php", June 2010.
[EVOLUTION]
Zhang, B., Zhang, L., and L. Wang, "Evolution Towards
Global Routing Scalability", Work in Progress,
October 2009.
[GROW-VA] Francis, P., Xu, X., Ballani, H., Jen, D., Raszuk, R., and
L. Zhang, "FIB Suppression with Virtual Aggregation", Work
in Progress, February 2011.
[INTAREA-SEAL]
Templin, F., Ed., "The Subnetwork Encapsulation and
Adaptation Layer (SEAL)", Work in Progress, February 2011.
[INTAREA-VET]
Templin, F., Ed., "Virtual Enterprise Traversal (VET)",
Work in Progress, January 2011.
[IVIP-ARCH]
Whittle, R., "Ivip (Internet Vastly Improved Plumbing)
Architecture", Work in Progress, March 2010.
[RADIR] Narten, T., "On the Scalability of Internet Routing", Work
in Progress, February 2010.
[RFC1070] Hagens, R., Hall, N., and M. Rose, "Use of the Internet as
a subnetwork for experimentation with the OSI network
layer", RFC 1070, February 1989.
[RFC2526] Johnson, D. and S. Deering, "Reserved IPv6 Subnet Anycast
Addresses", RFC 2526, March 1999.
[RFC3068] Huitema, C., "An Anycast Prefix for 6to4 Relay Routers",
RFC 3068, June 2001.
[RFC4192] Baker, F., Lear, E., and R. Droms, "Procedures for
Renumbering an IPv6 Network without a Flag Day", RFC 4192,
September 2005.
[RFC4271] Rekhter, Y., Li, T., and S. Hares, "A Border Gateway
Protocol 4 (BGP-4)", RFC 4271, January 2006.
[RFC4548] Gray, E., Rutemiller, J., and G. Swallow, "Internet Code
Point (ICP) Assignments for NSAP Addresses", RFC 4548,
May 2006.
[RFC4605] Fenner, B., He, H., Haberman, B., and H. Sandick,
"Internet Group Management Protocol (IGMP) / Multicast
Listener Discovery (MLD)-Based Multicast Forwarding
("IGMP/MLD Proxying")", RFC 4605, August 2006.
[RFC5214] Templin, F., Gleeson, T., and D. Thaler, "Intra-Site
Automatic Tunnel Addressing Protocol (ISATAP)", RFC 5214,
March 2008.
[RFC5522] Eddy, W., Ivancic, W., and T. Davis, "Network Mobility
Route Optimization Requirements for Operational Use in
Aeronautics and Space Exploration Mobile Networks",
RFC 5522, October 2009.
[RFC5720] Templin, F., "Routing and Addressing in Networks with
Global Enterprise Recursion (RANGER)", RFC 5720,
February 2010.
[RFC5743] Falk, A., "Definition of an Internet Research Task Force
(IRTF) Document Stream", RFC 5743, December 2009.
[RFC5887] Carpenter, B., Atkinson, R., and H. Flinck, "Renumbering
Still Needs Work", RFC 5887, May 2010.
[RFC6115] Li, T., "Recommendation for a Routing Architecture",
RFC 6115, February 2011.
[RFC6139] Russert, S., Fleischman, E., and F. Templin, "Routing and
Addressing in Networks with Global Enterprise Recursion
(RANGER) Scenarios", RFC 6139, February 2011.
[RO-CR] Bernardos, C., Calderon, M., and I. Soto, "Correspondent
Router based Route Optimisation for NEMO (CRON)", Work
in Progress, July 2008.
[SAMPLE] Carpenter, B. and S. Jiang, "Legacy NAT Traversal for
IPv6: Simple Address Mapping for Premises Legacy Equipment
(SAMPLE)", Work in Progress, June 2010.
[TTRMOB] Whittle, R. and S. Russert, "TTR Mobility Extensions for
Core-Edge Separation Solutions to the Internet's Routing
Scaling Problem,
http://www.firstpr.com.au/ip/ivip/TTR-Mobility.pdf",
August 2008.
[V6OPS-TUN-SEC]
Krishnan, S., Thaler, D., and J. Hoagland, "Security
Concerns With IP Tunneling", Work in Progress,
October 2010.Appendix A. IRON VPs over Internetworks with Different Address Families 🧚
okay so the IRON gets to lean on the routing system for its main trick 💅 It gives generally shortest-path routing for packets with EPA addresses coming from VPs that match the address family of the underlying Internetwork. but when the VPs are in an address family that the underlying Internetwork can't route (like OSI/NSAP [RFC4548] VPs running over an IPv4 Internetwork), you need a global mapping database so Servers can map each VP to a companion prefix from an address family that is routable. it's giving teamwork 🫶
For example, an IPv6 VP like 2001:DB8::/32 could be paired with a companion IPv4 prefix like 192.0.2.0/24, so encapsulated IPv6 packets can get forwarded over IPv4-only Internetworks 🌈
Every VP in the IRON is required to show up in a globally distributed Master VP database (MVPd) that keeps the VP-to-companion prefix mappings for all of them. The MVPd is run by a globally managed assigned numbers authority, the same way the Internet Assigned Numbers Authority (IANA) keeps the master list of all top-level IPv4 and IPv6 delegations. The database can be copied across multiple servers for load balancing, kinda like how FTP mirror sites handle software distributions 🍓
When a Server first starts up, it finds the full set of VPs for the IRON by reading the MVPd. It reads from a nearby server and checks back now and then for changes since the last read. Once it's read everything, the Server has the complete list of VP-to-companion prefix mappings ✨
The Server can then forward packets toward EPAs covered by a VP by encapsulating them in an outer header from the VP's companion prefix address family, and using any address from the companion prefix as the outer destination address. so the companion prefix basically acts as an anycast prefix 🦋
Possible encapsulations in this model include IPv6-in-IPv4, IPv4-in-IPv6, OSI/CLNP-in-IPv6, OSI/CLNP-in-IPv4, etc. 💖
Appendix B. Scaling Considerations 📈
okay the scaling stuff is so important for whether this works in real deployments 🔥 You have to think about a bunch of directions at once: interdomain core routing, handling tons of customer EUNs, traffic, and state requirements, etc. 💀
For routing scaling, each VPC advertises one or more VPs into the global Internet routing system, and the EPs get handed out from those VPs to customer EUNs. so routing scaling is at its best when each VP covers a lot of EPs. For example, the IPv6 prefix 2001:DB8::/32 contains 2^24 ::/56 EP prefixes for assignment to EUNs. That means the IRON could fit 2^32 ::/56 EPs with only 2^8 ::/32 VPs advertised into the interdomain routing core. (And when even longer EP prefixes are used, like /64s given to individual handsets on a cellular provider network, a huge number of EUNs can fit inside just one VP 🤯) Each VP also has an associated anycast companion prefix, so there's one anycast prefix advertised into the global routing system for each VP 🎀
For traffic scaling on Relays: each Relay represents an ASBR of a "shell" enterprise network. it just points arriving traffic packets with EPA destination addresses toward the Servers that look after customer EUNs. The Relay also sheds traffic bound for EPAs through redirection, which takes it off the path for the vast majority of traffic packets 🙌 Still, each Relay has to handle all traffic packets going between its customer EUNs and the non-IRON Internet. The scaling concerns for that second kind of traffic are no different from ASBR routers that connect big enterprise networks to the Internet 💅 For traffic scaling on Servers, each Server looks after a set of the VPC overlay network's customer EUNs. A Server handles all traffic packets destined to its EUNs, but it only handles the initial packets of flows that start from the EUNs and head to EPAs. So traffic scaling for EPA-addressed traffic is asymmetric, and it's proportional to how many EUNs each Server serves 🧠
For state on Relays: each Relay keeps a list of all Servers in the VPC overlay network, plus FIB entries for all customer EUNs that each Server serves. that means the state is mostly driven by how many EUNs are in the VPC overlay network. So you need to size the Relay to hold state for all EUNs when you're planning the VPC overlay network 📋 For state on Servers: each Server keeps tunnel-neighbor state for each of the customer EUNs it serves, but it doesn't need state for every EUN in the VPC overlay network. And neither Relays nor Servers need to keep state for final destinations of outbound traffic 🌸
Clients source and sink all traffic packets that come from or go to the customer EUN. So traffic scaling for Clients is the same as for any site border router 🫶 Clients also keep state for the Servers used for final destinations of outbound traffic flows. This can be managed as soft state, since stale entries that get purged from the cache will be refreshed the next time new traffic packets are sent 🦋
Author's Address
Fred L. Templin (editor)
Boeing Research & Technology
P.O. Box 3707 MC 7L-49
Seattle, WA 98124
USA
EMail: fltemplin@acm.org📦 where this page comes from
- ietf: rfc6179 2011-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms