RFC 6820
Address Resolution Problems in Large Data Center Networks
RFC 6820: Address Resolution Problems in Large Data Center Networks 💖
okay so bestie, this one is from January 2013 and it's all about address resolution (how a machine finds the link-layer address behind an IP address) when a data center has a massive number of hosts 🖥️✨ it's an Informational doc, so it's a "here's what's going wrong" write-up, not a rulebook.
Here's the title block tea ☕ It was written by T. Narten (IBM Corporation), M. Karir (Merit Network Inc.), and I. Foo (Huawei Technologies). The category is Informational, the ISSN is 2070-1721, and it's dated January 2013 🎀. The title block doesn't list anything it obsoletes or updates, so it doesn't replace or change any other RFC 🦋
Abstract ✨
okay so the abstract is giving "here's the scope, babe" 💅 It looks at address resolution issues that come up when data centers scale to a very large number of hosts. The focus is pretty narrow: address resolution, which means ARP for IPv4 and Neighbor Discovery (ND) for IPv6, happening inside a data center.
Status of This Memo 📜
this section is the fine print, so read it carefully bestie 🙈
This document is not an Internet Standards Track specification. It's published for informational purposes.
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are a candidate for any level of Internet Standard; see Section 2 of RFC 5741.
Info about the current status of this document, any errata, and how to give feedback on it lives at http://www.rfc-editor.org/info/rfc6820 🌸
Copyright Notice 🌈
Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
Table of Contents 🗂️
1. Introduction ....................................................3
2. Terminology .....................................................3
3. Background ......................................................4
4. Address Resolution in IPv4 ......................................6
5. Address Resolution in IPv6 ......................................7
6. Generalized Data Center Design ..................................7
6.1. Access Layer ...............................................8
6.2. Aggregation Layer ..........................................8
6.3. Core .......................................................9
6.4. L3/L2 Topological Variations ...............................9
6.4.1. L3 to Access Switches ...............................9
6.4.2. L3 to Aggregation Switches ..........................9
6.4.3. L3 in the Core Only ................................10
6.4.4. Overlays ...........................................10
6.5. Factors That Affect Data Center Design ....................11
6.5.1. Traffic Patterns ...................................11
6.5.2. Virtualization .....................................11
6.5.3. Summary ............................................12
7. Problem Itemization ............................................12
7.1. ARP Processing on Routers .................................12
7.2. IPv6 Neighbor Discovery ...................................14
7.3. MAC Address Table Size Limitations in Switches ............15
8. Summary ........................................................15
9. Acknowledgments ................................................16
10. Security Considerations .......................................16
11. Informative References ........................................161. Introduction ✨
okay so this section sets the stage 🎬 The doc is about scaling large data centers, and it zooms in on address resolution (ARP in IPv4, Neighbor Discovery in IPv6) inside the data center.
Here's the thing that's kind of tricky: strictly speaking, address resolution only works within a single L2 broadcast domain (ARP runs at L2, below IP). But it gets messy because routers have lots of interfaces that each need address resolution, and IEEE 802.1Q networks have individual VLANs that each act as their own L2 broadcast domain 🧩 So the scope of address resolution ends up covering both the L2 link and all the devices attached to those links. main character energy, honestly 💅
This doc points out potential issues with address resolution in data centers that have a huge number of hosts. The scope is intentionally pretty narrow because it matches the charter of the Address Resolution for Massive numbers of hosts in the Data center (ARMD) WG 🌷 It lists "pain points" that people are actually running into in current data centers. The goal is to stay focused on address resolution and not every other broader issue that could pop up in data centers 🙌
2. Terminology 📖
these are the vocab words you need to know, bestie 🍓
- Address Resolution: The process of determining the link-layer address corresponding to a given IP address. In IPv4, address resolution is performed by ARP [RFC0826]; in IPv6, it is provided by Neighbor Discovery (ND) [RFC4861].
- Application: Software that runs on either a physical or virtual machine, providing a service (e.g., web server, database server, etc.).
- L2 Broadcast Domain: The set of all links, repeaters, and switches that are traversed to reach all nodes that are members of a given L2 broadcast domain. In IEEE 802.1Q networks, a broadcast domain corresponds to a single VLAN.
- Host (or server): A computer system on the network.
- Hypervisor: Software running on a host that allows multiple VMs to run on the same host.
- Virtual Machine (VM): A software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Applications (generally) do not know they are running on a VM as opposed to running on a "bare" host or server, though some systems provide a paravirtualization environment that allows an operating system or application to be aware of the presence of virtualization for optimization purposes.
- ToR: Top-of-Rack Switch. A switch placed in a single rack to aggregate network connectivity to and from hosts in that rack.
- EoR: End-of-Row Switch. A switch used to aggregate network connectivity from multiple racks. EoR switches are the next level of switching above ToR switches.
part 1 of 5 done! 🎀 more parts coming, so stay tuned, bestie 🫶
3. Background ✨
okay so big flat L2 networks have had scaling problems forever 😭. The bigger an L2 broadcast domain gets, the more broadcast traffic protocols like ARP create. That's a real burden because every device (switch, host, and router) has to process that traffic and possibly act on it. In extreme cases you get "broadcast storms" 💀, where the broadcast traffic hits a level that basically takes down part or all of a network. Poor loop detection and prevention, or misconfigurations, can set this up as network conditions change.
The conventional wisdom was basically "don't do that" 🙈. Split big L2 networks into lots of smaller ones, each running as its own L3/IP subnet. Plenty of data center networks were designed this way, like putting each rack in its own IP subnet. That keeps the broadcast domain (and address resolution) confined to one ToR switch, which scales really well. Unfortunately, it clashes with the current trend toward dynamic workload shifting and more virtualization, which we'll get into.
Workload placement is a challenge in data centers 😩. Ideally you'd dynamically reassign workloads to optimize server utilization, add servers when demand goes up, and so on. But servers are often preconfigured with a given set of IP addresses, so where they can be placed is limited by the data center's IP addressing restrictions. For example, servers configured with addresses from a particular subnet can only be placed where they connect to that IP subnet. If each ToR switch acts as the gateway for its own subnet, a server can only connect to that one ToR switch. That gateway switch is the L2/L3 boundary. A similar constraint shows up in virtualized environments, as we'll see next.
Server virtualization is fast becoming the norm ✨. Each physical server supports multiple virtual machines (VMs), and each VM runs its own operating system, middleware, and applications. Virtualization is a huge enabler of workload agility 🦋: any server can host any application (on its own VM), and VMs can be added, shrunk, or moved within the physical infrastructure. The perks are awesome, including higher utilization, increased data security, reduced user downtime, even significant power conservation, and the promise of a more flexible and dynamic computing environment 🌈.
This discussion focuses on VM placement and migration. But keep in mind that even without virtualization, many of the same issues apply to individual workloads on standalone machines. For example, when you increase the number of servers running a particular workload to meet demand, placement can be constrained by IP subnet numbering, as discussed earlier.
The greatest flexibility in VM and workload management happens when you can place a VM (or workload) anywhere in the data center, no matter what IP addresses it uses or how the physical network is laid out 🧚. In practice, moving VMs is easiest when placement and movement don't conflict with the IP subnet boundaries of the data center's network. That way the VM's IP address doesn't need to change to reflect its actual point of attachment from an L3/IP perspective. If a VM moves to a new IP subnet, its address must change, and clients will need to be made aware of that change. From a VM management perspective, things are simplified if all servers are on a single large L2 network 🫶.
With virtualization, it's not uncommon for a single physical server to host ten or more VMs, each with its own IP and Media Access Control (MAC) addresses 🍓. So the number of addresses per machine (and hence per subnet) is going up, even when the number of physical machines stays constant. In a few years, the numbers will likely be even higher 📈.
Back in the day, applications were static in the sense that they tended to stay in one physical place 🐢. An application installed on a physical machine stayed there, because moving it somewhere else was generally expensive. Physical servers hosting applications were also placed to facilitate communication locality. Applications were physically located near the servers hosting the applications they talked to most. Network traffic patterns could be optimized, sometimes keeping significant traffic local to one network segment. In these more static and carefully managed environments, it was possible to build networks that approached scaling limitations without actually crossing the threshold.
Today, with the proliferation of VMs, traffic patterns are getting more diverse and less predictable 🌀. There can easily be less locality, because VMs hosting applications get moved for reasons like reducing overall power usage (by consolidating VMs and powering off idle machines) or moving a VM to a physical server with more capacity or a lower load. In today's changing environments, it's getting harder to engineer networks, since traffic patterns keep shifting as VMs move around 💅.
So in summary, both the size and the density of L2 networks are increasing 📊. Increasingly dynamic workloads and more VM usage are pushing for ever-larger L2 networks. Today, there are already data centers with over 100,000 physical machines and many times that number of VMs, and this number will only increase going forward. Traffic patterns within a data center are also constantly changing 💜. Ultimately, the issues described in this document might be observed at any scale, depending on the particular design of the data center.
4. Address Resolution in IPv4
In IPv4 over Ethernet, ARP provides the function of address resolution 🎀. To determine the link-layer address of a given IP address, a node broadcasts an ARP Request. The request is delivered to all portions of the L2 network, and the node with the requested IP address responds with an ARP Reply. ARP is an old protocol and, by current standards, it's sparsely documented. For example, there are no clear requirements for retransmitting ARP Requests when there are no replies. Consequently, implementations vary in the details of what they actually implement [RFC0826][RFC1122].
From a scaling perspective, ARP has a bunch of problems 😬. First, it uses broadcast, so any network with a large number of attached hosts will see a correspondingly large amount of broadcast ARP traffic. The second problem is that it's not feasible to change host implementations of ARP, because current implementations are too widely entrenched. Any changes to host ARP implementations would take years to become deployed enough to matter. That said, it may be possible to change ARP implementations in hypervisors, L2/L3 boundary routers, and/or ToR access switches, to leverage techniques like Proxy ARP. Finally, ARP implementations need to take steps to flush out stale or otherwise invalid entries.
Unfortunately, existing standards don't provide clear implementation guidelines for how to do this 😵. So implementations vary significantly, and some are "chatty," meaning they just periodically flush caches every few minutes and send new ARP queries.
5. Address Resolution in IPv6
Broadly speaking, from an address resolution perspective, IPv6's Neighbor Discovery (ND) behaves a lot like ARP, with a few notable differences 💅. First, ARP uses broadcast, whereas ND uses multicast. When querying for a target IP address, ND maps the target address into an IPv6 Solicited Node multicast address. Multicast has the benefit over broadcast that multicast frames don't necessarily need to go to all parts of the network. They can be sent only to segments where listeners for the Solicited Node multicast address live 🦋.
In the case where multicast frames are delivered to all parts of the network, sending to a multicast address still has an advantage. Most (if not all) nodes will filter out the unwanted multicast query via filters installed in the Network Interface Card (NIC), rather than burdening host software with the need to process such packets. So whereas all nodes must process every ARP query, ND queries are only processed by the nodes they're intended for ✨. In cases where multicast filtering can't effectively be implemented in the NIC (e.g., on hypervisors supporting virtualization), filtering would need to be done in software (e.g., in the hypervisor's vSwitch) 🫶.
6. Generalized Data Center Design
There are many different ways data center networks might be designed 🏗️. The designs are usually engineered to suit the particular workloads being deployed in the data center. For example, a large web server farm might be engineered in a very different way than a general-purpose multi-tenant cloud hosting service. However, in most cases the designs can be abstracted into a typical three-layer model consisting of an access layer, an aggregation layer, and the Core. The access layer generally refers to the switches closest to the physical or virtual servers. The aggregation layer serves to interconnect multiple access-layer devices. The Core switches connect the aggregation switches to the larger network core 👑.
Figure 1 shows a generalized data center design, which captures the essential elements of various alternatives 🧚.
+-----+-----+ +-----+-----+
| Core0 | | Core1 | Core
+-----+-----+ +-----+-----+
/ \ / /
/ \----------\ /
/ /---------/ \ /
+-------+ +------+
+/------+ | +/-----+ |
| Aggr11| + --------|AggrN1| + Aggregation Layer
+---+---+/ +------+/
/ \ / \
/ \ / \
+---+ +---+ +---+ +---+
|T11|... |T1x| |TN1| |TNy| Access Layer
+---+ +---+ +---+ +---+
| | | | | | | |
+---+ +---+ +---+ +---+
| |... | | | | | |
+---+ +---+ +---+ +---+ Server Racks
| |... | | | | | |
+---+ +---+ +---+ +---+
| |... | | | | | |
+---+ +---+ +---+ +---+
Typical Layered Architecture in a Data Center
Figure 16.1. Access Layer 🧚
okay so the access switches are the ones that hook up directly to physical and virtual servers 🖥️✨ the access layer can be built by wiring the servers in a rack to a ToR switch, or (less common, ngl) the servers can get wired straight to an EoR switch 🎀 a server rack might have one single uplink to one access switch, or it can have dual uplinks to two different access switches 💖
6.2. Aggregation Layer 🦋
in a typical data center, aggregation switches are the ones that interconnect a whole bunch of ToR switches 🌸 usually there are multiple parallel aggregation switches serving the same group of ToRs, so the load gets shared 🫶 and it's no longer uncommon to see aggregation switches connecting hundreds of ToR switches in big data centers 😭🔥
6.3. Core 👑
core switches are the bridge between aggregation switches and the main data center network 🌈 they interconnect different sets of racks, and they also provide connectivity to data center gateways that lead out to external networks 💅
6.4. L3/L2 Topological Variations 🍓
6.4.1. L3 to Access Switches
okay this one is giving simple 🧚 the L3 domain gets extended all the way from the core network down to the access switches 💜 each rack enclosure is a single L2 domain, and that L2 domain is confined to the rack itself 🎀 generally there are no significant ARP/ND scaling issues here, because the L2 domain can't grow very large 🙈 this topology has benefits when the servers attached to a particular access switch mostly run VMs that are confined to a single subnet 🦋 those VMs and the apps they host aren't moved (migrated) to other racks that might be attached to different access switches (and different IP subnets) 🚫 a small server farm or a very static compute cluster might be super well served by this design ✨
6.4.2. L3 to Aggregation Switches
when the L3 domain only extends to aggregation switches, hosts in any of the IP subnets configured on the aggregation switches can be reached via L2 through any access switch, if the access switches enable all the VLANs 🌸 this gives a greater level of flexibility, because servers attached to any access switch can run any VMs that have been provisioned with IP addresses configured on the aggregation switches 💖 in this kind of environment, VMs can migrate between racks without IP address changes 🫶 the drawback, though, is that multiple VLANs have to be enabled on all access switches and all access-facing ports on aggregation switches 😬 even though L2 traffic is still partitioned by VLANs, the fact that all VLANs are enabled on all ports can lead to broadcast traffic on all VLANs that traverse all links and ports 📢 which has the same effect as one big L2 domain on the access-facing side of the aggregation switch 💀 on top of that, the internal traffic itself might have to cross different L2 boundaries, which can result in significant ARP/ND load at the aggregation switches 🔥 overall this design provides a good tradeoff between flexibility and L2 domain size ⭐ a moderate-sized data center might use this approach to provide high-availability services at a single location 🌈
6.4.3. L3 in the Core Only
sometimes a wider range of VM mobility is desired (so, a greater number of racks where VMs can move around without IP address changes) 🦋 in that case, the L3 routed domain might be terminated at the core routers themselves 👑 VLANs can then span multiple groups of aggregation switches, which lets hosts move among a greater number of server racks without IP address changes 🧚 this scenario results in the largest ARP/ND performance impact, as explained later 😭 a data center with very rapid workload shifting may consider this kind of design 💅
6.4.4. Overlays
there are several approaches where overlay networks can be used to build very large L2 networks, which helps enable VM mobility 🌈 overlay networks using various L2 or L3 mechanisms let the interior switches/routers mask host addresses 🙈 L3 overlays can also help the data center designer control the size of the L2 domain, and they enhance the ability to provide multi-tenancy in data center networks 💜 however, using overlays does not eliminate traffic associated with address resolution 🙅♀️ it just moves it into regular data traffic 🔁 that means address resolution is implemented in the overlay and isn't directly visible to the switches of the data center network 👀
a potential problem in a large data center is that when a large number of hosts talk to their peers in different subnets, all of those hosts send (and receive) data packets to their respective L2/L3 boundary nodes 😱 that's because the traffic flows are generally bidirectional 🔄 this has the potential to further highlight any scaling problems 📈 these L2/L3 boundary nodes have to process ARP/ND requests sent from originating subnets, and resolve physical (MAC) addresses in the target subnets for what are generally bidirectional flows 🥹 so, for maximum flexibility in managing the data center workload, it's often desirable to use overlays to place related groups of hosts in the same topological subnet, which avoids the L2/L3 boundary translation 🫶 the use of overlays in the data center network can be a useful design mechanism for managing a potential bottleneck at the L2/L3 boundary, by redefining where that boundary exists 🎀
6.5. Factors That Affect Data Center Design 💅
6.5.1. Traffic Patterns
expected traffic patterns play a huge role in designing appropriately sized access, aggregation, and core networks 📊 traffic patterns also vary based on the expected use of the data center 🌸
broadly speaking, it's desirable to keep as much traffic as possible on the access layer, so the bandwidth usage at the aggregation layer stays low 🧚 if the data center is expected to serve as a large web server farm, where thousands of nodes are doing similar things and the traffic is mostly in and out of the data center, an access layer with EoR switches might be used 🖥️ it minimizes complexity, allows servers and databases to be located in the same L2 domain, and provides maximum density 💖
a data center that's expected to host a multi-tenant cloud hosting service might have some completely unique requirements 🦄 to isolate inter-customer traffic, smaller L2 domains might be preferred, and even though the overall size of the data center might be comparable to the previous example, the multi-tenant nature of the cloud hosting app requires a smaller and more compartmentalized access layer 🔐 a multi-tenant environment might also require L3 all the way to the access-layer ToR switch 👑
yet another example of a unique traffic pattern is a high-performance compute cluster, where most of the traffic stays within the cluster but there's a high degree of crosstalk between the nodes 🔥 this would once again call for a large access layer, to minimize the requirements at the aggregation layer 💜
6.5.2. Virtualization
using virtualization in the data center further increases the densities that can be achieved 📈 but virtualization also complicates the requirements on the access layer 😵💫 that's because it restricts the scope of server placement when there's server failover from hardware failures, or server migration for load balancing or other reasons 🌪️
virtualization can also put additional requirements on the aggregation switches, in terms of address resolution table size and the scalability of any address-learning protocols that might be used on those switches 🧠 it often also requires additional VLANs for high-availability beaconing, and those VLANs need to span the entire virtualized infrastructure 🛰️ which means the access layer also has to span the entire virtualized infrastructure 🫠
6.5.3. Summary
the designs in this section all come with a bunch of tradeoffs 🎭 the "L3 to access switches" design from Section 6.4.1 is the only design that constrains L2 domain size in a way that avoids ARP/ND scaling problems ✨ but that design has limitations, and it doesn't address some of the other requirements that lead to configurations using larger L2 domains 😤 so ARP/ND scaling issues are a real problem in practice 💀
7. Problem Itemization 🔥
this section lays out some specific problems, or "pain points", that are related to large data centers 💅🫶
7.1. ARP Processing on Routers ✨
okay so the first pain point is kinda dramatic 😭. When a big L2 broadcast domain has a ton of ARP traffic, the routers plugged into it can end up doing a LOT of ARP work. Most of that traffic isn't even for the router, but it still has to read enough of each ARP Request to figure out it can ignore it 💅. ARP says a recipient must update its ARP cache if it gets an ARP query from a source it already has an entry for [RFC0826] 🦋.
ARP processing on routers is usually handled by a "slow path" software processor, not by a hardware Application-Specific Integrated Circuit (ASIC) like the one that forwards packets 🔥. That design really limits how fast ARP traffic gets handled, way slower than how fast the ASICs can forward stuff. At the time of this writing, current implementations can support ARP processing in the low thousands of ARP packets per second 💀. In some deployments, people have said that this limit is a real problem 🙈.
To cut down the ARP load, some routers have added extra optimizations in their forwarding ASIC paths 🧚. For example, some routers can be set up to discard ARP Requests for target addresses other than the ones assigned to the router. That way, the router's software processor only sees ARP Requests for addresses it owns and has to answer 👑. This can seriously reduce how many ARP Requests the router has to process 💜.
Another optimization is about reducing the number of ARP queries aimed at routers, whether they're for address resolution or to check existing cache entries 🫶. Some routers can be set up to broadcast periodic gratuitous ARPs [RFC5227]. When a gratuitous ARP comes in, implementations mark the matching entry as "fresh" and reset its aging timer to the max setting 🌸. So sending out periodic gratuitous ARPs can actually stop nodes from needing to send ARP Requests just to recheck stale entries for a router. The end result is fewer ARP queries hitting the router 🎀. Gratuitous ARPs get broadcast to every node in the L2 broadcast domain, so they might also pre-populate ARP caches on neighboring devices, which cuts down on ARP traffic even more ✨. But it is not believed that most implementations support pre-population of ARP entries, because the ARP specification [RFC0826] only recommends updating ARP entries that already exist when ARP messages arrive. It doesn't call for creating new entries when none exist yet 🙃.
Last but not least, there's the overhead of processing IP packets when there's no ARP entry for them 😩. Existing standards say that one or more IP packets with no ARP entry should be queued while address resolution finishes [RFC1122] [RFC1812]. Once the ARP query is resolved, the queued packets can be forwarded on 🦋. But again, handling these packets happens in the "slow path," which limits how fast a router can deal with ARP "cache misses," and that's seen as a problem in some deployments today 💅. Also, if no response comes back, the router may send the ARP/ND query multiple times. If there's still no response after a number of ARP/ND requests, the router needs to drop any queued data packets and may send an ICMP destination unreachable message as well [RFC0792]. This whole process can be CPU intensive 🥵.
Even though address resolution traffic stays local to one L2 network, some data center designs end L2 domains at individual aggregation switches/routers (for example, see Section 6.4.2) 🍓. Those routers can have a lot of interfaces connected (say, 100 or more) 🔥. The address resolution traffic on any one interface might be fine, but the total across all interfaces can become a real problem 😭.
There's also a variant where individual routers serve a small number of interfaces, but each interface is attached to a very large subnet 🌈. Once again, the problem is the total ARP traffic across all of the router's interfaces. It's basically the same pain point as above, just with a few big IP subnets instead of lots of smaller ones 💖.
When hosts in two different subnets under the same L2/L3 boundary router need to talk to each other, the L2/L3 router doesn't just have to start ARP/ND requests toward the target's subnet. It also has to process the ARP/ND requests coming from the originating subnet 🙈. That adds even more to the overall ARP processing load 💀.
7.2. IPv6 Neighbor Discovery ✨
IPv6's Neighbor Discovery acts a lot like ARP, but there are a few big differences, and those lead to a different set of potential issues 🧚. From an L2 point of view, an important difference is that ND address resolution requests get sent via multicast. That means ND queries only get processed by the nodes they're meant for 👑. Compared to broadcast ARPs, that cuts down the total number of ND packets an implementation has to receive 💜.
Another key difference is about revalidating stale ND entries 🫶. ND requires nodes to periodically revalidate any entries they're using, so bad entries get timed out fast enough that TCP doesn't kill the connection 🌸. Because of this, some implementations send out "probe" ND queries to validate in-use ND entries as often as every 35 seconds [RFC4861] 🎀. Unlike ARP, these probes are sent via unicast. But on larger networks, probes can make routers receive a ton of these queries, way more than with ARP, which doesn't specify that kind of behavior ✨. Unfortunately, the IPv4 trick of sending gratuitous ARPs (described in Section 7.1) doesn't work in IPv6 😭. The ND specification says outright that gratuitous ND "updates" can't make an ND entry "valid." Instead, those entries get marked "probe," which makes the receiving node (eventually) send a probe back to the sender. That's exactly the behavior the router is trying to prevent 💀!
Routers implementing Neighbor Unreachability Discovery (NUD) (for neighboring destinations) will need to process neighbor cache state changes, like moving entries from REACHABLE to STALE 🦋. How that's implemented can affect how well ND scales on a router 🔥. For instance, one possible approach is to have the forwarding operation detect when an ND entry gets referenced that needs to move from REACHABLE to STALE, then signal an event the software processor has to handle. That kind of implementation could increase the load on the service processor, a lot like high rates of ARP requests have caused problems on some routers 😩.
It should be noted that ND doesn't require sending probes in every case 💅. Section 7.3.1 of [RFC4861] describes a technique where hints from TCP can be used to confirm that an existing ND entry is working fine and doesn't need to be revalidated 🫶.
Finally, IPv6 and IPv4 are often run at the same time, in parallel on the same network, which is called dual-stack mode 👯. In those environments, the IPv4 and IPv6 issues listed above make each other worse 🙈.
7.3. MAC Address Table Size Limitations in Switches ✨
L2 switches keep L2 MAC address forwarding tables for all the sources and destinations passing through the switch 🍓. These tables get filled in through learning, and they're used to forward L2 frames to the right destination 🌈. The bigger the L2 domain, the bigger the tables have to be 📈. In theory, a switch only needs to track addresses it's actively using (sometimes called "conversational learning"). But switches flood broadcast frames (like the ones from ARP), multicast frames (like the ones from Neighbor Discovery), and unicast frames going to unknown destinations 🌊. Switches also add entries for the source addresses of those flooded frames to their forwarding tables 💖. So MAC address table size can become a problem as the L2 domain grows 😭. It gets even worse with VMs, where one physical machine hosts many VMs (in the 10's today, but growing fast as the number of cores per CPU goes up), because every VM has its own MAC address that switches can see 🙈.
When L3 extends all the way to access switches (see Section 6.4.1), MAC address table size in switches generally isn't a problem 🫶. When L3 only extends to aggregation switches (see Section 6.4.2), though, MAC table size limits can be a real issue 💀.
8. Summary 🎀
This document has laid out a bunch of issues related to address resolution in large data centers 🌸. It covered different scenarios where those issues might pop up and what the potential problems are, and it also explained the fundamental factors that cause them 🔥. The hope is that describing specific pain points will kick off a discussion about whether they should be addressed and how to do it best 👑.
9. Acknowledgments 🫶
This document got a major glow-up thanks to comments from Manav Bhatia, David Black, Stewart Bryant, Ralph Droms, Linda Dunbar, Donald Eastlake, Wesley Eddy, Anoop Ghanwani, Joel Halpern, Sue Hares, Pete Resnick, Benson Schliesser, T. Sridhar, and Lucy Yong 💖. Igor Gashinsky deserves extra credit 🏆. He highlighted some of the ARP-related pain points and cleared up the difference between what the standards require and what some router vendors have actually implemented in response to operator requests ✨.
10. Security Considerations 💀
This document does not create any security implications nor does it have any security implications 🙈. The security vulnerabilities in ARP are well known, and this document does not change or mitigate them in any way 😭. Security considerations for Neighbor Discovery are discussed in [RFC4861] and [RFC6583] 🔒.
11. Informative References 📚
[RFC0792] Postel, J., "Internet Control Message Protocol", STD 5,
RFC 792, September 1981.
[RFC0826] Plummer, D., "Ethernet Address Resolution Protocol: Or
converting network protocol addresses to 48.bit Ethernet
address for transmission on Ethernet hardware", STD 37,
RFC 826, November 1982.
[RFC1122] Braden, R., "Requirements for Internet Hosts -
Communication Layers", STD 3, RFC 1122, October 1989.
[RFC1812] Baker, F., "Requirements for IP Version 4 Routers",
RFC 1812, June 1995.
[RFC4861] Narten, T., Nordmark, E., Simpson, W., and H. Soliman,
"Neighbor Discovery for IP version 6 (IPv6)", RFC 4861,
September 2007.
[RFC5227] Cheshire, S., "IPv4 Address Conflict Detection", RFC 5227,
July 2008.
[RFC6583] Gashinsky, I., Jaeggli, J., and W. Kumari, "Operational
Neighbor Discovery Problems", RFC 6583, March 2012.Authors' Addresses 🎀
okay bestie, the authors are the squad who wrote this whole thing, so here's how to reach them. 💖
Thomas Narten
IBM Corporation
3039 Cornwallis Ave.
PO Box 12195
Research Triangle Park, NC 27709-2195
USA
EMail: narten@us.ibm.com
Manish Karir
Merit Network Inc.
EMail: mkarir@merit.edu
Ian Foo
Huawei Technologies
EMail: Ian.Foo@huawei.comThomas, Manish, and Ian are literally the legends who put this RFC together, so slide into their inboxes if you've got questions 🦋✨
📦 where this page comes from
- ietf: rfc6820 2013-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms