RFC 7342
Practices for Scaling ARP and Neighbor Discovery (ND) in Large Data Centers
Practices for Scaling ARP and Neighbor Discovery (ND) in Large Data Centers ✨
okay bestie, this RFC is an Independent Submission 💅 (so it's not from the official IETF stream), numbered RFC 7342, and it's Informational 🎀. That means it's sharing practical tips, not a rule you have to follow. The ISSN is 2070-1721 and it came out in August 2014 🌸. The authors are L. Dunbar (Huawei), W. Kumari (Google), and I. Gashinsky (Yahoo) 👑. The title block doesn't list any Obsoletes or Updates entries, so this doc isn't replacing or changing anything else.
Abstract 🦋
This memo writes down some operational practices that help ARP and Neighbor Discovery (ND) scale up in data center environments 💖.
Status of This Memo 📜
This document is not an Internet Standards Track specification, so it's published purely for info 🧚.
This is a contribution to the RFC Series, separate from any other RFC stream. The RFC Editor picked to publish it at their discretion and doesn't say anything about how useful it is for implementing or deploying 🫶. Anything the RFC Editor approves for publication is not a candidate for any level of Internet Standard, see Section 2 of RFC 5741 💜.
If you want the current status, any errata, or a way to give feedback, check http://www.rfc-editor.org/info/rfc7342 🌈.
Copyright Notice 🎀
Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved 💅.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info), as they were in effect on the day this was published. Read those carefully, because they explain what rights and restrictions you have with this doc 🙈.
Table of Contents 🗂️
1. Introduction ....................................................2
2. Terminology .....................................................4
3. Common DC Network Designs .......................................4
4. Layer 3 to Access Switches ......................................5
5. Layer 2 Practices to Scale ARP/ND ...............................5
5.1. Practices to Alleviate APR/ND Burden on L2/L3
Boundary Routers ...........................................5
5.1.1. Communicating with a Peer in a Different Subnet .....6
5.1.2. L2/L3 Boundary Router Processing of Inbound
Traffic .............................................7
5.1.3. Inter-Subnet Communications .........................8
5.2. Static ARP/ND Entries on Switches ..........................8
5.3. ARP/ND Proxy Approaches ....................................9
5.4. Multicast Scaling Issues ...................................9
6. Practices to Scale ARP/ND in Overlay Models ....................10
7. Summary and Recommendations ....................................10
8. Security Considerations ........................................11
9. Acknowledgements ...............................................11
10. References ....................................................12
10.1. Normative References .....................................12
10.2. Informative References ...................................131. Introduction ✨
Okay so the big idea is that this memo writes down some practices that help ARP/ND scale in data centers 🧚.
As [RFC6820] explains, data centers keep getting more into rapid workload shifting and server virtualization 🔥. That means servers get loaded (or reloaded) with different Virtual Machines (VMs) or applications at different times. VMs on the same physical server can have different IP addresses, and they might even be in different IP subnets 🦋.
So a physical server can hold VMs in different subnets, and VMs can move between server racks without anyone reconfiguring their IP addresses 💖. To make that work, the networks need to turn on multiple broadcast domains (lots of VLANs) on the interfaces of L2/L3 boundary routers and Top-of-Rack (ToR) switches, and let some subnets span multiple router ports 🌸.
Note: L2/L3 boundary routers, as this document talks about them, can forward IEEE 802.1 Ethernet frames (Layer 2) without changing the Media Access Control (MAC) header 👀. When subnets span multiple ports on those routers, they still count as "single-link" subnets, specifically the multi-access link model that [RFC4903] recommends 💅. They're different from the "multi-link" subnets described in [Multi-Link] and RFC 4903. Those are subnets where different physical media share the same prefix and connect to one router. Inside a "multi-link" subnet from RFC 4903, Layer 2 frames from one port can't be natively forwarded to another port without a header change 🙅♀️.
Here's the catch 😭: when the total number of VMs (or hosts) across all those subnets gets big, address resolution (IPv4 ARP and IPv6 ND) can start to have scaling issues 💀. Three main issues come up with ARP/ND address resolution when subnets span multiple L2/L3 boundary router ports 🫶:
- ARP/ND messages get flooded to lots of physical link segments, which can cut down the bandwidth left for user traffic 🌈.
- ARP/ND processing puts load on the L2/L3 boundary routers 🔥.
- In IPv4, every end station in a subnet receives ARP broadcast messages from every other end station in that subnet. IPv6 ND fixed this problem by using multicast 💜.
Most data center servers are moving to 1G or 10G ports now 🎀. So even when ARP/ND messages are flooded to all physical links, they use a negligible amount of bandwidth compared to the link capacity ✨. IGMP/MLD (Internet Group Management Protocol and Multicast Listener Discovery) snooping [RFC4541] can also cut the ND multicast traffic going to some physical link segments 🧚.
Modern servers have more computing power every year 💪, so handling lots of ARP broadcast messages matters a lot less to them now. For example, lab testing showed that 2000 ARP requests per second only uses 2% of a single-core CPU server 🙈. That's why the impact of ARP broadcasts on end stations isn't significant on today's servers 💅.
Statistics from Merit Network [ARMD-Statistics] show that a big pile of mobile VMs in a data center mostly hits the L2/L3 boundary routers 😭. That's issue 2 from the list above 🔥.
So this memo documents some simple practices that can scale ARP/ND in a data center, especially the ones that cut the processing load on L2/L3 boundary routers 👑✨.
2. Terminology 📚
This document reuses a lot of terminology from [RFC6820]. A bunch of the definitions are repeated here so you don't have to flip back and forth 🫶.
- ARP: IPv4 Address Resolution Protocol [RFC826] 🌸
- Aggregation Switch: A Layer 2 switch that interconnects ToR switches 🦋
- Bridge: An IEEE802.1Q-compliant device. In this document, "Bridge" is used interchangeably with "Layer 2 switch" 🧚
- DC: Data Center 💖
- DA: Destination Address 🎀
- End Station: A VM or physical server whose address is either the destination or the source of a data frame 👀
- EoR: End-of-Row switches in a data center 🌈
- NA: IPv6 Neighbor Advertisement 💅
- ND: IPv6 Neighbor Discovery [RFC4861] ✨
- NS: IPv6 Neighbor Solicitation 🫶
- SA: Source Address 💜
- ToR: Top-of-Rack Switch (also known as an access switch) 🍓
- UNA: IPv6 Unsolicited Neighbor Advertisement 🙈
- VM: Virtual Machine 🔥
- Subnet: Refers to the multi-access link subnet referenced by RFC 4903 🌸
3. Common DC Network Designs 🏗️
There are a few common network designs for a data center 💅:
- Layer 3 connectivity to the access switch 🎀
- Large Layer 2 🌈
- Overlay models 🧚
Honestly, there's no single network design that fits every case 🙃. The next sections go through some of the common practices for scaling address resolution under each design 💖.
4. Layer 3 to Access Switches 🛰️
This design puts Layer 3 on the access switches, so the access switches become the L2/L3 boundary routers for the VMs attached to them 👑.
As [RFC6820] describes, lots of data centers are built so that ARP/ND broadcast/multicast messages stay confined to a few ports (interfaces) on the access switches (the ToR switches) 🫶.
There's another version of the Layer 3 solution, where Layer 3 runs all the way down to the servers (or even to the VMs). This keeps the ARP/ND broadcast/multicast messages confined to the small number of VMs inside a server 🦋.
Advantage: Both ARP and ND scale well here. There's no address resolution issue in this design ✨.
Disadvantage: The main downside of this design shows up during VM movement 😬. When VMs get moved to a different location, either the VMs need an address change, or the switches/routers need a configuration change 💀.
Summary: This solution fits best for data centers with a static workload, and/or network operators who can reconfigure IP addresses/subnets on switches before any workload change happens 🎀. No protocol changes are suggested 💅.
5. Layer 2 Practices to Scale ARP/ND 🌊
5.1. Practices to Alleviate APR/ND Burden on L2/L3 Boundary Routers 🛡️
Broadcast and multicast ARP/ND messages inside a Layer 2 domain can hurt the L2/L3 boundary routers, especially when there are lots of VMs and subnets 😩. This section covers some practices people commonly use to cut down the ARP/ND processing those routers have to do 🫶✨.
5.1.1. Communicating with a Peer in a Different Subnet ✨
Scenario: okay so, when the originating end station doesn't have its default gateway MAC address in its ARP/ND cache and needs to talk to a peer in a different subnet, it has to send ARP/ND requests to its default gateway router to resolve the router's MAC address. If the gateway router has lots of subnets, and a huge number of end stations in those subnets don't have the gateway MAC address in their ARP/ND caches, the gateway router ends up processing a massive number of ARP/ND requests 😭. This is often CPU intensive, since ARP/ND messages are usually processed by the CPU (and not in hardware).
Note: any centralized configuration that preloads the default MAC addresses is not included in this scenario 🙈.
Solution: for IPv4 networks, a practice to ease this problem is to have the L2/L3 boundary router send periodic gratuitous ARP [GratuitousARP] messages, so all the connected end stations can refresh their ARP caches 🌸. As a result, most (if not all) end stations won't need to send ARP requests for the gateway routers when they want to talk to external peers.
For the scenario above, IPv6 end stations are still required to send unicast ND messages to their default gateway router (even with those routers periodically sending Unsolicited Neighbor Advertisements) because IPv6 requires bidirectional path validation 🔐.
Advantage: this practice reduces the ARP requests the L2/L3 boundary router has to process for IPv4 💖.
Disadvantage: this practice doesn't reduce ND processing on the L2/L3 boundary router for IPv6 traffic 💀.
Recommendation: if the network is an IPv4-only network, this approach can be used. For an IPv6 network, one needs to consider the work described in [RFC7048]. Note: ND and Secure Neighbor Discovery (SEND) [RFC3971] use the bidirectional nature of queries to detect and prevent security attacks 🛡️.
5.1.2. L2/L3 Boundary Router Processing of Inbound Traffic 🦋
Scenario: when an L2/L3 boundary router gets a data frame destined for a local subnet, and the destination isn't in the router's ARP/ND cache, some routers hold the packet and trigger an ARP/ND request to resolve the L2 address. The router may need to send multiple ARP/ND requests until either a timeout is reached or an ARP/ND reply is received, and only then forward the data packets towards the target's MAC address. This process is not only CPU intensive but also buffer intensive 🥵.
Solution: to protect a router from getting overburdened by resolving target MAC addresses, one option is for the router to limit the rate at which it resolves target MAC addresses for inbound traffic whose target isn't in the router's ARP/ND cache. When the rate is exceeded, the incoming traffic whose target isn't in the ARP/ND cache gets dropped 🚫.
For an IPv4 network, another common practice to ease the pain from this problem is for the router to snoop ARP messages between other hosts, so its ARP cache can be refreshed with active addresses in the L2 domain ✨. As a result, the router is more likely to already have the IP-MAC entry when it receives data frames from external peers. [RFC6820] Section 7.1 gives a full description of this problem.
For IPv6 end stations, routers are supposed to send Router Advertisements (RAs) unicast even if they have snooped UNAs/NSs/NAs from those stations 🎀. So this practice lets an L2/L3 boundary send unicast RAs to the target instead of multicasts. [RFC6820] Section 7.2 has a full description of this problem.
Advantage: this practice reduces the number of ARP requests routers have to send when they receive IPv4 packets, and the number of IPv4 data frames from external peers that routers have to hold because their targets aren't in the ARP cache 🙌.
Disadvantage: the amount of ND processing on routers for IPv6 traffic isn't reduced. IPv4 routers still need to hold data packets from external peers and trigger ARP requests if the targets of those data packets either don't exist or aren't very active. In this case, IPv4 processing or IPv4 buffers aren't reduced 😩.
Recommendation: if there's a higher chance of routers getting data packets destined for nonexistent or inactive targets, alternative approaches should be considered 🧐.
5.1.3. Inter-Subnet Communications 🌈
The router could get hit with ARP/ND requests twice when the originating and destination stations are in different subnets attached to the same router, and those hosts don't talk to external peers often enough 🫠. The first hit is when the originating station in subnet-A starts an ARP/ND request to the L2/L3 boundary router, if the router's MAC isn't in the host's cache (Section 5.1.1 above). The second hit is when the L2/L3 boundary router starts ARP/ND requests to the target in subnet-B, if the target isn't in the router's ARP/ND cache (Section 5.1.2 above).
Again, the practices described in Sections 5.1.1 and 5.1.2 can ease some problems in some IPv4 networks 💜.
For IPv6 traffic, the practices described above don't reduce the ND processing on L2/L3 boundary routers 🥀.
Recommendation: consider the recommended approaches described in Sections 5.1.1 and 5.1.2. However, any solutions that relax the bidirectional requirement of IPv6 ND disable the security that the two-way ND communication exchange provides 🔒.
5.2. Static ARP/ND Entries on Switches 🧚
In a data center environment, the placement of L2 and L3 addressing may be orchestrated by Server (or VM) Management System(s). So it may be possible for static ARP/ND entries to be configured on routers and/or servers 🍓.
Advantage: this methodology has been used to reduce ARP/ND fluctuations in large-scale data center networks 👑.
Disadvantage: when some VMs are added, deleted, or moved, many switches' static entries need to be updated 😵. In a data center with virtualized servers, those events can happen frequently. For example, if one VM is added to one server and that VM's subnet spans 15 access switches, all 15 of them need to be updated. Network management mechanisms (SNMP, the Network Configuration Protocol (NETCONF), or proprietary mechanisms) are available to provide updates or incremental updates. However, there's no well-defined approach for switches to sync their content with the management system for efficient incremental updates 🙃.
Recommendation: additional work may be needed within IETF working groups (e.g., NETCONF, NVO3, I2RS, etc.) to get prompt incremental updates of static ARP/ND entries when changes occur 📣.
5.3. ARP/ND Proxy Approaches 🪄
RFC 1027 [RFC1027] specifies one ARP Proxy approach, called "Proxy ARP". However, RFC 1027 doesn't discuss a scaling mechanism. Since RFC 1027 was published in 1987, many variants of Proxy ARP have been deployed 🕰️. RFC 1027's Proxy ARP technique lets a gateway return its own MAC address on behalf of the target station.
[ARP_Reduction] describes a type of "ARP Proxy" that lets a ToR switch snoop ARP requests and return the target station's MAC if the ToR has the information in its cache. However, [RFC4903] doesn't recommend the caching approach described in [ARP_Reduction], because such a cache prevents any type of fast mobility between Layer 2 ports and breaks Secure Neighbor Discovery [RFC3971] 🙅♀️.
IPv6 ND Proxy [RFC4389] specifies a proxy used between an Ethernet segment and other segments, such as wireless or PPP segments 📶. ND Proxy [RFC4389] doesn't let a proxy send NA messages on behalf of the target, so the proxy doesn't interfere with hosts moving from one segment to another. Therefore, ND Proxy [RFC4389] doesn't reduce the number of ND messages to an L2/L3 boundary router 🤷♀️.
Bottom line, the term "ARP/ND Proxy" means different things depending on vendors and/or environments 💅.
Recommendation: for IPv4, even though those Proxy ARP variants (not RFC 1076) have been used to reduce ARP traffic in various environments, there are many issues with caching ⚠️.
The IETF should consider making proxy recommendations for data center environments as a transition issue, to help DC operators move to IPv6 🚀. Section 7 of [RFC4389] ("Guidelines to Proxy Developers") should be considered when developing any new proxy protocols to scale ARP 🌟.
5.4. Multicast Scaling Issues 🎶
Multicast snooping (IGMP/MLD) has different implementations and scaling issues 🌀. [RFC4541] notes that multicast IGMPv2/v3 snooping has trouble with subnets that include both IGMPv2 and IGMPv3. [RFC4541] also notes that MLDv2 snooping requires either destination MAC (DMAC) address filtering or deeper inspection of frames/packets to scale 🔍.
MLDv2 snooping needs to be re-examined for scaling within the DC 🧐. Efforts like IGMP/MLD explicit tracking [IGMP-MLD-Tracking] for downstream hosts need to provide better scaling than IGMP/MLDv2 snooping 💫.
6. Practices to Scale ARP/ND in Overlay Models 🦋
There are several documents on using overlay networks to scale large Layer 2 networks (or avoid the need for large L2 networks) and enable mobility (e.g., [L3-VM-Mobility], [VXLAN]) 💖. Transparent Interconnection of Lots of Links (TRILL) and IEEE 802.1ah (Mac-in-Mac) are other types of overlay networks that can scale Layer 2 🌸.
Overlay networks hide the VMs' addresses from the interior switches and routers, which greatly reduces the number of addresses exposed to the interior switches and router 🫶. The overlay edge nodes that perform the network address encapsulation/decapsulation still handle all remote stations' addresses that communicate with the locally attached end stations 📦.
For a large data center with many applications, those applications' IP addresses need to be reachable by external peers 🌐. So the overlay network may have a bottleneck at the gateway node(s) when resolving target stations' physical addresses (MAC or IP) and the overlay edge address within the data center 🚧.
Here are two approaches that can be used to minimize this problem 👇:
- Use static mapping as described in Section 5.2.
- Have multiple L2/L3 boundary nodes (i.e., routers), with each handling a subset of stations' addresses that are visible to external peers (e.g., Gateway #1 handles a set of prefixes, Gateway #2 handles another subset of prefixes, etc.) 👑
7. Summary and Recommendations ✨
okay so this memo is basically a big ol' catalogue of common practices that help with the address resolution mess on L2/L3 gateway routers 🫶
and bestie, there is no one-size-fits-all in data centers 💅 this memo ran through a bunch of different scenarios and laid out the pros and cons of every practice in each one.
in some of those scenarios, the practices could get even better if we extended or created IETF protocols. here are the protocol change recommendations:
- Relax the bidirectional requirement of IPv6 ND in some environments. Heads up though: relaxing it brings other issues along with it 😬 so you'd need a thorough study of the possible problems before making that change.
- Create an incremental "update" scheme for efficient static ARP/ND entries. 🔥
- Develop IPv4 ARP/IPv6 ND Proxy standards for use in the data center 🦋 when you're building any new proxy protocol to scale ARP/ND, go read Section 7 of [RFC4389] ("Guidelines to Proxy Developers") first, it's giving wisdom 🧚
- Look at the scaling issues with IGMP/MLD snooping and figure out whether new alternatives could scale better 🌈
8. Security Considerations 💀
this memo documents solutions that already exist, and it also proposes extra work that could extend various IETF protocols so ARP/ND scales better for data centers. the vibe is practical, but the stakes are high 😭
security is a MAJOR issue in data centers, so bestie, you need to seriously think about it whenever you design a future protocol extension 🔒 no skipping this part, it's not optional energy 🙈
9. Acknowledgements 🌸
huge thank-you to the ARMD WG 🫶 and these legends who gave valuable input to this document: Joel Jaeggli, Dave Thaler, Susan Hares, Benson Schliesser, T. Sridhar, Ron Bonica, Kireeti Kompella, and K.K. Ramakrishnan 👑
10. References 📚
10.1. Normative References
[GratuitousARP]
Cheshire, S., "IPv4 Address Conflict Detection", RFC 5227,
July 2008.
[RFC826] Plummer, D., "Ethernet Address Resolution Protocol: Or
Converting Network Protocol Addresses to 48.bit Ethernet
Address for Transmission on Ethernet Hardware", STD 37,
RFC 826, November 1982.
[RFC1027] Carl-Mitchell, S. and J. Quarterman, "Using ARP to
implement transparent subnet gateways", RFC 1027,
October 1987.
[RFC3971] Arkko, J., Ed., Kempf, J., Zill, B., and P. Nikander,
"SEcure Neighbor Discovery (SEND)", RFC 3971, March 2005.
[RFC4389] Thaler, D., Talwar, M., and C. Patel, "Neighbor Discovery
Proxies (ND Proxy)", RFC 4389, April 2006.
[RFC4541] Christensen, M., Kimball, K., and F. Solensky,
"Considerations for Internet Group Management Protocol
(IGMP) and Multicast Listener Discovery (MLD) Snooping
Switches", RFC 4541, May 2006.
[RFC4861] Narten, T., Nordmark, E., Simpson, W., and H. Soliman,
"Neighbor Discovery for IP version 6 (IPv6)", RFC 4861,
September 2007.
[RFC4903] Thaler, D., "Multi-Link Subnet Issues", RFC 4903,
June 2007.
[RFC6820] Narten, T., Karir, M., and I. Foo, "Address Resolution
Problems in Large Data Center Networks", RFC 6820,
January 2013.10.2. Informative References
[ARMD-Statistics]
Karir, M. and J. Rees, "Address Resolution Statistics",
Work in Progress, July 2011.
[ARP_Reduction]
Shah, H., Ghanwani, A., and N. Bitar, "ARP Broadcast
Reduction for Large Data Centers", Work in Progress,
October 2011.
[IGMP-MLD-Tracking]
Asaeda, H., "IGMP/MLD-Based Explicit Membership Tracking
Function for Multicast Routers", Work in Progress,
December 2013.
[L3-VM-Mobility]
Kumari, W. and J. Halpern, "Virtual Machine mobility in L3
Networks", Work in Progress, August 2011.
[Multi-Link]
Thaler, D. and C. Huitema, "Multi-link Subnet Support in
IPv6", Work in Progress, June 2002.
[RFC1076] Trewitt, G. and C. Partridge, "HEMS Monitoring and Control
Language", RFC 1076, November 1988.
[RFC7048] Nordmark, E. and I. Gashinsky, "Neighbor Unreachability
Detection Is Too Impatient", RFC 7048, January 2014.
[VXLAN] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
L., Sridhar, T., Bursell, M., and C. Wright, "VXLAN: A
Framework for Overlaying Virtualized Layer 2 Networks over
Layer 3 Networks", Work in Progress, April 2014.Authors' Addresses 🎀
Linda Dunbar
Huawei Technologies
5340 Legacy Drive, Suite 175
Plano, TX 75024
USA
Phone: (469) 277 5840
EMail: ldunbar@huawei.com
Warren Kumari
Google
1600 Amphitheatre Parkway
Mountain View, CA 94043
USA
EMail: warren@kumari.net
Igor Gashinsky
Yahoo
45 West 18th Street 6th floor
New York, NY 10011
USA
EMail: igor@yahoo-inc.comthat's the whole doc, bestie! 🌈💖
📦 where this page comes from
- ietf: rfc7342 2014-08
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms