RFC 7586

The Scalable Address Resolution Protocol (SARP) for Large Data Centers

📅 published June 2015, from ietf

RFC 7586: The Scalable Address Resolution Protocol (SARP) for Large Data Centers 💖

📄 Stream: Independent Submission 🏷️ Category: Experimental 🔢 ISSN: 2070-1721 📅 Date: June 2015 👩‍💻 Authors: Y. Nachum, L. Dunbar (Huawei), I. Yerushalmi, T. Mizrahi (Marvell) 🚫 Obsoletes: nothing listed 🔄 Updates: nothing listed

Abstract ✨

okay so this doc introduces the Scalable Address Resolution Protocol (SARP) 🦋, which is an architecture that uses proxy gateways to scale big data center networks. SARP is built on fast proxies that seriously shrink switches' Filtering Database (FDB) tables and lower the impact of ARP and Neighbor Discovery (ND) on network elements. This matters in situations where hosts in one subnet (or VLAN) can be spread across lots of different locations. SARP is aimed at massive data centers with a huge number of Virtual Machines (VMs) that can move between physical locations. it's giving scale 🔥

Independent Submissions Editor Note 📝

this is an Experimental document, and the experiment ends two years after the RFC is published. at that point, the RFC authors will try to figure out how widely SARP has been implemented and used 👀

IESG Note 🎀

the IESG points out that the problems described in RFC 6820 can already be handled by simply combining existing standardized or other published techniques. those include Layer 2 VPN (RFC 4664), proxy ARP (RFC 925), proxy Neighbor Discovery (RFC 4389), IGMP and MLD snooping (RFC 4541), and ARP mediation for IP interworking of Layer 2 VPNs (RFC 6575). so yeah, the problem is already solvable 💅

Status of This Memo 🌸

this document is not an Internet Standards Track specification. it's published for examination, experimental implementation, and evaluation.

this document defines an Experimental Protocol for the Internet community. it's a contribution to the RFC Series, independently of any other RFC stream. the RFC Editor chose to publish it at their discretion and makes no statement about its value for implementation or deployment. documents approved for publication by the RFC Editor are not a candidate for any level of Internet Standard; see Section 2 of RFC 5741.

info about the current status of this document, any errata, and how to send feedback can be found at http://www.rfc-editor.org/info/rfc7586 🔗

Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved.

this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication. please review these carefully, because they describe your rights and restrictions with respect to this document 🫶

Table of Contents 🧚

   1. Introduction ....................................................3
      1.1. SARP Motivation ............................................4
      1.2. SARP Overview ..............................................7
      1.3. SARP Deployment Options ....................................8
      1.4. Comparison with Existing Solutions .........................9
   2. Terms and Abbreviations Used in This Document ..................10
   3. SARP: Theory of Operation ......................................11
      3.1. Control Plane: ARP/ND .....................................11
           3.1.1. ARP/NS Request for a Local VM ......................11
           3.1.2. ARP/NS Request for a Remote VM .....................12
           3.1.3. Gratuitous ARP and Unsolicited Neighbor
                  Advertisement (UNA) ................................13
      3.2. Data Plane: Packet Transmission ...........................13
           3.2.1. Local Packet Transmission ..........................13
           3.2.2. Packet Transmission between Sites ..................13
      3.3. VM Migration ..............................................14
           3.3.1. VM Local Migration .................................14
           3.3.2. VM Migration from One Site to Another ..............14
                  3.3.2.1. Impact on IP-to-MAC Mapping Cache
                           Table of Migrated VMs .....................16
      3.4. Multicast and Broadcast ...................................17
      3.5. Non-IP Packet .............................................17
      3.6. High Availability and Load Balancing ......................17
      3.7. SARP Interaction with Overlay Networks ....................18
   4. Security Considerations ........................................18
   5. References .....................................................19
      5.1. Normative References ......................................19
      5.2. Informative References ....................................20
   Acknowledgments ...................................................21
   Authors' Addresses ................................................21

1. Introduction ✨

okay bestie, so this doc describes a proxy gateway technique called the Scalable Address Resolution Protocol (SARP) 💅. it cuts down switches' Filtering Database (FDB) size and the ARP/Neighbor Discovery impact on network elements, in an environment where hosts within one subnet (or VLAN) can be spread across different access domains in data centers.

the main idea is that SARP represents all VMs (or hosts) under each access domain by the MAC address of their corresponding access node (or aggregation node) 🧚. for example (see Figure 1), say host A in the west site needs to talk to host B, which is on the same VLAN but hanging off a different access domain (the east site). SARP makes host A use the MAC address of SARP proxy 2 instead of host B's address. that's the whole trick 🙈

because of this, switches in each domain don't have to keep a list of MAC addresses for every VM (or host) in the other access domains. every switch only needs to know MAC addresses in its own domain, plus the addresses of the remote SARP proxy gateways. so the switches' FDB size stays limited no matter how many access domains there are 🍓 no cap, that's iconic.

     +-------+     +-------+    _   __       +-------+     +-------+
     |       |     | SARP  |   / \_/  \_     | SARP  |     |       |
     |host A |<===>| proxy |<=>\_       \<==>| proxy |<===>|host B |
     |       |     |   1   |   /       _/    |   2   |     |       |
     +-------+     +-------+   \__   _/      +-------+     +-------+
                                  \_/
     <------West Site------>                 <------East Site------>

Figure 1: A Brief Overview of SARP 🦋

1.1. SARP Motivation ✨

Okay so, [RFC6820] is the one that digs into the impacts and scaling issues that pop up in data center networks when subnets stretch across multiple Layer 2 / Layer 3 (L2/L3) boundary routers. 💅

Unfortunately, when the combined number of VMs (or hosts) across all those subnets gets big, it can cause a total explosion in the size of the switches' MAC address table and really hit network elements hard 💀

There are four major issues when subnets span across multiple L2/L3 boundary router ports 🎀

  1. Explosion of the size of the intermediate switches' MAC address table (FDB). When hosts in a VLAN (or subnet) span across multiple access domains, and each access domain has hosts in different VLANs, each access switch has to enable multiple VLANs. That means those access switches are exposed to all MAC addresses across all VLANs.

For example, an access switch with 40 attached physical servers, where each server has 100 VMs, has 4,000 attached MAC addresses. If hosts/VMs can really be moved anywhere, the worst case for the access switch is when all those 4,000 VMs belong to different VLANs, meaning the access switch has 4000 VLANs enabled. If each VLAN has 200 hosts, this access switch's MAC address table could potentially have 200 * 4,000 = 800,000 entries 😭

It's important to note that this example is relevant regardless of whether IPv4 or IPv6 is used. 🌈

This scenario is worse than what today's L2/L3 gateway has to face. In today's environment, where each subnet is limited to a few access switches, the number of MAC addresses the gateway has to learn is significantly smaller in scale. 🧚

  1. ARP/ND processing load impact on the L2/L3 boundary routers. All VMs periodically send NDs to their corresponding gateway nodes to get the gateway nodes' MAC addresses. When the combined number of VMs across all the VLANs is large, processing the responses to the ND requests from those VMs can easily exhaust the gateway's CPU utilization 🔥

An L2/L3 boundary router could be hit with ARP/ND twice when the originating and destination stations are in different subnets attached to the same router, and when those hosts don't communicate with external peers very frequently. The first hit is when the originating station in subnet 1 initiates an ARP/ND request to the L2/L3 boundary router. The second hit is when the L2/L3 boundary router initiates an ARP/ND request to the target in subnet 2, if the target is not in the router's ARP/ND cache. 🙈

  1. Broadcast load on end stations in IPv4. In IPv4, every end station in a subnet receives ARP broadcast messages from all other end stations in the subnet. IPv6 ND eliminated this issue by using multicast. 🦋

However, most devices support a limited number of multicast addresses, due to the scaling of multicast filtering. Once the number of multicast addresses exceeds the multicast filter limit, the multicast addresses have to be processed by the devices' CPUs (i.e., the slow path). 🐢

It's less of an issue in data centers without VM mobility, since each port is only dedicated to one (or a small number of) VLANs. So the number of multicast addresses hitting each port is significantly lower. 💜

  1. ARP/ND flooding reduces bandwidth for user traffic. The ARP/ND messages are flooded to many physical link segments, which can reduce the bandwidth utilization for user traffic. Honestly though, ARP/ND flooding is, in most cases, an insignificant issue in today's data center networks, because the majority of data center servers are shifting towards 1G or 10G Ethernet ports. The bandwidth used by ARP/ND, even when flooded to all physical links, becomes negligible compared to the link bandwidth. Furthermore, IGMP and Multicast Listener Discovery (MLD) snooping [RFC4541] can further reduce the ND multicast traffic to some physical link segments. 🍓

Statistics gathered by Merit Network [ARMDStats] have shown that the major impact of a large number of VMs in data centers is on the L2/L3 boundary routers, i.e., issue 2 above. An L2/L3 boundary router could be hit with ARP/ND twice when 1) the originating and destination stations are in different subnets attached to the same router, and 2) those hosts don't communicate with external peers often enough. Main character energy for the routers, honestly 👑

Overlay approaches, e.g., [RFC7364], can hide addresses of hosts (VMs) in the core, but they don't prevent the MAC address table explosion problem (issue 1) unless the Network Virtualization Edge (NVE) is on a server. ✨

The scaling practices documented in [RFC7342] can only reduce some ARP impact on L2/L3 boundary routers in some scenarios, but not all. Close, but no cap, not quite enough 🥺

In order to protect router CPUs from being overburdened by target resolution requests, some routers rate-limit the target MAC resolution requests to the router's CPU. When the rate limit is exceeded, the incoming data frames are dropped. In traditional data centers this issue is less significant, since the number of hosts attached to one L2/L3 boundary router is limited by the number of physical ports of the switches/routers. But when servers are virtualized to support 30+ VMs, the number of hosts under one router can grow by a factor of 30+. 😳

Furthermore, in traditional data center networks, each subnet is neatly bound to a limited number of server racks, i.e., switches only need to be familiar with MAC addresses of hosts that reside in that small number of subnets. In contemporary data center networks, subnets are spread across many server racks, so switches are exposed to VLAN/MAC addresses of many subnets, which greatly increases the size of switches' FDB tables. It's giving chaos, periodt 💀

The solution proposed in this document can eliminate or reduce the likelihood of inter-subnet data frames being dropped, and it reduces the number of host MAC addresses that intermediate switches are exposed to, thus reducing switches' FDB table sizes. Iconic behaviour 🫶

1.2. SARP Overview ✨

The SARP approach uses proxy gateways to address the problems discussed above 🦋

Note: The guidelines to proxy developers [RFC4389] have been carefully considered for SARP. Section 3.3 discusses how SARP works when VMs are moved from one segment to another.

In order to enable VMs to be moved across servers while ensuring their MAC/IP addresses remain unchanged, the Layer 2 network (e.g., VLAN) that interconnects those VMs may spread across different server racks, different rows of server racks, or even different data center sites. Spread out, bestie 🌸

A multisite data center network is made up of two main building blocks: an interconnecting segment and an access segment. While the access network is, in most cases, a Layer 2 network, the interconnecting segment is not necessarily a Layer 2 network.

The SARP proxies are located at the boundaries where the access segment connects to its interconnecting segment. The boundary node can be a hypervisor virtual switch, a top-of-rack switch, an aggregation switch (or end-of-row switch), or a data center core switch. Figure 2 depicts an example of two remote data centers that are managed as a single, flat Layer 2 domain. SARP proxies are implemented at the edge devices connecting the data center to the transport network. SARP significantly reduces the ARP/ND transmissions over the interconnecting network. 🧚

                            *-------------------*
                            |                   |
                    +-------| Interconnecting   |-------+
                    |       |     network       |       |
                    |       *-------------------*       |
                    |                                   |
           *-----------------*                  *----------------*
           |  SARP Proxies   |                  |  SARP Proxies  |
           *-----------------*                  *----------------*
              |           |                        |           |
          *-------*   *-------*                *-------*   *-------*
          |Access |   |Access |                |Access |   |Access |
          *-------*   *-------*                *-------*   *-------*
              |
         *----------*
         |Hypervisor|
         *----------*
              |
          *--------*
          |Virtual |
          |Machine |
          *--------*

             (West Site)                          (East Site)

Figure 2: SARP: Network Architecture Example

1.3. SARP Deployment Options ✨

SARP deployment is tightly coupled with the data center architecture. SARP proxies are located at the point where the Layer 2 infrastructure connects to its Layer 2 cloud using overlay networks. SARP proxies can be located at the data center edge (as Figure 2 depicts), data center core, or data center aggregation (denoted by "Agg" in the figure). SARP can also be implemented by the hypervisor (as Figure 3 depicts). So many options, so little time 🎀

To simplify the description, we'll focus on data centers that are managed as a single, flat Layer 2 network, where SARP proxies are located at the boundary where the data center connects to the transport network (as Figure 2 depicts).

                            *-------------------*
                            |                   |
                    +-------|     TRANSPORT     |-------+
                    |       |                   |       |
                    |       *-------------------*       |
                    |                                   |
           *-----------------*                  *----------------*
           |   Edge Device   |                  |  Edge Device   |
           *-----------------*                  *----------------*
                    |                                   |
           *-----------------*                  *----------------*
           |       Core      |                  |      Core      |
           *-----------------*                  *----------------*
              |           |                        |           |
          *-------*   *-------*                *-------*   *-------*
          |  Agg  |   |  Agg  |                |  Agg  |   |  Agg  |
          *-------*   *-------*                *-------*   *-------*
              |
         *----------*
         |Hypervisor|
         *----------*

             (West Site)                          (East Site)

Figure 3: SARP Deployment Options

1.4. Comparison with Existing Solutions ✨

The IETF has developed several mechanisms to address issues associated with Layer 2 networks over multiple geographic locations. For example, there's Layer 2 VPN [RFC4664], proxy ARP [RFC925] [ProxyARP], proxy Neighbor Discovery [RFC4389], IGMP and MLD snooping [RFC4541], and ARP mediation for IP interworking of Layer 2 VPNs [RFC6575]. Lots of options, all with their own vibe 💅

However, all those solutions work well when hosts within one subnet are placed together under one access domain, so that the intermediate switches in each access domain are only exposed to host addresses from a limited number of subnets. SARP is here to provide a solution for when hosts within one subnet are spread across multiple access domains, and each access domain has hosts from many subnets. In this environment, the intermediate switches in each access domain are exposed to the combined hosts of all the subnets that are enabled by the access domain. That's the whole plot twist 🌈

2. Terms and Abbreviations Used in This Document ✨

okay so let's get the vocab down first bestie 📚🫶

  • ARP: Address Resolution Protocol [ARP] 🔍
  • FDB: Filtering Database, which is used for Layer 2 switches [802.1Q]. Layer 2 switches flood data frames when the Destination Address (DA) is not in the FDB 🌊, whereas routers drop data frames when the DA is not in the Forwarding Information Base (FIB) 🗑️. That's literally why the FDB is a thing for Layer 2 switches 💅
  • FIB: Forwarding Information Base 🧭
  • Hypervisor: a software layer that creates and runs virtual machines on a server 🖥️✨
  • IP-D: IP address of the destination virtual machine 🎯
  • IP-S: IP address of the source virtual machine 🎀
  • MAC-D: MAC address of the destination virtual machine 💜
  • MAC-E: MAC address of the East Proxy SARP Device 🌅
  • MAC-S: MAC address of the source virtual machine 🌸
  • NA: IPv6 ND's Neighbor Advertisement 📣
  • ND: IPv6 Neighbor Discovery Protocol [ND]. In this document, ND also refers to Neighbor Solicitation, Neighbor Advertisement, and Unsolicited Neighbor Advertisement messages defined by RFC 4861 🧚
  • NS: IPv6 ND's Neighbor Solicitation 👋
  • SARP Proxy: The components that participate in SARP 🦋
  • UNA: IPv6 ND's Unsolicited Neighbor Advertisement [ND] 📢
  • VM: Virtual Machine 💖

3. SARP: Theory of Operation 🌈

3.1. Control Plane: ARP/ND 🧠

this section is all about how the ARP/ND stuff goes down 💅 the first scenario is when the source and destination VMs are chilling in the same access segment 🏠. in the second one, the source VM is in the local access segment and the destination VM lives at the remote access segment ✈️

in every scenario, the VMs (source and destination) share the same L2 broadcast domain 🫶

3.1.1. ARP/NS Request for a Local VM 🎀

when the source and destination VMs are at the same access segment (Figure 4), the address resolution process works the way it's described in [ARP] and [ND] 📖. host A sends an ARP request or an IPv6 Neighbor Solicitation (NS) to learn the IP-to-MAC mapping of host B 🔎, and then it gets a reply from host B with the IP-D to MAC-D mapping 💌

  +-------+      _   __       +-------+      _   __
  |host A |     / \_/  \_     | SARP  |     / \_/  \_
  | IP-S  |<--->\_access \<==>| proxy |<===>\_interc.\
  | MAC-S |     /network_/    |   1   |     /network_/
  +-------+  +->\__   _/      +-------+     \__   _/
             |     \_/                         \_/
  +-------+  |
  |host B |<-+
  | IP-D  |
  | MAC-D |
  +-------+

  <--------------West Site------------>

        Figure 4: SARP: Two Hosts in the Same Access Segment

3.1.2. ARP/NS Request for a Remote VM 🌍

now when the source and destination VMs are at different access segments, the address resolution goes like this 👀

  +-------+     +-------+    _   __       +-------+     +-------+
  |host A |     | SARP  |   / \_/  \_     | SARP  |     |host B |
  | IP-S  |<===>|proxy 1|<=>\_       \<==>|proxy 2|<===>| IP-D  |
  | MAC-S |     | MAC-W |   /       _/    | MAC-E |     | MAC-D |
  +-------+     +-------+   \__   _/      +-------+     +-------+
                               \_/
  <------West Site------>                 <------East Site------>

     Figure 5: SARP: Two Hosts That Reside in Different Segments

in the example in Figure 5, the source VM is at the west access segment and the destination VM is at the east access segment 🌅

when host A sends an ARP/NS request to find out the IP-to-MAC mapping of host B 💌:

  1. if SARP proxy 1 doesn't have IP-D in its ARP cache 🙈, the ARP/NS request gets propagated to all access segments that might have VMs in the same virtual network as the originating VM 📡, including the east access segment 🌅.
  2. as SARP proxy 1 forwards the ARP/NS message, it swaps the source MAC address, MAC-S, for its own MAC address, MAC-W 🎭. so all the switches in the interconnecting segment never get exposed to MAC-S 🫣.
  3. the ARP/NS request reaches SARP proxy 2 📬.
  4. if SARP proxy 2 doesn't have IP-D in its ARP cache, the ARP/NS request is forwarded to the east access network 🚀. host B responds with an ARP reply (IPv4) or a Neighbor Advertisement (IPv6) to the request with MAC-D 💖.
  5. when the response message reaches SARP proxy 2, it replaces MAC-D with MAC-E 🔄, so the response reaches SARP proxy 1 with MAC-E ✨.
  6. as SARP proxy 1 forwards the response to host A, it changes the destination address from MAC-W to MAC-S 🎯.

SARP Proxy ARP/ND Cache 🗃️

SARP proxies keep a cache of the IP-to-MAC mapping 💾. this cache is built from the ARP/ND messages that hosts send and that traverse the SARP proxies 🛤️.

in steps 1 and 4 above, if the SARP proxy has IP-D in its ARP cache, it responds with MAC-E 🙌 without forwarding the ARP/NS request 🛑.

this caching approach seriously cuts down the amount of ARP/ND traffic on the network 📉 and makes the round-trip time of ARP/ND requests way faster ⚡.

when the west SARP proxy caches the IP-to-MAC mapping entries for remote VMs, the expiration timers should be set to relatively low values 🕰️ so you don't end up with stale entries when remote VMs get moved or deleted 🗑️. in environments where VMs move more often, it is not recommended for SARP proxies to cache the IP-to-MAC mapping entries of remote VMs 🙅‍♀️.

3.1.3. Gratuitous ARP and Unsolicited Neighbor Advertisement (UNA) 📣

hosts (or VMs) send out Gratuitous ARP (IPv4) [TcpIp] and Unsolicited Neighbor Advertisement (UNA) (IPv6) messages 📨 so that other nodes can refresh their IP-to-MAC entries in their caches 🔁.

the local SARP proxy handles the Gratuitous ARP or UNA message the same way it handles an ARP reply or IPv6 NA, i.e., it swaps the MAC addresses in the same manner 🤝.

3.2. Data Plane: Packet Transmission 📦

3.2.1. Local Packet Transmission 🏡

when a VM sends packets to a destination VM at the same site (Figure 4), the data plane is totally unaffected by SARP 🙌. packets go from (IP-S, MAC-S) to (IP-D, MAC-D) 💅.

3.2.2. Packet Transmission between Sites 🌉

packets sent between sites (Figure 5) go through the SARP proxy of both sites 🛣️.

a packet sent from host A to host B goes through this procedure 📝:

  1. host A sends a packet to IP-D, and based on its ARP table, it uses the MAC addresses {MAC-E, MAC-S} 🎯.
  2. SARP proxy 1 receives the packet and replaces the source MAC address, so the packet includes {MAC-E, MAC-W} 🔄.
  3. SARP proxy 2 receives the packet and replaces the destination MAC address, and the packet is sent to host B with {MAC-D, MAC-W} 🚀.

SARP proxy 1 swaps the source MAC address for its own because the switches in the interconnecting segment only know about SARP proxy MAC addresses and don't know about host addresses 🧐.

Note: it is a common security practice in data center networks to use access lists, allowing each VM to communicate only with a list of authorized peer VMs 🔐. in most cases, such access control lists are based on IP addresses and, hence, are not affected by the MAC address replacement in SARP 🫶.

3.3. VM Migration 🚚

3.3.1. VM Local Migration 🏠

when a VM migrates locally within its access segment, SARP doesn't need any special behavior 😌. VM migration is handled entirely by the Layer 2 mechanisms 🧱.

3.3.2. VM Migration from One Site to Another 🧳

this section focuses on a scenario where a VM moves from the west site to the east site while keeping its MAC and IP addresses 🎀.

VM migration can affect networking elements depending on where they are 👀:

  • origin site (west site)
  • destination site (east site)
  • other sites
  +-------+     +-------+    _   __       +-------+     +-------+
  |host A |     | SARP  |   / \_/  \_     | SARP  |     |host A |
  | IP-D  |<===>|proxy 1|<=>\_       \<==>|proxy 2|<===>| IP-D  |
  | MAC-D |     | MAC-W |   /       _/    | MAC-E |     | MAC-D |
  +-------+     +-------+   \__   _/      +-------+     +-------+
                               \_/
  <------West Site------>                 <------East Site------>
        Origin Site                          Destination Site

     Figure 6: SARP: Host A Migrates from West Site to East Site

Origin Site 🌇

the origin site is where the VM lives before the migration (the west site) 🏚️.

before the VM (IP=IP-D, MAC=MAC-D) gets moved, all VMs at the west site that have an ARP entry of IP-D in their ARP table have the IP-D -> MAC-D mapping 🗺️. VMs on other access segments have an ARP entry of IP-D -> MAC-W mapping, where MAC-W is the MAC address of the SARP proxy on the west access segment 🧭.

after the VM (IP-D) in the west site moves to the east site, if a Gratuitous ARP (IPv4) or an Unsolicited Neighbor Advertisement (IPv6) message is sent out by the destination hypervisor on behalf of the VM (IP-D) 📣, then the IP-to-MAC mapping cache of the VMs in all access segments gets updated to IP-D -> MAC-E, where MAC-E is the MAC address of the SARP proxy on the east site 🌅. if no Gratuitous ARP or UNA message is sent out by the destination hypervisor, the IP-to-MAC cache on the VMs in the west site (and other sites) eventually ages out ⏳.

until the IP-to-MAC mapping cache tables are updated, the source VMs from the west site keep sending packets locally to MAC-D 😬, and switches at the west site are still configured with the old location of MAC-D 🙃. this transient mess can be fixed by having the VM manager send out a fake Gratuitous ARP or UNA message on behalf of the destination hypervisor 🪄. another option is to have a shorter aging timer configured for the IP-to-MAC cache table ⏱️.

Destination Site 🌆

the destination site is the site the VM migrated to, i.e., the east site in Figure 6 🎯.

before any Gratuitous ARP or UNA messages are sent out by the destination hypervisor, all VMs at the east site (and all other sites) might have an IP-D -> MAC-W mapping in their IP-to-MAC mapping cache 🫠. the IP-to-MAC mapping cache gets updated by aging or by a Gratuitous ARP or UNA message sent by the destination hypervisor 🔄. until those caches are updated, VMs from the east site keep sending packets to MAC-W 📭. this can be fixed by having the VM manager send out a fake Gratuitous ARP or UNA message immediately after the VM migration 🚨, or by redirecting the packets from the SARP proxy of the east site back to the migrated VM by updating the destination MAC of the packets to MAC-D 🎯.

Other Sites 🌏

all VMs at the other sites that have an ARP entry of IP-D in their ARP table have the IP-D -> MAC-W mapping 🗂️. the ARP mapping gets updated by aging or by a Gratuitous ARP message sent by the destination hypervisor of the migrated VM, and modified by the SARP proxy of the east site to an IP-D -> MAC-E mapping 🔧. until the ARP tables are updated, VMs from other sites keep sending packets to MAC-W 🫤.

3.3.2.1. Impact on IP-to-MAC Mapping Cache Table of Migrated VMs ✨

okay so when a VM (IP-D) gets moved from one site to another, its IP-to-MAC mapping entries for VMs at other sites (i.e., neither the east site nor the west site) are still valid 🫶. Most guest OSs (or VMs) will refresh their IP-to-MAC cache after migration anyway, but the entries are still good in the meantime, which is lowkey iconic 💅.

The migrated VM's IP-to-MAC mapping entries for VMs at the east site can stay exactly as they are until the ARP aging time, if they don't get refreshed after migration 🌸. That's because these entries are mapped to MAC-E. All traffic the migrated VM sends to VMs at the east site goes through the SARP proxy of the east site, and that proxy can redirect it back to the right destinations there 🦋. On top of that, an ARP/UNA message sent by the SARP proxy of the east site, or by the VMs on the east site, can refresh the matching entries in the migrated VM's IP-to-MAC cache. Main character energy, fr fr 👑.

The migrated VM's ARP entries for VMs at the west site stay unchanged until either the ARP entries age out or new data frames arrive from the remote sites 🙈. Here's the tea: all the MAC addresses of VMs at the west site are unknown to the east site. So during the transient period before the ARP entries age out, all unknown traffic from the VM gets intercepted by the SARP proxy of the east site and forwarded on to the SARP proxy of the west site 💀. This transient behavior can be avoided if the SARP proxy has the destination IP address in its ARP cache. Then, when it receives a packet with an unknown destination MAC address, it could send a Gratuitous ARP or UNA message to the migrated VM 🧚.

Note that overlay networks providing Layer 2 network virtualization services configure their edge-device MAC aging timers to be greater than the ARP request interval 🍓.

3.4. Multicast and Broadcast 🌈

Multicast and broadcast traffic gets forwarded by SARP proxies like this:

  • SARP proxies modify the source MAC address of multicast and broadcast packets as described in Section 3.2.
  • SARP proxies do not modify the destination MAC address of multicast and broadcast packets.

3.5. Non-IP Packet 🎀

The L2/L3 boundary routers in this document can forward non-IP IEEE 802.1 Ethernet frames (Layer 2) without changing the MAC headers ✨. When subnets span across multiple ports of those routers, they still count as a single link, or a multi-access link model recommended by [RFC4903]. They're different from the "multi-link" subnets described in [MultLinkSub] and [RFC4903]. Those refer to different physical media with the same prefix connected to a router, where Layer 2 frames can't be natively forwarded without changing the headers 🙅‍♀️.

3.6. High Availability and Load Balancing 💜

The SARP proxy sits at the boundary where the local Layer 2 infrastructure connects to the interconnecting network 🌸. All traffic from the local site to the remote sites goes through the SARP proxy. The SARP proxy is subject to high-availability and bandwidth requirements, so it's not something to slack on 😤.

The SARP architecture supports multiple SARP proxies connecting a single site to the transport network 🔥. In SARP, all proxies can be active and can back up one another. The architecture is robust, and it lets network admins allocate proxies according to bandwidth and high-availability needs 👑.

Traffic gets segregated between SARP proxies by using VLANs 💖. An SARP proxy is the Master SARP proxy of one set of VLANs and the Backup SARP proxy of another set.

Let's say the SARP proxies of the west site are SARP proxy 1 and SARP proxy 2. The west site supports VLAN 1 and VLAN 2. SARP proxy 1 is the Master SARP proxy of VLAN 1 and the Backup SARP proxy of VLAN 2, while SARP proxy 2 is the Master SARP proxy of VLAN 2 and the Backup SARP proxy of VLAN 1. Both proxies are members of VLAN 1 and VLAN 2 🦋.

The Master SARP proxy updates its Backup SARP proxy with all the ARP reply messages 📨. The Backup SARP proxy keeps a backup database for all the VLANs it's the Backup SARP proxy of 🧚.

The Master and the Backup SARP proxies maintain a keepalive mechanism 💓. If something fails, the Backup SARP proxy becomes the Master SARP proxy. The failure decision is made per VLAN ⭐. When the Master and the Backup SARP proxies switch over, the Backup SARP proxy can use the MAC address of the Master SARP proxy. It then sends a Gratuitous ARP message locally, carrying the MAC address of the Master SARP proxy, to update the forwarding tables on the local switches 🔁. The Backup SARP proxy also tells the remote SARP proxies about the change 🫶.

3.7. SARP Interaction with Overlay Networks 🧚‍♀️

SARP can be used over overlay networks that provide L2 network virtualization, such as IP, Virtual Private LAN Service (VPLS), Transparent Interconnection of Lots of Links (TRILL), Overlay Transport Virtualization (OTV), Network Virtualization using GRE (NVGRE), and Virtual eXtensible Local Area Network (VXLAN) 🌈. Mapping SARP onto overlay networks is pretty straightforward: the VM maps the destination IP to the SARP proxy MAC address, and the overlay networks take care of mapping the proxy MAC to its correct tunnel 💅.

SARP massively cuts down the complexity of overlay networks and transport networks by shrinking the mapping tables down to the number of SARP proxies 📉.

4. Security Considerations 🔐

SARP proxies sit at the boundaries of access networks, where the local Layer 2 infrastructure connects to its Layer 2 cloud 🌥️. SARP proxies interoperate with overlay network protocols that extend the Layer 2 subnet across data centers, or between different systems within a data center 🧩.

SARP does not expose the network to security threats beyond those that exist whether or not SARP is present ✅.

SARP proxies may be exposed to denial-of-service (DoS) attacks by means of ARP/ND message flooding 🚨. So SARP proxies must have sufficient resources to support the SARP control plane, without making the network more vulnerable to DoS than it was without SARP proxies. This one is serious, bestie 😭.

SARP adds security to the data plane when it comes to network reconnaissance 🕵️‍♀️. It hides all the local Layer 2 MAC addresses from potential attackers located on the interconnecting network, and it significantly limits the number of addresses exposed to an attacker at a remote site 🔒.

5. References 📚

5.1. Normative References

5.2. Informative References

  • [802.1Q] IEEE, "IEEE Standard for Local and metropolitan area networks -- Bridges and Bridged Networks", IEEE Std 802.1Q.
  • [ARMDStats] Karir, M., and J. Rees, "Address Resolution Statistics", Work in Progress, draft-karir-armd-statistics-01, July 2011.
  • [MultLinkSub] Thaler, D., and C. Huitema, "Multi-link Subnet Support in IPv6", Work in Progress, draft-ietf-ipv6-multi-link-subnets-00, June 2002.
  • [RFC4903] Thaler, D., "Multi-Link Subnet Issues", RFC 4903, DOI 10.17487/RFC4903, June 2007, <http://www.rfc-editor.org/info/rfc4903>.
  • [RFC6820] Narten, T., Karir, M., and I. Foo, "Address Resolution Problems in Large Data Center Networks", RFC 6820, DOI 10.17487/RFC6820, January 2013, <http://www.rfc-editor.org/info/rfc6820>.
  • [RFC7342] Dunbar, L., Kumari, W., and I. Gashinsky, "Practices for Scaling ARP and Neighbor Discovery (ND) in Large Data Centers", RFC 7342, DOI 10.17487/RFC7342, August 2014, <http://www.rfc-editor.org/info/rfc7342>.
  • [RFC7364] Narten, T., Ed., Gray, E., Ed., Black, D., Fang, L., Kreeger, L., and M. Napierala, "Problem Statement: Overlays for Network Virtualization", RFC 7364, DOI 10.17487/RFC7364, October 2014, <http://www.rfc-editor.org/info/rfc7364>.
  • [TcpIp] Stevens, W., "TCP/IP Illustrated, Volume 1: The Protocols", Addison-Wesley, 1994.

Acknowledgments 🫶

The authors thank Ted Lemon, Eric Gray, and Adrian Farrel for providing valuable comments and suggestions for the document 💖.

Authors' Addresses 💌

Youval Nachum
EMail: youval.nachum@gmail.com


Linda Dunbar
Huawei Technologies
5430 Legacy Drive, Suite #175
Plano, TX  75024
United States
Phone: (469) 277 5840
EMail: ldunbar@huawei.com


Ilan Yerushalmi
Marvell
6 Hamada St.
Yokneam, 20692
Israel
EMail: yilan@marvell.com


Tal Mizrahi
Marvell
6 Hamada St.
Yokneam, 20692
Israel
EMail: talmi@marvell.com

That's the whole RFC, bestie! 🌈✨

📦 where this page comes from