RFC 9161

Operational Aspects of Proxy ARP/ND in Ethernet Virtual Private Networks

📅 published January 2022, from ietf

RFC 9161 💖 Operational Aspects of Proxy ARP/ND in Ethernet Virtual Private Networks

okay bestie, RFC 9161 is a Standards Track document from January 2022 🦋 It was written by J. Rabadan (editor), S. Sathappan, K. Nagaraj, G. Hankins (at Nokia), and T. King (at DE-CIX). It's published as ISSN 2070-1721, and it updates RFC 7432 (the EVPN spec) ✨ It doesn't name anything it obsoletes, so RFC 7432 is the only one it's touching. Now let's get into it 💅

Abstract 🌸

ngl this abstract is giving "here's the whole vibe in four sentences." It describes the EVPN Proxy ARP/ND function, boosted by the ARP/ND Extended Community 🧚 So this doc updates the EVPN spec to explain how Proxy ARP/ND actually works, in way more detail. The whole point is helping operators of Internet Exchange Points, Data Centers, and other networks deal with IPv4 and IPv6 address resolution headaches in huge Broadcast Domains. How? By reducing and even suppressing the flooding that address resolution causes across the EVPN network 🔥

Status of This Memo 🎀

This is an Internet Standards Track document 💜

It's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community. It went through public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more on Internet Standards, check out Section 2 of RFC 7841 🫶

For the current status, any errata, and how to send feedback, go to https://www.rfc-editor.org/info/rfc9161 🌈

Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved. 🔒

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication. Please read these carefully, because they spell out your rights and restrictions for this document 📜 Code Components pulled out of this document must include Revised BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as the Revised BSD License describes 💅

Table of Contents 🧚

1.  Introduction
  1.1.  The Data Center Use Case
  1.2.  The Internet Exchange Point Use Case
2.  Terminology
3.  Solution Description
  3.1.  Proxy ARP/ND Sub-functions
  3.2.  Learning Sub-function
    3.2.1.  Proxy ND and the NA Flags
  3.3.  Reply Sub-function
  3.4.  Unicast-Forward Sub-function
  3.5.  Maintenance Sub-function
  3.6.  Flood (to Remote PEs) Handling
  3.7.  Duplicate IP Detection
4.  Solution Benefits
5.  Deployment Scenarios
  5.1.  All Dynamic Learning
  5.2.  Dynamic Learning with Proxy ARP/ND
  5.3.  Hybrid Dynamic Learning and Static Provisioning with Proxy
        ARP/ND
  5.4.  All Static Provisioning with Proxy ARP/ND
  5.5.  Example of Deployment in Internet Exchange Points
  5.6.  Example of Deployment in Data Centers
6.  Security Considerations
7.  IANA Considerations
8.  References
  8.1.  Normative References
  8.2.  Informative References
Acknowledgments
Contributors
Authors' Addresses

1. Introduction ✨

okay so, as [RFC7432] lays out, the IP Address field in an EVPN MAC/IP Advertisement route can optionally carry one of the IP addresses tied to the MAC address 💖 A Provider Edge (PE) can learn local IP->MAC pairs and advertise them in those EVPN MAC/IP Advertisement routes. Then the remote PEs that import those routes into the same Broadcast Domain (BD) can add those IP->MAC pairs to their Proxy ARP/ND tables. Once they're in there, a remote PE can reply to local ARP Requests or Neighbor Solicitations, or "unicast-forward" those packets to the owner MAC. The result is less flooding, and in some cases the flooding gets fully suppressed across the EVPN network 🦋

EVPN and its Proxy ARP/ND function are seriously useful in Data Centers (DCs) and Internet Exchange Points (IXPs) with large Broadcast Domains. There, the amount of ARP/ND flooded traffic causes real problems on the connected routers and Customer Edges (CEs) 🔥 [RFC6820] describes the address resolution problems in large DC networks, so go read that if you want the full tea ☕

This document describes the Proxy ARP/ND function in [RFC7432] networks, boosted by the ARP/ND Extended Community [RFC9047]. From that angle, this document updates [RFC7432] 🌸

Proxy ARP/ND may be implemented to help IXPs, DCs, and other operators deal with the issues that come from address resolution in large Broadcast Domains 💜

1.1. The Data Center Use Case 🎀

As [RFC6820] describes, IPv4 and IPv6 address resolution can cause a LOT of issues in large DCs 💀 The problems created by IPv4 Address Resolution Protocol procedures can be especially significant.

First, ARP Requests use broadcast MAC addresses. That means any Tenant System in a large Broadcast Domain sees a huge pile of ARP traffic that isn't even addressed to most of the receivers 😭

It gets worse. If some Tenant Systems disappear from the Broadcast Domain, some implementations will keep sending ARP Requests over and over. As [RFC6820] states, there are no clear requirements for retransmitting ARP Requests when no replies come back. So an implementation may choose to keep retrying endlessly even with no replies 🙈

The flooding that address resolution creates can be reduced by using EVPN and its Proxy ARP/ND function 🦋

1.2. The Internet Exchange Point Use Case 🌈

The implementation described in this document is especially useful in IXP networks 🧚

A typical IXP gives access to a large Layer 2 Broadcast Domain for peering purposes (called "the peering network" here), and that's where (hundreds of) Internet routers are connected. In this section, we call these Internet routers CE devices. Because every router has to connect to a single Layer 2 network, the peering networks use IPv4 addresses in length ranges from /21 to /24 (and even bigger for IPv6), which can make very large Broadcast Domains. The peering network is transparent to the CEs, so it floods any ARP Requests or NS messages to all the CEs in the network. Gratuitous ARP and NA messages get flooded to all the CEs too 😵‍💫

In these IXP networks, most of the CEs are typically peering routers, and roughly all of the Broadcast, Unknown Unicast, and Multicast (BUM) traffic comes from the ARP and ND address resolution procedures. This ARP/ND BUM traffic creates significant data volumes that reach every single router in the peering network. ARP/ND messages get processed in "slow path" software processors and they take high priority in the routers, so heavy loads of ARP/ND traffic can make some routers run out of resources. When CEs disappear from the network, address resolution explosions can happen, and a router with limited processing power may fail to keep BGP sessions running 😱

IPv6 routers might handle this better if Multicast Listener Discovery (MLD) snooping is enabled, since ND uses an SN-multicast address in NS messages. ARP, though, uses broadcast and has to be processed by all the routers in the network 🫠 Some routers may also be configured to broadcast periodic Gratuitous ARPs (GARPs) [RFC5227]. For IPv6, the fact that IPv6 CEs have more than one IPv6 address also adds to the growth of ND flooding in the network. The amount of ARP/ND flooded traffic grows linearly with the number of IXP participants, so the issue can only get worse as new CEs are added 📈

IXPs have built various solutions over the past years to deal with this. While those solutions may ease address resolution issues in large Broadcast Domains, EVPN gives IXPs new and more efficient possibilities 🌟 EVPN and its Proxy ARP/ND function may solve the issue in a distributed and scalable way, fully integrated with the PE network 👑

2. Terminology 💅

Here's the key-word situation, and it's important 🎀 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174], but only when they appear in all capitals, exactly as shown here 🔥

Here's the abbreviation glossary, copied exactly so nothing gets lost in translation 💖

ARP:        Address Resolution Protocol

AS-MAC:     Anti-spoofing MAC.  It is a special MAC configured on all
            the PEs attached to the same BD and used for the
            duplicate IP detection procedures.

BD:         Broadcast Domain

BUM:        Broadcast, Unknown Unicast, and Multicast Layer 2 traffic

CE:         Customer Edge router

DAD:        Duplicate Address Detection, as per [RFC4861]

DC:         Data Center

EVI:        EVPN Instance

EVPN:       Ethernet Virtual Private Network, as per [RFC7432]

GARP:       Gratuitous ARP

IP->MAC:    An IP address associated to a MAC address.  IP->MAC
            entries are programmed in Proxy ARP/ND tables and may be
            of three different types: dynamic, static, or EVPN-
            learned.

IXP:        Internet Exchange Point

IXP-LAN:    The IXP's large Broadcast Domain to where Internet
            routers are connected.

LAG:        Link Aggregation Group

MAC or IP DA:  MAC or IP Destination Address

MAC or IP SA:  MAC or IP Source Address

ND:         Neighbor Discovery

NS:         Neighbor Solicitation

NA:         Neighbor Advertisement

NUD:        Neighbor Unreachability Detection, as per [RFC4861]

O Flag:     Override Flag in NA messages, as per [RFC4861]

PE:         Provider Edge router

R Flag:     Router Flag in NA messages, as per [RFC4861]

RT2:        EVPN Route type 2 or EVPN MAC/IP Advertisement route, as
            per [RFC7432]

S Flag:     Solicited Flag in NA messages, as per [RFC4861]

SN-multicast address:  Solicited-Node IPv6 multicast address used by
            NS messages

TLLA:       Target Link Layer Address, as per [RFC4861]

VPLS:       Virtual Private LAN Service

This document assumes you're already familiar with the terminology used in [RFC7432] 🦋

3. Solution Description ✨

Okay bestie, let's look at a real example 🎀 Figure 1 shows an EVPN network where the Proxy ARP/ND function is turned on.

                                                         BD1
                                                     Proxy ARP/ND
                                                    +------------+
   IP1/M1          +----------------------------+   |IP1->M1 EVPN|
    GARP --->Proxy ARP/ND                       |   |IP2->M2 EVPN|
   +---+      +--------+   RT2(IP1/M1)          |   |IP3->M3 sta |
   |CE1+------|   BD1  |    ------>      +------+---|IP4->M4 dyn |
   +---+      +--------+                 |          +------------+
                  PE1                    | +--------+ Who has IP1?
                   |           EVPN      | |   BD1  | <-----  +---+
                   |           EVI1      | |        | ----->  |CE3|
   IP2/M2          |                     | |        | IP1->M1 +---+
    GARP  --->Proxy ARP/ND               | +--------+   |    IP3/M3
     +---+    +--------+   RT2(IP2/M2)   |              |
     |CE2+----|   BD1  |    ------>      +--------------+
     +---+    +--------+                       PE3|    +---+
                  PE2                           | +----+CE4|
                   +----------------------------+      +---+
                                                 <---IP4/M4 GARP

                   Figure 1: Proxy ARP/ND Network Example

Every BD (Broadcast Domain) on the EVPN PEs that has Proxy ARP/ND enabled gets its own Proxy table 💅 Each table can hold three flavours of Proxy ARP/ND entries:

  • Dynamic entries: these get learned by snooping a CE's ARP and ND messages. Figure 1 has IP4->M4 as an example. 👀
  • Static entries: these are provisioned on the PE by the management system. Figure 1 has IP3->M3 as an example. 🧚
  • EVPN-learned entries: these come from the IP/MAC info inside the RT2's that arrive from remote PEs. Figure 1 has IP1->M1 and IP2->M2 as examples. 🦋

Now the walkthrough, which is so satisfying 🌈 We'll say IP1, IP2, and IP3 are IPv4 addresses:

  1. Proxy ARP/ND gets switched on in BD1 of PE1, PE2, and PE3. 🔥
  2. The PEs start filling their Proxy tables with dynamic, static, and EVPN-learned entries:
    • a. PE3 adds IP1->M1 and IP2->M2 based on the EVPN routes it gets from PE1 and PE2. Those entries were learned as dynamic entries on PE1 and PE2 first, then advertised in BGP EVPN. ✨
    • b. PE3 adds IP4->M4 as dynamic, because it snooped the matching ARP messages sent by CE4. 👂
    • c. An operator provisions the static entry IP3->M3. 🛠️
  3. CE3 sends an ARP Request asking for the MAC address of IP1, and PE3 does this:
    • a. It intercepts the ARP Request and runs a Proxy ARP lookup for IP1. 🕵️‍♀️
    • b. If the lookup works (like in Figure 1), PE3 sends back an ARP Reply with IP1->M1. The ARP Request doesn't get flooded to the EVPN network or to any other local CEs. Iconic, no notes 💖
    • c. If the lookup fails, PE3 floods the ARP Request out to the EVPN network and the other local CEs. 🌊

Now let's do the same thing with IPv6. Say IP1, IP2, IP3, and IP4 are all IPv6 addresses now, and Proxy ARP/ND is turned on in BD1:

  1. PEs add entries in a similar way to IPv4, but there are a few differences, so watch out 👀
    • a. IP1->M1 and IP2->M2 are learned as dynamic entries on PE1 and PE2 by snooping NA messages, not NS messages. With IPv4, any ARP frame can be snooped to learn a dynamic Proxy ARP entry. For IPv6, the R and O Flags in the snooped NA messages also get copied into the Proxy ND entries. 🎀
    • b. PE1 and PE2 advertise those entries in EVPN MAC/IP Advertisement routes, and they include the learned R and O Flags in the ARP/ND Extended Community. 📣
    • c. PE3 also adds IP4->M4 as dynamic after it snoops an NA message sent by CE4. 💜
  2. CE3 sends an NS message asking for the MAC address of IP1. PE3 acts just like the IPv4 example: it intercepts the NS, does a lookup on the IP, and replies with an NA if the lookup works. If it works, the NS doesn't get flooded to the EVPN PEs or to any other local CEs. 🫶
  3. If the lookup fails, PE3 floods the NS to the remote EVPN PEs attached to the same BD and to the other local CEs, same as in the IPv4 case. 🌸

As PE3 collects more and more host entries in its Proxy ARP/ND table, it floods fewer ARP Request messages among PEs, and sometimes it doesn't flood them at all 🥹 In a network where most of the participating CEs aren't moving between PEs and keep announcing themselves with GARPs or unsolicited-NA messages, ARP/ND flooding among PEs can basically disappear. Unknown unicast flooding can too. In an EVPN-based IXP network, where every entry is static, the ARP/ND flooding among PEs is totally suppressed. Main character energy 👑

Heads up: when CEs move between PEs, the Proxy ARP/ND function depends on the CE announcing its new location with a GARP or unsolicited-NA message, so the tables update right away. 🚨

If a CE moves "silently", meaning it doesn't send any GARP or NA message after it attaches to the destination PE, the mechanisms in Section 3.5 make sure the Proxy ARP/ND tables eventually catch up ⭐

3.1. Proxy ARP/ND Sub-functions

The Proxy ARP/ND function can be broken into six sub-functions, or procedures 🧩

  1. Learning sub-function
  2. Reply sub-function
  3. Unicast-forward sub-function
  4. Maintenance sub-function
  5. Flood handling sub-function
  6. Duplicate IP detection sub-function

A Proxy ARP/ND implementation MUST at least support the Learning, Reply, Maintenance, and duplicate IP detection sub-functions. 🫶 The next sections go through each sub-function one at a time.

3.2. Learning Sub-function

A Proxy ARP/ND implementation in an EVPN BD MUST support dynamic and EVPN-learned entries, and SHOULD support static entries. 💅

Static entries come from the management plane. A static entry gets configured on the PE attached to the host that uses the IP address in that entry. The provisioned static IP->MAC entry MUST be advertised in EVPN with an ARP/ND Extended Community where the Immutable ARP/ND Binding Flag (I) is set to 1, as per [RFC9047]. When the I Flag in the ARP/ND Extended Community is 1, the advertising PE is saying the IP address must not be associated to a MAC other than the one in the EVPN MAC/IP Advertisement route. Setting I = 1 in the ARP/ND Extended Community works with any value of the Sticky bit (S) or sequence number in the [RFC7432] MAC Mobility Extended Community. 🔒 Heads up, these two bits are different: the I bit is about the fixed, configured IP-to-MAC pairing in the IP->MAC binding, while the S bit says the advertised MAC isn't subject to the [RFC7432] mobility procedures. Don't mix them up, bestie 🙈

An entry can link one configured static IP to a list of possible MACs, like IP1->(MAC1,MAC2..MACN). Until a frame (including a local ARP/NA message) comes in from the CE, the PE won't advertise any IP1->MAC in EVPN. ⏳ When traffic arrives from the CE, the PE checks that the source MAC, for example MAC1, is in the list of allowed MACs. Only then does the PE activate IP1->MAC1, and it advertises just that IP1 and MAC1 in an EVPN MAC/IP Advertisement route. ✅

The PE MUST create EVPN-learned entries from the valid EVPN MAC/IP Advertisement routes it receives that contain a MAC and an IP address. 📬

Dynamic entries are learned in different ways depending on whether the entry has an IPv4 or an IPv6 address:

Proxy ARP dynamic entries:

  • The PE MUST snoop all ARP packets (that is, all frames with Ethertype 0x0806) received from the CEs attached to the BD in order to learn dynamic entries. ARP packets from remote EVPN PEs attached to the same BD don't get snooped. The Learning function copies the sender MAC and sender IP from the snooped ARP packet into the Proxy ARP table. Note that a MAC or an IP address with value 0 SHOULD NOT be learned. 🙅‍♀️

Proxy ND dynamic entries:

  • The PE MUST snoop the NA messages (Ethertype 0x86dd, ICMPv6 type 136) received from the CEs attached to the BD, and learn dynamic entries from the Target Address and TLLA information. NA messages from remote EVPN PEs don't get snooped. A PE implementing Proxy ND as in this document MUST NOT create dynamic IP->MAC entries from NS messages, because NS messages don't have the R Flag that the Proxy ND reply function requires. See Section 3.2.1 for more about the R Flag. 🚫
  • This document describes an "anycast" capability that you can configure for the PE's Proxy ND function. It changes how dynamic Proxy ND entries are learned, based on the O Flag of the snooped NA messages. If the O Flag is zero in the received NA message, the IP->MAC SHOULD only be learned when the IPv6 "anycast" capability is enabled in the BD. Either way, an NA message with O Flag = 0 still gets forwarded normally by the PE, based on a MAC DA lookup. 🌈

The following procedure for the Learning sub-function is RECOMMENDED:

  • When a new Proxy ARP/ND EVPN or static active entry is learned (or provisioned), the PE SHOULD send a GARP or unsolicited-NA message to all the connected access CEs. The PE SHOULD send a GARP or unsolicited-NA message for dynamic entries only if the ARP/NA message that created the entry on the PE was NOT flooded to all the local connected CEs before. This GARP/unsolicited-NA message makes sure the CE ARP/ND caches get updated, even when the ARP/NS/NA messages from CEs connected to remote PEs aren't flooded in the EVPN network. 📢

Note that if a static entry is provisioned with the same IP as an existing EVPN-learned or dynamic entry, the static entry wins 👑

After a PE reboots, the static and EVPN entries get re-added as soon as the PE is back online and has received all the EVPN routes for the BD. The dynamic entries, though, are gone. 😭 Because of that, the PE floods new NS and ARP Requests to remote PEs, and the dynamic entries gradually get relearned 🔄

3.2.1. Proxy ND and the NA Flags ✨

okay so [RFC4861] is the one that explains how the R Flag works in IPv6 address resolution 💅 basically:

  • Nodes capable of routing IPv6 packets must reply to NS messages with NA messages where the R Flag is set (R Flag = 1).
  • Hosts that are not able to route IPv6 packets must indicate that inability by replying with NA messages that contain R Flag = 0.

and this part is lowkey a big deal for default gateways 🌸 the R Flag in NA messages changes how hosts pick their default gateways when they send packets off-link, as per [RFC4861]:

  • Hosts build a Default Router List based on the received RAs and NAs with R Flag = 1. Each cache entry has an IsRouter flag, which must be set for received RAs and is set based on the R Flag in the received NAs. A host can choose one or more Default Routers when sending packets off-link.
  • In those cases where the IsRouter flag changes from TRUE to FALSE as a result of an NA update, the node must remove that router from the Default Router List and update the Destination Cache entries for all destinations using that neighbor as a router, as specified in Section 7.3.3 of [RFC4861]. This is needed to detect when a node that is used as a router stops forwarding packets due to being configured as a host. 🫶

The R and O Flags for a Proxy ARP/ND entry get learned in these ways:

  • The R Flag information SHOULD be added to the static entries by the management interface. The O Flag information MAY also be added by the management interface. If the R and O Flags are not configured, the default value is 1.
  • Dynamic entries SHOULD learn the R Flag and MAY learn the O Flag from the snooped NA messages used to learn the IP->MAC itself.
  • EVPN-learned entries SHOULD learn the R Flag and MAY learn the O Flag from the ARP/ND Extended Community [RFC9047] received from EVPN along with the RT2 used to learn the IP->MAC itself. If no ARP/ND Extended Community is received, the PE will add a configured R Flag / O Flag to the entry. These configured R and O Flags MAY be an administrative choice with a default value of 1. The configuration of this administrative choice provides a backwards-compatible option with EVPN PEs that follow [RFC7432] but do not support this specification. 🦋

Note that, typically, IP->MAC entries with O = 0 won't get learned, so the Proxy ND function replies to NS messages with NA messages that contain O = 1. BUT this document lets you configure the "anycast" capability in the BD where the Proxy ND function is enabled, and that's where it gets spicy 🔥 If "anycast" is enabled in the BD and an NA message with O = 0 is received, the associated IP->MAC entry will be learned with O = 0. If this "anycast" capability is enabled in the BD, duplicate IP detection must be disabled so that the PE is able to learn the same IP mapped to different MACs in the same Proxy ND table. If the "anycast" capability is disabled, NA messages with O Flag = 0 will not create a Proxy ND entry (although they still get forwarded normally), so no EVPN advertisement with ARP/ND Extended Community will be generated. 💀

3.3. Reply Sub-function

This sub-function replies to address resolution requests/solicitations once a lookup in the Proxy ARP/ND table succeeds for a given IP address 🧚 The following considerations should be taken into account, assuming that the ARP Request / NS lookup hits a Proxy ARP/ND entry IP1->MAC1:

a. When replying to ARP Requests or NS messages:

  • The PE SHOULD use the Proxy ARP/ND entry MAC address MAC1 as MAC SA. This is RECOMMENDED so that the resolved MAC can be learned in the MAC forwarding database of potential Layer 2 switches sitting between the PE and the CE requesting the address resolution.
  • For an ARP reply, the PE MUST use the Proxy ARP entry IP1 and MAC1 addresses in the sender Protocol Address and Hardware Address fields, respectively.
  • For an NA message in response to an address resolution NS or DAD NS, the PE MUST use IP1 as the IP SA and Target Address. M1 MUST be used as the Target Link Local Address (TLLA). 👑

b. A PE SHOULD NOT reply to a request/solicitation received on the same attachment circuit over which the IP->MAC is learned. In this case, the requester and the requested IP are assumed to be connected to the same Layer 2 CE/access network linked to the PE's attachment circuit, so the requested IP owner will receive the request directly. 🙈

c. A PE SHOULD reply to broadcast/multicast address resolution messages, i.e., ARP Requests, ARP probes, NS messages, as well as DAD NS messages. An ARP probe is an ARP Request constructed with an all-zero sender IP address that may be used by hosts for IPv4 Address Conflict Detection as specified in [RFC5227]. A PE SHOULD NOT reply to unicast address resolution requests (for instance, NUD NS messages). 🌈

d. When replying to an NS, a PE SHOULD set the Flags in the NA messages as follows:

  • The R bit is set as it was learned for the IP->MAC entry in the NA messages that created the entry (see Section 3.2.1).
  • The S Flag will be set/unset as per [RFC4861].
  • The O Flag will be set in all the NA messages issued by the PE, except in the case where the BD is configured with the "anycast" capability and the entry was previously learned with O = 0. If "anycast" is enabled and there is more than one MAC for a given IP in the Proxy ND table, the PE will reply to NS messages with as many NA responses as "anycast" entries there are in the Proxy ND table. 🍓

e. For Proxy ARP, a PE MUST only reply to ARP Requests with the format specified in [RFC0826]. 💜

f. For Proxy ND, a PE MUST reply to NS messages with known options with the format and options specified in [RFC4861] and MAY reply, discard, forward, or unicast-forward NS messages containing other options. An administrative choice to control the behavior for received NS messages with unknown options ("reply", "discard", "unicast-forward", or "forward") MAY be supported.

  • The "reply" option implies that the PE ignores the unknown options and replies with NA messages, assuming a successful lookup on the Proxy ND table. An unsuccessful lookup will result in a "forward" behavior (i.e., flood the NS message based on the MAC DA).
  • If "discard" is available, the operator should assess if flooding NS unknown options may be a security risk for the EVPN BD (and if so, enable "discard") or, on the contrary, if not forwarding/flooding NS unknown options may disrupt connectivity. This option discards NS messages with unknown options irrespective of the result of the lookup on the Proxy ND table.
  • The "unicast-forward" option is described in Section 3.4.
  • The "forward" option implies flooding the NS message based on the MAC DA. This option forwards NS messages with unknown options irrespective of the result of the lookup on the Proxy ND table. The "forward" option is RECOMMENDED by this document. ✨

3.4. Unicast-Forward Sub-function

As discussed in Section 3.3, sometimes the operator wants to "unicast-forward" certain ARP Requests and NS messages instead of replying to them 🦋 The implementation of a "unicast-forward" function is RECOMMENDED. This option can be enabled with one of the following parameters:

a. unicast-forward always

b. unicast-forward unknown-options

If "unicast-forward always" is enabled, the PE performs a Proxy ARP/ND table lookup and, on a hit, forwards the packet to the owner of the MAC found in the Proxy ARP/ND table. This happens irrespective of the options carried in the ARP/ND packet. This option provides total transparency in the BD and yet reduces the amount of flooding significantly 🌸

If "unicast-forward unknown-options" is enabled, upon a successful Proxy ARP/ND lookup, the PE performs a "unicast-forward" action only if the ARP Requests or NS messages carry unknown options, as explained in Section 3.3. The "unicast-forward unknown-options" configuration allows the support of new applications using ARP/ND in the BD while still reducing the flooding. 🧚

Irrespective of the enabled option, if there is no successful Proxy ARP/ND lookup, the unknown ARP Request / NS message will be flooded in the context of the BD, as per Section 3.6. 💅

3.5. Maintenance Sub-function

The Proxy ARP/ND tables SHOULD follow a number of maintenance procedures, so that the dynamic IP->MAC entries are kept if the owner is active, and flushed (and the associated RT2 withdrawn) if the owner is no longer in the network. The following procedures are RECOMMENDED:

Age-time: A dynamic Proxy ARP/ND entry MUST be flushed out of the table if the IP->MAC has not been refreshed within a given age-time. The entry is refreshed if an ARP or NA message is received for the same IP->MAC entry. The age-time is an administrative option, and its value should be carefully chosen depending on the specific use case. In IXP networks (where the CE routers are fairly static), the age-time may normally be longer than in DC networks (where mobility is required). ⏰

Send-refresh option: The PE MAY send periodic refresh messages (ARP/ND "probes") to the owners of the dynamic Proxy ARP/ND entries, so that the entries can be refreshed before they age out. The owner of the IP->MAC entry would reply to the ARP/ND probe, and the corresponding entry age-time gets reset. The periodic send-refresh timer is an administrative option and is RECOMMENDED to be a third of the age-time, or a half of the age-time in scaled networks. 🔥

An ARP refresh issued by the PE will be an ARP Request message with the sender's IP = 0, sent from the PE's MAC SA. If the PE has an IP address in the subnet (for instance, on an Integrated Routing and Bridging (IRB) interface), then it MAY use it as a source for the ARP Request (instead of sender's IP = 0). An ND refresh will be an NS message issued from the PE's MAC SA and a Link Local Address associated to the PE's MAC. 🫶

The refresh request messages SHOULD be sent only for dynamic entries, and not for static or EVPN-learned entries. Even though the refresh request messages are broadcast or multicast, the PE SHOULD only send the message to the attachment circuit associated to the MAC in the IP->MAC entry. 🌈

The age-time and send-refresh options are used in EVPN networks to avoid unnecessary EVPN RT2 withdrawals. If refresh messages are sent before the corresponding BD Bridge-Table and Proxy ARP/ND age-time for a given entry expires, inactive but existing hosts will reply, refreshing the entry and therefore avoiding unnecessary EVPN MAC/IP Advertisement withdrawals in EVPN. Both entries (MAC in the BD and IP->MAC in the Proxy ARP/ND) are reset when the owner replies to the ARP/ND probe. If there is no response to the ARP/ND probe, the MAC and IP->MAC entries will be legitimately flushed and the RT2s withdrawn. 😭

3.6. Flood (to Remote PEs) Handling 🌊✨

okay so the Proxy ARP/ND function already helps cut down on flooding of ARP Requests and NS messages to remote PEs in an EVPN network 💖. But in certain use cases, the flooding of ARP/NS/NA messages (and even unknown unicast flooding) to remote PEs can be suppressed completely in an EVPN network 🙈🔥.

For instance, in an IXP network, all the participant CEs are well known and won't move to a different PE, so the IP->MAC entries for the local CEs can all be provisioned on the PEs by a management system 🎀. If the entries for the CEs are all provisioned on the local PE, a given Proxy ARP/ND table only has static and EVPN-learned entries 🧚. In this case, the operator may choose to suppress the flooding of ARP/NS/NA from the local PE to the remote PEs completely 💅.

The flooding can also be suppressed completely in IXP networks with dynamic Proxy ARP/ND entries, as long as all the CEs are directly connected to the PEs and they all announce their presence with a GARP/unsolicited-NA when they connect to the network 🦋. But if either of those two assumptions isn't true, and any of the PEs may not learn all the local Proxy ARP/ND entries, then flooding of the ARP/NS/NA messages from the local PE to the remote PEs SHOULD NOT be suppressed 😭, or the address resolution process for some CEs will never finish 💀.

In networks where fast mobility is expected (the DC use case), it is NOT RECOMMENDED to suppress the flooding of unknown ARP Requests / NS messages or GARPs/unsolicited-NAs 🏃‍♀️💨. Unknown ARP Requests / NS messages are those for which the Proxy ARP/ND lookups for the requested IPs don't succeed 🔍.

To give the operator the choice to suppress or allow flooding to remote PEs, a PE MAY support administrative options to individually suppress or allow the flooding of:

  • Unknown ARP Requests and NS messages.
  • GARP and unsolicited-NA messages.

The operator will pick these options based on the expected behavior of the CEs 👑.

3.7. Duplicate IP Detection 🕵️‍♀️💜

The Proxy ARP/ND function MUST support duplicate IP detection as described in this section, so that ARP/ND-spoofing attacks or duplicate IPs caused by human errors can be detected 🚨. For IPv6 addresses, CEs will keep carrying out the DAD procedures as per [RFC4862]. The solution in this section is an extra security mechanism run by the PEs that makes sure IPv6 address moves between PEs are legit and not the result of an attack 🛡️. [RFC6957] describes a solution for the IPv6 Duplicate Address Detection Proxy, but it's defined for point-to-multipoint topologies with split-horizon forwarding, where the "CEs" have no direct communication within the same L2 link. So it's not suitable for EVPN Broadcast Domains 🙅‍♀️. The solution here also uses the AS-MAC for extra security 🔐.

ARP/ND spoofing is when an attacker sends "fake" ARP/ND messages onto a Broadcast Domain 😈. The goal is usually to link the attacker's MAC address to another host's IP address, so any traffic meant for that IP gets sent to the attacker instead 😱.

Because EVPN and Proxy ARP/ND are distributed, duplicated IPs are easy to detect in a similar way to the MAC duplication detection function that [RFC7432] supports for MAC addresses 🧠✨.

Duplicate IP detection watches for "IP-moves" in the Proxy ARP/ND table like this:

a. When an existing active IP1->MAC1 entry gets modified, a PE starts an M-second timer (default value of M = 180). If it detects N IP moves before the timer expires (default value of N = g5), it concludes that a duplicate IP situation has happened 🚩. An IP move is seen when, for instance, IP1->MAC1 gets replaced by IP1->MAC2 in the Proxy ARP/ND table 🔄. Static IP->MAC entries, meaning locally provisioned or EVPN-learned entries with I = 1 in the ARP/ND Extended Community, are not subject to this procedure 📌. Static entries MUST NOT be overridden by dynamic Proxy ARP/ND entries 🙅‍♀️.

b. To detect the duplicate IP faster, the PE SHOULD send a Confirm message to the former owner of the IP 🏃‍♀️. A Confirm message is a unicast ARP Request / NS message that the PE sends to the MAC addresses that previously owned the IP, when the MAC changes in the Proxy ARP/ND table 📬. It uses a sender's IP of 0.0.0.0 in the case of ARP (if the PE has an IP address in the subnet, then it MAY use that one instead) 🎯, and an IPv6 Link Local Address in the case of NS 🌐. If the PE doesn't get an answer within a given time, the new entry is confirmed and activated ✅. The default RECOMMENDED time to receive the confirmation is 30 seconds ⏱️. In the case of spoofing, say IP1->MAC1 moves to IP1->MAC2, the PE may send a unicast ARP Request / NS message for IP1 with MAC DA = MAC1 and MAC SA = PE's MAC 🕵️. This forces the legitimate owner to reply if the move to MAC2 was spoofed, and makes the PE send another Confirm message, this time with MAC DA = MAC2 🔁. If both the legitimate owner and the spoofer keep replying to the Confirm message, the PE will detect the duplicate IP within the M-second timer, and a response gets triggered like this:

  • If the IP1->MAC1 pair was previously owned by the spoofer and the new IP1->MAC2 came from a valid CE, then the Confirm message triggers a response from the spoofer 😈.
  • If it's the other way around, meaning IP1->MAC1 was previously owned by a valid CE, the Confirm message triggers a response from the CE 🥰.

Either way, if this keeps going, duplicate detection will kick in 💥.

c. Upon detecting a duplicate IP situation:

  1. The entry in the duplicate detected state can't be updated with new dynamic or EVPN-learned entries for the same IP 🔒. The operator MAY override the entry, though, with a static IP->MAC 👑.
  2. The PE SHOULD alert the operator and stop responding to ARP/NS for the duplicate IP until a corrective action is taken 🛑.
  3. Optionally, the PE MAY associate an "anti-spoofing-mac" (AS-MAC) with the duplicate IP in the Proxy ARP/ND table 🧁. The PE sends a GARP/unsolicited-NA message with IP1->AS-MAC to the local CEs, as well as an RT2 (with IP1->AS-MAC) to the remote PEs 📣. This updates the ARP/ND caches on all the CEs in the BD, so all the CEs in the BD use the AS-MAC as MAC DA when sending traffic to IP1 🌈. This stops the spoofer from attracting any traffic for IP1 🙌. Since the AS-MAC is a managed MAC address known by all the PEs in the BD, all the PEs MAY apply filters to drop and/or log any frame with MAC DA = AS-MAC 🗂️. Advertising the AS-MAC as a "drop-MAC" (using an indication in the RT2) that can be used directly in the BD to drop frames is for further study 🔮.

d. The duplicate IP situation is cleared when the operator takes a corrective action or, alternatively, after a HOLD-DOWN timer (default value of 540 seconds) ⏳🧹.

The values of M, N, and the HOLD-DOWN timer SHOULD be a configurable administrative option, so there's the flexibility needed for different scenarios 🎛️💖.

For Proxy ND, the duplicate IP detection described in this section SHOULD only monitor IP moves for IP->MACs learned from NA messages with O Flag = 1 🎯. NA messages with O Flag = 0 wouldn't override the ND cache entries for an existing IP, so this procedure wouldn't detect duplicate IPs 🙃. This duplicate IP detection for IPv6 SHOULD be disabled when the IPv6 "anycast" capability is activated in a given BD 🌸.

4. Solution Benefits ✨🏆

The solution in this document gives you the following benefits:

a. It may completely suppress the flooding of the ARP/ND messages in the EVPN network, as long as all the CE IP->MAC addresses local to the PEs are known or provisioned on the PEs from a management system 💯. Note that in this case, unknown unicast flooded traffic can also be suppressed, since all the expected unicast traffic will go to known MAC addresses in the PE BDs 🧼.

b. It significantly reduces the flooding of the ARP/ND messages in the EVPN network, as long as some or all the CE IP->MAC addresses are learned on the data plane by snooping ARP/ND messages sent by the CEs 👀.

c. It refreshes the CE IP->MAC entries learned through the data plane periodically, so they don't get withdrawn by EVPN when they age out, unless the CE isn't active anymore ♻️. This option cuts down the EVPN control plane overhead in a network with active CEs that don't send packets very often 🐢💜.

d. It gives you a mechanism to detect duplicate IP addresses and avoid ARP/ND-spoof attacks, or the effects of duplicate addresses caused by human errors 🛡️🫶.

5. Deployment Scenarios 🗺️🌸

Four deployment scenarios with different levels of ARP/ND control are available to operators using this solution, depending on how they want to manage ARP/ND: all dynamic learning, all dynamic learning with Proxy ARP/ND, hybrid dynamic learning and static provisioning with Proxy ARP/ND, and all static provisioning with Proxy ARP/ND 🎀.

5.1. All Dynamic Learning 🥱

In this scenario, for minimum security and mitigation, EVPN is deployed in the BD with the Proxy ARP/ND function shut down 🔓. PEs don't intercept ARP/ND requests and flood all requests sent by the CEs, since a conventional Layer 2 network among those CEs would be enough 🪄. Even though no ARP/ND mitigation is used here, the operator can still use EVPN features like control plane learning and all-active multihoming in the peering network 🙌.

Although this option doesn't require any of the procedures in this document, it's included as a baseline/default option for completeness 📚. This option is equivalent to VPLS as far as ARP/ND is concerned 🤝. The options described in Sections 5.2, 5.3, and 5.4 are only possible in EVPN networks together with their Proxy ARP/ND capabilities 💅.

5.2. Dynamic Learning with Proxy ARP/ND 🧠💫

This scenario minimizes flooding while still enabling dynamic learning of IP->MAC entries 🦋. The Proxy ARP/ND function is enabled in the BDs of the EVPN PEs, so the PEs snoop ARP/ND messages sent by the CEs and respond to CE ARP Requests / NS messages 📨.

PEs will flood requests if the entry isn't in their Proxy table 🌊. Any unknown source IP->MAC entries get learned and advertised in EVPN, and traffic to unknown entries is discarded at the ingress PE 🗑️.

This scenario uses the Learning, Reply, and Maintenance sub-functions, with optional use of the Unicast-forward and duplicate IP detection sub-functions 🧩. The Flood handling sub-function uses default flooding for unknown ARP Requests / NS messages 🌈.

5.3. Hybrid Dynamic Learning and Static Provisioning with Proxy ARP/ND ✨

Some IXPs and other operators want to protect particular hosts on the BD 🛡️ while still letting CE addresses be dynamically learned. For example, an operator might want to set up static IP->MAC entries for management and infrastructure hosts that provide critical services 💜. In this setup, the static entries are provisioned from the management plane for the protected IP->MAC addresses, and dynamic learning with Proxy ARP/ND is turned on for the BD, just like in Section 5.2 🎀.

This scenario uses the same sub-functions as Section 5.2, except that the static entries get added by the Learning sub-function 🦋.

5.4. All Static Provisioning with Proxy ARP/ND 🔒

For a solution that maximizes security and gets rid of flooding and unknown unicast in the peering network, every IP->MAC entry is provisioned from the management plane 💅. The Proxy ARP/ND function is enabled in the BDs of the EVPN PEs, so the PEs intercept and respond to CE requests 🧚. Dynamic learning and ARP/ND snooping are disabled, so ARP Requests and NS messages to unknown IPs get discarded at the ingress PE 🙈. This scenario gives an operator the most control over IP->MAC entries, and lets them manage every entry from a management system 👑.

In this scenario, the Learning sub-function is limited to static entries. The Maintenance sub-function won't need any procedures because of the static entries 🌸. The Flood handling sub-function completely suppresses unknown ARP Requests / NS messages, as well as GARP and unsolicited-NA messages 💀.

5.5. Example of Deployment in Internet Exchange Points 🌐

Nowadays, almost all IXPs install some security rules to protect the peering network (BD) 🔐. These rules are often called port security. Port security is a bundle of operational steps that limit access to the IXP-LAN and the customer router, and control the kinds of traffic the routers are allowed to exchange (e.g., Ethernet, IPv4, and IPv6) 🍓. Because of this, the deployment scenario from Section 5.4, "All Static Provisioning with Proxy ARP/ND", is the predominant scenario for IXPs 👑✨.

On top of the "All Static Provisioning" behavior, IXP networks are recommended to configure the Reply sub-function to "discard" ARP Requests / NS messages with unrecognized options 🙅‍♀️.

IXP customers usually go through a certain operational life cycle 🔄. Each step of that life cycle has its own specific operational procedures.

Here's what the operational procedures look like to guarantee port security across a customer's life cycle, with a focus on EVPN features 💖:

  1. A new customer is connected the first time to the IXP:

Before the connection between the customer router and the IXP-LAN is activated, the MAC of the router is allowlisted on the IXP's switch port. All other MAC addresses get blocked 🚫. Pre-defined IPv4 and IPv6 addresses of the IXP peering network space are configured on the customer router. The IP->MAC static entries (IPv4 and IPv6) are configured in the IXP's management system for the customer's port, so Proxy ARP/ND is supported 🎀.

If a customer uses multiple ports aggregated into a single logical port (LAG), some vendors randomly pick the MAC address of the LAG from the different MAC addresses assigned to the ports 🎲. In this case, the static entry gets used and linked to a list of allowed MACs 📋.

  1. Replacement of customer router:

If a customer router is about to be replaced, the new MAC address(es) must be installed in the management system, alongside the MAC address(es) of the currently connected router 🔁. This lets the customer swap the router without any active involvement from the IXP operator 🙌. Static entries are used for this too. Once the replacement is done, the MAC address(es) of the replaced router can be removed 🧹.

  1. Decommissioning a customer router:

If a customer router is decommissioned, it gets disconnected from the IXP PE 🔌. Right after that, the router's MAC address(es) and IP->MAC bindings can be removed from the management system 👋.

5.6. Example of Deployment in Data Centers 🏢

DCs normally have different requirements from IXPs when it comes to Proxy ARP/ND 💻. Some of the differences are listed below 💜:

a. The mobility that virtualized DCs require makes the "Dynamic Learning" or "Hybrid Dynamic and Static Provisioning" models more appropriate than the "All Static Provisioning" model 🦋.

b. IPv6 "anycast" may be required in DCs, while it typically isn't a requirement in IXP networks 🌈. So, if the DC needs IPv6 anycast addresses, the "anycast" capability gets explicitly enabled in the Proxy ND function, and the Proxy ND sub-functions get modified to match. For instance, if IPv6 "anycast" is enabled in the Proxy ND function, the duplicate IP detection procedure in Section 3.7 must be disabled 🙈.

c. DCs may require special options on ARP/ND, as opposed to the address resolution function, which is the only one typically required in IXPs 🧠. Based on that, the Reply sub-function may be modified to forward or discard unknown options 🔀.

6. Security Considerations 🔐

The security considerations of [RFC7432] and [RFC9047] apply to this document too. EVPN does not inherently provide cryptographic protection (including confidentiality protection). Be careful out there, bestie 😭

The procedures in this document cut down the amount of ARP/ND message flooding, which by itself protects "slow path" software processors of routers and Tenant Systems in large BDs 🛡️. ARP/ND requests that get replied to by the Proxy ARP/ND function (and so aren't flooded) are normally aimed at existing hosts in the BD 🎯. ARP/ND requests aimed at absent hosts are still normally flooded. However, the suppression of unknown ARP Requests and NS messages described in Section 3.6 can add an extra layer of security against ARP Requests / NS messages sent to non-existing hosts 🕵️‍♀️.

The unicast-forward and/or flood suppression sub-functions give the BD an added security mechanism, but they can also raise the risk of blocking service for a CE if the EVPN PEs can't provide the ARP/ND resolution that the CE needs 🙃.

The solution also protects against Denial-of-Service (DoS) attacks that use ARP/ND spoofing as a first step 🥷. The duplicate IP detection and the use of an AS-MAC, as explained in Section 3.7, protect the BD against ARP/ND spoofing 💜.

The Proxy ARP/ND function specified in this document doesn't allow an IP address to be learned mapped to multiple MAC addresses in the same table, unless the "anycast" capability is enabled (and only for Proxy ND). When "anycast" is enabled in the Proxy ND function, the number of allowed entries for the same IP address MUST be limited by the operator 🚧. That stops DoS attacks from trying to fill the Proxy ND table with loads of entries for the same IP 🙅‍♀️.

This document gives some examples and guidelines that IXPs can use in their EVPN BDs 📚. When EVPN and its associated Proxy ARP/ND function are used in IXP networks, they provide ARP/ND security and mitigation 🌸. IXPs must still employ additional security mechanisms that protect the peering network, following established BCPs such as the ones described in [EURO-IX-BCP] 🫶. For example, IXPs should disable all unneeded control protocols and block unwanted protocols from CEs, so that only IPv4, ARP, and IPv6 Ethertypes are permitted on the peering network 🚦. Port security features and ACLs can add yet another layer of security ✨.

Finally, it's worth noting that the Proxy ARP/ND solution in this document won't work if there's a mechanism securing ARP/ND exchanges among CEs, because the PE isn't able to secure the "proxied" ND messages 💀.

7. IANA Considerations 📝

This document has no IANA actions 🌷.

8. References 📖

8.1. Normative References

[RFC0826]  Plummer, D., "An Ethernet Address Resolution Protocol: Or
           Converting Network Protocol Addresses to 48.bit Ethernet
           Address for Transmission on Ethernet Hardware", STD 37,
           RFC 826, DOI 10.17487/RFC0826, November 1982,
           <https://www.rfc-editor.org/info/rfc826>.

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119,
           DOI 10.17487/RFC2119, March 1997,
           <https://www.rfc-editor.org/info/rfc2119>.

[RFC4861]  Narten, T., Nordmark, E., Simpson, W., and H. Soliman,
           "Neighbor Discovery for IP version 6 (IPv6)", RFC 4861,
           DOI 10.17487/RFC4861, September 2007,
           <https://www.rfc-editor.org/info/rfc4861>.

[RFC5227]  Cheshire, S., "IPv4 Address Conflict Detection", RFC 5227,
           DOI 10.17487/RFC5227, July 2008,
           <https://www.rfc-editor.org/info/rfc5227>.

[RFC7432]  Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
           Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
           Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
           2015, <https://www.rfc-editor.org/info/rfc7432>.

[RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
           2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
           May 2017, <https://www.rfc-editor.org/info/rfc8174>.

[RFC9047]  Rabadan, J., Ed., Sathappan, S., Nagaraj, K., and W. Lin,
           "Propagation of ARP/ND Flags in an Ethernet Virtual
           Private Network (EVPN)", RFC 9047, DOI 10.17487/RFC9047,
           June 2021, <https://www.rfc-editor.org/info/rfc9047>.

8.2. Informative References

[EURO-IX-BCP]
           Euro-IX, "European Internet Exchange Association",
           <https://www.euro-ix.net/en/forixps/set-ixp/ixp-bcops>.

[RFC4862]  Thomson, S., Narten, T., and T. Jinmei, "IPv6 Stateless
           Address Autoconfiguration", RFC 4862,
           DOI 10.17487/RFC4862, September 2007,
           <https://www.rfc-editor.org/info/rfc4862>.

[RFC6820]  Narten, T., Karir, M., and I. Foo, "Address Resolution
           Problems in Large Data Center Networks", RFC 6820,
           DOI 10.17487/RFC6820, January 2013,
           <https://www.rfc-editor.org/info/rfc6820>.

[RFC6957]  Costa, F., Combes, J-M., Ed., Pougnard, X., and H. Li,
           "Duplicate Address Detection Proxy", RFC 6957,
           DOI 10.17487/RFC6957, June 2013,
           <https://www.rfc-editor.org/info/rfc6957>.

Acknowledgments 🫶

The authors want to thank Ranganathan Boovaraghavan, Sriram Venkateswaran, Manish Krishnan, Seshagiri Venugopal, Tony Przygienda, Robert Raszuk, and Iftekhar Hussain for their review and contributions 🌟. Thank you to Oliver Knapp as well for his detailed review 💖.

Contributors 🎀

In addition to the authors listed on the front page, the following coauthors have also contributed to this document 🦋:

Wim Henderickx, Nokia 🌸

Daniel Melzer, DE-CIX Management GmbH 🌈

Erik Nordmark, Zededa ✨

Authors' Addresses 🎀

okay bestie, these are the people who wrote this spec, and they're all giving Nokia and DE-CIX energy 💅✨ here's where you can find them:

Jorge Rabadan (editor) 👑 Nokia 777 Middlefield Road Mountain View, CA 94043 United States of America Email: jorge.rabadan@nokia.com

Senthil Sathappan 🦋 Nokia 701 E. Middlefield Road Mountain View, CA 94043 United States of America Email: senthil.sathappan@nokia.com

Kiran Nagaraj 🌸 Nokia 701 E. Middlefield Road Mountain View, CA 94043 United States of America Email: kiran.nagaraj@nokia.com

Greg Hankins 🧚 Nokia Email: greg.hankins@nokia.com

Thomas King 💜 DE-CIX Management GmbH Email: thomas.king@de-cix.net

and that's a wrap on the whole RFC!! 🫶 the protocol nerds behind it are real ones, no cap 😭🔥

📦 where this page comes from