RFC 7094

Architectural Considerations of IP Anycast

📅 published January 2014, from ietf

Architectural Considerations of IP Anycast 💖

  • Series: Internet Architecture Board (IAB) 🎀
  • RFC number: 7094 ✨
  • Authors: D. McPherson (Verisign, Inc.), D. Oran (Cisco Systems), D. Thaler (Microsoft Corporation), E. Osterweil (Verisign, Inc.) 🫶
  • Category: Informational 📚
  • ISSN: 2070-1721
  • Date: January 2014 🗓️
  • Obsoletes / Updates: this RFC doesn't list any documents it obsoletes or updates 🙈

Abstract

okay so this memo is all about the architectural side of IP anycast 🦋 it looks at what anycast means for the design of the Internet and also gives a little history of how different IETF protocols have used it over the years 🧚

Status of This Memo 🌸

so first things first, this is NOT a standards track spec. it's published for information only, bestie 💅

this document is a product of the Internet Architecture Board (IAB), and it's basically the IAB saying "this is valuable info, let's keep it on the permanent record" 📜 it reflects the IAB's consensus. and any document the IAB approves for publication can't be a candidate for any level of Internet Standard. see Section 2 of RFC 5741 if you want the details 🤓

if you want the current status of this doc, any errata, or a way to send feedback, check http://www.rfc-editor.org/info/rfc7094 🔗

Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. 🔒

this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they were in effect on the date it was published. please actually read those, because they spell out your rights and restrictions for this document 📖

Table of Contents 🗂️

1. Overview ........................................................2
2. Background ......................................................3
   2.1. Anycast History ............................................3
   2.2. Anycast in IPv6 ............................................6
   2.3. DNS Anycast ................................................6
   2.4. BCP 126 on Operation of Anycast Services ...................8
3. Principles ......................................................8
   3.1. Layering and Resiliency ....................................8
   3.2. Anycast Addresses as Destinations ..........................9
   3.3. Anycast Addresses as Sources ..............................10
   3.4. Service Discovery .........................................10
4. Analysis .......................................................11
   4.1. Regarding Widespread Anycast Use ..........................11
   4.2. Transport Implications ....................................11
   4.3. Stateful Firewalls, Middleboxes, and Anycast ..............12
   4.4. Security Considerations ...................................12
   4.5. Deployment Considerations .................................15
5. Conclusions ....................................................16
6. Acknowledgements ...............................................16
7. Informative References .........................................16
   Appendix A. IAB Members at the Time of Approval ...................21

1. Overview 🌈

okay, IP anycast has a long history and some seriously interesting engineering challenges 🤯 but at its core it's a pretty simple idea ✨ as BCP 126 [RFC4786] describes it, anycast in general means making one particular Service Address available in multiple discrete, autonomous locations, so that datagrams sent to that address get routed to one of the available locations 📬

IP anycast is used for at least one critical Internet service, and that's the Domain Name System [RFC1035] root servers 👑 by late 2007, at least 10 of the 13 root name servers were already using IP anycast [RSSAC29]. and it's growing for other stuff too 📈 it's been used for DNS resolution services for over a decade and it's still used by several DNS Top Level Domain (TLD) operators today. it's also used in operational environments for other services, including Network Time Protocol (NTP) [RFC5905] services ⏰

here's the big vibe check: anycast addresses look exactly like unicast addresses 🫠 syntactically they're indistinguishable. anycast addressing is basically the same as unicast addressing, just with the same address showing up in multiple locations 🗺️ destination-based routing does best-effort delivery of a packet to one interface out of the set of interfaces claiming reachability for that address. the expectation is delivery to the "closest" instance, as judged by unicast routing topology metrics. there's also a chance that load-balancing techniques (for example per-packet or per-microflow) get used across multiple equal-cost routes to spread the load for an anycasted prefix ⚖️

unlike IP unicast, it is not considered an error to assert the same anycast address on multiple interfaces, whether those interfaces are in the same system or in multiple systems 🙌

now, when you use IP anycast, there are SO many pitfalls and subtleties 😭 with applications, with transports, and with routing configuration and operation. in this doc, we're aiming to capture a lot of the architectural implications of IP anycast 🧠

BCP 126 [RFC4786] talks about several different deployment models for IP anycast. this document adds two more distinctions: "off-link anycast" and "on-link anycast" 💅

"off-link anycast" uses routing protocol preferences and the IP hop-by-hop destination-based forwarding paradigm to steer packets toward the "closest" destination. this is the traditional kind of anycast that BCP 126 [RFC4786] mostly focuses on, and it can be used for IPv4 and IPv6 🌍

"on-link anycast" is formal support for anycast in the address resolution (duplicate address detection) protocol. it's only standardized for IPv6 🌟 it works with designated anycast addresses on the anycasted hosts, plus the Override flag in Neighbor Discovery (ND) Neighbor Advertisements (NAs) [RFC4861]. there is no standardized mechanism for this in IPv4 🙃

2. Background 🦋

as of this writing, the term "anycast" shows up in 176 RFCs and 144 active Internet-Drafts 📚🤯 the following sections capture some of the key appearances and discussions of anycasting within the IETF over the years 💜

2.1. Anycast History ✨

okay so the very first formal spec for anycast came from "Host Anycasting Service" [RFC1546] 💅 and honestly those authors were so on it. They caught most of the issues IP anycast still has today 🫶

One of the earliest known uses was in 1994 for a "Video Registry" experiment [IMR9401] 🎬 A UDP query got sent to an anycasted address to find the topologically closest "supposedly equivalent network resource":

      A video resource (for example, a catalog server that lists
      available video clips) sends an anycast UDP datagram to locate the
      nearest video registry.  At most one registry responds with a
      unicast UDP datagram containing the registry's IP address.  Said
      resource then opens a TCP connection to that [the received
      registry address] address and sends a request to register itself.
      Every 5 minutes or so, each registry multicasts to all other
      registries all of the resources it knows from local registration
      requests.  It also immediately announces newly registered
      resources.  Remotely registered resources not heard about for 20
      minutes are dropped.

There's also talk that ISPs started using anycast for DNS resolution around the same time 👀 but no public references exist to back that up, so we'll just leave it there 🤷‍♀️

In 1997, the IAB clarified that IPv4 anycast addresses were pure "locators" and could never serve as "identifiers" of hosts or interfaces [RFC2101] 🧭 that's a big deal, bestie.

In 1998, the IAB ran a routing workshop [RFC2902] 🛠️ One of the output action items was an Anycast section, which lives in Section 2.10.3. The big asks were describing the advantages and disadvantages of anycast, and a belief that local-scoped well-known anycast addresses would be useful to some applications 💜 The next section then outlined an action item suggesting a BOF be held to plan anycast work. If a working group formed, a paper on the advantages and disadvantages of anycast was supposed to be part of its charter.

Because of that recommendation in [RFC2902], an Anycast BOF [ANYCASTBOF] happened at IETF 46 in November of 1999 🌸 A bunch of uses got discussed. No firm conclusion came out about using TCP with anycasted services, but people noticed anycasting was useful for DNS, even though it brought some new complexities. Global anycast wasn't expected to scale (more on that in Section 4.1 below), so it was expected to stay limited to a small number of key uses.

In 2001, the Multicast and Anycast Group Membership [MAGMA] WG was chartered 🦋 Its job was host-to-router signaling, including initial authentication and access control for multicast and anycast group membership. Other anycast stuff, like architecture and routing, was out of scope.

Simple Network Time Protocol (SNTP) Version 4 [RFC2030] explained how to use SNTP anycast for server discovery ⏰ That was extended in [RFC4330] as an NTP-specific "manycast" service, where anycast handles the discovery part.

IPv6 reserved some subnet anycast addresses [RFC2526] and gave one to "Mobile IPv6 Home-Agents" [RFC3775] (which [RFC6275] later obsoleted) 🧚

The original IPv6 transition mechanism [RFC2893] used IPv4 anycast addresses as tunnel endpoints for IPv6 encapsulated in IPv4, but that got removed later [RFC4213] 🗑️ The 6to4 tunneling protocol [RFC3056] got a 6to4 relay anycast prefix [RFC3068] to make 6to4 routers easier to configure 🔧 Lowkey though, 6to4 deployment ran into a fair number of operational and security issues [RFC3964] that came from using anycast as a discovery mechanism 😬 One takeaway is that operational care is needed so anycast addresses get advertised and/or filtered in a way that gives the intended scope. For example, you might only advertise a route for your 6to4 relay to Autonomous Systems (ASes) that follow your own acceptable usage policy. That can get operationally expensive really fast 💸

In 2002, DNS' use of anycast was first specified in "Distributing Authoritative Name Servers via Shared Unicast Addresses" [RFC3258] 🌐 Notably, it used the term "shared unicast address" instead of "anycast address." That was because of how IPv6 handles the on-link model. "Shared unicast" addresses are unicast (not multicast) in the IPv6 model, so they support the off-link anycast model (described earlier) but not the on-link anycast model 🎀 Around the same time, site-local-scoped well-known addresses started being used for recursive resolvers [DNS-DISC], but that use never got standardized (more on that in Section 3.4 below).

Anycast was also used for routing to rendezvous points (RPs) for PIM [RFC4610] 📡

"Operation of Anycast Services" BCP 126 [RFC4786] covers how the routing system interacts with anycast services and how those services get operated 🛰️

"Requirements for a Mechanism Identifying a Name Server Instance" [RFC4892] pointed to anycast with DNS as the motivation for identifying individual name server instances. That's where the Name Server ID (NSID) option came from [RFC5001] 🆔 You could look at NSID as a case of locator and identifier separation, where the anycast address is the locator and the NSID is the identifier. Iconic, honestly 👑

The IAB's "Reflections on Internet Transparency" [RFC4924] briefly mentions that violating transparency can also hurt global services that use anycast 🫣

2.2. Anycast in IPv6 💖

Originally, the IPv6 addressing architecture [RFC1884] [RFC2373] [RFC3513] seriously restricted anycast addresses 😤 Specifically, the architecture said anycast addresses must not be used as source addresses and must not be assigned to IPv6 hosts (so only routers could have them). Those restrictions got lifted in 2006 [RFC4291] 🥳

The more recent "IPv6 Transition/Co-existence Security Considerations" [RFC4942] overview now recommends this:

      To avoid exposing knowledge about the internal structure of the
      network, it is recommended that anycast servers now take advantage
      of the ability to return responses with the anycast address as the
      source address if possible.

As the Overview covered, "on-link anycast" is used explicitly in IPv6 via ND NAs. See Section 7.2.7 of [RFC4861] for more info 📚

2.3. DNS Anycast 🌍

"Distributed Authoritative Name Servers via Shared Unicast Addresses" [RFC3258] described how to reach authoritative name servers using multiple unicast addresses, each configured on a different set of servers. In Section 2.3, it said:

      This document presumes that the usual DNS failover methods are the
      only ones used to ensure reachability of the data for clients.  It
      does not advise that the routes be withdrawn in the case of
      failure; it advises instead that the DNS process shutdown so that
      servers on other addresses are queried.  This recommendation
      reflects a choice between performance and operational complexity.
      While it would be possible to have some process withdraw the route
      for a specific server instance when it is not available, there is
      considerable operational complexity involved in ensuring that this
      occurs reliably.  Given the existing DNS failover methods, the
      marginal improvement in performance will not be sufficient to
      justify the additional complexity for most uses.

Here's the tea 🫖 In anycast more generally, most of the benefits can't be realized without route withdrawals, because traffic keeps going to the link with the failed server 😭 When multiple unicast addresses are used with different sets of servers, a client can still fail over to a different server address and, therefore, a different set of servers. Reliability problems can still happen, though, when each set contains a failed server. If all servers in the same set are on the same subnet, those problems could be minimized, since address resolution within the subnet will send traffic to an available server.

Other assertions in that document included:

  • It asserted (as an advantage) that no routing changes were needed ✅
  • It recommended stopping DNS processes instead of withdrawing routes to deal with failures, data synchronization issues, and failover, as in the quoted text above 🛑 The idea was that DNS resolvers may (indeed) reach out and query unavailable DNS name servers, but as their queries time out, they'll choose to pin themselves to other server addresses and, therefore, different servers.
  • It argued that failure modes involving state were not serious, because:
    • the vast majority of DNS queries are UDP
    • large routing metric disparity among authoritative server instances would localize queries to a single instance for most clients
    • when the resolver tries TCP and it breaks, the resolver will try to move to a different server address. To make sure this is possible, the DNS zone should be configured with multiple server addresses for different sets of name servers. Section 3.3 of [DNS-DISC] goes into more detail on why multiple addresses matter.

"Unique Per-Node Origin ASNs for Globally Anycasted Services" [RFC6382] gives recommendations on using per-node unique origin Autonomous System Numbers (ASNs) for globally anycasted critical infrastructure services 🏗️ The goal is to provide routing system discriminators for a given anycasted prefix. That lets network management and monitoring techniques, or other operational mechanisms, use this new origin AS as a discriminator however it fits their environment, either for detection or for policy tied to a given anycasted node 🔍

2.4. BCP 126 on Operation of Anycast Services 📘

"Operation of Anycast Services" BCP 126 [RFC4786] came out of the IETF's GROW working group 🌱 The main design constraint was that routing "be stable" for much longer than a "transaction time." A "transaction time" is loosely defined as "a single interaction between a single client and a single server." It doesn't take a position on which applications are good candidates for anycast 🤍

It also treats anycast service disruptions as an operational problem: "Operators should be aware that, especially for long running flows, there are potential failure modes using anycast that are more complex than a simple 'destination unreachable' failure using unicast" ⚠️

The document mainly covers global Internet-wide services provided by anycast. When it discusses internal topology issues, it's mostly about routing implications rather than application design implications 🗺️ BCP 126 also calls networks that use per-packet load balancing on equal cost paths "pathological," which [RFC2991] also discussed 🙈

3. Principles 🌟

3.1. Layering and Resiliency 🧚

Okay bestie, keeping IP protocols in a modular layered design is literally critical for the Internet to stay successful and flexible 💖. One big question is whether an application should have to adapt when the routing system changes 🤔.

Applications should make minimal assumptions about routing stability, just like they should make minimal assumptions about congestion and packet loss ✨. When designing applications, it would perhaps be safe to assume that the routing system may deliver each anycast packet to a different service instance, in any pattern, with temporal reordering being a not-so-rare phenomenon 🌀.

Most stateful transport protocols (e.g., TCP) don't understand anycast's properties on their own, so without modification they will fail probabilistically, but possibly catastrophically, when anycast addresses meet "normal" routing dynamics 💀. Specifically, if datagrams for an active transaction get routed to a new anycasted end system that has no state for that transaction, the session gets reset and has to be reinitiated 😭. Different networks also have different routing properties, so they'll struggle under different conditions. That means a protocol can work fine in, say, a test lab and still break on the global Internet 🙈.

3.2. Anycast Addresses as Destinations 🎀

When an anycast address is the destination, different packets with the same destination IP address may reach different destination hosts, even when the packets come from the same source host 🦋. Anycast addresses are "safe" to use as destination addresses for an application if all of the following design points are met:

  • A request message or "one shot" message is self-contained in a single transport packet.
  • A stateless transport (e.g., UDP) is used for the above.
  • Replies are always sent to a unicast address. These can be multipacket, since the unicast destination is presumed to be associated with a single "stable" end system and not an anycasted source address. Note that this constrains the use of anycast as source addresses in request messages, since reply messages sent back to that address may reach a device that was not the source that initially triggered it 👀.
  • The server side of the application keeps no hard state across requests.
  • Retries are idempotent. In addition to not assuming server state, they don't encode any assumptions about loss of requests versus loss of replies 🔁.

It's worth noting, though, that even under the above circumstances, ICMP messages against packets with anycast source addresses may be routed to servers other than the ones expected 😬. Path Maximum Transmission Unit Discovery (PMTUD) can also run into complications with anycast addresses, since iterations in the PMTU discovery process may have packets routed to different anycast service instances 🌪️.

3.3. Anycast Addresses as Sources 🌸

When an anycast address is the source address, it doesn't uniquely identify the source host, so replies might be sent to a different host 🙃. As noted earlier, this concept is sometimes referred to (e.g., in [RFC3258]) as a "shared unicast address" 🫶. Anycast addresses are "safe" to use as source addresses for an application if all of the following design points are met:

  • No response message is generated by the receiver with the anycast source used as a destination, unless the application has some private state synchronization that allows the response message to arrive at a different instance 🔐.
  • The source anycast address is reachable via the interface address if unicast reverse path forwarding (RPF) [RFC4778] checking is on, or the service address is explicitly provisioned to bypass RPF checks. In addition to the application defined in [RFC4778], Section 4.4.5 of BCP 126 [RFC4786] gives explicit consideration to RPF checks in anycasting operations 📚.

3.4. Service Discovery 🔍

Applications that can handle an extra round-trip time (RTT) to learn a unicast destination address for multipacket exchanges might safely use anycast destination addresses for service instance discovery 💅. For example, "instance discovery" messages are sent to an anycast destination address, and then a reply is sent from the unique unicast source address of the interface that received the discovery message. Or, a reply is sent from the anycast source address of the interface that received the message, containing the unicast address to be used to invoke the service. Only the latter of these will avoid potential NAT binding and stateful firewall issues 🧱.

[DNS-DISC] discussed several options for the need to configure DNS servers, including a "Well-known Anycast Address" for recursive DNS service configuration in clients. The idea was to ease configuration and let those systems ship with these well-known addresses configured "from the beginning, as, say, factory default" 🏭. The proposal was later dropped, but the analysis was used when publishing [RFC4339] 🧠.

After the final round of revisions to [DNS-DISC] was made, [RFC4339] was published with a very similar focus and overlapping content 👯. The difference was that [RFC4339] focused on analysis, while [DNS-DISC] covered both the analysis and a specific proposal. The proposal details were removed in what became [RFC4339], although Section 3.3 of that RFC still discusses the approach of using a well-known anycast address in this scenario 🙌. During publication, the IESG requested that the following "IESG Note" be contained in the document:

   This document describes three different approaches for the
   configuration of DNS name resolution server information in IPv6
   hosts.

   There is not an IETF consensus on which approach is preferred.
   The analysis in this document was developed by the proponents for
   each approach and does not represent an IETF consensus.

   The 'RA option' and 'Well-known anycast' approaches described in
   this document are not standardized.  Consequently the analysis for
   these approaches might not be completely applicable to any
   specific proposal that might be proposed in the future.

4. Analysis ⚖️

4.1. Regarding Widespread Anycast Use 🌍

Widespread use of anycast for global Internet-wide services or inter-domain services has some scaling challenges 📈. Similar in some ways to multicast, each service generates at least one unique route in the global BGP routing system. As a result, additional anycast instances create additional paths for a given prefix, which scales super-linearly as a function of the denseness of inter-domain interconnection within the routing system. In other words, more paths mean more resources, and more network interconnections mean more paths 🕸️.

This is why the Anycast BOF concluded that "the use of global anycast addresses was not expected to scale and hence was expected to be limited to a small number of key uses" 🚦.

One interesting note, though, is that multiple anycast services can share a route if they are all located in a single announced prefix and all the servers of all the services are always collocated 🏠. If the announced prefix is aggregated differently in different locations, longest-match routing might leave some anycast locations unreachable 😱. So extra precaution must be taken when aggregating prefixes used by anycast services 🧐.

4.2. Transport Implications 🚚

UDP is the "lingua franca" for anycast today 🗣️. Stateful transports could be enhanced to be more anycast friendly, and this was anticipated in Host Anycasting Services [RFC1546], specifically:

   The solution to this problem is to only permit anycast addresses
   as the remote address of a TCP SYN segment (without the ACK bit
   set).  A TCP can then initiate a connection to an anycast address.
   When the SYN-ACK is sent back by the host that received the
   anycast segment, the initiating TCP should replace the anycast
   address of its peer, with the address of the host returning the
   SYN-ACK.  (The initiating TCP can recognize the connection for
   which the SYN-ACK is destined by treating the anycast address as a
   wildcard address, which matches any incoming SYN-ACK segment with
   the correct destination port and address and source port, provided
   the SYN-ACK's full address, including source address, does not
   match another connection and the sequence numbers in the SYN-ACK
   are correct.)  This approach ensures that a TCP, after receiving
   the SYN-ACK is always communicating with only one host.

The reason for these considerations can be shown with an example. One operationally observed shortcoming of using the Transmission Control Protocol (TCP) [RFC0793] with anycast nodes in DNS is that, even during TCP connection establishment, IP control packets from a DNS client may initially be routed to one anycast instance. Subsequent IP packets may then be delivered to a different anycast instance if (for example) a route has changed 😵. In such a case, the TCP connection will likely elicit a connection reset, but it will certainly result in the disruption of the connection 💔.

Multi-address transports (e.g., SCTP) might be more amenable to such extensions than TCP 🧩.

The features needed for address discovery when doing multihoming in the transport layer are similar to those needed to support anycast 🤝.

4.3. Stateful Firewalls, Middleboxes, and Anycast 🔥

Middleboxes (e.g., NATs) and stateful firewalls cause problems when used in conjunction with some ways of using anycast 🚧. In particular, a server-side transition from an anycast source IP address to a unique unicast address may require new or additional session state. That state may not exist in the middlebox, as discussed previously in Section 3.4 🫣.

4.4. Security Considerations 🔒

okay bestie, security is where anycast gets dramatic 😭 Anycast is often deployed to mitigate or at least localize the effects of distributed denial-of-service (DDoS) attacks 🛡️ For example, with the Netgear NTP fiasco [RFC4085], anycast was used in a distributed sinkhole model [RFC3882] to mitigate the effects of embedded globally routed Internet addresses in network elements 🕳️✨

"Internet Denial-of-Service Considerations" [RFC4732] notes that: "A number of the root nameservers have since been replicated using anycast to further improve their resistance to DoS". Iconic, honestly 👑

"Operation of Anycast Services" BCP 126 [RFC4786] cites DoS mitigation, constraining DoS to localized regions, and identifying attack sources using spoofed addresses as some motivations to deploy services using anycast 🎯 Multiple anycast service instances, like the ones used by the root name servers, also add resiliency when network partitioning occurs (for example, because of transoceanic fiber cuts or natural disasters) 🌊🌪️

Now the big warning 🚨 When using anycast, care must be taken not to simply withdraw an anycast route in the presence of a sustained DoS attack 🙅‍♀️ The result would simply move the attack to another service instance, potentially causing a cascaded failure 💀 Anycast adds resiliency when such an attack is instead constrained to a single service instance 💖

It should be noted that there is a significant man-in-the-middle (MITM) exposure in either variant of anycast discovery (see Section 3.4). In many applications, this may necessitate the need for end-to-end security models (for example, using IPsec [RFC6071] or even DNSSEC [RFC4033]) that enable end systems to authenticate one another, or the data itself 🔐

But wait, there's a complication 🥺 When considering the above suggestion of enabling end systems to authenticate each other, things can get messy. If the service nodes of an anycast deployment are administered by separate authorities, any server-side authentication credentials that are used must (necessarily) be shared across the administrative boundaries in the anycast deployment 🤝 This would likely also be the case with Secure Neighbor Discovery, described in [RFC5909] 🧩

Furthermore, as discussed earlier in this document, operational consideration needs to be given to making sure anycast addresses get advertised and/or filtered in a way that produces the intended scope 🎀 For example, only advertise a route to your 6to4 relay to ASes that conform to your own Acceptable Use Policy (AUP) 📜 This seems to be operationally expensive, and it is often vulnerable to errors outside of the local routing domain 😩 That's especially true when anycasted services are deployed with the intent to scope associated announcements within some local or regional boundary 🗺️

As previously discussed, [RFC6382] makes recommendations regarding the use of per-node unique origin ASNs for globally anycasted critical infrastructure services 🏗️ The goal is to provide routing system discriminators for a given anycasted prefix. Network management and monitoring techniques, or other operational mechanisms, may then use this new discriminator in whatever manner fits their operating environment, whether for detection or for policy associated with a given anycasted node 🔍

The per-node unique origin ASNs also have the additional benefit of overcoming complications that might arise with the potential deployment of the Resource Public Key Infrastructure (RPKI) [RFC6480] 🔑 Without per-node unique origin ASNs, the cryptographic certificates needed to attest to the Route Origin Authorizations (ROAs) of a multi-administrative anycast deployment would need to be shared 😬 But if each service instance has a separate ASN, those ASNs can be managed separately in the RPKI 🥳

Now for the tea ☕ Unlike multicast (but like unicast), anycast allows traffic stealing 🫣 With multicast, joining a multicast group doesn't stop anyone else who was already receiving the traffic from continuing to receive it 🙌 With anycast, adding an anycasted node to the routing system can stop a previous recipient from continuing to receive traffic, because the traffic may now be delivered to the new node instead 😳 So, if an unauthorized anycast node can inject a route into the network, or be resolved using ARP/Neighbor Discovery on a link with an authorized anycast node, traffic can be diverted, and that can trigger DoS or other attacks 🚩 Section 6.3 of BCP 126 [RFC4786] gives expanded discussion on "Service Hijacking" and "traffic stealing" 📖 Meanwhile, [FanInfocom13] discusses measured instances of anycast nodes and "benign masquerading or hostile hijacking of anycast services" by unauthorized nodes 👀

Unlike unicast (but like multicast), the desire is to allow applications to cause route injection 🧃 In multicast, one often allows arbitrary applications on hosts to join multicast groups, which results in multicast routing state. Trying to apply that same model to anycast would present new security concerns, which is why [MAGMA] only got so far 🫠 The security concerns include:

  1. Allowing route injection can cause DOS to a legitimate address owner 💔
  2. Allowing route injection consumes routing resources and can hence cause DOS to the routing system, and impact legitimate communications as a result 📉

These are two of the core issues that were part of the discussion during [RFC1884], the [ANYCASTBOF], and the MAGMA [MAGMA] chartering 🗣️ Additional security considerations are scattered throughout the list of references provided herein 📚✨

4.5. Deployment Considerations 🚀

BCP 126 [RFC4786] provides some very solid guidance related to operations of anycasted services and, in particular, the operations of DNS 🧠💖

This document covers issues associated with the architectural implications of anycast 🏛️ It does not address, in any depth, the fact that there are deployed services with TCP transport using anycast today 🙈 Evidence exists to suggest that such practice is not "safe" in the traditional and architectural sense (as described in Section 4.2) 😬 These sorts of issues are relative, though, and we recognize that sometimes the unpredictability in the routing system beyond the local administrative domain can be manageable 🤷‍♀️ Basically, despite the inherent architectural problems of using anycast with stateful transport and connection-oriented protocols, there is expanding deployment (for example, in content distribution networks) 📦 Situations exist where it may make sense too, such as service discovery, short-lived transactions, or cases where dynamically directing traffic to topologically optimal service instances is required 🎯 In general, operators should consider the content and references provided herein, and evaluate the benefits and implications of anycast in their specific environments and applications 🧐

Also, as noted in Section 2.3, the issue of whether to withdraw anycast routes when there is a service failure is only briefly broached in [RFC3258] 🫥 The advice given is that routes should not be withdrawn, in order to reduce operational complexity. However, the issue of route advertisements and service outages deserves greater attention 🔎

There's an inherent trade-off between the operational complexity of matching service outages with anycast route withdrawals, and allowing anycast routes to persist for services that are no longer available ⚖️ [RFC3258] maintains that DNS' inherent failure recovery mechanism is sufficient to overcome failed nodes, but even this advice admits that these decisions are both application-specific and subject to the operational needs of each deployment 💅 For example, the routing system plays a bigger role in DNS when services are anycast. Therefore, operational consideration must be given to the fact that relying on anycast for DNS deployment optimizations means there are operational trade-offs related to keeping route advertisements (and withdrawals) symmetric with service availability 🪞 For instance, to make sure the DNS resolvers in a failed anycast instance's catchment [RFC4786] are able to fail over and reach a non-failed catchment, a route withdrawal is almost certainly required 🏃‍♀️ On the other hand, instability of a DNS process that triggers frequent route advertisement and withdrawal might result in suppression of legitimate paths to available nodes, for example because of route flap damping [RFC2439] 🌀

Rather than prescribing advice that tries to fit all situations, it should simply be recognized that when using anycast with network services that provide redundancy or resilience at other layers of the protocol stack, operators should carefully consider the optimal layer(s) at which to provide said functions 🍰

As noted in Section 2.3, use of anycast within a subnet does not necessarily suffer from the potential issues with route withdrawals 🏠 So, using anycast to reach servers that reside in the same subnet can be made more reliable than using anycast to reach topologically disparate server instances 💪 Within a subnet, though, care must be taken as stated in Section 5.4 of [RFC4862]: "Duplicate Address Detection MUST NOT be performed on anycast addresses" 🚫 Hence, the servers must be configured appropriately 🛠️

5. Conclusions 🎀

In summary, operators and application vendors alike should consider the benefits and implications of anycast in their specific environments and applications 🌸 They should also give forward consideration to how new network protocols and application functions may take advantage of anycast, or how they may be negatively impacted if anycasting is employed 🔮

6. Acknowledgements 🫶

Many thanks to Kurtis Lindqvist for his early review and feedback on this document 💜 Thanks to Brian Carpenter, Alfred Hoenes, and Joe Abley for their usual careful review and feedback, as well as Mark Smith, Lixia Zhang, Stephane Bortzmeyer, Masataka Ohta, and S. Moonesamy for their detailed reviews 🦋 Helpful feedback was also received from others, including Edward Lewis, Jean-Michel Combes, Wolfgang Nagele, Mark Townsley, and Abdussalam Baryun ✨

7. Informative References ✨

okay bestie, this is the receipts section 🧾💅 every outside doc the RFC leans on is listed here, so you can go read the source material whenever you want. I've kept every tag, title, date and URI exactly as written, with the wrapped URLs joined back up so they actually work when you copy them 🔗

  • [ANYCASTBOF] Deering, S., "IAB Anycast BOF Announcement", October 1999, <http://www.ietf.org/mail-archive/web/ietf/current/msg11182.html>.
  • [DNS-DISC] Durand, A., Hagino, J., and D. Thaler, "Well known site local unicast addresses for DNS resolver", Work in Progress, September 2002.
  • [FanInfocom13] Fan, X., Heidemann, J., and R. Govindan, "Evaluating Anycast in the Domain Name System", Proceedings of the IEEE Infocom 2013, April 2013.
  • [IMR9401] RFC Editor, "INTERNET MONTHLY REPORT", January 1994, <ftp://ftp.rfc-editor.org/in-notes/museum/imr/imr9401.txt>.
  • [MAGMA] MAGMA (concluded), "Multicast and Anycast Group Membership (MAGMA)", April 2006, <http://www.ietf.org/wg/concluded/magma>.
  • [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981.
  • [RFC1035] Mockapetris, P., "Domain names - implementation and specification", STD 13, RFC 1035, November 1987.
  • [RFC1546] Partridge, C., Mendez, T., and W. Milliken, "Host Anycasting Service", RFC 1546, November 1993.
  • [RFC1884] Hinden, R. and S. Deering, "IP Version 6 Addressing Architecture", RFC 1884, December 1995.
  • [RFC2030] Mills, D., "Simple Network Time Protocol (SNTP) Version 4 for IPv4, IPv6 and OSI", RFC 2030, October 1996.
  • [RFC2101] Carpenter, B., Crowcroft, J., and Y. Rekhter, "IPv4 Address Behaviour Today", RFC 2101, February 1997.
  • [RFC2373] Hinden, R. and S. Deering, "IP Version 6 Addressing Architecture", RFC 2373, July 1998.
  • [RFC2439] Villamizar, C., Chandra, R., and R. Govindan, "BGP Route Flap Damping", RFC 2439, November 1998.
  • [RFC2526] Johnson, D. and S. Deering, "Reserved IPv6 Subnet Anycast Addresses", RFC 2526, March 1999.
  • [RFC2893] Gilligan, R. and E. Nordmark, "Transition Mechanisms for IPv6 Hosts and Routers", RFC 2893, August 2000.
  • [RFC2902] Deering, S., Hares, S., Perkins, C., and R. Perlman, "Overview of the 1998 IAB Routing Workshop", RFC 2902, August 2000.
  • [RFC2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and Multicast Next-Hop Selection", RFC 2991, November 2000.
  • [RFC3056] Carpenter, B. and K. Moore, "Connection of IPv6 Domains via IPv4 Clouds", RFC 3056, February 2001.
  • [RFC3068] Huitema, C., "An Anycast Prefix for 6to4 Relay Routers", RFC 3068, June 2001.
  • [RFC3258] Hardie, T., "Distributing Authoritative Name Servers via Shared Unicast Addresses", RFC 3258, April 2002.
  • [RFC3513] Hinden, R. and S. Deering, "Internet Protocol Version 6 (IPv6) Addressing Architecture", RFC 3513, April 2003.
  • [RFC3775] Johnson, D., Perkins, C., and J. Arkko, "Mobility Support in IPv6", RFC 3775, June 2004.
  • [RFC3882] Turk, D., "Configuring BGP to Block Denial-of-Service Attacks", RFC 3882, September 2004.
  • [RFC3964] Savola, P. and C. Patel, "Security Considerations for 6to4", RFC 3964, December 2004.
  • [RFC4033] Arends, R., Austein, R., Larson, M., Massey, D., and S. Rose, "DNS Security Introduction and Requirements", RFC 4033, March 2005.
  • [RFC4085] Plonka, D., "Embedding Globally-Routable Internet Addresses Considered Harmful", BCP 105, RFC 4085, June 2005.
  • [RFC4213] Nordmark, E. and R. Gilligan, "Basic Transition Mechanisms for IPv6 Hosts and Routers", RFC 4213, October 2005.
  • [RFC4291] Hinden, R. and S. Deering, "IP Version 6 Addressing Architecture", RFC 4291, February 2006.
  • [RFC4330] Mills, D., "Simple Network Time Protocol (SNTP) Version 4 for IPv4, IPv6 and OSI", RFC 4330, January 2006.
  • [RFC4339] Jeong, J., "IPv6 Host Configuration of DNS Server Information Approaches", RFC 4339, February 2006.
  • [RFC4610] Farinacci, D. and Y. Cai, "Anycast-RP Using Protocol Independent Multicast (PIM)", RFC 4610, August 2006.
  • [RFC4732] Handley, M., Rescorla, E., and IAB, "Internet Denial-of-Service Considerations", RFC 4732, December 2006.
  • [RFC4778] Kaeo, M., "Operational Security Current Practices in Internet Service Provider Environments", RFC 4778, January 2007.
  • [RFC4786] Abley, J. and K. Lindqvist, "Operation of Anycast Services", BCP 126, RFC 4786, December 2006.
  • [RFC4861] Narten, T., Nordmark, E., Simpson, W., and H. Soliman, "Neighbor Discovery for IP version 6 (IPv6)", RFC 4861, September 2007.
  • [RFC4862] Thomson, S., Narten, T., and T. Jinmei, "IPv6 Stateless Address Autoconfiguration", RFC 4862, September 2007.
  • [RFC4892] Woolf, S. and D. Conrad, "Requirements for a Mechanism Identifying a Name Server Instance", RFC 4892, June 2007.
  • [RFC4924] Aboba, B. and E. Davies, "Reflections on Internet Transparency", RFC 4924, July 2007.
  • [RFC4942] Davies, E., Krishnan, S., and P. Savola, "IPv6 Transition/Co-existence Security Considerations", RFC 4942, September 2007.
  • [RFC5001] Austein, R., "DNS Name Server Identifier (NSID) Option", RFC 5001, August 2007.
  • [RFC5905] Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network Time Protocol Version 4: Protocol and Algorithms Specification", RFC 5905, June 2010.
  • [RFC5909] Combes, J-M., Krishnan, S., and G. Daley, "Securing Neighbor Discovery Proxy: Problem Statement", RFC 5909, July 2010.
  • [RFC6071] Frankel, S. and S. Krishnan, "IP Security (IPsec) and Internet Key Exchange (IKE) Document Roadmap", RFC 6071, February 2011.
  • [RFC6275] Perkins, C., Johnson, D., and J. Arkko, "Mobility Support in IPv6", RFC 6275, July 2011.
  • [RFC6382] McPherson, D., Donnelly, R., and F. Scalzo, "Unique Origin Autonomous System Numbers (ASNs) per Node for Globally Anycasted Services", BCP 169, RFC 6382, October 2011.
  • [RFC6480] Lepinski, M. and S. Kent, "An Infrastructure to Support Secure Internet Routing", RFC 6480, February 2012.
  • [RSSAC29] "RSSAC 29 Meeting Minutes", December 2007, <http://www.icann.org/en/groups/rssac/meetings/rssac-29-en.pdf>.

Appendix A. IAB Members at the Time of Approval ✨

the squad that approved this one 🫶 these are the IAB members at the time:

  • Bernard Aboba
  • Jari Arkko
  • Marc Blanchet
  • Ross Callon
  • Alissa Cooper
  • Joel Halpern
  • Russ Housley
  • Eliot Lear
  • Xing Li
  • Erik Nordmark
  • Andrew Sullivan
  • Dave Thaler
  • Hannes Tschofenig

Authors' Addresses 💖

and here are the people who wrote it, with their contact details 💌

Danny McPherson
Verisign, Inc.
12061 Bluemont Way
Reston, VA
USA

EMail: dmcpherson@verisign.com
Dave Oran
Cisco Systems
USA

EMail: oran@cisco.com
Dave Thaler
Microsoft Corporation
One Microsoft Way
Redmond, WA
USA

EMail: dthaler@microsoft.com
Eric Osterweil
Verisign, Inc.
12061 Bluemont Way
Reston, VA
USA

EMail: eosterweil@verisign.com

and that's the whole RFC, bestie! 🌈👑 thank u for being the most patient study buddy ever 🦋

📦 where this page comes from