RFC 4786

Operation of Anycast Services

📅 published December 2006, from ietf

RFC 4786 📜✨ (BCP 126) — Operation of Anycast Services, part 1 of 5 💖

  • Title: Operation of Anycast Services 🦋
  • Authors: J. Abley (Afilias Canada) and K. Lindqvist (Netnod Internet Exchange) 🧑‍💻👩‍💻
  • Category: Best Current Practice (BCP: 126) 👑
  • Date: December 2006 📅
  • Obsoletes / Updates: the title block doesn't list anything it obsoletes or updates 🙈

Status of This Memo 🗂️

okay bestie, this one is a Best Current Practice for the Internet Community 🌸 It's basically a "here's how we do it well right now" guide, and it's open for discussion and suggestions for improvements 💬 Anyone can share it freely, no cap 🫶

The IETF Trust owns the copyright on this one, and the year is 2006 🌈

Abstract 🌟

So the internet keeps getting bigger, and more systems and services inside companies are becoming super common too 🌍 That means a lot of services now need to be really available, like always-on available 😭 Those demands make the infrastructure underneath them need to be more reliable, which is a lot of pressure 💅

People have tried all kinds of tricks to keep services available on the internet 🔥 This document is a sort of commentary-and-recommendations piece about spreading services out using anycast 🎀

Table of Contents 📚

   1. Introduction ....................................................3
   2. Terminology .....................................................4
   3. Anycast Service Distribution ....................................5
      3.1. General Description ........................................5
      3.2. Goals ......................................................5
   4. Design ..........................................................6
      4.1. Protocol Suitability .......................................6
      4.2. Node Placement .............................................7
      4.3. Routing Systems ............................................8
           4.3.1. Anycast within an IGP ...............................8
           4.3.2. Anycast within the Global Internet ..................9
      4.4. Routing Considerations .....................................9
           4.4.1. Signalling Service Availability .....................9
           4.4.2. Covering Prefix ....................................10
           4.4.3. Equal-Cost Paths ...................................10
           4.4.4. Route Dampening ....................................12
           4.4.5. Reverse Path Forwarding Checks .....................13
           4.4.6. Propagation Scope ..................................13
           4.4.7. Other Peoples' Networks ............................14
           4.4.8. Aggregation Risks ..................................14
      4.5. Addressing Considerations .................................15
      4.6. Data Synchronisation ......................................15
      4.7. Node Autonomy .............................................16
      4.8. Multi-Service Nodes .......................................17
           4.8.1. Multiple Covering Prefixes .........................17
           4.8.2. Pessimistic Withdrawal .............................17
           4.8.3. Intra-Node Interior Connectivity ...................18
      4.9. Node Identification by Clients ............................18
   5. Service Management .............................................19
      5.1. Monitoring ................................................19
   6. Security Considerations ........................................19
      6.1. Denial-of-Service Attack Mitigation .......................19
      6.2. Service Compromise ........................................20
      6.3. Service Hijacking .........................................20
   7. Acknowledgements ...............................................21
   8. References .....................................................21
      8.1. Normative References ......................................21
      8.2. Informative References ....................................21

1. Introduction 🎀

This doc is written for network operators who are thinking about deploying or running a distributed service with anycast 💖 It explains the best current practice for doing that, but it does not say whether any particular service should or should not use anycast 🤷‍♀️ That part is on you, bestie 💅

So here's the basic idea 🦋 To distribute a service with anycast, you first tie the service to a stable set of IP addresses. Then you advertise reachability to those addresses in a routing system from several independent service nodes 🌸 There are a few different techniques for this, and they're covered in [RFC1546], [ISC-TN-2003-1], and [ISC-TN-2004-1] ✨

The techniques and considerations in this doc apply to services reachable over both IPv4 and IPv6 🌈

Anycast has been getting super popular for adding redundancy to DNS servers 🔥 It works alongside the redundancy the DNS architecture already has built in 👑 Several root DNS server operators have spread their servers all over the internet, and both resolver and authority servers are often distributed inside service provider networks 🛰️ Commercial DNS authority server operators have been using anycast distribution for several years now 💅 Anycast isn't limited to DNS, though. Using it does put some extra limits on the service being distributed, like transaction longevity, transaction state held on servers, and data synchronisation capabilities 🙈

Okay, the scary part: anycast is conceptually simple, but actually running it comes with some pitfalls 😭 For example, monitoring whether the service is up gets harder. The availability you see depends on where the client is in the network, and which clients end up at which anycast nodes is neither static nor reliably deterministic 💀

This document covers using anycast for both local scope distribution of services, using an Interior Gateway Protocol (IGP), and global distribution, using the Border Gateway Protocol (BGP) [RFC4271] 🌍 Monitoring and data synchronisation issues are mostly shared between the two, but the deployment issues are pretty different 🫶

2. Terminology 📖

Service Address: an IP address tied to a particular service (for example, the destination address DNS resolvers use to reach a particular authority server) 🏷️

Anycast: the practice of making a particular Service Address available in multiple, discrete, autonomous locations, so that datagrams sent to it get routed to one of the several available locations 🦋

Anycast Node: an internally-connected collection of hosts and routers that together provide service for an anycast Service Address 🧚 An Anycast Node could be as simple as one host taking part in a routing system with adjacent routers, or it could be a bunch of hosts connected in a fancier way. Either way, to the routing system carrying the anycast, each Anycast Node presents a unique path to the Service Address ✨ The whole anycast system for a service is made up of two or more separate Anycast Nodes 👯

Catchment: in physical geography, this is an area drained by a river, also called a drainage basin 🌊 By analogy, in this document it means the topological region of a network where packets aimed at an Anycast Address get routed to one particular node 💜

Local-Scope Anycast: reachability information for the anycast Service Address is propagated through a routing system in such a way that a particular anycast node is only visible to a subset of the whole routing system 🔍

Local Node: an Anycast Node providing service using a Local-Scope Anycast Address 🏡

Global-Scope Anycast: reachability information for the anycast Service Address is propagated through a routing system in such a way that a particular anycast node is potentially visible to the whole routing system 🌐

Global Node: an Anycast Node providing service using a Global-Scope Anycast Address 🌏

3. Anycast Service Distribution 🚀

3.1. General Description 💫

Anycast is the name for making a Service Address available to a routing system at Anycast Nodes in two or more discrete locations 🎀 The service each node provides is usually consistent no matter which node the routing system picks for a given request. Some services might benefit from deliberate differences in how individual nodes behave, so they can do locality-specific stuff, but that's an exception (see Section 4.6) 🧁

For services distributed with anycast, there's no inherent need for referrals to other servers, or for name-based distribution ("round-robin DNS") 🙃 Those techniques could be combined with anycast if an application really needed them 💅 The routing system decides which node handles each request, based on how the routing system is designed and where in the network the request starts 🗺️

The Anycast Node that ends up handling a particular query can be influenced by the traffic engineering features of the routing protocols that make up the routing system 🎛️ How much control the node's operator has depends on how big the routing system is that the Service Address is anycast through 📏

Load-balancing between Anycast Nodes is typically hard to pull off, since load between nodes is usually unbalanced in terms of requests and traffic 😩 Spreading load between nodes for reliability can often be done, though, and so can coarse-grained load spreading to help popular services scale 📈

The routing system a service is anycast through can be anything from a small Interior Gateway Protocol (IGP) connecting a handful of components, all the way up to the Border Gateway Protocol (BGP) [RFC4271] connecting the global internet 🌍 Which one you use depends on what kind of service distribution you need 🦋

⋯ to be continued in part 2 💖

3.2. Goals ✨

okay so a service can be anycast for a bunch of different reasons 🎀 Here are the common goals people go for:

  1. Coarse ("unbalanced") distribution of load across nodes 🌸 This lets the infrastructure grow to handle more queries and survive sudden spikes in traffic 📈
  2. Mitigating non-distributed denial-of-service attacks by keeping the damage contained to single Anycast Nodes 🛡️
  3. Constraining distributed denial-of-service attacks or flash crowds so they stay local to the regions around Anycast Nodes 🌍 Anycast also means traffic can get handled closer to where it came from, maybe over high-performance peering links instead of oversubscribed, paid transit circuits 💸
  4. Getting extra info about where traffic is coming from when attack (or query) traffic uses spoofed source addresses 🕵️‍♀️ This works because of a property of anycast: which Anycast Node picks up a given query can be tied to where the request topologically came from 🧭
  5. Improving query response time by cutting the network distance between client and server with a local Anycast Node ⚡ How much faster things get depends on how the routing system picks nodes for clients. Being topologically close inside the routing system doesn't generally line up with round-trip performance across the network, so in some cases response times see no reduction, and they may even go up 😭
  6. Turning a list of servers into one distributed address 💜 For example, a huge number of authoritative nameservers for a zone can be deployed behind a small set of anycast Service Addresses. This makes zone data easier to reach in the DNS without making the referral response from the parent zone's nameserver any bigger 🧚

4. Design 💅

4.1. Protocol Suitability 🦋

When a service is anycast between two or more nodes, the routing system is the one making the node selection decision on the client's behalf 🤖 A single client-server interaction is usually required to happen between a client and the same server node for the whole transaction. So the routing system's node selection ought to stay stable for a lot longer than the transaction takes, if the service is going to be reliable 🫶

Some services have super short transactions, and they might even be done with one request packet and one reply packet (think DNS transactions over UDP transport) ⚡ Other services have way longer-lived transactions, like bulk file downloads and audio-visual media streaming 🎬

Services can be anycast inside very predictable routing systems that stay stable for long stretches, like anycast within a well-managed and topologically-simple IGP, where node selection only changes when a node fails 🌈 Other deployments are way less predictable (see Section 4.4.7) 😬

The stability of the routing system and the transaction time of the service should be carefully compared when you're deciding whether a service is suitable for anycast distribution 🔍 In some cases, for new protocols, it might be practical to split a large transaction into an initialisation phase handled by anycast servers, and a sustained phase handled by non-anycast servers, maybe picked during that initialisation phase 🧩

This document deliberately avoids prescribing which protocols or services are suitable for anycast distribution. Trying to do that would be presumptuous 🙈

Operators should be aware that, especially for long running flows, anycast can have potential failure modes that are more complex than a simple 'destination unreachable' failure with unicast 💀

4.2. Node Placement 📍

Where Anycast Nodes should be placed depends a lot on what the service distribution is trying to achieve 🎯 For example:

  • A DNS recursive resolver service might be distributed within an ISP's network, with one Anycast Node per site 🏠
  • A root DNS server service might be distributed throughout the Internet. Anycast Nodes could be put in regions with poor external connectivity, so DNS keeps working well enough within the region during external network failures 🌧️
  • An FTP mirror service might include local nodes at exchange points, so ISPs connected to that exchange point can download bulk data more cheaply than if they had to use expensive transit circuits 💰

In general, node placement decisions should take into account the likely traffic requirements, the potential for flash crowds or denial-of-service traffic, the stability of the local routing system, and the failure modes around node failure or local routing system failure 🧠

4.3. Routing Systems 🛤️

4.3.1. Anycast within an IGP 🏡

There are several common reasons to distribute a Service Address within the scope of an IGP:

  1. to improve service response times by hosting a service close to other users of the network;
  2. to improve service reliability by providing automatic fail-over to backup nodes; and
  3. to keep service traffic local so it doesn't congest wide-area links 🚦

In each case, network engineers can make the decisions about where and how services are provisioned without needing operational complexities like regional variances in the configuration of client computers, or deliberate DNS incoherence (where DNS queries return different answers depending on where they came from) 🙅‍♀️

When a service is anycast within an IGP, the same organisation that provides the service typically controls the routing system 👑 So the relationship between service transaction characteristics and network stability is likely to be well-understood. This technique is therefore usable for a larger number of applications than Internet-wide anycast service distribution (see Section 4.1) ✨

An IGP generally has no inherent restriction on the length of prefix that can be introduced to it. So there's no need to build a covering prefix for particular Service Addresses. Host routes corresponding to the Service Address can be introduced to the routing system instead 🧚‍♀️ See Section 4.4.2 for more discussion of the requirement for a covering prefix.

IGPs often have little or no aggregation of routes, partly because of the algorithmic complexity of supporting aggregation. In many networks' IGPs there's little motivation for aggregation, since the amount of routing information carried in the IGP is small enough that scaling concerns in routers don't come up 🤷‍♀️ For a discussion of aggregation risks in other routing systems, see Section 4.4.8.

By reducing the scope of the IGP to just the hosts providing the service (together with one or more gateway routers), this technique can be applied to building server clusters 🖥️ This application is discussed in some detail in [ISC-TN-2004-1].

4.3.2. Anycast within the Global Internet 🌐

Service Addresses may be anycast within the global Internet routing system to spread services across the entire network 🌏 The main differences between this and the IGP-scope distribution discussed in Section 4.3.1 are:

  1. the routing system is, in general, controlled by other people; and
  2. the routing protocol concerned (BGP), along with commonly-accepted practices in its deployment, imposes some additional constraints (see Section 4.4) 🚧

4.4. Routing Considerations 🧭

4.4.1. Signalling Service Availability 📣

When a routing system gets reachability information for a Service Address from an individual node, packets addressed to that Service Address will start arriving at the node 📦 Since it's essential for the node to be ready to accept requests before they start arriving, it's desirable to couple the routing information to the availability of the service at a particular node 💖

When a routing advertisement from a node corresponds to a single Service Address, this coupling might look like this: the service becoming available triggers the route advertisement, and the service becoming unavailable triggers a route withdrawal 🎀 This can be done with routing protocol implementations on the same server. These implementations provide the service being distributed, and they're configured to advertise and withdraw the route in step with the availability (and health) of the software on the host that processes service requests 🫧 An example of this arrangement for a DNS service is included in [ISC-TN-2004-1].

When a routing advertisement from a node corresponds to two or more Service Addresses, it may not be appropriate to trigger a route withdrawal just because one service is unavailable 🤔 Another approach, when the service is down at one Anycast Node, is to route requests to a different Anycast Node where the service is working normally 🔁 This approach is discussed in Section 4.8.

Rapid advertisement/withdrawal oscillations can cause operational problems 😵‍💫 So nodes should be configured to avoid rapid oscillations (for example, by implementing a minimum delay after a withdrawal before the service can be re-advertised) ⏳ See Section 4.4.4 for a discussion of route oscillations in BGP.

4.4.2. Covering Prefix 🧁

In some routing systems (like the BGP-based routing system of the global Internet), you generally can't propagate a host route and be confident it'll reach the whole network 🥺 This comes down to operational policy, not a protocol restriction.

So in those cases you need to propagate a route that covers the Service Address, with a prefix short enough that it won't be thrown out by commonly-deployed import policies 📏 For IPv4 Service Addresses this is often a 24-bit prefix. There are other well-documented examples of IPv4 import policies that filter on Regional Internet Registry (RIR) allocation boundaries, so some experimentation may be prudent 🧪 Corresponding import policies for IPv6 prefixes also exist. See Section 4.5 for more discussion of IPv6 Service Addresses and corresponding anycast routes 🌸

Propagating a single route per service has some associated scaling issues, which are discussed in Section 4.4.8 📈

When multiple Service Addresses are covered by the same covering route, there's no longer a tight coupling between advertising that route and the individual services behind the covered host routes 🧵 The resulting impact on signalling the availability of individual services is discussed in Section 4.4.1 and Section 4.8.

4.4.3. Equal-Cost Paths ✨

okay bestie, so some routing systems can send traffic to the same destination over equal-cost paths 🛣️ and that's where things get spicy. If those equal-cost paths lead to different Anycast Nodes, then different request packets from the same transaction could end up at more than one node 😭. Services over TCP [RFC0793] are especially exposed to this, because the TCP setup handshake means a transaction is made up of multiple request packets from the start 🫶.

For services spread across the global Internet using BGP, equal-cost paths are normally not a consideration 💅. BGP's exit selection algorithm usually picks a single, consistent exit for a single destination, even when several candidate paths exist. Implementations of BGP do exist that support multi-path exit selection, though 🦋.

Equal-cost paths are commonly supported in IGPs 🌸. Multi-node selection for a single transaction can be avoided in most cases by being careful with IGP link metrics, or by using equal-cost multi-path (ECMP) selection algorithms, which cause a single node to be selected for a single multi-packet transaction 🎀. For an example of hash-based ECMP selection in anycast service distribution, check out [ISC-TN-2004-1] 🤓.

Other ECMP selection algorithms are also commonly available, including ones where packets from the same flow are not guaranteed to go to the same destination 🙈. ECMP algorithms that pick a route per packet instead of per flow are commonly called "Per Packet Load Balancing" (PPLB) ✨.

With respect to anycast service distribution, some uses of PPLB may cause different packets from a single multi-packet transaction sent by a client to land on different Anycast Nodes, which effectively makes the anycast service unavailable 💀. Whether this affects a specific anycast service depends on how and where Anycast Nodes are deployed in the routing system, and on where the PPLB is happening 👑:

  1. PPLB across multiple, parallel links between the same pair of routers should cause no node selection problems 💖;
  2. PPLB across diverse paths within a single autonomous system (AS), where the paths converge to a single exit as they leave the AS, should cause no node selection problems 🌈;
  3. PPLB across links to different neighbour ASes, where the neighbour ASes have selected different nodes for a particular anycast destination will, in general, cause request packets to be distributed across multiple Anycast Nodes. This will have the effect that the anycast service is unavailable to clients downstream of the router performing PPLB 😭.

The uses of PPLB that have the potential to interact badly with anycast service distribution can also cause persistent packet reordering 🌀. A network path that persistently reorders segments will degrade the performance of traffic carried by TCP [Allman2000]. According to several documented measurements, TCP accounts for the bulk of traffic carried on the Internet ([McCreary2000], [Fomenkov2004]) 📊. So, in many cases, it's reasonable to consider networks making that kind of PPLB use to be pathological 🙅‍♀️.

4.4.4. Route Dampening 🌊

Frequent advertisements and withdrawals of individual prefixes in BGP are known as flaps 🦋. Rapid flapping can lead to CPU exhaustion on routers quite far from the source of the instability, which is why rapid route oscillations are often "dampened", as described in [RFC2439] 😩.

A dampened path gets suppressed by routers for an interval that grows with how often the oscillation is observed, and a suppressed path will not propagate 🧊. So a single router can stop a flapping prefix from spreading across an autonomous system, which protects the other routers in the network from the instability 🛡️.

Some implementations of flap dampening penalise oscillating advertisements based on the observed AS_PATH, rather than on Network Layer Reachability Information (NLRI; see [RFC4271]) 🤔. This means network instability that causes route flapping from a single Anycast Node will not generally cause advertisements from other nodes (which have different AS_PATH attributes) to be dampened by those implementations 🙌.

To limit the chance of such implementations penalising advertisements from different Anycast Nodes because of oscillations from just one node, care should be taken to make the AS_PATH attributes on routes from different nodes as diverse as possible 🌈. For example, Anycast Nodes should use the same origin AS for their advertisements, but might have different upstream ASes 💅.

Where different implementations of flap dampening are common, one node's instability may cause stable nodes to become unavailable 😱. To help with that, the following measures may be useful:

  1. Carefully deploying Local Nodes alongside especially stable Global Nodes (with high inter-AS path splay, redundant hardware, power, etc.) may help keep oscillation problems inside the Local Nodes' limited regions of influence 🏡;
  2. Aggressive flap-dampening of the service prefix close to the origin (e.g., within an Anycast Node, or in adjacent ASes of each Anycast Node) may also help reduce the chance that remote ASes see oscillations at all 🔥.

4.4.5. Reverse Path Forwarding Checks 🔍

Reverse Path Forwarding (RPF) checks, first described in [RFC2267], are commonly deployed as part of ingress interface packet filters on routers across the Internet, so that packets with spoofed source addresses get denied (see also RFC 2827 [RFC2827]) 🕵️‍♀️. Deployed RPF implementations come with several modes of operation (e.g., "loose" and "strict") 🎛️.

Some RPF modes can deny non-spoofed packets when they come from multi-homed sites, since the selected paths might legitimately not match the ingress interface of those non-spoofed packets 😬. This issue is discussed in [RFC3704] 📖.

A collection of Anycast Nodes spread across the Internet looks pretty much like a distributed, multi-homed site to the routing system 🌍. So this risk also exists for Anycast Nodes, even when individual nodes aren't multi-homed 🫠. Care should be taken to make sure each Anycast Node is treated as a multi-homed network, and that the matching recommendations in [RFC3704] about RPF checks are followed 🫶.

4.4.6. Propagation Scope 📣

In the context of anycast service distribution across the global Internet, Global Nodes are the ones that can provide service to clients anywhere in the network 🌎. Reachability information for the service is propagated globally, without restriction, by advertising the routes covering the Service Addresses for global transit to one or more providers 🚀.

More than one Global Node can exist for a single service (and honestly this is often the case, for redundancy and load-sharing) 💜.

On the other hand, sometimes it's desirable to deploy an Anycast Node that only serves a local catchment of autonomous systems, and that is deliberately not available to the entire Internet 🏘️. Such nodes are called Local Nodes in this document 🧁. One example of when a Local Node might be a good fit is a node designed to serve a region with rich internal connectivity but unreliable, congested, or expensive access to the rest of the Internet 🌸.

Local Nodes advertise covering routes for Service Addresses in a way that restricts their propagation 🔒. This might be done using well-known community string attributes such as NO_EXPORT [RFC1997] or NOPEER [RFC3765], or by arranging with peers to apply a conventional "peering" import policy instead of a "transit" import policy, or some suitable combination of these measures 🎀.

Advertising reachability to Service Addresses from Local Nodes should ideally be done using a routing policy that requires explicit attributes for propagation, rather than relying on implicit (default) policy 💅. Accidentally letting a route leak beyond its intended horizon can cause capacity problems for Local Nodes, which might degrade service performance network-wide 😵.

4.4.7. Other Peoples' Networks 🌐

When anycast services are deployed across networks run by others, their reachability depends on routing policies and topology changes (planned and unplanned) 🎢. These are unpredictable and sometimes hard to identify 🫣. Since the routing system may include networks operated by multiple, unrelated organisations, unforeseen interactions can also happen when unrelated changes combine 🤯.

The stability and predictability of such a routing system should be taken into consideration when assessing whether anycast is suitable as a distribution strategy for particular services and protocols (see also Section 4.1) 📝.

To help manage this, routing policies used by Anycast Nodes across such routing systems should be conservative 🧐. Individual nodes' internal and external/connecting infrastructure should be scaled to support loads far in excess of the average 💪. And the service should be monitored proactively from many points, so you avoid unpleasant surprises (see Section 5.1) 👀.

4.4.8. Aggregation Risks 📉

Propagating a single route for each anycast service doesn't scale well in routing systems where carrying routing information is a concern, and where there are potentially many services to distribute 😩. For example, an autonomous system that provides services to the Internet with N Service Addresses covered by a single exported route would need to advertise (N+1) routes, if each of those services were distributed using anycast 🧮.

The common practice of applying minimum prefix-length filters in import policies on the Internet (see Section 4.4.2) means that for a route covering a Service Address to be usefully propagated, the prefix length has to be substantially shorter than what would be needed to advertise just the host route 🙃. Widespread advertisement of short prefixes for individual services therefore also hurts address conservation 🏠.

Both of these issues can be eased to some extent by using a single covering prefix to hold multiple Service Addresses, as described in Section 4.8 🧩. This does mean decoupling the route advertisement from individual service availability (see Section 4.4.1), though, which comes with risks to the stability of the service as a whole (see Section 4.7) ⚠️.

In general, the scaling problems described here stop anycast from being a useful, general approach for service distribution on the global Internet 🚫. It's still a useful technique for distributing a limited number of Internet-critical services, and in smaller networks where these aggregation concerns don't apply 🌟.

4.5. Addressing Considerations ✨

okay so first things first, bestie: a Service Address should be unique within the routing system that connects all Anycast Nodes to all possible clients of the service 🫶 It also has to be picked so that the matching routes are actually allowed to spread around inside that routing system. no routes, no service, no cap 😭

For an IPv4-numbered service running across the Internet, you might grab an address from a block where the minimum RIR allocation size is 24 bits 🌸 Then reachability can come from originating the covering 24-bit prefix. it's giving tidy, honestly 💅

For an IPv4-numbered service running inside a private network, you might pick a locally-unused [RFC1918] address 🎀 Then you can signal reachability to it with a (32-bit) host route. very main character energy for a private setup 🦋

For IPv6-numbered services, Anycast Addresses are not scoped any differently from unicast addresses 💜 So the address-suitability guidelines we just talked about for IPv4 also apply to IPv6. And get this, the old rules that banned anycast distribution of services over IPv6 were removed from the IPv6 addressing spec in [RFC4291]. the way that's been cleaned up is so iconic ✨

4.6. Data Synchronisation 🌈

Some services have been rolled out in a localised form, where people in certain regions get regionally-relevant content 🌍 But lots of services are meant to give consistent answers no matter where the request comes from 💅 For a service that's distributed with anycast, that means different Anycast Nodes have to behave in a consistent manner. And if that consistent behaviour depends on a data set, then that data has to be synchronised between the nodes 🫶

How the data gets synchronised depends on what the service is. Some examples are zone transfers for authoritative DNS servers, and rsync for FTP archives 🧚 In general, synchronising data between Anycast Nodes involves transactions happening between non-anycast addresses. so the sync traffic is its own little side-quest 💖

Synchronising data across public networks should be done with proper authentication and encryption 🔐 no skipping that step, bestie 🙈

4.7. Node Autonomy 💜

If you're deploying anycast because you want better reliability through redundancy, you really want to make sure one defect can't take down many (or all) of your nodes 😭 You also don't want one node failing to cause a cascading failure that knocks out the next node, and the next, until the whole service is defeated 💀

The trick is to avoid co-dependencies by making each node as autonomous and self-sufficient as it can be ✨ How well nodes can survive failures elsewhere depends on the service itself. But for services that can handle disconnected operation, like the timed propagation of changes between master and slave servers in the DNS, you can get a really high degree of autonomy 👑

Cascading failure from load can also get less likely if you deploy both Global and Local Nodes for a single service 🍓 That's because the effective fail-over path for traffic is generally from Local Node to Global Node. Traffic that might sink one Local Node is unlikely to sink all of the Local Nodes, except in the most degenerate cases 🙈

The chance of cascading failure from a software defect in an operating system or server can often be reduced by running different implementations of the operating system, server software, routing protocol software, and so on 🌸 That way, a defect in one component doesn't take down the whole system. diversity is everything, periodt 🦋

Just so you know, these approaches to node autonomy are, to different degrees, kind of at odds with the practical goal of making a deployed service easy to operate 😅 A service that's overly complex is more likely to suffer from operator error than one that's simpler to run. So give all of these aspects careful thought so you can strike an appropriate balance ⚖️

4.8. Multi-Service Nodes 🎀

For a service distributed across a routing system where covering prefixes are required to announce reachability to a single Service Address (see Section 4.4.2), things get extra spicy when multiple services need to be distributed across one set of nodes 🌶️ That's because you have to signal the availability of each individual service to the routing system, so requests don't land on nodes that can't process them (see Section 4.4.1). it's the whole vibe check 💅

Several approaches are described in the following sections 🧚

4.8.1. Multiple Covering Prefixes

Each Service Address is chosen so that only one Service Address is covered by each advertised prefix 💖 Advertising and withdrawing a single covering prefix can then be tightly coupled to the availability of that one associated service 🫶

This is the most straightforward approach 🌸 But it makes very poor use of globally-unique addresses, so it's only suitable for a small number of critical, infrastructural services, like root DNS servers 👑 Deploying services Internet-wide with this approach will not scale. not me saying "it's not scaling" 💀

4.8.2. Pessimistic Withdrawal

Multiple Service Addresses are chosen so they're all covered by a single prefix 🦋 The advertisement and withdrawal of that single covering prefix is coupled to the availability of all the associated services. if any individual service goes unavailable, the covering prefix gets withdrawn 😭

The tricky part is that all Service Addresses have to be available for this to work. The announcement needs to be triggered by the presence of all the component routes, not just a single covered route 🧩

Here's the catch: one malfunctioning service takes every deployed service on that node offline 💀 That might make this approach unsuitable for lots of applications 🙈

4.8.3. Intra-Node Interior Connectivity

Multiple Service Addresses are chosen so they're covered by a single prefix 🌈 This time, the advertisement and withdrawal of the covering prefix is coupled to the availability of any one service. Nodes have interior connectivity, for example using tunnels 🚇 Host routes for Service Addresses are distributed using an IGP that extends to include routers at all nodes 🗺️

If a service is unavailable at one node but available at other nodes, a request may be routed over the interior network from the receiving node towards some other node for processing 🔁

Now the spicy bit: if some local services in a node are down and the node is disconnected from other nodes, continuing to advertise the covering prefix might make requests get black-holed 🕳️ yikes 😭

This approach gives reasonable address utilisation of the netblock covered by the announced prefix. the cost is reduced autonomy for individual nodes, since the IGP that all nodes participate in can be viewed as a single point of failure 🥲

4.9. Node Identification by Clients 🔍

From time to time, all clients of deployed services run into problems, and those problems need diagnosing 🩺 A service distributed using anycast adds one extra variable to that diagnostic process compared to a simple unicast service: which particular Anycast Node is handling the client's request 🤔

Sometimes common network-level diagnostic tools, like traceroute, are enough to figure out which node a client is using 🗺️ But those tools can be beyond what users on the client side of a transaction are able to do. Plus, network conditions can change by the time anyone gets around to using the tools, so the evidence is gone 💨

Troubleshooting anycast services gets SO much easier if mechanisms for identifying a node are designed right into the protocol 🫶 Examples include the NSID option in DNS [NSID], and the common practice of including hostname information in an SMTP server's initial greeting at session initiation [RFC2821] 📬

Providing these kinds of in-band mechanisms for node identification is strongly recommended for services distributed using anycast 💖

5. Service Management 📋

5.1. Monitoring 📊

Monitoring a distributed service is more complex than monitoring a non-distributed one 🧐 The reason is that the observed accuracy and availability of the service can be different depending on which part of the network the clients are attached to. And when a problem pops up, it isn't always obvious which node served the request, and so which node is misbehaving 🕵️‍♀️

It's recommended that distributed services are monitored from probes spread representatively across the routing system 🌐 Wherever possible, record the identity of the node answering each request alongside the performance and availability statistics 💜 The RIPE NCC DNSMON service [DNSMON] is an example of this kind of monitoring for the DNS 🫶

Monitoring the routing system can also give you useful diagnostics for troubleshooting service availability 🔭 Where perspective matters, do it from a variety of places. You can use dedicated probes, or public route measurement facilities on the Internet, like the RIPE NCC Routing Information Service [RIS] and the University of Oregon Route Views Project [ROUTEVIEWS] 🦋

Monitoring the health of the component devices in an anycast deployment (hosts, routers, and so on) is pretty straightforward ✨ You can use the same tools and techniques you'd use for any other network-connected infrastructure, without the extra complexity of monitoring anycast Service Addresses 💅

6. Security Considerations 🔐

6.1. Denial-of-Service Attack Mitigation 🛡️

This document describes mechanisms for deploying services on the Internet that can help mitigate vulnerability to attack 💖 Here's how:

  1. An Anycast Node can act as a sink for attack traffic originating within its sphere of influence. That keeps nodes elsewhere from having to deal with that traffic 🧱
  2. Dealing with attack traffic whose sources are widely distributed is itself spread across all the nodes that contribute to the service 🌸 Since sorting legitimate traffic from attack traffic is distributed too, this may scale better than a service that isn't distributed 📈

6.2. Service Compromise 🚨

Distributing a service across several (or many) autonomous nodes adds monitoring work and a bigger systems administration burden for the service operator 😩 That extra load might reduce how effective host and router security is 😬

The upside is that you can take compromised servers offline without compromising the service. But that only works if you have working procedures to do it quickly and reliably ⚡ no procedures, no save 💀

6.3. Service Hijacking 🦋

okay bestie, so it's possible that some unauthorised party could advertise routes for anycast Service Addresses across a network 😳. Then they could scoop up legit request traffic, or process requests in a way that wrecks the service (or both!!). A rogue Anycast Node can be super hard for clients or the operator of the service to spot 🙈

The risk of service hijacking through messing with the routing system is there whether or not the service uses anycast 💀. BUT the fact that legit Anycast Nodes are visible in the routing system can make it harder to catch the rogue ones. 🔥

Lots of protocols with authentication or integrity protection do it really robustly 💅, like with periodic re-authentication within a single session, or integrity protection at the channel level (e.g., [RFC2845], [RFC3207]) or the message level (e.g., [RFC4033], [RFC2311]). Less robust protocols may be more open to session hijacking 😬. Since there's a bigger chance of undetected session hijacking with anycast services, using robust protocols is recommended for anycast services that need authentication or integrity protection 🫶.

7. Acknowledgements 🌸

The authors gratefully acknowledge the contributions from various participants of the grow working group, and in particular Geoff Huston, Pekka Savola, Danny McPherson, Ben Black, and Alan Barrett 💖.

This work was supported by the US National Science Foundation (research grant SCI-0427144) and DNS-OARC 🎀.

8. References ✨

8.1. Normative References

  • [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981.
  • [RFC1918] Rekhter, Y., Moskowitz, R., Karrenberg, D., Groot, G., and E. Lear, "Address Allocation for Private Internets", BCP 5, RFC 1918, February 1996.
  • [RFC1997] Chandrasekeran, R., Traina, P., and T. Li, "BGP Communities Attribute", RFC 1997, August 1996.
  • [RFC2439] Villamizar, C., Chandra, R., and R. Govindan, "BGP Route Flap Damping", RFC 2439, November 1998.
  • [RFC2827] Ferguson, P. and D. Senie, "Network Ingress Filtering: Defeating Denial of Service Attacks which employ IP Source Address Spoofing", BCP 38, RFC 2827, May 2000.
  • [RFC3704] Baker, F. and P. Savola, "Ingress Filtering for Multihomed Networks", BCP 84, RFC 3704, March 2004.
  • [RFC4271] Rekhter, Y., Li, T., and S. Hares, "A Border Gateway Protocol 4 (BGP-4)", RFC 4271, January 2006.
  • [RFC4291] Hinden, R. and S. Deering, "IP Version 6 Addressing Architecture", RFC 4291, February 2006.

8.2. Informative References

Authors' Addresses 💌

Joe Abley
Afilias Canada, Corp.
204 - 4141 Yonge Street
Toronto, ON  M2P 2A8
Canada

Phone: +1 416 673 4176
EMail: jabley@ca.afilias.info
URI:   http://afilias.info/


Kurt Erik Lindqvist
Netnod Internet Exchange
Bellmansgatan 30
118 47 Stockholm
Sweden

EMail: kurtis@kurtis.pp.se
URI:   http://www.netnod.se/

Copyright (C) The IETF Trust (2006).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST, AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property 🧚

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.

Acknowledgement 🌈

Funding for the RFC Editor function is currently provided by the Internet Society. 🫶

📦 where this page comes from