RFC 2187

Application of Internet Cache Protocol (ICP), version 2

📅 published September 1997, from ietf

RFC 2187 🎀 Application of Internet Cache Protocol (ICP), version 2

bestie, look at this ID card 💅 This doc is RFC 2187, titled Application of Internet Cache Protocol (ICP), version 2. It came out of the Network Working Group and was written by D. Wessels and K. Claffy, from the National Laboratory for Applied Network Research/UCSD. It's Informational and dated September 1997. The header doesn't list any Obsoletes or Updates, so it doesn't replace or change anything else 🌸

Status of This Memo ✨

This memo is here to give info to the Internet community 🦋 It does not specify an Internet standard of any kind. Distribution of this memo is unlimited 💜

Abstract 🌈

So this doc describes how to use ICPv2 (Internet Cache Protocol version 2, RFC2186) for Web caching 🧚 ICPv2 is a lightweight message format that Web caches use to talk to each other. Several independent caching implementations use ICP[3,5] now, so it's important to write down the existing practical uses of ICP for anyone trying to implement, deploy, and extend it 🙌

ICP queries and replies are about whether a URL (or object) exists in a neighbor cache 👀 Caches swap ICP messages and use what they learn to pick the best place to grab an object from. A companion document (RFC2186) explains the format and syntax of the protocol itself. This doc is all about ICP deployment, efficiency, security, and how it interacts with the rest of Web traffic behavior 💖

Table of Contents 📜

   1.   Introduction.................................................  2
   2.   Web Cache Hierarchies........................................  3
   3.   What is the Added Value of ICP?..............................  5
   4.   Example Configuration of ICP Hierarchy.......................  5
     4.1. Configuring the `proxy.customer.org' cache.................  6
     4.2. Configuring the `cache.isp.com' cache......................  6
   5.   Applying the Protocol........................................  7
     5.1. Sending ICP Queries........................................  8
     5.2. Receiving ICP Queries and Sending Replies.................. 10
     5.3. Receiving ICP Replies...................................... 11
     5.4. ICP Options................................................ 13
   6.   Firewalls.................................................... 14
   7.   Multicast.................................................... 14
   8.   Lessons Learned.............................................. 16
     8.1. Differences Between ICP and HTTP........................... 16
     8.2. Parents, Siblings, Hits and Misses......................... 16
     8.3. Different Roles of ICP..................................... 17
     8.4. Protocol Design Flaws of ICPv2............................. 17
   9.   Security Considerations...................................... 18
     9.1. Inserting Bogus ICP Queries................................ 19
     9.2. Inserting Bogus ICP Replies................................ 19
     9.3. Eavesdropping.............................................. 20
     9.4. Blocking ICP Messages...................................... 20
     9.5. Delaying ICP Messages...................................... 20
     9.6. Denial of Service.......................................... 20
     9.7. Altering ICP Fields........................................ 21
     9.8. Summary.................................................... 22
   10.  References................................................... 23
   11.  Acknowledgments.............................................. 24
   12.  Authors' Addresses........................................... 24

1. Introduction 🌸

ICP is a lightweight message format for talking among Web caches 💅 It's used to swap hints about whether URLs exist in neighbor caches. Caches send ICP queries and replies to gather info so they can pick the most appropriate place to retrieve an object from ✨

This doc describes how ICP is implemented in software 🧚 For the protocol and message format, check out the companion document (RFC2186). We're not making judgments about whether or how ICP should be used in particular Web caching setups 🫶 ICP might be a "net win" in some situations and a "net loss" in others. We know some practices in this doc are suboptimal 💀 A few exist for historical reasons, and some parts got better in later versions. Since this doc only describes current practices, we're focusing on documenting rather than evaluating 📝 We do address known security problems and other shortcomings though 🔥

Here's the plan for the rest of the doc 🦋 First we cover Web cache hierarchies, explain why you'd use ICP, and show how to configure it in cache hierarchies. Then we walk through a step-by-step ICP query-response transaction. After that we talk about how ICP works with firewalls and briefly touch on multicasting ICP. We end with the lessons we've learned during the protocol's development and deployement so far, plus the canonical security considerations 👑

ICP was first developed by Peter Danzig, et. al. at the University of Southern California, as a central part of hierarchical caching in the Harvest research project[3] 🌈

2. Web Cache Hierarchies 🍓

A single Web cache cuts down the traffic generated by the clients behind it 🥰 A group of Web caches can get the same kind of benefit by sharing another cache. The Harvest researchers imagined that connecting Web caches hierarchically would matter a lot 🙈 In a cache hierarchy (or mesh), each cache sets up peering relationships with its neighbor caches. There are two types of relationship: parent and sibling. A parent cache is basically one level up in the hierarchy, and a sibling cache is on the same level. The terms "neighbor" and "peer" both mean either a parent or a sibling that's a single "cache-hop" away 💖 Figure 1 shows a simple hierarchy setup.

But what does it even mean to be "on the same level" or "one level up?" 🤔 Document requests generally flow up the hierarchy. When a cache doesn't hold a requested object, it may ask via ICP whether any of its neighbor caches has it ✨ If a neighbor does have the object (i.e., a "neighbor hit"), the cache will request it from them. If none of the neighbors has the object (a "neighbor miss"), then the cache must forward the request either to a parent or directly to the origin server 😭 The big difference between a parent and a sibling is that a "neighbor hit" may be fetched from either one, but a "neighbor miss" may NOT be fetched from a sibling 🚫 In other words, in a sibling relationship, a cache can only ask to retrieve objects the sibling already has cached. The same cache can ask a parent to retrieve any object, whether or not it's cached 👑 A parent cache's role is

     T H E   I N T E R N E T
   ===========================
       |          ||
       |          ||
       |          ||
       |          ||
       |      +----------------------+
       |      |                      |
       |      |        PARENT        |
       |      |        CACHE         |
       |      |                      |
       |      +----------------------+
       |          ||
     DIRECT       ||
   RETRIEVALS     ||
       |          ||
       |         HITS
       |         AND
       |        MISSES
       |       RESOLVED
       |          ||
       |          ||
       |          ||
       V          \/
   +------------------+                    +------------------+
   |                  |                    |                  |
   |      LOCAL       |/--------HITS-------|     SIBLING      |
   |      CACHE       |\------RESOLVED-----|      CACHE       |
   |                  |                    |                  |
   +------------------+                    +------------------+
      |  |  |  |  |
      |  |  |  |  |
      |  |  |  |  |
      V  V  V  V  V
   ===================
      CACHE CLIENTS

FIGURE 1: A Simple Web cache hierarchy. The local cache can retrieve hits from sibling caches, hits and misses from parent caches, and some requests directly from origin servers.

to provide "transit" for the request if necessary 🚗 That's why parent caches are ideally located within or on the way to a transit Internet service provider (ISP).

Squid and Harvest both allow for complex hierarchical configurations 🧠 For example, you could say a given neighbor is only used for a certain class of requests, like URLs from a specific DNS domain.

It's also possible to treat a neighbor as a sibling for some requests and as a parent for others 🦋

The cache hierarchy model described here has a bunch of features to keep top-level caches from becoming choke points ✨ One is the ability to restrict parents, as we just covered (by domains). Another optimization is that the cache only forwards cachable requests to its neighbors 🙌 A big chunk of Web requests are inherently uncachable, including requests that need certain types of authentication, session-encrypted data, highly personalized responses, and certain types of database queries. Lower level caches should handle these requests directly rather than burdening parent caches 🫶

3. What is the Added Value of ICP? 💜

Sure, you can maintain cache hierarchies without ICP, but the lack of ICP or something similar rules out sibling meta-communicative relationships, i.e., mechanisms for querying nearby caches about a given document 🙅‍♀️

One concern about ICP is the extra delay that an ICP query/reply exchange adds to an HTTP transaction 😬 But if the ICP query can locate the object in a nearby neighbor cache, the ICP delay may be more than offset by how much faster the data arrives from that neighbor 🚀 To keep ICP delays down, the caches (and the protocol itself) are designed to return ICP requests quickly. In fact, the application does minimal processing of the ICP request, so most ICP-related delay comes from transmission on the network 💨

ICP also tells you whether a neighbor is reachable 👀 If ICP replies from a neighbor don't arrive, then either the network path is congested (or down), or the cache application isn't running on the ICP-queried neighbor machine. Either way, the cache should not use this neighbor at this time 🚫 Also, because an idle cache can turn replies around faster than a busy one, all other things being equal, ICP provides some form of load balancing ⚖️

4. Example Configuration of ICP Hierarchy 🎀

Configuring caches within a hierarchy means setting up peering relationships, which currently involves manual configuration at both peering endpoints 🛠️ One cache must indicate that the other is a parent or sibling. The other cache will most likely have to add the first cache to its access control lists 🔐

Below, we show some sample configuration lines for a hypothetical situation 🧚 We have two caches: one operated by an ISP, and another operated by a customer. First we describe how the customer would configure their cache to peer with the ISP. Second, we describe how the ISP would allow the customer access to its cache 💖

4.1. Configuring the `proxy.customer.org' cache

okay so, to set up parents and siblings in a hierarchy in Squid, you add a cache_host directive to the config file 💅 The format is:

    cache_host hostname type http-port icp-port [options]

where type is either parent', sibling', or `multicast'. For our example, it would be:

    cache_host cache.isp.com parent 8080 3130

🎀 This config makes the customer cache send most cache misses up to the parent. cgi-bin requests and non-GET requests skip the parent and go directly to the origin. Routing everything through the parent isn't always what you want, though. Servers that also live in the customer.org domain are a good example. To always handle those local domains directly, the customer adds this to the config file:

    local_domain customer.org

Sometimes the customer only wants the ISP cache for certain DNS domains. Limiting requests like this is actually more common higher up in cache hierarchies, but we'll show it here anyway ✨ To limit the ISP cache to a subset of DNS domains, the customer uses:

    cache_host_domain cache.isp.com com net org

After that, any requests that are NOT in the .com, .net, or .org domains get handled directly. 🌸

4.2. Configuring the `cache.isp.com' cache

🧚 To set up the query-receiving side of a cache peer relationship, you use access lists, which work a lot like the ones used for routing peers. Access lists give you tons of room to customize the peering relationship. If there are no access lines present, the cache allows the request by default.

Note that the cache.isp.com cache doesn't need to name the customer cache as a peer, and the type of relationship isn't encoded in the ICP query itself. The access control entries are what control the relationships between this cache and its neighbors. For our example, the ISP would use:

    acl src Customer  proxy.customer.org
    http_access allow Customer
    icp_access  allow Customer

This defines an access control entry named `Customer' that specifies a source IP address of the customer cache machine. The customer cache is then allowed to make any request to both the HTTP and ICP ports, including cache misses. This setup means the ISP cache is a parent of the customer. 💜

If the ISP wants to enforce a sibling relationship, it has to deny access to cache misses. It does that like this:

    miss_access deny Customer

🙈 The ISP should also tell the customer about this, so the customer can switch their config from parent to sibling. Otherwise, if the customer requests an object that isn't in the ISP cache, an error message gets generated. Nobody wants that drama!

5. Applying the Protocol

🦋 The following sections describe the ICP implementation in the Harvest[3] (research version) and Squid Web cache[5] packages. In terms of version numbers, that means version 1.4pl2 for Harvest and version 1.1.10 for Squid.

The basic sequence of events in an ICP transaction goes like this:

  1. Local cache receives an HTTP[1] request from a cache client.
  2. The local cache sends ICP queries (section 5.1).
  3. The peer cache(s) receive the queries and send ICP replies (section 5.2).
  4. The local cache receives the ICP replies and decides where to forward the request (section 5.3).

5.1. Sending ICP Queries

5.1.1. Determine whether to use ICP at all

Not every HTTP request needs an ICP query, and that's honestly a relief 🙌 Cache hits don't need ICP, because the request is satisfied right away. For origin servers very close to the cache, we don't want to use any neighbor caches. In Squid and Harvest, the administrator decides what counts as a local' server using the local_domain' and `local_ip' configuration options. The cache always contacts a local server directly and never queries a peer cache.

There are other classes of requests that the cache (or the administrator) might prefer to forward directly to the origin server. In Squid and Harvest, one of those classes is all non-GET request methods. A Squid cache can also be set up to not use peers for URLs that match the `hierarchy_stoplist'.

For an HTTP request to lead to an ICP transaction, it must:

  • not be a cache hit
  • not be to a local server
  • be a GET request, and
  • not match the `hierarchy_stoplist' configuration.

We call this a "hierarchical" request. A "non-hierarchical" request is one that doesn't generate any ICP traffic. To avoid processing requests that are likely to hurt cache efficiency, you can configure the cache to skip the hierarchy for URLs that contain certain strings (e.g. `cgi_bin').

5.1.2. Determine which peers to query

🔥 By default, a cache sends an ICP_OP_QUERY message to each peer, unless any one of the following is true:

  • Restrictions prevent querying a peer for this request, based on the configuration directive cache_host_domain', which specifies a set of DNS domains (from the URLs) for which the peer should or should not be queried. In Squid, a more flexible directive (cache_host_acl') supports restrictions on other parts of the request (method, port number, source, etc.).
  • The peer is a sibling, and the HTTP request includes a "Pragma: no-cache" header. This is because the sibling would be asked to transit the request, which is not allowed.
  • The peer is configured to never be sent ICP queries (i.e. with the `no-query' option).

If the determination yields only one queryable ICP peer, and the Squid configuration directive `single_parent_bypass' is set, then you can skip waiting for the single ICP response and just send the HTTP request directly to the peer cache.

The Squid configuration option `source_ping' tells a Squid cache to send a ping to the original source at the same time as its ICP queries, in case the origin is closer than any of the caches. 👑

5.1.3. Calculate the expected number of ICP replies

Harvest and Squid want to maximize the chance of getting a HIT reply from one of the peers. So the cache waits for all ICP replies to be received. Normally, we expect an ICP reply for each query sent, except in these cases:

  • When the peer is believed to be down. If the peer is down, Squid and Harvest keep sending it ICP queries, but don't expect the peer to reply. When an ICP reply is received from the peer again, its status changes to up.

The way up/down status is determined has shifted a bit as the Harvest and Squid software evolved. Both Harvest and Squid mark a peer down when it fails to reply to 20 consecutive ICP queries. Squid also marks a peer down when a TCP connection fails, and marks it up again when a diagnostic TCP connection succeeds.

  • When sending to a multicast address. In this case we'll probably expect more than one reply, and we have no way to definitively determine how many to expect. We discuss multicast issues in section 7 below.

5.1.4. Install timeout event

🍓 ICP uses UDP as its underlying transport, so ICP queries and replies may sometimes get dropped by the network. The cache installs a timeout event in case not all of the expected replies arrive. By default, Squid and Harvest use a two-second timeout. If object retrieval hasn't started when the timeout occurs, a source is selected as described in section 5.3.9 below.

5.2. Receiving ICP Queries and Sending Replies

When an ICP_OP_QUERY message is received, the cache examines it and decides which reply message to send. It sends one of the following reply opcodes, tested in the order listed:

5.2.1. ICP_OP_ERR

The URL is extracted from the payload and parsed. If parsing fails, an ICP_OP_ERR message is returned.

5.2.2. ICP_OP_DENIED

The access controls are checked. If the peer isn't allowed to make this request, ICP_OP_DENIED is returned. Squid counts the number of ICP_OP_DENIED messages sent to each peer. If more than 95% of more than 100 replies have been denied, then no reply is sent at all. This keeps misconfigured caches from endlessly bouncing unnecessary ICP messages back and forth. 🙅‍♀️

5.2.3. ICP_OP_HIT

If the cache reaches this point without already matching one of the previous opcodes, it means the request is allowed, and we have to figure out whether it will be a HIT or a MISS. So we check whether the URL exists in the local cache. If it does, and the cached entry is fresh for at least the next 30 seconds, we can return an ICP_OP_HIT message. The stale/fresh determination uses the local refresh (or TTL) rules.

⚠️ Note that a race condition exists for ICP_OP_HIT replies to sibling peers. The ICP_OP_HIT means that a subsequent HTTP request for the named URL would result in a cache hit. We assume the HTTP request will come very quickly after the ICP_OP_HIT. However, there's a slight chance the object might be purged from this cache before the HTTP request arrives. If that happens, and the replying peer has applied Squid's `miss_access' configuration, then the user will get a very confusing access denied message.

5.2.3.1. ICP_OP_HIT_OBJ

Before returning the ICP_OP_HIT message, we check whether we can send an ICP_OP_HIT_OBJ message instead. We can use ICP_OP_HIT_OBJ if:

  • The ICP_OP_QUERY message had the ICP_FLAG_HIT_OBJ flag set.
  • The entire object (plus URL) will fit in an ICP message. The maximum ICP message size is 16 Kbytes, but an application may choose to set a smaller maximum value for ICP_OP_HIT_OBJ replies.

Normally ICP replies are sent right after the query is received, but the ICP_OP_HIT_OBJ message can't be sent until the object data is available to copy into the reply message. For Squid and Harvest, this means the object must be "swapped in" from disk if it isn't already in memory. So, on average, an ICP_OP_HIT_OBJ reply will have higher latency than ICP_OP_HIT. ⏳

5.2.4. ICP_OP_MISS_NOFETCH

At this point we have a cache miss. ICP has two types of miss replies. If the cache doesn't want the peer to request the object from it, it sends an ICP_OP_MISS_NOFETCH message.

5.2.5. ICP_OP_MISS

Finally, an ICP_OP_MISS reply is returned as the default. If the replying cache is a parent of the querying cache, the ICP_OP_MISS is an invitation to fetch the URL through the replying cache. 🫶

5.3. Receiving ICP Replies

Some ICP replies get ignored, specifically when any of the following is true:

  • The reply message originated from an unknown peer.
  • The object named by the URL does not exist.
  • The object is already being fetched.

5.3.1. ICP_OP_DENIED

If more than 95% of more than 100 replies from a peer cache have been ICP_OP_DENIED, then a denial rate that high most likely means a configuration error, either locally or at the peer. For this reason, no further queries are sent to the peer for the duration of the cache process. 🚫

5.3.2. ICP_OP_HIT

Object retrieval starts immediately from the replying peer. 🏃‍♀️

5.3.3. ICP_OP_HIT_OBJ

The object data is extracted from the ICP message, and the retrieval is complete. If there's some problem with the ICP_OP_HIT_OBJ message (e.g. missing data), the reply is treated like a standard ICP_OP_HIT. 🩹

5.3.4. ICP_OP_SECHO ✨

Okay so retrieval kicks off immediately from the origin server 🏃‍♀️💨, because the ICP_OP_SECHO reply showed up before any ICP_OP_HIT's did. If an ICP_OP_HIT had landed first, this ICP_OP_SECHO reply gets ignored, since the retrieval already started. 🙈

5.3.5. ICP_OP_DECHO 🦋

An ICP_OP_DECHO reply gets handled just like an ICP_OP_MISS 💅. Non-ICP peers must always be configured as parents, because a non-ICP sibling makes no sense 🙅‍♀️.

One serious problem with ICP_OP_DECHO is that it bounces messages off the peer's UDP echo port. So it does not tell you the peer cache is actually running 💀. It only tells you that network connectivity exists between the pair. 🌸

5.3.6. ICP_OP_MISS 💔

If the peer is a sibling, the ICP_OP_MISS reply is ignored. Otherwise, the peer may be "remembered" for future use, in case no HIT replies show up later (section 5.3.9) 🧚.

Harvest and Squid remember the first parent to return an ICP_OP_MISS message. With Squid, the parents may be weighted, so the "first parent to miss" might not actually be the first reply received. We call this the FIRST_PARENT_MISS ✨. Remember that sibling misses are entirely ignored, since we only care about misses from parents. The parent miss RTT's can be weighted, because sometimes the closest parent isn't the one people want to use 🫶.

Also, recent versions of Squid may remember the parent with the lowest RTT to the origin server, using the ICP_FLAG_SRC_RTT option. We call this the CLOSEST_PARENT_MISS 👑.

5.3.7. ICP_OP_MISS_NOFETCH 🙃

This reply is essentially ignored. A cache must not forward a request to a peer that returns ICP_OP_MISS_NOFETCH 🚫.

5.3.8. ICP_OP_ERR 🤫

Silently ignored 🫥.

5.3.9. When all peers MISS. 🎯

For ICP_OP_HIT and ICP_OP_SECHO, the request gets forwarded immediately 🚀. For ICP_OP_HIT_OBJ there's no need to forward the request at all. For all other reply opcodes, we wait until the expected number of replies have come in. When we have all of the expected replies, or when the query timeout occurs, it's time to forward the request ⏰.

Since MISS replies came back from all peers, we must either select a parent cache or the origin server 👀.

  • If the peers are using the ICP_FLAG_SRC_RTT feature, we forward the request to the peer with the lowest RTT to the origin server. If the local cache is also measuring RTT's to origin servers, and is closer than any of the parents, the request goes directly to the origin server 🏆.
  • If there is a FIRST_PARENT_MISS parent available, the request will be forwarded there 💖.
  • If the ICP query/reply exchange didn't produce any appropriate parents, the request will be sent directly to the origin server (unless firewall restrictions prevent it) 🌈.

5.4. ICP Options 🎀

The following options were added to Squid to support some new features while keeping backward compatibility with the Harvest implementation 🍓.

5.4.1. ICP_FLAG_HIT_OBJ 🧁

This flag is off by default. It gets set in an ICP_OP_QUERY message only if these three criteria are met:

  • It is enabled in the cache configuration file with `udp_hit_obj on'.
  • The peer must be using ICP version 2.
  • The HTTP request must not include the Pragma: no-cache header.

5.4.2. ICP_FLAG_SRC_RTT 📡

This flag is off by default. It gets set in an ICP_OP_QUERY message only if these two criteria are met:

  • It is enabled in the cache configuration file with `query_icmp on'.
  • The peer must be using ICP version 2.

6. Firewalls 🧱

Running a Web cache behind a firewall or in a private network poses some interesting problems 🤔. The hard part is figuring out whether the cache can connect to the origin server. Harvest and Squid provide an inside_firewall' configuration directive to list DNS domains on the near side of a firewall. Everything else is assumed to be on the far side of a firewall. Squid also has a firewall_ip' directive, so inside hosts can be specified by IP addresses too 💅.

In a simple configuration, a Squid cache behind a firewall will have only one parent cache (which lives on the firewall itself). In this case, Squid must use that parent for all servers beyond the firewall, so there's no need to use ICP 🙌.

In a more complex configuration, there may be a number of peer caches also behind the firewall. Here, ICP may be used to check for cache hits in the peers. Occasionally, when ICP is being used, no replies come back at all 😭. If the cache weren't behind a firewall, the request would be forwarded directly to the origin server. But in this situation, the cache has to pick a parent cache, either randomly or based on configuration information. For example, Squid lets you designate a parent cache as the default choice when no others are available 🫶.

7. Multicast 📣

For efficient distribution, a cache may deliver ICP queries to a multicast address, and neighbor caches may join the multicast group to receive those queries 🦋.

Current practice is that caches send ICP replies only to unicast addresses, for several reasons:

  • Multicasting ICP replies would not reduce the number of packets sent.
  • It prevents other group members from receiving unexpected replies.
  • The reply should follow unicast routing paths to show (unicast) connectivity between the receiver and the sender, since the subsequent HTTP request will be unicast routed 🛣️.

Trust is a big deal in inter-cache relationships 🔐. A Web cache should not automatically trust any cache that replies to a multicast ICP query. Caches should ignore ICP messages from addresses that aren't specifically configured as neighbors. Otherwise, someone could easily pollute a cache mesh by running an illegitimate cache, having it join a group, returning ICP_OP_HIT for all requests, and then delivering bogus content 😈.

When sending to multicast groups, cache administrators must be careful to use the minimum multicast TTL required to reach all group members. Joining a multicast group requires no special privileges, and there's no way to stop anyone from joining "your" group 😱. Two groups of caches using the same multicast address could overlap, which would cause a cache to receive ICP replies from unknown neighbors. The unknown neighbors wouldn't be used to retrieve the object data, but the cache would constantly receive ICP replies it must always ignore 🙄.

To prevent an overlapping cache mesh, caches should limit the scope of their ICP queries with appropriate TTLs. An application such as mtrace[6] can determine appropriate multicast TTLs 🔍.

As mentioned in section 5.1.3, we need to estimate the number of expected replies for an ICP_OP_QUERY message. For unicast, we expect one reply per query if the peer is up. For multicast, though, we generally expect more than one reply, but we have no way of knowing exactly how many 🤷‍♀️. Squid regularly (every 15 minutes) sends out test ICP_OP_QUERY messages to only the multicast group peers. As with a real ICP query, a timeout event is installed and the replies are counted until the timeout occurs. We've found that the received count varies a lot 📉. So the number of replies to expect is calculated as a moving average, rounded down to the nearest integer ✨.

8. Lessons Learned 📝

8.1. Differences Between ICP and HTTP 🆚

ICP is really different from HTTP 👀. HTTP supports a rich and sophisticated set of features. ICP, in contrast, was designed to be simple, small, and efficient 🐣. HTTP request and reply headers are lines of ASCII text delimited by a CRLF pair, whereas ICP uses a fixed size header and represents numbers in binary. The only thing ICP and HTTP have in common is the URL 🔗.

Note that the ICP message doesn't even include the HTTP request method 😬. The original implementation assumed that only GET requests would be cachable, and there would be no need to look up non-GET requests in neighbor caches. So the current version of ICP doesn't accommodate non-GET requests, although the next version of this protocol will likely include a field for the request method 🔜.

HTTP defines features that matter for caching but can't be expressed with the current ICP protocol. Among these are Pragma: no-cache, If-Modified-Since, and all of the Cache-Control features of HTTP/1.1 🙅. An ICP_OP_HIT_OBJ message may deliver an object that doesn't obey all of the request header constraints. These differences between ICP and HTTP are the reason we discourage the use of the ICP_OP_HIT_OBJ feature ⚠️.

8.2. Parents, Siblings, Hits and Misses 👨‍👩‍👧

Note that the ICP message has no field to indicate the intent of the querying cache 🫣. Nowhere in the ICP request or reply does it say that the two caches have a sibling or parent relationship. A sibling cache can only respond with HIT or MISS, not "you can retrieve this from me" or "you can not retrieve this from me." So the querying cache has to apply the HIT or MISS reply to its local configuration, so it doesn't resolve misses through a sibling cache 🧩. This is awkward, because this aspect of the relationship can only be configured in the cache originating the requests, and indirectly via the access controls configured in the queried cache, as described earlier in section 4.2 😩.

8.3. Different Roles of ICP 🎭

There are two different understandings of what ICP's role in a cache mesh really is 🤨. One understanding is that ICP's role is only object location, specifically to provide hints about whether or not a named object exists in a neighbor cache. The implied assumption is that cache hits are highly desirable, and ICP is used to maximize the chance of getting them. If an ICP message is lost due to congestion, nothing significant is lost, because the request will be satisfied regardless 🙌.

ICP is increasingly being asked to take on a more complex role: conveying cache usage policy 📜. For example, many organizations (e.g. universities) install a Web cache on the border of their network. Such organizations may be happy to establish sibling relationships with other, nearby caches, subject to the following terms:

  • Any of the organization's customers or users may request any object (cached or not).
  • Anyone may request an object already in the cache.
  • Anyone may request any object from the organization's servers behind the cache.
  • All other requests are denied; specifically, the organization will not provide transit for requests in which neither the client nor the server falls within its domain 🚧.

To successfully convey policy, the ICP exchange must very accurately predict the result (hit, miss) of a subsequent HTTP request 🎯. That result may often depend on other request fields, such as Cache-Control. So it's not possible for ICP to accurately predict the result without more, or perhaps all, of the HTTP request 💭.

8.4. Protocol Design Flaws of ICPv2 🔧

okay so the authors were honest about this part 🫶 they listed the things about the original ICP design that they'd do differently, so future versions don't repeat the same mistakes. lowkey respect 👑

  • The NULL-terminated URL in the payload field makes you step through the message one octet at a time just to find some of the fields (like where the object data starts in an ICP_OP_HIT_OBJ message). not it 💀
  • Two fields, Sender Host Address and Requester Host Address, are IPv4 specific. Neither one is actually used in practice, since they're normally zero-filled 🙈 If IP addresses are going to matter in an ICP message, every address needs an address family descriptor, and clients need a way to say whether they want IPv6 responses or not 🌈
  • Options are capped at 32 option flags and 32 bits of option data. The vibe they want instead is more like TCP, with an option descriptor followed by option data ✨
  • The URL string is still used as the cache key, but it doesn't really cut it anymore 😭 Some HTTP responses now change depending on the requestor's User-Agent and other headers. A cache key has to include all non-transport headers in the client's request, and all non-hop-by-hop request headers should be sent in an ICP query 🎀
  • ICPv2 uses different opcode values for queries and responses. ICP should use the same opcode on both sides of a two-sided transaction, with a "query/response" indicator saying which side is which 🦋
  • ICPv2 has no authentication fields at all 🫠

9. Security Considerations 🚨

okay bestie, this section is SERIOUS, so I'm keeping it dramatic but accurate. ICP over UDP is connectionless, which makes it a sitting duck for attacks. Below we look at the different vulnerabilities and attack methods, and what each one means 🔥

Our first line of defense is checking the source IP address of the ICP message, for example as given by recvfrom(2). ICP query messages should be processed if the access control rules let the querying address access the cache. ICP reply messages, though, must only be accepted from known neighbors, and a cache must ignore replies from unknown addresses 🛡️

Here's the catch: because we trust that an address in an IP packet is valid, ICP is open to IP address spoofing 😬 This document addresses some of the consequences of spoofing. Normally, spoofed addresses can only be detected by routers, not by hosts. However, the IP Authentication Header[7,8] can be used underneath ICP to provide cryptographic authentication of the entire IP packet that carries the ICP protocol, which gets rid of the risk of IP address spoofing 🔐

9.1. Inserting Bogus ICP Queries 🎭

Processing an ICP_OP_QUERY message has no known security implications, as long as the requesting address is granted access to the cache 💅

9.2. Inserting Bogus ICP Replies 👀

this one is about a third party generating ICP reply messages that reach the querying cache before the real reply shows up, or before any replies arrive at all. The third party can slip in bogus ICP replies that look like they came from legit neighbors. There are three vulnerabilities here:

  • Preventing a certain neighbor from being used

If a third party could send an ICP_OP_MISS_NOFETCH reply back before the real reply arrived, the (falsified) neighbor would not be used.

A third party could blast a cache with ICP_OP_DENIED messages until the threshold described in section 5.3.1 is reached, which would temporarily terminate the neighbor relationship 😤

  • Forcing a certain neighbor to be used

If a third party could send an ICP_OP_HIT reply back before the real reply arrived, the (falsified) neighbor would be used. This may violate the terms of a sibling relationship, since ICP_OP_HIT replies mean a subsequent HTTP request will also be a hit 🙃

Similarly, if bogus ICP_OP_SECHO messages can be generated, the cache would retrieve requests directly from the origin server 🏃‍♀️

  • Cache poisoning ☠️

The ICP_OP_HIT_OBJ message is especially sensitive to security issues, since it contains actual object data. Combined with IP address spoofing, this option makes it quite likely that the cache could get polluted with invalid objects 🤢

9.3. Eavesdropping 👂

Multicasting ICP queries is a super easy way for others to "snoop" on ICP messages 👀 If you enable multicast, cache administrators should configure the application to use the minimum required multicast TTL, using a tool such as mtrace[6]. Also, the IP Encapsulating Security Payload [7,9] mechanism can be used to protect ICP messages against eavesdropping 🔒

Eavesdropping on ICP traffic can hand third parties a list of URLs that cache users are browsing 😳 Because Squid and Harvest zero-fill the Requestor Host Address, those URLs can't be mapped back to individual host systems 🌸

By default, Squid and Harvest don't send ICP messages for URLs containing cgi-bin' or ?'. Those URLs sometimes carry sensitive information in their argument parameters 🙈 Cache administrators need to know that changing the configuration to send ICP queries for such URLs may expose sensitive information to outsiders, especially when multicast is in use ⚠️

9.4. Blocking ICP Messages 🚫

Intentionally blocked (or discarded) ICP queries or replies look exactly like link failure or congestion. They prevent the use of a neighbor and lead to timeouts (see section 5.1.4) 😩 If all messages are blocked, the cache assumes the neighbor is down and removes it from the selection algorithm. But if, say, every other query is blocked, the neighbor stays "alive," and every other request suffers the ICP timeout. ugh, so sneaky 🫣

9.5. Delaying ICP Messages 🐌

The neighbor selection algorithm normally waits for all ICP MISS replies to arrive. Delaying queries or replies so they arrive later than they normally would adds delay to the subsequent HTTP request. And of course, if messages are delayed until after the timeout, the behavior is the same as "blocking" above ⏰

9.6. Denial of Service 🌊

A denial-of-service attack, where the ICP port gets flooded with a continuous stream of bogus messages, has three vulnerabilities:

  • The application may log every bogus ICP message and eventually fill up a disk partition 💾
  • The socket receive queue may fill up, causing legitimate messages to be dropped 📭
  • The host may waste some CPU cycles receiving the bogus messages 🔥

9.7. Altering ICP Fields ✏️

Here we assume a third party can change one or more of the ICP reply message fields 😈

Opcode

Changing the opcode field is much like inserting the bogus messages described above. Changing a hit to a miss would prevent the peer from being used. Changing a miss to a hit would force the peer to be used.

Version

Altering the ICP version field may have unpredictable consequences if the new version number is recognized and supported. The receiving application should ignore messages with invalid version numbers. At the time of this writing, both version numbers 2 and 3 are in use. These two versions use some fields (e.g. Options) in a slightly different manner.

Message Length

An incorrect message length should be detected by the receiving application as an invalid ICP message.

Request Number

The request number is often used as part of the cache key. Harvest doesn't use the request number. Squid uses the request number in conjunction with the URL to create a cache key. Altering the request number will cause a lookup of the cache key to fail, which is similar to blocking the ICP reply altogether.

There's no requirement that a cache use both the URL and the request number to locate HTTP requests with outstanding ICP queries (though both Squid and Harvest do). The request number must always be the same in the query and the reply. However, if the querying cache uses only the request number to locate pending requests, there's some chance that a replying cache increments the request number in the reply to give the false impression that the two caches are closer than they really are. In other words, if there are a few ICP requests "in flight" at any given time, incrementing the reply request number can trick the querying cache into seeing a smaller round-trip time than really exists 🏎️

Options

There's little risk in having the Options bitfields altered. Any option bit must only be set in a reply if it was also set in a query. Changing a bit from clear to set is detectable by the querying cache, and such a message must be ignored. Changing a bit from set to clear is allowed and has no negative side effects 🌤️

Option Data

ICP_FLAG_SRC_RTT is the only option which uses the Option Data field. Altering the RTT values returned here can affect the neighbor selection algorithm, either forcing or preventing the use of a neighbor 😵‍💫

URL

The URL and Request Number are used to generate the cache key. Altering the URL will cause a lookup of the cache key to fail, and the ICP reply to be entirely ignored. This is similar to blocking the ICP reply altogether 🙃

9.8. Summary 📋

  • ICP_OP_HIT_OBJ is particularly vulnerable to security problems because it includes object data. For this reason, and others, its use is discouraged 🚫
  • Falsifying, altering, inserting, or blocking ICP messages can cause an HTTP request to fail in only two situations:
    • If the cache is behind a firewall and cannot directly connect to the origin server.
    • If a false ICP_OP_HIT reply causes the HTTP request to be forwarded to a sibling, where the request is a cache miss, and the sibling refuses to keep forwarding the request on behalf of the originating cache.
  • Falsifying, altering, inserting, or blocking ICP messages can easily cause HTTP requests to be forwarded (or not forwarded) to certain neighbors. If the neighbor cache has also been compromised, it could serve bogus content and pollute a cache hierarchy 🦠
  • Blocking or delaying ICP messages can cause HTTP requests to be further delayed, but they'll still be satisfied 🥹

10. References 📚

Okay bestie, this is the reading list, and it's giving "well-researched" 🤓✨ Every single one of these gets listed exactly as written, so nobody has to go hunting.

  • [1] Fielding, R., et. al, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2068, UC Irvine, January 1997.
  • [2] Berners-Lee, T., Masinter, L., and M. McCahill, "Uniform Resource Locators (URL)", RFC 1738, CERN, Xerox PARC, University of Minnesota, December 1994.
  • [3] Bowman M., Danzig P., Hardy D., Manber U., Schwartz M., and Wessels D., "The Harvest Information Discovery and Access System", Internet Research Task Force - Resource Discovery, http://harvest.transarc.com/.
  • [4] Wessels D., Claffy K., "ICP and the Squid Web Cache", National Laboratory for Applied Network Research, http://www.nlanr.net/~wessels/Papers/icp-squid.ps.gz.
  • [5] Wessels D., "The Squid Internet Object Cache", National Laboratory for Applied Network Research, http://squid.nlanr.net/Squid/
  • [6] mtrace, Xerox PARC, ftp://ftp.parc.xerox.com/pub/net-research/ipmulti/.
  • [7] Atkinson, R., "Security Architecture for the Internet Protocol", RFC 1825, NRL, August 1995.
  • [8] Atkinson, R., "IP Authentication Header", RFC 1826, NRL, August 1995.
  • [9] Atkinson, R., "IP Encapsulating Security Payload (ESP)", RFC 1827, NRL, August 1995.

11. Acknowledgments 🫶

Okay the thank-you list is my favourite part, because these people literally made this doc better 💖 The authors shout out:

  • Paul A Vixie <paul@vix.com> for giving excellent feedback on this document 🙌
  • Martin Hamilton <martin@mrrl.lut.ac.uk> for pushing the development of multicast ICP 🚀
  • Eric Rescorla <ekr@terisa.com> and Randall Atkinson <rja@home.net> for helping out with security issues 🔐
  • and especially Allyn Romanow for keeping us on the right track 👑 no notes, she's the GOAT

12. Authors' Addresses 🏡

Here's where you can find the authors, bestie 💌

Duane Wessels National Laboratory for Applied Network Research 10100 Hopkins Drive La Jolla, CA 92093

EMail: wessels@nlanr.net

K. Claffy National Laboratory for Applied Network Research 10100 Hopkins Drive La Jolla, CA 92093

EMail: kc@nlanr.net

That's the whole doc, babe 🎀 Thanks for hanging in there through all five parts. You're literally a whole networking genius now 🦋💅

📦 where this page comes from