RFC 3040
Internet Web Replication and Caching Taxonomy
RFC 3040 🎀 Internet Web Replication and Caching Taxonomy
Okay bestie, this is RFC 3040, and it's called "Internet Web Replication and Caching Taxonomy" 💖 It came out in January 2001 from the Network Working Group. The authors are I. Cooper (Equinix, Inc.), I. Melve (UNINETT), and G. Tomlinson (CacheFlow Inc.) 🧚 The category is Informational, and the title block doesn't list anything it obsoletes or updates.
Status of this Memo
This memo is giving info to the Internet community 💅 It does not specify an Internet standard of any kind, and distribution of it is unlimited 🌸
Copyright Notice
Copyright (C) The Internet Society (2001). All Rights Reserved. 🔥
Abstract
Okay so this memo is the one that sets the standard terminology and the taxonomy for web replication and caching infrastructure, the way it's actually deployed today ✨ It brings in standard concepts and the protocols people use in this space right now. It shows the solutions out there that use these technologies so we can build a standard taxonomy 🦋
Known problems with caching proxies are NOT part of this one, though. Those live in the doc titled "Known HTTP Proxy/Caching Problems" 🙈 This document presents open protocols and points you to published material for each of them 💜
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3
2.1 Base Terms . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 First order derivative terms . . . . . . . . . . . . . . . 6
2.3 Second order derivatives . . . . . . . . . . . . . . . . . 7
2.4 Topological terms . . . . . . . . . . . . . . . . . . . . 7
2.5 Automatic use of proxies . . . . . . . . . . . . . . . . . 8
3. Distributed System Relationships . . . . . . . . . . . . . 9
3.1 Replication Relationships . . . . . . . . . . . . . . . . 9
3.1.1 Client to Replica . . . . . . . . . . . . . . . . . . . . 9
3.1.2 Inter-Replica . . . . . . . . . . . . . . . . . . . . . . 9
3.2 Proxy Relationships . . . . . . . . . . . . . . . . . . . 10
3.2.1 Client to Non-Interception Proxy . . . . . . . . . . . . . 10
3.2.2 Client to Surrogate to Origin Server . . . . . . . . . . . 10
3.2.3 Inter-Proxy . . . . . . . . . . . . . . . . . . . . . . . 11
3.2.3.1 (Caching) Proxy Meshes . . . . . . . . . . . . . . . . . . 11
3.2.3.2 (Caching) Proxy Arrays . . . . . . . . . . . . . . . . . . 12
3.2.4 Network Element to Caching Proxy . . . . . . . . . . . . . 12
4. Replica Selection . . . . . . . . . . . . . . . . . . . . 13
4.1 Navigation Hyperlinks . . . . . . . . . . . . . . . . . . 13
4.2 Replica HTTP Redirection . . . . . . . . . . . . . . . . . 14
4.3 DNS Redirection . . . . . . . . . . . . . . . . . . . . . 14
5. Inter-Replica Communication . . . . . . . . . . . . . . . 15
5.1 Batch Driven Replication . . . . . . . . . . . . . . . . . 15
5.2 Demand Driven Replication . . . . . . . . . . . . . . . . 16
5.3 Synchronized Replication . . . . . . . . . . . . . . . . . 16
6. User Agent to Proxy Configuration . . . . . . . . . . . . 17
6.1 Manual Proxy Configuration . . . . . . . . . . . . . . . . 17
6.2 Proxy Auto Configuration (PAC) . . . . . . . . . . . . . . 17
6.3 Cache Array Routing Protocol (CARP) v1.0 . . . . . . . . . 18
6.4 Web Proxy Auto-Discovery Protocol (WPAD) . . . . . . . . . 18
7. Inter-Proxy Communication . . . . . . . . . . . . . . . . 19
7.1 Loosely coupled Inter-Proxy Communication . . . . . . . . 19
7.1.1 Internet Cache Protocol (ICP) . . . . . . . . . . . . . . 19
7.1.2 Hyper Text Caching Protocol . . . . . . . . . . . . . . . 20
7.1.3 Cache Digest . . . . . . . . . . . . . . . . . . . . . . . 21
7.1.4 Cache Pre-filling . . . . . . . . . . . . . . . . . . . . 22
7.2 Tightly Coupled Inter-Cache Communication . . . . . . . . 22
7.2.1 Cache Array Routing Protocol (CARP) v1.0 . . . . . . . . . 22
8. Network Element Communication . . . . . . . . . . . . . . 23
8.1 Web Cache Control Protocol (WCCP) . . . . . . . . . . . . 23
8.2 Network Element Control Protocol (NECP) . . . . . . . . . 24
8.3 SOCKS . . . . . . . . . . . . . . . . . . . . . . . . . . 25
9. Security Considerations . . . . . . . . . . . . . . . . . 25
9.1 Authentication . . . . . . . . . . . . . . . . . . . . . . 26
9.1.1 Man in the middle attacks . . . . . . . . . . . . . . . . 26
9.1.2 Trusted third party . . . . . . . . . . . . . . . . . . . 26
9.1.3 Authentication based on IP number . . . . . . . . . . . . 26
9.2 Privacy . . . . . . . . . . . . . . . . . . . . . . . . . 26
9.2.1 Trusted third party . . . . . . . . . . . . . . . . . . . 26
9.2.2 Logs and legal implications . . . . . . . . . . . . . . . 27
9.3 Service security . . . . . . . . . . . . . . . . . . . . . 27
9.3.1 Denial of service . . . . . . . . . . . . . . . . . . . . 27
9.3.2 Replay attack . . . . . . . . . . . . . . . . . . . . . . 27
9.3.3 Stupid configuration of proxies . . . . . . . . . . . . . 28
9.3.4 Copyrighted transient copies . . . . . . . . . . . . . . . 28
9.3.5 Application level access . . . . . . . . . . . . . . . . . 28
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . 28
References . . . . . . . . . . . . . . . . . . . . . . . . 28
Authors' Addresses . . . . . . . . . . . . . . . . . . . . 31
Full Copyright Statement . . . . . . . . . . . . . . . . . 321. Introduction ✨
Okay so, ever since the World-Wide Web showed up in 1990, it's gone from a simple client-server thing into a seriously complex distributed architecture 🌈 Honestly, a lot of that glow-up is because of scaling problems that came with exponential growth 📈 Different paradigms and solutions popped up to handle specific needs. Two core infrastructure pieces got us through all that growth: replication and caching 🍓 A lot of the time, web caches and replicated services need to be able to coexist, and that's a whole vibe 🫶
This memo sets the standard terminology and the taxonomy of web replication and caching infrastructure as it's deployed on the Internet today 💅 The main goal is to give everyone a shared understanding and one reference point for this whole application domain ✨
It's also expected that this document will feed into building a standard architectural framework. The point is efficient, reliable, and predictable service on a web that includes both replicas and caches 🦋
Some of the protocols this memo looks at are only specified in company technical white papers or work in progress documents 🙈 Those references are here to show that the protocols exist, that they're being tried out on the Internet today, or to help you understand the tech 💜
There are tons of protocols, both open and proprietary, being used in web replication and caching right now 🔥 Most of the open ones are DNS [8], Cache Digests [21][10], CARP [14], HTTP [1], ICP [2], PAC [12], SOCKS [7], WPAD [13], and WCCP [18][19]. We'll talk about how these protocols work and how they fit into caching and replication environments below 🧚
2. Terminology 💅
Okay, this part is the glossary bit, and it's giving main character energy 👑 The following terminology defines common terms used within the web replication and caching community 🌸 Base terms are taken from the HTTP/1.1 specification [1] wherever possible, and they're included here for reference. First- and second-order derivatives are built from those base terms to help define the relationships in this area 🎀
Terms that are in common usage and which are contrary to definitions in RFC 2616 and this document are highlighted. 🫶
2.1 Base Terms
Okay so most of these terms are borrowed straight from RFC 2616 [1] 💅 I've kept them all here so you've got everything in one spot, bestie 🎀
client (taken from [1]): A program that establishes connections for the purpose of sending requests. 💖
server (taken from [1]): An application program that accepts connections in order to service requests by sending back responses. Any given program may be capable of being both a client and a server, but we only care about the role the program is playing for a particular connection, not what it can do in general. 🧚 Likewise, any server may act as an origin server, proxy, gateway, or tunnel, switching behavior based on the nature of each request. It's giving shapeshifter 🦋
proxy (taken from [1]): An intermediary program that acts as both a server and a client so it can make requests on behalf of other clients. Requests get serviced internally or passed along, with possible translation, to other servers. A proxy MUST implement both the client and server requirements of this specification. ✨ A "transparent proxy" is a proxy that doesn't modify the request or response beyond what's needed for proxy authentication and identification. A "non-transparent proxy" is a proxy that modifies the request or response to add some extra service for the user agent, like group annotation services, media type transformation, protocol reduction, or anonymity filtering. Unless transparent or non-transparent behavior is explicitly stated, the HTTP proxy requirements apply to both kinds of proxies. 🌸
Note: The term "transparent proxy" refers to a semantically transparent proxy as described in [1], not what the caching community usually means by it. We recommend that the term "transparent proxy" is always prefixed to avoid confusion (e.g., "network transparent proxy"). However, see the definition of "interception proxy" below. 🙈
The above condition requiring implementation of both the server and client requirements of HTTP/1.1 is only appropriate for a non-network transparent proxy. 💜
cache (taken from [1]): A program's local store of response messages, plus the subsystem that controls how those messages are stored, retrieved, and deleted. A cache stores cacheable responses so future, equivalent requests get answered faster and use less network bandwidth. Any client or server may include a cache, though a cache cannot be used by a server that is acting as a tunnel. 🍓
Note: The term "cache" used alone often means "caching proxy".
Note: There are additional motivations for caching, for example reducing server load (as a further means to reduce response time).
cacheable (taken from [1]): A response is cacheable if a cache is allowed to store a copy of the response message for use in answering subsequent requests. The rules for working out whether HTTP responses are cacheable are defined in section 13. Even if a resource is cacheable, there may be extra constraints on whether a cache can use the cached copy for a particular request. 🔥
gateway (taken from [1]): A server that acts as an intermediary for some other server. Unlike a proxy, a gateway receives requests as if it were the origin server for the requested resource, so the requesting client may not even realize it's talking to a gateway. 👀
tunnel (taken from [1]): An intermediary program that acts as a blind relay between two connections. Once it's active, a tunnel is not considered a party to the HTTP communication, even though the tunnel may have been started by an HTTP request. It ceases to exist when both ends of the relayed connections are closed. 🦋
replication: "Creating and maintaining a duplicate copy of a database or file system on a different computer, typically a server." - Free Online Dictionary of Computing (FOLDOC) 📚
inbound/outbound (taken from [1]): Inbound and outbound refer to the request and response paths for messages. "Inbound" means "traveling toward the origin server", and "outbound" means "traveling toward the user agent". 🚀
network element: A network device that introduces multiple paths between source and destination, transparent to HTTP. 🧠
2.2 First order derivative terms
These terms are built on top of the base terms above, so they're giving "level two" energy 🎀
origin server (taken from [1]): The server on which a given resource resides or is to be created. 🏠
user agent (taken from [1]): The client that initiates a request. These are often browsers, editors, spiders (web-traversing robots), or other end user tools. 💅
caching proxy: A proxy with a cache, acting as a server to clients and as a client to servers. 🧁
Caching proxies are often called "proxy caches" or simply "caches". The term "proxy" is also frequently misused when referring to caching proxies. 🙄
surrogate: A gateway co-located with an origin server, or at a different point in the network, that has been delegated the authority to operate on behalf of one or more origin servers, and that typically works in close co-operation with them. Responses are typically delivered from an internal cache. ✨
Surrogates may derive cache entries from the origin server or from another of the origin server's delegates. In some cases a surrogate may tunnel such requests. 🌈
Where close co-operation between origin servers and surrogates exists, this enables modifications of some protocol requirements, including the Cache-Control directives in [1]. Such modifications have yet to be fully specified. 🤷♀️
Devices commonly known as "reverse proxies" and "(origin) server accelerators" are both more properly defined as surrogates. 👑
reverse proxy: See "surrogate". 🫶
server accelerator: See "surrogate". 🫶
2.3 Second order derivatives
These terms build further on the first order derivatives 🌸
master origin server: An origin server on which the definitive version of a resource resides. 👑
replica origin server: An origin server holding a replica of a resource, which may act as an authoritative reference for client requests. 🪞
content consumer: The user or system that initiates inbound requests through the use of a user agent. 🍿
browser: A special instance of a user agent that acts as a content presentation device for content consumers. 🌐
2.4 Topological terms
These definitions were added to describe caching device topology, so buckle up 🎢
user agent cache: The cache within the user agent program. 💜
local caching proxy: The caching proxy that a user agent connects to. 📍
intermediate caching proxy: Seen from the content consumer's view, all caches taking part in the caching mesh that aren't the user agent's local caching proxy. 🕸️
cache server: A server for requests made by local and intermediate caching proxies, which does not act as a proxy. 🖥️
cache array: A cluster of caching proxies that act logically as one service and partition the resource name space across the array. Also known as "diffused array" or "cache cluster". 🧩
caching mesh: A loosely coupled set of co-operating proxy and (optionally) caching servers, or clusters, that act independently but share cacheable content between themselves using inter-cache communication protocols. 🌿
2.5 Automatic use of proxies
Network admins may want to force or make it easier for clients to use proxies 🫣 They can set this up inside the network itself or inside automatic systems in user agents, so the content consumer doesn't have to know any of the configuration stuff ever happened ✨ The terms that describe these setups are below.
automatic user-agent proxy configuration: The technique of discovering that one or more proxies are available and automatically configuring the user agent to use them. Using a proxy is transparent to the content consumer but not to the user agent. The term "automatic proxy configuration" is also used in this sense. 🤖
traffic interception: The process of using a network element to examine network traffic and decide whether it should be redirected. 👀
traffic redirection: Redirecting client requests from a network element that's doing traffic interception over to a proxy. It's used to deploy (caching) proxies without having to manually reconfigure individual user agents, or to force the use of a proxy where that use wouldn't otherwise happen. 🔀
interception proxy (a.k.a. "transparent proxy", "transparent cache"): The term "transparent proxy" has been used in the caching community for proxies that work with zero configuration in the user agent. That use is somewhat transparent to user agents. But because of discrepancies with [1] (see the definition of "proxy" above) and objections to the word "transparent", we introduce the term "interception proxy" for proxies that receive redirected traffic flows from network elements doing traffic interception. 🥹
Interception proxies receive inbound traffic flows through the process of traffic redirection. (Such proxies are deployed by network administrators to facilitate or require the use of appropriate services the proxy offers.) Problems associated with deploying interception proxies are described in the document "Known HTTP Proxy/Caching Problems" [23]. Using interception proxies requires zero configuration of the user agent, which acts as though it's communicating directly with an origin server. 🎀
3. Distributed System Relationships
This section identifies the relationships that exist in a distributed replication and caching environment. Once those relationships are defined, later sections describe the communication protocols used in each one 🗺️
3.1 Replication Relationships
The following sections describe relationships between clients and replicas, and between replicas themselves 💞
3.1.1 Client to Replica
A client may communicate with one or more replica origin servers, as well as with master origin servers. (If there are no replica servers, the client interacts directly with the origin server, which is the normal case.) 🌟
------------------ ----------------- ------------------
| Replica Origin | | Master Origin | | Replica Origin |
| Server | | Server | | Server |
------------------ ----------------- ------------------
\ | /
\ | /
-----------------------------------------
| Client to
----------------- Replica Server
| Client |
-----------------Protocols that let the client use one of the replicas can be found in Section 4. 🦋
3.1.2 Inter-Replica 🦋
okay so this one is the relationship between the master origin server(s) and the replica origin servers 🫶 their whole job is replicating the data sets that clients are reading in the relationship from Section 3.1.1. the way it's set up is a chain, with replicas on each end and the master in the middle ✨
------------------ ----------------- ------------------
| Replica Origin |-----| Master Origin |-----| Replica Origin |
| Server | | Server | | Server |
------------------ ----------------- ------------------the protocols for this one are in Section 5 💅
3.2 Proxy Relationships 🌸
ngl this is where it gets busy 😭 there are a bunch of ways (caching) proxies and cache servers talk to each other, and also to user agents.
3.2.1 Client to Non-Interception Proxy 🎀
a client can talk to zero or more proxies for some or all of its requests 🧚 if the communication ends up using no proxy at all, then the relationship is just client and (replica) origin server (see Section 3.1.1). main character energy for the origin server 👑
----------------- ----------------- -----------------
| Local | | Local | | Local |
| Proxy | | Proxy | | Proxy |
----------------- ----------------- -----------------
\ | /
\ | /
-----------------------------------------
|
-----------------
| Client |
-----------------also, a user agent can interact with an extra server, one that's run on behalf of a proxy, just so it can do automatic user agent proxy configuration 🙈
schemes and protocols used in these relationships are in Section 6 🦋
3.2.2 Client to Surrogate to Origin Server 💖
so a client can talk to zero or more surrogates for requests meant for one or more origin servers 💜 if there's no surrogate, the client just talks straight to an origin server. if there is a surrogate, the client acts like it's talking to an origin server (the surrogate is giving it "origin server" vibes 💅). the surrogate either answers from its own internal cache, or it acts as a gateway or tunnel to the real origin server 🔥
-------------- -------------- --------------
| Origin | | Origin | | Origin |
| Server | | Server | | Server |
-------------- -------------- --------------
\ | /
\ | /
-----------------
| Surrogate |
| |
-----------------
|
|
------------
| Client |
------------3.2.3 Inter-Proxy 🧚
inter-proxy relationships come in two flavours: meshes (loosely coupled) and clusters (tightly coupled) 🍓
3.2.3.1 (Caching) Proxy Meshes 🦋
inside a loosely coupled mesh of (caching) proxies, communication can happen at the same level between peers, and also with one or more parents 🫶
--------------------- ---------------------
-----------| Intermediate | | Intermediate |
| | Caching Proxy (D) | | Caching Proxy (E) |
|(peer) --------------------- ---------------------
-------------- | (parent) / (parent)
| Cache | | ------/
| Server (C) | | /
-------------- | /
(peer) | ----------------- ---------------------
-------------| Local Caching |-------| Intermediate |
| Proxy (A) | (peer)| Caching Proxy (B) |
----------------- ---------------------
|
|
----------
| Client |
----------Client included for illustration purposes only
so an inbound request can get routed to one of a bunch of intermediate (caching) proxies, depending on whether that parent is the better fit for resolving the request 🧠
for example, in the figure above, Cache Server C and Intermediate Caching Proxy B are peers of Local Caching Proxy A 🎀 they're only used when the resource A is asking for already exists on either B or C. Intermediate Caching Proxies D & E are parents of A, and A gets to choose which one to use for a particular request 👑
the A & B relationship only makes sense in a caching environment 💅 the A & D and A & E relationships also work fine when D or E are non-caching proxies 🌈
protocols used in these relationships are in Section 7.1 ✨
3.2.3.2 (Caching) Proxy Arrays 🌈
if a user agent would normally have a relationship with a proxy, it might instead have a relationship with an array of proxies arranged in a tightly coupled mesh 😭
----------------------
---------------------- |
--------------------- | |
| (Caching) Proxy | |-----
| Array |----- ^ ^
--------------------- ^ ^ | |
^ ^ | |--- |
| |----- |
--------------------------protocols used in this relationship are in Section 7.2 🫶
3.2.4 Network Element to Caching Proxy 💀
a network element doing traffic interception can choose to redirect a client's requests to a specific proxy inside an array 🙈 (it can also choose not to redirect the traffic, and then the relationship is client and (replica) origin server, see Section 3.1.1 💖)
----------------- ----------------- -----------------
| Caching Proxy | | Caching Proxy | | Caching Proxy |
| Array | | Array | | Array |
----------------- ----------------- -----------------
\ | /
-----------------------------------------
|
--------------
| Network |
| Element |
--------------
|
///
|
------------
| Client |
------------the interception proxy might be directly in-line with the traffic flow, which means the intercepting network element and interception proxy are part of the same hardware system 🔥 or it can be out-of-path, where the intercepting network element has to redirect traffic to another network segment. in that second case, communication protocols let the intercepting network element stop and start redirecting traffic when the interception proxy goes (un)available 😭 details are in Section 8 🎀
4. Replica Selection 👑
this section is about the schemes and protocols used when clients and replica origin web servers cooperate and communicate 🧚 the dream is to find an optimal replica origin server for clients to talk to. "optimal" is a policy-based decision, usually based on proximity, but it can also depend on other stuff like load ✨
4.1 Navigation Hyperlinks 🔗
Best known reference: This memo.
Description: the simplest client-to-replica communication mechanism 🎀 it uses hyperlink URIs embedded in web pages that point to the individual replica origin servers. the content consumer manually picks the link to whichever replica origin server they want to use 💅
Security: relies on the protocol security tied to the appropriate URI scheme 🔐
Deployment: probably the most commonly deployed client-to-replica communication mechanism 🌸 ubiquitous interoperability with humans, obviously 🫶
Submitter: Document editors.
4.2 Replica HTTP Redirection 🦋
Best known reference: This memo.
Description: a simple, super common way to connect clients with replica origin servers is HTTP redirection 🚀 clients get redirected to an optimal replica origin server using HTTP [1] response codes, like 302 "Found" or 307 "Temporary Redirect". a client starts HTTP communication with one of the replica origin servers, and that first server can either accept the service or redirect the client again 🌈 for more on HTTP response codes, check section 10.3 in HTTP/1.1 [1] 📚
Security: relies entirely on HTTP security 🔐
Deployment: observed at a number of large web sites 💜 the extent of usage on the Internet is unknown 🤷♀️
Submitter: Document editors.
4.3 DNS Redirection 🧚
Best known references:
- RFC 1794 DNS Support for Load Balancing Proximity [8]
- This memo
Description: the Domain Name Service (DNS) is a more sophisticated client-to-replica mechanism 🍓 DNS servers sort resolved IP addresses based on quality of service policies. when a client resolves the name of an origin server, the enhanced DNS server sorts the available IP addresses of the replica origin servers, starting with the most optimal replica and ending with the least optimal one ✨
Security: relies entirely on DNS security, plus any other protocols used in determining the sort order 🔐
Deployment: observed at a number of large web sites and large ISP web hosted services 🌸 the extent of usage on the Internet is unknown, but it's believed to be increasing 📈 iykyk 😏
Submitter: Document editors.
5. Inter-Replica Communication 🍓
this section describes the cooperation and communication between master- and replica- origin servers 🫶 it's used for replicating data sets between origin servers.
5.1 Batch Driven Replication 🔥
Best known reference: This memo.
Description: the replica origin server that needs updating starts the communication with a master origin server 💅 the communication happens at intervals based on queued transactions that are scheduled for deferred processing ⏰ the scheduling policies vary, but they usually recur at a specified time. once the communication is established, data sets get copied to the initiating replica origin server 💖
Security: relies on the protocol used to transfer the data set 🔐 FTP [4] and RDIST are the most commonly observed protocols 👑
Deployment: very common for synchronizing mirror sites on the Internet 🪞
Submitter: Document editors.
5.2 Demand Driven Replication 💖
Best known reference: This memo.
Description: Okay so here's the vibe 🫶: replica origin servers only grab content when clients actually ask for it. If a client requests a resource that isn't in the data set of the replica origin server/surrogate, the server tries to sort it out by fetching the resource from the master origin server and handing it back to the client who asked 💅.
Security: This relies on whatever protocol is used to transfer the resources. The most common ones spotted are FTP [4], Gopher [5], HTTP [1] and ICP [2] 🔥.
Deployment: Spotted at several large websites. How much it's used across the Internet overall? Unknown 🙈.
Submitter: Document editors ✨.
5.3 Synchronized Replication 🎀
Best known reference: This memo.
Description: Replicated origin servers work together using synchronized strategies and specialized replica protocols to keep their replica data sets coherent 🧚. The strategies range from tightly coherent (a few minutes) to loosely coherent (a few or more hours) ⏰. Updates happen between replicas based on the synchronization time constraints of the coherency model being used, and they're generally just deltas 🦋.
Security: All of the known protocols use strong cryptographic key exchange methods, which are either based on the Kerberos shared secret model or the public/private key RSA model 🔐.
Deployment: Spotted at a few sites, mostly university campuses 🎓.
Submitter: Document editors 💜.
Note: The editors know about at least two open source protocols, AFS and CODA, plus the proprietary NRS protocol from Novell 🌈.
6. User Agent to Proxy Configuration 🌸
This section is all about how user agents and proxies configure, cooperate and talk to each other 💬.
6.1 Manual Proxy Configuration ✍️
Best known reference: This memo.
Description: Each user must configure her user agent by supplying information about proxied protocols and local policies.
Security: The potential for doing wrong is high 😬, since each user individually sets their own preferences.
Deployment: Widely deployed and used in all current browsers 💻. Most browsers also support additional options.
Submitter: Document editors ✨.
6.2 Proxy Auto Configuration (PAC) 🧠
Best known reference: "Navigator Proxy Auto-Config File Format" [12]
Description: A JavaScript script retrieved from a web server gets executed for each URL accessed, to work out which proxy (if any) should be used for the resource. User agents must be configured to request this script upon startup. There's no bootstrap mechanism, so manual configuration is necessary 🙃.
Despite the manual configuration, the process of proxy configuration gets simpler, because it's centralized within a script at a single location 🎯.
Security: A common policy per organization is possible, but it still requires initial manual configuration. PAC is better than "manual proxy configuration" since PAC administrators can update the proxy configuration without further user intervention 👑.
Interoperability of PAC files isn't high, since different browsers have slightly different interpretations of the same script, which can lead to undesired effects 💀.
Deployment: Implemented in Netscape Navigator and Microsoft Internet Explorer 🧃.
Submitter: Document editors.
6.3 Cache Array Routing Protocol (CARP) v1.0 🗺️
Best known references:
- "Cache Array Routing Protocol" [14] (work in progress)
- "Cache Array Routing Protocol (CARP) v1.0 Specifications" [15]
- "Cache Array Routing Protocol and Microsoft Proxy Server 2.0" [16]
Description: User agents may use CARP directly as a hash function based proxy selection mechanism. They need to be configured with the location of the cluster information.
Security: Security considerations are not covered in the specification works in progress. Yes, this is a serious gap, so keep it in mind 🚨.
Deployment: Implemented in Microsoft Proxy Server and Squid. Implemented in user agents via PAC scripts.
Submitter: Document editors.
6.4 Web Proxy Auto-Discovery Protocol (WPAD) 🔍
Best known reference: "The Web Proxy Auto-Discovery Protocol" [13] (work in progress)
Description: WPAD uses a collection of pre-existing Internet resource discovery mechanisms to perform web proxy auto-discovery.
The only goal of WPAD is to locate the PAC URL [12]. WPAD does not specify which proxies will be used. WPAD supplies the PAC URL, and then the PAC script operates as defined above to choose proxies per resource request ✨.
The WPAD protocol specifies the following:
- how to use each mechanism for the specific purpose of web proxy auto-discovery
- the order in which the mechanisms should be performed
- the minimal set of mechanisms which must be attempted by a WPAD compliant user agent
The resource discovery mechanisms utilized by WPAD are as follows:
- Dynamic Host Configuration Protocol DHCP
- Service Location Protocol SLP
- "Well Known Aliases" using DNS A records
- DNS SRV records
- "service: URLs" in DNS TXT records
Security: Relies upon DNS and HTTP security 🔒.
Deployment: Implemented in some user agents and caching proxy servers. There are more than two independent implementations 🎉.
Submitter: Josh Cohen 💜.
7. Inter-Proxy Communication 🦄
7.1 Loosely coupled Inter-Proxy Communication 🤝
This section describes the cooperation and communication between caching proxies.
7.1.1 Internet Cache Protocol (ICP) 👀
Best known reference: RFC 2186 Internet Cache Protocol (ICP), version 2 [2]
Description: ICP is used by proxies to query other (caching) proxies about web resources, to see if the requested resource is present on the other system.
ICP uses UDP. Since UDP is an uncorrected network transport protocol, an estimate of network congestion and availability may be calculated by ICP loss. This rudimentary loss measurement provides, together with round trip times, a load balancing method for caches ⚖️.
Security: See RFC 2187 [3]
ICP does not convey information about HTTP headers associated with resources. HTTP headers may include access control and cache directives. Since proxies ask for the availability of resources, and subsequently retrieve them using HTTP, false cache hits may occur. An object present in a cache but not accessible to a sibling is one example 🙈.
ICP suffers from all the security problems of UDP 💀.
Deployment: Widely deployed. Most current caching proxy implementations support ICP in some form 🌍.
Submitter: Document editors.
See also: "Internet Cache Protocol Extension" [17] (work in progress)
7.1.2 Hyper Text Caching Protocol 🕵️
Best known reference: RFC 2756 Hyper Text Caching Protocol (HTCP/0.0) [9]
Description: HTCP is a protocol for discovering HTTP caching proxies and cached data, managing sets of HTTP caching proxies, and monitoring cache activity.
HTCP requests include HTTP header material, while ICPv2 does not. That lets HTCP replies describe more accurately the behaviour that would occur as a result of a subsequent HTTP request for the same resource 🧐.
Security: Optionally uses HMAC-MD5 [11] shared secret authentication. The protocol is subject to attack if authentication is not used ⚠️.
Deployment: Implemented in Squid and the "Web Gateway Interceptor" 🦋.
Submitter: Document editors.
7.1.3 Cache Digest 🙌
Best known references:
- "Cache Digest Specification - version 5" [21]
- "Summary Cache: A Scalable Wide-Area Web Cache Sharing Protocol" [10] (see note)
Description: Cache Digests are a response to the problems of latency and congestion associated with previous inter-cache communication mechanisms, such as the Internet Cache Protocol (ICP) [2] and the Hyper Text Cache Protocol [9]. Unlike these protocols, Cache Digests support peering between caching proxies and cache servers without a request-response exchange taking place for each inbound request. Instead, a summary of the contents in cache (the Digest) is fetched by other systems that peer with it. Using Cache Digests, it's possible to determine with a relatively high degree of accuracy whether a given resource is cached by a particular system 🎯.
Cache Digests are both an exchange protocol and a data format 📦.
Security: If the contents of a Digest are sensitive, they should be protected. Any methods which would normally be applied to secure an HTTP connection can be applied to Cache Digests.
A 'Trojan horse' attack is currently possible in a mesh. System A can build a fake peer Digest for system B and serve it to B's peers if requested. This way A can direct traffic toward/from B. The impact of this problem is minimized by the 'pull' model of transferring Cache Digests from one system to another 😱.
Cache Digests provide knowledge about peer cache content on a URL level. Hence, they do not dictate a particular level of policy management, and they can be used to implement various policies on any level (user, organization, etc.) 🧩.
Deployment: Cache Digests are supported in Squid. Cache Meshes: NLANR Mesh; TF-CACHE Mesh (European Academic networks.
Submitter: Alex Rousskov for [21], Pei Cao for [10].
Note: The technology of Summary Cache [10] is patent pending by the University of Wisconsin-Madison.
7.1.4 Cache Pre-filling 📡
Best known reference: "Pre-filling a cache - A satellite overview" [20] (work in progress)
Description: Cache pre-filling is a push-caching implementation. It's especially well suited to IP-multicast networks, because it allows preselected resources to be inserted simultaneously into caches within the targeted multicast group. Different implementations already exist, especially in satellite contexts. However, there is still no standard for this kind of push-caching, and vendors propose solutions either based on dedicated equipment or on public domain caches extended with a pre-filling module.
Security: Relies on the inter-cache protocols being employed.
Deployment: Observed in two commercial content distribution service providers.
Submitter: Ivan Lovric.
7.2 Tightly Coupled Inter-Cache Communication 🔥
7.2.1 Cache Array Routing Protocol (CARP) v1.0 🎀
Also see Section 6.3 💅
Best known references:
- "Cache Array Routing Protocol" [14] (work in progress)
- "Cache Array Routing Protocol (CARP) v1.0 Specifications" [15]
- "Cache Array Routing Protocol and Microsoft Proxy Server 2.0" [16]
Description: Okay bestie, so CARP is a hashing function that splits up URL-space across a cluster of proxies 🧚 It also defines a Proxy Array Membership Table and ways to download that info. Iconic.
A user agent that implements CARP v1.0 can allocate requests for URLs and intelligently route them to any member of the Proxy Array. Because requests get sorted through these proxies, duplicated cache contents get eliminated, and global cache hit rates may be improved. The way that works is literally giving efficiency 📈✨
Security: Security considerations are not covered in the specification works in progress. 🙈
Deployment: Implemented in caching proxy servers. More than two independent implementations. 🔥
Submitter: Document editors.
8. Network Element Communication 🌸
This section describes the cooperation and communication between proxies and network elements. Examples of such network elements include routers and switches. It's generally used for deploying interception proxies and/or diffused arrays. Main character energy 👑
8.1 Web Cache Control Protocol (WCCP)
Best known references:
- "Web Cache Control Protocol" [18][19] (work in progress)
Note: The name used for this protocol varies. It's sometimes referred to as the "Web Cache Coordination Protocol", but frequently just "WCCP" to avoid confusion. Iykyk 🙄
Description: WCCP V1 runs between a router functioning as a redirecting network element and out-of-path interception proxies. The protocol lets one or more proxies register with a single router to receive redirected traffic. It also lets one of the proxies, the designated proxy, tell the router how redirected traffic gets distributed across the array. Lowkey a whole power move 💜
WCCP V2 additionally runs between multiple routers and the proxies. ✨
Security: WCCP V1 has no security features. 💀 WCCP V2 provides optional authentication of protocol packets.
Deployment:
- Network elements: WCCP is deployed on a wide range of Cisco routers.
- Caching proxies: WCCP is deployed on a number of vendors' caching proxies.
Submitter: David Forster, Document editors.
8.2 Network Element Control Protocol (NECP)
Best known reference:
- "NECP: The Network Element Control Protocol" [22] (work in progress)
Description: NECP gives network elements methods to learn about server capabilities, availability, and hints about which flows can and cannot be serviced. This lets network elements do load balancing across a farm of servers, redirect to interception proxies, and cut-through flows that the farm can't serve. It's giving organization 🧚🎀
Security: It optionally uses HMAC-SHA-1 [11] shared secret authentication along with complex sequence numbers to provide moderately strong security. The protocol is subject to attack if authentication is not used. 😭
Deployment: Unknown at present. Several network element and caching proxy vendors have expressed intent to implement the protocol. 🦋
Submitter: Gary Tomlinson
8.3 SOCKS
Best known reference:
- RFC 1928 SOCKS Protocol Version 5 [7]
Description: SOCKS is primarily used as a caching proxy to firewall protocol. Firewalls don't conform to the narrowly defined network element definition above, but they are an integral part of the network infrastructure, so it still counts 💅 When used with a firewall, SOCKS provides an authenticated tunnel between the caching proxy and the firewall. 🫶
Security: An extensive framework provides for multiple authentication methods. Currently, SSL, CHAP, DES, and 3DES are known to be available. 🔥
Deployment: SOCKS is widely deployed in the Internet. 🌈
Submitter: Document editors.
9. Security Considerations 🚨
Bestie, this whole section is serious. Read it all, no skipping!! 😭
This document provides a taxonomy for web caching and replication. Recommended practice, architecture, and protocols are not described in detail.
By definition, replication and caching involve copying resources. There are legal implications of making and keeping transient or permanent copies. These are not covered here. ⚖️
Information on the security of each protocol referred to by this memo is provided in the preceding sections and in their accompanying documentation. HTTP security is discussed in section 15 of RFC 2616 [1], the HTTP/1.1 specification, and to a lesser extent in RFC 1945 [6], the HTTP/1.0 specification. RFC 2616 contains security considerations for HTTP proxies.
Caching proxies have the same security issues as other application level proxies. Application level proxies are not covered in these security considerations. IP number based authentication is problematic when a proxy is involved in the communications. Details are not discussed here. 💔
9.1 Authentication 🔐
Requests for web resources, and responses to such requests, may be directed to replicas and/or may flow through intermediate proxies. The integrity of communication needs to be preserved to ensure protection from both loss of access and unintended change. Not okay to mess with that 😤
9.1.1 Man in the middle attacks 🕵️♀️
HTTP proxies are men-in-the-middle, which makes them the perfect place for a man-in-the-middle attack. A discussion of this is found in section 15 of RFC 2616 [1]. Scary, but real 😭
9.1.2 Trusted third party
A proxy must either be trusted to act on behalf of the origin server and/or client, or it must act as a tunnel. When presenting cached objects to clients, the clients need to trust the caching proxy to act on behalf of the origin server. 🫡
A replica may get accreditation from the origin server. ✨
9.1.3 Authentication based on IP number
Authentication based on the client's IP number is problematic when connecting through a proxy, since the authenticating device only has access to the proxy's IP number. One (problematic) solution to this is for the proxy to spoof the client's IP number for inbound requests. Yikes, no thank you 🙅♀️
Authentication based on IP number assumes that the end-to-end properties of the Internet are preserved. This is typically not the case for environments containing interception proxies. 🙈
9.2 Privacy 🫣
9.2.1 Trusted third party
When using a replication service, one must trust both the replica origin server and the replica selection system. 🤝
Redirection of traffic, either by automated replica selection methods or within proxies, may introduce third parties that the end user and/or origin server must trust. With interception proxies, those third parties are often unknown to both endpoints of the communication. Unknown third parties may have security implications. 👀
Both proxies and replica selection services may have access to aggregated access information. A proxy typically knows about accesses by each client using it. That's more sensitive than the information held by a single origin server. 😬
9.2.2 Logs and legal implications 📜
Logs from proxies should be kept secure, since they provide information about users and their patterns of behaviour. A proxy's log is even more sensitive than a web server log, because every request from the user population goes through the proxy. Logs from replica origin servers may need to be amalgamated to get aggregated statistics from a service, and transporting logs across borders may have legal implications. Log handling is restricted by law in some countries. 🌍
Requirements for object security and privacy are the same in a web replication and caching system as they are in the Internet at large. The only reliable solution is strong cryptography. End-to-end encryption frequently makes resources uncacheable, as in the case of SSL encrypted web sessions. 🔒💖
9.3 Service security 🛡️
9.3.1 Denial of service
Any redirection of traffic is susceptible to denial of service attacks at the redirect point, and both proxies and replica selection services may redirect traffic. 🚫
By attacking a proxy, access to all servers may be denied for a large set of clients. Chaos 💀
It has been argued that introducing an interception proxy is a denial of service attack, since the end-to-end nature of the Internet is destroyed without the content consumer's knowledge. Honestly, that's a whole mood 😤
9.3.2 Replay attack
A caching proxy is, by definition, a replay attack. 🔁
9.3.3 Stupid configuration of proxies
It is quite easy to have a stupid configuration that will harm service for content consumers. This is the most common security problem with proxies. Bestie, please double-check your configs 🙏
9.3.4 Copyrighted transient copies
The legislative forces of the world are considering whether transient copies, like those kept in replication and caching systems, should be legal. The legal implications of replication and caching are subject to local law. ⚖️
Caching proxies need to preserve the protocol output, including headers. Replication services need to preserve the source of the objects. 🎀
9.3.5 Application level access
Caching proxies are application level components in the traffic flow path. They may give intruders access to information that was previously only available at the network level in a proxy-free world. Some network level equipment may have required physical access to get sensitive information. Introducing application level components may require additional system security. 🔐✨
10. Acknowledgements 💜
The editors would like to thank the following for their assistance: David Forster, Alex Rousskov, Josh Cohen, John Martin, John Dilley, Ivan Lovric, Joe Touch, Henrik Nordstrom, Patrick McManus, Duane Wessels, Wojtek Sylwestrzak, Ted Hardie, Misha Rabinovich, Larry Masinter, Keith Moore, Roy Fielding, Patrik Faltstrom, Hilarie Orman, Mark Nottingham and Oskar Batuner. 🫶🌈
References 📚✨
okay so these are all the receipts 🧾 every single doc the spec leans on, bestie. all 23 are listed below, numbering and details exactly as written so nobody gets lost 💅
- [1] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P. and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. 🌸
- [2] Wessels, D. and K. Claffy, "Internet Cache Protocol (ICP), Version 2", RFC 2186, September 1997. 🧚
- [3] Wessels, D. and K. Claffy, "Application of Internet Cache Protocol (ICP), Version 2", RFC 2187, September 1997. 🦋
- [4] Postel, J. and J. Reynolds, "File Transfer Protocol (FTP)", STD 9, RFC 959, October 1985. 📁
- [5] Anklesaria, F., McCahill, M., Lindner, P., Johnson, D., Torrey, D. and B. Alberti, "The Internet Gopher Protocol", RFC 1436, March 1993. 🐿️
- [6] Berners-Lee, T., Fielding, R. and H. Frystyk, "Hypertext Transfer Protocol -- HTTP/1.0", RFC 1945, May 1996. 🌈
- [7] Leech, M., Ganis, M., Lee, Y., Kuris, R., Koblas, D. and L. Jones, "SOCKS Protocol Version 5", RFC 1928, March 1996. 🧦
- [8] Brisco, T., "DNS Support for Load Balancing", RFC 1794, April 1995. ⚖️
- [9] Vixie, P. and D. Wessels, "Hyper Text Caching Protocol (HTCP/0.0)", RFC 2756, January 2000. 💜
- [10] Fan, L., Cao, P., Almeida, J. and A. Broder, "Summary Cache: A Scalable Wide-Area Web Cache Sharing Protocol", Proceedings of ACM SIGCOMM'98 pp. 254-265, September 1998. 📄
- [11] Krawczyk, H., Bellare, M. and R. Canetti, "HMAC: Keyed-Hashing for Message Authentication", RFC 2104, February 1997. 🔐
- [12] Netscape, Inc., "Navigator Proxy Auto-Config File Format", March 1996,
<URL:http://www.netscape.com/eng/mozilla/2.0/relnotes/demo/proxy-live.html>. 🧭 - [13] Gauthier, P., Cohen, J., Dunsmuir, M. and C. Perkins, "The Web Proxy Auto-Discovery Protocol", Work in Progress. 🚧
- [14] Valloppillil, V. and K. Ross, "Cache Array Routing Protocol", Work in Progress. 🚧
- [15] Microsoft Corporation, "Cache Array Routing Protocol (CARP) v1.0 Specifications, Technical Whitepaper", August 1999,
<URL:http://www.microsoft.com/Proxy/Guide/carpspec.asp>. 🪟 - [16] Microsoft Corporation, "Cache Array Routing Protocol and Microsoft Proxy Server 2.0, Technical White Paper", August 1998,
<URL:http://www.microsoft.com/proxy/documents/CarpWP.exe>. 🪟 - [17] Lovric, I., "Internet Cache Protocol Extension", Work in Progress. 🚧
- [18] Cieslak, M. and D. Forster, "Cisco Web Cache Coordination Protocol V1.0", Work in Progress. 🚧
- [19] Cieslak, M., Forster, D., Tiwana, G. and R. Wilson, "Cisco Web Cache Coordination Protocol V2.0", Work in Progress. 🚧
- [20] Goutard, C., Lovric, I. and E. Maschio-Esposito, "Pre-filling a cache - A satellite overview", Work in Progress. 🛰️
- [21] Hamilton, M., Rousskov, A. and D. Wessels, "Cache Digest specification - version 5", December 1998,
<URL:http://www.squid-cache.org/CacheDigest/cache-digest-v5.txt>. 🍪 - [22] Cerpa, A., Elson, J., Beheshti, H., Chankhunthod, A., Danzig, P., Jalan, R., Neerdaels, C., Shroeder, T. and G. Tomlinson, "NECP: The Network Element Control Protocol", Work in Progress. 🚧
- [23] Cooper, I. and J. Dilley, "Known HTTP Proxy/Caching Problems", Work in Progress. 🚧
Authors' Addresses 🖊️💌
these are the people who wrote it, with their contact details exactly as published 🫶
Ian Cooper
Equinix, Inc.
2450 Bayshore Parkway
Mountain View, CA 94043
USA
Phone: +1 650 316 6065
EMail: icooper@equinix.comIngrid Melve
UNINETT
Tempeveien 22
Trondheim N-7465
Norway
Phone: +47 73 55 79 07
EMail: Ingrid.Melve@uninett.noGary Tomlinson
CacheFlow Inc.
12034 134th Ct. NE, Suite 201
Redmond, WA 98052
USA
Phone: +1 425 820 3009
EMail: gary.tomlinson@cacheflow.comFull Copyright Statement ⚖️🔒
this part is legal stuff, so I've kept the wording exactly as it is 🙈
Copyright (C) The Internet Society (2001). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement 🫶🌷
funding for the RFC Editor function is currently coming from the Internet Society, so a big thank you to them 💖
📦 where this page comes from
- ietf: rfc3040 2001-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms