RFC 3507
Internet Content Adaptation Protocol (ICAP)
RFC 3507: Internet Content Adaptation Protocol (ICAP) ✨
okay bestie, let's get into it 💅 this one is RFC 3507, and the title is Internet Content Adaptation Protocol (ICAP) 🎀. The author header lists J. Elson and A. Cerpa, both from UCLA 🧚. It's Informational (not a standard), dated April 2003, and the header lists no obsoletes or updates, so it doesn't replace or change any other RFC 🦋. It's published in the Network Working Group stream, which is giving "open collab, no gatekeeping" vibes 🌈.
Status of this Memo
ngl this part is so chill 🫶 the memo is for the Internet community as information. It does not specify an Internet standard of any kind, and distribution is unlimited, so share it freely 💜.
Copyright Notice
Copyright is held by The Internet Society (2003), and all rights are reserved 🌸.
IESG Note
okay the drama here is real 😭 the Open Pluggable Services (OPES) working group was chartered to write a standards track spec for a protocol meant to do the same kinds of jobs as ICAP. BUT ICAP was already everywhere, so the IESG decided it made more sense to write down how people were already using it, and publish ICAP as an informational document 🍓.
The IESG also flagged that ICAP was built before RFC 3238 came out, so it doesn't cover the architectural and policy issues that RFC describes 👑. Main character of a timeline, honestly 💀.
Abstract
so what IS ICAP? ✨ It's the Internet Content Adaption Protocol (yes, that's how it's spelled in the original, fun quirk 🙈). It's a protocol for simple, object-based content vectoring for HTTP services 🦋. Think of it as a lightweight way to run a "remote procedure call" on HTTP messages 💖.
Here's the flow 🌸: an ICAP client sends HTTP messages to an ICAP server, which does some kind of transformation or other processing (that's the "adaptation") and then sends responses back to the client, usually with modified messages 🎀. The adapted messages are typically either HTTP requests or HTTP responses ⭐.
Table of Contents
Table of Contents
1. Introduction............................................3
2. Terminology.............................................5
3. ICAP Overall Operation..................................8
3.1 Request Modification..............................8
3.2 Response Modification............................10
4. Protocol Semantics.....................................11
4.1 General Operation................................11
4.2 ICAP URIs........................................11
4.3 ICAP Headers.....................................12
4.3.1 Headers Common to Requests and
Responses................................12
4.3.2 Request Headers..........................13
4.3.3 Response Headers.........................14
4.3.4 ICAP-Related Headers in HTTP
Messages.................................15
4.4 ICAP Bodies: Encapsulation of HTTP
Messages.........................................16
4.4.1 Expected Encapsulated Sections...........16
4.4.2 Encapsulated HTTP Headers................18
4.5 Message Preview..................................18
4.6 "204 No Content" Responses outside of
Previews.........................................22
4.7 ISTag Response Header............................22
4.8 Request Modification Mode........................23
4.8.1 Request..................................23
4.8.2 Response.................................24
4.8.3 Examples.................................24
4.9 Response Modification Mode.......................27
4.9.1 Request..................................27
4.9.2 Response.................................27
4.9.3 Examples.................................28
4.10 OPTIONS Method...................................29
4.10.1 OPTIONS request..........................29
4.10.2 OPTIONS response.........................30
4.10.3 OPTIONS examples.........................33
5. Caching................................................33
6. Implementation Notes...................................34
6.1 Vectoring Points.................................34
6.2 Application Level Errors.........................35
6.3 Use of Chunked Transfer-Encoding.................37
6.4 Distinct URIs for Distinct Services..............37
7. Security Considerations................................37
7.1 Authentication...................................37
7.2 Encryption.......................................38
7.3 Service Validation...............................38
8. Motivations and Design Alternatives....................39
8.1 To Be HTTP, or Not to Be.........................39
8.2 Mandatory Use of Chunking........................39
8.3 Use of the null-body directive in the
Encapsulated header..............................40
9. References.............................................40
10. Contributors...........................................41
Appendix A BNF Grammar for ICAP Messages..................45
Authors' Addresses..........................................48
Full Copyright Statement....................................49that's the whole map of the doc 🗺️ sections 1 through 10, the BNF grammar in Appendix A, and then the addresses and copyright at the very end 💜. part 1 is done, so more parts are coming, bestie 🧚✨
1. Introduction ✨
okay so bestie, the internet keeps getting bigger 📈 and popular web servers are being asked to hand out content to hundreds of millions of users, all on faster and faster connections 🚀. the old model where one big centralized server handles every single part of every client's request is lowkey reaching the end of its useful life 💀
so the move is toward setups that scale better using replication, distribution, and caching 🌸. on the content provider side, replication and load-balancing let the work of client requests get spread across a myriad of servers 🦋. providers are also deploying geographically diverse content distribution networks that bring origin-servers closer to the "edge" of the network, where clients actually live 🌈. these networks of distributed origin-servers (also called "surrogates") let the content provider distribute their content while still keeping control over the integrity of that content 🫶. the distributed nature of this setup, plus how close a surrogate is to the end-user, means the content provider can offer extra services based on things like geography, which would've been way harder with a single centralized service 🧚
now, ICAP (the Internet Content Adaption Protocol) 💖 is a protocol built for simple object-based content vectoring for HTTP services. it's basically a lightweight protocol for running a "remote procedure call" on HTTP messages ✨. ICAP clients pass HTTP messages to ICAP servers for some kind of transformation or other processing (that's the "adaptation" part) 🎀. the server runs its transformation service on the messages and sends responses back to the client, usually with modified messages 💅. the adapted messages can be either HTTP requests or HTTP responses. transformations on other non-HTTP content might be possible, but those are beyond the scope of this document 🙈
this kind of Remote Procedure Call (RPC) is super useful in a bunch of ways. for example:
- Simple transformations of content can happen near the edge of the network instead of needing an updated copy of an object from an origin server 🍓. like, a content provider might want a popular web page to show a different advertisement every time someone views it 👀. right now, providers handle this by marking such pages as non-cachable and tracking user cookies, which piles extra load onto the origin server and the network 😩. in our architecture, the page can be cached once near the edges of the network, and those edge caches make an ICAP call to a nearby ad-insertion server every time the page gets served to a client 🔥.
other transformations by edge servers are possible too, either with cooperation from the content provider (as in a content distribution network) or as a value-added service from a client's network provider (as in a surrogate) 💜. examples include translating web pages into different human languages, or into formats that suit special physical devices (e.g., PDA-based or cell-phone-based browsers) 📱.
- Surrogates or origin servers can skip expensive operations by shipping the work off to other servers instead 🧠. this helps spread load across multiple machines 💪. for example, say a user is trying to download an executable program through a surrogate (e.g., a caching proxy) 🕵️. the surrogate, acting as an ICAP client, can ask an external server to check the executable for viruses before it's accepted into the cache 🛡️.
- Firewalls or surrogates can act as ICAP clients and send outgoing requests to a service that checks that the URI in the request is allowed (for example, in a system that allows parental control of web content viewed by children) 🧸. in this case, it's a request that's being adapted, not an object returned by a response 👑.
In all of these examples, ICAP is helping to reduce or distribute the load on origin servers, surrogates, or the network itself 🌟. sometimes ICAP makes transformations happen near the edge of the network, which allows greater cachability of the underlying content 💾. other times, devices like origin servers or surrogates can lighten their load by handing expensive operations off to other machines 🙌. honestly, in every case, ICAP also creates a standard interface for content adaptation, which gives way more flexibility in content distribution or the addition of value added services in surrogates 🥰.
there are two major components in our architecture:
- Transaction semantics -- "How do I ask for adaptation?" 💬
- Control of policy -- "When am I supposed to ask for adaptation, what kind of adaptation do I ask for, and from where?" 🤔
right now, ICAP only defines the transaction semantics 🎯. for example, this document specifies how to send an HTTP message from an ICAP client to an ICAP server, how to specify the URI of the ICAP resource being requested along with other resource-specific parameters, and how to receive the adapted message 📨.
this wire-protocol is a necessary building block, but it's of limited use without the second part: an accompanying application framework for it to operate in 🧱. the trickier policy issue is beyond the scope of the current ICAP protocol, but it's planned for future work 🔮.
in the initial implementations, we expect that manual, implementation-specific configuration will be used to define policy 🛠️. this includes the rules for recognizing messages that need adaptation, the URIs of available adaptation resources, and so on 📋. for ICAP clients and servers to interoperate, the exact method used to define policy doesn't need to match across implementations, as long as the policy itself is consistent 🫶.
IMPORTANT: Note that right now, without a policy-framework, it is strongly RECOMMENDED that transformations SHOULD only be performed on messages with the explicit consent of either the content-provider or the user (or both). Deploying transformation services without the consent of either one leads to, at best, unpredictable results 😬. for more discussion of these issues, see Section 7. 🙏
once the full range of typical policy decisions is better understood through experience with these initial implementations, later follow-ons to this architecture may define an additional policy control protocol 📚. this future protocol may offer a standard policy definition interface that complements the ICAP transaction interface defined here 💅.
2. Terminology 🎀
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC 2119 [2]. 📖
the special terminology used in this document is defined below 👇. most of these terms are taken as-is from HTTP/1.1 [4] and are reproduced here for reference. a thorough understanding of HTTP/1.1 is assumed on the part of the reader, so bestie, go brush up if you need to 💖.
connection: a transport layer virtual circuit established between two programs for the purpose of communication 🔌.
message: the basic unit of HTTP communication, made of a structured sequence of octets matching the syntax defined in Section 4 of HTTP/1.1 [4] and transmitted via the connection 📦.
request: an HTTP request message, as defined in Section 5 of HTTP/1.1 [4] 📤.
response: an HTTP response message, as defined in Section 6 of HTTP/1.1 [4] 📥.
resource: a network data object or service that can be identified by a URI, as defined in Section 3.2 of HTTP/1.1 [4]. resources may be available in multiple representations (e.g., multiple languages, data formats, size, resolutions) or vary in other ways 🌈.
client: a program that establishes connections for the purpose of sending requests 🙋.
server: an application program that accepts connections in order to service requests by sending back responses. any given program may be capable of being both a client and a server. our use of these terms refers only to the role the program is performing for a particular connection, not to the program's capabilities in general. likewise, any server may act as an origin server, surrogate, gateway, or tunnel, switching behavior based on the nature of each request 🔄.
origin server: the server on which a given resource resides or is to be created 🏠.
proxy: an intermediary program that acts as both a server and a client for the purpose of making requests on behalf of other clients. requests are serviced internally or passed on, with possible translation, to other servers. a proxy MUST implement both the client and server requirements of this specification 🧚♀️.
cache: a program's local store of response messages, plus the subsystem that controls its message storage, retrieval, and deletion. a cache stores cachable responses in order to reduce response time and network bandwidth consumption on future, equivalent requests. any client or server may include a cache, though a cache cannot be used by a server that is acting as a tunnel 🗄️.
cachable: a response is cachable if a cache is allowed to store a copy of the response message for use in answering subsequent requests. the rules for determining the cachability of HTTP responses are defined in Section 13 of [4]. even if a resource is cachable, there may be additional constraints on whether a cache can use the cached copy for a particular request 🧐.
surrogate: a gateway co-located with an origin server, or at a different point in the network, that has been delegated the authority to operate on behalf of, and typically works in close co-operation with, one or more origin servers. responses are typically delivered from an internal cache. surrogates may derive cache entries from the origin server or from another of the origin server's delegates. in some cases a surrogate may tunnel such requests 🦸.
where close co-operation between origin servers and surrogates exists, this enables modifications of some protocol requirements, including the Cache-Control directives in [4]. such modifications have yet to be fully specified 🚧.
devices commonly known as "reverse proxies" and "(origin) server accelerators" are both more properly defined as surrogates 💁.
New definitions:
ICAP resource: similar to an HTTP resource as described above, but the URI refers to an ICAP service that performs adaptations of HTTP messages 🆕.
ICAP server: similar to an HTTP server as described above, except that the application services ICAP requests 🖥️.
ICAP client: a program that establishes connections to ICAP servers for the purpose of sending requests. an ICAP client is often, but not always, a surrogate acting on behalf of a user 🧑💻.
3. ICAP Overall Operation 🌸
before we get into ICAP's semantics in detail, let's do a quick overview of the protocol's major functions and expected uses 👀. like we said earlier, ICAP focuses on modification of HTTP requests (Section 3.1) and modification of HTTP responses (Section 3.2) 💕.
3.1. Request Modification 🎀
Okay so in "request modification" (reqmod) mode 💅, an ICAP client sends an HTTP request to an ICAP server, and then the server has three possible moves:
- Send back a modified version of the request. The ICAP client may then perform the modified request by contacting an origin server; or, pipeline the modified request to another ICAP server for further modification.
- Send back an HTTP response to the request. This is used to provide information useful to the user in case of an error (e.g., "you sent a request to view a page you are not allowed to see").
- Return an error.
ICAP clients MUST be able to handle all three types of responses 🫶. However, in line with the guidance provided for HTTP surrogates in Section 13.8 of [4], ICAP client implementors do have flexibility in handling errors. If the ICAP server returns an error, the ICAP client may (for example) return the error to the user, execute the unadapted request as it arrived from the client, or re-try the adaptation again.
Let's do a content filtering example 🍓, because it's so relatable. Picture a surrogate that receives a request from a client for a web page on an origin server. The surrogate, acting as an ICAP client, sends the client's request to an ICAP server that performs URI-based content filtering. If access to the requested URI is allowed, the request comes back to the ICAP client unmodified. However, if the ICAP server decides to disallow access to the requested resources, it may either:
- Modify the request so that it points to a page containing an error message instead of the original URI.
- Return an encapsulated HTTP response that indicates an HTTP error.
This method can be used for a variety of other applications 🌈; for example, anonymization, modification of the Accept: headers to handle special device requirements, and so forth.
Typical data flow:
origin-server
| /|\
| |
5 | | 4
| |
\|/ | 2
ICAP-client --------------> ICAP-resource
(surrogate) <-------------- on ICAP-server
| /|\ 3
| |
6 | | 1
| |
\|/ |
client- A client makes a request to a ICAP-capable surrogate (ICAP client) for an object on an origin server.
- The surrogate sends the request to the ICAP server.
- The ICAP server executes the ICAP resource's service on the request and sends the possibly modified request, or a response to the request back to the ICAP client.
If Step 3 returned a request:
- The surrogate sends the request, possibly different from original client request, to the origin server.
- The origin server responds to request.
- The surrogate sends the reply (from either the ICAP server or the origin server) to the client.
3.2. Response Modification 🌸
In the "response modification" (respmod) mode 🦋, an ICAP client sends an HTTP response to an ICAP server. (The response sent by the ICAP client typically has been generated by an origin server.) The ICAP server may then:
- Send back a modified version of the response.
- Return an error.
The response modification method is intended for post-processing performed on an HTTP response before it is delivered to a client. Examples include formatting HTML for display on special devices, human language translation, virus checking, and so forth ✨.
Typical data flow:
origin-server
| /|\
| |
3 | | 2
| |
\|/ | 4
ICAP-client --------------> ICAP-resource
(surrogate) <-------------- on ICAP-server
| /|\ 5
| |
6 | | 1
| |
\|/ |
client- A client makes a request to a ICAP-capable surrogate (ICAP client) for an object on an origin server.
- The surrogate sends the request to the origin server.
- The origin server responds to request.
- The ICAP-capable surrogate sends the origin server's reply to the ICAP server.
- The ICAP server executes the ICAP resource's service on the origin server's reply and sends the possibly modified reply back to the ICAP client.
- The surrogate sends the reply, possibly modified from the original origin server's reply, to the client.
4. Protocol Semantics 👑
4.1. General Operation 🦋
ICAP is a request/response protocol, similar in semantics and usage to HTTP/1.1 [4]. Despite the similarity, ICAP is not HTTP, nor is it an application protocol that runs over HTTP. This means, for example, that ICAP messages can not be forwarded by HTTP surrogates. Our reasons for not building directly on top of HTTP are discussed in Section 8.1.
ICAP uses TCP/IP as a transport protocol. The default port is 1344, but other ports may be used. The TCP flow is initiated by the ICAP client to a passively listening ICAP server.
ICAP messages consist of requests from client to server and responses from server to client. Requests and responses use the generic message format of RFC 2822 [3] -- that is, a start-line (either a request line or a status line), a number of header fields (also known as "headers"), an empty line (i.e., a line with nothing preceding the CRLF) indicating the end of the header fields, and a message-body.
The header lines of an ICAP message specify the ICAP resource being requested, as well as other meta-data such as cache control information. The message body of an ICAP request contains the (encapsulated) HTTP messages that are being modified.
As in HTTP/1.1, a single transport connection MAY (perhaps even SHOULD) be re-used for multiple request/response pairs. The rules for doing so in ICAP are the same as described in Section 8.1.2.2 of [4]. Specifically, requests are matched up with responses by allowing only one outstanding request on a transport connection at a time. Multiple parallel connections MAY be used as in HTTP 💜.
4.2. ICAP URIs 🧚
All ICAP requests specify the ICAP resource being requested from the server using an ICAP URI. This MUST be an absolute URI that specifies both the complete hostname and the path of the resource being requested. For definitive information on URL syntax and semantics, see "Uniform Resource Identifiers (URI): Generic Syntax and Semantics," RFC 2396 [1], Section 3. The URI structure defined by ICAP is roughly:
ICAP_URI = Scheme ":" Net_Path [ "?" Query ]
Scheme = "icap"
Net_Path = "//" Authority [ Abs_Path ]
Authority = [ userinfo "@" ] host [ ":" port ]ICAP adds the new scheme "icap" to the ones defined in RFC 2396. If the port is empty or not given, port 1344 is assumed. An example ICAP URI line might look like this:
icap://icap.example.net:2000/services/icap-service-1An ICAP server MUST be able to recognize all of its host names, including any aliases, local variations, and numeric IP addresses of its interfaces.
Any arguments that an ICAP client wishes to pass to an ICAP service to modify the nature of the service MAY be passed as part of the ICAP-URI, using the standard "?"-encoding of attribute-value pairs used in HTTP. For example:
icap://icap.net/service?mode=translate&lang=french4.3. ICAP Headers 🎀
The following sections define the valid headers for ICAP messages. Section 4.3.1 describes headers common to both requests and responses. Request-specific and response-specific headers are described in Sections 4.3.2 and 4.3.3, respectively.
User-defined header extensions are allowed. In compliance with the precedent established by the Internet mail format [3] and later adopted by HTTP [4], all user-defined headers MUST follow the "X-" naming convention ("X-Extension-Header: Foo"). ICAP implementations MAY ignore any "X-" headers without loss of compliance with the protocol as defined in this document.
Each header field consists of a name followed by a colon (":") and the field value. Field names are case-insensitive. ICAP follows the rules describe in section 4.2 of [4].
4.3.1. Headers Common to Requests and Responses 🌷
The headers of all ICAP messages MAY include the following directives, defined in ICAP the same as they are in HTTP:
Cache-Control
Connection
Date
Expires
Pragma
Trailer
UpgradeNote in particular that the "Transfer-Encoding" option is not allowed. The special transfer-encoding requirements of ICAP bodies are described in Section 4.4.
The Upgrade header MAY be used to negotiate Transport-Layer Security on an ICAP connection, exactly as described for HTTP/1.1 in [4].
The ICAP-specific headers defined are:
Encapsulated (See Section 4.4)4.3.2. Request Headers 🫶
Similar to HTTP, ICAP requests MUST start with a request line that contains a method, the complete URI of the ICAP resource being requested, and an ICAP version string. The current version number of ICAP is "1.0".
This version of ICAP defines three methods:
REQMOD - for Request Modification (Section 4.8)
RESPMOD - for Response Modification (Section 4.9)
OPTIONS - to learn about configuration (Section 4.10)The OPTIONS method MUST be implemented by all ICAP servers. All other methods are optional and MAY be implemented.
User-defined extension methods are allowed. Before attempting to use an extension method, an ICAP client SHOULD use the OPTIONS method to query the ICAP server's list of supported methods; see Section 4.10. (If an ICAP server receives a request for an unknown method, it MUST give a 501 error response as described in the next section.)
Given the URI rules described in Section 4.2, a well-formed ICAP request line looks like the following example:
RESPMOD icap://icap.example.net/translate?mode=french ICAP/1.0A number of request-specific headers are allowed in ICAP requests, following the same semantics as the corresponding HTTP request headers (Section 5.3 of [4]). These are:
Authorization
Allow (see Section 4.6)
From (see Section 14.22 of [4])
Host (REQUIRED in ICAP as it is in HTTP/1.1)
Referer (see Section 14.36 of [4])
User-AgentIn addition to HTTP-like headers, there are also request headers unique to ICAP defined:
Preview (see Section 4.5)4.3.3. Response Headers ✨
okay so ICAP responses MUST start with an ICAP status line, which looks a lot like HTTP's, with the ICAP version and a status code 💅
ICAP/1.0 200 OKThe status codes mean the same thing they do in HTTP (Section 6.1.1 and 10 of [4]), except where this document says otherwise. The ones to watch are 100 (Section 4.5) and 204 (Section 4.6) 👀
Here are the ICAP error codes that are different from their HTTP cousins 🫶
100- Continue after ICAP Preview (Section 4.5). 🌸204- No modifications needed (Section 4.6). ✨400- Bad request. 🙈404- ICAP Service not found. 😭405- Method not allowed for service. Like, RESPMOD was requested for a service that only supports REQMOD 💀408- Request timeout. The ICAP server gave up waiting for a request from the ICAP client ⏰500- Server error. Something broke on the ICAP server, like running out of disk space 💔501- Method not implemented. This one is illegal for an OPTIONS request, since OPTIONS support is mandatory, so no sending this back for OPTIONS 🚫502- Bad Gateway. This is an ICAP proxy, and the proxying itself produced an error 🌉503- Service overloaded. The ICAP server hit a maximum connection limit for this service, and the ICAP client should not exceed that limit in the future 🚦505- ICAP version not supported by server. 🧚
Same vibe as HTTP here: the 4xx class is for client errors, and the 5xx class is for server errors 🎀
ICAP response headers let the server pack extra info into the response that can't fit in the status line 📦
A response-specific header is allowed in ICAP responses, and it follows the same semantics as the matching HTTP response headers (Section 6.2 of [4]). That header is:
Server(see Section 14.38 of [4]) 🖥️
ICAP also has one response header of its own, on top of the HTTP-style ones:
ISTag(see Section 4.7) 🏷️
4.3.4. ICAP-Related Headers in HTTP Messages 🌈
So when an ICAP-enabled HTTP surrogate sends an HTTP request to an origin server, it's often super useful to tell the origin server about the surrogate's ICAP capabilities 🧠 The origin server can use that info to tailor its response. For example, an origin server might skip inserting an ad into a page if it knows a downstream ICAP server can insert the ad instead. Iconic behavior 📢🚫
Now, this ICAP spec can't dictate how HTTP gets used between HTTP clients and servers, but we do suggest a convention 🌸 If such headers are used, they SHOULD start with "X-ICAP". HTTP clients with ICAP services SHOULD minimally include an "X-ICAP-Version: 1.0" header along with their application-specific headers 💖
4.4. ICAP Bodies: Encapsulation of HTTP Messages 📦✨
The ICAP encapsulation model is a lightweight way to pack any number of HTTP message sections into one encapsulating ICAP message-body. This lets you send requests, responses, and request/response pairs to an ICAP server for vectoring 🦋
Basically you glue the interesting message parts (the encapsulated sections) together into a single ICAP message-body (the encapsulating message). The encapsulated sections can be the headers or the bodies of HTTP messages 🧩
Encapsulated bodies MUST be transferred using the "chunked" transfer-coding described in Section 3.6.1 of [4]. However, encapsulated headers MUST NOT be chunked. So the ICAP message-body switches from non-chunked to chunked as it moves from the encapsulated header section to the encapsulated body section 🔄 (See Examples in Sections 4.8.3 and 4.9.3.) The reasoning behind this choice is explained in Section 8.2 🤓
4.4.1. The "Encapsulated" Header 🎀
The offset of each encapsulated section's start, measured from the start of the encapsulating message's body, is noted in the "Encapsulated" header. This header MUST be included in every ICAP message 📍 For example, this header:
Encapsulated: req-hdr=0, res-hdr=45, res-body=100means the message encapsulates a group of request headers, then a group of response headers, then a response body. Each one starts at the byte-offset listed. The byte-offsets are in decimal, to match HTTP's Content-Length header 🔢
The special entity "null-body" means there is no encapsulated body in the ICAP message 🫥
The syntax of an Encapsulated header is:
encapsulated_header: "Encapsulated: " encapsulated_list
encapsulated_list: encapsulated_entity |
encapsulated_entity ", " encapsulated_list
encapsulated_entity: reqhdr | reshdr | reqbody | resbody | optbody
reqhdr = "req-hdr" "=" (decimal integer)
reshdr = "res-hdr" "=" (decimal integer)
reqbody = { "req-body" | "null-body" } "=" (decimal integer)
resbody = { "res-body" | "null-body" } "=" (decimal integer)
optbody = { "opt-body" | "null-body" } "=" (decimal integer)There are also semantic rules on top of the syntax. The encapsulated parts MUST appear in the encapsulating message-body in the same order they're named in the Encapsulated header. Put differently, the offsets in the Encapsulated line MUST be monotonically increasing 📈 Plus, which forms are legal depends on the method (REQMOD, RESPMOD, or OPTIONS). Specifically:
REQMOD request encapsulated_list: [reqhdr] reqbody
REQMOD response encapsulated_list: {[reqhdr] reqbody} |
{[reshdr] resbody}
RESPMOD request encapsulated_list: [reqhdr] [reshdr] resbody
RESPMOD response encapsulated_list: [reshdr] resbody
OPTIONS response encapsulated_list: optbodyIn that grammar, encapsulated headers are always optional 🙌 At most one body per encapsulated message is allowed 🚫👜 If there's no encapsulated body, you use the "null-body" directive instead. This is handy because it still tells you how long the header section is 💡
Here are some legal Encapsulated headers, bestie 👑
/* REQMOD request: This encapsulated HTTP request's headers start
* at offset 0; the HTTP request body (e.g., in a POST) starts
* at 412. */
Encapsulated: req-hdr=0, req-body=412
/* REQMOD request: Similar to the above, but no request body is
* present (e.g., a GET). We use the null-body directive instead.
* In both this case and the previous one, we can tell from the
* Encapsulated header that the request headers were 412 bytes
* long. */
Encapsulated: req-hdr=0, null-body=412
/* REQMOD response: ICAP server returned a modified request,
* with body */
Encapsulated: req-hdr=0, req-body=512
/* RESPMOD request: Request headers at 0, response headers at 822,
* response body at 1655. Note that no request body is allowed in
* RESPMOD requests. */
Encapsulated: req-hdr=0, res-hdr=822, res-body=1655
/* RESPMOD or REQMOD response: header and body returned */
Encapsulated: res-hdr=0, res-body=749
/* OPTIONS response when there IS an options body */
Encapsulated: opt-body=0
/* OPTIONS response when there IS NOT an options body */
Encapsulated: null-body=04.4.2. Encapsulated HTTP Headers 📝
By default, ICAP messages can encapsulate HTTP message headers and entity bodies 🎁 HTTP headers MUST start with the request-line (for requests) or the status-line (for responses), followed by the interesting HTTP headers 🌟
The encapsulated headers MUST end with a blank line. That keeps them human readable, and it also ends line-by-line HTTP parsers properly 🙃
HTTP/1.1 makes a distinction between end-to-end headers and hop-by-hop headers (see Section 13.5.1 of [4]). End-to-end headers matter to the ultimate recipient of a message. Hop-by-hop headers only matter for a single transport-layer connection, and they include things like Connection and Keep-Alive 🔗 All end-to-end HTTP headers SHOULD be encapsulated, and all hop-by-hop headers MUST NOT be encapsulated 🚧
Even with those rules, the hop-by-hop Proxy-Authenticate and Proxy-Authorization headers MUST be forwarded to the ICAP server in the ICAP header section, not inside the encapsulated message 🔐 This lets client credentials that were sent to the ICAP client get passed along, which matters when the ICAP client is also an HTTP surrogate. This doesn't contradict HTTP/1.1, which explicitly says "A proxy MAY relay the credentials from the client request to the next proxy if that is the mechanism by which the proxies cooperatively authenticate a given request." (Section 14.34) ✨
The Via header of an encapsulated message SHOULD be modified by an ICAP server as if the message were traveling through an HTTP surrogate 🛤️ And any Via header the ICAP server adds MUST specify the protocol as ICAP/1.0 💜
4.5. Message Preview 🦋
okay bestie, so ICAP REQMOD or RESPMOD requests that the ICAP client sends to the ICAP server can include a "preview" ✨ This lets the server peek at the beginning of a transaction and then decide if it wants to opt out early, instead of getting the rest of the request message. Previewing can be a total glow-up for performance in a bunch of situations 💅 like these:
- Virus-checkers can certify a big chunk of files as "clean" just by looking at the file type, the file name extension, and the first few bytes of the file. Only the leftover files need to be sent to the virus-checking ICAP server in their entirety 🛡️
- Content filters can use Preview to decide if an HTTP entity needs to be inspected 👀 (the HTTP file type alone is not enough, because sometimes "text" actually turns out to be graphics data). The magic numbers at the front of the file can identify a file as a JPEG or GIF 🖼️
- If an ICAP server wants to transcode all GIF87 files into GIF89 files, it can quickly detect the GIF87 files by looking at the first few body bytes of the file 🎞️
- If an ICAP server wants to force all cacheable files to expire in 24 hours or less, this can be done by selecting HTTP messages with expiries more than 24 hours in the future ⏰
ICAP servers SHOULD use the OPTIONS method (see Section 4.10) to specify how many bytes of preview are needed for a particular ICAP application on a per-resource basis 📋 Clients SHOULD be able to provide Previews of at least 4096 bytes. Clients furthermore SHOULD provide a Preview when using any ICAP resource that has indicated a Preview is useful. (This indication might be provided via the OPTIONS method, or some other "out-of-band" configuration.) Clients SHOULD NOT provide a larger Preview than a server has indicated it is willing to accept 🙅♀️
To effect a Preview, an ICAP client MUST add a "Preview:" header to its request headers, indicating the length of the preview 🎀 The ICAP client then sends:
- all of the encapsulated header sections, and
- the beginning of the encapsulated body section, if any, up to the number of bytes advertised in the Preview (possibly 0).
After the Preview is sent, the client stops and waits for an intermediate response from the ICAP server before continuing ⏸️ This is kind of like the "100-Continue" feature in HTTP, except that the stop-and-wait point can be within the message body. HTTP, on the other hand, requires that the point be the boundary between the headers and body. Not the same vibe at all 🙈
For example, to effect a Preview consisting of only encapsulated HTTP headers, the ICAP client would add the following header to the ICAP request:
Preview: 0This tells the server that the ICAP client will only send the encapsulated header sections to the ICAP server. Then it sends a zero-length chunk and stops and waits for a "go ahead" before sending any more encapsulated body bytes to the ICAP server 🚦
Similarly, the ICAP header:
Preview: 4096indicates that the ICAP client will attempt to send 4096 bytes of origin server data in the encapsulated body of the ICAP request to the ICAP server. It's important to note that the actual transfer may be less, because the ICAP client is acting like a surrogate and is not looking ahead to find the total length of the origin server response 👀 The entire ICAP encapsulated header section(s) will be sent, followed by up to 4096 bytes of encapsulated HTTP body. The chunk body terminator 0\r\n\r\n is always included in these transactions 🧚
After sending the preview, the ICAP client will wait for a response from the ICAP server. The response MUST be one of the following:
- 204 No Content. The ICAP server does not want to (or can not) modify the ICAP client's request. The ICAP client MUST treat this the same as if it had sent the entire message to the ICAP server and an identical message was returned 🫶
- ICAP reqmod or respmod response, depending on what method was the original request. See Section 4.8.2 and 4.9.2 for the format of reqmod and respmod responses 📦
- 100 Continue. If the entire encapsulated HTTP body did not fit in the preview, the ICAP client MUST send the remainder of its ICAP message, starting from the first chunk after the preview. If the entire message fit in the preview (detected by the "EOF" symbol explained below), then the ICAP server MUST NOT respond with 100 Continue 🚫
When an ICAP client is performing a preview, it may not yet know how many bytes will ultimately be available in the arriving HTTP message that it is relaying to the HTTP server 🤷♀️ So ICAP defines a way for ICAP clients to indicate "EOF" to ICAP servers if one unexpectedly arrives during the preview process. This is a super useful optimization if a header-only HTTP response arrives at the ICAP client (i.e., zero bytes of body). Only a single round trip is needed for the complete ICAP server response, which is so efficient it's giving 🔥
We define an HTTP chunk-extension of "ieof" to indicate that an ICAP chunk is the last chunk (see [4]). The ICAP server MUST strip this chunk extension before passing the chunk data to an ICAP application process 🧹
For example, consider an ICAP client that has just received HTTP response headers from an origin server and initiates an ICAP RESPMOD transaction to an ICAP server. It doesn't know yet how many body bytes will be arriving from the origin server, because the server is not using the Content-Length header 😵💫 The ICAP client tells the ICAP server it will be sending a 1024-byte preview using a "Preview: 1024" request header. If the HTTP origin server then closes its connection to the ICAP client before sending any data (i.e., it provides a zero-byte body), the corresponding zero-byte preview for that zero-byte origin response would look like this:
\r\n
0; ieof\r\n\r\nIf an ICAP server sees this preview, it knows from the presence of "ieof" that the client will not be sending any more chunk data. In this case, the server MUST respond with the modified response or a 204 No Content message right away. It MUST NOT send a 100-Continue response in this case. (In contrast, if the origin response had been 1 byte or larger, the "ieof" would not have appeared. In that case, an ICAP server MAY reply with 100-Continue, a modified response, or 204 No Content.) 🌸
In another example, if the preview is 1024 bytes and the origin response is 1024 bytes in two chunks, the encapsulation would look like this:
200\r\n
<512 bytes of data>\r\n
200\r\n
<512 bytes of data>\r\n
0; ieof\r\n\r\n
<204 or modified response> (100 Continue disallowed due to ieof)If the preview is 1024 bytes and the origin response is 1025 bytes (and the ICAP server responds with 100-continue), then these chunks would appear on the wire:
200\r\n
<512 bytes of data>\r\n
200\r\n
<512 bytes of data>\r\n
0\r\n
<100 Continue Message>
1\r\n
<1 byte of data>\r\n
0\r\n\r\n <no ieof because we are no longer in preview mode>Once the ICAP server receives the eof indicator, it finishes reading the current chunk stream 🏁
Note that when offering a Preview, the ICAP client is committing to temporarily buffer the previewed portion of the message so that it can honor a "204 No Content" response 🧺 The remainder of the message is not necessarily buffered. It might be pipelined directly from another source to the ICAP server after a 100-Continue 🚰
4.6. "204 No Content" Responses outside of Previews
An ICAP client MAY choose to honor "204 No Content" responses for an entire message. This is the client's decision, because it puts a burden on the client of buffering the entire message 🧠
An ICAP client MAY include "Allow: 204" in its request headers, indicating that the server MAY reply to the message with a "204 No Content" response if the object does not need modification ✅
If an ICAP server receives a request that does not have "Allow: 204", it MUST NOT reply with a 204. In this case, an ICAP server MUST return the entire message back to the client, even though it is identical to the message it received 📨
The ONLY EXCEPTION to this rule is in the case of a message preview, as described in the previous section. If this is the case, an ICAP server can respond with a 204 No Content message in response to a message preview EVEN if the original request did not have the "Allow: 204" header 💖
4.7. ISTag Response Header
The ISTag ("ICAP Service Tag") response-header field gives ICAP servers a way to send a service-specific "cookie" to ICAP clients that represents a service's current state 🍪 It's a 32-byte-maximum alphanumeric string of data (not including the null character) that may, for example, be a representation of the software version or configuration of a service. An ISTag validates that previous ICAP server responses can still be considered fresh by an ICAP client that may be caching them 💾 If a change on the ICAP server invalidates previous responses, the ICAP server can invalidate portions of the ICAP client's cache by changing its ISTag. The ISTag MUST be included in every ICAP response from an ICAP server 🫡
For example, consider a virus-scanning ICAP service. The ISTag might be a combination of the virus scanner's software version and the release number of its virus signature database. When the database is updated, the ISTag can be changed to invalidate all previous responses that had been certified as "clean" and cached with the old ISTag 🧼
ISTag is similar to the HTTP ETag, but not identical 👯 While an ETag is a validator for a particular entity (object), an ISTag validates all entities generated by a particular service (URI). A change in the ISTag invalidates all the other entities provided by a service with the old ISTag, not just the entity whose response contained the updated ISTag. Big difference, bestie 😮
The syntax of an ISTag is simply:
ISTag = "ISTag: " quoted-stringIn this document we use the quoted-string definition defined in section 2.2 of [4]. For example:
ISTag: "874900-1994-1c02798"4.8. Request Modification Mode
In this method, described in Section 3.1, an ICAP client sends an HTTP request to an ICAP server 📤 The ICAP server returns a modified version of the request, an HTTP response, or (if the client indicates it supports 204 responses) an indication that no modification is required 🙌
4.8.1. Request
In REQMOD mode, the ICAP request MUST contain an encapsulated HTTP request. The headers and body (if any) MUST both be encapsulated, except that hop-by-hop headers are not encapsulated 🦋
4.8.2 Response
okay so the ICAP server has four possible answers to a Request Modification request 💅:
- An error indication,
- A 204 indicating that the ICAP client's request requires no adaptation (see Section 4.6 for limitations of this response),
- An encapsulated, adapted version of the ICAP client's request, or
- An encapsulated HTTP error response. Note that Request Modification requests may only be satisfied with HTTP responses in cases when the HTTP response is an error (e.g., 403 Forbidden).
The first line of the response message MUST be a status line as described in Section 4.3.3. If the return code is a 2XX, the ICAP client SHOULD continue its normal execution of the request. If the ICAP client is a surrogate, this may include serving an object from its cache or forwarding the modified request to an origin server. Note it is valid for a 2XX ICAP response to contain an encapsulated HTTP error response, which in turn should be returned to the downstream client by the ICAP client 🎀.
For other return codes that indicate an error, the ICAP client MAY (for example) return the error to the downstream client or user, execute the unadapted request as it arrived from the client, or re-try the adaptation again 🦋.
The modified request headers, if any, MUST be returned to the ICAP client using appropriate encapsulation as described in Section 4.4 🫶.
4.8.3 Examples
okay, this is the fun part 🧚. We're going to watch a surrogate receive a plain GET request from a client. The surrogate acts as an ICAP client and forwards that request to an ICAP server for modification. The ICAP server edits several headers and strips the cookie out of the original request 🍓.
In all of our examples, we include the extra meta-data added to the message due to chunking the encapsulated message body (if any). We assume that end-of-line terminations, and blank lines, are two-byte "CRLF" sequences.
ICAP Request Modification Example 1 - ICAP Request
REQMOD icap://icap-server.net/server?arg=87 ICAP/1.0
Host: icap-server.net
Encapsulated: req-hdr=0, null-body=170
GET / HTTP/1.1
Host: www.origin-server.com
Accept: text/html, text/plain
Accept-Encoding: compress
Cookie: ff39fk3jur@4ii0e02i
If-None-Match: "xyzzy", "r2d2xxxx"
ICAP Request Modification Example 1 - ICAP Response
ICAP/1.0 200 OK
Date: Mon, 10 Jan 2000 09:55:21 GMT
Server: ICAP-Server-Software/1.0
Connection: close
ISTag: "W3E4R7U9-L2E4-2"
Encapsulated: req-hdr=0, null-body=231
GET /modified-path HTTP/1.1
Host: www.origin-server.com
Via: 1.0 icap-server.net (ICAP Example ReqMod Service 1.1)
Accept: text/html, text/plain, image/gif
Accept-Encoding: gzip, compress
If-None-Match: "xyzzy", "r2d2xxxx"
The second example is a lot like the first, except the request being modified is a POST instead of a GET 💜. Notice that the encapsulated Content-Length argument has been updated to match the modified body of the POST message. The outer ICAP message doesn't need a Content-Length header because it uses chunking (not shown) ✨.
In this second example, the Encapsulated header marks the split between the forwarded header and forwarded body, for both the request and the response.
ICAP Request Modification Example 2 - ICAP Request
REQMOD icap://icap-server.net/server?arg=87 ICAP/1.0
Host: icap-server.net
Encapsulated: req-hdr=0, req-body=147
POST /origin-resource/form.pl HTTP/1.1
Host: www.origin-server.com
Accept: text/html, text/plain
Accept-Encoding: compress
Pragma: no-cache
1e
I am posting this information.
0
ICAP Request Modification Example 2 - ICAP Response
ICAP/1.0 200 OK
Date: Mon, 10 Jan 2000 09:55:21 GMT
Server: ICAP-Server-Software/1.0
Connection: close
ISTag: "W3E4R7U9-L2E4-2"
Encapsulated: req-hdr=0, req-body=244
POST /origin-resource/form.pl HTTP/1.1
Host: www.origin-server.com
Via: 1.0 icap-server.net (ICAP Example ReqMod Service 1.1)
Accept: text/html, text/plain, image/gif
Accept-Encoding: gzip, compress
Pragma: no-cache
Content-Length: 45
2d
I am posting this information. ICAP powered!
0
Finally, this third example shows an ICAP server sending back an error response when it gets a Request Modification request 🙈.
ICAP Request Modification Example 3 - ICAP Request
REQMOD icap://icap-server.net/content-filter ICAP/1.0
Host: icap-server.net
Encapsulated: req-hdr=0, null-body=119
GET /naughty-content HTTP/1.1
Host: www.naughty-site.com
Accept: text/html, text/plain
Accept-Encoding: compress
ICAP Request Modification Example 3 - ICAP Response
ICAP/1.0 200 OK
Date: Mon, 10 Jan 2000 09:55:21 GMT
Server: ICAP-Server-Software/1.0
Connection: close
ISTag: "W3E4R7U9-L2E4-2"
Encapsulated: res-hdr=0, res-body=213
HTTP/1.1 403 Forbidden
Date: Wed, 08 Nov 2000 16:02:10 GMT
Server: Apache/1.3.12 (Unix)
Last-Modified: Thu, 02 Nov 2000 13:51:37 GMT
ETag: "63600-1989-3a017169"
Content-Length: 58
Content-Type: text/html
3a
Sorry, you are not allowed to access that naughty content.
0
4.9 Response Modification Mode
In this method, described in Section 3.2, an ICAP client sends an origin server's HTTP response to an ICAP server, along with the original client request that caused that response (if it's available) 🌸. Just like the Request Modification method, the ICAP server can answer with an adapted HTTP response, an error, or a 204 response code saying that no adaptation is needed.
4.9.1 Request
Using the encapsulation described in Section 4.4, the header and body of the HTTP response to be modified MUST be included in the ICAP body. If the original client request header is available, it SHOULD also be included. As with the other method, the hop-by-hop headers of the encapsulated messages MUST NOT be forwarded. The Encapsulated header MUST indicate the byte-offsets of the beginning of each of these four parts 🔥.
4.9.2 Response
The response from the ICAP server looks just like a reply in the Request Modification method (Section 4.8). In other words, it's one of these:
- An error indication,
- An encapsulated and potentially modified HTTP response header and response body, or
- An HTTP response 204 indicating that the ICAP client's request requires no adaptation.
The first line of the response message MUST be a status line as described in Section 4.3.3. If the return code is a 2XX, the ICAP client SHOULD continue its normal execution of the response. The ICAP client MAY re-examine the headers in the response's message headers in order to make further decisions about the response (e.g., its cachability) 👑.
For other return codes that indicate an error, the ICAP client SHOULD NOT return these directly to the downstream client, since these errors only make sense in the ICAP client/server transaction 💀.
The modified response headers, if any, MUST be returned to the ICAP client using appropriate encapsulation as described in Section 4.4 🫶.
4.9.3 Examples
In Example 4, an ICAP client asks for modification of an entity that came back from a client GET. The original client GET went to an origin server at "www.origin-server.com", and the ICAP server lives at "icap.example.org" 🌈.
ICAP Response Modification Example 4 - ICAP Request
RESPMOD icap://icap.example.org/satisf ICAP/1.0
Host: icap.example.org
Encapsulated: req-hdr=0, res-hdr=137, res-body=296
GET /origin-resource HTTP/1.1
Host: www.origin-server.com
Accept: text/html, text/plain, image/gif
Accept-Encoding: gzip, compress
HTTP/1.1 200 OK
Date: Mon, 10 Jan 2000 09:52:22 GMT
Server: Apache/1.3.6 (Unix)
ETag: "63840-1ab7-378d415b"
Content-Type: text/html
Content-Length: 51
33
This is data that was returned by an origin server.
0
ICAP Response Modification Example 4 - ICAP Response
ICAP/1.0 200 OK
Date: Mon, 10 Jan 2000 09:55:21 GMT
Server: ICAP-Server-Software/1.0
Connection: close
ISTag: "W3E4R7U9-L2E4-2"
Encapsulated: res-hdr=0, res-body=222
HTTP/1.1 200 OK
Date: Mon, 10 Jan 2000 09:55:21 GMT
Via: 1.0 icap.example.org (ICAP Example RespMod Service 1.1)
Server: Apache/1.3.6 (Unix)
ETag: "63840-1ab7-378d415b"
Content-Type: text/html
Content-Length: 92
5c
This is data that was returned by an origin server, but with
value added by an ICAP server.
0
4.10 OPTIONS Method
The ICAP "OPTIONS" method is how an ICAP client pulls configuration info from an ICAP server 🍭. The client sends a request aimed at a specific ICAP resource and gets back a response listing options specific to the service that URI names. All OPTIONS requests MAY also return options that are global to the server (i.e., apply to all services) ✨.
4.10.1 OPTIONS Request
The OPTIONS method has a request-line, as described in Section 4.3.2, like this example:
OPTIONS icap://icap.server.net/sample-service ICAP/1.0 User-Agent:
ICAP-client-XYZ/1.001Other headers are also allowed, as described in Section 4.3.1 and Section 4.3.2 (for example, Host) 💖.
4.10.2. OPTIONS Response ✨
okay so the OPTIONS response is a status line (check section 4.3.3 🎀) followed by a series of header name-value pairs, and optionally an opt-body 💅. Multiple values in the value field MUST be separated by commas. If an opt-body shows up in the OPTIONS response, the Opt-body-type header tells you its format.
these are the OPTIONS headers this version of the protocol supports 🧚
Methods 🍓 The method this service supports. This header MUST be included in the OPTIONS response. The OPTIONS method MUST NOT be in the Methods' list since it MUST be supported by all the ICAP server implementations. Each service should have a distinct URI and support only one method in addition to OPTIONS (see Section 6.4).
For example: Methods: RESPMOD
Service 🌸 A text description of the vendor and product name. This header MAY be included in the OPTIONS response.
For example: Service: XYZ Technology Server 1.0
ISTag 🦋 See section 4.7 for details. This header MUST be included in the OPTIONS response.
For example: ISTag: "5BDEEEA9-12E4-2"
Encapsulated 🫶 This header MUST be included in the OPTIONS response; see Section 4.4.
For example: Encapsulated: opt-body=0
Opt-body-type 💜 A token that identifies the format of the opt-body. (Valid opt-body types are not defined by ICAP.) This header MUST be included in the OPTIONS response ONLY if an opt-body type is present.
For example: Opt-body-type: XML-Policy-Table-1.0
Max-Connections 🔥 The maximum number of ICAP connections the server can support. This header MAY be included in the OPTIONS response.
For example: Max-Connections: 1500
Options-TTL ⏳ The time (in seconds) for which this OPTIONS response is valid. If none is specified, the OPTIONS response does not expire. This header MAY be included in the OPTIONS response. The ICAP client MAY reissue an OPTIONS request once the Options-TTL expires.
For example: Options-TTL: 3600
Date 📅 The server's clock, given as an RFC 1123 compliant date/time string. This header MAY be included in the OPTIONS response.
For example: Date: Fri, 15 Jun 2001 04:33:55 GMT
Service-ID 🪪 A short label that identifies the ICAP service. It MAY be used in attribute header names. This header MAY be included in the OPTIONS response.
For example: Service-ID: xyztech
Allow ✅ A directive that declares a list of optional ICAP features this server has implemented. This header MAY be included in the OPTIONS response. In this document we define the value "204" to indicate that the ICAP server supports a 204 response.
For example: Allow: 204
Preview 👀 The number of bytes the ICAP client sends during a preview. This header MAY be included in the OPTIONS response.
For example: Preview: 1024
Transfer-Preview 🎀 A list of file extensions that should be previewed to the ICAP server before being sent in their entirety. This header MAY be included in the OPTIONS response. Multiple file extension values should be separated by commas. The wildcard value "" specifies the default behavior for all the file extensions not specified in any other Transfer- header (see below).
For example: Transfer-Preview: *
Transfer-Ignore 🙈 A list of file extensions that should NOT be sent to the ICAP server. This header MAY be included in the OPTIONS response. Multiple file extensions should be separated by commas.
For example: Transfer-Ignore: html
Transfer-Complete 🧁 A list of file extensions that should be sent in their entirety (without preview) to the ICAP server. This header MAY be included in the OPTIONS response. Multiple file extension values should be separated by commas.
For example: Transfer-Complete: asp, bat, exe, com, ole
Note: If any of Transfer-* are sent, exactly one of them MUST contain the wildcard value "" to specify the default. If no Transfer- are sent, all responses will be sent in their entirety (without Preview).
4.10.3. OPTIONS Examples 🌈
in example 5, an ICAP Client sends an OPTIONS Request to an ICAP Service named icap.server.net/sample-service so it can get the configuration info for that service 🍓
ICAP OPTIONS Example 5 - ICAP OPTIONS Request
----------------------------------------------------------------
OPTIONS icap://icap.server.net/sample-service ICAP/1.0
Host: icap.server.net
User-Agent: BazookaDotCom-ICAP-Client-Library/2.3
----------------------------------------------------------------ICAP OPTIONS Example 5 - ICAP OPTIONS Response
----------------------------------------------------------------
ICAP/1.0 200 OK
Date: Mon, 10 Jan 2000 09:55:21 GMT
Methods: RESPMOD
Service: FOO Tech Server 1.0
ISTag: "W3E4R7U9-L2E4-2"
Encapsulated: null-body=0
Max-Connections: 1000
Options-TTL: 7200
Allow: 204
Preview: 2048
Transfer-Complete: asp, bat, exe, com
Transfer-Ignore: html
Transfer-Preview: *
----------------------------------------------------------------5. Caching 💾
ICAP servers' responses MAY be cached by ICAP clients, just like any other surrogate might cache HTTP responses. Similar to HTTP, ICAP clients MAY always store a successful response (see sections 4.8.2 and 4.9.2) as a cache entry, and MAY return it without validation if it is fresh. ICAP servers use the caching directives described in HTTP/1.1 [4].
In Request Modification mode, the ICAP server MAY include caching directives in the ICAP header section of the ICAP response (NOT in the encapsulated HTTP request of the ICAP message body). In Response Modification mode, the ICAP server MAY add or modify the HTTP caching directives located in the encapsulated HTTP response (NOT in the ICAP header section). Consequently, the ICAP client SHOULD look for caching directives in the ICAP headers in case of REQMOD, and in the encapsulated HTTP response in case of RESPMOD.
In cases where an ICAP server returns a modified version of an object created by an origin server, such as in Response Modification mode, the expiration of the ICAP-modified object MUST NOT be longer than that of the origin object. In other words, ICAP servers MUST NOT extend the lifetime of origin server objects, but MAY shorten it.
In cases where the ICAP server is the authoritative source of an ICAP response, such as in Request Modification mode, the ICAP server is not restricted in its expiration policy.
Note that the ISTag response-header may also be used to providing caching hints to clients; see Section 4.7.
6. Implementation Notes 🧚
6.1. Vectoring Points 🔮
The definition of the ICAP protocol itself only describes two different adaptation channels: modification (and satisfaction) of requests, and modifications of replies. However, an ICAP client implementation is likely to actually distinguish among four different classes of adaptation:
- Adaptation of client requests. This is adaptation done every time a request arrives from a client. This is adaptation done when a request is "on its way into the cache". Factors such as the state of the objects currently cached will determine whether or not this request actually gets forwarded to an origin server (instead of, say, getting served off the cache's disk). An example of this type of adaptation would be special access control or authentication services that must be performed on a per-client basis.
- Adaptation of requests on their way to an origin server. Although this type of adaptation is also an adaptation of requests similar to (1), it describes requests that are "on their way out of the cache"; i.e., if a request actually requires that an origin server be contacted. These adaptation requests are not necessarily specific to particular clients. An example would be addition of "Accept:" headers for special devices; these adaptations can potentially apply to many clients.
- Adaptations of responses coming from an origin server. This is the adaptation of an object "on its way into the cache". In other words, this is adaptation that a surrogate might want to perform on an object before caching it. The adapted object may subsequently be served to many clients. An example of this type of adaptation is virus checking: a surrogate will want to check an incoming origin reply for viruses once, before allowing it into the cache -- not every time the cached object is served to a client.
- Adaptation of responses coming from the surrogate, heading back to the client. Although this type of adaptation, like (3), is the adaptation of a response, it is client-specific. Client reply adaptation is adaptation that is required every time an object is served to a client, even if all the replies come from the same cached object off of disk. Ad insertion is a common form of this kind of adaptation; e.g., if a popular (cached) object that rarely changes needs a different ad inserted into it every time it is served off disk to a client. Note that the relationship between adaptations of type (3) and (4) is analogous to the relationship between types (2) and (1).
Although the distinction among these four adaptation points is critical for ICAP client implementations, the distinction is not significant for the ICAP protocol itself. From the point of view of an ICAP server, a request is a request -- the ICAP server doesn't care what policy led the ICAP client to generate the request. We therefore did not make these four channels explicit in ICAP for simplicity.
6.2 Application Level Errors 💅
Okay bestie, so Section 4 is all about the "on the wire" protocol errors, and those MUST be standardized across implementations so everyone can actually talk to each other 🫶. This section is about the errors that get passed between ICAP software and the clients and servers it runs on. These are implementation dependent and don't have to be standardized since they're "within the box" 📦. We still share them as advice for future implementors, based on what past implementations have taught us ✨.
Error name Value
====================================================
ICAP_CANT_CONNECT 1000
ICAP_SERVER_RESPONSE_CLOSE 1001
ICAP_SERVER_RESPONSE_RESET 1002
ICAP_SERVER_UNKNOWN_CODE 1003
ICAP_SERVER_UNEXPECTED_CLOSE_204 1004
ICAP_SERVER_UNEXPECTED_CLOSE 1005- 1000
ICAP_CANT_CONNECT🔌: the message is"Cannot connect to ICAP server". The ICAP server isn't connected on the socket. Maybe it's dead, or maybe it just isn't connected on the socket 💀. - 1001
ICAP_SERVER_RESPONSE_CLOSE🚪: the message is"ICAP Server closed connection while reading response". The ICAP server does a TCP-shutdown on the connection before the ICAP client has sent all the body data 😭. - 1002
ICAP_SERVER_RESPONSE_RESET💥: the message is"ICAP Server reset connection while reading response". The ICAP server TCP-resets the connection before the ICAP client has sent all the body data 😩. - 1003
ICAP_SERVER_UNKNOWN_CODE❓: the message is"ICAP Server sent unknown response code". The ICAP client got an unknown ICAP response code (see Section 4.x) 🙈. - 1004
ICAP_SERVER_UNEXPECTED_CLOSE_204🙃: the message is"ICAP Server closed connection on 204 without 'Connection: close' header". An ICAP server MUST send theConnection: closeheader if it intends to close after the current transaction. Literally a must-have 📝. - 1005
ICAP_SERVER_UNEXPECTED_CLOSE🫠: the message is"ICAP Server closed connection as ICAP client wrote body preview".
6.3 Use of Chunked Transfer-Encoding 🧩
For simplicity, ICAP messages MUST use the chunked transfer-encoding inside the encapsulated body section, as defined in HTTP/1.1 [4] 🎀. This means ICAP client implementations have to convert incoming objects "on the fly" into chunked from whatever transfer-encoding they arrived with. Don't worry, the conversion is actually simple ✨:
- For objects arriving with
Content-Lengthheaders, you can make one big chunk the same size as theContent-Lengthheader says 📏. - For objects arriving with a TCP close to signal the end of the object, each group of bytes you read from the OS can become a chunk. Write the length of the bytes read, then the bytes themselves 🧁.
- For objects arriving with chunked encoding, they can be retransmitted as is, no re-chunking needed 🦋.
6.4 Distinct URIs for Distinct Services 🔖
ICAP servers SHOULD assign unique URIs to each service they provide, even if those services could theoretically be told apart by their method 💜. So a REQMOD service and a RESPMOD service should never share the same URI, even if they do something that's conceptually the same 🙅♀️.
This is kind of like HTTP, where in theory you could GET or POST to the same URI and expect two different results 🤯. Overloading URIs like that only causes confusion, so just don't do it 🌸.
7. Security Considerations 🔐
7.1 Authentication 🗝️
Authentication in ICAP is very similar to proxy authentication in HTTP, as specified in RFC 2617 🧠. Specifically, these rules apply:
WWW-Authenticatechallenges and responses are for end-to-end authentication between a client (user) and an origin server. Like any proxy, ICAP clients and ICAP servers MUST forward these headers without modification 📨.- If authentication is required between an ICAP client and an ICAP server, hop-by-hop Proxy Authentication as described in RFC 2617 MUST be used 🔒.
There are potential applications where a user (as opposed to an ICAP client) might have rights to access an ICAP service 👑. In this version of the protocol, we assume ICAP clients and ICAP servers are under the same administrative domain and contained in a single trust domain. So in these cases, it's enough for users to authenticate themselves to the ICAP client, which is a surrogate from the user's point of view 🫡. This type of authentication is also Proxy Authentication as described in RFC 2617 ✨.
This standard explicitly excludes any method for a user to authenticate directly to an ICAP server. The ICAP client MUST be involved as described above 🚨.
7.2 Encryption 🕵️♀️
Users of ICAP should note well that ICAP messages are not encrypted for transit by default 👀. Without some other form of encryption at the link or network layers, eavesdroppers may be able to record the unencrypted transactions between ICAP clients and servers 😱. As described in Section 4.3.1, the Upgrade header MAY be used to negotiate transport-layer security for an ICAP connection [5] 🔐.
End-to-end encryption between a client and an origin server is likely to rule out value-added services from intermediaries like surrogates 🙅♀️. An ICAP server that can't decrypt a client's messages will, of course, be unable to transform them at all 🥲.
7.3 Service Validation 🧐
Normal HTTP surrogates, when operating correctly, should not affect the end-to-end semantics of messages that pass through them 💅. That gives us a well-defined way to check that a surrogate is working right: a message should look the same before the surrogate as it does after 🪞.
ICAP is different, though, because it's meant to change the semantics of messages on their way from origin servers to users 🌪️. So the criteria for a correctly operating surrogate aren't as easy to define anymore. Validating ICAP services is going to be a lot harder as a result 😮💨. Bad adaptations may lead to security vulnerabilities that weren't there in the unadapted content 🚩.
8. Motivations and Design Alternatives 🧠
This section digs into some of our design decisions and the ideas and motivations behind them 📖. It doesn't define protocol requirements. It's just here to shed light on the requirements from earlier sections ✨. Nothing in this section carries the "force of law" or is part of the formal protocol specification, so it's just the tea 🍵.
Our guiding principle was to make ICAP the simplest possible protocol that would still do the job, and no simpler 💖. Some features were rejected where non-protocol-based solutions could be found 🙌. We also intentionally left a number of issues up to the implementor, since we believe that doesn't compromise interoperability 🦋.
8.1 To Be HTTP, or Not To Be 🤔
ICAP was first designed as an application-layer protocol built to run on top of HTTP 🌐. That made sense for a bunch of reasons. HTTP is well understood in the community and has tons of software infrastructure already built for it (clients, servers, parsers, etc.) 🛠️. Our early designs were all about leveraging that existing work. We hoped ICAP services could be implemented simply, using CGI scripts run by existing web servers 🧚.
But the devil (as always) was in the details 😩. Some features we thought were important were impossible to implement with HTTP. For example, ICAP clients can stop and wait for a "100 Continue" message in the middle of a message-body, but HTTP clients can only wait between the header and the body 🙃. On top of that, some transformations of HTTP messages by surrogates are legal (and harmless for HTTP) but caused problems with ICAP's "header-in-header" encapsulation and other features 🤦♀️.
In the end, the tangle of workarounds needed to squeeze ICAP into HTTP was more complex and confusing than just moving away from HTTP and defining a new (but similar) protocol 🦋✨.
8.2 Mandatory Use of Chunking 🍰
Chunking is mandatory in ICAP encapsulated bodies, for three reasons 💁♀️. First, efficiency matters, and chunked encoding lets both the client and the server keep the transport-layer connection open for later reuse ♻️. Second, ICAP servers (and their developers) should be encouraged to produce "incremental" responses where possible, which cuts down the latency users feel 🏃♀️. Chunked encoding is the only way to support that kind of implementation 🎯. Finally, by standardizing on a single encapsulation mechanism, we avoid the complexity client and server software would need to support multiple ones 🧼. That keeps ICAP simple, especially for the "body preview" feature described in Section 4.5 👑.
Chunking of encapsulated bodies is mandatory, but encapsulated headers are not chunked 🙈. There are two reasons for that. First, when a chunked HTTP message body is being encapsulated in an ICAP message, the ICAP client (HTTP server) can copy it straight from the HTTP client to the ICAP server without un-chunking and then re-chunking it 🚀. Second, a lot of header-parser implementations have trouble with headers that come in multiple chunks 😵. Earlier drafts of this document required that a chunk boundary not come within a header. For clarity, chunking of encapsulated headers has simply been disallowed 🚫.
8.3 Use of the null-body directive in the Encapsulated header 📏
There is a downside to not using the chunked transfer-encoding for the encapsulated header part of an ICAP message 😬. Parsers don't know in advance how much header data is coming (for example, for buffer allocation). ICAP doesn't allow chunking in the header part, for the reasons described in Section 8.2. To make up for it, the null-body directive lets the final header's length be determined, even though it isn't chunked 🧩✨.
9. References 📚
- [1] Berners-Lee, T., Fielding, R. and L. Masinter, "Uniform Resource Identifiers (URI): Generic Syntax and Semantics", RFC 2396, August 1998.
- [2] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
- [3] Resnick, P., "Internet Message Format", RFC 2822, April 2001.
- [4] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P. and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.
- [5] Khare, R. and S. Lawrence, "Upgrading to TLS Within HTTP/1.1", RFC 2817, May 2000.
10. Contributors 🌸
okay so ICAP started with an original idea from John Martin and Peter Danzig 💖 but so many people helped build it out, past and present, and they all deserve shoutouts 🫶
- Lee Duggs ✨
- Network Appliance, Inc., 495 East Java Dr., Sunnyvale, CA 94089 USA
- Phone: (408) 822-6000
- EMail: lee.duggs@netapp.com
- Paul Eastham 💅
- Network Appliance, Inc., 495 East Java Dr., Sunnyvale, CA 94089 USA
- Phone: (408) 822-6000
- EMail: eastham@netapp.com
- Debbie Futcher 🎀
- Network Appliance, Inc., 495 East Java Dr., Sunnyvale, CA 94089 USA
- Phone: (408) 822-6000
- EMail: deborah.futcher@netapp.com
- Don Gillies 🦋
- Network Appliance, Inc., 495 East Java Dr., Sunnyvale, CA 94089 USA
- Phone: (408) 822-6000
- EMail: gillies@netapp.com
- Steven La 🌈
- Network Appliance, Inc., 495 East Java Dr., Sunnyvale, CA 94089 USA
- Phone: (408) 822-6000
- EMail: steven.la@netapp.com
- John Martin 👑
- Network Appliance, Inc., 495 East Java Dr., Sunnyvale, CA 94089 USA
- Phone: (408) 822-6000
- EMail: jmartin@netapp.com
- Jeff Merrick 🧚
- Network Appliance, Inc., 495 East Java Dr., Sunnyvale, CA 94089 USA
- Phone: (408) 822-6000
- EMail: jeffrey.merrick@netapp.com
- John Schuster 🍓
- Network Appliance, Inc., 495 East Java Dr., Sunnyvale, CA 94089 USA
- Phone: (408) 822-6000
- EMail: john.schuster@netapp.com
- Edward Sharp ⭐
- Network Appliance, Inc., 495 East Java Dr., Sunnyvale, CA 94089 USA
- Phone: (408) 822-6000
- EMail: edward.sharp@netapp.com
- Peter Danzig 🔥
- Akamai Technologies, 1400 Fashion Island Blvd, San Mateo, CA 94404 USA
- Phone: (650) 372-5757
- EMail: danzig@akamai.com
- Mark Nottingham 💜
- Akamai Technologies, 1400 Fashion Island Blvd, San Mateo, CA 94404 USA
- Phone: (650) 372-5757
- EMail: mnot@akamai.com
- Nitin Sharma 🌸
- Akamai Technologies, 1400 Fashion Island Blvd, San Mateo, CA 94404 USA
- Phone: (650) 372-5757
- EMail: nitin@akamai.com
- Hilarie Orman 🦋
- Novell, Inc., 122 East 1700 South, Provo, UT 84606 USA
- Phone: (801) 861-7021
- EMail: horman@novell.com
- Craig Blitz ✨
- Novell, Inc., 122 East 1700 South, Provo, UT 84606 USA
- Phone: (801) 861-7021
- EMail: cblitz@novell.com
- Gary Tomlinson 💅
- Novell, Inc., 122 East 1700 South, Provo, UT 84606 USA
- Phone: (801) 861-7021
- EMail: garyt@novell.com
- Andre Beck 🎀
- Bell Laboratories / Lucent Technologies, 101 Crawfords Corner Road, Holmdel, New Jersey 07733-3030
- Phone: (732) 332-5983
- EMail: abeck@bell-labs.com
- Markus Hofmann 🌈
- Bell Laboratories / Lucent Technologies, 101 Crawfords Corner Road, Holmdel, New Jersey 07733-3030
- Phone: (732) 332-5983
- EMail: hofmann@bell-labs.com
- David Bryant 🫶
- CacheFlow, Inc., 650 Almanor Avenue, Sunnyvale, California 94086
- Phone: (888) 462-3568
- EMail: david.bryant@cacheflow.com
Appendix A. BNF Grammar for ICAP Messages 🎀
this grammar is written in augmented Backus-Naur Form (BNF), which is pretty much the same notation HTTP/1.1 uses (see Section 2.1 of [4]) 🤓 so implementors need to be comfy with the notation before reading on, iykyk
a lot of header values have the exact same grammar and semantics as in HTTP/1.1, so they aren't repeated here, bc why would we 💀
ICAP-Version = "ICAP/1.0"
ICAP-Message = Request | Response
Request = Request-Line
*(Request-Header CRLF)
CRLF
[ Request-Body ]
Request-Line = Method SP ICAP_URI SP ICAP-Version CRLF
Method = "REQMOD" ; Section 4.8
| "RESPMOD" ; Section 4.9
| "OPTIONS" ; Section 4.10
| Extension-Method ; Section 4.3.2
Extension-Method = token
ICAP_URI = Scheme ":" Net_Path [ "?" Query ] ; Section 4.2
Scheme = "icap"
Net_Path = "//" Authority [ Abs_Path ]
Authority = [ userinfo "@" ] host [ ":" port ]
Request-Header = Request-Fields ":" [ Generic-Field-Value ]
Request-Fields = Request-Field-Name
| Common-Field-Name
; Header fields specific to requests
Request-Field-Name = "Authorization" ; Section 4.3.2
| "Allow" ; Section 4.3.2
| "From" ; Section 4.3.2
| "Host" ; Section 4.3.2
| "Referer" ; Section 4.3.2
| "User-Agent" ; Section 4.3.2
| "Preview" ; Section 4.5
; Header fields common to both requests and responses
Common-Field-Name = "Cache-Control" ; Section 4.3.1
| "Connection" ; Section 4.3.1
| "Date" ; Section 4.3.1
| "Expires" ; Section 4.3.1
| "Pragma" ; Section 4.3.1
| "Trailer" ; Section 4.3.1
| "Upgrade" ; Section 4.3.1
| "Encapsulated" ; Section 4.4
| Extension-Field-Name ; Section 4.3
Extension-Field-Name = "X-" token
Generic-Field-Value = *( Generic-Field-Content | LWS )
Generic-Field-Content = <the OCTETs making up the field-value
and consisting of either *TEXT or
combinations of token, separators,
and quoted-string>
Request-Body = *OCTET ; See Sections 4.4 and 4.5 for semantics
Response = Status-Line
*(Response-Header CRLF)
CRLF
[ Response-Body ]
Status-Line = ICAP-Version SP Status-Code SP Reason-Phrase CRLF
Status-Code = "100" ; Section 4.5
| "101" ; Section 10.1.2 of [4]
| "200" ; Section 10.2.1 of [4]
| "201" ; Section 10.2.2 of [4]
| "202" ; Section 10.2.3 of [4]
| "203" ; Section 10.2.4 of [4]
| "204" ; Section 4.6
| "205" ; Section 10.2.6 of [4]
| "206" ; Section 10.2.7 of [4]
| "300" ; Section 10.3.1 of [4]
| "301" ; Section 10.3.2 of [4]
| "302" ; Section 10.3.3 of [4]
| "303" ; Section 10.3.4 of [4]
| "304" ; Section 10.3.5 of [4]
| "305" ; Section 10.3.6 of [4]
| "306" ; Section 10.3.7 of [4]
| "307" ; Section 10.3.8 of [4]
| "400" ; Section 4.3.3
| "401" ; Section 10.4.2 of [4]
| "402" ; Section 10.4.3 of [4]
| "403" ; Section 10.4.4 of [4]
| "404" ; Section 4.3.3
| "405" ; Section 4.3.3
| "406" ; Section 10.4.7 of [4]
| "407" ; Section 10.4.8 of [4]
| "408" ; Section 4.3.3
| "409" ; Section 10.4.10 of [4]
| "410" ; Section 10.4.11 of [4]
| "411" ; Section 10.4.12 of [4]
| "412" ; Section 10.4.13 of [4]
| "413" ; Section 10.4.14 of [4]
| "414" ; Section 10.4.15 of [4]
| "415" ; Section 10.4.16 of [4]
| "416" ; Section 10.4.17 of [4]
| "417" ; Section 10.4.18 of [4]
| "500" ; Section 4.3.3
| "501" ; Section 4.3.3
| "502" ; Section 4.3.3
| "503" ; Section 4.3.3
| "504" ; Section 10.5.5 of [4]
| "505" ; Section 4.3.3
| Extension-Code
Extension-Code = 3DIGIT
Reason-Phrase = *<TEXT, excluding CR, LF>
Response-Header = Response-Fields ":" [ Generic-Field-Value ]
Response-Fields = Response-Field-Name
| Common-Field-Name
Response-Field-Name = "Server" ; Section 4.3.3
| "ISTag" ; Section 4.7
Response-Body = *OCTET ; See Sections 4.4 and 4.5 for semanticsAuthors' Addresses 🦋
- Jeremy Elson 💜
- University of California Los Angeles, Department of Computer Science, 3440 Boelter Hall, Los Angeles CA 90095
- Phone: (310) 206-3925
- EMail: jelson@cs.ucla.edu
- Alberto Cerpa 🌸
- University of California Los Angeles, Department of Computer Science, 3440 Boelter Hall, Los Angeles CA 90095
- Phone: (310) 206-3925
- EMail: cerpa@cs.ucla.edu
ICAP discussion currently happens at icap-discussions@yahoogroups.com, and more info lives at http://groups.yahoo.com/group/icap-discussions/ 🧚
Full Copyright Statement
Copyright (C) The Internet Society (2003). All Rights Reserved. 🌈
you can copy and share this document, and translations of it too. you can also write derivative works that comment on or explain it, or help with implementing it, and publish and distribute those however you like, in whole or in part, no restrictions, as long as the copyright notice and this paragraph stay on every copy and derivative work 💖
the document itself can't be modified in any way, like removing the copyright notice or the references to the Internet Society or other Internet organizations. the one exception is when it's needed for developing Internet standards, in which case the Internet Standards process copyright procedures have to be followed. translating it into languages other than English is also allowed 🫶
these permissions are perpetual and won't be revoked by the Internet Society or its successors or assigns 👑
the document and the info in it are provided on an "AS IS" basis, and the following disclaimer applies:
THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 🙈
Acknowledgement ✨
Funding for the RFC Editor function is currently provided by the Internet Society. thank you, Internet Society, you're literally keeping the lights on 💅🌸
📦 where this page comes from
- ietf: rfc3507 2003-04
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms