RFC 3229

Delta encoding in HTTP

๐Ÿ“… published January 2002, from ietf

Okay so bestie, this is part 1 of 10 for RFC 3229, and it's the one that sets the whole vibe ๐ŸŽ€ Let's get into it!

Network Working Group โœจ

Author team: J. Mogul (Compaq WRL), B. Krishnamurthy and F. Douglis (AT&T), A. Feldmann (Univ. of Saarbruecken), Y. Goland and A. van Hoff (Marimba), and D. Hellerstein (ERS/USDA). That's a whole squad ๐Ÿซถ

Category: Standards Track ๐Ÿฆ‹ which means it's an actual Internet standards track protocol for the community, and it's still open for discussion and suggestions for improvements ๐Ÿ’ฌ

Date: January 2002 ๐ŸŒธ

Title: Delta encoding in HTTP ๐ŸŒˆโœจ

The full title block also lists the RFC number as 3229. This one doesn't obsolete or update any other RFCs, so there's no drama there, no cap ๐Ÿ’…


Status of this Memo ๐Ÿ“

Okay so this section is the official "what kind of doc is this" part. It's a standards track protocol for the Internet community, and it's basically asking everyone for feedback. For the current standardization state and status, check the "Internet Official Protocol Standards" (STD 1) doc. And distribution of this memo is unlimited, so share away ๐Ÿฅฐ

Copyright (C) The Internet Society (2002). All Rights Reserved. Legally boring, but it's gotta be here ๐Ÿ™ˆ

Abstract ๐Ÿงš

Lowkey the whole point: this doc describes how delta encoding can be supported as a compatible extension to HTTP/1.1 ๐Ÿ”ฅ

Here's the tea. Lots of HTTP requests end up pulling down resources the client already has a cache entry for, just slightly modified. And research says those updates are super frequent, with changes usually way smaller than the whole entity. So HTTP would use bandwidth way better if it sent a minimal description of the changes instead of the entire new version of the resource. That's what's called "delta encoding" โœจ It's giving efficiency, bestie ๐Ÿ’–

Table of Contents ๐ŸŽ€

1 Introduction....................................................  3
     1.1 Related research and proposals...........................  4
2 Goals...........................................................  5
3 Terminology.....................................................  6
4 The HTTP message-generation sequence............................  8
     4.1 Relationship between deltas and ranges................... 11
5 Basic mechanisms................................................ 13
     5.1 Background: an overview of HTTP cache validation......... 13
     5.2 Requesting the transmission of deltas.................... 14
     5.3 Choice of delta algorithm and format..................... 16
     5.4 Identification of delta-encoded responses................ 16
     5.5 Guaranteeing cache safety................................ 17
     5.6 Transmission of delta-encoded responses.................. 18
     5.7 Examples of requests combining Range and delta encoding.. 19
6 Encoding algorithms and formats................................. 22
7 Management of base instances.................................... 23
     7.1 Multiple entity tags in the If-None-Match header......... 24
     7.2 Hints for managing the client cache...................... 25
8 Deltas and intermediate caches.................................. 27
9 Digests for data integrity...................................... 28
10 Specification.................................................. 28
     10.1 Protocol parameter specifications....................... 28
     10.2 IANA Considerations..................................... 30
     10.3 Basic requirements for delta-encoded responses.......... 30
     10.4 Status code specifications.............................. 30
          10.4.1 226 IM Used...................................... 31
     10.5 Header specifications................................... 31
          10.5.1 Delta-Base....................................... 31
          10.5.2 IM............................................... 32
          10.5.3 A-IM............................................. 33
     10.6 Caching rules for 226 responses......................... 35
     10.7 Rules for deltas in the presence of content-codings..... 36
          10.7.1 Rules for generating deltas in the presence of
                 content-codings.................................. 37
          10.7.2 Rules for applying deltas in the presence of
                 content-codings.................................. 37
          10.7.3 Examples for using A-IM, IM, and content-codings. 38
     10.8 New Cache-Control directives............................ 40
          10.8.1 Retain directive................................. 40
          10.8.2 IM directive..................................... 40
     10.9 Use of compression with delta encoding.................. 41
     10.10 Delta encoding and multipart/byteranges................ 42
11 Quantifying the protocol overhead.............................. 42
12 Security Considerations........................................ 44
13 Acknowledgements............................................... 44
14 Intellectual Property Rights................................... 44
15 References..................................................... 44
16 Authors' addresses............................................. 47
17 Full Copyright Statement....................................... 49

1. Introduction ๐ŸŒธ

The World Wide Web is a distributed system, so it's always leaning on caching to cut down retrieval delays. Pulling a Web resource (like a document, image, icon, or applet) over the Internet or other wide-area networks usually takes long enough that you actually notice it, often seconds ๐Ÿ˜ญ Caching can often wipe that delay out or make it way smaller ๐Ÿ™Œ

Many Web resources change over time, so a practical caching approach needs a coherency mechanism so users don't get served stale info ๐Ÿ™ˆ HTTP originally had very little caching support, but under real-world pressure it quickly picked up a simple mechanism for keeping cache coherency. Iconic glow-up ๐Ÿ’…

In HTTP/1.0 [2], the server can include a "last-modified" timestamp with a response. If a client stores that response in a cache entry and wants to reuse it later, it can send a request with an "If-modified-since" field holding that timestamp. This is called a conditional retrieval ๐Ÿฆ‹ When the server gets a conditional request, it can either send a full response, or, if the resource hasn't changed, send a shorter reply saying the client's cache entry is still good โœจ HTTP/1.0 also has an "Expires" timestamp, which tells the client how long a response stays valid. If that's the case, the client can use its cached copy until then without validating it with a conditional retrieval first ๐Ÿงš

HTTP/1.1 [10] adds a ton of new features for cache coherency and performance. But it keeps the all-or-none model for responses to conditional retrievals: the server either says the resource hasn't changed at all, or it has to send the entire current value ๐Ÿ’€

Here's the thing tho: traces confirm that even when a Web resource does change, the new version is often super similar to the old one ๐Ÿ‘€ If the difference, or "delta", between the two versions got sent instead of the whole new thing, a client with a cached copy of the old one could apply the delta and rebuild the new version. With limited bandwidth, cutting the response size and delay could be a huge deal ๐Ÿ”ฅ

Think of deltas as a way to squeeze every drop of value out of client and proxy caches ๐Ÿ“ Instead of treating a whole response as the "cache line", deltas let us treat arbitrary pieces of a cached response as the replaceable unit, so we skip sending pieces that haven't changed. Main character energy ๐Ÿ‘‘

This document proposes a set of compatible extensions to HTTP/1.1 that let clients and servers use delta encoding with minimal overhead ๐ŸŽ€

We assume the reader is already familiar with the HTTP/1.1 specification, so no need to explain the basics again, periodt ๐Ÿ’–


Part 1 of 10. More parts coming, so stay tuned, bestie ๐Ÿซถ

okay so delta encoding is NOT a new idea bestie ๐Ÿ’… people have been doing the "only send what changed" thing for ages. The MPEG-1 video standard sends the occasional full still-image frame, but most frames are encoded (to oversimplify, obvi) as changes from an adjacent frame ๐ŸŽฌ SCCS and RCS [27], the software version control systems, store intermediate versions as deltas. SCCS starts with an original version and encodes later ones as forward deltas, while RCS encodes previous versions as reverse deltas from their successors ๐Ÿง  Jacobson's technique for compressing IP and TCP headers over slow links [17] is a clever, highly specialized form of delta encoding, and honestly iconic ๐Ÿซถ

In spite of all that history, it looks like it took several years before anyone thought to apply delta encoding to HTTP ๐Ÿ™ˆ probably because HTTP caching has been kind of haphazard in its development. The first published suggestion came from Williams et al. in a paper about HTTP cache removal policies [30], but those authors didn't go into their design until later [29] ๐ŸŒธ

The WebExpress project [15] seems to be the first published description of an implementation of delta encoding for HTTP, which they call "differencing" ๐Ÿ’œ WebExpress is aimed specifically at wireless environments and includes a bunch of orthogonal optimizations. It also doesn't propose changing the HTTP protocol itself. Instead it uses a pair of interposed proxies to convert the HTTP message stream into an optimized form โœจ The results they reported for WebExpress differencing are seriously impressive, but they're limited to a few selected benchmarks, so bestie, keep that in mind ๐Ÿงš

Banga et al. [1] describe optimistic deltas, where a layer of interposed proxies on either end of a slow link work together to reduce latency ๐Ÿฆ‹ If the client-side proxy already has a cached copy of a resource, the server-side proxy can simply send a delta (or a 304 [Not Modified] response). If only the server-side proxy has a cached copy, it may optimistically send its (possibly stale) copy to the client-side proxy, followed by a delta if necessary once the server-side proxy has validated its own cache entry with the origin server. Here's the tea: optimistic deltas, unlike delta encoding, actually increase the number of bytes sent over the network. They do it to improve latency by anticipating a "Not Modified" response from the origin server ๐Ÿ˜ญ The optimistic delta paper, like the WebExpress paper, didn't propose a change to the HTTP protocol itself, and it only reported results for a small set of selected URLs.

Mogul et al. [23] collected lengthy traces, at two different sites, of the full contents of HTTP messages to quantify the potential benefits of delta-encoded responses ๐Ÿ”ฅ They showed that delta encoding can provide remarkable improvements in response size and response delay for an important subset of HTTP content types. They proposed a set of HTTP extensions, but without the level of detail a specification needs. Douglis et al. [8] used the same full-content traces to quantify how fast resources change on the Web ๐Ÿ“

The HTTP Distribution and Replication Protocol (DRP), proposed to W3C by Marimba, Netscape, Sun, Novell, and At Home, aims to add a collection of new features to HTTP to support "the efficient replication of data over HTTP" [13] ๐Ÿ‘‘ One part of the DRP proposal is "differential downloading," which is basically a form of delta encoding. The original DRP proposal takes a different approach than the one described here, but a forthcoming revision of DRP will be revised to conform to the proposal in this document ๐Ÿ’…

Tridgell and Mackerras [28] describe the "rsync" algorithm, which does something similar to delta encoding ๐ŸŒˆ In rsync, the client breaks a cache entry into a series of fixed-size blocks, computes a digest value for each block, and sends the series of digest values to the server as part of its request. The origin server does the same block-based computation and returns only the blocks whose digest values differ. We believe it might be possible to support rsync using the "instance manipulation" framework described later in this document, but nobody has worked that out in any detail yet ๐Ÿงš


2 Goals ๐ŸŽ€

The goals of this proposal are:

  1. Reduce the mean size of HTTP responses, thereby improving latency and network utilization.
  2. Avoid any extra network round trips.
  3. Minimize the amount of per-request and per-response overheads.
  4. Support a variety of encoding algorithms and formats.
  5. Interoperate with HTTP/1.0 and HTTP/1.1.
  6. Be fully optional for clients, proxies, and servers.
  7. Allow moderately simple implementations.

The goals do NOT include: ๐Ÿ™…โ€โ™€๏ธ

  • Reducing the number of HTTP requests sent to an origin server.
  • Reducing the size of every HTTP message.
  • Increasing the cache-hit ratio of HTTP caches.
  • Allowing excessively simplistic implementations of delta encoding.
  • Delta encoding of request messages, or of responses to methods other than GET.

Nothing in this specification specifically precludes the use of a delta encoding for the body of a PUT request. However, no mechanism currently exists for the client to discover whether the server can interpret such messages, so we don't attempt to specify how they might be used ๐Ÿซ 


3 Terminology ๐Ÿฆ‹

HTTP/1.1 [10] defines the following terms:

resource        A network data object or service that can be
                identified by a URI, as defined in section 3.2.
                Resources may be available in multiple
                representations (e.g. multiple languages, data
                formats, size, resolutions) or vary in other ways.

entity          The information transferred as the payload of a
                request or response.  An entity consists of
                metainformation in the form of entity-header fields
                and content in the form of an entity-body, as
                described in section 7.

variant         A resource may have one, or more than one,
                representation(s) associated with it at any given
                instant.  Each of these representations is termed a
                `variant.' Use of the term `variant' does not
                necessarily imply that the resource is subject to
                content negotiation.

The dictionary definition for "entity" is "something that has separate and distinct existence and objective or conceptual reality" [21]. Unfortunately, the definition for "entity" in HTTP/1.1 is similar to the one used in MIME [12], based on a false analogy between MIME and HTTP ๐Ÿ™„

In MIME, electronic mail messages do have distinct and separate existences. MIME defines "entity" as something that "refers specifically to the MIME-defined header fields and contents of either a message or one of the parts in the body of a multipart entity."

In HTTP, however, a response message to a GET does not have a distinct and separate existence. Rather, it reflects the current state of a resource (or a variant, subject to a set of constraints). The HTTP/1.1 specification has no term to describe "the value that would be returned in response to a GET request at the current time for the selected variant of the specified resource." This leads to awkward wording in the HTTP/1.1 specification in places where this concept is necessary ๐Ÿ˜ฉ

To express this concept, we define a new term, for use in this document:

instance        The entity that would be returned in a status-200
                response to a GET request, at the current time, for
                the selected variant of the specified resource, with
                the application of zero or more content-codings, but
                without the application of any instance manipulations
                (see below) or transfer-codings.

It is convenient to think of an entity tag, in HTTP/1.1, as being associated with an instance rather than an entity. That is, for a given resource, two different response messages might include the same entity tag, but two different instances of the resource should never be associated with the same (strong) entity tag ๐Ÿท๏ธ

We will informally use the term "delta," in this document, to mean an HTTP response encoded as the difference between two instances ๐Ÿ’–

More formally, delta encodings are members of a potentially larger class of transformations on instances, which leads to this new term:

instance manipulation
                An operation on one or more instances which may
                result in an instance being conveyed from server to
                client in parts, or in more than one response
                message.  For example, a range selection or a delta
                encoding.  Instance manipulations are end-to-end, and
                often involve the use of a cache at the client.

For reasons that will become clear later on, it is convenient to think about subrange selection as a form of instance manipulation. In some contexts, compression might also be treated as an instance manipulation, rather than as a content-coding or transfer-coding. ๐Ÿง 

4. The HTTP message-generation sequence โœจ

okay bestie, HTTP/1.1 can do a bunch of different transformations on the body of a value ๐ŸŽ€ there are three main ones:

  • Content-coding ๐ŸŽ The spec says: "Content coding values indicate an encoding transformation that has been or can be applied to an entity. Content codings are primarily used to allow a document to be compressed or otherwise usefully transformed without losing the identity of its underlying media type and without loss of information. Frequently, the entity is stored in coded form, transmitted directly, and only decoded by the recipient." Content-codings are normally end-to-end, so once the sender applies one, only the ultimate recipient takes it off. An intermediate server is allowed to apply one too, when it makes sense ๐Ÿ’…
  • Transfer-coding ๐Ÿšš The spec says: "Transfer coding values are used to indicate an encoding transformation that has been, can be, or may need to be applied to an entity-body in order to ensure "safe transport" through the network. This differs from a content coding in that the transfer coding is a property of the message, not of the original entity." Transfer-codings are explicitly hop-by-hop. (Bonus, but it's a sneaky one: an intermediate proxy can store the transfer-coded version of a message as an optimization, as long as that doesn't clash with its externally visible function ๐Ÿฆ‹)
  • Ranges ๐Ÿฐ A client using the Range header can ask the server for one or more subranges of the instance instead of the whole thing. HTTP/1.1 only supports byte-ranges, though future extensions might add other kinds of range-specifiers, like chapters of a document ๐Ÿ“–

A client says it's willing to receive a content-coding by sending an Accept-Encoding header, which lists the content-codings it understands. It can optionally include info about which ones it prefers ๐Ÿซถ If a server uses any non-identity content-coding(s), it adds a Content-Encoding header field to the response, listing those content-codings in the order they were applied โœจ

RFC 2068 [9] didn't have an equivalent for negotiating transfer-codings, though it did include a "Transfer-Encoding" header for marking the response. A new TE header was later added to HTTP/1.1 [10], which is the Accept-Encoding header's transfer-coding cousin ๐Ÿ‘ฏ

In this document, we add new, optional message headers to support instance manipulations ๐Ÿงš A client says it's willing to receive an instance-manipulation by sending an A-IM header (short for "Accept-Instance-Manipulation", which is far too long to spell out ๐Ÿ™ˆ), which is basically the Accept-Encoding idea again. Similarly, a server lists the instance-manipulations it has applied using an IM header ๐Ÿ’œ

To see how delta encoding works on HTTP responses, you need to get how these transformations relate to each other. Conceptually, they get applied in this order:

  1. Upon receiving a GET request, the server uses the URI in the request to identify the requested resource.
  2. Optionally, it uses information from the request (and perhaps additional information) to select a variant of that resource.
  3. At this point, the server may apply a non-identity content-coding to the instance, or one might have been inherent in its generation. This also results in a Content-Encoding header.
  4. The result of the first three steps, at the time when the request is processed, is an instance. The instance includes a body (possibly empty) and possibly some instance headers. The entity tag, if any, is assigned at this point. That is, an entity tag is associated with an instance, NOT an entity.
  5. The server may then apply an instance-manipulation. For example, if the request included a Range header, the server may optionally produce a range response, consisting of the original set of headers, a Content-Range header, and the appropriate range(s) from the (possibly encoded) body. Delta encodings are instance-manipulations, and are computed at this stage.
  6. The result of the fifth step becomes the entity, consisting of entity headers and an entity body.
  7. The server may then apply a non-identity transfer-coding; on-the-fly compression could be done in this step. If so, a Transfer-Encoding header is added to the message.
  8. The results of the seventh step is the message, consisting of a message body (the transfer-coded version of the entity body), the entity headers, and additional response and general headers.
Note: Section 14.13 of the HTTP/1.1 specification [10] says "The Content-Length entity-header field indicates the size of the entity-body." In other words, Content-Length measures the length of an entity, not of an instance or of a variant. For example, if the message is a delta encoding, Content-Length gives the length of the delta encoding, not the length of the current instance. ๐Ÿค“

Here's the whole sequence as a diagram ๐Ÿ—บ๏ธ

    datatype        operation leading to next datatype
    ========        ==================================
    resource
                |   choose acceptable variant, if needed
                v
    variant
                |   apply content-coding, if any
                v

                |   compute/assign entity tag
                v
    instance
                |   apply instance manipulation, if any
                v      (delta encoding, range selection, etc.)
    entity-body
                |   apply transfer-coding, if any
                v
    message-body

This formalization of the HTTP message generation sequence hadn't been described before ๐ŸŒˆ but it's clear that Range selection has to happen after the entity tag is assigned and after any content-coding is applied, and before any transfer-coding. So this formalization is fully consistent with previous practice and specification โœจ

4.1. Relationship between deltas and ranges ๐ŸŽ€

If both Ranges and delta encodings are forms of instance manipulation, which should be applied first? ๐Ÿค” It depends on how the Range is being used!

Ranges serve two main purposes, and the requesting client gets to decide:

  1. to complete a partial response after a premature termination of a message transmission.
  2. to obtain just selected sections of an instance.

For the first use, Range has to be applied after any delta encoding, since the goal is to recover an intact copy of the delta-encoded instance. For the second use, Range has to be applied before any delta encoding, because otherwise the offsets in the Range request would be meaningless (the client generally can't know how a server's delta encoding maps instance byte offsets to entity byte offsets) ๐Ÿ˜ญ

So we need a way for the client to specify the order in which two or more instance-manipulations get applied ๐Ÿ’– This fits neatly into the A-IM header (see section 10.5.3), where the server is required to apply instance-manipulations in the order they're listed in the A-IM header. We also include a range literal in the set of registered instance-manipulations, so the client can say, by ordering it relative to the others, whether range selection happens before or after delta encoding ๐Ÿฆ‹

We also need a way for the server to say which order it applied two or more instance-manipulations in. This is part of the specification of the IM header (see section 10.5.2), which follows the same practice as the Content-Encoding header: the IM header lists the instance-manipulations in the order they were applied (including, perhaps, the special range literal) ๐Ÿ“‹

A similar issue comes up when Ranges are combined with compression ๐Ÿ—œ๏ธ If the client is using a Range to complete a partial response after a premature termination of a compressed message, the Range has to be applied after the compression. That works in unmodified HTTP/1.1, because the compression can be done as a content-coding. But if the client is using a Range to get selected sections of an instance, it would normally only be able to specify offsets in terms of the uncompressed variant. If the selected portion is big enough to be worth compressing, the client could request a compressed transfer-coding, but that's hop-by-hop and not the most efficient approach (especially if an HTTP/1.0 proxy is in the path) ๐Ÿ™ƒ

We can fix this by supporting compression as an instance-manipulation (as well as a content-coding or transfer-coding), and by using the new mechanism that lets the client specify that the compression instance-manipulation happens after the Range instance-manipulation ๐ŸŽ‰

This also lets the client control whether compression happens before or after delta encoding ๐Ÿ“ Some simple differencing algorithms (like the UNIX "diff" command) need their output compressed afterward to get the best results.

5. Basic mechanisms ๐ŸŒธ

In this section, we explain the concepts behind delta encoding ๐Ÿ’… This isn't meant as a formal specification of the proposed extensions; see section 10 for that.

5.1. Background: an overview of HTTP cache validation โœจ

okay so bestie, say your cache is holding a response and you want to know if it's still current ๐Ÿงš HTTP/1.1 lets the client do a "conditional GET" ๐Ÿ’… using one of two kinds of "cache validators." The classic one, which works in both HTTP/1.0 and HTTP/1.1, goes like this: the client can send the "If-Modified-Since" request-header with the "Last-Modified" timestamp (if any) that the server gave along with the response ๐Ÿ•ฐ๏ธ If the server's timestamp for the resource hasn't changed, it may send back a 304 (Not Modified) status code, which does not send the body of the resource ๐Ÿ™ˆ If the timestamp has changed, the server would normally send a 200 (OK) with a complete copy of the resource and a brand new Last-Modified timestamp ๐Ÿ’Œ

ngl, timestamps are kinda messy ๐Ÿ˜ญ because of the lack of timestamp resolution, if a resource changes twice during one second, the change might not be detectable at all. So HTTP/1.1 also lets the server attach an entity tag to a response ๐Ÿท๏ธ An entity tag is an opaque string that the server builds however it likes, and the protocol specification puts only a bare minimum of requirements on it. (Especially, a "strong" entity tag must change if the value of the resource changes ๐Ÿ”ฅ) Then the client can validate its cache entry by sending a conditional request with the "If-None-Match" request-header, presenting the entity tag that goes with the cached response ๐Ÿ’œ (The protocol defines a few other ways to send entity tags too, like the "If-Range" header, which is used for short-circuiting an otherwise necessary round trip ๐Ÿฆ‹) If the entity tag matches the server's current tag for the resource, the server should send a 304 (Not Modified) response. Otherwise, the server should send a 200 (OK) response along with a complete copy of the resource.

In the existing HTTP protocol (HTTP/1.0 or HTTP/1.1), a client sending a conditional request can get back either of two responses:

  • status = 200 (OK), with a full copy of the resource, because the server's copy is presumably different from the client's cached copy.
  • status = 304 (Not Modified), with no body, because the server's copy is presumably the same as the client's cached copy.

Informally, you could think of these as "deltas" of 100% and 0% of the resource, respectively ๐ŸŒˆ Note that these deltas are relative to one specific cached response. In other words, a client can't ask for a delta without saying, somehow, which two instances of a resource are being compared ๐Ÿ‘€ The "new" instance is implicitly whatever the current instance is that the server would return for an unconditional request, and the "old" instance is the one sitting in the client's cache. The cache validator (last-modified time or entity tag) is what tells the server which old instance we mean ๐ŸŽ€

5.2. Requesting the transmission of deltas ๐Ÿ“

okay so to actually support sending real deltas, an extension to HTTP/1.1 needs these four features:

  1. A way to mark a request as conditional.
  2. A way to specify the old instance, to which the delta will be applied by the client.
  3. A way to indicate that the client is able to apply one or more specific forms of delta encoding.
  4. A way to mark a response as being delta-encoded in a particular format.

Features one and two are already covered by HTTP/1.1 ๐Ÿ™Œ Having a conditional request-header (like "If-Modified-Since" or "If-None-Match") marks a request as conditional, and the value of that header uniquely identifies the old instance (ignoring the problem of last-modified timestamp granularity).

We'll put off talking about the fourth feature until section 5.6 โณ

The third feature is a way for the client to say it can apply deltas (other than the trivial 0% and 100% ones) ๐Ÿ’– That's done by sending a list of acceptable delta-encoding formats in a request-header field, specifically the "A-IM" header. Having that list in a conditional request tells the server the client can handle delta-encoded cache updates.

For example, a client might send this request:

GET /foo.html HTTP/1.1
Host: bar.example.net
If-None-Match: "123xyz"
A-IM: vcdiff, diffe, gzip

Here's what that request means ๐Ÿซถ

  • The client wants the current value of /foo.html.
  • It already has a cached response (instance) for that resource, whose entity tag is "123xyz".
  • It's willing to accept delta-encoded updates in either of two formats: "diffe" (that's output from the UNIX "diff -e" command) and "vcdiff". (Encoding algorithms and formats, like "vcdiff", are described in section 6 ๐ŸŒธ)
  • It's willing to accept responses that have been compressed with "gzip", whether or not they're also delta-encoded. (It might be useful to compress the output of "diff -e".) However, based on the mandatory ordering constraint in section 10.5.3, if delta encoding and compression both get applied, this "A-IM" request header says compression should be done last ๐Ÿ’…

If, in this example, the server's current entity tag for the resource is still "123xyz", it should just return a 304 (Not Modified) response, same as a traditional server would โœจ

If the entity tag has changed (presumably, but not necessarily, because the resource was modified), the server could instead compute the delta between the instance whose entity tag was "123xyz" and the current instance ๐Ÿฆ‹

We'll talk about what the server needs to store in order to compute deltas later, in section 7 ๐Ÿ“ฆ

Now, if a client says it's willing to accept deltas but the server doesn't support this kind of instance manipulation, the server will simply ignore that part of the request ๐Ÿ™ˆ (HTTP always lets an implementation ignore a header that isn't required by a specification the implementation complies with, and the "A-IM" spec lets the server ignore an instance-manipulation it doesn't understand.) So if a server either doesn't implement the A-IM header at all, or doesn't implement any of the instance manipulations listed in it, it acts as if the client never asked for a delta-encoded response. The server generates a status-200 response instead ๐Ÿ’Œ

5.3. Choice of delta algorithm and format ๐ŸŽ€

The server is not required to send a delta-encoded response ๐Ÿ™…โ€โ™€๏ธ For example, the result might end up bigger than the current size of the resource. The server might not be able to compute a delta for this type of resource (like a compressed binary format). It might not have enough CPU cycles for the delta computation. It might not support any of the delta formats the client supports. Or the network bandwidth might be high enough that the delay from computing the delta isn't worth the delay it saves by sending a smaller response ๐ŸŒˆ

But, if the server does want to compute a delta, and the set of encodings it supports has more than one encoding in common with the set the client offered, which one should it use? That's mostly up to the server, although the client can express preferences using "Quality Values" (or "qvalues") in the "A-IM" header ๐Ÿ’œ The HTTP/1.1 specification [10] describes qvalues in more detail. (Clients might prefer one delta encoding format over another that produces a smaller encoding, if decoding costs for the first format are lower and the client is resource-constrained ๐Ÿ“)

Server implementations have a few possible approaches ๐Ÿ‘‘ For example, if CPU cycles are plentiful and network bandwidth is scarce, the server might compute every possible encoding and then send the smallest result. Or the server might use heuristics to pick an encoding format, based on things like the content-type of the resource, its current size, and the expected amount of change between instances of the resource โœจ

Also, it might be worth caching the deltas inside the server ๐Ÿง  if a resource gets requested by several different delta-capable clients between modifications. In that case, the cost of computing a delta can be spread out over many responses, so the server might go for a more expensive computation.

5.4. Identification of delta-encoded responses ๐Ÿฆ‹

A response that uses delta encoding must be identified as such ๐Ÿท๏ธ That's done with the "IM" response-header, which is specified in section 10.5.2.

However, a simplistic use of this approach would cause serious problems if a delta-encoded response flows through an intermediate (proxy) cache that doesn't know about the delta mechanism ๐Ÿ˜ฑ The Internet still has a significant number of HTTP/1.0 caches, which might never be fully replaced, and the HTTP specifications insist that message recipients ignore any header field they don't understand. So a non-delta-capable proxy cache that receives a delta-encoded response might store it, and might later hand it to a non-delta-capable client that requested the same resource. That naive client would believe it had received a valid copy of the entire resource, with predictably unpleasant results ๐Ÿ’€

To fix this, we propose that delta-encoded responses (actually, all instance-manipulated responses) get identified with a new HTTP status code ๐ŸŽ‰ For clarity in the discussion that follows, we'll use the (currently unassigned) code 226, with a reason phrase of "IM Used". (We see no benefit in spelling out "Instance Manipulation Used", since that just means sending unnecessary bytes, and this Reason-phrase shouldn't normally be seen by human users ๐Ÿ™ˆ) There's some precedent for this approach: HTTP/1.1 introduces the 206 (Partial Content) status code for sending sub-ranges of a resource. Existing proxies apparently forward responses with unknown status codes, and don't try to cache them ๐ŸŒธ

An alternative to a new status code would be to use the "Expires" header to stop HTTP/1.0 caches from storing the response, and then use Cache-Control: max-age (defined in HTTP/1.1) to let more modern caches store delta-encoded responses. But that adds a lot of bytes to the response headers, which would reduce how effective delta encoding is ๐Ÿ“‰ It's also not entirely clear that this approach stops all caching by all HTTP/1.0 proxies ๐Ÿค”

We were reluctant to define an additional status code as part of the support for delta encoding. However, we don't see any other efficient way to stay compatible with the deployed base of HTTP/1.0 cache implementations ๐Ÿฅบ

5.5. Guaranteeing cache safety ๐Ÿ›ก๏ธ

okay so this one is about making sure caches don't mess up a 226 response ๐Ÿ˜ญ Although we aren't aware of any HTTP/1.1 proxy implementations that would try to cache a response with an unknown 2xx status code, the HTTP/1.1 spec does allow that behavior if the response has an Expires or Cache-Control header that explicitly allows caching. And that would be a problem when a 226 (IM Used) response carries those headers ๐Ÿ’€

The fix is to use the Cache Control Extensions mechanism from HTTP/1.1. We define a brand new cache-directive, "im" ๐Ÿฆ‹ It says the "no-store" cache-directive may be ignored by implementations that follow the spec for the IM and A-IM headers.

For example, this response:

HTTP/1.1 226 IM Used
ETag: "489uhw"
IM: vcdiff
Date: Tue, 25 Nov 1997 18:30:05 GMT
Cache-Control: no-store, im, max-age=30

...

"MUST NOT" be stored by a cache that complies with the HTTP/1.1 specification, which says the max-age cache-directive "implies that the response is cacheable [...] unless some other, more restrictive cache directive is also present." ๐Ÿ™…โ€โ™€๏ธ BUT a cache that does follow the spec for the im cache-directive (so it follows the spec for the A-IM and IM header fields, and the 226 status code) ignores the no-store directive. So it sees max-age as allowing caching โœจ

We are not entirely sure that all HTTP/1.1 caches obey the rule that the max-age directive is overridden by the no-store directive. If operational testing reveals this to be a problem, more elaborate solutions are possible.
Warning to origin server implementors: it does not suffice to send Vary: If-None-Match, A-IM in status-226 responses. We have discovered at least one scenario where this does not prevent a proxy cache that does not implement IM and A-IM from incorrectly "validating" a cached 226 response. ๐Ÿ˜ญ Not the vibe, so don't rely on it!

5.6. Transmission of delta-encoded responses ๐Ÿ“ฆ

a delta-encoded response is different from a standard response in four ways, bestie:

  1. It carries a status code of 226 (IM Used). ๐ŸŽ€
  2. It carries an "IM" response-header field, which tells you which delta encoding is used in this response. ๐ŸŒธ
  3. Its message-body is a delta encoding of the current instance, instead of a full copy of the instance. ๐Ÿงš
  4. It might carry several other new headers, as described later in this document. โœจ

For example, a response to the request given in section 5.2 might look like:

HTTP/1.1 226 IM Used
ETag: "489uhw"
IM: vcdiff
Date: Tue, 25 Nov 1997 18:30:05 GMT

...

(We do not show the actual contents of the response body, since this is a binary format. ๐Ÿ™ˆ)

Note: the Etag header in a 226 response with a delta encoding provides the entity tag of the current instance of the resource variant. It is not meaningful to associate an entity tag with the delta value, which is not an instance. ๐Ÿ’…

5.7. Examples of requests combining Range and delta encoding ๐Ÿ“

In the example used in section 5.2, the client sends:

GET /foo.html HTTP/1.1
Host: bar.example.net
If-None-Match: "123xyz"
A-IM: vcdiff, diffe, gzip

and the server either responds with a 304 (Not Modified) response, or with the appropriate delta encoding. ๐Ÿ’–

Here are a few more examples to clarify how the client request should be interpreted. ๐Ÿ‘‘

If the client sends:

GET /foo.html HTTP/1.1
Host: bar.example.net
If-None-Match: "123xyz"
A-IM: vcdiff, diffe, gzip, range
Range: bytes=0-99

then the meaning is the same as in the example above, except that after the delta encoding (and compression, if any) is computed, the server then returns only the first 100 bytes of the output of the delta encoding. (If it is shorter than 100 bytes, the entire delta encoding is returned.) Because the "range" token appears last in the "A-IM" header, this tells the origin server to apply any range selection after the other instance-manipulations. ๐Ÿฆ‹

The interaction between the If-Range mechanism and delta encoding is somewhat complex. (If-Range means, informally, "if the entity is unchanged, send me the part(s) that I am missing; otherwise, send me the entire new entity." ๐Ÿง ) Here is an example that should clarify the use of this combination.

Suppose that the client wants to have the complete current instance of http://bar.example.net/foo.html. It already has a (complete) cache entry for this URI, with entity tag "A", so it issues this request:

GET /foo.html HTTP/1.1
host: bar.example.net
If-None-Match: "A"
A-IM: vcdiff

Suppose that the server's current instance has entity tag "B", and that the server also has retained a copy of the instance with entity tag "A". Then, the server could compute the difference between "B" and "A", and respond with:

HTTP/1.1 226 IM Used
Etag: "B"
IM: vcdiff
Date: Tue, 25 Nov 1997 18:30:05 GMT
Content-Length: 1000

...

but the network connection is terminated after the client has received exactly 900 bytes of the message body for the delta-encoded content. ๐Ÿ˜ญ Rip.

The client wants to retrieve the remaining 100 bytes of the delta encoding that was being sent in the interrupted response. It therefore should send:

GET /foo.html HTTP/1.1
host: bar.example.net
If-None-Match: "A"
If-Range: "B"
A-IM: vcdiff,range
Range: bytes=900-

This rather elaborate request has a well-defined meaning, which depends on the current entity tag Tcur of the instance when the server receives the request. ๐Ÿงฉ

  • Tcur = "A" (i.e., for some reason, the instance has reverted to the value already in the client's cache). The server should return a 304 (Not Modified) response, as required by the HTTP/1.1 specification for "If-None-Match". ๐Ÿ”
  • Tcur = "B" (i.e., the instance has not changed again). The HTTP/1.1 specification for "If-None-Match", in this case, is that the header field is ignored (by a server that does not understand delta encoding). Therefore, this is equivalent to the client's previous request, except that the Range selection is applied after the vcdiff instance manipulation (if both are to be applied). So the (delta-aware) server again computes the delta between the "A" instance and the "B" instance (or uses a cached computation of the delta), then applies the Range selection, and returns a 226 (IM Used) response, with an message-body containing bytes 900 to 999 of the result of the vcdiff encoding, with an IM:vcdiff,range response header. ๐ŸŽฏ
  • Tcur = "C" (i.e., the instance has changed again). In this case, the HTTP/1.1 specification for "If-None-Match" again means that this is equivalent to an unconditional request for the current instance. The specification for "If-Range" requires the server to return the entire current instance. However, a delta-aware server can construct the delta between the "A" instance described by the "If-None-Match" field and the current ("C") instance, and return a 226 (IM Used) response, with an IM:vcdiff response header. ๐ŸŒˆ

If the client's request had not included the If-None-Match: "A" header field, the server could not have computed a delta, since it would not have known which entire instance was already available to the client. ๐Ÿค” If the request had not included the If-Range: "B" header field, the server could not have distinguished between the latter two cases (Tcur = "B" or Tcur = "C") and would not have been able to apply the Range selection to the result of delta encoding. Main character energy for those headers, no cap ๐Ÿ’…

On the other hand, suppose that the client has a cache entry for the "A" instance of http://bar.example.net/foo.html, and it has already received the first 900 bytes of a new instance "B" (perhaps as the result of an aborted transfer). Now the client wants to receive the entire current instance, so it could send this request:

GET /foo.html HTTP/1.1
host: bar.example.net
If-None-Match: "A"
If-Range: "B"
A-IM: range,vcdiff
Range: bytes=900-

In this example, as in the previous example, if Tcur = "A" then the server should send 304 (Not Modified), and if Tcur = "C", then the server should send the entire new instance, either as a 200 response or as a delta encoding against instance "A". โœจ

However, if Tcur = "B", in this case the server should first select the specified range (bytes 900 through the end) from both instances "A" and "B", then compute the delta encoding between these ranges (using vcdiff), and then transmit the result using a 226 (IM Used) response with an IM:range,vcdiff response header. ๐Ÿซถ It's giving precision.

6 Encoding algorithms and formats โœจ

okay so the literature has a whole buffet of delta encoding algorithms and formats ๐Ÿ“ here's the tea:

diff -e         The UNIX "diff" program is ubiquitously available,
                and is relatively fast for both encoding and decoding
                (decoding is actually done using the "ed" program).
                However, the size of the resulting deltas is
                relatively large.  This algorithm can only be used on
                text-format files.

diff -e | gzip  Running the output of "diff" through a compression
                algorithm such as "gzip" [5] (or, perhaps better,
                "deflate" [7, 6]) yields a more compact encoding, but
                the costs of encoding and decoding are much higher
                than for "diff" by itself.  This algorithm can only
                be used on text-format files.

vcdiff (vdelta) The algorithm that generates the "vcdiff" format [19,
                20] inherently compresses its output, and generally
                produces smaller results than the combination of
                "diff" and "gzip".  The algorithm also runs much
                faster, and can be applied to binary-format input.
                The "vcdiff" format is based on previous work on an
                algorithm named "vdelta."  (Note that the "vcdiff"
                format can be used either for delta encoding or as a
                compressed format, so two different instance-
                manipulation values would have to be registered in
                order to distinguish these two uses, should its use
                as a compressed format be adopted.)  The most recent
                published study suggests that "vdelta" is the best
                overall delta algorithm [16].

gdiff           The gdiff format [14] was specified as a generic,
                algorithm-independent format for expressing deltas.
                Because it is more generic it is easy to implement,
                but it may not be the most compact encoding format.

Our proposal does not recommend any specific algorithm or format, but rather encourages client and server implementors to choose the most appropriate one(s) ๐Ÿ’–. However, to avoid the possibility of excessively long "A-IM" headers, we suggest that, after some period of experimentation, it might be reasonable to specify a "recommended" set of delta formats for general-purpose HTTP implementations ๐Ÿซถ.

We suspect that it should be possible to devise a delta encoding algorithm appropriate for use on typical image encodings, such as GIF and JPEG ๐Ÿฆ‹. Although experiments with vdelta have not shown much potential [23], this may simply be because these experiments used vdelta directly on the already-compressed forms of these encodings. However, it might be necessary to devise a delta encoding algorithm that is aware of the two-dimensional nature of images. We have some expectation that this is possible, since MPEG compression relies on computing deltas between successive frames of a video stream ๐ŸŽ€.


7 Management of base instances ๐Ÿงš

okay this section is about the server's "base instances" ๐Ÿ‘‘ so, if a resource changes more often than a client's cache throws stuff out, then the "old instance" in a client's conditional request might not be the most recent prior instance ๐Ÿ™ˆ that raises a big question: how many old instances should the server keep, if any? we call these old instances "base instances," and there are lots of options for server folks. for example:

  • The server might not store any old instances, and so would never respond with a delta.
  • The server might only store the most recent prior instance; requests attempting to validate this instance could be answered with a delta, but requests attempting to validate older instances would be answered with a full copy of the resource.
  • The server might store all prior instances, allowing it to provide a delta response for any client request.
  • The server might store only a subset of the prior instances. The use of a Least Recently Used (LRU) algorithm to determine this kind of subset has proved effective in some similar circumstances, such as cache replacement.

The server might not have to store prior instances explicitly ๐Ÿ’…. It might, instead, store just the deltas between specific base instances and subsequent instances (or the inverse deltas between base instances and prior instances). This approach might be integrated with a cache of computed deltas.

None of these approaches necessarily requires additional protocol support. However, if a server administrator wants to store only a subset of the prior instances, but would like the server to be able to respond using deltas as often as possible, then the client needs some additional information. Otherwise, the client's "If-None-Match" header might specify a base instance not stored at the server, even though an appropriate base instance is held in the client's cache ๐Ÿ˜ญ.

We identify two additional protocol changes to help solve this problem.

7.1 Multiple entity tags in the If-None-Match header

okay so the earlier examples only showed one entity tag in an If-None-Match header, but HTTP/1.1 actually lets the header carry more than one entity-tag ๐ŸŒˆ. This feature was put in HTTP/1.1 to support efficient caching of multiple variants of a resource, but it's not limited to that use.

say a client has kept more than one instance of a resource in its cache. That is, not only does it keep the most recent instance, but it also holds onto copies of one or more prior, invalid instances. (Alternatively, it might retain sufficient delta or inverse-delta information to reconstruct older instances.) In this case, it could use its conditional request to tell the server about all of the instances it could apply a delta to. For example, the client might send:

GET /foo.html HTTP/1.1
host: bar.example.net
If-None-Match: "123xyz", "337pey", "489uhw"
A-IM: vcdiff

to indicate that it has three instances of this resource in its cache ๐Ÿง. If the server is able to generate a delta from any of these prior instances, it can select the appropriate base instance, compute the delta, and return the result to the client โœจ.

In this case, however, the server must also tell the client which base instance to use, and so we need to define a response header, named "Delta-Base", for this purpose ๐Ÿ’œ. For example, the server might reply:

HTTP/1.1 226 IM Used
ETag: "1acl059"
IM: vcdiff
Delta-Base: "337pey"
Date: Tue, 25 Nov 1997 18:30:05 GMT

This response tells the client to apply the delta to the cached response with entity tag "337pey", and to associate the entity tag "1acl059" with the result ๐Ÿฆ‹.

Of course, if the server has retained more than one of the prior instances identified by the client, this could complicate the problem of choosing the optimal delta to return, since now the server has a choice not only of the delta format, but also of the base instance to use ๐Ÿ™ƒ.

7.2 Hints for managing the client cache

so, supporting multiple entity tags for choosing the base instance means a client might benefit from storing multiple old instances of a resource in its cache ๐Ÿ“. A client with finite space would not want to keep all old instances, so it must manage its cache for maximal effectiveness by saving those instances most likely to be useful for future deltas ๐Ÿซถ. Although this could be done using information purely local to the client (e.g., an LRU algorithm), certain "hint" information from the server could improve the client's ability to manage its cache. The use of hints for improving Web cache performance has been described previously [4, 22].

If the server intends to retain certain instances and not others, it can label the responses that transmit the retained instances ๐Ÿท๏ธ. This would help the client manage its cache, since it would not have to retain all prior instances on the possibility that only some of them might be useful later. The label is a hint to the client, not a promise that the server will indefinitely retain an instance.

We propose adding a new directive to the existing "Cache-Control" header for this purpose, named "retain" ๐ŸŒธ. For example, in response to an unconditional request, the server might send:

HTTP/1.1 200 OK
ETag: "337pey"
Date: Tue, 25 Nov 1997 18:30:05 GMT
Cache-Control: retain

to suggest that a delta-capable client should retain this instance. The "retain" directive could also appear in a delta response, referring to the current instance:

HTTP/1.1 226 IM Used
ETag: "1acl059"
Date: Tue, 25 Nov 1997 18:30:05 GMT
Cache-Control: retain
IM: vcdiff
Delta-Base: "337pey"

The "retain" directive includes an optional timeout parameter, which the server can use if it expects to delete an old base instance at a particular time. For example,

HTTP/1.1 200 OK
ETag: "337pey"
Date: Tue, 25 Nov 1997 18:30:05 GMT
Cache-Control: retain=3600

means that the server intends to retain this base instance for one hour โฐ.

Another situation where a server can provide a hint to a client is where the server supports the delta mechanism in general, but does not intend to provide delta-encoded responses for a particular resource ๐Ÿ™…. By sending a "retain=0" directive, it indicates that the client should not waste request-header bytes attempting to obtain a delta-encoded response using this base instance (and, by implication, for this resource). It also indicates that the client ought not waste cache space on this instance after it has become stale. To avoid wasting response-header bytes, a server ought not send "retain=0", except in reply to a request that attempts to obtain a delta-encoded response ๐Ÿ’€.

Note that the "retain" directive is orthogonal to the "max-age" directive. The "max-age" directive indicates how long a cache entry remains fresh (i.e.,can be used without contacting the origin server for revalidation); the "retain" directive is of interest to a client AFTER the cache entry has become stale. ๐ŸŒˆ

In practice, the "Cache-Control" response-header field might already be present, so the cost (in bytes) of sending this directive might be smaller than these examples implies ๐ŸŽ€.

8. Deltas and intermediate caches โœจ

okay so we designed delta-encoded responses so that naive proxy caches won't store them ๐Ÿ™ˆ but if a proxy does understand the delta mechanism, it can be lowkey amazing and actually join in on sending and receiving deltas ๐Ÿ’–

there are a few independent ways a proxy could participate:

  • Besides forwarding a delta-encoded response, the proxy might store it and then use it to answer a later request with a compatible If-None-Match field (one that's either a superset of the field from the request that first got the response, or one that includes the Delta-Base value in the cached response). It also needs a compatible IM response-header field (one that includes the actual delta-encoding format used in the response). Obviously all the other HTTP rules about whether a cache entry is valid still apply ๐Ÿงš
  • Besides forwarding a delta-encoded response, the proxy might apply the delta to the matching entry in its own cache. That updated entry could then be used for later responses, even for clients that can't do deltas ๐ŸŒธ
  • When the proxy gets a conditional request from a delta-capable client, and it has a complete copy of an up-to-date ("fresh," in HTTP/1.1 terminology) response in its cache, it could generate a delta locally and send it back to the client ๐Ÿ‘‘
  • When the proxy gets a request from a non-delta-capable client, it might turn that into a delta request before forwarding it to the server. Then, after applying the resulting delta response to one of its own cache entries, it would return a full-body response to the client (or a response with status code 206 or 304, whichever fits) ๐Ÿฆ‹

all of these optional tricks make proxy software more complicated, and might need more storage or CPU. BUT if they're done carefully, they should help cut down the latency end users feel and the load on the network ๐Ÿซถ Generally, CPU speed and disk costs are improving faster than network latencies, so we expect complex proxy implementations to keep getting more valuable ๐Ÿ”ฅ


9. Digests for data integrity ๐Ÿ”

when a recipient puts a complete HTTP response back together from several separate messages, it might be necessary to check that the whole thing is intact ๐Ÿ˜ญ For example, the client's cache might be corrupt, or the delta encoding (at the client or the server) might have a bug ๐Ÿ’€

HTTP/1.1 already has ways to check the integrity of individual messages. A message may include a Content-MD5 response header, which gives an MD5 message digest of the body of the message (but not the headers). The Digest Authentication mechanism [11] gives a similar message-digest function, except that it also covers certain header fields. Neither one does anything for a set of data sent over several messages, which is exactly what happens with the result of applying a delta-encoded response (or, for that matter, a Range response) ๐Ÿ™ƒ

so data integrity for reassembled messages needs a brand-new message header. A mechanism like that is proposed in a separate document [24]. You might still want to use the Digest Authentication mechanism, or something stronger, to protect delta messages against tampering ๐Ÿ”’


10. Specification ๐Ÿ’…

in this specification, the key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" are to be interpreted as described in RFC 2119 [3] ๐ŸŽ€

10.1. Protocol parameter specifications

this spec defines a new HTTP parameter type, the instance-manipulation ๐Ÿง

instance-manipulation = token [imparams]

imparams = ";" imparam-name [ "=" ( token | quoted-string ) ]
imparam-name = token

heads up: the imparam-name MUST NOT be "q", so it doesn't get mixed up with qvalues (see [10]) โš ๏ธ

the instance-manipulation values start out as:

  • vcdiff A delta using the "vcdiff" encoding format [19, 20] โœจ
  • diffe The output of the UNIX "diff -e" command [26] ๐Ÿš
  • gdiff The GDIFF encoding format [14] ๐ŸŒˆ
  • gzip Same definition as the HTTP "gzip" content-coding ๐Ÿ—œ๏ธ
  • deflate Same definition as the HTTP "deflate" content-coding ๐Ÿซง
  • range A token that says the result is partial content, as the result of a range selection โœ‚๏ธ
  • identity A token used only in the A-IM header (not in the IM header), to say whether or not the identity instance-manipulation is acceptable ๐Ÿ™Œ

for convenience later in the spec, we define a subset of these as delta-coding values ๐Ÿง 

delta-coding = "vcdiff" | "diffe" | "gdiff" | token

future instance-manipulation values might also get added to this list ๐Ÿ”ฎ

10.2. IANA Considerations

the Internet Assigned Numbers Authority (IANA) runs the name space for instance-manipulation values ๐Ÿ›๏ธ Values and their meaning must be documented in an RFC or other peer-reviewed, permanent, and readily available reference, with enough detail that interoperability between independent implementations is possible. Subject to those constraints, name assignments are First Come, First Served (see RFC 2434 [25]) ๐Ÿฅ‡

this spec also adds a new value to the IANA HTTP Status Code Registry (see RFC 2817 [18]) ๐Ÿ“‹ The details for that code are in section 10.4.1 ๐Ÿ‘€

10.3. Basic requirements for delta-encoded responses

a server MAY send a delta-encoded response if all of these are true:

  1. The server would be able to send a 200 (OK) response for the request.
  2. The client's request includes an A-IM header field listing at least one delta-coding.
  3. The client's request includes an If-None-Match header field listing at least one valid entity tag for an instance of the Request-URI (a "base instance") ๐Ÿซถ

a delta-encoded response:

  • MUST carry a status code of 226 (IM Used) ๐ŸŽฏ
  • MUST include an IM header field listing, at least, the delta-coding employed ๐Ÿ“
  • MAY include a Delta-Base header field listing the entity tag of the base-instance ๐Ÿท๏ธ

10.4. Status code specifications

the following new status code is defined for HTTP ๐Ÿ†•

10.4.1. 226 IM Used

the server has fulfilled a GET request for the resource, and the response is a representation of the result of one or more instance-manipulations applied to the current instance ๐ŸŽ The actual current instance might not be available except by combining this response with other previous or future responses, as appropriate for the specific instance-manipulation(s). If that's the case, the headers of the resulting instance come from combining the headers from the status-226 response and the other instances, following the rules in section 13.5.3 of the HTTP/1.1 specification [10] ๐Ÿงฉ

the request MUST have included an A-IM header field listing at least one instance-manipulation. The response MUST include an Etag header field giving the entity tag of the current instance ๐Ÿ’ฏ

a response received with a status code of 226 MAY be stored by a cache and used to answer a later request, subject to the HTTP expiration mechanism, any Cache-Control headers, and the requirements in section 10.6 ๐Ÿ—ƒ๏ธ

a response received with a status code of 226 MAY be used by a cache, together with a cache entry for the base instance, to create a cache entry for the current instance ๐Ÿช„

10.5. Header specifications

the following headers are defined for use as entity-headers. (Because of the terminology mix-up discussed in section 3, some entity-headers are more properly associated with instances than with entities ๐Ÿค“)

10.5.1. Delta-Base

the Delta-Base entity-header field is used in a delta-encoded response to specify the entity tag of the base instance ๐Ÿ”–

Delta-Base = "Delta-Base" ":" entity-tag

a Delta-Base header field MUST be included in a response with an IM header that includes a delta-coding, if the request included more than one entity tag in its If-None-Match header field ๐Ÿšจ

any response with an IM header that includes a delta-coding MAY include a Delta-Base header ๐Ÿ™†

we aren't aware of other cases where a delta-encoded response MUST or SHOULD include a Delta-Base header, but we haven't done an exhaustive or formal analysis. Implementors might be wise to include a Delta-Base header in every delta-encoded response ๐Ÿง

a cache or proxy that gets a delta-encoded response lacking a Delta-Base header MAY add one whose value is the entity tag given in the If-None-Match field of the request (but only if that field lists exactly one entity tag) ๐Ÿ› ๏ธ

10.5.2. IM

the IM response-header field is used to indicate the instance-manipulations, if any, that have been applied to the instance represented by the response ๐Ÿชž Typical instance manipulations include delta encoding and compression ๐Ÿ—œ๏ธ

IM = "IM" ":" #(instance-manipulation)

instance-manipulations are defined in section 10.1 ๐Ÿ“–

as a special case, if the instance-manipulations include both range selection and at least one other non-identity instance-manipulation, the IM header field MUST be used to indicate the order in which all of these instance-manipulations, including range selection, were applied ๐Ÿงฎ If the IM header lists the "range" instance-manipulation, the response MUST include either a Content-Range header or a multipart/byteranges Content-Type in which each part contains a Content-Range header. (See section 10.10 for specific discussion of combining delta encoding and multipart/byteranges ๐ŸŽ€)

responses that include an IM header MUST carry a response status code of 226 (IM Used), as specified in section 10.4.1 ๐ŸŽฏ

the server SHOULD omit the IM header if it would list only the "range" instance-manipulation. Such responses would normally be sent with response status code 206 (Partial Content), as specified by HTTP/1.1 [10] โœ‚๏ธ

here are some examples of the IM header in action ๐Ÿ‘‡

IM: vcdiff

this one says the entity-body is a delta encoding of the instance, using vcdiff ๐Ÿฆ‹

IM: diffe, deflate, range

this one says the instance was first delta-encoded using diffe, then that result was compressed with deflate, and finally one or more ranges of that compressed encoding were selected ๐Ÿ“

IM: range, vcdiff

this one says one or more ranges of the instance were selected, and the result was then delta encoded against identical ranges of a previous base instance ๐Ÿงš

a cache that uses a response received for one request to answer a later request MUST follow the rules in section 10.6 if the cached response includes an IM header field ๐Ÿซถ

10.5.3. A-IM โœจ

Okay bestie, so A-IM is a request header that's basically like Accept, except it's for instance-manipulations ๐Ÿ’… It restricts which ones are acceptable in the response. Remember from 10.5.2 that a response can be the result of applying several instance-manipulations stacked together ๐Ÿงš

A-IM = "A-IM" ":" #( instance-manipulation
                         [ ";" "q" "=" qvalue ] )

When an A-IM request-header field includes one or more delta-coding values, the request MUST contain an If-None-Match header field, listing one or more entity tags from prior responses for the request-URI. ๐Ÿ˜ญ No delta-coding without that If-None-Match, periodt.

A server checks whether an instance-manipulation it can actually use is acceptable, given an A-IM header field, using these rules:

  1. If the instance-manipulation is listed in the A-IM field, then it is acceptable, unless it is accompanied by a qvalue of 0. (As defined in section 3.9 of the HTTP/1.1 specification [10], a qvalue of 0 means "not acceptable.") A server MUST NOT use a non-identity instance-manipulation for a response unless the instance-manipulation is listed in an A-IM header in the request. ๐Ÿ™ˆ
  2. If multiple but incompatible instance-manipulations are acceptable, then the acceptable instance-manipulation with the highest non-zero qvalue is preferred. ๐Ÿ”ฅ
  3. The "identity" instance-manipulation is always acceptable, unless specifically refused because the A-IM field includes "identity;q=0". ๐ŸŒธ

If an A-IM field is present in a request, and the server cannot send a response that's acceptable according to the A-IM header, then the server SHOULD send an error response with the 406 (Not Acceptable) status code. ๐Ÿ’€

If a response uses more than one instance-manipulation, the instance-manipulations MUST be applied in the order in which they appear in the A-IM request-header field. ๐Ÿ‘‘

The server's choice about whether to apply an instance-manipulation SHOULD be independent of its choice to apply any subsequent two-input instance-manipulations to the response. (Two-input instance-manipulations include delta-codings, because they take two different values as input. Compression and "range" instance-manipulations take only one input. Other instance-manipulations may be defined in the future.) ๐Ÿฆ‹

Note: the intent of this requirement is to prevent the server from generating a delta-encoded response that the client can only decode by first applying an instance-manipulation encoding to its cached base instance. A server implementor might wish to consider what the client would logically have in its cache, when deciding which instance-manipulations to apply prior to a delta-coding.

Examples:

A-IM: vcdiff, gdiff

This example means that the client will accept a delta encoding in either vcdiff or gdiff format. ๐Ÿ’–

A-IM: vcdiff, gdiff;q=0.3

This example means that the client will accept a delta encoding in either vcdiff or gdiff format, but prefers the vcdiff format. โœจ

A-IM: vcdiff, diffe, gzip

This example means that the client will accept a delta encoding in either vcdiff or diffe format, and will accept the output of the delta encoding compressed with gzip. It also means that the client will accept a gzip compression of the instance, without any delta encoding, because A-IM provides no way to insist that gzip be used only if diffe is used. ๐Ÿค”

It's left to the server implementor to choose useful combinations of acceptable instance-manipulations. For example, following diffe by gzip is useful, but following vcdiff by gzip probably is not useful. ๐Ÿง 


10.6. Caching rules for 226 responses ๐ŸŽ€

When a client or proxy receives a 226 (IM Used) response, it MAY use this response to create a cache entry in three ways:

  1. It MAY decode all of the instance-manipulations to recover the original instance, and store that instance in the cache. In this case, the recovered instance is stored as a status-200 response, and MUST be used in accordance with the normal HTTP caching rules.
  2. It MAY decode all of the instance-manipulations except for range selection(s), and store the result in the cache. In this case, the result is stored as a status-206 response, and MUST be used in accordance with the normal HTTP caching rules for Partial Content.
  3. It MAY store the status-226 (IM Used) response as a cache entry.

A status-226 cache entry MUST NOT be used in response to a subsequent request under any of these conditions. (A cache that never stores status-226 responses may ignore these tests.)

  1. If any of the instance-manipulation values from the IM header field in the cached response do not appear in the subsequent request's A-IM header field. The comparison between the headers is done using an exact match on each instance-manipulation value, including any associated imparams values (see section 10.1). ๐Ÿ”
  2. If the order of instance-manipulation values appearing in the cached IM header field differs from the order of that set of instance-manipulations in the A-IM header field of the subsequent request. ๐Ÿ”„
  3. If the cache implementation is not aware of, or is not at least conditionally compliant with, the specification of any of the instance-manipulation values in the cached IM header field.
Note: This rule allows for extending the set of instance-manipulations without causing deployed cache implementations to commit errors. The specification of new instance-manipulations may include additional caching rules to improve cache-hit rates in cognizant implementations. ๐ŸŒˆ
  1. If any of the instance-manipulation values in the cached IM header field is a delta-coding, and the cache entry includes a Delta-Base header field, and that Delta-Base entity tag is not one of the entity tags listed in an If-None-Match header field of the subsequent request. ๐Ÿซถ
  2. If any of the instance-manipulation values in the cached IM header field is a delta-coding, the cache entry does not include a Delta-Base header field, and the If-None-Match header field of the request that led to that cache entry does not match the If-None-Match header field of the subsequent request. ๐Ÿ’…

If the IM header field of the cached response includes the "range" instance-manipulation, then a status-226 cache entry MUST NOT be used in response to a subsequent request if the cached response is inconsistent with the Range header field value(s) in the request, as would be the case for a cached 206 (Partial Content) response. ๐Ÿ™ƒ

Note: we know of no existing, published formal specification for deciding if a cached status-206 response is consistent with a subsequent request. We believe that either of these conditions is sufficient: 1. The ranges specified in the headers of the request that led to the cached response are the same as specified in the headers of the subsequent request. 2. The ranges specified in the cached response are the same as specified in the headers of the subsequent request. Further analysis might be necessary. ๐Ÿง

10.7. Rules for deltas in the presence of content-codings ๐Ÿ“

The use of delta encoding with content-encoded instances adds some slight complexity. When a client (perhaps a proxy) has received a delta encoded response, either or both of that new response and a cached previous response may have non-identity content-codings. So we specify rules for the server and client, to prevent situations where the client is unable to make sense of the server's response. ๐Ÿงฉ

10.7.1. Rules for generating deltas in the presence of content-codings ๐Ÿ› ๏ธ

When a server generates a delta-encoded response, the list of content-codings the server uses (i.e., the value of the response's Content-Encoding header field) SHOULD be a prefix of the list of content-codings the server would have used had it not generated a delta encoding. โœจ

This requirement allows a client receiving a delta-encoded response to apply the delta to a cached base instance without having to apply any content-codings during the process. (Although the client might, of course, be required to decode some content-codings.) ๐Ÿ’œ

10.7.2. Rules for applying deltas in the presence of content-codings ๐Ÿฆ„

When a client receives a delta response with one or more non-identity content codings:

  1. If both the new (delta) response and the cached response (instance) have exactly the same set of content-codings, the client applies the delta response to the cached response without removing the content-codings from either response. ๐Ÿ’–
  2. If the new (delta) response and the cached response have a different set of content-codings, then before applying the delta the client decodes one or more content-codings from the cached response, until the result has the same set of content-codings as the delta response. ๐Ÿ”ง
  3. If a proxy or cache is forwarding the result of applying the delta response to a cached base instance response, or later forwards this result from a cache entry, the forwarded response MUST carry the same Content-Encoding header field as the new (delta) response (and so it must be content-encoded as indicated by that header field). ๐ŸŽฏ

The intent of these rules (and in particular, rule #3) is that the results are always consistent with the rule that the entity tag is associated with the result of the content-coding. Any recipient after the application of the delta-coding receives exactly the same response it would have received as a status-200 response from the origin server (without any delta-coding). That's the whole vibe, fr fr ๐Ÿฅฐ

10.7.3 Examples for using A-IM, IM, and content-codings

okay bestie, let's do the examples ๐ŸŽ€ Say a client with an empty cache sends this request:

GET /foo.html HTTP/1.1
Host: example.com
Accept-encoding: gzip

and the origin server replies with:

HTTP/1.1 200 OK
Date: Wed, 24 Dec 1997 14:00:00 GMT
Etag: "abc"
Content-encoding: gzip

We write URI;entity-tag to name a specific instance, so this response means the client stores GZIP(foo.html;"abc") in its cache ๐Ÿงš

A minute later, the client sends this conditional request:

GET /foo.html HTTP/1.1
Host: example.com
If-none-match: "abc"
Accept-encoding: gzip
A-IM: vcdiff

If the server can make a delta-encoded response, it has two options ๐Ÿ’… The first is to compute the delta from the compressed instances, which might not give the most efficient coding:

HTTP/1.1 226 IM Used
Date: Wed, 24 Dec 1997 14:01:00 GMT
Etag: "def"
Delta-base: "abc"
Content-encoding: gzip
IM: vcdiff

The body is VCDIFF_DELTA(GZIP(foo.html;"abc"), GZIP(foo.html;"def")). The client rebuilds GZIP(foo.html;"def") by applying the delta to its previous cache entry, then stores that as the new cache entry โœจ

The server's other option is to compute the delta from the uncompressed values:

HTTP/1.1 226 IM Used
Date: Wed, 24 Dec 1997 14:01:00 GMT
Delta-base: "abc"
Etag: "ghi"
IM: vcdiff

The body is VCDIFF_DELTA(GUNZIP(GZIP(foo.html;"abc")), foo.html;"ghi"), or more simply VCDIFF_DELTA(foo.html;"abc", foo.html;"ghi"). The client applies the delta to its old cache entry and stores foo.html;"ghi", which has no content-coding ๐Ÿฆ‹

Quick heads up: the new foo.html (at 14:01:00 GMT) without gzip has to get a different entity tag from the compressed copy of the same file. Different vibes, different tag, no mix-ups ๐Ÿ’œ

The client's second request might look like this:

GET /foo.html HTTP/1.1
Host: example.com
If-none-match: "abc"
Accept-encoding: gzip
A-IM: diffe, gzip

The client lists gzip in both Accept-Encoding and A-IM. Why? If the server doesn't support delta encoding, the client still gets the compression benefits as a content-coding ๐Ÿซถ But if the server supports the diffe delta-coding, the client wants the result compressed, and that has to happen as an instance-manipulation.

A server that supports diffe might reply:

HTTP/1.1 226 IM Used
Date: Wed, 24 Dec 1997 14:01:00 GMT
Delta-base: "abc"
Etag: "ghi"
IM: diffe, gzip

The body is GZIP(DIFFE_DELTA(GUNZIP(GZIP(foo.html;"abc")), foo.html;"ghi")), or more simply GZIP(DIFFE_DELTA(foo.html;"abc", foo.html;"ghi")). Here the gzip is an instance-manipulation and not a content-coding, so it's not kept when the reassembled response is stored or forwarded ๐Ÿ™ˆ The client ends up storing foo.html;"ghi" with no content-coding and no compression at all ๐Ÿ‘‘


10.8 New Cache-Control directives

Okay, we're adding two new cache-directives ๐ŸŒธ (see section 14.9 of RFC 2616 [10] for the specification of cache-directive).

10.8.1 Retain directive

The cache-response-directive values now include the retain directive:

cache-response-directive = ...
        | "retain" [ "=" delta-seconds ]

A retain directive is always a "hint" from a server to a client. It never sets a mandatory action for the recipient, so no pressure ๐Ÿ’–

If retain shows up, a delta-capable client ought to keep that instance in its cache, space permitting. It also ought to use the matching entity tag in a future request for a delta-encoded response. In other words, the server will probably send delta-encoded responses based on that instance as the base ๐Ÿงš So if a client has cached a bunch of instances of one resource and only some are marked "retain", there's no point keeping the unmarked ones.

If retain comes with a delta-seconds value, the server will likely stop using that instance as a base after that many seconds. The client ought not use the entity tag in a future delta request after that window closes. The clock starts when the response is generated, so the client should count the response's Age in its math โฐ

If the retain directive has a delta-seconds value of zero, a client SHOULD NOT use the matching entity tag in a future request for a delta-encoded response ๐Ÿ’€

Note: We recommend that server implementors consider the bandwidth implications of sending the "retain=0" directive to clients or proxies that might not have the ability to make use of it.

10.8.2 IM directive

The cache-response-directive values now include the im directive:

cache-response-directive = ...
        | "im"

A cache that follows the spec for the IM header, the A-IM header, and the 226 response-status code SHOULD ignore a no-store cache-directive when an im directive is in the same response ๐Ÿฆ‹ Every other implementation MUST ignore the im directive, which means it MUST observe a no-store directive if one is present. Not me getting confused by the double negatives ๐Ÿ˜ญ


10.9 Use of compression with delta encoding

Compressing the diffe and gdiff delta codings has been shown to shrink the message bodies a lot in many cases ๐Ÿ“ (The vcdiff coding is already compressed on its own and doesn't get much out of extra compression.) So it is strongly recommended that implementations that support diffe and/or gdiff also support the gzip and/or deflate compression codings. (deflate gives a more compact result.) However, this is not a requirement for using delta encoding. The main reason is that the CPU cost of compressing and decompressing might be too much in some environments ๐Ÿ’…

A client that supports both delta encoding and compression as instance-manipulations says so like this:

A-IM: diffe, deflate

The ordering rule in section 10.5.3 says that if the server uses both instance-manipulations in the response, the compression has to be applied to the result of the delta encoding, not the other way around. So the response here would include:

IM: diffe, deflate

Also, a client might accept compression as either a content-coding or an instance-manipulation. For example:

Accept-Encoding: gzip
A-IM: gzip, gdiff

In this case the server can apply gzip before delta encoding, either as a content-coding or as an instance-manipulation. Remember that the entity tag is assigned after content-coding but before instance-manipulation, so this choice really does change how delta encoding behaves โœจ


10.10 Delta encoding and multipart/byteranges

A client can ask for several non-contiguous byte ranges in one request ๐ŸŒˆ The server answers with the "multipart/byteranges" media type (section 19.2 of [10]) to send those ranges together. If a multipart/byteranges response is delta encoded (meaning it uses a delta-coding as an instance-manipulation), the delta-related headers apply to the whole response, not to each part. There's only one base instance and one current instance involved, so that makes sense ๐Ÿงš A delta-encoded response with multiple ranges MUST use the same delta-coding for all of the ranges.

If a server decides to use delta encoding for a multipart/byteranges response, it MUST generate the response following these rules.

When a multipart/byteranges response uses a delta-coding before a range selection, the A-IM and IM header fields list the delta-coding before the "range" literal. (Remember, this is the approach used to get a partial response after a message transmission ends early.) The server first generates the sequence of bytes that is the difference (delta) between the base instance and the current instance. Then it selects the specified byte ranges and sends each range in its own part of the multipart/byteranges media type ๐Ÿ’–

When a multipart/byteranges response uses a delta-coding after a range selection, the A-IM and IM header fields list the delta-coding after the "range" literal. (This is the approach used to get an updated version of just some sections of an instance.) The server first selects the specified ranges from the current instance, and selects the same ranges from the base instance too. (Some of these selected ranges might be empty, if the instance isn't long enough.) Then it generates the individual differences (deltas) between each pair of ranges, and sends each difference in its own part of the multipart/byteranges media type ๐Ÿฆ‹

11. Quantifying the protocol overhead โœจ

okay so let's talk about how much extra weight these changes add, bestie ๐ŸŽ€ spoiler: not a lot. The proposed protocol changes make HTTP message headers a teeny bit bigger. In the simplest case, a conditional request (aka one for a URI the client already has a cache entry for) gets one more header ๐Ÿ’–:

A-IM:vcdiff

That's about 13 extra bytes. A recent study [23] reported mean request sizes from two different traces of 281 and 306 bytes, so the net increase in request size works out to between 4% and 5% ๐Ÿ“ˆ.

Because a client needs an existing cache entry to use as a base for a delta-encoded response, it would never send "A-IM: vcdiff" (or listing other delta encoding formats) on its unconditional requests ๐Ÿ™…โ€โ™€๏ธ. The same study showed that at least 46% of the requests in lengthy traces were for URLs not seen previously in the trace. So no more than about half of typical client requests could be conditional (and the actual fraction is likely to be smaller, given the finite size of real caches) ๐Ÿฅฒ.

The study also showed that 64% of the responses in a lengthy trace were for image content-types (GIF and JPEG) ๐Ÿ“ธ. As noted in section 6, we do not currently know of a delta-encoding format suitable for such image types. Unless a client did support such a delta-encoding format, it would presumably not ask for a delta when making a conditional request for image content-types.

Taken together, these factors suggest that the mean increase in request header size would be much less than 5%, and probably below 1% ๐ŸŽ‰.

Delta-encoded responses carry slightly longer headers ๐Ÿงš. In the simplest case, a response carries one more header, e.g.:

IM:vcdiff

That's about 11 bytes. Other headers (such as "Delta-Base") might also be included. However, none of these extra headers would be included except in cases where a delta encoding is actually employed, and the sender of the response can avoid sending a delta encoding if this results in a net increase in response size. Thus, a delta-encoded response should never be larger than a regular response for the same request ๐Ÿซถ.

Simulations suggest that, when delta encoding pays off at all, it saves several thousand bytes [23]. So adding a few dozen bytes to the response headers should almost never cancel out the savings in the message-body size ๐Ÿ”ฅ.

Finally, the use of the "retain" Cache-Control directive might cause some additional overhead. Some server heuristics might be successful in limiting the use of these headers to situations where they would probably optimize future responses. Neither of these headers is necessary for the simpler uses of delta encoding ๐ŸŒธ.


12. Security Considerations ๐Ÿ”

We are not aware of any aspects of the basic delta encoding mechanism that affect the existing security considerations for the HTTP/1.1 protocol. Nothing spicy here, just a clean vibe ๐Ÿ˜Œโœจ.

13. Acknowledgements ๐Ÿซถ

Phong Vo has provided a great deal of guidance in the choice of delta encoding algorithms and formats. Issac Goldstand and Mike Dahlin provided a number of useful comments on the specification. Dave Kristol suggested many textual corrections. Thank you all, you are literally the MVPs ๐Ÿ‘‘๐Ÿ’•.

14. Intellectual Property Rights โš–๏ธ

The IETF has been notified of intellectual property rights claimed in regard to some or all of the specification contained in this document. For more information, consult the online list of claimed rights, at http://www.ietf.org/ipr.html.

The IETF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document, or the extent to which any license under such rights might or might not be available. Neither does it represent that it has made any effort to identify any such rights. Information on the IETF's procedures with respect to rights in standards-track and standards-related documentation can be found in BCP 11. Copies of claims of rights made available for publication, and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementors or users of this specification, can be obtained from the IETF Secretariat ๐Ÿ“„.

15. References ๐Ÿ“š

  1. Gaurav Banga, Fred Douglis, and Michael Rabinovich. Optimistic Deltas for WWW Latency Reduction. Proc. 1997 USENIX Technical Conference, Anaheim, CA, January, 1997, pp. 289-303.
  2. Berners-Lee, T., Fielding, R. and H. Frystyk, "Hypertext Transfer Protocol -- HTTP/1.0", RFC 1945, May 1996.
  3. Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  4. Edith Cohen, Balachander Krishnamurthy, and Jennifer Rexford. Improving End-to-End Performance of the Web Using Server Volumes and Proxy Filters. Proc. SIGCOMM '98, September, 1998, pp. 241-253.
  5. Deutsch, P., "GZIP file format specification version 4.3", RFC 1952, May 1996.
  6. Deutsch, P., "DEFLATE Compressed Data Format Specification version 1.3", RFC 1951, May 1996.
  7. Deutsch, P. and J-L. Gailly, "ZLIB Compressed Data Format Specification version 3.3", RFC 1950, May 1996.
  8. Fred Douglis, Anja Feldmann, Balachander Krishnamurthy, and Jeffrey Mogul. Rate of Change and Other Metrics: a Live Study of the World Wide Web. Proc. Symposium on Internet Technologies and Systems, USENIX, Monterey, CA, December, 1997, pp. 147-158.
  9. Fielding, R., Gettys, J., Mogul, J., Nielsen, H. and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2068, January 1997.
  10. Fielding, R., Gettys, J., Mogul, J., Nielsen, H., Masinter, L., Leach, P. and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.
  11. Franks, J., Hallam-Baker, P., Hostetler, J., Leach, P., Luotonen, A., Luotonen, L. and L. Stewart, "HTTP Authentication: Basic and Digest Access Authnetication", RFC 2617, June 1999.
  12. Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies", RFC 2045, November 1996.
  13. Arthur van Hoff, John Giannandrea, Mark Hapner, Steve Carter, and Milo Medin. The HTTP Distribution and Replication Protocol. Technical Report NOTE-DRP, World Wide Web Consortium, August, 1997.
  14. Arthur van Hoff and Jonathan Payne. Generic Diff Format Specification. Technical Report NOTE-GDIFF, World Wide Web Consortium, August, 1997.
  15. Barron C. Housel and David B. Lindquist. WebExpress: A System for Optimizing Web Browsing in a Wireless Environment. Proc. 2nd Annual Intl. Conf. on Mobile Computing and Networking, ACM, Rye, New York, November, 1996, pp. 108-116.
  16. James J. Hunt, Kiem-Phong Vo, and Walter F. Tichy. An Empirical Study of Delta Algorithms. IEEE Soft. Config. and Maint. Workshop, 1996.
  17. Jacobson, V., "Compressing TCP/IP Headers for Low-Speed Serial Links", RFC 1144, February 1990.
  18. Khare, R. and S. Lawrence, "Upgrading to TLS Within HTTP/1.1", RFC 2817, May 2000.
  19. David G. Korn and Kiem-Phong Vo. A Generic Differencing and Compression Data Format. Technical Report HA1630000-021899-02TM, AT&T Labs - Research, February, 1999.
  20. Korn, D. and K. Vo, "The VCDIFF Generic Differencing and Compression Data Format", Work in Progress.
  21. Merriam-Webster. Webster's Seventh New Collegiate Dictionary. G. & C. Merriam Co., Springfield, MA, 1963.
  22. Jeffrey C. Mogul. Hinted caching in the Web. Proc. Seventh ACM SIGOPS European Workshop, Connemara, Ireland, September, 1996, pp. 103-108.
  23. Jeffrey C. Mogul, Fred Douglis, Anja Feldmann, and Balachander Krishnamurthy. Potential benefits of delta encoding and data compression for HTTP. Research Report 97/4, DECWRL, July, 1997.
  24. Mogul, J. and A. Van Hoff, "Instance Digests in HTTP", RFC 3230, January 2002.
  25. Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 2434, October 1998.
  26. The Open Group. The Single UNIX Specification, Version 2 - 6 Vol Set for UNIX 98. Document number T912, The Open Group, February, 1997.
  27. W. Tichy. "RCS - A System For Version Control". Software - Practice and Experience 15, 7 (July 1985), 637-654.
  28. Andrew Tridgell and Paul Mackerras. The rsync algorithm. Technical Report TR-CS-96-05, Department of Computer Science, Australian National University, June, 1996.
  29. Stephen Williams. Personal communication. http://ei.cs.vt.edu/~williams/DIFF/prelim.html.
  30. Stephen Williams, Marc Abrams, Charles R. Standridge, Ghaleb Abdulla, and Edward A. Fox. Removal Policies in Network Caches for World-Wide Web Documents. Proc. SIGCOMM '96, Stanford, CA, August, 1996, pp. 293-305.

16. Authors' addresses ๐Ÿ—บ๏ธ

Jeffrey C. Mogul Western Research Laboratory Compaq Computer Corporation 250 University Avenue Palo Alto, California, 94305, U.S.A.

Phone: 1 650 617 3304 (email preferred) EMail: JeffMogul@acm.org

Balachander Krishnamurthy AT&T Labs - Research 180 Park Ave, Room D-229 Florham Park, NJ 07932-0971, U.S.A.

EMail: bala@research.att.com

Fred Douglis AT&T Labs - Research 180 Park Ave, Room B-137 Florham Park, NJ 07932-0971, U.S.A.

Phone: 1 973 360-8775 EMail: douglis@research.att.com

Anja Feldmann University of Saarbruecken, Germany, Computer Science Department Im Stadtwald, Geb. 36.1, Zimmer 310 D-66123 Saarbruecken, Germany

EMail: anja@cs.uni-sb.de Yaron Y. Goland

Email: yaron@goland.org

Arthur van Hoff Marimba, Inc. 440 Clyde Avenue Mountain View, CA 94043, U.S.A.

Phone: 1 650 930 5283 EMail: avh@marimba.com

Daniel M. Hellerstein Economic Research Service, USDA 1909 Franwall Ave, Wheaton MD 20902

Phone: 1 202 694-5613 or 1 301 649-4728 EMail: danielh@crosslink.net or webmaster@srehttp.org

Copyright (C) The Internet Society (2002). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement

Funding for the RFC Editor function is currently provided by the Internet Society. ๐Ÿ’œ

๐Ÿ“ฆ where this page comes from