RFC 3230

Instance Digests in HTTP

📅 published January 2002, from ietf

Title Block 📜✨

Network Working Group, Request for Comments 3230, Category: Standards Track 🎀. Okay bestie, this one dropped in January 2002 🧚. The authors are J. Mogul from Compaq WRL and A. Van Hoff from Marimba 💜. The block doesn't have an Obsoletes or Updates line, so nothing is listed there 🫶.

Instance Digests in HTTP 💖

Status of this Memo 🌸

This document is an Internet standards track protocol for the Internet community, and it's asking for discussion and suggestions for improvements 💅. If you want the current standardization state and status of this protocol, check the "Internet Official Protocol Standards" (STD 1) doc 🔥. Distribution of this memo is unlimited ✨.

Copyright (C) The Internet Society (2002). All Rights Reserved. 💜

Abstract 🌈

Okay so, HTTP/1.1 has a Content-MD5 header that lets a server include a digest of the response body 🍓. Lowkey though, it only covers the body of that one message, not the whole file. That's a problem if the response is a Content-Range or uses a delta encoding, because the full file could look totally different 😭. Also, Content-MD5 is stuck on one digest algorithm, and something like SHA-1 (Secure Hash Standard) might be a better fit in some situations 🔥. And finally, HTTP/1.1 has zero explicit way for a client to ask for a digest 💀. This doc proposes HTTP extensions that fix all three of these issues 👑.

Table of Contents 🧚

   1 Introduction....................................................  2
        1.1 Other limitations of HTTP/1.1............................  3
   2 Goals...........................................................  4
   3 Terminology.....................................................  5
   4 Specification...................................................  6
        4.1 Protocol parameter specifications........................  6
             4.1.1 Digest algorithms.................................  6
        4.2 Instance digests.........................................  7
        4.3 Header specifications....................................  8
             4.3.1 Want-Digest.......................................  8
             4.3.2 Digest............................................  9
   5 Negotiation of Content-MD5......................................  9
   6 IANA Considerations............................................. 10
   7 Security Considerations......................................... 10
   8 Acknowledgements................................................ 10
   9 References...................................................... 10
   10 Authors' Addresses............................................. 12
   11 Full Copyright Statement....................................... 13

1 Introduction 🎀

Okay, so HTTP usually sits on top of a reliable transport like TCP, but that doesn't guarantee the info gets from sender to receiver intact 😭. Stuff like undetected transmission errors, programming bugs, corrupted stored data, and malicious interference can all mess up what's transmitted 💀.

The usual fix for data integrity in a network protocol or distributed system (HTTP included) is digests, checksums, or hash values ✨. The sender computes a digest and sends it along with the data 🫶. The recipient computes its own digest of what it got, then checks integrity by comparing the two 💅.

Checksums show up at basically every layer of the IP stack, but different layers use different digest algorithms 🦋. That's because of computational cost, the size and nature of the data being protected, and the different threats to integrity at each layer 🌸. For example, Ethernet uses a Cyclic Redundancy Check (CRC) 🧚. IPv4 uses a ones-complement checksum over the IP header (but not the rest of the packet) 🍓. TCP uses a ones-complement checksum over the TCP header and data, and it adds a "pseudo-header" to catch certain kinds of programming errors 🔥.

HTTP/1.1 [4] has a built-in integrity mechanism, the Content-MD5 header 🎀. Honestly, that header is defined for MIME-conformant messages in a standalone spec [10], and HTTP/1.1 just borrowed it 💜. Per the HTTP/1.1 spec:

The Content-MD5 entity-header field [...] is an MD5 digest of the entity-body for the purpose of providing an end-to-end message integrity check (MIC) of the entity-body.

HTTP/1.1 grabbed Content-MD5 from the MIME world because MIME messages (like email) and HTTP messages (requests to or responses from an HTTP server) kinda feel similar 🌈.

We'll get into this more in section 3, but that analogy has caused some real confusion 🙈. A MIME message is self-contained, while an HTTP message might not include the whole current representation of a resource 😭. (More precisely, an HTTP response might not contain an entire "instance"; check section 3 for the definition of that term 💖.)

There are at least two situations where this difference is a real problem ⭐:

  1. When an HTTP server sends a 206 (Partial Content) response, as defined in HTTP/1.1. The client might build its own view of an instance (like an HTML document) by combining a cache entry with the partial content in the message 🫶.
  2. When an HTTP server uses a "delta encoding", as proposed in a separate document [9]. A delta encoding represents the changes between the current instance of a resource and a previous one, and it's a super efficient way to cut down on the bandwidth needed for cache updates ✨. The client builds its view of an instance by applying the delta in the message to one of its cache entries 👑.

We're calling both of these kinds of transformations a potentially broader category: "instance manipulations" 🧚.

In each of these cases, the server could use a Content-MD5 header to protect the integrity of the response message 🍓. But here's the tea: the MIC in a Content-MD5 header only applies to the entity in that message, not the entire instance being reassembled 😭. So it can't protect against corrupted cache entries, programming errors (like applying a partial content or delta incorrectly), certain malicious attacks [9], or corruption of certain HTTP headers in transit 💀.

So yeah, Content-MD5 is useful and sufficient in lots of cases, but it's not enough for verifying instance integrity in every use of HTTP 🔥.

The Digest Authentication mechanism [5] does something similar to Content-MD5 (among its other goals), with a message-digest function that also covers certain header fields 🌸. Like Content-MD5, though, it covers one specific message, not an entire instance 🙈.

1.1 Other limitations of HTTP/1.1 🦋

Checksums aren't free, bestie 💸. Computing a digest eats CPU and can add latency to generating a message 😩. (Careful caching at the sender's end can dodge some of these costs, but in lots of cases that cache wouldn't have a useful hit ratio 🙃.) Sending a digest also uses up HTTP header space, which means more latency and bandwidth 📉. If the recipient isn't going to use the digest anyway, why should the sender waste resources computing and sending it 🤷‍♀️?

The Content-MD5 header, obviously, means you're using the MD5 algorithm [15]. Other algorithms might fit some purposes better 🌷. These include the SHA-1 algorithm [12] and various "fingerprinting" algorithms [7] 🧚. Right now HTTP offers zero standardized support for any of these 💀.

HTTP/1.1 kinda assumes the sender decides whether to generate a digest, and it gives the recipient no way to say whether a checksum would be useful or which checksum algorithms it understands 😭🫶.

2 Goals 💅

The goals of this proposal are:

  1. Digest coverage for entire instances communicated via HTTP. 🎯
  2. Support for multiple digest algorithms. 🌈
  3. Negotiation of the use of digests. 🤝

The goals do not include:

  • header integrity 🙅‍♀️ The digest mechanisms described here cover only the bodies of instances, and do not protect the integrity of associated "entity headers" or other message headers.
  • authentication 🙅‍♀️ The digest mechanisms described here are not meant to support authentication of the source of a digest or of a message or instance. These mechanisms, therefore, are not sufficient defense against many kinds of malicious attacks.
  • privacy 🙅‍♀️ Digest mechanisms do not provide message privacy.
  • authorization 🙅‍♀️ The digest mechanisms described here are not meant to support authorization or other kinds of access controls.

The Digest Access Authentication mechanism [5] can give some integrity for certain HTTP headers, and it does provide authentication 🔐✨.

3. Terminology ✨

okay so HTTP/1.1 [4] already has a few terms it's using, and we're gonna borrow them 💅

  • resource 🌸: a network data object or service you can identify with a URI, as defined in section 3.2. A resource can come in multiple representations (different languages, data formats, sizes, resolutions) or vary in other ways.
  • entity 🎀: the info sent as the payload of a request or response. It's made of metainformation (entity-header fields) plus content (the entity-body), as described in section 7.
  • variant 🦋: a resource can have one or more representations attached to it at any given moment, and each of those is called a variant. Using the word "variant" doesn't automatically mean the resource does content negotiation, ngl.

okay so the dictionary says "entity" means "something that has separate and distinct existence and objective or conceptual reality" [8]. Unfortunately, HTTP/1.1's definition of "entity" is basically copied from MIME [6], and it's built on a totally false analogy between MIME and HTTP 🙈

In MIME, emails actually do have their own distinct existence. MIME says an entity "refers specifically to the MIME-defined header fields and contents of either a message or one of the parts in the body of a multipart entity."

In HTTP though? A response to a GET doesn't have a distinct existence at all. It's describing the current state of a resource (or a variant, with some constraints). HTTP/1.1 has no word for "the value that would be returned in response to a GET request at the current time for the selected variant of the specified resource," so the spec ends up with awkward wording in places where that idea is needed 😭

It's too late to fix the terminology in HTTP/1.1, so we're defining a new term for this document:

  • instance 💖: the entity that would be returned in a status-200 response to a GET request, at the current time, for the selected variant of the specified resource. It has zero or more content-codings applied, but no instance manipulations or transfer-codings.

It's handy to think of an entity tag in HTTP/1.1 as belonging to an instance instead of an entity. So for a given resource, two different response messages might carry the same entity tag, but two different instances should never share the same (strong) entity tag ✨

We also define this one:

  • instance manipulation 🧚: an operation on one or more instances that might get an instance sent from server to client in parts, or across more than one response message. Range selection and delta encoding are two examples. Instance manipulations are end-to-end, and they often involve a cache at the client 🍓

4. Specification 👑

Heads up, the key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" in this spec should be read the way RFC 2119 [2] describes 🫶

4.1. Protocol parameter specifications 🎀

4.1.1. Digest algorithms 🌈

Digest algorithm values are how we say which specific digest computation is happening. Some algorithms can take one or more parameters too.

digest-algorithm = token

The grammar for "parameter" is the same one used in RFC 2616 [4]. Also, all digest-algorithm values are case-insensitive, so md5 and MD5 are the same thing 💅

The Internet Assigned Numbers Authority (IANA) acts as the registry for digest-algorithm values. To start, the registry has these tokens:

  • MD5 🌸: the MD5 algorithm, as specified in RFC 1321 [15]. Its output gets encoded with base64 [1].
  • SHA 💖: the SHA-1 algorithm [12]. Its output gets encoded with base64 [1].
  • UNIXsum 🦋: the algorithm the UNIX "sum" command computes, as defined by the Single UNIX Specification, Version 2 [13]. The output is an ASCII decimal-digit string holding the 16-bit checksum, which is the first word of the output of the UNIX "sum" command.
  • UNIXcksum 🧚: the algorithm the UNIX "cksum" command computes, as defined by the Single UNIX Specification, Version 2 [13]. The output is an ASCII digit string holding the 32-bit CRC, which is the first word of the output of the UNIX "cksum" command.

If other digest-algorithm values get defined, their encoding MUST either be represented as a quoted string, or MUST NOT include ";" or "," in the character sets used for the encoding. Rules are rules, bestie 🫶

4.2. Instance digests 🔥

An instance digest is the representation of a digest algorithm's output, plus an indication of which algorithm was used (and any parameters).

instance-digest = digest-algorithm "="
                        <encoded digest output>

The digest is computed on the entire instance associated with the message. The instance is a snapshot of the resource before any instance manipulation or transfer-coding is applied (see section 3). The byte order used to compute the digest is the transmission byte order defined for the content-type of the instance 🌸

Note: the digest is computed before the application of any instance manipulation. If a range or a delta-coding [9] is used, the computation of the digest after the computation of the range or delta would not provide a digest useful for checking the integrity of the reassembled instance. So it's giving "do the math first" 🙈

The encoded digest output uses whatever encoding format is defined for that specific digest-algorithm. For example, if the digest-algorithm is "MD5", the encoding is base64. If the digest-algorithm is "UNIXsum", the encoding is an ASCII string of decimal digits ✨

Examples:

MD5=HUXZLQLMuI/KZ5KDcJPcOA==
sha=thvDyvhfIqlvFe+A9MYgxAfm1q5=
UNIXsum=30637

4.3. Header specifications 🦋

These headers are defined:

4.3.1. Want-Digest 💜

The Want-Digest message header field lets the sender say it wants an instance digest on messages associated with the Request-URI. Main character energy 👑

Want-Digest = "Want-Digest" ":"
                 #(digest-algorithm [ ";" "q" "=" qvalue])

If a digest-algorithm doesn't come with a qvalue, it's treated as if its qvalue were 1.0 ✨

The sender is willing to accept a digest-algorithm if and only if it's listed in a Want-Digest header field of a message and its qvalue is non-zero. So a zero qvalue is basically a "no thanks" 💀

If there are multiple acceptable digest-algorithm values, the sender's preferred one (or ones) is the one with the highest qvalue 🌈

Examples:

Want-Digest: md5
Want-Digest: MD5;q=0.3, sha;q=1

4.3.2. Digest 🌸

The Digest message header field gives a message digest of the instance described by the message 🫶

Digest = "Digest" ":" #(instance-digest)

The instance described by a message might be fully contained in the message-body, partially contained in it, or not contained in it at all. The instance is specified by the Request-URI and any cache-validator in the message 🍓

A Digest header field MAY contain multiple instance-digest values. This could be useful for responses that are going to live in caches shared by users with different browsers, for example 🧚

A recipient MAY ignore any or all of the instance-digests in a Digest header field. It's not rude, it's allowed 💅

A sender MAY send an instance-digest using a digest-algorithm without knowing whether the recipient supports that digest-algorithm, or even knowing that the recipient will ignore it. Send it anyway, periodt 🙈

Examples:

Digest: md5=HUXZLQLMuI/KZ5KDcJPcOA==
Digest: SHA=thvDyvhfIqlvFe+A9MYgxAfm1q5=,unixsum=30637

5. Negotiation of Content-MD5 🎀

HTTP/1.1 has a Content-MD5 header field, but no way to ask for it to be used (or not used). The Want-Digest header field from this document is the foundation for that kind of mechanism 🌈

First, we add the token "contentMD5" to the set of digest-algorithm values in section 4.1.1. There's a catch though: this digest-algorithm MUST NOT be used in a Digest header field 💀

If "contentMD5" shows up with a non-zero qvalue in a Want-Digest header field, the sender is saying it wants a Content-MD5 header on messages associated with the Request-URI ✨

If "contentMD5" shows up with a zero qvalue in a Want-Digest header field, the sender is saying it will ignore Content-MD5 headers on messages associated with the Request-URI 🙈

6. IANA Considerations 🔥

IANA administers the name space for digest-algorithm values. Each value and what it means has to be documented in an RFC or some other peer-reviewed, permanent, and readily available reference, with enough detail that independent implementations can actually interoperate. Subject to those constraints, name assignments are First Come, First Served (see RFC 2434 [11]) 👑

7. Security Considerations 💀

Okay, this part is serious, so pay attention 😭 This document specifies a data integrity mechanism that protects HTTP instance data, but not HTTP entity headers, from certain kinds of accidental corruption. It's also useful for detecting at least one spoofing attack [9]. However, it is not intended as general protection against malicious tampering with HTTP messages.

Warning: this is NOT a shield against someone deliberately messing with your HTTP messages. It's a corruption checker, not a bodyguard 🚨

The HTTP Digest Access Authentication mechanism [5] does provide some protection against malicious tampering, so go look at that one too 🫶

8. Acknowledgements 🌸

Nobody is totally sure who first realized that the Content-MD5 header field isn't enough for data integrity once ranges or deltas are in the mix. Such a mystery 🕵️‍♀️

Laurent Demailly may have been the first to suggest an algorithm-independent checksum header for HTTP [3]. And Dave Raggett suggested using the term "digest" instead of "checksum" [14], which is honestly iconic of him 💖

9. References 📚

okay bestie, this is the bibliography 🤓 every source the spec leans on is listed here, so you can go read the originals if you want the full story 💜

  • [1] Freed, N. and N. Borenstein, N., "MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies", RFC 2049, November 1996.
  • [2] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [3] Laurent Demailly. Re: Revised Charter. http://www.ics.uci.edu/pub/ietf/http/hypermail/1995q4/0165.html.
  • [4] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P. and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1.", RFC 2616, June 1999.
  • [5] Franks, J., Hallam-Baker, P., Hostetler, J., Lawrence, S., Leach, P., Luotonen, A. and L. Stewart, "HTTP Authentication: Basic and Digest Access Authentication", RFC 2617, June 1999.
  • [6] Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies", RFC 2045, November 1996.
  • [7] Nevin Heintze. Scalable Document Fingerprinting. Proc. Second USENIX Workshop on Electronic Commerce, USENIX, Oakland, CA, November, 1996, pp. 191-200. http://www.cs.cmu.edu/afs/cs/user/nch/www/koala/main.html.
  • [8] Merriam-Webster. Webster's Seventh New Collegiate Dictionary. G. & C. Merriam Co., Springfield, MA, 1963.
  • [9] Mogul, J., Krishnamurthy, B., Douglis, F., Feldmann, A., Goland, Y. and A. van Hoff, "Delta encoding in HTTP", RFC 3229, December 2001.
  • [10] Myers, J. and M. Rose, "The Content-MD5 Header Field", RFC 1864, October 1995.
  • [11] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 2434, October 1998.
  • [12] National Institute of Standards and Technology. Secure Hash Standard. FEDERAL INFORMATION PROCESSING STANDARDS PUBLICATION 180-1, U.S. Department of Commerce, April, 1995. http://csrc.nist.gov/fips/fip180-1.txt.
  • [13] The Open Group. The Single UNIX Specification, Version 2 - 6 Vol Set for UNIX 98. Document number T912, The Open Group, February, 1997.
  • [14] Dave Raggett. Re: Revised Charter. http://www.ics.uci.edu/pub/ietf/http/hypermail/1995q4/0182.html.
  • [15] Rivest, R., "The MD5 Message-Digest Algorithm", RFC 1321, April 1992.

10. Authors' Addresses 🎀

these are the people who wrote it, so if you ever have a question, these are the humans to slide into the DMs 💌

Jeffrey C. Mogul
Western Research Laboratory
Compaq Computer Corporation
250 University Avenue
Palo Alto, California, 94305, U.S.A.

EMail: JeffMogul@acm.org
Phone: 1 650 617 3304 (email preferred)

Arthur van Hoff
Marimba, Inc.
440 Clyde Avenue
Mountain View, CA 94043

EMail: avh@marimba.com
Phone: 1 (650) 930 5283

legal stuff, but it's important so read it 🙈

Copyright (C) The Internet Society (2002). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement 🫶

funding for the RFC Editor function is currently provided by the Internet Society, so a big thank you to them for keeping the lights on 🌈✨

that's the whole spec, bestie! you made it through 🎉💖👑

📦 where this page comes from